putting models at the heart of testing

Intelligent Definition and Assurance 2-Nov-15

© 2013 Gerrard Consulting Limited 1

Putting Models at the Heart of Testing

Paul Gerrard

Gerrard Consulting

New Model for TestingPutting Models at the Heart of Testing

Twitte

r: @pau

l_ge

rrard

Paul [email protected]

gerrardconsulting.com



Helping clients transform their testing through

INNOVATION, COACHING and LEADERSHIP

Our CLIENTS

– Want to be agile rather than follow Agile dogma

– Have a pragmatic approach and are focused on delivery

– Want a solution that fits, not a badly fitting suit.

Agenda

• Motivation

• A New Model for Testing

• Exploratory Testing

• Some Simple Model Examples

• The Scope of Models

• Models for Code

• Characteristics of Test Models

• Using Test Models

• Traceability, Models of Models

• Close

Intelligent Definition and Assurance Slide 4



Introduce Yourselves

And think of a Team Name

Motivation

Why a workshop on test models?




• All testing is based on models

• Trainers and testers focus on standard techniques

• Techniques are called test DESIGN techniques, but really, they are clerical processes

• Testers think techniques are ‘advanced’ (and don’t use them anyway)

• Testers believe they can’t invent new techniques because they don’t understand concept of modelling

• Most tests are still guessed, not designed.


Teaching test techniques

Example technique – boundary value analysis

1. For input (or output or hidden) domains that are defined as ranges, each treated by different rules

2. Identify the ranges and the extreme boundaries of each one

3. Use each of these boundary values as test values

4. Use the appropriate rule to define the outcome.

Training exercise:“Using this prepared requirement fragment (that happens to fit

technique X - sort of), create tests using technique X”




Test technique usage

• Certified courses place an emphasis on the classic test

techniques

• Everybody knows them

• But hardly anyone uses them (in a systematic way, at least)

• When testers encounter ‘real’ (i.e. messy incomplete)

requirements they guess what technique might be useful.

Maybe.


The “Transfer Problem”

• In the Domain Testing Workbook*, the authors comment on student performance in a final exam which contained:– Questions that were similar to those the students had already experienced

in class and homework assignments

– One question that required students to combine skills and knowledge in a way that was mentioned in lectures, but they had not practiced

• Almost every student handled the first type of question very well, but every student failed the more challenging question

• The “transfer problem” relates to how a student applies the knowledge they gain in class to different situations or the “real-world”.

Intelligent Definition and Assurance

* Domain Testing Workbook, http://contextdrivenpress.com/Slide 10

http://contextdrivenpress.com/



Teaching test models should help

• Real life and requirements are messy and large

– Not simple and tiny

• Requirements don't usually tell you which technique to use

• We need to select techniques to fit the requirement

• Modelling will help us to simplify the problem, and select

techniques (or invent new ones).


A New Model for Testing



Forget Logistics(for the time being)

Document or not?

Automated or manual?

Agile v waterfall?

This business or that business?

This technology v that technology?

ALL Testing is Exploratory

We explore sources of knowledge ...

... to build test models ...

... that inform our testing.



All Testing is Based on Models

Models Pub Quiz



Models are innate, essential, human


Judgement, exploring and testing

Testing(the system)

Our model(s) are adequate

Our model(s) are not adequate

Exploring (sources)

Judgement

Creates testmodels

Uses testmodels

We explore sources of knowledge to build test models that inform our testing

BTW – Do Developers explore the same way? I think so.




Exploration

ExplorationDefinitions specs/stories

People(& you)

Sources

Require-ments

TestModels

Enquiring

Challenging

Sources:People, documents,experience, system under test

Modelling

Test Models:Can be documented

or mental models

Predicting

System under test


Testing

TestingSystem

Under Test

Refining

Informing Applying

Interpreting

TestModels

Revise the System

More exploring Reporting




New Model Testing

29 page paper: http://dev.sp.qa/download/newModel


Capabilities

Enquiring, Modelling, Predicting, Challenging

Informing, Applying, Interpreting, Refining

Reporting and Logging

http://dev.sp.qa/download/newModel



• Analysis, enquiry and elicitation

• Modelling

• Creation of custom models, using

heuristics, guesses, brainstorming,

ideation, creative thinking

• Custom test design techniques

• Comparison of models, value, advantages,

disadvantages, compromises

• Identification, validation and use of

oracles

• Predicate logic and proof

• Hypothesis and inference

• Socratic method

• Rapid Review and Inspection techniques

• Test case design

• Test models and the meaning of coverage

• Testing as controlled experiment

• Observation, Note taking, recording

A very different skillset• Basic data analysis and statistics

• Decision-making with incomplete data

• Computer forensics

• Fault tree analysis

• Failure diagnosis

• Bug advocacy, triage processes and

negotiation

• Meaningful software and test metrics

• Visual presentation of data

• Reporting and presentation skills

• Understanding stakeholders

• Test analytics

• Risk management, risk-based testing and

decision-making

• Critical Thinking

• Interpersonal skills

• Dealing with uncertainty/fallibility


Testing Career Development(speculative)

Foundations

Technical

Management

Strategic

Test Strategy Project Intelligence

Test Assurance

Exploration Forensics Interpretation

Scripting/Programming

Test Automation Technical (Excel, SQL, OS utils etc)

Stakeholder management

Analytics & visualisation

Managing uncertainty

Critical Thinking

ISTQB etc...

Supplier Management

Test Process Management

Methodology




Fallible Sources of Knowledge

Our sources of knowledge are fallible and

incomplete

• Testers depend on sources of knowledge to test

– Written documentation, specs, standards

– Verbally communicated knowledge and advice

– Experience of using similar systems

– Experience of using the current business process or system (that is to be replaced)

– Intelligence, intuition, perception, beliefs, prejudices etc

• All of our sources of knowledge are fallible and incomplete.




Evolution

• Our brains are wonderfully adept at interpreting fuzzy data for

the purpose of catching mammoths for dinner

• We are much less good at communicating in a formal, precise

way and interpreting what we perceive without error

• Eliciting information, capturing, interpreting, communicating and

acting on it are all highly error-prone

• These errors are the cause of the defects testers seek to

prevent or detect.


How testers gather knowledge

• We question people, documents, the system under test to

understand what is required

• We challenge people, documents, the system to check whether

we understand and trust that knowledge

• We EXPLORE our sources to acquire the knowledge required

to test

• In this respect, ALL testing is exploratory.




'Exploratory' Testing

What if our source of knowledge is mostly the

system?

• Exploratory testing is the approach to use when there is little

documentation

• In our model of testing our ‘source’ is also the ‘system under

test’

• So we need to merge our source of knowledge and system

under test into one element on the diagram.




"Exploratory" testing

Testing System Under Test

RefineModel

Applytests

Interpretoutcome

TestModels

Exploration

Enquiry

Challenge

Imaginemodel

MakePredictions

Informtests


The first test application

• Some tiny test applications to get you in the mood…

• http://dev.sp.qa/workshop

• There are two test application on the menu

– Example 1

– Example 2

• Let’s look at “Example 1”

• DON’T CLICK SUBMIT YET!


http://dev.sp.qa/workshop



Exercise

• Explore the form that has just the input field and submit button and work out how it works

• As you proceed, write down your ideas of how it seems to work

• Pause every three clicks

• Watch your model evolve

• You can only click “Submit” twenty times

• How many 'clicks' to figure out how it works?


Exploration process

ExplorationTest

Models

Enquiring

Challenging

Modelling

Predicting

System under test

1. We ask questions of the system to see what happens.

(We don't know yet)

4. We compare the behaviour of the system with our predictions.

(Is our model correct?Is our source correct?)

2. We build a mental model of the system from our findings

3. We use our model to make predictions

(The system should do ....)Intelligent Definition and Assurance Slide 34



You have explored and perhaps

you know how it works

But have you tested it?

Now, the testing begins, doesn't it?

Explore this and describe it?

You explore, and know what it is

"But I wanted Paella!"




Exploration process

ExplorationDefinitions

specs/stories

People(& you)

Sources

Require-ments

TestModels

Enquiring

Challenging

Sources:People, documents,experience, system under test

Modelling

Test Models:Can be documented

or mental models

Predicting

System under test


Exploring a system as a the only source of

knowledge

• Firstly – it's impossible to ignore:

– You have your own knowledge and experience

– You know of conventions, standards

– You were asked to test for a reason, aren't you?

– When is there ever no written documentation?

• If you ignore these sources

– The system does what you have seen it do

– Your model (a reverse engineered spec/requirements) is based on this evidence

– So how could you ever find a bug?

No!




Explore this?


Exploration of (complex) systems

• We normally have multiple sources of knowledge– Stakeholders, users, customers, advisors

– Designers, architects, developers

– Our own experience and knowledge, as testers

– and we *may* have the system too

• We use all our sources, select, use critical thinking to derive:– What is required

– Oracles (how it is expected to behave)

• The need and oracles come from 'any of the above' except the system itself.




Some Simple Model Examples

Example: Requirements Document

• What element types

can you seen?

• What can you

count?

•

•

•

•

•

1.1 _________________________________________________________________________________________________________________________

1.2 ____________________________________________________________________________________________________________________________________________

1.3 ______________________________________________________________________________________________________




Flowchart

• What element types

can you see?

• What can you count?

•

•

•

•


Models and coverage

• Models have items that can be identified and counted

• If a coverage item can be objectively defined, we could use that measure to set targets and measure progress (are we done?)

– Decisions, boundaries, paths, states, transitions, reconciliations, outcomes

– Two testers using the same model would derive similar tests

• If a coverage item is subjective, e.g. risks, then

– We can't measure coverage in any meaningful way

– Different testers would identify different risks, different tests.




“all models are wrong, but some

are useful”

George Box, Statistician

The Scope of Models



Test design is based on models

• Test Analysis: we create models of the environment, the

system, usage, human nature, goals, risks and the tests

themselves

• Test design: we select from the infinite number possible, the

tests that we believe will be most valuable to our stakeholders

• Our test model helps us to select tests in a systematic way.


We model and test systems

• A system could be a toothbrush

or an airport

• The system could be a class

method or the entire

business process

• Different sub-systems can be

a hierarchy or ...


Class

Sub-system

System

Process

Business Unit

Method

Scope as a hierarchy

Slide 48



A system of systems


Scope as clustersSlide 49

Scope as a perspective




Setting scope in familiar ways

• Scope = “features that...”

– ...are used in a path through a business process

– ...share a common aspect, e.g. sales data (orders, sales ledger, stock...)

– ...depend on a particular infrastructure component

– ...are used by the same user group

• The states and valid transitions of an object

• Objects that interact on a single web page

• The range of values of a numeric variable

• Etc. etc. etc.


When we select items in scope – we deselect

or ignore other items

We choose models to

simplify situations, but by

excluding important items,

do we make a model invalid?

“Everything should be

made as simple as

possible, but no

simpler.”

Albert Einstein (allegedly)




The second test application

• Some tiny test applications to get you in the mood…

• http://dev.sp.qa/workshop

• Let’s look at “Example 2”

• Try working in PAIRs?


Models for Code

http://dev.sp.qa/workshop



Models for code

• Treat the code as blocks of sequential statements delineated by branches/decisions and entry points

• The path of execution follows the branches

• We want to know

– Are the right decisions taken at the right time?

– Is the flow of control correct?

• Coverage:

– We test all the code blocks (statement testing)

– We test all the branches (branch/decision testing.


Some Python code...

• Note to PG: Use the 'Deep Black' theme


dictutils.py



A model for code

• How many branch outcomes are in this control flowgraph?

• Path 1: A, B, C, E, D, C, E, FCoverage= 66%, 75%, 87.5%, 100%

• Path II: A, B, C, E, D,FCoverage = 50%, 62.5%, 75%, 100%

• Path 1 + Path IICoverage= 62.5%, 75%, 83.3%, 100%

• Which branches are left to cover?

• How many more test cases are required?

A

C

D

E

B

F


Another, different model for code

• Definition-use – Data Flow Analysis

• We look for sequences of:

– Definition (of variables)

– Usage (in calculations, assignments, decisions)

– Kill (deletion, release of memory)

• We model ‘the flow of use of data’ rather than the flow of control

• Coverage: Definition-Use pairs:

– dd, du, dk, ud, uu, uk, kd, ku, kk

– Some are valid, some are not.




Data flow


a = 3

...

if a > 10 then

doThis()

else

doThat()

end if

...

c = a + b

...

a = None

Definition - d

Predicate use – u

Calculation use - u

Kill - k

Slide 59

Data flow analysis - pairsdd

du

dk

ud

uu

uk

kd

ku

kkIntelligent Definition and Assurance

wrong

normal

suspicious

wrong

normal

normal

normal

wrong

suspicious

• We use tools to

scan the code and

report these

anomalies

• A 'static analysis'

techniques

Slide 60



Disadvantages of these models?

• Control flow graphs/branch coverage

–

–

–

• Data Flow/definition-use pairs

–

–

–


Characteristics of Test Models



Characteristics of test models

value

coverage format

perspectivescope

Let’s look at each aspect in turn...


Scope

• Scope sets the boundary of our model

• What is included and what is not

• ‘Draws a line’ around the elements that comprise the system scope

• Examples:– Life histories of customer orders, assets, stock items, airline flights or

other entity

– Business or system processes or flows or use-cases

– Menu trees, user types or personas

– Data tables, columns or patterns

– Permutations of a selected input values.




Using models to define, reduce and limit scope

• How do you test a very large system?

• Think of ways to model a small, testable subset of Microsoft

Word (or other WP)

• Here are a couple of example I thought of

• Now, you draw a picture that limits the scope of your subset

(or sub-system)


Long composition sequences, undos, redos




Inserting text and repagination


Ways of testing Word

Create a model that represents each of the following:

1. A workflow or sequence of actions that a user might perform on a regular basis

2. A series of operations on an object in a document (e.g. table, image, a paragraph, object, drawing...)

3. An interaction with the operating or file system.

4. An interaction with another device.




Different perspectives

• Stakeholder perspective

– What do users see, use or regard as important?

– Business or user processes, goals, concerns/risks?

– Usage patterns, failure modes, reliability, performance

– What do architects, designers, operations, developers regard as

important?


Different perspectives 2

• System perspective

– Structure and design – from control-flow, data flow in code

– Flow of control and data between architectural elements

– Consistency of use of data

– Security, authority and scope of influence of function.





Examples: ATM Requirements

• An ATM has four functions:

– Accept card and validate PIN

– Request a cash withdrawal

– Request an online bank balance

– Request a paper balance to be posted to my address

• What could possibly go wrong?




We are the bank that likes to say,

“We don’t care”

Now, we are the Customer of

the bank



Formats of models

• Inventories, lists, tables– Risk or hazard registers, checklists, decision tables, data item validations, attacks, error-guesses

• Flows, journeys or tours– Business or system processes and paths, life histories, code control flow, user journeys

• Business/User Stories– Model requirements as a set of features with scenarios

• Networks of systems, sub-systems, features...– Collaborations, state models, messaging, reconciliations

• Analytics– Use of language, usage of features, patterns in data input, patterns in outputs

• Perspectives– Roles, personas, stakeholder positions

• Can you think of others?


Hazard – risk – impact model

Hazard Failure mode Impact

Untrained users are entering poor quality data

? ?

?The system cannot respond quickly enough for users

?

? ?We can’t collect money from online customers

What might

trigger a

failure?

What might

fail?

What’s the

problem?




Story HeaderFeature: ship orders

As a orders clerk

I want to acknowledge and ship the order

So that we fulfil a book order

Scenario: ship a single book from stock

Given I select a valid order

And the ordered book is in stock

When I choose ‘acknowledge and ship’

Then order status is changed to ‘shipped’

And an address label is printed

Modelling features using structured stories


Key word

Story text

Each Story has multiple Scenarios

Scenarios can be data driven

http://businessstorymethod.com

Anatomy of a business story header (the

feature description)Feature A label for the FACILITY a user needs to support

their business objective

As a… The ROLE of the user who needs to achieve the goal

I want (to)… The TASK the user wants to perform with the help of the system

So that… The GOAL of the user

• Note that roles can sometimes vary, but it is often better to reference ‘personas’ that have multiple roles.

• Personas could be “18 year old male gamer” or “65 year old female retired nursery school teacher” for

example.

• The story brings together these aspects so we can view the feature from

different viewpoint to explore it


http://businessstorymethod.com/



Anatomy of a scenario

Given… Pre-conditions that define the state of the system or scenario when the user performs the task

When… The values, inputs, commands or actions executed by the user to complete the task

Then… The outcome of the task (given the preconditions and the user inputs, actions etc.)

• The parallel with test cases is obvious:

• given=precondition(s), when=steps, then=outcome(s)

• A scenario maps directly to a test case – but we haven’t used the

word test yet.

• If I do – stop me.


Stories may have many scenariosFeature: Ship an Order

In order to fulfil a book order

As a orders clerk

I want to acknowledge and ship the order

Scenario: ship a single book from stock


And the ordered book is in stock

When I choose ‘acknowledge and ship’

Then order status is changed to ‘shipped’

And an address label is printed

Scenario: advise a book is out of stock


And the ordered book is out of stock

When I choose ‘message the purchaser’

Then Enter message to purchaser advising the order status

And an email is sent to the purchasers email address

Scenario: advise an item is discontinued


And the ordered book is discontinued

……

Etc. etc.




Value

• Value to who? The stakeholder

• A model has value to a stakeholder if it is meaningful to them:

– Business processes, outcomes processes and paths have meaning to

business stakeholders

– Collaboration diagrams, state-models, architectural designs have

meaning to designers, architects

– Control-flowgraphs, decision tables, statement coverage have meaning

to programmers.


Coverage - revisited

• Models usually have something that can be objectively

identified and counted

• If a coverage item can be defined, we can use them to set

targets and measure progress

– Decisions, boundaries, paths, states, transitions, reconciliations,

outcomes

• If coverage item is subjective, e.g. risks, then one can only use

them in a check list.




Working the model

• The model exposes opportunities for tests

– These are the test or coverage items

– Choose tests to exercise each coverage item in turn

• If the model is blobs and lines we trace paths

• Lists – we use as a checklist

• Stories – each scenario becomes a test case

• Progress can be monitored in terms of a model that has

meaning to... some stakeholder.


Exercise Choice

• Model a maze-solving robot

• Define tests for a search engine

• Test the triangle program.




Testing Triangles

How many tests do you think you need to cover

this requirement?

Triangle Testing

Charter?next slide




Specification

1. The user will enter the lengths of three sides of a triangle.

2. Given this data, the program will determine whether a triangle can be formed from the three sides.

3. For valid data, the program will determine whether the triangle is:a) equilateral (all sides of equal length)

b) isosceles (two sides equal)

c) scalene (all sides unequal)

4. The program will calculate the area of the triangle to an accuracy of four significant digits using the following formulae:

a) S = (a+b+c)/2 where a, b, c are the side lengths – this is half the perimeter.

b) Area = SQRT(S*(S-a)*(S-b)*(S-c))

5. If the triangle has a right angle (a “right triangle”), the program will report this.

6. If the triangle has all sides of integer values and has a right angle, it is a “perfect right triangle”• E.g. a triangle with side lengths of 5, 12 and 13 is a perfect right triangle

• The triangle obeys Pythagoras theorem (52 + 122 = 132) for integer values.


Using Test Models



Working with models

• Inventories, lists, flows, journeys, networks, analytics

• A model the tester creates or could be an existing model

provided in a spec

• Some models are documented

• Some are mental models that guide the tester in their

exploration of a system.


We use test models to…

• Simplify the context of the test

– Irrelevant or negligible details are ignored in the model

• Focus attention on selected aspect of the system

• Generate a set of unique tests (within the context of the

model) that are diverse (with respect to that model)

• Perhaps also enable test coverage to be planned, monitored

and evaluated.




Sources of models

• The test basis: we analyse the text or diagrams or information that describe required behaviour

• Stakeholders: interviews, discussions, challenges

• The architecture of the system: we identify testable items in its user-interface, structure or internal design.

• Modes of failure: we identify potential ways in which the system might fail that are of concern to stakeholders.

• Usage patterns: we focus on the way the system will be used, operated and interacted with in a business context.

• Everything looks fine – doesn’t it?


Models over-simplify (they have to)

• Models incorporate implicit assumptions and are approximate representations

• They are heuristic, useful in some situations, always incomplete and fallible

• Before we adopt a model, we need to know:

– What aspects of the behaviour, design, modes of failure or usage the model helps us to identify

– What assumptions and simplifications it includes (explicitly or implicitly).




Example model

• Think of a model for this

form

• Flowchart, boundary values,

decision table would be

good examples

• Now think of THREE things

that the model IGNORES.


Formality

• Formal test models

– Derived from analyses of requirements or code

– Quantitative coverage measure can be obtained from a formal model

• Informal test models

– E.g. some models are just lists of modes of failure, risks or vulnerabilities.

– Informal models cannot be used to define quantitative coverage measures

• Ad-hoc models

– Some models can be ad-hoc, invented by the tester just before or even

during testing.




Stakeholders and test models

• Stakeholders may not tell testers to use specific test models

• But to understand coverage and traceability, they need to

understand your models

• The challenge:

– Stakeholders may be of the opinion that the models you propose

generate too few tests to be meaningful or too many to be economic

– We need to engage stakeholders and ask them to help us to build

our models.


Traceability



Traceability

• Got requirements?

– An overarching view of how the 'flow' from requirements through to

tests, test outcomes, changes and acceptance are evidenced and

auditable (in both directions)

• No requirements?

– Agile? Exploratory environment?

– We need a more general definition.


Traceability – more general

• Sources of knowledge identified

• Test Models identified, terms of use understood

• Provenance of tests can be explained

• Test (and re-tests) run, outcomes, incidents can be linked.




Questions on traceability

• Does traceability ‘matter’ in your organisation?

• How do you connect/associate your sources of knowledge

through to executable tests?

• Traceability: in one direction only?

• Traceability: a one off – changes cannot be traced?

• Can/do you use traceability to perform impact analyses of

proposed changes?


Models of Models

Speculative



A model of models

User model office, business simulation

Focus groupUsability

Programmer ad-hoc testing

Business workflowsEnd to end testing

Reconciliation-based

Automated Unit testingAutomated integration testing

Formal methods‘Model-based’ test automation

Usage

relatedTechnology

related

Informal

Formal

Exploratory

Through UIExploratory

Through API

Scripted manual Developer test

automation


For another day ...

identifying

articulating

selecting

evaluating

discarding

covering

exploiting




Close

New Model Testing

29 page paper: http://dev.sp.qa/download/newModel


http://dev.sp.qa/download/newModel




• All testing is based on models but we don’t teach them enough

• Testers should use models naturally

– To select the right test technique

– To invent new techniques of their own

• Models aren’t documents, bureaucratic

• Modelling is the creative part of testing

• With models, tests are designed, not guessed.


Finally...

• If you want the slides

– Drop me an email

– Connect with me on LinkedIn

– Follow me on Twitter

• Register on:

– gerrardconsulting.com

– tkbase.com




New Model for TestingPutting Models at the Heart of Testing

Twitte

r: @pau

l_ge

rrard

Paul [email protected]

gerrardconsulting.com

putting models at the heart of testing

Documents