no struggle with test design (presentation at testexpo 2015 denmark)

No struggle with test design using the new practical grouping of

coverage types

Rik Marselis Copenhagen

29 January 2015

1

TMap HD

© Capgemini Sogeti

Question 1

Using Test Design Techniques is difficult…

Agree

Disagree

2

© Capgemini Sogeti

Question 2

Using Test Design Techniques is FUN…

Agree

Disagree

3

© Capgemini Sogeti

What Test Design Techniques do you know?

4

© Capgemini Sogeti

What Test Design Techniques do you know? TMap NEXT • Equivalence classes • Boundary value analysis • Decision points (CC, DC, CDC, MCDC, MCC)

• Orthogonal arrays • Pairwise testing • Data cycle test (CRUD) • Statistical usage (op.prof. • Right paths / Fault paths • Checklist • Decision table • Data Combination Test • Elementary comparison tst • Process Cycle test • Real-life test • Semantic test • Syntactic test • Use case test • Error Guessing • Exploratory testing • Reviews • Walkthroughs •  Inspections

5

ISTQB • Equivalence partitioning • Boundary value analysis • Decision table testing • Cause Effect graphing • State transition testing • Use case testing • User story testing • Structure based (SC, CC, DC, CDC, MCDC, MCC)

• Error Guessing • Exploratory testing • Orthogonal arrays • Pairwise testing • Domain analysis • Defect based techniques • Checklist • Combinatorial / Class. tree •  Informal review • Technical review • Walkthrough •  Inspection • Static analysis

Torbjorn Ryber • Exploratory testing (this is an approach, not a technique)

•  Inspection • Walkthrough • Technical review •  Informal review • Modelling • Equivalence partitions • Boundary values • Domain tests • Business process testing • Use case testing • State based testing • Decision tables & trees • Elementary comparison • Combinatorial testing • Data cycle testing • Syntax testing • Time cycle testing • Program Logic (SC, BC, PC)

• Data flow testing

Boris Beizer • Control-flow testing • Loop testing • Data-flow testing • Transaction-flow testing • Domain testing • Syntax testing • Finite-state testing

And more…

© Capgemini Sogeti

What techniques do you actually use?

What are your experiences with using the techniques?

How many techniques do you need in your day-to-day work?

6

© Capgemini Sogeti

Do you have difficulty selecting techniques?

So this is what TMap HD will help you with.

TMap HD has 5 elements:

7

People

Integrate

Simplify

Industrialize

Confidence

Skills: Creative, Curious, Social, IT, Testing, etc.

© Capgemini Sogeti

Simplify

To help you in the selection of test design techniques: •  Two approaches

•  These can be combined

•  Focus on the type of problem you are solving

8

© Capgemini Sogeti

Two approaches and their subdivision

9

Neil’s Quest for Quality: Page 86, Building Block 9

Next presentation by Gitte Ottosen

© Capgemini Sogeti

Coverage based testing

•  Why coverage? –  To be sure you haven’t forgotten something important

•  Four groups of coverage

–  Process

–  Conditions

– Data

–  Appearance

FIRST STEP: Decide what type of problem you’re solving

and what type of risk you’re covering

10

© Capgemini Sogeti

Example: Price of travel in public transport

11

< 4 year OR

soldier Free Travel

Start

End

> 4 year AND < 18 OR > 65 OR card

50% off

100%

Yes

Yes

No

No

© Capgemini Sogeti


•  How is this functional specification presented?

•  Indeed: a process flow

•  So what test design technique will we use?

•  Indeed: Process Cycle Test

•  Or would you choose another? If so, which one?

12

© Capgemini Sogeti 13

Coverage group example 1

Process

Process Cycle Test

© Capgemini Sogeti

Example PCT: Price of travel in public transport

14

< 4 year OR

soldier Free Travel

Start

End


50% off

100%

Yes

Yes

No

No

1

2

3

4

5

TM – 1 (every path): Testcase a: 1,2 Testcase b: 1,3,4 Testcase c: 1,3,5

A

B

TM – 2 (every pathcombi.)

A: 1-2, 1-3 B: 3-4, 3-5

Testcase a: 1-2 Testcase b: 1-3 & 3-4 Testcase c: 1-3 & 3-5

(so, in this simple example, no difference, both 3 testcases)

Process Cycle Test

© Capgemini Sogeti

Example PCT: Price of travel in public transport

15

< 4 year OR

soldier Free Travel

Start

End


50% off

100%

Yes

Yes

No

No

1

2

3

4

5

TM – 1 (every path):

Also guarantees:

-  Statement coverage *

-  Decision coverage

A

B

*) Of all statements as shown in the graph

BUT:

ARE YOU CONFIDENT THIS IS GOOD ENOUGH COVERAGE?

© Capgemini Sogeti


•  Process cycle test nicely covers the process but doesn’t cover the combined conditions very well.

•  So what other test design technique will we use?

•  Let’s see what would be the result for these techniques: -  Decision table -  Elementary Comparison Test -  Data Combination Test

16

Remember: decide what is your problem / risk



Conditions

Decision Table Test

© Capgemini Sogeti

Decision table “condensed”

18

Age < 4 Y N N N N N

Age >= 4 and < 18 N - Y N N N

Age >= 18 and < 65 N - N N Y Y

Soldier - Y N N N N

Card - - N N Y N

0% (free) X X

50% X X X

100% X

This is the way ISTQB Foundation teaches us


Decision table “complete” (MCC)

Decision table “collapsed for impossible ages”

32 test cases, of which 16 not possible

20 test cases, of which 4 not possible

This is the way TMap NEXT describes


Decision table “collapsed for irrelevant card”

Decision table “collapsed for irrelevant soldier” 12 test cases

13 test cases

Remaining: 8 test cases with “normal” outcome (compare these with results of other techniques)

Beware: Only collapse if this

doesn’t increase your risk to an

unacceptable level



Conditions

Elementary Comparison Test

© Capgemini Sogeti

ECT & MCDC

22

Steps in Elementary Comparison Test: 1.  Identify the decision points 2.  Determine test situations per decision point, using MCDC

(MCDC = Modified Decision Coverage) 3.  Create Logical Test Cases 4.  Create Physical Test Cases 5.  Create Test Script

D1

D2

© Capgemini Sogeti

MCDC with 2 conditions

23

FALSE (0) = go to D2

A B

TRUE (1) = 0%

A B

(A) < 4 year

(B) soldier

1 .

. 1

0 .

. 0

D1 = < 4 year OR soldier

0 Neutral value

0

0

0

0

D1.1

D1.2

D1.3

STEP 2: D1.1, D1.2 and D1.3 are our Test Situations

© Capgemini Sogeti

MCDC with 4 conditions

24

FALSE (0) = 100 %

A B C D

TRUE (1) = 50 %

A B C D

(A) > 4

(B) < 18

1 . . .

. 1 . .

. . 1 .

. . . 1

0 . . .

. 0 . .

. . 0 .

. . . 0

D2 = > 4 AND < 18 OR > 65 OR CARD

(C) > 65

Neutral values (AND = 1, OR = 0 )

(D) CARD

1 0 0

1 0 0

1 0 0

1 0 0

1 0 0

1 0 0

1 0 0

1 0 0

1 0 0

D2.1

D2.2

D2.3

D2.4

D2.5

© Capgemini Sogeti

ECT & MCDC Graph

25

D1: < 4 year OR

soldier

D2: > 4 year AND < 18

OR > 65 OR CARD

END

D1.1 D1.2

D1.3

D2.1 D2.2

D2.3 D2.4 D2.5

STEP 3: Create Logical Test Cases, Sub-step 1: create graph

© Capgemini Sogeti

ECT & MCDC Testcases

26

D1.3 : > 4 YEAR, NOT soldier D2.4 : < 4 YEAR, < 18 YEAR, < 65 YEAR, NO card

This is a GOOD Logical Testcase IMPOSSIBLE Physical Testcase

So we have 7 Logical Testcases And 6 Physical Testcases

STEP 3: Create Logical Test Cases, Sub-step 2: combine test situations STEP 4 and 5: Create Physical Test Cases and Test Scripts



Data

Data Combination Test

© Capgemini Sogeti


28

3 data elements: -  Age 4 values ( < 4, > 4 & < 18, > 18 & < 65, > 65 ) -  Soldier 2 values (YES & NO) -  Card 2 values (YES & NO)

Draw a classification tree: Price of travel

Age Soldier Card

< 4 > 4 & < 18 > 18 & < 65 > 65 YES NO YES NO

© Capgemini Sogeti


29

Create test cases with coverage: Every value of each data item

TC1

TC2

TC3

TC4

0%

0%

100%

0%

© Capgemini Sogeti


30

Create test cases with coverage: Pairwise (every combi. of 2 data items)

TC1 TC2 TC3 TC4 TC5 TC6 TC7 TC8

0% 0% 100%

0%

0%

50%

50% 0%

© Capgemini Sogeti

50%


31

Create test cases with coverage: Pairwise (every combi. of 2 data items)


0% 0%

100%

0%

0%

50%

50% 0%

0%

© Capgemini Sogeti

50%


32

Allthough you now have all combinations of all data-values, you still can’t guarantee that all input values have triggered outputs


0%

100%

0%

0%

0%

50% 0%


Conclusion

© Capgemini Sogeti

Let’s compare the achieved coverage

•  PCT 3 test cases; •  DT 6 test cases with condensed table or 32 with

MCC, of which 16 are possible or 8 + 4 with collapsed table;

•  ECT 7 logical / 6 physical test cases with MCDC; •  DCoT 4 (all values) or 8 (all pairs) test cases;

Which one gives good coverage? That depends on your assessment of risk and

“problem”

34

© Capgemini Sogeti

Each coverage type serves one or more goals

•  Process test relatively light technique but with guaranteed coverage of process paths

•  Decision table (MCC, optimised for “not possible”) heavy technique, covers conditions but also all feasible paths and most data combinations

•  Elementary comparison test medium technique that covers all important combinations of conditions and all process paths

•  Data combination test, all values quick technique

•  Data combination test, all pairs medium technique, good when focus is on data

35

© Capgemini Sogeti

Beware: Don’t overdo it !!

•  Lesson1: choose smart, based on the real problem you are facing, and save time by knowing what other coverage(s) you achieve as well.

•  Lesson 2: be aware that the obvious technique may not be the best. (PCT is not very effective, DT-MCC is not very efficient)

•  Experience based testing should be combined with coverage based testing: –  When you for example come across some combined conditions,

do a quick MCDC –  When you see some boundary, do boundary value analysis –  When you have a process, cover every path

•  This way you will achieve both effective and efficient testing without an overload of activities

36

© Capgemini Sogeti

So why do we call this “SIMPLIFY”?

37

1. Determine your problem

2. Decide on the approach

3. Decide on the technique(s)

Don’t always just apply the same technique (or only error guessing)

Know how coverage-rates of techniques compare / overlap

And thus: Do effective and efficient testing with a

minimum of effort

© Capgemini Sogeti

Wanna know more? Read our article in Testing Experience

38

Or join one of our training courses

Download: www.capgeminisogeti.dk/TVCT

Use test approaches & test design techniques

to your advantage

And have fun !!

© Capgemini Sogeti

Rik Marselis

40

Management Consultant Quality & Testing at Over 30 years IT experience, of which > 15 years quality & testing

Advisor, process-improver & coach at many organizations Prince2 Practitioner, CMMI and CISA Trainer for various courses, e.g. Agile testing

TMap, TPI en ISTQB accredited

Research Author of various books & articles Fellow of SogetiLabs, Speaker at various conferences

And next to this:

Chairman (association of testers, 1700 members)

@rikmarselis

no struggle with test design (presentation at testexpo 2015 denmark)

Technology