testing techniques

Testing Techniques

Equivalence class analysis Cause effect graphs All pair testing

Equivalence classes

Triangle Problem

• Read 3 integer values in range [1..200]. • These 3 values represent the length of the

sides of a triangle. The programme displays a message which establishes that the triangle is isosceles, equilateral or scalene.

Triangle Problem• we are interested in 4 questions:

– Is it a triangle?– Is it an isosceles?– Is it a scalene?– Is it an equilateral?

• We may define the input test data by defining the equivalence class through “viewing” the 4 output groups:– input sides <a, b, c> do not form a triangle– input sides <a, b ,c> form an isosceles triangle– input sides <a, b, c> form a scalene triangle– input sides <a, b, c> form an equilateral triangle

Valid Classes

Non Valid Classes

Non Valid Classes

Could you find other non valid classes ?

Cause effect graph

Methodology

Triangle example

Triangle Problem

Cause Effect Graph

Decision Table

Test cases

All pairs testing

• Based on orthogonal Array• Drastically reduces number of test cases• A combination of worst cases and equivalence

class testing• Well supported in the agile methods

community• Supported by commercial and non-

commercial products

What Does All Pairs Do?

• Input is a set of equivalence classes for each variable.• Sometimes the equivalence classes are those used in

Boundary Value testing (min, min+, nominal, max-, max)• Output is a set of (partial) test cases that approximate an

orthogonal array, plus some pairing information.• Why partial? No expected outputs are provided. natural

enough, but a tester still needs to do this step.

All Pairs Assumptions

• Variables have clear equivalence classes.

• Variables are independent.

• Failures are the result of the interaction of a pair of variable values.

Example

• Suppose that we have three parameters.• For each parameter, there are two possible values.

– Values are :• A, B for parameter 1.• J, K for parameter 2.• Y, Z for parameter 3.

• Degree of interaction coverage is 2.– We want to cover all potential 2-way interactions

among parameter values.

Set of all possible test configurations

A J Y

A K Z

A K Y

A J Z

K YB

B ZJ

B YJ

K ZB

There are 23 = 8 possible test configurations.

Three parameters, twovalues for each.

Set of all possible degree 2interaction elements

• Coverage measure:– Percentage of interaction elements covered.

There are ( ) 2 2 = 12 possible interaction elements. 32

A J

B K

B J

A K

A Y

A Z

B Z

B Y

J Y

K Z

K Y

J Z

Test configurations assets of interactions

A J Y

A J

A Y

J Y

One test configuration...

… covers 3 possibleinteraction elements.

…using a subset ofall test configurations.

Interaction test coverage goal

Goal: cover all interactionelements…

A J Y

K YB

A K ZA K YA J Z

B ZJB YJ

K ZB

A J

B KB JA K

A YA Z

B ZB Y

J Y

K ZK YJ Z

A J Y

K YB

A K ZA K YA J Z

B ZJB YJ

K ZB

A J

B KB JA K

A YA Z

B ZB Y

J Y

K ZK YJ Z

Selection of test configurations forcoverage of interaction elements

Interaction elements Test configurations

A J A Y J YA J Y

Degree 2 coverage: 3 / 12 = 25%

Degree 3 coverage: 1 / 8 = 12.5%

A KA K

A J Y

K YB

A K ZA K YA J Z

B ZJB YJ

K ZB

A J

B KB J

A YA Z

B ZB Y

J Y

K ZK YJ Z



A K Z

A Z

K Z



A J Y

K YB

A K ZA K YA J Z

B ZJB YJ

K ZB

A J Y

A K Z

A J

B KB JA K

A YA Z

B ZB Y

J Y

K ZK YJ Z



B ZJ

B JB Z

J Z


Degree 3 coverage: 3 / 8 = 37.5%

A J Y

K YB

A K ZA K YA J Z

B ZJB YJ

K ZB

A J Y

A K Z

B ZJ

A J

B KB JA K

A YA Z

B ZB Y

J Y

K ZK YJ Z



B KB Y K Y

K YBDegree 2 coverage: 12 / 12 = 100%


Choosing the degree of coverage• In one experiment, covering 2 way interactions resulted in the following

average code coverage:– 93% block coverage.– 83% decision coverage.– 76% c-use coverage.– 73% p-use coverage.– Source: Cohen, et al, “The combinatorial design approach to

automatic test generation”, IEEE Software, Sept. 1996.• Another experience report investigating interactions among 2-4

components:– Dunietz, et al, “Applying design of experiments to software testing”,

Proc. Of ICSE ‘97.

Online Mortgage Application(Thanks, Bernie Berger, STAREast)

NY L 1 fam A+ Pri 80% Yes Yes Yes Cust Yes Yes

NJ M 2 fam A Vac 90% No No No Bank No No

FL H 3 fam A- Inv 100%

TX H+1 4 fam B

CA H+2 Coop <B

DC H+3 Condo

Other

Regi

on

tier

Prop

erty

Cred

it

Resid

ence

LTV

NIV

NAV

Refin

ance

Clos

ing

Cost

Intr

o Ra

te

Bank

Em

p

Twelve variables, with varying numbers of values, have7 x 6 x 6 x 5 x 3 x 3 x 2 x 2 x 2 x 2 x 2 x 2 = 725,760combinations of values.

“All Pairs does it in 50.” (Bernie Berger, STAREast2003)

Total number of combinations

1234

47484950

NYNYNJNJNY

OtherOtherOther

LMLM

H+1H

H+2H+1

1 fam2 fam2 fam1 fam

~Coop2 fam3 fam

Condo

A+A

A-BB

~B~A-<B

PriVacInvPri

~Inv~Vac~Pri

~Vac

80%90%

100%90%

~80%~100%~80%~90%

YesNo

YesNo

~No~No

~Yes~No

YesNoNo

Yes~No~No~No~No

YesNo

YesNo

~Yes~Yes~No

~Yes

CustBankBankCust

~Cust~Cust~Cust~Cust

YesNo

YesNo

~Yes~No~No~No

YesNoNo

Yes~Yes~No~No~No

666654461112

Selected All Pairs Test Cases (note fault resolution!)

Mortgage Application Observations

• The example uses values, not equivalence classes.• Values are independent,• System level testing (apparently).• The efficiency comes at the expense of fault isolation.

There are 2C12 = 12!/(2! x (12 - 2)!) = 66 pairs in the first row of the table. If that test case fails, which of the 66 pairs caused the failure?

• NB: excellent candidate for regression testing

What if the fault involves more than just a pair of variables?

pCn1

12

å =2^12 - 1=4095

pCn =n!

p! (n - p)!

Number of combinations of n things taken p at a time is

Worst case (for first test case):

testing techniques

Documents

set of equivalence classes

clear equivalence classes

possible interaction

degree of interaction

possible values

parameter values

input test data

set of partial test