the classification tree method with combinatorial …...test object 1 test object n test object 2...
TRANSCRIPT
![Page 1: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/1.jpg)
Integrating the classification tree method with combinatorial testing: test selection strategy
Eduardo Miranda, PhD.CMU/MSE
MSE/NIST SeminarJune 7th, 2011Pittsburgh, PA
![Page 2: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/2.jpg)
Test Object 1
Test Object n
Test Object 2
The Classification Tree Method illustrated
P Q
Test Object 1
a b c d e
a b c
Test all pairs = {bd, cd, ae, be}Test for b = {bd, be}
(2, AA, 34.5, F)
2, AA, 34.5, F, T,
32,BB,1,……..
ae
cd
cebe
bdad
ae
cd
cebe
bdad
e
d
Revealing subset
(2,*,*,F)
2011 © Eduardo Miranda 2
![Page 3: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/3.jpg)
Partition and boundary value analysis
• Two types of faults:– Computation faults: The wrong function is applied to some subdomain Si in the implementation
– Domain faults: The boundary between two subdomains Si, Sj in the implementation is wrong
• Partition testing– Test inputs are designed to find computation faults
• Boundary value analysis– BVA aims to find domain faults by using test inputs near the boundaries
R. Hierons, Avoiding Coincidental Correctness in Boundary Value Analysis, 2006
2011 © Eduardo Miranda 3
![Page 4: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/4.jpg)
How do we build confidence that a subset is revealing?• If it is possible to establish an order relationship on the members of a given subset we can use boundary value analysis– Quantities– Lengths– Collection of elements– Positions– Distance
• Test catalogues• Unless the subset has only one member, the absolute minimum number of values to test is 2
2011 © Eduardo Miranda 4
![Page 5: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/5.jpg)
Simple test catalog
Software Testing and Analysis—Process, Principles, and Techniques, M. Pezzè & M. Young, 2008
2011 © Eduardo Miranda 5
![Page 6: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/6.jpg)
At what level should we employ interaction testing?
• If we are using BVA to establish confidence in a subdomain:– Should we use the BVA values (left‐, left, middle, right, right +) in the interactions?
– Should we first establish confidence on the equivalence class and then test for interactions?
– What happens if we are not using BVA?
• What do we do with invalid values?
2011 © Eduardo Miranda 6
![Page 7: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/7.jpg)
To answer the previous questions we will look at three scenarios
Range for each aspect [1,999]
2011 © Eduardo Miranda 7
![Page 8: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/8.jpg)
Number of test cases required by different strategies (1)
0102030405060708090
100
0 5 10 15 20
Test cases
Number of aspects
1 equivalence class per aspect, all pairsBVA once each
Comb. Equiv. class
BVA once each +Comb. equiv. class
Nominal + BVA
Nominal + BVA +Comb. equiv. class
Comb. all
2011 © Eduardo Miranda 8
![Page 9: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/9.jpg)
Number of test cases required by different strategies (2)
0
20
40
60
80
100
120
140
0 5 10 15 20
Test cases
Number of aspects
2 equivalence classes per aspect, all pairsBVA once each
Comb. Equiv. class
BVA once each +Comb. equiv. class
Nominal + BVA
Nominal + BVA +Comb. equiv. class
Comb. all
2011 © Eduardo Miranda 9
![Page 10: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/10.jpg)
Number of test cases required by different strategies (3)
0
50
100
150
200
250
0 5 10 15 20
Test cases
Number of aspects
3 equivalence classes per aspect, all pairsBVA once each
Comb. Equiv. class
BVA once each +Comb. equiv. class
Nominal + BVA
Nominal + BVA +Comb. equiv. class
Comb. all
2011 © Eduardo Miranda 10
![Page 11: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/11.jpg)
Comparison of different strategies on the basis of fault finding effectiveness (1)
intended behavior1. X = “Don’t know”2. if T then
1. if 1<= V <= 99 then 1. x = “V1”
2. endif3. else
1. If 100 <= V <= 200 then1. x = “V2”
2. endif4. endif5. print (x)
programmed behavior1. X = “Don’t know”2. if T then
1. if 1<= V < 99 then 1. x = “V1”
2. endif3. else
1. If 100 <= V < 200 then1. x = “V2”
2. endif4. endif5. print (x)
Fault “A”
Fault “B”
2011 © Eduardo Miranda 11
![Page 12: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/12.jpg)
Comparison of different strategies on the basis of fault finding effectiveness (2)
Will not trigger fault “A”
Coverage will depend on the values generated
Will not trigger any fault
Will trigger both faults2011 © Eduardo Miranda 12
![Page 13: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/13.jpg)
What about negative testing
• Should we mix positive and negative testing?• Should we test for combinations of invalid values or it suffices with all singles testing?
2011 © Eduardo Miranda 13
![Page 14: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/14.jpg)
Fault masking
if V = “Invalid” thendisplay messageexit
endif..if z = z1 then
x := 6 / (u ‐ 4)endif
The fault is masked by the invalid value in row 4
All pairs array
2011 © Eduardo Miranda 14
![Page 15: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/15.jpg)
Exercising invalid combinations
if V1 is invalid thendisplay Error1exit
endifif V2 is invalid then
display Error2exit
endifIf V3 is invalid then
display Error3exit
endif
if V1 is invalid thenV1 = DefaultValue1
endifif V2 is invalid then
V2 = DefaultValue2endifif V3 is invalid then
V3 = DefaultValue3endif
Is V1 = invalid, V2 = invalid and V3 = invalid a good test case?
2011 © Eduardo Miranda 15
![Page 16: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/16.jpg)
Summary
• In testing you get what you pay for. Do not compare the cost of a method to the cost of not testing, because the latter will always require less test cases
• Combinatorial testing– Very good and efficient at finding computational faults– Very good, but expensive at finding domain problems. Still, what would
be a better approach is an open issue• Unless there is a fault model conjecture for doing something else:
– Keep negative testing separate from positive one– Don not combine invalid values
• To verify equivalence within a class– Combinations might act as confounders– For 1 or 2 equivalence classes within each aspect, combinatorial testing
is feasible and effective. For three or more, the number of test cases required becomes very large and all singles might be a good alternative
– One at a time testing will eliminate confounding but interactions might mask faults and will require a large number of test cases
2011 © Eduardo Miranda 16
![Page 17: the classification tree method with combinatorial …...Test Object 1 Test Object n Test Object 2 The Classification Tree Method illustrated P Q Test Object 1 a bc d e abc Test all](https://reader034.vdocuments.mx/reader034/viewer/2022050312/5f74a5716af3782c7735861a/html5/thumbnails/17.jpg)
Questions?
2011 © Eduardo Miranda 17