chapter 5 retrospective on functional testing 322 235 software testing by wararat...
TRANSCRIPT
Chapter 5 Retrospective on Functional Testing
322 235 Software Testing
ByWararat Songpan(Rungworawut),PH.D.
Department of Computer Science, Faculty of Science, Khon Kaen University
1322 235 การทดสอบซอฟต์�แวร�
Functional testing or Black Box
• We have seen 3 main types of functional testing, BVT, EC, and DT.• Boundary Value Testing (BVT):
o Boundary Value Analysis (BVA) – 5 extreme values – single assumptiono Robustness Testing (RT) – 7 extreme values – single assumptiono Worst-case Testing (WT) – 5 extreme values – Multiple assumptiono Robust Worst-case Testing (RWT) - 7 extreme values – Multiple
assumption
• Equivalence Class Testing (EC): o Weak Normal (WN) - valid EC(try no repeat) -> single assumptiono Strong Normal (SN) - valid EC (Every) -> multiple assumptiono Weak Robust (WR) - valid & Invalid EC -> single assumptiono Strong Robust (SR) - valid & Invalid EC-> multiple assumption
• Decision Table-based Testing (DT): o It finds out of some test cases that do not make sense
(Impossible test cases)
322 235 การทดสอบซอฟต์�แวร� 2
Test effort (1)
322 235 การทดสอบซอฟต์�แวร� 3
high
low
Number of Test Cases
Sophistication
Boundaryvalue
Equivalenceclass
Decisiontable
• Look at questions related to testing effort, testing efficiency, and testing effectiveness between 3 techniques.
Test effort (1)
322 235 การทดสอบซอฟต์�แวร� 4
high
low
Effort to Identify Test Cases
Boundaryvalue
Equivalenceclass
Decisiontable
Sophistication
Test Identification Effort VS Test Execution
Effort (BVT)
322 235 การทดสอบซอฟต์�แวร� 5
high
low
Number of Test Cases
SophisticationBoundary
valueEquivalence
classDecision
table
high
low
Effort to Identify Test Cases
Boundaryvalue
Equivalenceclass
Decisiontable
Sophistication
BVA: They are mechanical on generating test cases and therefore are easy to automate.
Test Identification Effort VS Test Execution
Effort (EC)
322 235 การทดสอบซอฟต์�แวร� 6
high
low
Number of Test Cases
SophisticationBoundary
valueEquivalence
classDecision
table
high
low
Effort to Identify Test Cases
Boundaryvalue
Equivalenceclass
Decisiontable
Sophistication
EC: The equivalence class techniques take into account data dependencies and the program logic however, more thought and care is required to define the equivalence relation, partition the domain, and identify the test cases.
Test Identification Effort VS Test Execution
Effort (EC)
322 235 การทดสอบซอฟต์�แวร� 7
high
low
Number of Test Cases
SophisticationBoundary
valueEquivalence
classDecision
table
high
low
Effort to Identify Test Cases
Boundaryvalue
Equivalenceclass
Decisiontable
Sophistication
DT: The decision table technique is the most sophisticated, because it requires that we consider both data and logical dependencies.
8
Functional Test Cases: BVA (15 TCs)Test case ID a b c Expected Results
1 100 100 1 Isosceles
2 100 100 2 Isosceles
3 100 100 100 Equilateral
4 100 100 199 Isosceles
5 100 100 200 Not a Trianle
6 100 1 100 Isosceles
7 100 2 100 Isosceles
8 100 100 100 Equilateral
9 100 199 100 Isosceles
10 100 200 100 Not a Triangle
11 1 100 100 Isosceles
12 2 100 100 Isosceles
13 100 100 100 Equilateral
14 199 100 100 Isosceles
15 200 100 100 Not a Triangle
9
Functional Test Cases: WN.EC (11
TCs)
Test case ID a b c Expected Results
1 5 5 5 Equilateral
2 5 5 3 Isosceles
3 5 3 5 Isosceles
4 3 5 5 Isosceles
5 3 4 5 Scalene
6 8 3 4 Not a triangle
7 7 3 4 Not a triangle
8 3 8 4 Not a triangle
9 3 7 4 Not a triangle
10 3 4 8 Not a triangle
11 3 4 7 Not a triangle
10
Functional Test Cases: DT (8 TCs.)
Test case ID a b c Expected Results
DT1 4 1 2 Not a Triangle
DT2 1 4 2 Not a Triangle
DT3 1 2 4 Not a Triangle
DT4 5 5 5 Equilateral
DT5 ? ? ? Impossible
DT6 ? ? ? Impossible
DT7 2 2 3 Isosceles
DT8 ? ? ? Impossible
DT9 2 3 2 Isosceles
DT10 3 2 2 Isosceles
DT11 3 4 5 Scalene
The number of test cases
0
50
100
150
200
250
300
350
RWT WT RT BVA SR.EC SN.EC WR.EC WN.EC DT
Test cases per method - Triangle Program
322 235 การทดสอบซอฟต์�แวร� 11
The number of test cases
322 235 การทดสอบซอฟต์�แวร� 12
0
50
100
150
200
250
300
350
RWT WT RT BVA SR.EC SN.EC WR.EC WN.EC DT
Test cases per method – NextDate Problem
The number of test cases
0
50
100
150
200
250
300
350
RWT WT RT BVA SR.EC SN.EC WR.EC WN.EC DT
Test cases per method –Commission Problem
322 235 การทดสอบซอฟต์�แวร� 13
Test cases Trend Line: All Program
322 235 การทดสอบซอฟต์�แวร� 14
Test case per method-all problem
0
50
100
150
200
250
300
350
RWT WT RT BVA SR.EC SN.EC WR.EC WN.EC DT
15
Testing Efficiency (1)• The data above, reveal the fundamental limitation of
functional testing: the twin possibilities of o gaps of untested functionalityo redundant tests
• For example:o The decision table technique for the NextDate program
generated 22 test cases (fairly complete)o The worst case boundary analysis generated 125 cases. These
are fairly redundant (check January 1 for five different years, only a few February cases but none on February 28, and February 29, and no major testing for leap years).
o The strong equivalence class test cases generated 36 test cases 11 of which are impossible.
• There are gaps and redundancy in functional test cases, and these are reduced by using more sophisticated techniques (i.e. Decision Tables).
16
Testing Efficiency (2)• The question is how can we quantify what we mean by the
term testing efficiency.
• The intuitive notion of an efficient testing technique is that it produces a set of test cases that are “just right” that is, with no gaps and no redundancy.
• We can even develop various ratios of useful test cases (i.e. not redundant, and with no gaps) with respect to the total number of test cases generated by method A and method B.
• One way to identify redundancy by annotating test cases with a brief purpose statement. Test cases with the same purpose provide a sense of redundancy measure.
• Detecting gaps is harder and this can be done only by comparing two different methods, even though there are no guarantees for complete detection of gaps.
• Overall, the structural methods (we will see later in the course), support interesting and useful metrics and these can provide a better quantification of testing efficiency.
17
Guidelines• Kinds of faults may reveal some pointers as to
which testing method to use.
• If we do not know the kinds of faults that are likely to occur in the program then the attributes most helpful in choosing functional testing methods are:
o Whether the variables represent physical or logical quantities
o Whether or not there are dependencies among variableso Whether single or multiple faults are assumedo Whether exception handling is prominent
18
Guidelines for Functional
Testing Technique Selection (1)• The following selection guidelines can be considered:
1. If the variables refer to physical quantities and/or are independent domain testing and equivalence testing can be considered.
2. If the variables are dependent, decision table testing can be considered
3. If the single-fault assumption is plausible to assume, boundary value analysis and robustness testing can be considered
4. If the multiple-fault assumption is plausible to assume, worst case testing, robust worst case testing, and decision table testing can be considered
5. If the program contains significant exception handling, robustness testing and decision table testing can be considered
6. If the variables refer to logical quantities, equivalence class testing and decision table testing can be considered
19
Guidelines for Functional
Testing Technique Selection (2)C1: variables Physical (P) Logical (L) P P P P P L L L L L
C2: Independent Variables? Y Y Y Y N Y Y Y Y N
C3: Single fault assumption? Y Y N N - Y Y N N -
C4: Exception handling? Y N Y N - Y N Y N -
A1: Boundary value analysis X
A2: Robustness testing X
A3: Worst case testing X
A4: Robust worst case testing X
A5: Weak equivalence testing X X X X
A6: Strong equivalence testing X X X X X X
A7: Decision table X X
Test Summary Report
322 235 การทดสอบซอฟต์�แวร� 20
Test Summary Report
322 235 การทดสอบซอฟต์�แวร� 21
Test Summary Report
322 235 การทดสอบซอฟต์�แวร� 22
Bug life Cycle
322 235 การทดสอบซอฟต์�แวร� 23
BTT= Bug Tracking
Tool
Basic of Testing effectiveness Metrics
• Test Coverage• Productivity• Defect Detection Effectiveness• Defect Acceptance Ratio• Estimation Accuracy
322 235 การทดสอบซอฟต์�แวร� 24
Test Coverage
322 235 การทดสอบซอฟต์�แวร� 25
Definition Calculation UnitPercentage of requirement that are covered in test Execution
= (Number of Requirement Covered in Test Execution/ Number of Requirement Specified ) * 100
Percentage (%)
Productivity
322 235 การทดสอบซอฟต์�แวร� 26
Definition Calculation UnitThe number of test cases developed or Test cases executed per person hour of effort
= (Number of Test Cases Developed/ Effort for Test Case development or Execution)
/hour
Defect Detection Effectiveness
322 235 การทดสอบซอฟต์�แวร� 27
Definition Calculation UnitPercentage of the total number of defects reported for the application that are reported during the testing stage
=(Number of Defects Reported during testing and Accepted)/ (Number of Defects Reported during testing and Accepted + Number of Defects Reported after the testing stage and Accepted) * 100
Percentage (%)
Defect Acceptance Ratio
322 235 การทดสอบซอฟต์�แวร� 28
Definition Calculation UnitPercentage of defects reported that are accepted as valid
=(Number of Defects Accepted as Valid)/ (Number of Defects Reported)
Percentage (%)
Estimation Accuracy
322 235 การทดสอบซอฟต์�แวร� 29
Definition Calculation UnitRatio of estimated effort to the actual effort for testing
=(Estimated Effort)/ (Actual Effort) x 100
Percentage (%)