coverage criteria
DESCRIPTION
Coverage Criteria. Drawn mostly from Ammann&Offutt and Pezze&Yooung. IEEE definition of V&V. Validation: The process of evaluating software at the end of software development to ensure compliance with intended usage - PowerPoint PPT PresentationTRANSCRIPT
Coverage CriteriaDrawn mostly from
Ammann&Offutt and Pezze&Yooung
2
IEEE definition of V&V
• Validation: The process of evaluating software at the end of software development to ensure compliance with intended usage• Verification: The process of determining whether
the products of a given phase of the software development process fulfill the requirements established during the previous phase
3
Static vs dynamic testing
• Static testing : Testing without executing the program• Include software inspections and some forms of
analyses• Very effective at finding certain kinds of problems –
especially “potential” faults, that is, problems that could lead to faults when the program is modified
• Dynamic testing : Testing by executing the program with real inputs
4
Testing vs debugging
• Testing : Finding inputs that cause the software to fail• Debugging : The process of finding a fault given a
failure
5
A test case
• Test case values: The values that directly satisfy one test requirement• Expected results: The result that will be produced
when executing the test if the program satisfies it intended behavior
6
Top-down vs bottom up testing• Top-down testing : Test the main procedure, then
go down through procedures it calls, and so on• Bottom-up testing : Test the leaves in the tree
(procedures that make no calls), and move up to the root.• Each procedure is not tested until all of its children have
been tested
7
Test requirement
• Test requirements: Specific things (software artifacts) that must be satisfied or covered during testing• Non-software example: test a bag of jelly beans• Come up with ways to test• Suppose the following flavors: Lemon (yellow), pistachio
(green), cantaloupe (orange), pear (white), tangerine (orange), apricot (yellow)• One requirement: test each flavor (six test
requirements)
8
Test criterion
• Test criterion: A collection of rules and a process that define test requirements
Flavor criterion: TR = {flavor=lemon, flavor=pistachio, flavor=cantaloupe, flavor=pear, flavor=tangerine, flavor=apricot}
9
Test coverage
• Given a set of test requirements TR for coverage criterion C, a test set T satisfies C coverage if and only if for every test requirement tr in TR, there is at least one test t in T such that t satisfies tr• A test case with 12 beans: 3 lemons, 1 pistachio, 2
cantaloupe, 1 pear, 1 tangerine, 4 apricot • OK to satisfy a test requirement with more than
one test
10
Coverage level
• Given a set of test requirements TR and a test set T, the coverage level is simply the ratio of the number of test requirements satisfied by T to the size of TR• Why? Sometime test requirements may be
infeasible• Example: suppose tangerine jelly beans are rare
and some bags may not contain any• Flavor criteria cannot be 100% satisfied• Maximum coverage level: 5/6 or 83%
11
Criteria vs subsumption
• Criteria subsumption : A test criterion C1 subsumes C2 if and only if every set of test cases that satisfies criterion C1 also satisfies C2• Must be true for every set of test cases• Example: color criteria for the jelly bean: {yellow,
green, orange, white}• If we satisfy flavor criteria, we’ll satisfy color criteria
• Example : If a test set has covered every branch in a program (satisfied the branch criterion), then the test set is guaranteed to also have covered every statement
12
Coverage is not size
• Coverage does not depend on the number of test cases • T0 , T1 : T1 >coverage T0 T1 <cardinality T0 • T1 , T2 : T2 =coverage T1 T2 >cardinality T1
• Small test cases make failure diagnosis easier• A failing test case in T2 gives more information for
fault localization than a failing test case in T1
13
Question 1
• Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.• Does T1 necessarily satisfy C2? Explain.
14
Question 2
• Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.• Does T1 necessarily satisfy C2? Explain.
Yes. This follows directly from the definition of subsumption.
15
Question 2
• Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.• Does T2 necessarily satisfy C1? Explain.
16
Question 2
• Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.• Does T2 necessarily satisfy C1? Explain.• No. There is no reason to expect test requirements
generated by C1 to be satisfied by T2.
17
Question 3
• Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.• If P contains a fault, and T2 reveals the fault, T1
does not necessarily also reveal the fault. Explain.
18
Question 3
• Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.• If P contains a fault, and T2 reveals the fault, T1
does not necessarily also reveal the fault. Explain.No. This is the hard question. Testers often think that test sets for strong criteria are at least as good at finding faults as test sets for weaker criteria. But subsumption is about criteria, not about test sets. In particular, there is no requirement that test set T2 be a subset of test set T1. So, it could happen that T2 contains that one test that reveals the fault, and T1 doesn't.
19
Statements vs branches
• Traversing all edges of a graph causes all nodes to be visited• So test suites that satisfy the branch adequacy criterion
for a program P also satisfy the statement adequacy criterion for the same program
• The converse is not true: A statement-adequate (or node-adequate) test suite may not be branch-adequate (edge-adequate)
20
“All branches” can still miss conditions• Sample fault: missing operator (negation)
digit_high == -1 || digit_low == -1• Branch adequacy criterion can be satisfied by
varying only digit_low• The faulty sub-expression might never determine the
result• We might never really test the faulty condition, even
though we tested both outcomes of the branch
21
Condition testing
• Branch coverage exposes faults in how a computation has been decomposed into cases• Intuitively attractive: check the programmer’s case
analysis• But only roughly: groups cases with the same outcome
• Condition coverage considers case analysis in more detail• Also individual conditions in a compound Boolean
expression• e.g., both parts of digit_high == 1 || digit_low == -1
22
Basic conditions vs branches• Basic condition adequacy criterion can be satisfied
without satisfying branch coverage• Branch and basic condition are not comparable
(neither implies the other)
Covering branches and conditions• Branch and condition adequacy: cover all conditions and all
decisions• Compound condition adequacy
• Cover all possible evaluations of compound conditions• Cover all branches of a decision tree
Ch 12, slide 23
24
Compound conditions: Exponential complexity
(((a || b) && c) || d) && e
Test a b c d e
Case
(1) T — T — T
(2) F T T — T
(3) T — F T T
(4) F T F T T
(5) F F — T T
(6) T — T — F
(7) F T T — F
(8) T — F T F
(9) F T F T F
(10) F F — T F
(11) T — F F —
(12) F T F F —
(13) F F — F —
Short-circuit evaluation often reduces this to a more manageable number, but not always
25
Multiple (compound) condition coverage• The multiple condition coverage of T is computed
as Cc/(Ce -Ci) , where Cc is the number of combinations covered, Ci is the number of infeasible simple combinations, and Ce is the total number of combinations in the program• Potentially a large number of test cases
26
Modified condition/decision (MC/DC)
• Motivation: Effectively test important combinations of conditions, without exponential blowup in test suite size • “Important” combinations means: Each basic condition
shown to independently affect the outcome of each decision• Compound condition as a whole evaluates to true for
one and false for the other
27
MC/DC: linear complexity• N+1 test cases for N basic conditions
(((a || b) && c) || d) && e
Test a b c d e outcome
Case
(1) true -- true -- true true
(2) false true true -- true true
(3) true -- false true true true
(6) true -- true -- false false
(11) true -- false false -- false
(13) false false -- false -- false
• Underlined values independently affect the output of the decision• Required by the RTCA/DO-178B standard
28
Comments on MC/DC
• MC/DC is • Basic condition coverage (C)• Branch coverage (DC)• Plus one additional condition (M): every condition must
independently affect the decision’s output
• It is subsumed by compound conditions and subsumes all other criteria discussed so far• Stronger than statement and branch coverage
• A good balance of thoroughness and test size (and therefore widely used)• Federal Aviation Administration’s requirement that test
suites be MC/DC adequate
Paths? (Beyond individual branches)• Should we explore sequences of branches (paths) in the
control flow?• Many more paths than branches
• A pragmatic compromise will be needed
Ch 12, slide 29
(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 30
Path adequacy
• Decision and condition adequacy criteria consider individual program decisions• Path testing focuses consider combinations of
decisions along paths• Adequacy criterion: each path must be executed at
least once • Coverage:
# executed paths # paths
(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 31
Practical path coverage criteria• The number of paths in a program with loops is
unbounded • the simple criterion is usually impossible to satisfy
• For a feasible criterion: Partition infinite set of paths into a finite number of classes• Useful criteria can be obtained by limiting • the number of traversals of loops• the length of the paths to be traversed• the dependencies among selected paths
(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 32
Cyclomatic adequacy
• Cyclomatic number: number of independent paths in the CFG• If e = #edges, n = #nodes, c = #connected components of a
graph, it is:• e - n + c for an arbitrary graph• e - n + 2 for a CFG
• Cyclomatic coverage counts the number of independent paths that have been exercised, relative to cyclomatic complexity