coverage criteria

Coverage CriteriaDrawn mostly from

Ammann&Offutt and Pezze&Yooung

2

IEEE definition of V&V

• Validation: The process of evaluating software at the end of software development to ensure compliance with intended usage• Verification: The process of determining whether

the products of a given phase of the software development process fulfill the requirements established during the previous phase

3

Static vs dynamic testing

• Static testing : Testing without executing the program• Include software inspections and some forms of

analyses• Very effective at finding certain kinds of problems –

especially “potential” faults, that is, problems that could lead to faults when the program is modified

• Dynamic testing : Testing by executing the program with real inputs

4

Testing vs debugging

• Testing : Finding inputs that cause the software to fail• Debugging : The process of finding a fault given a

failure

5

A test case

• Test case values: The values that directly satisfy one test requirement• Expected results: The result that will be produced

when executing the test if the program satisfies it intended behavior

6

Top-down vs bottom up testing• Top-down testing : Test the main procedure, then

go down through procedures it calls, and so on• Bottom-up testing : Test the leaves in the tree

(procedures that make no calls), and move up to the root.• Each procedure is not tested until all of its children have

been tested

7

Test requirement

• Test requirements: Specific things (software artifacts) that must be satisfied or covered during testing• Non-software example: test a bag of jelly beans• Come up with ways to test• Suppose the following flavors: Lemon (yellow), pistachio

(green), cantaloupe (orange), pear (white), tangerine (orange), apricot (yellow)• One requirement: test each flavor (six test

requirements)

8

Test criterion

• Test criterion: A collection of rules and a process that define test requirements

Flavor criterion: TR = {flavor=lemon, flavor=pistachio, flavor=cantaloupe, flavor=pear, flavor=tangerine, flavor=apricot}

9

Test coverage

• Given a set of test requirements TR for coverage criterion C, a test set T satisfies C coverage if and only if for every test requirement tr in TR, there is at least one test t in T such that t satisfies tr• A test case with 12 beans: 3 lemons, 1 pistachio, 2

cantaloupe, 1 pear, 1 tangerine, 4 apricot • OK to satisfy a test requirement with more than

one test

10

Coverage level

• Given a set of test requirements TR and a test set T, the coverage level is simply the ratio of the number of test requirements satisfied by T to the size of TR• Why? Sometime test requirements may be

infeasible• Example: suppose tangerine jelly beans are rare

and some bags may not contain any• Flavor criteria cannot be 100% satisfied• Maximum coverage level: 5/6 or 83%

11

Criteria vs subsumption

• Criteria subsumption : A test criterion C1 subsumes C2 if and only if every set of test cases that satisfies criterion C1 also satisfies C2• Must be true for every set of test cases• Example: color criteria for the jelly bean: {yellow,

green, orange, white}• If we satisfy flavor criteria, we’ll satisfy color criteria

• Example : If a test set has covered every branch in a program (satisfied the branch criterion), then the test set is guaranteed to also have covered every statement

12

Coverage is not size

• Coverage does not depend on the number of test cases • T0 , T1 : T1 >coverage T0 T1 <cardinality T0 • T1 , T2 : T2 =coverage T1 T2 >cardinality T1

• Small test cases make failure diagnosis easier• A failing test case in T2 gives more information for

fault localization than a failing test case in T1

13

Question 1

• Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.• Does T1 necessarily satisfy C2? Explain.

14

Question 2

• Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.• Does T1 necessarily satisfy C2? Explain.

Yes. This follows directly from the definition of subsumption.

15

Question 2

• Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.• Does T2 necessarily satisfy C1? Explain.

16

Question 2

• Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.• Does T2 necessarily satisfy C1? Explain.• No. There is no reason to expect test requirements

generated by C1 to be satisfied by T2.

17

Question 3

• Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.• If P contains a fault, and T2 reveals the fault, T1

does not necessarily also reveal the fault. Explain.

18

Question 3

• Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose that test set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.• If P contains a fault, and T2 reveals the fault, T1

does not necessarily also reveal the fault. Explain.No. This is the hard question. Testers often think that test sets for strong criteria are at least as good at finding faults as test sets for weaker criteria. But subsumption is about criteria, not about test sets. In particular, there is no requirement that test set T2 be a subset of test set T1. So, it could happen that T2 contains that one test that reveals the fault, and T1 doesn't.

19

Statements vs branches

• Traversing all edges of a graph causes all nodes to be visited• So test suites that satisfy the branch adequacy criterion

for a program P also satisfy the statement adequacy criterion for the same program

• The converse is not true: A statement-adequate (or node-adequate) test suite may not be branch-adequate (edge-adequate)

20

“All branches” can still miss conditions• Sample fault: missing operator (negation)

digit_high == -1 || digit_low == -1• Branch adequacy criterion can be satisfied by

varying only digit_low• The faulty sub-expression might never determine the

result• We might never really test the faulty condition, even

though we tested both outcomes of the branch

21

Condition testing

• Branch coverage exposes faults in how a computation has been decomposed into cases• Intuitively attractive: check the programmer’s case

analysis• But only roughly: groups cases with the same outcome

• Condition coverage considers case analysis in more detail• Also individual conditions in a compound Boolean

expression• e.g., both parts of digit_high == 1 || digit_low == -1

22

Basic conditions vs branches• Basic condition adequacy criterion can be satisfied

without satisfying branch coverage• Branch and basic condition are not comparable

(neither implies the other)

Covering branches and conditions• Branch and condition adequacy: cover all conditions and all

decisions• Compound condition adequacy

• Cover all possible evaluations of compound conditions• Cover all branches of a decision tree

Ch 12, slide 23

24

Compound conditions: Exponential complexity

(((a || b) && c) || d) && e

Test a b c d e

Case

(1) T — T — T

(2) F T T — T

(3) T — F T T

(4) F T F T T

(5) F F — T T

(6) T — T — F

(7) F T T — F

(8) T — F T F

(9) F T F T F

(10) F F — T F

(11) T — F F —

(12) F T F F —

(13) F F — F —

Short-circuit evaluation often reduces this to a more manageable number, but not always

25

Multiple (compound) condition coverage• The multiple condition coverage of T is computed

as Cc/(Ce -Ci) , where Cc is the number of combinations covered, Ci is the number of infeasible simple combinations, and Ce is the total number of combinations in the program• Potentially a large number of test cases

26

Modified condition/decision (MC/DC)

• Motivation: Effectively test important combinations of conditions, without exponential blowup in test suite size • “Important” combinations means: Each basic condition

shown to independently affect the outcome of each decision• Compound condition as a whole evaluates to true for

one and false for the other

27

MC/DC: linear complexity• N+1 test cases for N basic conditions

(((a || b) && c) || d) && e

Test a b c d e outcome

Case

(1) true -- true -- true true

(2) false true true -- true true

(3) true -- false true true true

(6) true -- true -- false false

(11) true -- false false -- false

(13) false false -- false -- false

• Underlined values independently affect the output of the decision• Required by the RTCA/DO-178B standard

28

Comments on MC/DC

• MC/DC is • Basic condition coverage (C)• Branch coverage (DC)• Plus one additional condition (M): every condition must

independently affect the decision’s output

• It is subsumed by compound conditions and subsumes all other criteria discussed so far• Stronger than statement and branch coverage

• A good balance of thoroughness and test size (and therefore widely used)• Federal Aviation Administration’s requirement that test

suites be MC/DC adequate

Paths? (Beyond individual branches)• Should we explore sequences of branches (paths) in the

control flow?• Many more paths than branches

• A pragmatic compromise will be needed

Ch 12, slide 29

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 30

Path adequacy

• Decision and condition adequacy criteria consider individual program decisions• Path testing focuses consider combinations of

decisions along paths• Adequacy criterion: each path must be executed at

least once • Coverage:

# executed paths # paths

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 31

Practical path coverage criteria• The number of paths in a program with loops is

unbounded • the simple criterion is usually impossible to satisfy

• For a feasible criterion: Partition infinite set of paths into a finite number of classes• Useful criteria can be obtained by limiting • the number of traversals of loops• the length of the paths to be traversed• the dependencies among selected paths

(c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 32

Cyclomatic adequacy

• Cyclomatic number: number of independent paths in the CFG• If e = #edges, n = #nodes, c = #connected components of a

graph, it is:• e - n + c for an arbitrary graph• e - n + 2 for a CFG

• Cyclomatic coverage counts the number of independent paths that have been exercised, relative to cyclomatic complexity