1

Software Testing & Quality Assurance

Lecture 10

Created by: Paulo AlencarModified by: Frank Xu

2

OverviewStructural Testing

Introduction – General ConceptsFlow Graph Testing

DD-PathsTest Coverage MetricsBasis Path TestingGuidelines and Observations

Data Flow TestingHybrid MethodsRetrospective on Structural Testing

3

Path Testing

• Path Testing is focusing on test techniques that are based on the selection of test paths though a program graph. If the set of paths is properly chosen, then we can claim that we have achieved a measure of test thoroughness.

• The fault assumption for path testing techniques is that something has gone wrong with the software that makes it take a different path than the one intended.

• Structurally, a path is a sequence of statements in a program unit. Semantically, a path is an execution instance of the program unit. For software testing we are interested in entry-exit paths.

4

Path Testing Process

• Input: – Source code and a path selection criterion

• Process:– Generation of a CFG

– Selection of Paths

– Generation of Test Input Data

– Feasibility Test of a Path

– Evaluation of Program’s Output for the Selected Test Cases

5

x = z+5 z = 4*3-y if(x > z) goto A; for( u=0; u < x; u++) { z = z+1; }; A: y = z + k

x = z+5 z = 4*3-y

x > z

z = z+1 u++

y = z+k

u = 0

f

f t

t

u < x

1

2

3

4

5

6

7

Example of a simple control flowgraph.

R1 R2

R3

V(G) = 3 Basis set: 1, 2, 3, 4, 6, 7 1, 2, 3, 4, 5, 4, 6, 7 1, 2, 6, 7

Flow Graphs – Use in determining Paths for Testing - Revisited

6

Program Graph to DD-Path Graph

4 5 6 7

12

19

16

17

14

109

15

13

21

8

23

11

1820

22

first

E

M

J

K

H

C

B

I

F

G

last

D

LN

O

7

Topological Paths

• We can identify eight possible paths through the previous diagram

1. first – A – B – C – E – F – G – O – last2. first – A – B – D – E – F – G – O – last3. first – A – B – C – E – F – H – I – N – O – last4. first – A – B – D – E – F – H – I – N – O – last5. first – A – B – C – E – F – H – J – K – M – N – O – last6. first – A – B – D – E – F – H – J – K – M – N – O – last7. first – A – B – C – E – F – H – J – L – M – N – O – last8. first – A – B – D – E – F – H – J – L – M – N – O – last

8

Feasible Paths

• If we assume that the logic of the program dictates that “if node C is traversed, then we must traverse node H, and if node D is traversed, then we must traverse Node G”

– These constraints will eliminate Paths 1, 4, 6 and 8

– We are left to consider four feasible paths:

2. first – A – B – D – E – F – G – O – last

3. first – A – B – C – E – F – H – I – N – O – last

5. first – A – B – C – E – F – H – J – K – M – N – O – last

7. first – A – B – C – E – F – H – J – L – M – N – O – last

9

• Let’s consider the Triangle program, its CFG, and its DD-Path graph.

[Jorgensen: Software Testing A Craftsman’s Approach]

1. program triangle (input, output);2. VAR a,b,c: integer;3. IsATriangle: boolean;4. BEGIN5. writeln(“Enter integers: “);6. readln(a,b,c);7. Writeln(“Side A is: “, a, “Side B is: “, b, “Side C is: “, c);8. IF(a <b+c) AND (b < a+c) AND (c<a+b) THEN9. IsATriangle = True;10. ELSE IsATriangle = False;11. IF IsATriangle) 12. THEN 13. BEGIN14. IF (a=b) XOR (a=c) XOR (b=c) AND NOT((a=b) AND (a=c)15. THEN writeln(“Equilateral”);16. IF (a=b) AND (b=c)17. THEN writeln(“Equilateral”);18. IF(a <> b) AND (a <> c) AND (a<>c) 19. THEN writeln(“Scalene”);20. END21. ELSE Writeln(“Not a Triangle”);22. END

first

A

B C

D

L

E

F G

H

A

I

J K

last

P1: A-B-D-E-G-I-J-K-LastP2: A-C-D-E-G-I-J-K-LastP3: A-B-D-L-LastP4: A-B-D-E-F-G-I-J-K-LastP5: A-B-D-E-F-G-H-I-J-K-LastP6: A-B-D-E-F-G-H-I-K-Last

10

Node B is traversed D, E are traversedNode C is traversed D, L are traversedNode E is traversed F, H, J are traversedNode F is traversed H, J are traversedNode H is traversed F, J are traversedNode J is traversed F, I are traversed

• Let’s consider the Triangle program, its CFG, and its DD-Path graph. [Jorgensen: Software Testing A Craftsman’s Approach]

1. program triangle (input, output); 2. VAR a,b,c: integer; 3. IsATriangle: boolean; 4. BEGINA5. writeln(“Enter integers: “);A6. readln(a,b,c);A7. Writeln(“Side A is: “, a, “Side B is: “, b, “Side C is: “, c);A8. IF(a < b+c) AND (b < a+c) AND (c < a+b) THENB9. IsATriangle = True;C10. ELSE IsATriangle = False;D11. IF IsATriangle) D12. THEN E13. BEGINE14. IF (a=b) XOR (a=c) XOR (b=c) AND NOT((a=b) AND (a=c)F15. THEN writeln(“Equilateral”);G16. IF (a=b) AND (b=c)H17. THEN writeln(“Equilateral”);I18. IF(a <> b) AND (a <> c) AND (a<>c) J19. THEN writeln(“Scalene”);K20. ENDL21. ELSE Writeln(“Not a Triangle”);22. END

Logically feasible paths:

P1: A-C-D-L-LastP2: A-B-D-E-F-G-I-K-LastP3: A-B-D-E-G-H-I-K-LastP4: A-B-D-E-G-I-J-K-Last

first

A

B C

D

L

E

F G

H

A

I

J K

last

11

Structured Programming Constructs

s1

s2

s3

s1

s2

s4

s3

s1

s2

s5

s4s3

s1

s3

s2

s1

s2

s1

s2

Sequence if-then-else case if-then loop-pre-test loop-post-test

12

Violations of Structured Programming Constructs

s1

s2

s3

s1

s3

s2

s0

s1

s2

s3

s0

s0

s1

s3

s2

s1

Branching into loop Branching out of loop Branching into decision Branching out of decision

13

Essential Complexity

• Essential complexity. This measures how much unstructured logic exists in a module (e.g., a loop with an exiting GOTO statement)

• The essential complexity can be computed on a program graph that has been condensed by collapsing structured programming constructs to a single node, until no more structured constructs can be found

14

Condensing A Program Graphfirst

A

B C

D

L

E

F G

H

A

I

J K

last

a

b

c

a

b

cH

L

last

first

d

a

dL

last

first efirst

e

V(G) = 1

15

Test Coverage Metrics - Revisited

Metric Description of Coverage

C0 Every Statement

C1 Every DD-Path

C1P Every predicate to each outcome

C2 C1 Coverage + loop coverage

Cd C1 Coverage + every dependent pair of DD-Paths

CMCC Multiple condition coverage

Cik Every program path that contains up to k repetitions of a loop (usually k=2)

Cstat “Statistically significant” fraction of paths

C∞ All possible execution paths

16

Some Fundamental Path Selection Criteria

• Exercise every statement or instruction at least once (statement coverage) C0

• Exercise every branch and case statement, in each direction, at least once (branch coverage) C1

• Exercise each condition in a decision (condition coverage) • Exercise each condition in a decision with all possible

outcomes C1P

• Exercise every compound predicate outcome (MCC) CMCC

• Exercise every path from entry to exit C∞

17

Example1

2

3

10 4

11 12

6

13

7

9

8

5

F T

FT

F

T

TF

FT

Statement Coverage C0:SCPath1: 1-2-3(F)-10(F)-11-13SCPath2: 1-2-3(T)-4(T)-5-6(T)- 7(T)-8-9-3(F)-10(T)-12-13

Branch or Decision Coverage C1:BCPath1: 1-2-3(F)-10(F)-11-13BCPath2: 1-2-3(T)-4(T)-5-6(T)- 7(T)-8-9-3(F)-10(T)-12-13BCPath3: 1-2-3(T)-4(F)-10(F)-11-13BCPath4: 1-2-3(T)-4(T)-5-6(F)-9-3(F)- 10(F)-11-13BCPath5: 1-2-3(T)-4(T)-5-6(T)-7(F)-9- 3(F)-10(F)-11-13

18

Condition Coverage

• Requires that each condition in a compound decision takes on all possible outcomes at least once

• Condition Coverage does not require that each decision takes on all possible outcomes

• Example: if Not (A or B) then CSetting A = True, B = False, then A = False and B = True

satisfies Condition Coverage, but statement C will never be executed

19

Multiple Condition Coverage

• Requires that each possible combination of inputs be tested for each decision.

• Example: “if (A and B)” requires 4 test cases:A = True, B = TrueA = True, B = FalseA = False, B = TrueA = False, B = False

• For n conditions, 2n test cases are needed, and this grows exponentially with n

20

Modified MCC

• Requires that each condition be shown to independently affect the outcome of a decision

• Generally, for n conditions, MC/DC requires only n + 1 test cases

• Example: if (A or B) then C 3 test cases: A = True, B = False (A dominates)

A = False, B = True (B dominates)A = False, B = False (false outcome)

21

Examples

• To test if (A or B)A: T F FB: F T F

• To test if (A and B)A: F T TB: T F T

• To test if (A xor B)A: T T FB: T F T

22

Guidelines and Observations• Functional testing focuses on properties that are “too far” or disassociated from

the source code being tested

• Path testing focuses too much on the graph and not the logic of the code

• Path testing very useful to measure the quality of the testing (coverage criteria)

• Basis path testing is giving us a lower bound of how much testing is necessary

• Path testing is providing a set of metrics that act as cross checks to functional testing

– When a program path is traversed by several functional test cases we suspect redundancy

– When we fail to attain DD-Path graph coverage, we know that there are gaps is the functional test cases

• Coverage metrics can be used as– Setting minimum testing standards and as– Mechanism to selectively test portions of the code more rigorously than others

23

Guidelines and Observations• We can distinguish between

– Specified program behavior (S),

– Programmed behavior (feasible paths) (P), and

– Topologically feasible paths (T)

S P

T

24

Guidelines and Observations• Region 1 contains specified behavior that is

implemented by feasible paths – most desirable

• Every feasible path is topologically possible so the parts 2,6 of set P should be empty

• Region 3 contains feasible paths that corresponds to unspecified behavior. Such extra functionality must be inspected

• Regions 4 and 7 contain the infeasible paths. Region 4 is problematic as it corresponds to specified behavior that has almost been implemented (topologically possible, yet infeasible program paths) – coding errors

• Region 7 refers to unspecified, infeasible, yet topologically possible paths – potential maintenance problem – could become Region3

S P

T

5

4

1

7

3

2

6

25

Software Testing Standards• IEEE 1008 Software Unit Testing• IEEE 1044 Classification for Software Anomalies• ISO/IEC 12207 Information Technology--Software Life Cycle Processes• ISO/IEC TR 15271 Guide for ISO/IEC 12207 (Software Life Cycle Processes)• AECL CE-1001-STD REV.1 Standard for Software Engineering of Safety Critical Software• BSI BS-7738 Specification for Information Systems Products Using SSADM• (Structured Systems Analysis and Design Method)• BSI BS-7925-1 Software Testing - Vocabulary• BSI BS-7925-2 Standard for Software Component Testing• DEF 00-55 (Part 1)/1 The Procurement of Safety Critical Software in Defence Equipment-Requirements• DIN VDE 0801 Principles for Computers in Safety-Related Systems• ESA PSS-05-01 Guide to the Software Engineering Standards, Issue 1• German Process-Model (V-Model) Software Life-Cycle Process Model• IAEA TRS-372 Development and Implementation of Computerized Operator Support Systems in Nuclear

Installations• IEC 60601-1-4 Medical Electrical Equipment--Part 1: General Requirements for Safety-4.Collateral Standard:

Programmable Electrical Medical Systems• IEC 60880 Software for Computers in the Safety Systems of Nuclear Power Stations• IEE 5 Software Inspection Handbook• NCC STARTS Purchasers' Handbook--Procuring Software-Based Systems• NRC NUREG/BR-0167 Software Quality Assurance Program and Guidelines• RTCA DO-178B/ED-12B Software Considerations in Airborne Systems and Equipment Certification