Software Testing Techniques

Software Testing

Testing is the process of exercising aprogram with the specific intent of finding

errors prior to delivery to the end user.

Software Development Process

AbstractConcept

Tangible Product

TestingTesting

Tries hard to demolish the software

Developer must discard the preconceived idea that his software is always correct.

Software Testing Objectives

• Uncovering different cases of errors

• Needing minimum effort and time

• Showing that software functions work according to specification

It cannot show absence of errors or defects. It can only show that errors are present

Software Testing Principles

• All tests should be traceable to customer requirements

• Tests should be planned long before testing begins

• The “pareto” principle applies to software testing

• Exhaustive testing is not possible• To be most effective, testing should be

conducted by a third party

Customer thinks - most serious defect is not meeting his

requirementsAll tests can be planned and designed before any

coding begins 80 % of errors

originate from 20% of

programs

Too many

paths in a

normal -size

program

Who Tests the Software?

developerdeveloper independent testerindependent tester

Understands the system Understands the system

but, will test "gently"but, will test "gently"

and, is driven by "delivery"and, is driven by "delivery"

Must learn about the systemMust learn about the system,,

but, will attempt to break itbut, will attempt to break it

and, is driven by qualityand, is driven by quality

What is a GOOD Test?

Has a high probability of

finding an error

Has a high probability of

finding an errorIt must not be

redundant. Every test must have a

different purpose

It must not be redundant. Every test must have a

different purposeIt should neither

be too simple, nor too complex.

It should neither be too simple,

nor too complex.

What Testing Showserrorserrors

requirements conformancerequirements conformance

performanceperformance

an indicationan indicationof qualityof quality

Testability• Operability—it operates cleanly• Observability—the results of each test case are

readily observed• Controlability—the degree to which testing can be

automated and optimized• Decomposability—testing can be targeted• Simplicity—reduce complex architecture and

logic to simplify tests• Stability—few changes are requested during

testing• Understandability—of the design

Test Case Design• Design of test cases is as challenging as

designing the software itself• A rich variety of test case design methods

have evolved• Methods provide a systematic approach o

testing• Methods provide a mechanism for that

ensures the completeness of tests and high likelihood of finding an error

Software Testing

Methods

Strategies

white-boxmethods

black-box methods

Close examination of procedural detailClose examination of procedural detail

Logical paths through the software are Logical paths through the software are tested by providing test cases that exercise tested by providing test cases that exercise specific sets of conditions and/or loopsspecific sets of conditions and/or loops

Status of the program is examined at Status of the program is examined at various various pointspoints to determine if the actual status to determine if the actual status matches the expected status matches the expected status

White Box TestingWhite Box Testing

Does White Box Testing lead to Does White Box Testing lead to 100% correct program?100% correct program?

The answer is:

NoNoThe answer is:

NoNo

It is not possible to exhaustively test every program path because the number paths is simply too large

Exhaustive Testing

loop < 20 Xloop < 20 X

There areThere are 10 possible paths! If we execute one possible paths! If we execute onetest per millisecond, it would take test per millisecond, it would take 3,170 years3,170 years to totest this program!!test this program!!

14

Selective Testing

loop < 20 Xloop < 20 X

Selected pathSelected path

Black Box TestingBlack Box Testing Know the specific function that the program is

designed to perform Test that each function is fully operational Search for more errors in the functions These tests are conducted at the software

interface Whether input is properly accepted and output is

correctly produced Integrity of external information (database) is

maintained Examines fundamental aspects of the system with

little regard to internal logical structure

The attributes of both black-box and white-boxwhite-box testing can be

combined to provide an approach that validates the software

interface and selectively ensures that the internal workings of the

software are correct

The attributes of both black-box and white-boxwhite-box testing can be

combined to provide an approach that validates the software

interface and selectively ensures that the internal workings of the

software are correct

White Box TestingWhite Box Testing

• Is also known as “Glass Box Testing”

• Uses the control structure of the procedural design to derive test cases

If a= b then

do c

else

do v

Glass Box

WhiteWhite Box TestingBox Testing

Exercise all logical decisions on their true and false sides

Exercise all loops at their boundaries and within their operational bounds

Guarantees that all independent paths within a module have been exercised at least once

Exercise internal data structures to ensure their validity

Test Case Design

"Bugs lurk in corners"Bugs lurk in corners and congregate at and congregate at boundaries ..."boundaries ..."

Boris BeizerBoris Beizer

OBJECTIVEOBJECTIVE

CRITERIACRITERIA

CONSTRAINTCONSTRAINT

to uncover errorsto uncover errors

in a complete mannerin a complete manner

with a minimum of effort and timewith a minimum of effort and time

Why Cover?logic errors and incorrect assumptionslogic errors and incorrect assumptions are inversely proportional to a path's are inversely proportional to a path's execution probabilityexecution probability

we often we often believebelieve that a path is not that a path is not likely to be executed; in fact, reality is likely to be executed; in fact, reality is often counter intuitiveoften counter intuitive

typographicaltypographical errors are random; it's errors are random; it's likely that untested paths will containlikely that untested paths will contain some some

Why need Why need White-boxWhite-box testing? testing? Why need Why need White-boxWhite-box testing? testing?

• Logic errors occur more frequently in the less probable paths of execution

• Everyday processing tends to be well understood while “special case “ processing tends to fall into cracks

• We often believe that a logical path is not likely to be executed when , in fact it may be executed on a regular basis

• Typos are random. It is as likely that a type will exist on an obscure logical path as on a mainstream path.

White-Box Testing

... our goal is to ensure that all... our goal is to ensure that all statements and conditions havestatements and conditions have been executed at least once ...been executed at least once ...

Basis Path TestingBasis Path Testing Basis Path TestingBasis Path Testing

• It is white-box testing techniqueIt is white-box testing technique

• Proposed by Tom MacabeProposed by Tom Macabe

• Derives a logical complexity measureDerives a logical complexity measure

• Test cases derived to exercise the basis set are Test cases derived to exercise the basis set are guaranteed to execute every statement in the guaranteed to execute every statement in the program program at least onceat least once during testing during testing

• Uses Uses flow-graph notationflow-graph notation

• Each structured construct has corresponding flow Each structured construct has corresponding flow graph symbolgraph symbol

Structured Constructs

Sequence

If-then-else

While-do

Repeat untilcase

Selection

Loops

if

thenelse

Flow Graph Symbols

Sequenceif while

caseuntil

• Each circle is called a flow graph node , represents one or more procedural statements

• A sequence of process boxes and a decision diamond can map into a single node

• An edge (arrow) must terminate at a node

• Areas bounded by edged and nodes are called regions

• Area outside graph is also a region

Flow Graph Symbols

• A compound condition occurs when one or more Boolean operators (AND,OR) is present in a conditional statement

• Separate node is created for each condition

• Each condition node is known as predicate node

• Each predicate node has two or more edges leaving it

Flow Graph for Compound Conditions

x

b

y

a

x

If a or b thenprocedure x

elseprocedure y

endif

If a or b thenprocedure x

elseprocedure y

endif

• Very useful software metric

• Foundation on graph theory

• Quantitative measure of the logical complexity of the program

• It defines the number of independent paths in the basis set of a program

• Provides s with an upper bound for the number of tests to be conducted

• Ensures that all statements have been executed at least once

Cyclomatic Complexity

Cyclomatic Complexity

A number of industry studies have indicatedA number of industry studies have indicated that the higher V(G), the higher the probabilitythat the higher V(G), the higher the probability or errors.or errors.

V(G)V(G)

modulesmodules

modules in this range aremodules in this range are more error pronemore error prone

• There are three ways:

Computation of Cyclomatic Complexity V(G) Cyclomatic Complexity V(G) for flow - graph G

V(G) = R where

R- no. of regions in G

V(G) = R where

R- no. of regions in G

V(G) = E - N + 2 where

W - no. of edges in G

N - no. of nodes in G

V(G) = E - N + 2 where

W - no. of edges in G

N - no. of nodes in G

V(G) = P+1 where

P - no. of predicate nodes in G

V(G) = P+1 where

P - no. of predicate nodes in G

Given flow-graph has 4 regions

V(G) = 11-9 + 2 = 4V(G) = 3 + 1 = 4

• Any path through the program that introduces at least one new set of processing statements or a new condition

• Must move along at least one edge that has not been traversed before the path is defined

• Each path introduces a new edge

• There are Four paths for the given example:

– Path 1: 1-11

– Path 2: 1-2-3-4-5-10-1-11

– Path-3: 1-2-3-6-8-9-10-1-11

– Path 4: 1-2-3-6-7-9-10-1-11

Independent Path

Basis Path TestingNext, we derive theNext, we derive the

independent paths:independent paths:

Since V(G) = 4,Since V(G) = 4,there are four pathsthere are four paths

Path 1Path 1: 1,2,3,6,7,8: 1,2,3,6,7,8Path 2Path 2: 1,2,3,5,7,8: 1,2,3,5,7,8Path 3Path 3: 1,2,4,7,8: 1,2,4,7,8Path 4Path 4: 1,2,4,7,2,4,...7,8: 1,2,4,7,2,4,...7,8

Finally, we derive testFinally, we derive testcases to exercise thesecases to exercise these paths.paths.

11

22

3344

55 66

77

88

• Using the code as the foundation , draw a corresponding flow graph

• Determine the cyclomatic complexity of the flow graph

• Determine a basis set of linearly independent paths

• Prepare test cases that will force execution of each path in the basis set

• Execute each test case and compared to expected results

• Some paths cannot be tested independently and must be tested as part of another path

Deriving Test Cases

Condition TestingCondition Testing Condition TestingCondition Testing

• It is a design method that exercises the logical conditions (Consisting of arithmetic, relational and Boolean expressions and operators) in a program module

• Types of error found in a condition:

– Boolean operator error

– Boolean variable error

– Boolean parenthesis error

– Relational operator error

– Arithmetic expression error

Condition TestingCondition Testing Condition TestingCondition Testing

• Branch Testing

– For a compound condition C, the true and false branches of C and every simple condition in C need to be executed at least once

• Domain Testing– Requires three or four tests for a relational

expression

– To detect errors in E1<relational-operator>E2

• three tests are required to detect error in the operator

• putting as minimum possible different values in E1 and E2 will help to detect errors in E1and E2

BRO TestingBRO Testing BRO TestingBRO Testing

• All relational operators and boolean variables must occur only once

• Uses condition constraint for a condition C which is defined as (D1, D2, …Dn) where Di specifies thea constraint on he outcome of ith condition of C

• Boolean variables have constraint value True or False

• For relational expression symbols <,>, = are used to specify the conditions

BRO Testing - ExamplesBRO Testing - Examples BRO Testing - ExamplesBRO Testing - Examples

Example 1: B1 & B2

C1= (D1,D2) = {(t,t),(t,f),(f,t)}

Example 1: B1 & B2

C1= (D1,D2) = {(t,t),(t,f),(f,t)}

Example 2: B1 & (E3=E4)

C2= (D1,D2) = {(t,=),(t,>),(t,<),(f,=)}

Example 2: B1 & (E3=E4)

C2= (D1,D2) = {(t,=),(t,>),(t,<),(f,=)}

Example 3: (E1 > E2) & (E3=E4)

C2= (D1,D2) = {(>,=),(<,=),(>,>),(>,<),(=,=)}

Example 3: (E1 > E2) & (E3=E4)

C2= (D1,D2) = {(>,=),(<,=),(>,>),(>,<),(=,=)}

Dataflow TestingDataflow Testing Dataflow TestingDataflow Testing

• Selects test paths according to the location of definitions and uses of variables in the program (S-statement no.)

• DEF(S) = {X| statement S contains a definition of X} USE(S) = (X| statement S contains a use of X} Assume

• For if or loop statement DEF set is always empty

• The definition of var X at S is said to be live at statement S’ if the path S-S’ contains no other definition of X

• A definition-use chain of var Xis of the form [X,S,S’] where X is DEF(S) and USE(S’) and X is live at S’.

DU TestingDU Testing DU TestingDU Testing

• Every DU chain should be covered at least once

• Useful for programs with nested if and loop statements

• Effective for error-detection

• Selecting test-paths is quite difficult

• Does not guarantee all branches of a program (e.g. if statement with no else statement and no definition in the then part.

DU TestingDU Testing DU TestingDU Testingproc x b1: do while C1 if C2

then if C4 then B4; else B5; endif;

else if C3

then B2; else B3; endif;endif;

enddo; B6; end proc;

proc x b1: do while C1 if C2

then if C4 then B4; else B5; endif;

else if C3

then B2; else B3; endif;endif;

enddo; B6; end proc;

Var X is defined in the last statement of blocks B1, B2, B3, B4, B5 and used in the first statements of B2, B3, B4, B5, B6There are 25 DU chains of var X

We need to cover only five paths from each B1, B2, B3, B4, B5 to B6

Loop TestingLoop Testing Loop TestingLoop Testing

• Loops are he cornerstone of vast majority of all software

• Focuses exclusively on the validity of loop constructs

• Four different classes of loops are defined:

– Simple loops

– Nested loops

– concatenated loops

– unstructured loops

Loop Testing

Nested Nested LoopsLoops

Concatenated Concatenated LoopsLoops UnstructuredUnstructured

LoopsLoops

Simple Simple looploop

Simple LoopsSimple Loops Simple LoopsSimple Loops

Set of Tests • Skip the loop entirely

• Only one pass through the loop

• Two passes through the loop

• M passes through the loop where m < n (n – max no. pf allowable passes)

• N-1, n, n+1 passes through the loop

Nested LoopsNested Loops Nested LoopsNested Loops

1. Start at the innermost loop. Set all other loops to minimum value

2. Conduct simple loop test for the innermost loop while outer loops have the minimum loop counter

3. Work outward, conducting tests for the next loop , keeping outer loops at minimum value and nested loops at typical values

4. Continue until all loops have been tested

Number of possible test is impractically large if simple loop approach is adopted.

Concatenated LoopsConcatenated Loops Concatenated LoopsConcatenated Loops

1. Simple loop approach can be adopted if each of the loops is independent of the others

2. If loop counter value of one loop is used as the initial loop counter value of another, then the loops are not independent. In such cases nested loop approach should be adopted