Testing the programs

In this part we look at• classification of faults• the purpose of testing• unit testing• integration testing strategies• when to stop testing

Concept change!!

• Many programmers view testing as a demonstration that their program performs properly.

• The idea of demonstrating correctness is really the reverse of what testing is all about.

• We test a program to demonstrate the existence of a fault! Because our goal is to discover faults, we consider a test successful only when a fault is discovered or a failure occurs as a result of our testing procedures.

Classification of faults

• In an ideal world, we produce programs where everything works flawlessly every time. Unfortunately this is not the case!

• We say that our software has failed, usually when its behaviour deviates from the one described in the requirements.

• First we identify the fault, i.e. determine what fault or faults caused the failure. Next we correct the fault by making changes to the system so that the fault is removed.

Classification of faults

• Why do we classify faults?– In order to improve our development

process!

• We would like to match a fault to a specific area of our development process.– In other words, we would like our

classification scheme to be orthogonal.

IBM Orthogonal Defect Classification

Fault type MeaningFunction Fault that affects capability, end-user interfaces,

product interfaces, interface with hardware architecture, or global data structure

Interface Fault in interacting with other components or drivers via calls, macros,control blocks or parameter lists

Checking Fault in program logic that fails to validate data and values properly beforethey are used

Assignment Fault in data structure or code block initialization.

IBM Orthogonal Defect Classification

Fault type Meaning

Timing/serialization Fault that involves timing of shared and real-time resources

Build/package/merge Fault that occurs because of problems in repositories, management changes,or version control

Documentation Fault that affects publications and maintenance notes

AlgorithmFault involving efficiency or correctness of algorithm or data structure butnot design

Hewlett-Packard fault classification

Testing steps

Views of Test Objects

• Black (opaque) box– In this type of testing, the test object is

viewed from the outside and its contents are unknown.

– Testing consists of feeding input to the object and noting what output is produced.

– The test's goal is to be sure that every kind of input is submitted and that the observed output matches the expected output.

Black Box (example)• Suppose we have a component that accepts as

input the three numbers a, b, c and outputs the two roots of the equation ax2 + bx + c = 0 or the message “no real roots”.

• It is impossible to test the component by submitting every possible triple of numbers (a,b,c).

• Representative cases may be chosen so that we have all combinations of positive, negative and zero for each of a, b, and c.

• Additionally we may select values that ensure that the discriminant, (b2 – 4ac) is positive, zero, or negative.

Black Box (example)

• If the tests reveal no faults, we have no guarantee that the component is fault-free!

• There are other reasons why failure may occur.

• For some components, it is impossible to generate a set of test cases to demonstrate correct functionality for all cases.

White box testing

• White (transparent) box– In this type of testing, we use the

structure of the test object to test in different ways.

– For example, we can devise test cases that execute all the statements or all the control paths within the component(s).

– Sometimes with many branches and loops it may be impractical to use this kind of approach.

Unit testing

Our goal is to find faults in components. There are several ways to do this:

• Examining the code– Code walkthroughs– Code inspections

• Proving code correct• Testing components

Examining the Code

• Code walkthroughs are an informal type of review.

• Your code and documentation is presented to a review team and the team comments on their correctness.

• You lead and control the discussion. The focus is on the code not the programmer

Examining the Code

• A code inspection is similar to a walkthrough but is more formal.

• Here the review team checks the code and documentation against a prepared list of concerns.– For example, the team may examine the

definition and use of data types and structures to see if their use is consistent with the design and with system standards and procedures.

• The team may review algorithms and computations for their correctness and efficiency

Proving code correct

• Proof techniques are not widely used. It is difficult to create proofs, these can sometimes be longer than the program itself!

• Additionally customers require demonstration that the program is working correctly.

• Whereas a proof tells us how a program will work in a hypothetical environment described by the design and requirements, testing gives us information on how the program works in its actual operating environment.

Testing components• Choosing test cases: To test a component, we select

input data and conditions and observe the output. • A test point or case is a particular choice of test

data.• A test is a finite collection of test cases. • Create tests that can convince ourselves and our

customers that the program works correctly, not only for the test cases but for all input.– We start by defining test objectives and define tests

designed to meet a specific objective.– One objective can be that all statements should execute

correctly another can be that every function performed by the code is done correctly.

Testing components

• As seen before we view the component as either a “white” or “black” box. – If we use “black” box testing, we supply all

possible input and compare the output with what was expected.

– For example, with the quadratic equation seen earlier we can choose values for the coefficients that range over combinations of positive, zero and negative numbers.

– Or select combinations based on the relative sizes e.g. a > b > c, b > c > a, c > b > a, ...etc

Testing components

• We can go further and select values based upon the discriminant.

• We even supply non-numeric input to determine the program's response.

• In total we have four mutually exclusive types of test input.

• We thus use the test objective to help us separate the input into equivalence classes.

Equivalence classes• Every possible input belongs to one of theclasses. That is, the classes cover the entireset of input data.• No input datum belongs to more than oneclass. That is, the classes are disjoint.• If the executing code demonstrates a faultwhen using a particular class member is usedas input, then the same fault can be detectedusing any other member of the class as input.That is , any element of the class represents allelements of that class.

Equivalence classes

• It is not always easy of feasible to tell if the third restriction can be met and it is usually rewritten to say:– if a class member is used to detect a

fault then the probability is high that the other elements in the class will reveal the same fault.

Common Practice• Usually 'white' box and 'black' box testing are

combined. • Suppose we have a component expects a

positive input value. Then, using 'black' box testing, we can have a test case for each of the following:– a very large positive integer– a positive integer– a positive, fixed point decimal– a number greater than 0 but less than 1– a negative number– a non numeric character

Common Practice

• Using 'white' box testing we can chose one or more of the following:– Statement testing: Every statement in the

component is executed at least once in some test.

– Branch testing: For every decision point in the the code, each branch is chosen at least once in some test.

– Path testing: Every distinct path through the code is executed at least once in some test.

White box testing• Statement testing

– choose X > K that produces a +ve result– 1-2-3-4-5-6-7

• Branch testing – choose two test cases to traverse each branch of the decision

points– 1-2-3-4-5-6-7– 1-2-4-5-6-1

• Path testing – four test cases needed– 1-2-3-4-5-6-7– 1-2-3-4-5-6-1– 1-2-4-5-6-7– 1-2-4-5-6-1

Integration testing

• When each component has been completed and tested, we can then combine them into a working system.

• This integration must be planned and coordinated so that in the case of a failure, we would be able to determine what may have caused it.

• Suppose we view the system as a hierarchy of components (shown on the following slide).

Integration testing

Bottom-up integration

Top-down integration

Big-bang integration

Sandwich integration

Comparison of integration strategies

When to Stop Testing?

Fault Seeding• We intentionally insert or “seed” a know

number of faults in a program. • Then another member of the team

locate as many faults as possible. • The number of undiscovered seeded

faults act as an indicator of the total number of faults(unseeded and seeded) remaining in the program.

• We say:

Fault Seeding

Problems:• It is assumed that the seeded faults are of

the same kind and complexity as the actual faults in the program. – This is difficult to do since we do not know what

are the typical faults until we have found them.

• We can attempt to overcome this by basing the seeded faults on historical data about previous faults. – This, however requires that we have built

similar systems before.

Fault SeedingSolution• Use two independent groups, Test Group 1 and

Test Group 2. • Let x be the number detected by Group 1 and y

the number detected by Group 2. • Some faults will be detected by both groups

say q, such that q <= x and q <= y.• Finally let n be the total number of faults in the

program which we want to estimate.• The effectiveness of each group can be given

by E1 = x/n and E2 = y/n

Fault Seeding• The group effectiveness measures the

group's ability to detect faults from among a set of existing faults.

• If we assume that Group 1 is just as effective at finding faults in any part of the program as in any other part, – we can look at the ratio of faults found by Group

1 from the set of faults found by Group 2.– E1 = x/n = q/y– E2 = y/n = q/x– Which gives n = (xy)/q = q/(E1 * E2)

Confidence in the Software

• If we seeded a program with S faults and we claim that the code has only N actual faults.

• Suppose we tested until all S faults have been found as well as n non-seeded faults, then a confidence level can be calculated as

1 , if n > NC = S/(S – N + 1) , if n N

Confidence in the Software

• With that approach we cannot predict the level of confidence until all the seeded faults are detected.

• Richards (1974) suggests a modification, where the confidence level can be estimated whether or not all the seeded faults have located. 1 , if n > N

• C= S S + N + 1 ,if n <= N s -1 N + s

Other stopping criteria

• We can use the test strategy to determine when to stop. – If we are doing statement, branch or path

testing, we can track how many statements, branches or paths yet need to be executed and gauge our progress in terms of those statements, branches or paths left to test.

• There are many tools that can calculate these coverage values for us.