Specification-based

regression test selection

with risk analysis Zheng-Wen Shen

2005/03/14

Reference

Specification-based regression test selection with

risk analysis

Yanping Chen, Robert L. Probert, D. Paul Sims

Source IBM Centre for Advanced Studies

Conference archive

Proceedings of the 2002 conference of the Centre

for Advanced Studies on Collaborative research

table of contents

Outline

1. Introduction

2. Background

3. Specification-based test case selection

4. Risk-based test case selection

5. Evaluation

6. Conclusions

1. Introduction

Regression testing is essential to ensure software

quality.

Regression test selection attempts to select a

cost-minimized subset of test cases.

White-box (code-based) V.S. Black-

box (specification-based)

Code-based regression test selection is good for unit testing, but it has a scalability problem.

Code-based regression techniques are not suitable for testing larger components at more abstract level.

Code-based regression techniques require testers to access and understand the code.

Code-based regression techniques are language-dependent.

Two kinds of regression tests

Targeted Tests, which ensure that important

current customer features are still supported

adequately in the new release.

Safety Tests, which are risk-directed, and ensure

the potential problem areas are properly handled.

2. Background

2.1 Regression testing

2.2 Controlled Regression Testing Assumption

(CRTA)

2.3 Risk Analysis

2.4 Requirement traceability

2.1 Regression testing

Given a program P, a modified version P’, and a set T of test cases used previously to test P, regression analysis and testing techniques attempt to make use of a subset of T to gain sufficient confidence in the correctness of P’ with respect to behaviors from P retained in P’.

With selective re-test techniques, we only return those test cases that test the affected entities of the modified program.

2.2 Controlled Regression Testing

Assumption (CRTA)

When P’ is tested with T, we hold all factors that might influence the output of P’, except for the code in P’, constant with respect to their states when we tested P with T.

Ideally, regression tests should be run while CRTA holds.

Testers must identify and document uncontrollable factors and the test cases that are potentially affected by them.

2.3 Risk Analysis

Risk is anything that threatens the successful

achievement of a project’s goals.

Using risk metrics to quantitatively measure the

quality of a test suite.

Risk Exposure model by Stale Amland.

RE(f) = P(f) * C(f)

1. P(f), the probability of a fault being present.

2. C(f), the cost (consequence or impact) of a

fault in the corresponding function if it occurs

in operation.

2.4 Requirement traceability

Traceability is a simple, common-sense book-

keeping property that can prevent a wide range

of problem.

Know which test case verifies a given requirements

attribute.

We use the activity diagram to represent the

desired system behavior.

The activity diagram and Test suite for the Get_Quote feature

Traceability matrix between the activity diagram and test suite

3. Specification-based test case

selection

3.1 Changes in code

3.2 Changes in specification

3.1 Changes in code

Changes only happen in the implementation without affecting the specification.

For a developer, any changes in code should be documented in a code change history.

For a tester, traceability needs to be established between the test case and the log of defects.

Test profile

The tester can choose test cases using the traceability matrix.

3.2 Changes in specification

Changes happen in requirements or design, the activity diagram need to be modified accordingly.

In a control flow graph (CFG), nodes are used to represent statements, and edges represent the control flow between the statement within a procedure.

We find that the activity diagram is quite similar to the CFG except that the nodes of activity diagram describe activities instead of statements.

A CFG-based regression test selection technique

Edge-coverage matrix for test suite T on CFG C

Regression analysis with the activity

diagram

1. We traverse the activity diagram to identify

affected edges.

2. Then we select test cases that execute the

affected edges based on the traceability matrix

to create our Targeted Tests.

4. Risk-based test case selection

4.1 Motivation for risk-based selection.

4.2 Model-based selection of Safety Tests.

4.3 Risk-based end-to-end test scenario selection.

4.1 Motivation for risk-based

selection

If the developer changes code that cannot be

identified by a test profile, and he does not

update the code change history, the tester might

miss some defects.

To achieve enough confidence, we would like to

include some test cases in addition to the

Targeted Tests for our regression test suite.

Useful in two situations

1. After running the Targeted Tests, if time and

resources are available, testers may want to

return some test cases to gain more confidence

in the modified system.

2. The time to delivery may be extremely short,

Safety Tests provide some assurance that the

remaining defects in the release will not bring

about serious failures.

4.2 Model-based selection of Safety

Tests

Uses Risk Exposure model for risk analysis.

There are four main steps in our approach.

1. Assess the cost of each test case.

2. Derive severity probability for each test case.

3. Calculate Risk Exposure for each test case.

4. Select Safety Tests.

Step 1. Assess the cost of each test case.

Cost means the cost of the requirements attributes that this test case covers.

Two kinds of costs will be taken into consideration.

1. The consequences of a fault as seen by the customer, that is, losing market share because of faults.

2. The consequences of a fault as seen by the vendor, that is, high software maintenance cost because of faults.

Cost of test cases

Step 2. Derive severity probability for each

test case

How important or serious the defect is.

Step 3. Calculate Risk Exposure for each test

case

Step 4. Select Safety Tests

We choose test cases that have the highest value

of RE(t) from the non-targeted tests to reach

our coverage target.

4.3 Risk-based end-to-end test

scenario selection

End-to-end scenarios simulate common user

profiles of system uses. Some are equivalent to

use cases.

More customer-directed.

Highly effective at finding regression faults.

One end-to-end consists of several test cases.

Every test case tests one or more requirements

attributes.

Selection strategy obeys two rules

1. Select scenarios that cover the most critical test

cases.

2. Have the suite of scenarios cover as many test

case as possible.

Traceability between test cases and scenarios

Step 1. Calculate Risk Exposure for each

scenario

Step 2. Select scenario that has the highest RE(s)

Step 3. Update table and rebuild table.

Step 4. Repeat Step 2 and Step 3 until we run

out of time and resources.

Step 1. Calculate Risk Exposure for

each scenario

RE (s) = ΣRE (t)i , {1 ≤ i ≤ n | test case i is

covered by this scenario}

Step 2. Select scenario that has the

highest RE(s)

The scenario with the highest RE(s) covers the

most critical test cases, that is, has the highest

coverage of test cases.

According to our selection rules, it should be

included in the regression test suite.

For example: s001 is put first into the regression

test suite.

Step 3. Update table and rebuild

table.

Rebuilt table of Risk Exposure for

scenarios

Step 4. Repeat Step 2 and Step 3 until

we run out of time and resources

The size of our test suite of scenarios is

dependent on the time and resources available.

Regression testing is terminated whenever we

run out of time and resources.

The final selected regression test suite is the

union of the Targeted Tests and the Safety Tests.

5. Evaluation

We applied our approach to three components

of IBM WebSphere Commerce 5.4. Each

component was owned by one experienced

tester. (306 test cases)

65 defects were opened in the first pass.

There were 9 defects found, which failed 28

test cases in the second pass.

Three assessment factors

1. Effectiveness

2. Cost-efficiency

3. Sensitivity to Risk

6. Conclusions

This technique is Customer-oriented and also Risk-

based.

It provides methods to obtain both Targted Tests

and Safety Tests.

Future work

To decide when to stop regression testing.

Implementing our approach in a production test

environment.