Test Suite Reduction for Regression Testing of Simple Interactions between

Two Software Modules

Dmitry Kichigin

Software regression integration testing

Software integration testing – testing of interactions between software elements being integrated

Software regression testing – retesting of software to verify that modifications have not caused unintended effects and that the software still complies with its specified requirements

Problem: high cost of regression testing

Solutions:• Automation of testing• Test suite reduction

Test suite reduction

“ seeks to reduce the number of test cases in a test suite while retaining a high percentage of the original suite’s fault detection effectiveness “

This reduces:• The costs associated with test suite maintenance• The costs of software execution on the tests

Result: reduction of overall costs associated with regression testing

Traditional approach

Involves a static analysis and/or instrumentation of a source code and consists of 3 main steps:

1. Instrumentation of software source code

2. Execution of software on a test suite and collecting coverage information

3. Using coverage information to reducing the test suite

Existing methods:

- Interface Mutation (Mutation Analysis)

- Structural methods:– Control-flow based– Data-flow based

Drawbacks

Recent trends in software development may limit their applicability:

• Component-based software development and use of commercial off-the-shelf (COTS) components:– the source code is unavailable

• Increasing size and complexity of the software:– source-code based coverage analysis is expensive and

hard to conduct for large software

Proposed solution

Test Suite Reduction Method for Regression Testing of Simple Interactions between Two Software Modules:

• Based on interactions behaviour monitoring• Uses sequences of interface function calls to analyse

interactions behaviour• Takes into account function parameters• Does not require access or instrumentation of a source

code

Definitions

Simple interaction - interaction via interface of functions which can take only scalar parameters

Interface functions are the functions, which form a part of module’s programming interface.

Interaction trace of interaction between module A and module B is a sequence of interface functions of module B, which are invoked by module A during software execution on a test case t.

…FILE* fp = fopen(…);for (int i=0; i<3; i++){ fread(…);}fclose(fp);…

…fopenfreadfreadfreadfclose…

Algorithm

T – original test suiteT’ – reduced test suite, T’ Tt – next test case from the

original test suite, t T

Mt – model of interaction behaviour on the test case t

MT’ – set of models of interaction behaviour on test cases from T’

Main steps:1. Pick the next test case from the

original test suite and execute the program

2. Build modules interactions model Mt

3. If Mt MT’, then add Mt to MT’ and add test case t to T’

4. Run until T is exhausted

yes

no

t := next(T)

Mt := build_model(t)

Mt MT’

yes

finish

no

? Mt MT’

?T ==

MT’ := MT’ Mt

T’ := T’ t

Mt empty?

Mt MT’

Interaction Behaviour Model

K-length sequence of interface functions is an arbitrary continuous sequence of length K, which can be located in an interaction trace.

Model of interaction between module A and module B on a test t - set of all possible K-length sequences, which can be located in the interaction trace for the interaction between A and B on t.

Use “sliding window” technique to build the model:

fopen fwrite fwrite fopen fwrite fclose fwrite fclose

fopen fwrite fwrite fopenfwrite fwrite fopen fwritefwrite fopen fwrite fclosefopen fwrite fclose fwritefwrite fclose fwrite fclose

Equality relation between models

Two given models are equal <=> their sequence sets are equal

Two sequences of functions are equal <=> their elements are equal pairwise

Function calls are equal when:• the functions have the same names, • the values of nominal parameters are the same, and • the values of numerical parameters belong to the same intervals.

]))interval(),(interval(),())name(),(name([

]))interval(),(interval(),())name(),(name([

),(

11

11

21

22 11 1

12

11 22 1

21

m

ni

kk

n

i

kk

Ms Ms

K

k

kk

m

ni

kk

n

i

kk

Ms Ms

K

k

kk

iiii

iiii

xyxyff

yxyxff

MM

Equality relation between models with taking into account parameter values

Function scalar parameters

Nominal Numerical

Nominal: • definition area is a finite set• no order relation and distance defined • equality relation: (x,y), where - Kronecker symbol

Numerical:• definition area is R• both order relation and distance are defined• equality relation: ))interval(),(interval(),( yxyx

Results of the experiment:level of test suit reduction

0

5

10

15

20

25

30

35

40

45

50

Original test suite size

Redu

ced

test

sui

te s

ize

0

10

20

30

40

50

60

70

Original test suite size

% o

f red

uctio

n

Subject programs: GNU Assembler & the C standard library

Results of the experiment:fault detection ability

0

5

10

15

20

25

Original test suite size

Num

ber o

f det

ecte

d fa

ults

Original test suiteOur methodRandom reduction method

0

20

40

60

80

100

120

Original test suite size

Erro

r det

ectio

n ra

te (i

n %

)Our methodRandom reduction method

Subject programs: GNU Assembler & the C standard library

Summary

The new test suite reduction method was presented, which: • does not require access or instrumentation of a source code• uses sequences of interface function calls as behaviour

characteristic• takes into account values of function parameters

The experimental evaluation demonstrated:• can successfully reduce the size of a test suite while

retaining its fault detection ability• the method outperforms the random test suite reduction

method

Thank you

Questions

?