coverage and - fbk · 2009-11-19 · coverage tool features compile time: automatic instrument code...

Coverage and Mutation testing tools

So#ware Analysis and Tes1ngCu Nguyen Duy (cunduy at :k dot eu)

Alessandro Marche?o (marche?o at :k dot eu)

Paolo Tonella (tonella at :k dot eu)

Mariano Ceccato (ceccato at :k dot eu)

Academic Year 2009-‐2010

mailto:[email protected]








Coverage testing (recall)

Coverage measures describe the degree to which a program has been testedtell the effectiveness in terms of coverage of the test

set, help improve software qualityinform quantitatively the project manager about the

progress of testingMany type of coverage measures

statementspathsmethods, classesrequirement specifications, etc. 2

Coverage tool features

Compile time: automatic instrument code for coverage recording

Runtime: structure-based coverage recording lines, blocks, conditions, methods and classes

Reportingquantitative coverage measure

percentage of code executedmost executed v.s. never executed code

visual navigationquickly navigate to code that is not executed to improve

the test set 3

Coverage tools

4source code in not-widely-used languages on a variety ofplatforms [30].

3.2. Instrumentation overhead

Coverage-testing tools capture coverage information by moni-toring program execution. Execution is monitored by insertingprobes into the program before or during its execution.A probe is typically a few lines of code that, when executed,generate a record or event that indicates that programexecution has passed through the point where the probe islocated. There are two kinds of overhead associated withinstrumenting a program with probes: the off-line overheadof inserting probes into the program, and the run time over-head of executing the probes to record the execution trace.

3.2.1. Off-line program analysis and instrumentationoverhead

Source code instrumentation, used by most of the toolsincluding BullseyeCoverage, Parasoft Insure!!, Intel CodeCoverage Tool, Semantic Designs and TestWork, requiresrecompilation, but provides more direct results and is moreadaptable to a wide variety of processors and platforms. Itcannot be used when the source code is not available, as isoften the case for third party code. C/C!! tools such asDynamic Memory Systems’ Dynamics, use runtime instru-mentation, which makes them feasible in a productionenvironment. They may be more efficient in terms of compi-lation time, but less portable. The Java coverage tool Koalog

Code Coverage does not require instrumentation, and there-fore no recompilation is needed [25]. It operates with the pro-duction binaries using the Java Debug Interface, which is partof the Java Platform Debugger Architecture (JPDA). KoalogCode Coverage is platform independent, but requires aJPDA compliant Java Virtual Machine (JVM). Agitar’s Agita-tor runs the code in a modified JVM, also using a dynamicinstrumentation approach. eXVantage uses source code instru-mentation for C/C!! and bytecode instrumentation for Java.As compared to the other 16 tools, it has the highest off-lineinstrumentation overhead because it analyzes the program insuch a way that it can select the least number of probes tobe inserted into the target program.

3.2.2. Run-time instrumentation overheadCompanies that provide tools for system software or embeddedsoftware tend to focus more on reducing run-time overhead, sothat their tools can be usable in real-time environments, e.g.CodeTEST [18]. TCAT claims that its TCAT C/C!!Version 3.2 maintains its overhead for execution size ratio at1.1–1.8 and execution speed ratio at 1.1–1.5 [29]; SemanticDesigns claims 1.1–1.3, varying according to language andcompiler, among the best in our survey. Clover claims thattheir execution speed overhead is highly variable, dependingon the nature of the application under test, and the nature ofthe tests. Typical execution speed ratio is 1.2–1.5. eXVantagehas different versions for different platforms, but claims aratio of 1.01 for versions optimized real-time, in some environ-ments, based on initial trials (Table 2) [24].

3.3. Additional features

Coverage testing tools can be used to assist in debugging, andsome of the coverage tools provide debugging assistance, suchas Agitar, Dynamic, JCover, Jtest and Semantic Designs. Eachuses a different solution. For example, Agitar provides a snap-shot and stack trace to help developers to track the cause ofbugs. JCover has the ability to do coverage differencing andcomparison to expose the erroneous code. Semantic Designsprovides slicing and dicing operations on test coverage datavia the GUI to allow code executed/not executed by arbitrarycombinations of test runs to be easily isolated visually.eXVantage uses a dynamic execution slicing approach. Itcreates an execution slice for each test case and reads resultsfrom a testing oracle to generate a bug localization reportautomatically whenever a failed test is detected [8].Coverage testing tools can also be used for program profil-

ing to identify heavily executed parts of programs. Profilingdata can be used in compiler optimization, program refactor-ing, performance-related debugging, etc. Many tools, includ-ing eXVantage, CodeTEST, Dynamic Code Coverage,JCover, PurifyPlus and Semantic Designs, support this feature.

TABLE 1: coverage tools and the languages to which they apply(alphabetical by tool name)

Tool name C!!/C Java Other

Agitar [14] X

Bullseye [15] X

Clover [16] X .net

Cobertura [17] X

CodeTEST [18] X

Dynamic [19] X

EMMA [20] X

eXVantage [21] X X

Gcov [22] X

Intel [23] X FORTRAN

JCover [24] X

Koalog [25] X

Parasoft (C!!test) [26] X

Parasoft (Jtest) [26] X

PurifyPlus [27] X X Basic, .net

Semantic Designs (SD)

[28]

X X C#, PHP,

COBOL,

PARLANSE

TCAT [29] X X

of 9 YANG et al.

THE COMPUTER JOURNAL, 2007

source: doi:10.1093/comjnl/bxm021

Clover

Is a commercial coverage toolIntegration with a wide range of development

environment: Ant, Eclipse, Maven, IntelliJInstallation

Eclipse update: http://update.atlassian.com/eclipse/clover/

Get an evaluation license

5

http://update.atlassian.com/eclipse/clover/




Clover license setting

6

Enable Clover to an Eclipse project

Right click on the project

Select Clover > Enable on this project

7

Clover Eclipse views

8

Run Unit test with Clover

In the Coverage Explorer view

Right click on the project

Select Run With Clover As > JUnit Test

9

Clover coverage report

10


10

report menu


10

report menu

projectinfo


10

report menu

projectinfo

overall coverage


10

report menu

projectinfo

overall coverage

testresults


10

report menu

projectinfo

overall coverage

testresults

coverage by package


10

report menu

projectinfo

overall coverage

testresults

tree mapcoverage by package

Exercise 1

Open project: Exercise-4-CurrentAccountEnable Clover on this projectExecute Unit TestGenerate coverage reportReport the test coverage of each class,

identify which lines of code have not executed

Task: improve the test cases to reach 100% coverage

11

Exercise 2

Similar to exercise 1, with Exercise-3-MathOperators

12

Mutation testing

Introduction

14

program

Test CaseTest CaseTest

CaseTest CaseTest

Case

Introduction

14

program


CaseTest CaseTest

Case

How good these test cases are?- fault detection - coverage

Introduction

14

program


CaseTest CaseTest

Case


what if no fault foundwhat if 100% covered

Introduction

14

program


CaseTest CaseTest

Case


what if no fault foundwhat if 100% covered

- mutant core with mutation testing

Mutation testing terms

Mutant: a copy of the original program with a small change (seeded fault)

Mutation operator: applied to make change (automatically) to the original program

Mutant killed: if its behaviors/outputs differ from those of the original program

Mutant score: of a test case = number of killed mutants / total number of mutants

15

Mutation operators

Method-level operators, example:Arithmetic operator, e.g. “-” -> “+”Relational operator, e.g. “ > ” -> “ < ”Conditional operator, e.g. “ && ” -> “ || ”

Class-level operators, example: EncapsulationInheritancePolymorphism

16http://cs.gmu.edu/~offutt/mujava/mutopsClass.pdf

http://cs.gmu.edu/~offutt/mujava/mutopsMethod.pdf

http://cs.gmu.edu/~offutt/mujava/mutopsClass.pdf




Mutant creation

17

program

mutant 1

mutant 2

mutant N

Mutation operator

Mutant creation

17

program

mutant 1

mutant 2

mutant N

Mutation operator

package math;

public class Add { public static int sum (int a, int b){ return a+b; } public static double sum (double a, double b){ return a+b; } public static long sum (long a, long b){ return a+b; }}

Mutant creation

17

program

mutant 1

mutant 2

mutant N

Mutation operator

package math;


package math;

public class Add {

public static int sum(int a, int b) { return ++a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return a + b; }}

Mutant score

18

program

mutant 1

mutant 2

mutant N

Mutant score

18

program

mutant 1

mutant 2

mutant N

TC1

Mutant score

18

program

mutant 1

mutant 2

mutant N

TC1

✔

✘

✔

✔

Mutant score

18

program

mutant 1

mutant 2

mutant N

TC1

✔

✘

✔

✔

MS(TC1) = killed mutants/ N

Example

19


public class Add { public static int sum(int a, int b) { return ++a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return a + b; }}

public class Add { public static int sum(int a, int b) { return a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return --a + b; }}

public class Add { public static int sum(int a, int b) { return a + b; } public static double sum(double a, double b) { return a - b; } public static long sum(long a, long b) { return a + b; }}

Example

19


public class Add { public static int sum(int a, int b) { return ++a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return a + b; }}

public class Add { public static int sum(int a, int b) { return a + b; } public static double sum(double a, double b) { return a + b; } public static long sum(long a, long b) { return --a + b; }}

public class Add { public static int sum(int a, int b) { return a + b; } public static double sum(double a, double b) { return a - b; } public static long sum(long a, long b) { return a + b; }}

TC1:Add o = new Add();print(o.sum(1,2));print(o.sum(1.0,2.0));

MutantScore(TC1) = ?

MuClipse

An Eclipse plugin for MuJava (http://cs.gmu.edu/~offutt/mujava/)

Main features:GGenerate mutantsRun testsReport mutant score

20

http://cs.gmu.edu/~offutt/mujava/




MuClipse: generate mutantsEclipse: Run > Run Configurations

21

MuClipse: run test

22

Eclipse: Run > Run Configurations

Exercise 1

Install MuClipsehttp://selab.fbk.eu/swat/projects/muclipse.tgz

Add java library to Exercise-3-MathOperatorshttp://selab.fbk.eu/swat/projects/mu-lib.tgz

Report the mutant score of the test case: TestCase1_add_Junit3_a of the project use only traditional mutation operators

Improve the mutant score of the test case23

http://selab.fbk.eu/swat/projects/muclipse.tgz




Exercise 2

Compare the coverage of the test case: TestCase1_add_Junit3_a before and after being improved using mutant score

Hints:Enable CloverExecute only the test case and report the

coverage

24

coverage and - fbk · 2009-11-19 · coverage tool features compile time: automatic instrument code...

Documents