occf: a framework for developing test coverage measurement tools supporting multiple programming...

ICST Testing Tools Track 2013

2013/03/20

Kazunori Sakamoto, Kiyofumi Shimojo, Ryohei Takasawa, Hironori Washizaki, Yoshiaki Fukazawa

Waseda University

Dept. Computer Science and Engineering

Motivating Example

• Coverage based tools have differences

2013/3/20 ICST Testing Tools Track 2013 2

int min(int a, int b) {

return a < b ? a : b

}

@assert void testMin() {

assertEquals(1, min(1, 2));

}

• Many tools say statement coverage is

100% (1/1)

• EMMA says line coverage

is 86% (6/7)

Java

JUnit

• Tools use different measurement units

Comparison of 25 coverage tools


Tool Mecha nism

Hierar chical view

Free State ment

Deci sion

Cond ition

Condit ion/D ecision

Lang uages

Cobertura Binary ✔ ✔ ✔ ✔ Java

EMMA Binary ✔ ✔ ✔ ✔ Java

JCover Code ✔ ✔ ✔ Java, Groovy

Clover Code ✔ ✔ ✔ Java

Agitar Binary ✔ ✔ ✔ Java

OpenCover Processor ✔ ✔ ✔ ✔ C#

NCover Source ✔ ✔ ✔ C#

dotCover Binary ✔ ✔ C#

gcov Compiler ✔ ✔ ✔ C++

COVTOOL Code ✔ ✔ C++



Tool Mecha nism

Hierar chical view

Free Statement

Decision

Condition


Lang uages

Bullseye Code ✔ ✔ ✔ ✔ ✔ C++

Intel Code Coverage Tool

Compiler ✔ ✔ Java

Squish Coco Code ✔ ✔ ✔ ✔ ✔ Java

TCAT Code ✔ ✔ C++, Java

Parasoft Test

Code ✔ ✔ ✔ ✔ ✔ C++, C# Java

PurifyPlus Binary ✔ ✔ ✔ ✔ ✔ C++, C# Java

Semantic Designs

Code ✔ ✔ ✔ C++, C# Java, etc.

Coverage Validator

Code ✔ ✔ ✔ C++, C# Java, etc.



Tool Mecha nism

Hierar chical view

Free Statement

Decision

Condition


Lang uages

ScriptCover Code ✔ ✔ JavaScript

Coverage.py Processor ✔ ✔ ✔ Python

rcov Processor ✔ ✔ ✔ Ruby

SimpleCov Processor ✔ ✔ ✔ Ruby

Devel::Cover Processor ✔ ✔ ✔ ✔ ✔ Perl

xdebug Code ✔ ✔ ✔ PHP

LuaCov Processor ✔ ✔ Lua

• Tools support different criteria and langs

• Differences between tools cause problems (*) *) Rüdiger Lincke, Jonas Lundberg and Welf Löwe: “Comparing Software Metrics Tools,” ISSTA '08 Proceedings of the 2008 international symposium on Software testing and analysis, pp. 131-142, 2008.

Problems and tool solutions

• Differences between tools

– Hard to measure coverage of software using multiple langs such as Web apps

– OCCF aids to provide consistent criteria

• High development costs

– Hard to develop coverage tools supporting new programming languages

– OCCF aids to develop new coverage tools


7

int func(int a) { if (branch(2, a == 0)) { stmt(0); printf("a == 0"); } else if (branch(3,false)){ stmt(1); printf("a != 0"); } }

• Inserts automatically • Outputs coverage info • Has no side effect • Measures completely

int func(int a) { if (a == 0) { printf("a == 0"); } else if (false) { printf("a != 0"); } }

Statement and

branch coverage

Function

printf("a == 0") printf("a != 0")

Statement

stmt (0) stmt (1)

Statement

Statement

Statement

int func(int a)

Inserts into AST

Abstraction syntax tree(AST)

void stmt(int id) { RECORD_STATEMENT(id); }

Measurement approach

2013/3/20 ICST Testing Tools Track 2013

OCCF

Architecture


AST Generator

C Java

AST Transformer

OCCF Core

C Java

Coverage Report

Tool

Fault Localization

Tool

Test Case Minimization

Tool

Python

Python Parsers (Code2Xml)

Cold spot

Trans formers

Tools

Process and design


OC

CF

Testin

g F

ram

ew

ork

~~~~ ~~~~ ~~~~

~~~~ ~~~~ ~~~~

Production code

Test code

Covera

ge

Rep

ort

Tool

Fau

lt Localiz

atio

n

Tool

Test C

ase

Min

imiz

atio

n

Tool

0101 0001 1110 1010

coverage data

~~~~ ~~~~ ~~~~ ~~~~

~~~~ ~~~~ ~~~~ ~~~~

Pass Fail Pass Fail

Test Results

Instrumentation code

Modified Production

code

Modified Test code Instrumentation

code

library

Instrumentation library

Process and design in detail


AST G

en

era

tor

AST F

inder

AST In

serte

r

AST AST AST

source code

~~~~ ~~~~ ~~~~

Cod

e G

en

era

tor

source code

~~~~ ~~~~ ~~~~ ~~~~

Java

C

Python

Instrumentation code

XML object in .NET Framework

Implementation

• OCCF is developed in C# – Works on .NET Framework and Mono

• AST Generator utilizes existing parser libraries • ANTLR for C, C#, Java, JavaScript, Lua

– http://www.antlr.org/ – Reuse existing grammar files on the repository

• Standard library (parser module) for Python 2, 3 – http://docs.python.org

• ruby_parser for Ruby 1.8, 1.9 – https://github.com/seattlerb/ruby_parser

• srcML for C++ – http://www.sdml.info/projects/srcml/


http://www.antlr.org/

http://www.antlr.org/

http://docs.python.org/

http://docs.python.org/

https://github.com/seattlerb/ruby_parser

https://github.com/seattlerb/ruby_parser

http://www.sdml.info/projects/srcml/

Demo

• Insert instrumentation code in eight languages: C, C++, C#, Java, JavaScript, Python, Ruby, Lua

• Let me show four materials in Java

– Original code

– Original AST in XML

– Modified AST in XML

– Modified code


Hot spot in OCCF

• Finding nodes to be inserted – Providing enhancements for LINQ to XML

e.g. root.Descendants("statement")

.Select(e => e.FirstElement())

• Instrumentation library – Recording statements and branches

• Text file, Binary file, Shared memory, TCP/IP, …

– Basically, implemented in each language

• Function call for instrumentation library – Function calls are inserted into each langs


Finding statements in C#


statement : declaration_statement | labeled_statement | embedded_statement ; embedded_statement : block | selection_statement // if, switch | … ; Grammar

var decls = root.Descendants("declaration_statement"); var stmts = root.Descendants("embedded_statement") .Where(e => e.FirstElement().Name() != "block"); return stmts.Concat(decls); Finder

Finding statements in JavaScript


statement : statementBlock | variableStatement | emptyStatement | expressionStatement | ifStatement | … ; Grammar

return root.Descendants("statement") .Select(e => e.FirstElement()) .Where(e => e.Name() != "statementBlock") .Where(e => e.Name() != "labeledStatement") .Where(e => e.Name() != "emptyStatement");

Finder

Finding branches in C


selection_statement : 'if' '(' expression ')' statement | 'switch' '(' expression ')' statement ; iteration_statement : 'while' '(' expression ')' statement | … ;

Grammar

return root.Descendants("selection_statement") .Where(e => e.FirstElement().Value == "if") .Select(e => e.NthElement(2));

Finder

We can consider while as a branch

Finding branches in Python


if_stmt : ’if’ test ’:’ suite (’elif’ test ’:’ suite)* (’else’ ’:’ suite)? ; while_stmt : "while" test ":" suite ("else" ":" suite)? ; Grammar

var ifs = root.Descendants("if_stmt") .SelectMany(e => e.Elements("test")); var whiles = root.Descendants("while_stmt") .SelectMany(e => e.Elements("test")); return ifs.Concat(whiles); Finder

Commonality between langs

• Finding statements

– Most of grammars have similar statement nodes

1. Finding statement nodes by a name

2. Excluding unused statement nodes

• Should consider what is a statement

• Finding branches

– Most of grammars have branch statements

1. Finding branch statement nodes by a name

2. Extracting condition nodes (or find directory)

• Should consider what is a branch (if, while, … ?)


Customization

• Changing AST Finders

e.g. Extracting only if statements


var ifs = root.Descendants("if_stmt") .SelectMany(e => e.Elements("test")); var whiles = root.Descendants("while_stmt") .SelectMany(e => e.Elements("test")); return ifs.Concat(whiles); Finder

return root.Descendants("if_stmt") .SelectMany(e => e.Elements("test"));

Finder

Sample applications for FizzBuzz


String answer(int n) { if (n % 15 == 0) { // should return fizzbuzz return "fizzzbuzzz"; } else if (n % 3 == 0) { return "fizz"; } else if (n % 5 == 0) { return "buzz"; } else { return "" + n; } }

Java

@Test void fizz3() { assertEquals("fizz", FizzBuzz.answer(3)); } @Test void fizz6() { assertEquals("fizz", FizzBuzz.answer(6)); } @Test void fizzBuzz15() { assertEquals("fizzbuzz", FizzBuzz.answer(15)); }

JUnit

Apps: Coverage reporter


1: executed 2: executed 3: executed 4: executed 5: 6: 7:


Java

Statement coverage

57% (4 / 7)

Apps: Fault localization


risk 1: 0.5 2: 1.0 3: 0 4: 0 5: NaN 6: NaN 7: NaN


Java

Calculate risk metric

We can change definition of

risk metric by writing script

Apps: Test minimization


@Test void fizz3() { assertEquals("fizz", FizzBuzz.answer(3)); } @Test void fizz6() { assertEquals("fizz", FizzBuzz.answer(6)); } @Test void fizzBuzz15() { assertEquals("fizzbuzz", FizzBuzz.answer(15)); }

JUnit

fizz3 and fizz6 are duplicated test cases because they execute

same path

But different test oracles have different detection capabilities ..

Limitation

• Data flow coverage

– OCCF cannot support to implement data flow analyzer due to language independence.

• Supportable languages

– OCCF cannot support languages where we cannot insert instrumentation code.

• Measurement environment

– OCCF works on only .NET Framework and Mono


Conclusion

• Problems – Differences between coverage tools

– High development costs for various langs

• OCCF Solution – Providing common code

– Extracting commonality between langs • Most of ASTs have similar structures

• Please contact us if you want to develop coverage tools with OCCF – We would like to support you


occf: a framework for developing test coverage measurement tools supporting multiple programming...

Technology