demand-driven structural testing with dynamic instrumentation (icse 2005)
TRANSCRIPT
Jonathan Misurda ([email protected]) James A. Clause, Juliya L. Reed, Bruce R. Childers
Department of Computer Science University of Pittsburgh Pittsburgh, Pennsylvania 15260 USA
Mary Lou Soffa Department of Computer Science University of Virginia Charlottesville, Virginia 22904 USA
Demand-Driven Structural Testing with Dynamic Instrumentation
Software Testing
n What is Software Testing? q Gather information about the behavior of the software being
developed or modified.
n Why test software? q Gain confidence and insights into software correctness q Create quality, robust programs
n Why is testing hard? q Testing is expensive
n 50-60% of the total cost of software development q Adds complexity to development process
n One testing technique is not enough
Structural Software Testing
n Structural Testing q The process of discovering run-time properties of a
program specifically pertaining to control-flow
q Demonstrate adequacy of the test cases
q Different granularities of structures: n Node coverage records statements (basic blocks) n Branch coverage records control-flow edges n Def-Use coverage records pairs of variable definitions
and uses
q Over multiple test cases until coverage criteria
Current Testing Tools
n E.g.: JCover, Clover, IBM Rational PurifyPlus
n Static instrumentation with test probes q Probes are injected instrumentation to gather coverage
information
n Limitations: q Expensive in both time and space q Only one type of test q Limited code regions and scopes q Cannot define their own ways to test
Our Approach
n Demand-driven structural testing q Specification driven: User written tests q Test plans: Recipe of how & where to test q Path specific: Instrument only what is needed q Dynamic: Insert & remove instrumentation
n Jazz: A scalable & flexible framework for testing q Automatically apply multiple test strategies q Handle large programs q Allows user customization of tests
Jazz Overview
Application
IDE & Test GUI
Test Specification
Test Planner
Test Plan
Test Analyzer and Result Reporter
Test Dynamic Instrumenter
Test Results
Select the regions and test types Allows user customization
Automatically generate where and how to instrument
Run the program with demand-driven instrumentation
Collect the results
Test Specifier
1a
1b
3
2
Specify the regions to test 1a. Highlight line regions 1b. Select classes and methods
List of desired tests
Click to create a test
Test Planner
n Test plan: Where and how to test program n Test storage, probe location table,
instrumentation payload n Can combine payloads to create custom strategies
Application
Test Planner
Test Specification
Test Instrumentation Payload
Test Plan
Global Storage & Probe Location Table
Test Planner
n Challenges: q When to insert test probes q Where to test/instrument q What to do at each probe
n A planner determines three things: q Seed Probe Set – probes which are initially inserted in a
test region. q Coverage Probe Set – probes that can be inserted and
removed on demand along a path. q Probe Lifetime – whether a probe must be re-inserted
after being hit by its “control flow successor” basic blocks.
Branch Coverage Example Plan public class simpleDemo {
public static void main(String[] args) { int evenSum = 0; int oddSum = 0;
for(int i = 0; i < 100; i++) { if(i%2==0) { evenSum += i; }
else { oddSum += i; } } System.out.println("The sum of the even numbers from 0 to 100 is: " + evenSum);
System.out.println("The sum of the odd numbers from 0 to 100 is: " + oddSum); }
}
DEFINITIONS { NAME: d_method, REGION_D, LOCATION: FILE simpleDemo.java {
CLASS simpleDemo, METHOD main }
} BODY {
DO BRANCH_TEST ON REGION d_method UNTIL: 85% }
simpleDemo.java
simpleDemo.testspec
Branch Coverage Planner
n Record which edges are executed q Determine (source, sink) edge pairs hit at run-time q Source is a branch q Sink can be taken & not-taken target of branch
Probe Set Members Seed Probe First block in testing region Coverage Probe Sinks of currently instrumented block Probe Lifetime Until all incoming edges have been
covered
Branch Coverage Example
Probe Loc Tab Storage Block Next Hit 1 2,3 2 4 3 4 4 1 Test Payload Mark edge hit Insert at next point Remove instrumentation
1
2 3
4 Coverage probe Hit
N, N
N
N
N
Y, Y
Y
Y
Y
Seed probe
Demo
n Branch coverage of music player q Test region is whole program (JOrbis) q Test input is a song
n Compared q Traditional approach with static instrumentation q Demand-driven approach with dynamic instrumentation
With Dynamic Instrumentation
With Static Instrumentation
Demo
n Branch coverage of music player q Test region is whole program (JOrbis) q Test input is a song
n Compared q Traditional approach with static instrumentation q Demand-driven approach with dynamic instrumentation
With Dynamic Instrumentation
With Static Instrumentation
Demo
n Branch coverage of music player q Test region is whole program (JOrbis) q Test input is a song
n Compared q Traditional approach with static instrumentation q Demand-driven approach with dynamic instrumentation
With Dynamic Instrumentation
With Static Instrumentation
Other Planners
n Node Coverage Planner q Record which statements are executed Probe Set Members Seed Probe First statement in the testing region Coverage Probe Next reachable statement Probe Lifetime Removed as soon as statement is executed
n Def-Use Coverage Planner q Record which variable definition reaches an executed use
Probe Set Members Seed Probe All variable definitions Coverage Probe Uses associated with all definitions Probe Lifetime Defs removed after all reachable uses are
covered. Uses are removed as executed
Dynamic Instrumentation
n Test plan targets an instrumentation API
n FIST instrumentation engine [WOSS’04] q Retargetable & reconfigurable q Dynamic insertion & removal of instrumentation q Binary level instrumentation (post JIT)
n Uses fast breakpoints [Kessler]: Replace existing instruction with a jump to instrumentation
Experimental Methodology
n Test Specification q Implemented as an Eclipse 3.0 plugin
n Test Planner & Dynamic Instrumenter q Implemented using the Jikes RVM version 2.1.1 q On the x86 platform
n SPECjvm98 benchmarks test input q unloaded 2.4 Ghz Pentium IV, 1GB of memory q RedHat Linux 7.3
n Traditional vs. Jazz (in same implementation) q Compared coverage, run-time, memory
Coverage Results
n Coverage – same reported by both tools
Branch Node Def-Use compress 58% 90.6% 89.8%
jess 46.8% 80.3% 71.8% db 44.4% 76.9% 75.0%
javac 38.9% 75.0% 66.9% mpegaudio 60.9% 88.7% 90.5%
mtrt 50.6% 90.3% 87.3% jack 55.6% 82.2% 73.4%
Run-time Performance
Static Branch Dynamic Branch
Static Node Static Def-Use Dynamic Node Dynamic Def-Use
Memory Requirements
Branch Node Def-Use Demand Static Demand Static Demand Static
compress 7.9 7.5 4.4 3.7 8.8 5.6 jess 50.2 60.3 34.1 29.7 70.3 44.8
db 9.7 8.9 5.0 4.1 9.8 5.1 javac 178.9 186 107.3 85.3 202.0 125.3
mpegaudio 24.7 29.5 15.0 14.7 34.8 22.3 mtrt 22.4 23 12.5 11.0 26.1 16.1 jack 73.4 78 46.5 37.6 89.0 46.8
• In Kilobytes
Summary
n New demand-driven approach q A tool (Jazz) for structural testing q Dynamic instrumentation guided by a program’s
execution q Minimally intrusive q User configurable and flexible testing
n Very low overhead q E.g., branch coverage tool is 3-4x faster than
traditional approaches
Test Results 1 Click to run test suite
2 Results
Static Instrumentation
n Shortcomings of static instrumentation: q Not scalable: Instrumentation left in place causes
unnecessary increase in overhead q Inflexible: Only certain tests, languages, and
platforms q Cumbersome: Requires recompilations for
dedicated testing q Intrusive: Addition of testing code may alter or
mask defects
Run-time Performance
-10%
30%
70%
110%
150%
190%
230%
270%
compress jess db javac mpeg mtrt jack avg
% s
low
dow
n ov
er b
asel
ine
Static Branch Dynamic Branch Static NodeDynamic Node Static Def-Use Dynamic Def-Use
Branch Testing
0102030405060708090
100
Tim
e (s
econ
ds)
compress jess db javac mpeg mtrt jack avg
Dynamic Static None
Node Testing
0102030405060708090
100
Tim
e (s
econ
ds)
compress jess db javac mpeg mtrt jack avg
Dynamic Static None
Def-Use Testing
0102030405060708090
100
Tim
e (s
econ
ds)
compress jess db javac mpeg mtrt jack avg
Dynamic Static None