guided test visualization: making sense of errors in concurrent programs saint wesonga & neha...
TRANSCRIPT
Guided Test Visualization: Making Sense of Errors in
Concurrent Programs
Saint Wesonga & Neha Rungta & Eric Mercer
Brigham Young University & NASA AMES, USA
State of the Art Stress Testing Dynamic Analysis Runtime monitoring Static Analysis Model Checking Symbolic Execution Software Model Checking
Guided-test [SPIN 2009]Slice And Dice [ICSE-NIER
2010]
Guided-test wants to find real errors
Find an error or you do not have much to say
Not intended to be exhaustive
Thus not complete
Yet sound
Our Solution
Possible errorslocations from
imprecise analysis tools
Generate backward
slicesfrom
possibleerror
locations
Runtime Environment
Sound error
detection
Add inter-thread dependenceinformation
Heuristics
Guided Symbolic Execution
Input
Possible errorslocations from
imprecise analysis tools
Generate backward
slicesfrom
possibleerror
locations
Runtime Environment
Sound error
detection
Add inter-thread dependenceinformation
Heuristics
Guided Symbolic Execution
Small exampleThread A {
void run (Element elem) { lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}}
Thread B {void run (Element elem) {
int x; // input variable
if( x > 18) {
lock(elem)
elem.reset();
unlock(elem)}
}}Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Small ExampleThread A {
void run (Element elem) { lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}}
Thread B {void run (Element elem) {
int x; // input variable
if( x > 18) {
lock(elem)
elem.reset();
unlock(elem)}
}}Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Small ExampleThread A {
void run (Element elem) { lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}}
Thread B {void run (Element elem) {
int x; // input variable
if( x > 18) {
lock(elem)
elem.reset();
unlock(elem)}
}}Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Small ExampleThread A {
void run (Element elem) { lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}}
Thread B {void run (Element elem) {
int x; // input variable
if( x > 18) {
lock(elem)
elem.reset();
unlock(elem)}
}}Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Small ExampleThread A {
void run (Element elem) { lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}}
Thread B {void run (Element elem) {
int x; // input variable
if( x > 18) {
lock(elem)
elem.reset();
unlock(elem)}
}}Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Small ExampleThread A {
void run (Element elem) { lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}}
Thread B {void run (Element elem) {
int x; // input variable
if( x > 18) {
lock(elem)
elem.reset();
unlock(elem)}
}}Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Small ExampleThread A {
void run (Element elem) { lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}}
Thread B {void run (Element elem) {
int x; // input variable
if( x > 18) {
lock(elem)
elem.reset();
unlock(elem)}
}}Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Could it really be?Thread A {
void run (Element elem) { lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}}
Thread B {void run (Element elem) {
int x; // input variable
if( x > 18) {
lock(elem)
elem.reset();
unlock(elem)}
}}Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Is it possible to arrive at this location on some
execution?
Initial Abstraction to Follow (not explore)
Possible errorslocations from
imprecise analysis tools
Generate backward
slicesto possible
error locations
Runtime Environment
Sound error
detection
Add inter-thread dependenceinformation
Heuristics
Guided Symbolic Execution
Slice along sequential executionThread A {
void run (Element elem) { lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}} Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Target Locations & Start of Programs
A.run lstart
Exception
Thread A {void run (Element elem) {
lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}} Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Call Sites and Start Locs of the callee
A.run lstart
check(elem)
A.check lstart
Exception
Object Element { int e;
Element() { e = 1;
}
void reset() {e = 11;
}}
Thread A {void run (Element elem) {
lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}}
Conditional Statements
A.run lstart
check(elem)
A.check lstart
if(elem.e > 9)
Exception
Thread A {void run (Element elem) {
lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}} Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Data Definitions
A.run lstart
check(elem)
A.check lstart
if(elem.e > 9)
Exception
Thread A {void run (Element elem) {
lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}} Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Synchronization Operations
A.run lstart
lock(elem)
check(elem)
unlock(elem) A.check lstart
if(elem.e > 9)
Exception
Thread A {void run (Element elem) {
lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}} Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Abstract System
A.run lstart
lock(elem)
check(elem)
unlock(elem) A.check lstart
if(elem.e > 9)
Exception
Thread A {void run (Element elem) {
lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}} Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Abstract Trace
A.run lstart
lock(elem)
check(elem)
unlock(elem) A.check lstart
if(elem.e > 9)
Exception
Thread A {void run (Element elem) {
lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}} Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Abstract Trace
A.run lstart
lock(elem)
check(elem)
A.check lstart
if(elem.e > 9)
Exception
Thread A {void run (Element elem) {
lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}} Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
One trace of many we could have selected: fair coin or
farm in parallel
Our Solution
Possible errorslocations from
imprecise analysis tools
Generate backward
slicesto possible
error locations
Runtime Environment
Sound error
detection
Add inter-thread dependenceinformation
Heuristics
Guided Symbolic Execution
Extract current program location of the
most recently executed thread
Map Concrete state to a location
Thread-0
StackLocalVars
Thread-1
StackLocalVars
Thread-n
StackLocalVars
HeapGlobalVars
Data Input VariablesSymbolic Representations
ϕpath Constraint
s1
l1
Goal is to match a concrete state to the next program
location in our trace
Follow the trace!
Guidance Strategys0
s2
s3
s1
s5 s6s4
l1
l2
ln
Distance heuristic to choose nearest to next location on the current trace
Our Solution
Possible errorslocations from
imprecise analysis tools
Generate backward
slicesto possible
error locations
Runtime Environment
Sound error
detection
Add inter-thread dependenceinformation (refine)
Heuristics
Guided Symbolic Execution
Followed trace to predicate…A.run lstart
lock(elem)
check(elem)
A.check lstart
if(elem.e > 9)
Exception
False in all successors
(zut!)
Where else is elem.e defined?Thread A {
void run (Element elem) { lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}}
Thread B {void run (Element elem) {
int x; // input variable
if( x > 18) {
lock(elem)
elem.reset();
unlock(elem)}
}}Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Redefine a global variableB.run lstart
e := 11
A.run lstart
lock(elem)
check(elem)
unlock(elem) A.check lstart
if(elem.e > 9)
Exception
Generate a new backward sliceB.run lstart
e := 11
if(x > 18)
lock(elem)
unlock(elem) Element.reset lstart
elem.reset()
Thread B {void run (Element elem) {
int x; // input variable
if( x > 18) {
lock(elem)
elem.reset();
unlock(elem)}
}} Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Updated Abstract SystemB.run lstart
e := 11
if(x > 18)
lock(elem)
unlock(elem) Element.reset lstart
elem.reset()
A.run lstart
lock(elem)
check(elem)
unlock(elem) A.check lstart
if(elem.e > 9)
Exception
Update Abstract Trace
B.run lstart
e := 11
if(x > 18)
lock(elem)
Element.reset lstart
elem.reset()
A.run lstart
lock(elem)
check(elem)
A.check lstart
if(elem.e > 9)
Exception
Check Lock Dependencies
B.run lstart
e := 11
if(x > 18)
lock(elem)
Element.reset lstart
elem.reset()
A.run lstart
lock(elem)
check(elem)
A.check lstart
if(elem.e > 9)
Exception
Update Abstract Trace
B.run lstart
e := 11
if(x > 18)
lock(elem)
Element.reset lstart
elem.reset()
unlock(elem)
A.run lstart
lock(elem)
check(elem)
A.check lstart
if(elem.e > 9)
Exception
Thread B {void run (Element
elem) { int x; //
input variableif( x > 18)
{
lock(elem)
elem.reset();
unlock(elem)}
}}
Update Abstract State
B.run lstart
e := 11
if(x > 18)
lock(elem)
Element.reset lstart
elem.reset()unlock(elem)
A.run lstart
lock(elem)
check(elem)
A.check lstart
if(elem.e > 9)
Exception
Restart with this new trace from the initial
state
Slice and dice wants to find real errors and prove correct…
Don’t find an error yet have much to say
Goal is still to find errors
so optimize search
yet useful in no error
Abstract Trace
A.run lstart
lock(elem)
check(elem)
unlock(elem) A.check lstart
if(elem.e > 9)
Exception
Thread A {void run (Element elem) {
lock(elem)
check(elem)unlock(elem)
}
void check(Element elem) {if(elem.e > 9)
Throw exception
}} Object Element {
int e;Element() { e = 1;
}
void reset() {e = 11;
}}
Slice and Dice Keep entire
abstraction Schedule only on
node in the abstraction
A.run lstart
lock(elem)
check(elem)
unlock(elem) A.check lstart
if(elem.e > 9)
Exception
When it is time to refine…
B.run lstart
e := 11
if(x > 18)
lock(elem)
Element.reset lstart
elem.reset()unlock(elem)
A.run lstart
lock(elem)
check(elem)
A.check lstart
if(elem.e > 9)
Exception…rather than
create a single trace…
Keep the entire abstractionB.run lstart
e := 11
if(x > 18)
lock(elem)
unlock(elem) Element.reset lstart
elem.reset()
A.run lstart
lock(elem)
check(elem)
unlock(elem) A.check lstart
if(elem.e > 9)
Exception
jpf-guided-test
50
Verification & Validation LabBrigham Young University
Provo, UT 84606, USA
http://vv.cs.byu.edu/
VV Lab, Brigham Young University, USA
Experimental Results A new GuidedSymbolic extension in Java
Pathfinder Uses the engine from the symbolic extension
(Pasareanu et. al. ISSTA 2008) Dynamic partial order reduction turned on Abstraction, refinement and heuristic
computation all performed on Java bytecode Libraries are part of the multi-threaded system
Multi-threaded Programs Reorder Benchmark
SLOC: 44, Reachability Airline Benchmark
SLOC: 31, Reachability VecDeadlock Real JDK 1.4 concurrent library
SLOC: 7267, Deadlock in Vector class VecDeadlock Real JDK 1.4 concurrent library
SLOC: 7169, Deadlock in Vector class VecRace Real JDK 1.4 concurrent library
SLOC: 7151, Race in ArrayList class
Error DiscoveryTi
me in m
inute
s
60
30
20
10
Out of PatienceGuided
DFS
VecDeadlock0
VecDeadlock1
VecRace
Error Discovery
56
Subject Guided Search
Thread # States Time (secs) Memory
Reorder(9,1) 11 205 1.67 7MB
Reorder(10,1) 12 239 1.67 7MB
Airline(15,3) 16 1210 3.23 5MB
Airline(20,2) 21 3279 7.46 5MB
Airline(20,1) 21 3609 7.46 5MB
VecDeadlock0 2 1370 4.56 66MB
VecDeadlock1 2 2948 6.89 66MB
VecRace 2 3120 7.98 65MB