falcon: fault localization in concurrent programs
TRANSCRIPT
1
Background Technique Studies Conclusion
Sangmin Park, Richard Vuduc, and Mary Jean Harrold
College of ComputingGeorgia Institute of Technology
Falcon: Fault Localization inConcurrent Programs
2
Background Technique Studies Conclusion
• Concurrency bugs are difficult to find• Non-deterministic behavior• Large thread interleaving space
• Godefroid, Nagappan [MSR-TR08] found that
Developers
Handle concurrencyissues
Motivation
3
Background Technique Studies Conclusion
• Bug finding for concurrency bugs• Static/dynamic data race detection
(e.g., [Ronesse99, Savage97, O’Callanhan03])
• Static/dynamic atomicity violation detection
(e.g., [Flanagan03, Wang06, Flanagan08])
• Dynamic pattern analysis (e.g., [Lu06, Hammer08])
• Fault localization• For sequential programs
(e.g., [Jones02, Liblit03, Liu05, Baah08])Techniques have not been applied to concurrent programsTechniques have not been applied to concurrent programs
1. Techniques target only one type of concurrency bug
2. Techniques report benign (harmless) results along with harmful results
1. Techniques target only one type of concurrency bug
2. Techniques report benign (harmless) results along with harmful results
Existing Techniques
4
Background Technique Studies Conclusion
Falcon Research Outline
• Background• Technique: patterns + suspiciousness• Empirical Studies• Conclusion
5
Background Technique Studies Conclusion
Concurrency ViolationsLu et al.’s [ASPLOS08] found that
Deadlock: 30%Deadlock: 30%Order andAtomicity violations:67%
Both occur due to undesirable memory-access sequences
Others: 3%
Both occur due to undesirable memory-access sequences
6
Background Technique Studies Conclusion
Order ViolationWhen desired order of accesses is reversed
Unintended program behavior
Thread T1
11: void M1()
12: {
13: lock(L);
14: A=new Object();
15: if (A) { /*…*/ }
16: unlock(L);
17: }
Thread T1
11: void M1()
12: {
13: lock(L);
14: A=new Object();
15: if (A) { /*…*/ }
16: unlock(L);
17: }
Thread T2
21: void M2()
22: {
23: lock(L);
24: x=A.getX();
25: unlock(L);
26: }
Thread T2
21: void M2()
22: {
23: lock(L);
24: x=A.getX();
25: unlock(L);
26: }
For A access Correct order Incorrect order
W14
R24 W14
R24
RT2 WT1
7
Background Technique Studies Conclusion
Atomicity ViolationWhen atomic region is interfered by another thread
Unintended program behavior
Thread T3
31: copy(Obj o)32: {33: …34: int sz=o.sz();35: …36: copy(sz, o, this.buffer);37: …38: }
Thread T3
31: copy(Obj o)32: {33: …34: int sz=o.sz();35: …36: copy(sz, o, this.buffer);37: …38: }
For o access Correct order
Incorrect order
RT3 WT4 RT3
Thread T4
41: access()42: {43: …44: o.clear();45: …46: }
Thread T4
41: access()42: {43: …44: o.clear();45: …46: }
R36
R36
W44
W44
R34
R34
8
Background Technique Studies Conclusion
Patterns for Concurrency Violations
1 RT1 * WT2
2 WT1 * RT2
3 WT1 * WT2
Patterns
for order violation
1 RT1 * WT2 * RT1
2 WT1 * WT2 * RT1
3 WT1 * RT2 * WT1
4 RT1 * WT2 * WT1
5 WT1 * WT2 * WT1
Patterns for atomicity violation[PPoPP06, ASPLOS06]
Note• Concurrency violations represented as patterns• Patterns not always concurrency violations
Note• Concurrency violations represented as patterns• Patterns not always concurrency violations
1 RT1 * WT2
2 WT1 * RT2
3 WT1 * WT2
1 RT1 * WT2 * RT1
2 WT1 * WT2 * RT1
3 WT1 * RT2 * WT1
4 RT1 * WT2 * WT1
5 WT1 * WT2 * WT1
9
Background Technique Studies Conclusion
Phase 1. Dynamic pattern recording• Input: a set of executions• Output: patterns, outcome (pass/fail)
Phase 2. Statistical analysis• Compute suspiciousness of each pattern
Technique For Concurrency—Falcon
Intuition Patterns appearing in failing executions are more suspicious than patterns appearing in passing executions
10
Background Technique Studies Conclusion
R11-W11 -R31
-31
y-Access : W12-W22-R32-R13
x-Access : W11-W21-R31-R13x-Access :
Falcon Example on Atomicity Violation
Initially x=0; y=0;
Thread 111:if(x==0) x=1;12:if(y==0) y=1;13:if(x==2&&y==2) assert(false);
Thread 221:if(x==1) x=2;22:if(y==1) y=2;
Thread 331:if(x==1) x=3;32:if(y==1) y=3;
Phase 1. Dynamic pattern recording• Monitor accesses per variable• Monitor accesses per thread• Create window per variable • Detect patterns inside window
x-Access :
y-Access :
Statements:
y-Access :
-21 -12-22-32-1311
-R21-W21 -R13
R12-W12-R22-W22-R32-R13
W11-W21-R31
W12-W22-R32-R13
-R13
x=0 y=0
x=1y=1x=2y=2
x=2y=2
x=2y=2
11
Background Technique Studies Conclusion
Falcon Example on Atomicity Violation
Initially x=0; y=0;
Thread 111:if(x==0) x=1;12:if(y==0) y=1;13:if(x==2&&y==2) assert(false);
Thread 221:if(x==1) x=2;22:if(y==1) y=2;
Thread 331:if(x==1) x=3;32:if(y==1) y=3;
Run 1 (r1):
x-access:
y-access:
r1 r2 r3 r4 Suspicious-ness
Pattern 1:W11-W31-R13
Pattern 2:W11-W21-R13
Pattern 3:W12-W32-R13
Pattern 4:W12-W22-R13
Phase 2. Statistical analysis• Use patterns and outcome• Compute suspiciousness
12
Background Technique Studies Conclusion
r1 r2 r3 r4 Suspicious-ness
Pattern 1:W11-W31-R13
Pattern 2:W11-W21-R13
Pattern 3:W12-W32-R13
Pattern 4:W12-W22-R13
P
Falcon Example on Atomicity Violation
Run 1 (r1): 11 31 21 12 32 22 13
x-access: W11 W31 R21 R13
y-access: W12 W32 R22 R13
r1 r2 r3 r4 Suspicious-ness
Pattern 1:W11-W31-R13
Pattern 2:W11-W21-R13
Pattern 3:W12-W32-R13
Pattern 4:W12-W22-R13
Initially x=0; y=0;
Thread 111:if(x==0) x=1;12:if(y==0) y=1;13:if(x==2&&y==2) assert(false);
Thread 221:if(x==1) x=2;22:if(y==1) y=2;
Thread 331:if(x==1) x=3;32:if(y==1) y=3;
Phase 2. Statistical analysis• Use patterns and outcome• Compute suspiciousnessRun 1 (r1): 11 31 21 12 32 22 13
x-access: W11 W31 R21 R13
y-access: W12 W32 R22 R13
13
Background Technique Studies Conclusion
Falcon Example on Atomicity Violation
Run 2 (r2): 11 31 21 12 22 32 13
x-access: W11 W31 R21 R13
y-access: W12 W22 R32 R13
r1 r2 r3 r4 Suspicious-ness
Pattern 1:W11-W31-R13
Pattern 2:W11-W21-R13
Pattern 3:W12-W32-R13
Pattern 4:W12-W22-R13
P P
Initially x=0; y=0;
Thread 111:if(x==0) x=1;12:if(y==0) y=1;13:if(x==2&&y==2) assert(false);
Thread 221:if(x==1) x=2;22:if(y==1) y=2;
Thread 331:if(x==1) x=3;32:if(y==1) y=3;
Phase 2. Statistical analysis• Use patterns and outcome• Compute suspiciousness
14
Background Technique Studies Conclusion
Falcon Example on Atomicity Violation
Run 3 (r3): 11 21 31 12 32 22 13
x-access: W11 W21 R31 R13
y-access: W12 W32 R22 R13
r1 r2 r3 r4 Suspicious-ness
Pattern 1:W11-W31-R13
Pattern 2:W11-W21-R13
Pattern 3:W12-W32-R13
Pattern 4:W12-W22-R13
P P P
Initially x=0; y=0;
Thread 111:if(x==0) x=1;12:if(y==0) y=1;13:if(x==2&&y==2) assert(false);
Thread 221:if(x==1) x=2;22:if(y==1) y=2;
Thread 331:if(x==1) x=3;32:if(y==1) y=3;
Phase 2. Statistical analysis• Use patterns and outcome• Compute suspiciousness
15
Background Technique Studies Conclusion
Falcon Example on Atomicity Violation
Run 4 (r4): 11 21 31 12 22 32 13
x-access: W11 W21 R31 R13
y-access: W12 W22 R32 R13
r1 r2 r3 r4 Suspicious-ness
Pattern 1:W11-W31-R13
Pattern 2:W11-W21-R13
Pattern 3:W12-W32-R13
Pattern 4:W12-W22-R13
P P P F
Initially x=0; y=0;
Thread 111:if(x==0) x=1;12:if(y==0) y=1;13:if(x==2&&y==2) assert(false);
Thread 221:if(x==1) x=2;22:if(y==1) y=2;
Thread 331:if(x==1) x=3;32:if(y==1) y=3;
Phase 2. Statistical analysis• Use patterns and outcome• Compute suspiciousness
16
Background Technique Studies Conclusion
r1 r2 r3 r4 Suspicious-ness
Pattern 1:W11-W31-R13
0.0
Pattern 2:W11-W21-R13
0.5
Pattern 3:W12-W32-R13
0.0
Pattern 4:W12-W22-R13
0.5
P P P F
r1 r2 r3 r4 Suspicious-ness
Pattern 1:W11-W31-R13
0.0
Pattern 2:W11-W21-R13
Pattern 3:W12-W32-R13
0.0
Pattern 4:W12-W22-R13
P P P F
Falcon Example on Atomicity Violation
r1 r2 r3 r4 Suspicious-ness
Pattern 1:W11-W31-R13
Pattern 2:W11-W21-R13
Pattern 3:W12-W32-R13
Pattern 4:W12-W22-R13
P P P F
Jaccard formula [Abreu07]
suspiciousness(P) =failed (P)
totalfailed + passed(P)
Initially x=0; y=0;
Thread 111:if(x==0) x=1;12:if(y==0) y=1;13:if(x==2&&y==2) assert(false);
Thread 221:if(x==1) x=2;22:if(y==1) y=2;
Thread 331:if(x==1) x=3;32:if(y==1) y=3;
17
Background Technique Studies Conclusion
Empirical Studies• Studies
1. Determine window size needed and its relationship to number of threads
2. Evaluate technique’s effectiveness
3. Compare technique with Cooperative Crug Isolation (CCI) [Thakur et al.,WODA09]
4. Evaluate technique’s efficiency (see paper)
• Empirical setup• Implemented in Java• Evaluated on a set of subjects
18
Background Technique Studies Conclusion
Subjects
Empirical SetupProgram LOC Failure Rate (%) Violation Type
Account 115 3 Atomicity
AirlineTickets 95 54 Atomicity
BubbleSort2 130 69 Atomicity
BufWriter 255 14 Atomicity
Lottery 359 43 Atomicity
MergeSort 375 84 Atomicity
Shop 273 2 Atomicity
ArrayList 5866 2 Atomicity
HashSet 7086 3 Atomicity
StringBuffer 1320 3 Atomicity
TreeSet 7532 3 Atomicity
Vector 709 2 Atomicity
Cache4j 3897 3 Order
RayTracer 1924 14 Atomicity
ContestBenchmark
Java Collection
Account 115 3 Atomicity
19
Background Technique Studies Conclusion
Goals To determine
• Window size to detect patterns for atomicity violations• Correlation of window size and number of threads
Method• For test case with 100 runs and subjects running with
p threads, where p={4,8,16,32,64}• recorded patterns during execution• computed window size needed to detect each
pattern• Aggregated data from all subjects
Empirical Study 1 – Window Size
20
Background Technique Studies Conclusion
4 8 16 32 640
10
20
30
40
50
60
Number of Threads
Win
do
w S
ize
to D
etec
t P
atte
rn
• Median window size 3
• Most patterns with window size < 10
• Window size and number of threads not positively correlated
Result – Window Size
21
Background Technique Studies Conclusion
Goal• To measure effectiveness of suspiciousness ranking
Techniques compared• Conflict Serializability (e..g, [Wang and Stoller,
PPoPP06])• AVIO [Lu et al., ASPLOS06]
Method• For test case with 100 runs and window size 5
• Computed suspiciousness of each pattern• Computed fault-localization expense (number of
patterns that must be examined to find fault)
Empirical Study 2 – Effectiveness
22
Background Technique Studies Conclusion
Result - Effectiveness
BufWriter 105 25 1
SubjectsConflict-Serializability AVIO Falcon
Patterns (Ranking) Patterns (Ranking) Patterns (Ranking)
Account 11 (-) 7 (-) 11 (2)
AirlineTickets 4 (-) 0 (-) 4 (1)
BubbleSort2 4 (-) 1 (-) 4 (1)
BufWriter 105 (-) 25 (-) 105 (1)
Lottery 4 (-) 4 (-) 4 (1)
MergeSort 76 (-) 42 (-) 76 (1)
Shop 10 (-) 0 (-) 10 (2)
ArrayList 1 (-) 1 (-) 1 (1)
HashSet 7 (-) 3 (-) 7 (1)
StringBuffer 2 (-) 1 (-) 2 (1)
TreeSet 9 (-) 3 (-) 9 (2)
Vector 1 (-) 1 (-) 1 (1)
Cache4j 23 (1)
RayTracer 14 (-) 14 (-) 14 (2)
Conflict-Serializability
Patterns (Ranking)
11 (-)
4 (-)
4 (-)
105 (-)
4 (-)
76 (-)
10 (-)
1 (-)
7 (-)
2 (-)
9 (-)
1 (-)
14 (-)
AVIO
Patterns (Ranking)
7 (-)
0 (-)
1 (-)
25 (-)
4 (-)
42 (-)
0 (-)
1 (-)
3 (-)
1 (-)
3 (-)
1 (-)
14 (-)
Falcon
Patterns (Ranking)
11 (2)
4 (1)
4 (1)
105 (1)
4 (1)
76 (1)
10 (2)
1 (1)
7 (1)
2 (1)
9 (2)
1 (1)
23 (1)
14 (2)
BufWriter 105 (-) 25 (-) 105 (1)
* (-) : Patterns are not ranked
23
Background Technique Studies Conclusion
Goal• To compare effectiveness of Falcon to CCI*
• CCI observes memory accesses, outcome (pass/fail)• CCI reports suspiciousness of each memory access
predicate (i.e., memory access and previous access type pair)
Method• Implemented the CCI technique in the Falcon toolset• For test case with 100 runs and subjects
• Executed the tool• Found the access and rank of actual bug
Empirical Study 3 – Case Study
* Cooperative Crug Isolation [Thakur, Sen, Liblit, Lu, WODA09]
24
Background Technique Studies Conclusion
Result – Case StudySubjects
CCI Falcon
Accesses of Bug (Rank) Pattern of Bug (Rank)
Account R (6), W (4), R (6) R-W-R (2)
AirlineTickets R (2), W (5), R (4) R-W-R (1)
BubbleSort2 W (1), R (7), W (2) W-R-W (1)
BufWriter R (5), W (1), R (5) R-W-R (1)
Lottery R (1), W (7), R (4) R-W-R (1)
MergeSort W (1), R (3), W (12) W-R-W (1)
Shop R (15), W (12), R (15) R-W-R (2)
ArrayList R (4), W (4), R (4) R-W-R (1)
HashSet R (15), W (15), R (1) R-W-R (1)
StringBuffer R (14), W (2), R (1) R-W-R (1)
TreeSet R (10), W (5), R (10) R-W-R (2)
Vector R (2), W (2), R (2) R-W-R (1)
Cache4j W (79), R (2) W-R (1)
RayTracer R (4), W (4), W (3) R-W-W (2)
HashSet R (15), W (15), R (1) R-W-R (1)
Shop R (15), W (12), R (15) R-W-R (2)
CCI
Accesses of Bug (Rank)
R (6), W (4), R (6)
R (2), W (5), R (4)
W (1), R (7), W (2)
R (5), W (1), R (5)
R (1), W (7), R (4)
W (1), R (3), W (12)
R (15), W (12), R (15)
R (4), W (4), R (4)
R (15), W (15), R (1)
R (14), W (2), R (1)
R (10), W (5), R (10)
R (2), W (2), R (2)
W (79), R (2)
R (4), W (4), W (3)
Falcon
Pattern of Bug (Rank)
R-W-R (2)
R-W-R (1)
W-R-W (1)
R-W-R (1)
R-W-R (1)
W-R-W (1)
R-W-R (2)
R-W-R (1)
R-W-R (1)
R-W-R (1)
R-W-R (2)
R-W-R (1)
W-R (1)
R-W-W (2)
Account R (6), W (4), R (6) R-W-R (2)
Falcon ranking1. WS1-WS3-RS2
2. RS6-WS2-RS6
3. WS4-RS6-WS2
4. RS1-WS4-WS6
5. WS6-WS4-RS6
6. RS2-WS4-WS6
7. …
Falcon ranking1. RS2-WS3-RS1
2. RS6-WS2-RS6
3. RS4-WS6-WS2
4. WS1-WS3-WS6
5. RS6-WS3-RS6
6. RS2-WS3-RS3
7. RS1-WS4-WS6
• CCI reports the actual bug with lower ranks• CCI does not provide an association of buggy
accesses
• CCI reports the actual bug with lower ranks• CCI does not provide an association of buggy
accesses
CCI ranking1. RS1
2. RS2
3. WS3
4. WS4
5. RS5
6. RS6
7. …
CCI ranking1. RS1
…
12. WS2
13. …
14. …
15. RS6
…
CCI ranking1. RS1
…
…
…
15. RS2
15. WS3
…
Falcon ranking1. WS1-WS3-RS2
2. RS6-WS4-RS6
3. WS4-WS6-RS2
4. RS1-WS4-RS6
5. WS6-WS4-WS6
6. RS2-WS4-WS6
7. …
25
Background Technique Studies Conclusion
Future Work
• Perform more empirical studies• With larger subjects• With more subjects especially with order violations• With considering multiple test cases
• Apply to other languages and systems• Other than Java language
• Extend the technique to support• Handling multi-variable related patterns for
atomicity violations• Localizing deadlocks
26
Background Technique Studies Conclusion
Contributions
• Falcon is the first technique• that reports and ranks patterns by
suspiciousness• that addresses order violations and
atomicity violations with patterns• Studies show that
• Falcon’s window size• Small constant • Not correlated with number of threads
• Falcon’s suspiciousness ranking technique• Effective in identifying faulty interleavings• More effective than ranking individual accesses
Questions?Questions?
27
Background Technique Studies Conclusion
Backup Slides
28
Background Technique Studies Conclusion
Result – EfficiencyConflict-Serializability (PPoPP 06)
AVIO(ASPLOS 06)
Atomic-Set-Serializability(ICSE 08)
Falcon(ICSE10)
Efficiency (Average slowdown)
38X (Median slowdown)
25X 13X(Used almost same set of subjects as Falcon)
9.9X