aditya v. nori, sriram k. rajamani microsoft research india
TRANSCRIPT
An Empirical Study of Optimizations in Yogi
Aditya V. Nori, Sriram K. Rajamani
Microsoft Research India
What is Yogi? An industrial strength program verifier
Idea: Synergize verification and testing
Synergy [FSE โ06], Dash [ISSTA โ08], SMASH [POPL โ10] algorithms to perform scalable analysis
Engineered a number of optimizations for scalability
Integrated with Microsoftโs Static Driver Verifier (SDV) toolkit and used internally
Motivation
Share our experiences in making Yogi robust, scalable and industrial strength
Several of the implemented optimizations are folklore Very difficult to design tools that are bug free evaluating
optimizations is hard! Our empirical evaluation gives tool builders information about
what gains can be realistically expected from optimizations
Vanilla implementation of algorithms: (flpydisk, CancelSpinLock) took 2 hours
Algorithms + engineering + optimizations: (flpydisk, CancelSpinLock) took less than 1 second!
Outline
Overview of Yogi
Overview of optimizations
Evaluation setup
Empirical Results
Summary
Property checking
void foo(){ *p = 4; *q = 5; if (condition) error();}
QuestionIs error() unreachable for all possible inputs?
Verification: can prove the absence of bugs, but can result in false errorsTesting: finds bugs, but canโt prove their absence
The Yogi algorithm
no
no
Can extend test beyond frontier?
Refine abstraction
Construct initial abstractionConstruct random tests
Test succeeded? Bug!
Abstractionsucceeded?
ฯ = error path in abstraction f = frontier of error path
yes
no
yes
Proof! yes
Input:Program P
Property ฯ
Example: Abstraction & Tests
no
no
Can extend test beyond frontier?
Refine abstraction
Construct initial abstractionConstruct random tests
Test succeeded? Bug!
Abstractionsucceeded?
ฯ = error path in abstraction f = frontier of error path
yes
no
yes
Proof! yes
Input:Program P
Property ฯ
void foo(int y){0: int x, lock = 0;1: do {2: lock = 1;3: x = y;4: if (*) {5: lock = 0;6: y = y+1; }7: } while (x != y);8: if (lock != 1)9: error();10:}
y = 101234
56
789
ร
ร ร
ร ร
ร ร
ร ร
ร
ร
ร ร
ร
10ร
Symbolic execution +
Theorem proving
Example: Refinement
no
no
Can extend test beyond frontier?
Refine abstraction
Construct initial abstractionConstruct random tests
Test succeeded? Bug!
Abstractionsucceeded?
ฯ = error path in abstraction f = frontier of error path
yes
no
yes
Proof! yes
Input:Program P
Property ฯ
void foo(int y){0: int x, lock = 0;1: do {2: lock = 1;3: x = y;4: if (*) {5: lock = 0;6: y = y+1; }7: } while (x != y);8: if (lock != 1)9: error();10:}
01234
56
78:ฯ
9
ร
ร ร
ร ร
ร ร
ร ร
ร
ร
ร ร
ร
10ร
8:ยฌฯร
Example: Proof!
no
no
Can extend test beyond frontier?
Refine abstraction
Construct initial abstractionConstruct random tests
Test succeeded? Bug!
Abstractionsucceeded?
ฯ = error path in abstraction f = frontier of error path
yes
no
yes
Proof! yes
Input:Program P
Property ฯ
void foo(int y){0: int x, lock = 0;1: do {2: lock = 1;3: x = y;4: if (*) {5: lock = 0;6: y = y+1; }7: } while (x != y);8: if (lock != 1)9: error();10:}
012
34:ยฌs5:ยฌs6:ยฌr
9
ร
ร ร
ร รรร
รร
ร
ร
7:ยฌqร
8:ยฌpร
4:s5:s6:r7:q8:pร
10
Optimizations
Initial abstraction from property predicates
Relevance heuristics for predicate abstraction Suitable predicates (SP) Control dependence predicates (CD)
Interprocedural analysis Global modification analysis Summaries for procedures
Thresholds for tests
Fine tuning environment models
Evaluation setup
Benchmarks: 30 WDM drivers and 83 properties (2490 runs) Anecdotal belief: most bugs in the tools are
usually caught with this test suite
Presentation methodology: Group optimizations logically such that related
optimizations are in the same group Total time taken, total number of defects found
for every possible choice of enabling/disabling each optimization in the group
Initial abstraction
state { enum {Locked = 0, Unlocked = 1} state = Unlocked;}
KeAcquireCancelSpinlock.Entry { if (state != Locked) { state = Locked; } else abort;}
KeReleaseCancelSpinlock.Entry { if (state == Locked) { state = Unlocked; } else abort;}
01
(๐ ๐ก๐๐ก๐โ ๐ฟ๐๐๐๐๐)
01
(๐ ๐ก๐๐ก๐=๐ฟ๐๐๐๐๐)
01๐
๐
Empirical resultsAbstractio
n using SLIC predicates
Total time
(minutes)
#defects #timeouts
yes 2160 241 77no 2580 241 86
16%
Relevance heuristics (SP)
Avoid irrelevant conjuncts
AC
๐
๐
B๐
D ๐ฟ
AC
๐
ยฌ๐
B๐
D ๐ฟ
C ๐
๐๐ ๐ ๐ข๐๐(๐)
๐๐ ๐ ๐ข๐๐(๐)
Irrelevant?
Relevance heuristics (CD)
Abstract assume statements that are not potentially relevant by skip statements
If Yogi proves that the program satisfies property, we are done.
Otherwise, validate the error trace
and refine the abstraction by putting back assume statements, if the error trace is spurious
Example: SP heuristic
int x;void foo() { bool protect = true; โฆ if (x > 0) protect = false; โฆ if (protect) KeAcquireCancelSpinLock(); for (i = 0; i < 1000; i++) { a[i] = readByte(i); } if (protect) KeReleaseCancelSpinLock();}
AC
๐
๐
B๐
D ๐ ๐ก๐๐ก๐=๐๐๐๐๐๐
AC
๐
ยฌ๐
B๐
D ๐ ๐ก๐๐ก๐=๐๐๐๐๐๐
C ๐
๐๐ ๐ ๐ข๐๐( ๐>1000)
๐๐ ๐ ๐ข๐๐( ๐>1000)
๐=(state=Locked )โง( ๐>1000)
Example: SP heuristic
int x;void foo() { bool protect = true; โฆ if (x > 0) protect = false; โฆ if (protect) KeAcquireCancelSpinLock(); for (i = 0; i < 1000; i++) { a[i] = readByte(i); } if (protect) KeReleaseCancelSpinLock();}
AC
๐
๐
B๐
D ๐ ๐ก๐๐ก๐=๐๐๐๐๐๐
AC
๐
ยฌ๐
B๐
D ๐ ๐ก๐๐ก๐=๐๐๐๐๐๐
C ๐
๐๐ ๐ ๐ข๐๐( ๐>1000)
๐๐ ๐ ๐ข๐๐( ๐>1000)
๐=(state=Locked )
Example: CD heuristic
int x;void foo() { bool protect = true; โฆ if (x > 0) protect = false; โฆ if (protect) KeAcquireCancelSpinLock(); for (i = 0; i < 1000; i++) { a[i] = readByte(i); } if (protect) KeReleaseCancelSpinLock();}
Empirical resultsSP
heuristic
CD heurist
ic
Total time
(minutes)
#defects
#timeouts
yes yes 2160 241 77yes no 2580 239 91no yes 2400 238 87no no 2894 235 174
10%
Empirical resultsSP
heuristic
CD heurist
ic
Total time
(minutes)
#defects
#timeouts
yes yes 2160 241 77yes no 2580 239 91no yes 2400 238 87no no 2894 235 174
16%
Empirical resultsSP
heuristic
CD heurist
ic
Total time
(minutes)
#defects
#timeouts
yes yes 2160 241 77yes no 2580 239 91no yes 2400 238 87no no 2894 235 174
25%
Interprocedural analysis
Yogi performs a compositional analysis : Is it possible to execute starting from
state and reach state ?
Global modification analysis
May-Must analysis (SMASH, POPL 2010)
Example
AC
๐
๐
B๐
D ๐ ๐ก๐๐ก๐=๐๐๐๐๐๐
AC
๐
ยฌ๐
B๐
D ๐ ๐ก๐๐ก๐=๐๐๐๐๐๐
C ๐
๐๐๐(โฆ)
foo(โฆ)
โจ๐1 , ๐๐๐ (โฆ) ,๐2 โฉ
Empirical resultsModificat
ion analysis
Summaries
Total time
(minutes)
#defects
#timeouts
yes yes 2160 241 77yes no 2760 239 109no yes 3180 237 134no no 3780 236 165
32%
Empirical resultsModificat
ion analysis
Summaries
Total time
(minutes)
#defects
#timeouts
yes yes 2160 241 77yes no 2760 239 109no yes 3180 237 134no no 3780 236 165
28%
Empirical resultsModificat
ion analysis
Summaries
Total time
(minutes)
#defects
#timeouts
yes yes 2160 241 77yes no 2760 239 109no yes 3180 237 134no no 3780 236 165
42%
Testing
Yogi relies on tests for โcheapโ reachability
Long tests avoiding several potential reachability
queries results in too many states and thus
memory consumption
Test thresholds: time vs. space tradeoff
Empirical evaluation
Test threshol
d
Total time
(minutes)
#defects
#timeouts
250 2600 236 92500 2160 241 771000 2359 240 881500 2400 239 89
Modeling the environment
if (DestinationString) { DestinationString->Buffer = SourceString;
// DestinationString->Length should be set to the // length of SourceString. The line below is missing // from the original stub SDV function DestinationString->Length = strlen(SourceString);}
if (SourceString == NULL){ DestinationString->Length = 0; DestinationString->MaximumLength = 0;}
Issue type #issues
Integers used as pointers
8Uninitialized
variables15
Type inconsistencies 9
Summary
Described optimizations implemented in Yogi Evaluated optimizations on the WDM test suite
Empirical data used to decide which optimizations to include in Yogi
We believe that this detailed empirical study of optimizations will enable tool builders to decide which optimizations to include and how to engineer their tools
http://research.microsoft.com/yogi