SAT-based Model Checking

Yakir VizelComputer Science Department,

Technion, Israel

Based on slides from K.L. McMillan, A.R. Bradley and Yakir Vizel

Outline

• Background– Symbolic Model Checking– DPLL-style SAT solvers– Bounded Model Checking

• SAT-based model checking methods– K-induction– Interpolation– Interpolation Sequence– IC3/PDR– There are more…

Model checking

• Problem definition:– Does every run of a (finite-state) transition system satisfy a given temporal property?

• Result:– Yes– No + counterexample

• Examples:– Is every request to this bus arbiter eventually acknowledged?

– Does this program every dereference a null pointer?

Transition systems

• Tuple (S,I,T), where:– S is the (finite) set of states– I S is the set of initial states– T SS is the set of transitions

• A run of (S,I,T) is S, where: 0I

– for all i 0, (i ,i+1) T

• That is, a run is an infinite path in the state graph strating with an initial state

Reachability

• Problem def:– Does a transition system have a finite run ending in a state contained in the failure set F?

– More precisely, does there exist 0...k Sk s.t.: 0 I and k F

• for all 0 i < k, (i ,i+1) T

• Using automata-theoretic methods, model checking safety properties reduces to reachability analysis.– Given a t.s. M and a property P, we can construct MP and FP such that M satisfies P exactly when FP is not reachable in MP.

State explosion problem

• Reachability analysis can be done by BFS or DFS on the state graph.

• However, |S| is exponential in system size– for example 2n, where n is number of registers

• Impractical to construct the state graph explicitly.

Our topic is essentially how to use a SAT solver to tackle this problem.

Symbolic transition systems

• Tuple (V,I,T), where:– V is a signature (set of variables),– I is a formula over V (the initial condition)

– T is a formula over VV' (the transition condition)

• States: = V{0,1} (a valuation to V)

• A run of (V,I,T) is , where:– I[0]

– for all i 0, T[i ,i+1]Note: T[i ,i+1] means T[i 'i+1]

Example

T is a conjunction of constraits, one per component.

ab cp

gg = a b

p = g c

c' = pT = { g = a b, p = g c, c' = p }

Symbolic Reachability

I FR1

R2...

R

= I Img(I,T)= R1 Img(R1,T)

Idea: represent reachable states by a formula

Essentially a BFS with symbolic representation.

Symbolic reachability, cont.

• Reachability fixed point:R0 = I

Ri+1 = Ri Img(Ri,T)

R = Ri

• F is reachable iff R F false• Image operator:

Img(Q,T) = V. (Q(V) T(V,V’))

We need a way to eliminate the quantifier, to get us back to an ordinary Boolean formula.

DPLL-style SAT solvers

• Objective:– Check satisfiability of a CNF formula

• literal: v or v• clause: disjunction of literals• CNF: conjunction of clauses

• Approach:– Branch: make arbitrary decisions– Propagate implication graph– Use conflicts to guide inference steps

SATO,GRASP,CHAFF,BERKMIN

SAT solvers can also generate refutation proofs!

The Implication Graph (BCP)

(a b) (b c d)

a

c

Decisions

b

Assignment: a b c d

d

Propositional Resolution

a b c a c d

b c d

When a conflict occurs, the implication graph isused to guide the resolution of clauses, so that thesame conflict will not occur again.

Conflict Clauses

(a b) (b c d) (b d)

a

c

Decisions

b

Assignment: a b c d

d

Conflict!

(b c )

resolve

Conflict!(a c)

resolve

Conflict!

Generating refutations

• Refutation = a proof of the null clause– Record a DAG containing all resolution steps performed during conflict clause generation.

– When null clause is generated, we can extract a proof of the null clause as a resolution DAG.

Original clauses

Derived clauses

Null clause

Circuit SAT

ab

c p

g

Can the circuit output be 1?

inputvariables output

variable

(a g) (b g)(a b g)

(g p) (c p)(g c p)

CNF(p)

p is satisfiable when theformula CNF(p) pis satisfiable

Bounded Model Checking

• Given– A finite transition system M– A property P

• Determine– Does M allow a counterexample to P of k transitions of fewer?

This problem can be translated to a SAT problem

BCCZ99

Symbolic Models (recall)

Transition system described by constraints...

ab cp

gg = a b

p = g c

c' = p

New notation: Q<n> means "add n primes to the symbols in Q"

T = { g = a b, p = g c, c' = p }

Bounded model checking

• Unfold the model k times:

U = T<0> T<1> ... T<k-1>

ab

cp

g ab

cp

g ab

cp

g

...I<0> F<k>

• Use SAT solver to check satisfiability of

I<0> U F<k>

• If unsatisfiable:• property has no Cex of length k• can produce a refutation proof P

Biere,et al. TACAS99

Bounded Model Checking

IR1

R2

= I Img(I,T)= R1 Img(R1,T)

F

I<0> T<0> T<1> … F<k>

……

BMC applications

• Debugging:– Can find counterexamples using a SAT solver

• Proving properties:– Only possible if a bound on the length of the shortest counterexample is known.• I.e., we need a diameter bound. The diameter is the maximum lenth of the shortest path between any two states.

– Worst case is exponential. Obtaining better bounds is sometimes possible, but generally intractable.

Unbounded Model Checking

• We consider a variety of methods to explioit SAT and BMC for unbounded model checking:– K-step induction– Abstraction

• Counterexample-based• Non-counterexample-based

– Exact image computations• SAT solver tests for fixed point• SAT solver computes image

– Over-approximate image computations

Induction

• The simple case: P is an inductive invariant– I => P– P T => P’

• Usually, P is not an inductive invariant

• BUT – a stronger inductive invariant R may exist (strengthening)– I => R– R T => R’– R => P

Induction

P

I

K-induction

• Induction:

SSS2000

P(s0)

i: P(si) P(si+1)

i: P(si)

• k-step induction:

P(s0..k-1)

i: P(si..i+k-1) P(si+k)

i: P(si)

K-induction with a SAT solver

• Recall: Uk = T<0> T<1> ... T<k-1>

• Two formulas to check:– Base case:

I<0> Uk-1 P<0>...P<k-1>

– Induction step:Uk P<0>...P<k-1>

P<k>

• If both are valid, then P always holds.

• If not, increase k and try again.

Simple path assumption

• Unfortunately, k-induction is not complete.– Some properties not k-inductive for any k.

• Simple path restriction:– There is a path to P iff there is a simple path to P (path with no repeated states).

P P P

Induction over simple paths

• Let simple(s0..k) be defined as:

i,j in 0..k : (i j) si sj

• k-induction over simple paths:

P(s0..k-1)

i: simple(s0..k) P(si..i+k-1) P(si+k)

i: P(si)

Must hold for k large enough, since a simple path cannot beunboundedly long. Length of longest simple path is calledrecurrence diameter.

...with a SAT solver

• For simple path restriction, let: Sk = t=0..k, u=t+1..k: v in V : vt = vu

(where V is the set of state variables).

• Two formulas to check:– Base case:

I<0> Uk-1 P<0>...P<k-1>

– Induction step:Sk Uk P<0>...P<k-1>

P<k>

• If both are valid, then P always holds.• If not, increase k and try again.

Termination

• Termination condition:k is the length of the longest simple path of the form

P* P

• This can be exponentially longer than the diameter.– example:

• loadable mod 2N counter where P is (count 2N-1)• diameter = 1• longest simple path = 2N

• Nice special cases:– P is a tautology (k=0)– P is inductive invariant (k=1)

Image computation methods

• Symbolic model checking without BDD's– Use SAT solver just for fixed-point detection• Abdulla, Bjesse and Een 2000• Williams, Biere, Clarke and Gupta 2000

– Adapt SAT solver to compute image directly• McMillan, 2002

Image over-approximation

• BMC and Craig interpolation allow us to compute image over-approximation relative to property.– Avoid computing exact image.– Maintain SAT solver's advantage of filtering out irrelevant facts.

Interpolation

• If A B = false, there exists an interpolant A' for (A,B) such that:

A A'A' B = false

A' refers only to common variables of A,B

• Example: – A = p q, B = q r, A' = q

• New result– given a resolution refutation of A

B, A' can be derived in linear time.

(Craig,57)

(Pudlak,Krajicek,97)

Interpolation-based MC

• Interpolation gives us– SAT-based algorithm for over-approximate image computation, using interpolation

– SAT-only symbolic model checking

(McMillan,2003)

Reachability

• Is there a path from I to F satisfying transitions T?

• Reachability fixed point:R0 = I

Ri+1 = Ri Img(Ri,T)

R = Ri

• Image operator:Img(Q,T) = V. (Q T)

• F is reachable iff R F false

Overapproximation

• An overapproximate image op. is Img' s.t.for all Q, Img(Q,T) implies Img'(Q,T)

• Overapprimate reachability:R'0 = I

R'i+1 = R'i Img'(R'i,T)

R' = R'i

• Img' is adequate (w.r.t.) F, when– if Q cannot reach F, Img’(Q,T) cannot reach F

• If Img' is adequate, then– F is reachable iff R' F false

Adequate image

Q F

Img(Q,T)

Reached from Q Can reach F

Img’(Q,T)

But how do you get an adequate Img'?

k-adequate image operator

• Img' is k-adequate (w.r.t.) F, when– if Q cannot reach F, Img’(Q,T) cannot reach F within k steps

• Note, if k > diameter, then k-adequate is equivalent to adequate.

Interpolation-based image• Idea -- use unfolding to enforce k-adequacy

A = Q T<0>

B = T<1> T<2> T<k-1> Fk

Fk = ¬P<1> ∨ ¬P<2> ∨ … ∨ ¬P<k>

Q FT T T T T T T

A B

t=1 t=k

Let Img'(Q)0= A', where A' is an interpolant for (A,B)...

Img' is k-adequate!

• Given the following BMC formula.

),()( 100 VVTVINIT

A B

A’

Huh?

• A A'– Img(Q,T) Img'(Q,T)

• A' B = false– Img’(Q,T) cannot reach F in k steps

• Hence Img' is k-adequate overapprox.

Q FT T T T T T T

A B

t=1 t=k

A'

But note, Img' is partial -- not defined if AB is sat.

k-adequate

IF1

F2

= I Img’(I,T)= F1 Img’(F1,T)

P

¬P

……k

……k

A1

A2

Using Interpolants

A’1

.

.

.

Using Interpolants (2)

I R1R1

I

R2R2 R3R3

¬P¬P

A’1A’1A’2A’2A’3A’3

Analogy To Reachability Analysis

Reachability algorithm

let k = 0repeat if I can reach F within k steps, answer reachable

R = I while Img'(R,T) F = false

R' = Img'(R,T) R if R' = R answer unreachable R = R' end while increase kend repeat

Termination

• Since k increases at every iteration, eventually k > d, the diameter, in which case Img' is adequate, and hence we terminate.

Notes:– don't need to know when k > d in order to terminate

– often termination occurs with k << d

Interpolation-based MC

• Fully SAT-based.• Inherits SAT solvers ability to concentrate on facts relevant to a property.

• Most effective when– Very large set of facts is available– Only a small subset are relevant to property

• For true properties, appears to converge for smaller k values.

Interpolation-Sequence

• If A1 A2 …Ak = false, there exists an interpolation-sequence A’0, A’1,…, A’k+1 for (A1,…Ak ) such that:

A’0=T and A’k+1=F

A’j Aj+1 A’j+1

A’j - over common variables of A1,…Aj and Aj+1,…Ak

• A’j equals the interpolant of A=A1…Aj and B=Aj+1 …Ak

– Given the same resolution graph

50

A1A2 A3 Ak Ak+1

A’1 A’2 A’3 A’k-1 A’k

• BMC formula partitioned in a different manner:

Interpolation-Sequence based MC(Vizel and Grumberg,2009)

51

I1,1

)(),(),()( 221100 VqVVTVVTVINIT

I1,

2 I2,2

I1I1

Using Interpolation-Sequence

52

INIT R1R1

INIT

R2R2 R3

R3

A1A1

AA A3A3

¬P¬P

)(),(),(),()( 33221100 VqVVTVVTVVTVINIT )(),()( 1100 VqVVTVINIT

A1,1A1,1

)(),(),()( 221100 VqVVTVVTVINIT

A2,2A2,2A1,2

A1,2

A1A1

A2A2

A3,3A3,3

A2,3A2,3A1,3

A1,3

Analogy to Forward Reachability

Conclusion

• SAT solvers are very effective at ignoring irrelevant facts

• SAT solvers can produce refutations• We can exploit in a number of ways:

– BMC– Abstraction for UMC (either CBA or PBA)– Abstract image computations using interpolation

This makes it possible to model check localizable properties large systems.

IC3 – The Breakthrough

• IC3 = Incremental Construction of Inductive Clauses for Indubitable Correctness

• The Goal: Find an Inductive Invariant stronger than P by learning relatively inductive facts (incrementally)– Recall: F is inductive invariant if

• I => F• F T => F’

– F is stronger than P, therefore F => P• F P T => F’ => P’

(Bradley,2010)

What Makes IC3 Special?

• No unrolling/unfolding of the transition function T is required

• All previous approaches require unrolling– Searching for an inductive invariant– Unrolling = A form of strengthening

• IC3 strengthen in a different way

Example

• M1:– I: x=1, y=1– T: x’= x+1, y’= y+x

• M2:– I: x=1, y=1– T: x’= x+y, y’= y+x

• P: y ≥ 1

Induction on M1

• I => P:– x=1 y=1 => y ≥ 1

• BUT: P T ≠> P’– y ≥ 1 x’=x+1 y’=x+y => y’ ≥ 1

• Incremental Proof:– F = x ≥ 0– x ≥ 0 x’=x+1 =>x’ ≥ 0 – Clearly, F P T => P’

Induction on M2

• I => P:– x=1 y=1 => y ≥ 1

• BUT: P T ≠> P’– y ≥ 1 x’=x+y y’=x+y => y’ ≥ 1

• Monolithic Proof:– Choose a new P*, stronger than P– Note that F = x ≥ 0 is also not inductive

• Set P* = x ≥ 0 y ≥ 1– I => P*– P* T => P*’

IC3 Basics

• Compute Over-approximated Reachability Sequence (OARS) <F0,F1,…,Fk+1> s.t.

– F0 = I

– Fi => P

– Fi => Fi+1

– Fi T => F’i+1

• Notations:– Cube s: conjunction of literals

• v1 v2 ¬v3 - Represents a state

– s is a cube => ¬s is a clause (DeMorgan)– P is inductive relative to F if F is inductive and F P T => P’

OARS

IR1

R2

= I Img(I,T)= R1 Img(R1,T)

PF1

F2

¬P

A Backward Search

• Search for a predecessor s to some error state: P T ¬P’– If none exists, property holds:

• P T ¬P’ = ¬(P T => P’)

• Try to block s– P = P ¬s

IC3 - Initialization

• Check the following two formulas:– I ¬P’– I T ¬P’

• If both are unsatisfiable then:– I => P– I T => P’

• Therefore– F0 = I, F1 = P

IC3 - Initialization

I

F0 P

F1

Iteration

IF1

F2

P

Fk+1

……

Fk Fk-1

IC3 - Iteration

• Given an OARS <F0,F1,…,Fk>, define Fk+1=P

• Apply a backward search– Find predecessor s in Fk that can reach a bad state• Fk T => P’ (Fk T ¬P’)

– If none exists, move to next iteration– If exists, try to find a predecessor t to s in Fk-1

• Fk-1 T => ¬s’ (Fk-1 T s’)

– If non exists, s can be removed from Fk

• Fk = Fk ¬s

– Otherwise: Recur on (t,k-1)• We call (t,k-1) a proof obligation

• If we can reach I, a CEX exists

That Simple?

• Looks simple• But this “simple” does NOT work• Simple = States Enumeration

– Too many states…

• Are we enumerating states?– Yes (when it doesn’t work)– But, not really

Observation 1

• A state s in Fk can reach a bad state in one transition

• Important Fact: s is not in Fk-1 (!!)

– Fk-1 T => Fk

– Fk => P

– If s was in Fk-1 we would have found it in an earlier iteration

• Therefore: Fk-2 T => ¬s’

Inductive Generalization

• A state s in Fk can reach a bad state in one transition

• Assuming s in not reachable:– Fk-1 T => ¬s’

• BUT, this is stronger: Fk-1 ¬s T => ¬s’

• This looks familiar!– I => ¬s

• Otherwise, CEX! (I ≠> ¬s s is in I)

– ¬s is inductive relative to Fk-1

Inductive Generalization

• So we know Fk-1 ¬s T => ¬s’

• And, ¬s is a clause• Generalize: Find a sub-clause c ¬s s.t. Fk-1 c T => c’ (and I => c)– Sub clause means less literals– Less literals implies less satisfying assignments• (a ∨ b ∨ c) vs. (a ∨ b)

– Stronger inductive fact

• More states are removed from Fk, making it stronger (closer to Rk)

Observation 2

• A state s in Fi can reach a bad state in a number of transitions

• s is also in Fj for j > i, a longer CEX may exist– Fi => Fj

– s may not be reachable in i steps, but it may be reachable in j steps

Push Forward

IF1

F2

P

Fk+1

……

Fk Fk-1

Push Forward

• s is removed from Fi – by conjoining a sub-clause c

– Fi = Fi c

• Pushing forward for j > i– Fj c T => c’ holds

– c is inductive relative to Fj

– Fj = Fj c

• s cannot be blocked at level j > i– Add a proof obligation (s,j)– If s is reachable from I, CEX!

IC3 – Key Ingredients

• Backward Search– Find a state s that can reach a bad state in a number of steps

– s may not be reachable (over-approximations)

• Block a State– Do it efficient, block more than s

• Generalization

• Push Forward– An inductive fact at frame i, may also be inductive at higher frames

– If not, a longer CEX is found

The End