1

LP, extended maxflow, TRWOR: How to understand

Vladimir’s most recent workRamin Zabih

Cornell University

2

Outline

• Linear programming and duality

• Extended maxflow

• TRW

Some slides from Ziv Bar-Yossef (Technion)

Much content from Vladimir Kolmogorov

3

Linear Programming

• Fast algorithms - O(n3) or so

• Numerous interesting special cases can be encoded as LP’s

• Many ways to convert between LP’s– Sometimes you can transform into a fast

special case

• Can sometimes solve integer programs – “Integrality gap”: IP versus LP solution

4

Example

5

0

2

4

6

8

10

0 2 4 6 8 10 0

2

4

6

8

10

0 2 4 6 8 10 0

2

4

6

8

10

0 2 4 6 8 10 0

2

4

6

8

10

0 2 4 6 8 10

6

Solving LP Graphically

Feasible region

Vertices

Objective function

Optimal solution

7

Notes on solutions

• Maximum is always at a vertex– Simplex algorithm

• Certain constraints are “tight”– So you can check if the solution is integral!

8

Convert any LP to Standard Form

•

•

•

•

9

Bounding the Optimal Solution

• Suppose we wish to find an upper bound on the optimal solution

• Multiply first constraint by 3: z ≤ 90• Add first and second constraints: z ≤ 54• What linear combination gives the best upper bound?

10

Bound Optimal Solution by LP

Primal

Dual

11

Primal vs. Dual

• Lemma: The dual of the dual is the primal.

• Theorem (Weak Duality)For any feasible solution x for the primal (cost vector c), and for any feasible solution y for the dual (cost b),

• Theorem (Strong Duality)If the primal has an optimal solution x*, then the dual also has an optimal solution y*, and

12

RDZ’s favorite dual

13

Notes

• Solutions to dual are not obviously cuts– Why are the yI,j values 0 or 1?

• Any flow · any cut– We can find the biggest flow fast

• By strong duality this is the value of the cheapest cut– But how do we actually find the cut?– How about in generality?

14

Complementary slackness

• Easy to go from dual optimum solution to primal optimal solution– No guarantees if dual solution isn’t optimal

• Dual variable yj is zero iff j’th constraint is slack (not tight)– So you know which constraint lines intersect

at the optimum!

15

LP for energy minimization

• We can formulate E(x) as an IP– Since IP’s are NP-hard

– We replace xp by indicator variables xp;k which are like probabilities (non negative, sum to 1)

• Similarly, need xpq;ij for adjacent pixels

• Parameters are costs

– We can then relax this IP to an LP and solve it• If integer solutions, we are done• Note that we first need to actually solve the LP!

16

Primal LP [Chekuri00]

17

Importance of this LP

• This LP is vital for both “extended maxflow” (quadratic pseudo-boolean optimization) and TRW(-S)– Both work on its dual

• So by solving the dual we are maximizing a lower bound on the energy

18

Extended maxflow vs TRW

• Maxflow is only for binary labels – I.e., for the expansion move algorithm

• Maxflow solves the LP exactly

• Solutions are [1 0] or [0 1] or [.5 .5]

• No ambiguities if problem is regular

• Automatically does “flipping”

• If not regular, solution is persistent

19

Extended maxflow

),(

),()()(vu

vuuvu

uuconst xxxE x

non-negative

const - lower bound on the energy:

xx constE )(

maximize

• Maximize lower bound on E

20

TRW idea• Decompose graph into a collection of trees

with different parameterizations

• Weak tree agreement (WTA): – Consider all trees incorporating a node. Their

parameterizations with cost 0 all agree.• Normal form tree parameterization

– node params are min marginals (after BP)

• Goal is normal form and WTA

constEjjx

ss

)|(min)( x

21

TRW(-S) properties

• WTA and normal form holds at the dual solution (max ()) – For binary labels only!

• TRW(-S) tries to achieve this– TRW doesn’t always converge to a solution

where this is true (TRW-S does)– What it converges for isn’t necessarily the

dual solution (except for 2 labels)– The primal solution may not be integral