relaxations and moves for map estimation in mrfs m. pawan kumar stanfordstanford vladimir...

Relaxations and Moves forMAP Estimation in MRFs

M. Pawan Kumar

STANFORD

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Vladimir Kolmogorov

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Philip Torr

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Daphne Koller

Our Problem

v1 v2 v3 v4

2

5

4

2

6

3

3

7

0

1 1

0

0

2 3

1

1

4 1

0Label l1

Label l2

Random Variables V = {v1, ... ,v4}

Label Set L = {l1, l2}

Labeling f: V L (shown in red)

Our Problem

v1 v2 v3 v4

2

5

4

2

6

3

3

7

0

1 1

0

0

2 3

1

1

4 1

0Label l1

Label l2

Random Variables V = {v1, ... ,v4}

Label Set L = {l1, l2}

Labeling f: V L (shown in red)

Energy of Labeling E(f) = 13 (shown in green)

Our Problem

v1 v2 v3 v4

2

5

4

2

6

3

3

7

0

1 1

0

0

2 3

1

1

4 1

0Label l1

Label l2

Find f* = argminf E(f)

Arbitrary topology, discrete label set, potentials (NP-hard)

Pairwise energy function: unary and pairwise potentials(still NP-hard)

Outline

• Convex Relaxations– Integer Programming Formulation– LP Relaxation– SDP Relaxation– SOCP Relaxation– Comparing Relaxations

• Move Making Algorithms

• Some Interesting Open Problems

Integer Programming Formulation

v1 v2

2

5

4

2

0

1 3

0

Unary Potentials

Unary Potential u = [ 5

Cost of v1 = 1

2

Cost of v1 = 2

; 2 4 ]

Labeling f shown in red

Label l1

Label l2

Label vector x = [ -1

v1 1

1

v1 = 2

; 1 -1 ]T

Recall that the aim is to find the optimal x


v1 v2

2

5

4

2

0

1 3

0

Unary Potentials


Label l1

Label l2

Unary Potential u = [ 5 2 ; 2 4 ]

Label vector x = [ -1 1 ; 1 -1 ]T

Sum of Unary Potentials = 12

∑i ui (1 + xi)


v1 v2

2

5

4

2

0

1 3

0

Unary Potentials


Label l1

Label l2

Unary Potential u = [ 5 2 ; 2 4 ]

0Cost of v1 = 1 and v1 = 1

0

00

0Cost of v1 = 1 and v2 = 1

3

Cost of v1 = 1 and v2 = 21 0

00

0 0

10

3 0

Pairwise Potential P


v1 v2

2

5

4

2

0

1 3

0

Pairwise Potentials


Label l1

Label l2


0 0

00

0 3

1 0

00

0 0

10

3 0

Sum of Pairwise Potentials14

∑ij Pij (1 + xi)(1+xj)


v1 v2

2

5

4

2

0

1 3

0

Pairwise Potentials


Label l1

Label l2

Sum of Pairwise Potentials14

∑ij Pij (1 + xi +xj + xixj)

14

∑ij Pij (1 + xi + xj + Xij)=

X = x xT Xij = xi xj


v1 v2

2

5

4

2

0

1 3

0

Pairwise Potentials


Label l1

Label l2


0 0

00

0 3

1 0

00

0 0

10

3 0

Constraints

• Uniqueness Constraint

∑ xi = 2 - |L|i va

• Integer Constraints

xi {-1,1}

X = x xT


x* = argmin 12

∑ ui (1 + xi) + 14

∑ Pij (1 + xi + xj + Xij)

∑ xi = 2 - |L|i va

xi {-1,1}

X = x xT

ConvexNon-Convex


Outline




LP Relaxation

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i va

xi {-1,1}

X = x xT

Retain Convex PartSchlesinger, 1976

Relax Non-ConvexConstraint

LP Relaxation

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i va

xi [-1,1]

X = x xT



LP Relaxation

X = x xT

Schlesinger, 1976

Xij [-1,1]

1 + xi + xj + Xij ≥ 0

∑ Xij = (2 - |L|) xij vb

LP Relaxation

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i va

xi [-1,1]

X = x xT



LP Relaxation

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i va

xi [-1,1],


Xij [-1,1]

1 + xi + xj + Xij ≥ 0

∑ Xij = (2 - |L|) xij vb

Outline




SDP Relaxation

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i va

xi {-1,1}

X = x xT

Retain Convex PartLasserre, 2000


SDP Relaxation

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i va

xi [-1,1]

X = x xT Relax Non-ConvexConstraint

Lasserre, 2000Retain Convex Part

x1

x2

xn

1

...

1 x1 x2... xn

1 xT

x X

=

Rank = 1

Xii = 1

Positive SemidefiniteConvex

Non-Convex

SDP Relaxation

x1

x2

xn

1

...

1 x1 x2... xn

1 xT

x X

=

Xii = 1

Positive SemidefiniteConvex

SDP Relaxation

SDP Relaxation

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i va

xi [-1,1]

X = x xT Relax Non-ConvexConstraint


SDP Relaxation

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i va

xi [-1,1]

Xii = 1 X - xxT 0

Accurate Inefficient


PositiveSemidefinite

Outline




SOCP Relaxation

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i va

xi [-1,1]

Xii = 1 X - xxT 0

Derive SOCP relaxation from the SDP relaxation

Further Relaxation

SOCP Relaxation

Choose a matrix C1 = UUT 0

Kim and Kojima, 2000

Choose a sub-graph G

Variables xG and XG

(XG - xGxGT)C1 ≥ 0


REPEAT

Outline




Dominating Relaxation

For all MAP Estimation problem (u, P)

A dominates B

A B

≥

Dominating relaxations are better

SOCP Relaxation




Variables xG and XG


If G is a tree, LP dominates SOCP

Examples

Muramatsu and Suzuki, 2003(MAXCUT)

Ravikumar and Lafferty, 2006 (QP Relaxation)

Kumar, Torr and Zisserman, 2006 (Equivalent SOCP Relaxation)

SOCP Relaxation




Variables xG and XG


If G is a cycle with non-negative P

SOCP Relaxation




Variables xG and XG


If G is an even cycle with non-positive P

SOCP Relaxation




Variables xG and XG


If G is an odd cycle with 1 non-positive P

SOCP Relaxation

What about other cycles?

Dominated by linear cycle inequalities

Cliques?

Dominated by clique inequalities

Kumar, Kolmogorov and Torr, 2007

Outline

• Convex Relaxations

• Move Making Algorithms– State of the Art– Comparison with LP Relaxation– Improved Moves


MRFs in Vision

va vb

li

lkPab(i,k)Pab(i,k) = wab min{ d(i-k), M }

wab is non-negative

Truncated Linear Truncated Quadratic

d(.) is a semi-metric distanceua(i) ub(k)

Move Making

Search Neighbourhood

Current Solution

Optimal Move

Solution Space

En

erg

y

Slide courtesy of Pushmeet Kohli

Outline




Expansion MoveVariables take label or retain current label

Boykov, Veksler, Zabih 2001Slide courtesy of Pushmeet Kohli

Sky

House

Tree

Ground

Initialize with TreeStatus: Expand GroundExpand HouseExpand Sky

[Boykov, Veksler, Zabih]

Expansion MoveVariables take label or retain current label

Boykov, Veksler, Zabih 2001Slide courtesy of Pushmeet Kohli

Outline




Multiplicative Bounds

LPMove-Making

Potts

Truncated Linear

Truncated Quadratic

Metric Labeling

2 2

2 + √2 2M

O(√M) 2M

O(log h) 2M

Expansion Bounds as bad as ICM Bounds

Outline




Randomized Rounding

0 y’0 y’i y’k y’h = 1

y’i = y0 + y1 + … + yi

Choose an interval of length L’

yi = (1 + xi)/2

Randomized Rounding

0 y’0 y’i y’k y’h = 1

Generate a random number r (0,1]

r

y’i = y0 + y1 + … + yi

yi = (1 + xi)/2

Randomized Rounding

0 y’0 y’i y’k y’h = 1

Assign label next to r (if within the interval)

r

y’i = y0 + y1 + … + yi

yi = (1 + xi)/2

Move Making

va vb

• Initialize the labeling

• Choose interval I of L’ labels

• Each variable can

• Retain old label

• Choose a label from I

• Choose best labeling

Iterate over intervals

Non-submodular move? Submodular overestimation

Truncated Convex Models

Pab(i,k) = wab min{ d(i-k), M }

Truncated Linear Truncated Quadratic

d(.) is convex d(x+1) - 2d(x) + d(x-1) ≥ 0

Move Making

va vb

• Choose interval I of L’ labels

• Each variable can

• Retain old label

• Choose a label from I

• Choose best labeling

Large L’ => Non-submodular

Move Making

va vb Submodular problem

Move Making

va vb Non-submodularProblem

Move Making

va vb Submodular problem

Ishikawa, 2003; Veksler, 2007

Move Making

va vb

am+1

am+2

an

t

am+2

bm+1

bm+2

bn

bm+2

s

LP Bounds

Kumar and Torr, NIPS 08

Type of Problem Bound

Potts 2

Truncated Linear 2 + √2

Truncated Quadratic O(√M)

Metric Labeling O(log h)

Kumar and Koller, UAI 09

Move Making

Outline




Problem 1

Relationship between rounding and move-making?

What happens when n < h ??(Should we even use move-making here??)

What about semi-metric MRFs??

Problem 2

Graph-cuts based image segmentation

Vicente, Kolmogorov, Rother, 2008

Problem 2

Image segmentation with connectivity prior

Vicente, Kolmogorov, Rother, 2008

Problem 2++

Kumar and Koller, 20??

Questions??

http://ai.stanford.edu/~pawan

relaxations and moves for map estimation in mrfs m. pawan kumar stanfordstanford vladimir...

Documents

x j x ij x

x x t x ij

x x t schlesinger

label vector x

lp relaxation x

integer constraints

ij p ij

red label