measuring uncertainty in graph cut solutions pushmeet kohli philip h.s. torr department of computing...

Measuring Uncertainty in Graph Cut Solutions

Pushmeet Kohli Philip H.S. Torr

Department of Computing Oxford Brookes University

Objective

Labelling problem

t

s

Graph Cut

st-mincut

Belief or Confidence

Most ProbableSolution

No uncertainty measureassociated with the solution

Outline

Inference in Graphical Models Inference using Graph Cuts Computing Min-marginals using Graph

Cuts Flow Potentials and Min-marginals Results

Inference in Graphical Models

x* = arg max Pr(x|D) x

E(x|D) = -log Pr(x|D) + constant

x* = arg min E(x|D) x

x: Set of latent variablesD: Observed Data

x*: Most Probable (MAP) SolutionPr: Joint Posterior Probability


.2 .1 .05

.35 .05 .2

.01 .03 .01

A

B 0 1 2

0

1

2

Joint Distribution

Pr(A,B)

MAP Solution

arg max Pr(A,B) {A=1, B=0}


Marginal- Sum the joint probability over all other variables.

.2 .1 .05

.35 .05 .2

.01 .03 .01

A

B 0 1 2

0

1

2

P(A=1) = 0.6


.2 .1 .05

.35 .05 .2

.01 .03 .01

A

B 0 1 2

0

1

2

Max- Marginals (µ)- Maximum joint probability over all other variables.

µA,1 = 0.35


.2 .1 .05

.35 .05 .2

.01 .03 .01

A

B 0 1 2

0

1

2

µA,1 = 0.35

µA,0 = 0.2

µA,2 = 0.03

Confidence or Belief (σ)- Normalized max-marginals σA,1 = µA,1 / Σx µA,x

= 0.35/ 0.58= 0.603


Min-Marginals Energies(ψ)- Minimize joint energy over all other variables.

- Related to max-marginals as:

µj = (1/z)*exp(-ψj)




- Can be used to compute confidence as:

σj = µj / Σa µa = exp(-ψi) / Σa exp(-ψa)





- Can be used to compute confidence as:

How to compute min-marginal energies using Graph Cuts?

σj = µj / Σa µa = exp(-ψi) / Σa exp(-ψa)



Graphical Model

Topology

Tree Graph with cycles

Belief Propagation and variants

Exact solution

True Marginals/ min-marginals

Approximate solution

Approximate Marginals/ min-marginals

Graph Cuts

No Marginals/

Min-Marginals

Class 1: Single Max-flow Computation, Exact Solution

Class 2: Expansions/ Swap Moves, Approximate Solution

Outline



• Given energy functions E(x|D),

Compute: arg min E(x|D)

Inference using Graph cuts

x


• Certain E(x|D) can be minimized using graph cuts exactly.


Compute: arg min E(x|D)x


• Class of energy function and graph contruction

- Binary random variables

- Submodular functions (Kolmogorov & Zabih, ECCV 2002)

- Multi-valued variables - Convex Pair-wise Terms (Ishikawa, PAMI 2003)

• Certain E(x|D) can be minimized using graph cuts exactly.


Compute: arg min E(x|D)x

EMRF(a1,a2)

Sink (0)

Source (1)

a1

a2

Graph Construction for Binary Random Variables


a1

a2

EMRF(a1,a2) = 2a1

2t-edges

(unary terms)


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1

a1

a2

2

5


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2

a1

a2

2

5

9

4


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2

a1

a2

2

5

9

4

2

n-edges(pair-wise term)


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a1

a2

2

5

9

4

2

1


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a1

a2

2

5

9

4

2

1

a1 = 1 a2 = 1

EMRF(1,1) = 11

Cost of st-cut = 11


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a1

a2

2

5

9

4

2

1

a1 = 1 a2 = 0

EMRF(1,0) = 8

Cost of st-cut = 8


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a1

a2

2

5

9

4

2

1

a1 = 1 a2 = 0

EMRF(1,0) = 8

Cost of st-cut = 8


MAP Solutiona1,map = 1 a2,map = 0 Sink (0)

Source (1)

Outline



Computing Min-marginals using Graph Cuts


Instead of minimizing E(.), minimize a projection of E(.) where the value of latent

variable xv is fixed to label j.

All projections of a sub-modular function aresub-modular [Kolmogorov and Zabih, ECCV 2002]


Instead of minimizing E(.), minimize a projection of E(.) where the value of latent

variable xv is fixed to label j.

Problem Solved? Not Really!


• Computing a min-marginal requires computation of a st-cut.

• Typical Segmentation problem- 640x480 image, 2 labels- Variables = 640x480 = 307200- Number of Min-marginals = 307200x2 = 614400 - Time taken for 1 graph cut = .3 seconds- Total computation time = 614400x0.3 = 184320 sec

= 51.2 hours!!!

Energy Projections and Graph Construction

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a1

a2

2

5

9

4

2

1

Sink (0)

Source (1)


EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a2= 1

EMRF(a1,1) = 2a1 + 5ā1+ 9 + ā1

a1

a2

2

5

9

4

2

1

Sink (0)

Source (1)


EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a1

a2

2

6

9

Sink (0)

Source (1) a2= 1

EMRF(a1,1) = 2a1 + 5ā1+ 9 + ā1

EMRF(a1,1) = 2a1 + 6ā1+ 9


EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 + Kā2

a1

a2

2

5

9

4

2

1

Sink (0)

Source (1)

∞

A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection.

Alternative Construction

K


a1

a2

2

5

9

4

2

1

Sink (0)

Source (1) A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection.

• The minimum value of the energy projection can be calculated by using the same graph.



∞K


a1

a2

2

5

9

4

2

1

Sink (0)

Source (1) A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection.



K> Sum of Outgoing/Incoming edges


∞K





• Results in a small change in the graph.

a1

a2

2

5

9

4

2

1

Sink (0)

Source (1)

∞K




a1

a2

2

5

9

4

2

1

Sink (0)

Source (1)

∞


K


• Results in a small change in the graph.

Solve using Dynamic Graph Cuts


Dynamic Graph Cuts

- Kohli and Torr [ICCV 2005]

Graph 1

Graph 2

Energy function

Projection of Energy function

Dynamic Graph Cuts


Graph 1

Graph 2

Energy function

Projection of Energy function

Graph 1 and 2 are similar

Dynamic Graph Cuts


Graph 1

Graph 2

Dynamic Graph Cuts


Graph 1 Residual Graph 1 st-cut 1


Computationally Expensive Procedure

Compute Max-flow

Compute Max-flow

Dynamic Graph Cuts




Re-parameterized Graph 2

Compute Max-flow

Dynamic Graph Cuts




Re-parameterized Graph 2same

solution

Dynamic Graph Cuts




Re-parameterized Graph 2similar edge

weights

Dynamic Graph Cuts





Extremely Fast Operation

Compute Max-flow

Compute Max-flow

Dynamic Graph Cuts





Extremely Fast Operation

300 msec

0.002msec

Summary of the Algorithm

• Construct graph G for minimizing energy E

• For computing min-marginals do:

- Obtain graph G* ≡ energy function projection E*(By adding constraining edges)

- Find the maximum flow in G* using dynamic graph cut algorithm [Kohli and Torr, ICCV 2005]

Extension to Multiple Labels

Graph construction for multi-label random variables (Ishikawa PAMI 2003)

Labels: l1 …. ln

Latent variables: x1 …. xn

x1 = l3 x2 = l2

MAP Labels

x3 = l2 x4 = l1

Extension to Multiple Labels

Graph construction for multi-label random variables (Ishikawa PAMI 2003)

Labels: l1 …. ln

Latent variables: x1 …. xn

x4 = l3

Graph for Projection

Outline



Min-Marginals and Flow Potentials

• Flow Potentials- maximum amount of flow that can be passed from the node to a particular terminal.


1 2 3

4 5

s

t

3

29

7

1 1 4

3 1

2 11

Graph



1 2 3

4 5

s

t

3

29

7

1 1 4

3 1

2 11

Graph


Source (s) flow potential of node 4


1 2 3

4 5

s

t

3

29

7

1 1 4

3 1

2 11

Graph

Source (s) flow potential of node 4

1 2 3

4 5

s

t

1 1

1 1



1 2 3

4 5

s

t

3

29

7

1 1 4

3 1

2 11

Graph


Sink (t) flow potential of node 4


1 2 3

4 5

s

t

3

29

7

1 1 4

3 1

2 11

Graph

Sink (t) flow potential of node 4

1 2 3

4 5

s

t

2

29



Relationship between Min-marginal of binary latent variables and flow potential.

MAP Solution energy

Flow potential in residual graph

+Min-marginal =

For details: See theorem 1.

Implications: Any algorithm for computing flow potentials can be used to compute min-marginals.

Outline



• Typical Segmentation problem- 640x480 image, 2 labels

• Computation Times- Naïve Approach = 51.2 hours!!!- Our Algorithm (Dynamic Graph Cuts) = 1.2 seconds

Experimental Evaluation

Experimental Evaluation

Computation Times (in seconds) for binary variables

Problem Size

(No. of Variables/

Neighbourhood)

MAP solution

(single maxflow computation)

Computing all min-marginals

1x105, 4-neighbourhood 0.18 0.70

2x105, 4-neighbourhood 0.46 1.34

4x105 , 4-neighbourhood 0.92 3.15






Min-Marginals in Image segmentation

Unary likelihood Contrast Term

Uniform Prior(Potts Model)

Image Segmentation Energy Boykov and Jolly [ICCV 2001], Blake et al. [ECCV 2004]

xi = binary variable representing label (‘fg’ or ‘bg’) of pixel i


ImageMAP Solution Belief - Foreground

Lowsmoothness

Highsmoothness

Moderatesmoothness

Colour Scale

1

0

0.5

[MSR]


encourage smoothness








How smoothness effects solutions?

encourage smoothness


Effect of increasing pair-wise terms of the energy function

Image


Effect of increasing pair-wise terms of the energy function

MAP Segmentation Foreground Confidence Map

Concluding Remarks

• Efficient method for computing exact min-marginals for certain labelling problems• Relationship between Flow-potentials and min-marginals

Applications• Computing M Most Probable Solutions• Min-marginals for parameter learning• Hierarchical Segmentation

Future Work• Efficient min-marginal computation for general problems

Thank You

measuring uncertainty in graph cut solutions pushmeet kohli philip h.s. torr department of computing...

Documents

marginals graph

max marginals

graphical models n inference

marginals class

marginals energies

marginals n results

graph cuts x slide

graphical models x