Transcript
Page 1: Submodularization for Binary  Pairwise  Energies

Submodularization for Binary Pairwise Energies

Lena Gorelick

joint work with

O. Veksler

I. Ben Ayed

A. Delong

Y. Boykov

Page 2: Submodularization for Binary  Pairwise  Energies

2

Νqp

xxp

pp qpxfE

,][1(x)

Example of Simple Binary Energy

Potts Modelf 1,0x

Page 3: Submodularization for Binary  Pairwise  Energies

3

Binary Pairwise EnergyQuadratic Form

Submodular Energy global optimum with graphcut (Boros &

Hammer, 2002)

qp

qppqp

pp xxvxuE,

(x) 1,0x

pqpq vv 0

Potts Model

Page 4: Submodularization for Binary  Pairwise  Energies

4

0, pqvpq

Non-Submodular Energy NP-hard

Binary Pairwise Energy Quadratic Form

const.)(,

qp

qppqp

pp xxvxuE x

Middlebury

Image credit: Carlos Hernandes

Page 5: Submodularization for Binary  Pairwise  Energies

5

Standard Optimization Methods

General energy - NP-hardApproximate methods:

Global Linearization: QPBO, TRWS, SRMP (Kolmogorov et al. 2006, 2014)

Local Linearization: parallel ICM, IPFP (Leordeanu, 2009)

Message Passing: BP (Pearl 1989)

Page 6: Submodularization for Binary  Pairwise  Energies

6

Related WorkGlobal Linearization

)(xE QPBO, TRWS, SRMP (Kolmogorov et al. 2006,

2014)

)(~min..

yyE

Cts

Linearize introducing large number ofvariables and constraintsSolve relaxed LPor its dual

Integrality Gap

*relaxedyRounding

*integer x

Page 7: Submodularization for Binary  Pairwise  Energies

Related WorkIterative Local Linearization

7

parallel ICM (Leordeanu, 2009) large steps weak min

IPFP (Leordeanu, 2009) controls step size by relaxation Integrality Gap

)(xE

x

txEt(x)~

1tx

N}1,0{Bounded domain of discrete configurations

Page 8: Submodularization for Binary  Pairwise  Energies

8

Local Submodular Approximation LSA

Local Submodular Approximation model

Non-linear

Two ways to control step size

Et(x)~

)(xE

x

tx1tx

N}1,0{Bounded domain of discrete configurations

Page 9: Submodularization for Binary  Pairwise  Energies

9

Trust Region Local submodular approximation

Auxiliary Functions = Surrogate Functions = Upper Bounds = Majorize-Minimize Local submodular upper bound

Never leave the discrete domain

Iterative Optimization Framework

LSA-AUX

LSA-TR

Page 10: Submodularization for Binary  Pairwise  Energies

10

Iterative Optimization FrameworkTrust Region:

Discrete High Order Energies Relaxed Quadratic Binary Energies Levenberg Marquardt

Auxiliary Functions=Surrogate Functions =Upper Bounds = Majorize-Minimize Discrete High Order Energies

Gorelick et al. 2012,2013

Ben Ayed et al. 2013

Olsson et al. 2008

Narasimhan & Bilmes 2005

Rother et al. 2006

Hartley & Zisserman 2004

Page 11: Submodularization for Binary  Pairwise  Energies

11

Local Submodular ApproximationLSA

qp

qppqp

pp xxvxuE,

)(x

)()()( sup xxx EEE sub

+- x

tx)(xE

Page 12: Submodularization for Binary  Pairwise  Energies

12

Local Submodular ApproximationLSA

)()()( sup xxx EEE sub

x

tx)(xE

Page 13: Submodularization for Binary  Pairwise  Energies

13

Approximate around

Local Submodular ApproximationLSA

)()()( sup xxx EEE sub

)(xE tx

)(~ x(x) subt EE

Et(x)~

x

tx)(xE

Page 14: Submodularization for Binary  Pairwise  Energies

14

Submodular functionLSA

Approximate around

Local Submodular ApproximationLSA

)()()( sup xxx EEE sub

)(xE tx

)()(~ xx(x) approxt

subt EEE

Linear Approximation

Et(x)~

x

tx)(xE

Page 15: Submodularization for Binary  Pairwise  Energies

15

0,)(sup pqqppqpq

vxxvE x

Linear Approximation of the Supermodular Term

Page 16: Submodularization for Binary  Pairwise  Energies

16

qppq xxv

Linear Approximation of the Supermodular Term

Page 17: Submodularization for Binary  Pairwise  Energies

17

Linear Approximation of the Supermodular Term

0 xy

Page 18: Submodularization for Binary  Pairwise  Energies

18

0

1

x

y

1

Linear Approximation of the Supermodular Term

1,0

0,0

1,1

0,1

0 xy

Page 19: Submodularization for Binary  Pairwise  Energies

19

Linear Approximation of the Supermodular Term

0

1

1

x

y

1,0

0 xy

Page 20: Submodularization for Binary  Pairwise  Energies

20

1

constyx 0

0

1

Linear Approximation of the Supermodular Term

x

y

1,0

0,0

1,1

Linear (Unary)

approximation

Page 21: Submodularization for Binary  Pairwise  Energies

21

Linear Approximation of the Supermodular Term

0

1

1

constyvxu

x

y

Page 22: Submodularization for Binary  Pairwise  Energies

22

LSA-TR:Trust Region Overview

)(xE

x

tx

)()()( sup xxx EEE sub

Page 23: Submodularization for Binary  Pairwise  Energies

23

LSA-TR:Trust Region Overview

)(xE

x

tx

Et(x)~

Newton Step

)()(~ xx(x) approxt

subt EEE

1tx

Page 24: Submodularization for Binary  Pairwise  Energies

LSA-TRTrust Region Sub-Problem

24

)(xE

x

tx

Et(x)~

Trust Region

)()(~ xx(x) approxt

subt EEE

Trust Region Sub-Problem

td ||||s.t. txx

24

NP-hard!Constrained Submodular Optimization

Page 25: Submodularization for Binary  Pairwise  Energies

25

fixed in each iteration inversely related to trust region size adjusted based on quality of approximation

LSA-TR: Approximate TR sub-problem

||||)()(

tt

approxt

subt EEL

xxxx(x)

Unary TermsBoykov et al. 2006

Gorelick et al. 2013

t

Submodular

Page 26: Submodularization for Binary  Pairwise  Energies

26

Experiments & Results

Page 27: Submodularization for Binary  Pairwise  Energies

27

Experiments & Results:Deconvolution Binary De-convolution All pairwise terms supermodular

Original Img Convolved Convolved+Noise

?

Page 28: Submodularization for Binary  Pairwise  Energies

Experiments & Results:Deconvolution

28

Noise:N(0,0.05)

SRMP

QPBOI

TRWS FTR-L

LBP

LSA-TR (0.3 sec.)E=21.13

LSA-AUX (0.04 sec)E=34.70

TRWS:5000 iter.E=65.07

LBP5000 iter.E=40.15

QPBO(0.1 sec.)

QPBO-I (0.2 sec.)E=66.44

IPFP(0.4 sec.)E=32.90

SRMP:5000 iter.E=39.06

Page 29: Submodularization for Binary  Pairwise  Energies

Experiments & Results:Segmentation of Thin Structures

29

QPBO

QPBO-IE= -77.08

LBPE= -84.54

IPFPE= 163.25

Image

SRMP

Potts, v<0(submodula

r)

with edge repulsion, v>0(non-submodular)

TRWSE= -67.21

LSA-TRE= -175.05

LSA-AUXE= -120.03

SRMPE= -101.61

Repulsion = Reward different labels across high contrast edges

Page 30: Submodularization for Binary  Pairwise  Energies

30

Experiments & Results:Inpainting dtf-chinesechar database

LSA-TRInput ImgGround Truth

Kappes et al., 2013

Page 31: Submodularization for Binary  Pairwise  Energies

31

Experiments & Results:In-painting Chinese Characters

Page 32: Submodularization for Binary  Pairwise  Energies

32

Curvature Regularization Efficient Squared Curvature model –

(Nieuwenhuis et al. 2014, poster on Friday)Potts Model Elastica

90-degree curvature

Heber et al. 2012

El-Zehiry&Grady, 2010

Our curvature Using LSA-TR

Page 33: Submodularization for Binary  Pairwise  Energies

33

Summary Two novel discrete optimization methods

Simple, efficient, state-of-art results The code is available online -

http://vision.csd.uwo.ca/code/

Extensions: Find new applications▪ Convexity Shape Prior (in ECCV14)

Alternative optimization framework with LSA▪ Pseudo-Bounds (in ECCV14)

Please come by our poster


Top Related