Submodularization for Binary Pairwise Energies

Lena Gorelick

joint work with

O. Veksler

I. Ben Ayed

A. Delong

Y. Boykov

2

Νqp

xxp

pp qpxfE

,][1(x)

Example of Simple Binary Energy

Potts Modelf 1,0x

3

Binary Pairwise EnergyQuadratic Form

Submodular Energy global optimum with graphcut (Boros &

Hammer, 2002)

qp

qppqp

pp xxvxuE,

(x) 1,0x

pqpq vv 0

Potts Model

4

0, pqvpq

Non-Submodular Energy NP-hard

Binary Pairwise Energy Quadratic Form

const.)(,

qp

qppqp

pp xxvxuE x

Middlebury

Image credit: Carlos Hernandes

5

Standard Optimization Methods

General energy - NP-hardApproximate methods:

Global Linearization: QPBO, TRWS, SRMP (Kolmogorov et al. 2006, 2014)

Local Linearization: parallel ICM, IPFP (Leordeanu, 2009)

Message Passing: BP (Pearl 1989)

6

Related WorkGlobal Linearization

)(xE QPBO, TRWS, SRMP (Kolmogorov et al. 2006,

2014)

)(~min..

yyE

Cts

Linearize introducing large number ofvariables and constraintsSolve relaxed LPor its dual

Integrality Gap

*relaxedyRounding

*integer x

Related WorkIterative Local Linearization

7

parallel ICM (Leordeanu, 2009) large steps weak min

IPFP (Leordeanu, 2009) controls step size by relaxation Integrality Gap

)(xE

x

txEt(x)~

1tx

N}1,0{Bounded domain of discrete configurations

8

Local Submodular Approximation LSA

Local Submodular Approximation model

Non-linear

Two ways to control step size

Et(x)~

)(xE

x

tx1tx

N}1,0{Bounded domain of discrete configurations

9

Trust Region Local submodular approximation

Auxiliary Functions = Surrogate Functions = Upper Bounds = Majorize-Minimize Local submodular upper bound

Never leave the discrete domain

Iterative Optimization Framework

LSA-AUX

LSA-TR

10

Iterative Optimization FrameworkTrust Region:

Discrete High Order Energies Relaxed Quadratic Binary Energies Levenberg Marquardt

Auxiliary Functions=Surrogate Functions =Upper Bounds = Majorize-Minimize Discrete High Order Energies

Gorelick et al. 2012,2013

Ben Ayed et al. 2013

Olsson et al. 2008

Narasimhan & Bilmes 2005

Rother et al. 2006

Hartley & Zisserman 2004

11

Local Submodular ApproximationLSA

qp

qppqp

pp xxvxuE,

)(x

)()()( sup xxx EEE sub

+- x

tx)(xE

12

Local Submodular ApproximationLSA

)()()( sup xxx EEE sub

x

tx)(xE

13

Approximate around

Local Submodular ApproximationLSA

)()()( sup xxx EEE sub

)(xE tx

)(~ x(x) subt EE

Et(x)~

x

tx)(xE

14

Submodular functionLSA

Approximate around

Local Submodular ApproximationLSA

)()()( sup xxx EEE sub

)(xE tx

)()(~ xx(x) approxt

subt EEE

Linear Approximation

Et(x)~

x

tx)(xE

15

0,)(sup pqqppqpq

vxxvE x

Linear Approximation of the Supermodular Term

16

qppq xxv

Linear Approximation of the Supermodular Term

17

Linear Approximation of the Supermodular Term

0 xy

18

0

1

x

y

1

Linear Approximation of the Supermodular Term

1,0

0,0

1,1

0,1

0 xy

19

Linear Approximation of the Supermodular Term

0

1

1

x

y

1,0

0 xy

20

1

constyx 0

0

1

Linear Approximation of the Supermodular Term

x

y

1,0

0,0

1,1

Linear (Unary)

approximation

21

Linear Approximation of the Supermodular Term

0

1

1

constyvxu

x

y

22

LSA-TR:Trust Region Overview

)(xE

x

tx

)()()( sup xxx EEE sub

23

LSA-TR:Trust Region Overview

)(xE

x

tx

Et(x)~

Newton Step

)()(~ xx(x) approxt

subt EEE

1tx

LSA-TRTrust Region Sub-Problem

24

)(xE

x

tx

Et(x)~

Trust Region

)()(~ xx(x) approxt

subt EEE

Trust Region Sub-Problem

td ||||s.t. txx

24

NP-hard!Constrained Submodular Optimization

25

fixed in each iteration inversely related to trust region size adjusted based on quality of approximation

LSA-TR: Approximate TR sub-problem

||||)()(

tt

approxt

subt EEL

xxxx(x)

Unary TermsBoykov et al. 2006

Gorelick et al. 2013

t

Submodular

26

Experiments & Results

27

Experiments & Results:Deconvolution Binary De-convolution All pairwise terms supermodular

Original Img Convolved Convolved+Noise

?

Experiments & Results:Deconvolution

28

Noise:N(0,0.05)

SRMP

QPBOI

TRWS FTR-L

LBP

LSA-TR (0.3 sec.)E=21.13

LSA-AUX (0.04 sec)E=34.70

TRWS:5000 iter.E=65.07

LBP5000 iter.E=40.15

QPBO(0.1 sec.)

QPBO-I (0.2 sec.)E=66.44

IPFP(0.4 sec.)E=32.90

SRMP:5000 iter.E=39.06

Experiments & Results:Segmentation of Thin Structures

29

QPBO

QPBO-IE= -77.08

LBPE= -84.54

IPFPE= 163.25

Image

SRMP

Potts, v<0(submodula

r)

with edge repulsion, v>0(non-submodular)

TRWSE= -67.21

LSA-TRE= -175.05

LSA-AUXE= -120.03

SRMPE= -101.61

Repulsion = Reward different labels across high contrast edges

30

Experiments & Results:Inpainting dtf-chinesechar database

LSA-TRInput ImgGround Truth

Kappes et al., 2013

31

Experiments & Results:In-painting Chinese Characters

32

Curvature Regularization Efficient Squared Curvature model –

(Nieuwenhuis et al. 2014, poster on Friday)Potts Model Elastica

90-degree curvature

Heber et al. 2012

El-Zehiry&Grady, 2010

Our curvature Using LSA-TR

33

Summary Two novel discrete optimization methods

Simple, efficient, state-of-art results The code is available online -

http://vision.csd.uwo.ca/code/

Extensions: Find new applications▪ Convexity Shape Prior (in ECCV14)

Alternative optimization framework with LSA▪ Pseudo-Bounds (in ECCV14)

Please come by our poster