Contour-Based Joint Clustering of Multiple Segmentations

Daniel Glasner *1 Shiv N. Vitaladevuni *2

Ronen Basri 1

* equal contribution authors

1 2

1

2 3

1

2 3

1

2 3

1

2 3

Joint-Clustering of Image SegmentsObjective: 1. Similar shape across frames 2. Coherent regions within frames

Segments /Super-pixels

Ourclusteringresult

Inputframe

Object contours can be matchedOversegmentation artifacts can not

• Large deformation histograms are similar

• Single object

• Inter image similarity:• Overlap• Color

Related Work This WorkCo-segmentation

[Rother et al.06, Bagon et al.08, Hochbaum et al.09, Joulin et al.10]

Co-clustering of image segments[Vitaladevuni and Basri

2010]• Small deformation

shapes are similar• Full image segmentation

# objects unknown• Inter image similarity:

• Shape• Intra Image similarity:

• Color / optical flow etc.

Different lighting conditions

Some ApplicationsVideo segmentation

EM slices[Vitaladevuni &Basri CVPR10]

( )

+ Coherency + Coherency( )

F(union) = Shape-sim( , )

Shape similarityacross frames ×Coherencywithin frame ✓

✓

Searching for a Good ClusterConvex functional of unions of segments 1. Bounding contours match - across frames2. Coherent regions - within frameExterior bounding contours match

Incorporating Shape

Input frames Shape-basedjoint clustering

IntersectionShape-basedsimilarity

Descriptor (segment) = bounding contour

A Novel Contour Descriptor

3

dimension =

# contour samples

in image 1

Image 1

0ei(θ+)

0eiα

ei

eiθ

3

3

Descriptor (union) = its bounding contour

Additive Descriptor & Contour Cancellation

ei

eiθ

0

ei(θ+)

union

ei

0+ =

dimension =

#contour samples

Comparing Shapes Across Images

B2B1

0

0eiθ

# contour elements image 2

# segments image 2

# contour elements image 1

# segments image 1

Image 2Image 1

Comparing Shapes Across Images

B2x2B1

= =

Contour descriptorin image 2

Contour descriptorin image 1

x1

Image 2Image 1

Binary indicator of segmentsin image 1

Binary indicator of segmentsin image 2

B2x2B1

= =x1

Shape-sim( , )

= ?

Correspondence

Matrix

W1,2

B1Hx1

T

?

# contour elements image1X

# contour elements image2

Comparing Shapes Across Images

Comparing Shapes Across Images

B2 x2

W1,2

x1T B1

H

Contour descriptor of shape in image 1

Contour descriptor of shape in image 2

Q1,2 =

# segments image1X

# segments image2

# contour elements image1X

# contour elements image2

1. Similar shape across frames2. Coherent regions within frames

2 wkl cos(k,l external contour

k l )=

= Shape-sim( , )

xT

x

Q1,2

Q1,2H

xTQx =

For arbitrary selection of segments

x x1

x2

Q1,1

Q2,2

( )

+ Coherency + Coherency( )

Optimization

maxX x 1 ... x c ),c

x jTQx j

j 1

c

s.t. x ij {0,1}

x ij2

j =1

c

1 i

Vitaladevuni and Basri CVPR 2010

NP-hard [Garey and Johnson 1979]

Convex Relaxation

• Efficient linear programming relaxation[Vitaladevuni & Basri 2010, Charikar et al. 2003]

• Q is hermitian objective is real

Video Dataset for Occlusion/Object Boundary Detection

A. Stein and M. Hebert. Occlusion boundaries from motion: low-level detection and mid-level reasoning. IJCV, 82(3), 2009. 2http://www.cs.cmu.edu/~stein/occlusion_data/

referenceframe

ground truth

Qualitative ResultsReference frame:

Our result:

Qualitative ResultsReference frame:

Our result:

Summary

• Joint segmentation of closely related images• Additive contour representation

– Ignores internal contours of unions of segments• Efficient convex optimization to find subsets

of segments with a similar shape• Combine inter and intra image similarity

Thank you!