spectral matting

Spectral Matting

A. Levin D. Lischinski and Y. Weiss. A Closed Form Solution to Natural Image Matting. IEEE Conf. on Computer Vision and

Pattern Recognition (CVPR), June 2006, New York

A. Levin, A. Rav-Acha, D. Lischinski. Spectral Matting. Best paper award runner up. IEEE Conf. on Computer Vision and Pattern

Recognition (CVPR), Minneapolis, June 2007

A. Levin1,2, A. Rav-Acha1, D. Lischinski1. Spectral Matting. IEEE Trans. Pattern Analysis and Machine Intelligence, Oct 2008.

1School of CS&Eng The Hebrew University2CSAIL MIT

1

Hard

segmentation compositing

Matte compositing

Source image

Hard segmentation and matting

2

Previous approaches to segmentation and matting

Unsupervised

Input Hard output Matte output

Spectral segmentation:Spectral segmentation: Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Zelnik and Perona 05 Tolliver and Miller 06

3

Previous approaches to segmentation and matting

Unsupervised

Input Hard output Matte output

Supervised

0

1

July and Boykov01 Rother et al 04 Li et al 04

4

Previous approaches to segmentation and matting

Unsupervised

Input Hard output Matte output

Supervised

0

1

Trimap interfaceTrimap interface: Bayesian Matting (Chuang et al 01) Poisson Matting (Sun et al 04) Random Walk (Grady et al 05)Scribbles interface:Scribbles interface: Wang&Cohen 05 Levin et al 06 Easy matting (Guan et al 06)

?

5

User guided interface

TrimapScribbles Matting result

6

Generalized compositing equation

iiiii BFI )1( 2 layers compositing

= x x+ 1 2L1L

7

Generalized compositing equation

iiiii BFI )1( 2 layers compositing

= x x+ 1 2L1L

Ki

Kiii LLLI

iii ...2211

K layers compositing

= x x+

+ x x+3 4 4L3L

1 2 2L1L

Matting components

8

Generalized compositing equation

1...21 K

iii

“Sparse” layers- 0/1 for most image pixels

Matting components:

Ki

Kiii LLLI

iii ...2211

K layers compositing

= x x+

+ x x+

10 ki

1

3 4

2 2L

4L3L

1L

9

Automatically computed matting components

Input

1 2 3 4

8765

Unsupervised matting

10

Building foreground object by simple components addition

=+ +

11

Spectral segmentation

22/

),(ji CC

ejiW

WDL

j

jiWiiD ),(),(

Spectral segmentation: Analyzing smallest eigenvectors of a graph Laplacian L

E.g.: Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Maila and shi 01 Zelnik and Perona 05 Tolliver and Miller 0612

Problem Formulation

= x x+ 1 2L1L

Assume a and b are constant in a small window

13

Derivation of the cost function

14

Derivation

LJ T )(

15

The matting Laplacian

LJ T )(

• semidefinite sparse matrix

• local function of the image:),( jiL

L

16

The matting affinity

17

The matting affinity

Color Distribution

Input

18

Matting and spectral segmentation

Typical affinity function Matting affinity function

19

Eigenvectors of input image

Input

Smallest eigenvectors 20

Spectral segmentationFully separated classes: class indicator vectors belong to Laplacian nullspace

General case: class indicators approximated as linear combinations of smallest eigenvectors

Null

Binary indicating

vectors

Laplacian matrix

21

Spectral segmentation

Fully separated classes: class indicator vectors belong to Laplacian nullspace

General case: class indicators approximated as linear combinations of smallest eigenvectors

Smallest eigenvectors- class indicators only up to linear transformation

33

RZero eigenvectors

Binary indicating

vectors

Laplacian matrix

Smallest eigenvecto

rs

Linear transformati

on

22

From eigenvectors to matting components

linear transformat

ion

23

From eigenvectors to matting components

Sparsity of matting components

Minimize

24

From eigenvectors to matting components

Minimize

Newton’s method

with initialization

25

From eigenvectors to matting components

Smallest eigenvectors

Projection into eigs space kCTk mEE

....

K-means

..

kCmle

1) Initialization: projection of hard segments

2) Non linear optimization for sparse components26

Extracted Matting Components

27

Brief Summary

LJ T )(

Construct Matting Laplacian

Smallest eigenvectors

Linear Transformation

Matting components

28

Grouping Components

=+ +

29

Grouping Components

Unsupervised matting User-guided matting

Complete foreground matte

=+ +

30

Unsupervised matting

LJ T )(

Matting cost function

Hypothesis:Generate indicating vector b

31

Unsupervised matting results

32

User-guided matting Graph cut method

Energy function

Unary term Pairwise termConstrained components

33

Components with the scribble interface

Components (our

approach)

Levin et al cvpr06

Wang&Cohen 05

Random Walk

Poisson 34

Components with the scribble interface

Components (our

approach)

Levin et al cvpr06

Wang&Cohen 05

Random Walk

Poisson 35

Direct component picking interface

=+ +

Building foreground object by simple components addition

36

Results

37

Quantitative evaluation

38

Spectral matting versus obtaining trimaps from a hard segmentation

39

Limitations Number of eigenvectors

Ground truth matte Matte from 70 eigenvectors

Matte from 400 eigenvectors40

Limitations Number of matting components

41

Conclusion Derived analogy between hard spectral

segmentation to image matting Automatically extract matting components

from eigenvectors Automate matte extraction process and

suggest new modes of user interaction

42