Image Matting with the Matting LaplacianChen-Yu Tseng 曾禎宇Advisor: Sheng-Jyh Wang

Image Matting with the Matting Laplacian• Matting Laplacian• A. Levin, D. Lischinski, Y. Weiss. A Closed Form Solution to Natural

Image Matting. IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008.

• Spectral Matting• A. Levin, A. Rav-Acha, D. Lischinski. Spectral Matting. IEEE T. PAMI,

vol. 30, no. 10, pp. 1699-1712, Oct. 2008.• Matting for Multiple Image Layers• D. Singaraju, R. Vidal. Estimation of Alpha Mattes for Multiple

Image Layers. IEEE T. PAMI, vol. 33, no. 7, pp. 1295-1309, July 2011.

Center for Imaging Science, Department of Biomedical Engineering,The Johns Hopkins University

Image Matting

• Extracting a foreground object from an image along with an opacity estimate for each pixel covered by the object

Input Image Conventional Segmentation

Result

Spectral Matting Result

Image Compositing Equation

Alpha Mattes Image Layers

x

x

x

=

+

+

Input Image

L1

L2

L3

α1

α2

α3

Ki

Kiii LLLI

iii ...2211

Methodology• Supervised Matting

• Unsupervised Matting• Spectral Matting

Input Image Trimap (user’s constraint) Alpha Matte

Matting ComponentsInput Image

Local Models for Alpha Mattes

= x x+ 1 2L1L

𝐼 𝑖=𝛼𝑖𝐹 𝑖+(1−𝛼 𝑖)𝐵𝑖

𝛼 𝑖=𝐼 𝑖−𝐵𝑖

𝐹 𝑖−𝐵𝑖≈𝑎 𝐼𝑖+𝑏 ,∀ 𝑖∈𝑤 Assume a and b are constant

in a small window

LJ T )(

, , and are unknown ill-posed problem

Color Line Assumption

Color Distributions

Input

Omer and M. Werman. Color Lines: Image Specific Color Representation. CVPR, 2004.

Local Models for Alpha Mattes for Multiple Layers

Local Models1. Two color lines2. A color point and a color point3. Two color points and a single color line4. Four color points

R

G

B

Local Models1. Two color lines2. A color plane and a color point3. Two color points and a single color line4. Four color points

Color point

Color plane

Unknown color point

𝐼 𝑖=𝛼𝑖𝐹 𝑖+(1−𝛼 𝑖)𝐵𝑖

𝐹 𝑖

𝐵𝑖

𝐼 𝑖

Local Models1. Two color lines2. A color plane and a color point3. Two color points and a single color line4. Four color points

𝐼 𝑗=𝛼 𝑗1𝐹 𝑗

1+𝛼 𝑗2𝐹 𝑗

2 Color point

Color plane

𝐹 𝑗1

𝐹 𝑗2

𝐼 𝑖=

=++

Local Models for Alpha Mattes for Multiple Layers

Local Models1. Two color lines2. A color point and a color point3. Two color points and a single color line4. Four color points

R

G

B

The Matting Laplacian

LJ T )(

.11),(),|(

1

3

kwjikkj

kk

Tki

kij ww

ji IUIL

Overview of Spectral MattingInput Data

Matting Laplacian Construction

Spectral GraphAnalysis

Data ComponentGeneration

Output Components Laplacian Matrix

Input Image Local Adjacency

Components

Spectral Clustering

Scatter plot of a 2D data set

K-means Clustering Spectral Clustering

U. von Luxburg. A tutorial on spectral clustering. Technical report, Max Planck Institute for Biological Cybernetics, Germany, 2006.

Graph Construction

njiijwW ,...,1,)(

),( EVG

},...,,{ 21 nvvvV Vertex Set

Similarity Graph

Weighted Adjacency Matrix

Connected Groups

Similarity Graph

Similarity Graph• ε-neighborhood Graph• k-nearest neighbor Graphs• Fully connected graph

Graph Laplacian

njiijwW ,...,1,)( W: adjacency matrix

D: degree matrix

n

jiji wd

1

L: Laplacian matrix

WDL

𝒇 𝑇 𝐿 𝒇 =12 ∑𝑖 , 𝑗=1

𝑛

𝑤 𝑖𝑗 ( 𝑓 𝑖− 𝑓 𝑗 )2

For every vector

Example

2

3

1

4

0 1 1 0 0

1 0 1 0 0

1 1 0 0 0

0 0 0 0 1

0 0 0 1 0

W: adjacency matrix

5

L: Laplacian matrix

Similarity Graph

2 -1 -1 0 0

-1 2 -1 0 0

-1 -1 2 0 0

0 0 0 1 -1

0 0 0 -1 1

𝒇 𝑇 𝐿 𝒇 =12 ∑𝑖 , 𝑗=1

𝑛

𝑤 𝑖𝑗 ( 𝑓 𝑖− 𝑓 𝑗 )2

1

1

1

0

0

Cost Function

𝒇1

1

0

1

0

*2

3

1

4

5

2

3

1

4

5

Good Assignment Poor Assignment

Laplacian Eigenvectors

s.t. =1 𝐿 𝒇 =λ 𝒇1. L is symmetric and positive semi-definite.2. The smallest eigenvalue of L is 0, the corresponding

eigenvector is the constant one vector 1.3. L has n non-negative, real-valued eigenvalues 0= λ 1 ≦ λ 2 . . . ≦ ≦ λ n.

: Eigenvector: Eigenvalue

Smallest eigenvectors

Input Image

From Eigenvectors to Matting Components

Smallest eigenvectors Projection into eigs space kCTk mEE

....

K-means

..

kCmle

Overview of Spectral MattingInput Data

GraphConstruction

Spectral GraphAnalysis

Data ComponentGeneration

Output Components Laplacian Matrix

Input Image Local Adjacency

Components

Matting Laplacian

iiiii BFI )1(

LJ T )(

= x+1-α Bα Fx

Matting Laplacian

Color Distribution

𝐼 𝑖

𝐼 𝑗 𝜇𝑘

Matting LaplacianTypical affinity function Matting affinity function

24

Brief Summary

Input Image

Laplacian Matrix

Smallest Eigenvectors Matting Components

K-means Clustering

&Linear

Transformation

Supervised Matting

LJ T )(

)()( )( TTLE

otherwise0

011),( iiii

5.001

i

i

i

TrimapInput

Foreground

Background

Unknown

Cost function with user-specified constraint:

Supervised Matting

𝜶𝑇 𝐿𝜶=12 ∑𝑖 , 𝑗=1

𝑛

𝑤𝑖𝑗 (𝛼 𝑖−𝛼 𝑗 )2

LJ T )(

)()( )( TTLE

Estimation Alpha Matte for Two Layers

Estimation Alpha Matte for Multi-Layers

Karusch-Kuhn-Tucker (KKT) condition

Assumption Construction

The vector of 1s lies in the null space of L,

the solution automatically satisfies the constraint

Constrained Alpha Matte Estimation

Image matting for n≥2 image layers with positivity + summation constraints

Karusch-Kuhn-Tucker (KKT) conditions

For 0 < < 1(i,i)=0 and (i,i)=0

Conventional Approaches Directly Clipping

Refinement is neglected in conventional approaches

Equivalent toIntroducing Lagrange Multipliers

Experiments

(a) Algorithm 1. (b) Algorithm 2. (c) Spectral Matting. (d) SM-enhance.

(a) Algorithm 1. (b) Algorithm 2. (c) Spectral Matting. (d) SM-enhance.

Summary

• Image Matting with the Matting Laplacian• Construction of the Matting Laplacian• Image Compositing Model• Local-Color Affine Model

• Supervised Closed-form Matting• Two-layer• Multiple-layer

• Spectral Matting• Extended Applications

Depth Estimation

)()( )( TTLEInput Image

Estimated Depth

Refined Result

Compositing ImageLikelihood Prior

L

Confidence Map

Prior

MAP

)()( )( TTLE

Input Image Transmission Prior

Refined TransmissionOutput Image

Graph Laplacian and Non-linear Filters

GlobalOptima

LocalOptima

Global Optima Local Optima

Gaussian-based Bilateral Filter

Matting-Laplacian-based Guided Filter (K. He, ECCV 2010)