Download - Model-Based Stereo with Occlusions
Model-Based Stereo with Occlusions
Fabiano Romeiro and Todd Zickler
Introduction
Varying illumination
Varying pose
Occlusions
Varying expressions
Introduction
Eigenfaces [Turk and Pentland, 1991]
[Belhumeur et al, 1997]Fisherfaces
Past Work: Image-based
For example:
3D Morphable Models (3DMMs)
[Blanz et al, 1999; Blanz et al, 2003; Blanz et al, 2005; Smet et al, 2006]
Introduction
2D AAMs
[Cootes et al, 1998; Baker et al, 2004; Mathews et al, 2004; Gross et al, 2006]
2D+3D AAMs
[Xiao et al, 2004]
Past Work: Model-based
Introduction
Pros
- Self-occlusion handled by model itself-- Allows direct modeling of illumination
Cons
- Difficult and expensive fitting process
Past Work: 3DMMs3D Morphable Models (3DMMs)
[Blanz et al, 1999; Blanz et al, 2003; Blanz et al, 2005; Smet et al, 2006]
Introduction
- Texture model not needed
Past Work: Stereo 3DMMs
Our work[Fransens et al, 2005]
- Stereo based cost
d(I 1; I 2)
-Stereo fitting with both shape and texture
→ Improved Accuracy
→ Robust to foreign-body occlusions
Outline
• 3DMM Background• Joint Shape and Texture Stereo Fitting• Handling Occlusions• Conclusions
Background
3DMMs
Vectorization of laser scans:
PCA performed:
F Si = [[X i1Y
i1Z i
1]T :::[X i
N Y iN Z i
N ]T ]T
- Basis for shape fSi gi=1;::;m
F Ti = [[R i1G
i1B
i1]
T :::[R iN Gi
N B iN ]T ]T
f Ti gi=1;::;m- Basis for texture[Blanz and Vetter, 1999]
Representation of face shape and texture:
3DMMs
Background
S = S0 +mX
i=1
®i Si
Prior probabilities on the coefficients:
P (~®) / exp(¡12
mX
i=1
(®i
¾i)2)
P (~̄) / exp(¡12
mX
i=1
(¯ i
°i)2) [Blanz and Vetter, 1999]
T = T0 +mX
i=1
¯ i Ti
Stereo Match
I 1(P1sk) ¼I 2(P2sk)
Shape (®)Pose (R;t)
sk
P1 P2
I 1 I 2
Texture Match
P1 P2
I m(k) ¼ ¹I (sk) = I 1(P1sk )+I 2(P2sk )2
I m(k) Texture (¯)Shape (®)
Lighting (K o®set, K amb, K dir)K =fR,G,Bg
skShape (®)Pose (R;t)
I 1 I 2
Joint Shape and Texture Stereo Fitting of 3DMMs
E =
P (Shape;Texture;P ose;L ightingjI 1; I 2) =
= P (Shape;P osejI 1; I 2)¢
X
kjvk 2V
jjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
| {z }Stereo Match
+mX
i=1
(®i
¾i)2
| {z }Shape Prior
+
+X
kjvk 2V
jjI m(k) ¡ ¹I (sk)jj2
¾2t
| {z }Texture Match
+mX
i=1
(¯ i
°i)2
| {z }TexturePrior
P (Texture;L ightingjI 1; I 2;Shape;P ose)
X
kjvk 2V
ha
µjjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
+jjI m(k) ¡ ¹I (sk)jj2
¾2t
¶X
kjvk 2V
jjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
| {z }Stereo Match
Binary occlusion map O : f1;::;N g ! f0;1gN
Robust Stereo Fitting of 3DMMs
E = +mX
i=1
(®i
¾i)2
| {z }Shape Prior
+
+X
kjvk 2V
jjI m(k) ¡ ¹I (sk)jj2
¾2t
| {z }TextureMatch
+mX
i=1
(¯ i
°i)2
| {z }Texture Prior
+jjI m(k) ¡ ¹I (sk)jj2
¾2t
+
P (Shape;Texture;P ose;L ighting;OjI 1; I 2) =
= P (Shape;P ose;OjI 1; I 2)¢P (Texture;L ightingjI 1; I 2;Shape;P ose;O)
Eo =X
kjvk 2V
ha
µjjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
+jjI m(k) ¡ ¹I (sk)jj2
¾2t
¶
+mX
i=1
(®i
¾i)2 +
mX
i=1
(¯ i
°i)2
Optimization Procedure
Initial fitE f- Fit Shape, Pose to minimize reprojection error
on selected feature points
- Rough initial estimates of Shape and Pose
Optimization procedure
Eo + ¸ ¢E f
Eo
4 experiments
Stereo and texture
Eo Stereo
Eo Robust Stereo and texture
Eo Robust Stereo
Eo =X
kjvk 2V
jjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
| {z }Stereo Match
+mX
i=1
(®i
¾i)2
| {z }ShapePrior
+
X
kjvk 2V
jjI m(k) ¡ ¹I (sk)jj2
¾2t
| {z }TextureMatch
+mX
i=1
(¯ i
°i)2
| {z }TexturePrior
Eo =X
kjvk 2V
ha
µjjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
¶
+mX
i=1
(®i
¾i)2
Eo =X
kjvk 2V
jjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
| {z }Stereo Match
+mX
i=1
(®i
¾i)2
| {z }Shape Prior
Results
480 recovered shape models (60 individuals, 8 poses)
K.U. Leuven Stereo face database
First 2 experiments: Stereo and Texture vs. Stereo
[Fransens et al, 2005]
ResultsQualitative Results
Stereo
Stereo and Texture
Stereo Matching CostStereo 280:77Stereo+Texture 340:17
Results
Stereo
Stereo+texture
Qualitative Results
Results
Stereo Stereo+Texturetr(S¡ 1
w Sb) 69:4101 104:0478det(S¡ 1
w Sb) 1:3418e¡ 11 2:9640e¡ 5
Quantitative Results
Results
Occluder classi¯cation Half Full near Full fartextureless (non-skincolor) Stereo Texture Texturetextured Stereo Texture Stereo+Texturetextureless (skincolor) X X X
Half-Occlusion Full Occlusion near Full-Occlusion far
Under Occlusions
Results
Occluder classi¯cation Half Full near Full fartextureless (non-skincolor) Stereo Texture Texturetextured Stereo Texture Stereo+Texturetextureless (skincolor) X X X
Under Occlusions
Input
Shape Estimate
Occlusion Map
Robust Stereo Robust S+T Robust Stereo Robust S+T
Conclusions
Robust stereo fitting of 3DMMs
- Uses both stereo constraint, texture information
- Increased accuracy of fit
- Ability to handle occlusions
Future Work
- More sophisticated stereo matching term
- Different feature spaces
- Break model into segments respecting occlusion boundaries