3d morphable face modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. ·...
TRANSCRIPT
![Page 1: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/1.jpg)
3D Morphable Face ModelsBernhard Egger
IPAM Workshop I: Geometric Processing
![Page 2: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/2.jpg)
2
Photo by Pete Souza / The White House
![Page 3: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/3.jpg)
![Page 4: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/4.jpg)
Graphics and Vision Research Group in Basel
4
gravis.unibas.ch
Thomas VetterSandro SchönbornAndreas ForsterAdam KortylewskyClemens BlumerAndreas SchneiderGhazi BouabeneMarcel LüthiThomas Gerig
![Page 5: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/5.jpg)
Probabilistic 3D Morphable Model
5
Probabilistic PCA
Gerig et al (2018). Morphable Face Models - An Open Framework, FGPaysan et al (2009). A 3D Face Model for Pose and Illumination Invariant Face Recognition, AVSSBlanz and Vetter (1999). A Morphable Model for the Synthesis of 3D Faces, SIGGRAPH
![Page 6: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/6.jpg)
Gaussian Process Registration
Marcel Lüthi, Christoph Jud, Thomas Gerig, Thomas Vetter ( 2017) Gaussian Process Morphable ModelsGerig et al (2018). Morphable Face Models - An Open Framework, FG
![Page 7: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/7.jpg)
Image Formation Process
7
![Page 8: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/8.jpg)
Goal: 3D Reconstruction from a Single Image
8
Source: KEYSTONE
![Page 9: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/9.jpg)
Approach: Analysis by Synthesis
9
≈Source: KEYSTONE
update
compare
target synthesized
![Page 10: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/10.jpg)
Analysis by Synthesis
10
Source: KEYSTONE
changing camera and illuminationparametrized representation
![Page 11: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/11.jpg)
Classical Face Model Adaptation
11
Schönborn, Egger, Morel-Forster, Vetter (2017). Markov Chain Monte Carlo for Automated Face Image Analysis IJCV
• Probabilistic Framework
𝑃 𝜃 ሚ𝐼) ∝ 𝑙(𝜃; ሚ𝐼) 𝑃(𝜃)
• Marko chain Monte Carlo-Sampling
= 𝑅𝑝(𝛼1 ∙ +𝛼2 ∙ +⋯+ 𝛼𝑛∙ )
Target ሚ𝐼
Fit 𝐼(𝜃)
![Page 12: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/12.jpg)
Color Illumination Ambiguity
12
![Page 13: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/13.jpg)
13
![Page 14: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/14.jpg)
Results: Qualitative
14
Source: LFW Database
Schönborn, Egger, Morel-Forster, Vetter (2017). Markov Chain Monte Carlo for Automated Face Image Analysis, IJCV
![Page 15: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/15.jpg)
Classical Morphable Model
15
Schönborn, Egger, Morel-Forster, Vetter (2017). Markov Chain Monte Carlo for Automated Face Image Analysis IJCV
𝑙 𝜃; ሚ𝐼 = ෑ
𝑖∈𝐹
𝑙𝑓𝑎𝑐𝑒(𝜃; ෩𝐼𝑖) ෑ
𝑖`∈𝐵
𝑏(෩𝐼𝑖`)
![Page 16: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/16.jpg)
Background model
16
Background modeling for generative image models (2015) CVIU,S Schönborn, B Egger, A Forster, T Vetter
• All A-by-S methods assume a background model
• Shrinking and the hacks
• Simple generic model is enough
RGB intensities
freq
uen
cy
![Page 17: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/17.jpg)
The good
Source: KEYSTONE
Schönborn, Egger, Morel-Forster, Vetter (2017). Markov Chain Monte Carlo for Automated Face Image Analysis IJCV
![Page 18: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/18.jpg)
Face Image Analysis under Occlusion
18
Source: AFLW Database Source: AR Face Database
![Page 19: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/19.jpg)
End-to-end learned Fitter
19
Tewari, Zollhöfer, Kim, Garrido, Bernard, Pérez, Theobalt, 2016. MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction arXiv
![Page 20: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/20.jpg)
The bad
20
![Page 21: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/21.jpg)
Semantic Morphable Model
21
![Page 22: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/22.jpg)
𝑙 𝜃; ሚ𝐼, 𝑧 =ෑ
𝑖
ෑ
𝑘
𝑙𝑘 𝜃; ෩𝐼𝑖𝑧𝑖𝑘
Semantic Model Adaptation
22
Egger, Schönborn, Schneider, Kortylewski, Morel-Forster, Blumer, Vetter (2018). Occlusion-aware Morphable Models and an Illumination Prior for Face Image Analysis, IJCV
with 𝑧𝑖𝑘 ∈ {0,1}and σ𝑘 𝑧𝑖𝑘 = 1 ∀ⅈ
![Page 23: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/23.jpg)
Semantic Model: Segmentation
23
𝑃 𝑧| ሚ𝐼, 𝜃 ∝ෑ
𝑖
ෑ
𝑘
𝑙𝑘 𝜃; ෩𝐼𝑖𝑧𝑖𝑘𝑃 𝑧𝑖𝑘|𝜃 𝑃 𝑐 ෑ
𝑗∈𝑛(𝑖)
𝑃 𝑧𝑖𝑘, 𝑧𝑗𝑘
Prior c
Label z𝑖
Pixels ෩𝐼𝑖
likelihoods prior smoothness
Egger, Schönborn, Schneider, Kortylewski, Morel-Forster, Blumer, Vetter (2018). Occlusion-aware Morphable Models and an Illumination Prior for Face Image Analysis, IJCV
![Page 24: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/24.jpg)
𝑙𝑓𝑎𝑐𝑒 𝜃; ෩𝐼𝑖 =
1
𝑁exp −
1
2𝜎2෩𝐼𝑖 − 𝐼𝑖 𝜃 if ⅈ ∈ 𝐹
1
𝛿ℎ𝑓𝑎𝑐𝑒 ෩𝐼𝑖 , 𝜃 if ⅈ ∈ ℬ
Generative Face Model
24
𝐼 𝜃
ሚ𝐼
Egger, Schönborn, Schneider, Kortylewski, Morel-Forster, Blumer, Vetter (2018). Occlusion-aware Morphable Models and an Illumination Prior for Face Image Analysis, IJCV
![Page 25: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/25.jpg)
Inference
25
Egger, Schönborn, Schneider, Kortylewski, Morel-Forster, Blumer, Vetter (2017). Occlusion-aware Morphable Models and an Illumination Prior for Face Image Analysis, IJCV (invited/under revision)
𝑙 𝜃; ሚ𝐼, 𝑧 =ෑ
𝑖
ෑ
𝑘
𝑙𝑘 𝜃; ෩𝐼𝑖𝑧𝑖𝑘
𝑃 𝑧| ሚ𝐼, 𝜃 ∝ෑ
𝑖
ෑ
𝑘
𝑙𝑘 𝜃; ෩𝐼𝑖𝑧𝑖𝑘𝑃 𝑧𝑖𝑘|𝜃 𝑃 𝑐 ෑ
𝑗∈𝑛(𝑖)
𝑃 𝑧𝑖𝑘, 𝑧𝑗𝑘
![Page 26: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/26.jpg)
Inference
26
Egger, Schönborn, Schneider, Kortylewski, Morel-Forster, Blumer, Vetter (2018). Occlusion-aware Morphable Models and an Illumination Prior for Face Image Analysis, IJCV
![Page 27: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/27.jpg)
Results
27
Targets: LFW Face Database
Target ሚ𝐼 Fit 𝐼(𝜃)Segmentation 𝑧
Target ሚ𝐼 Fit 𝐼(𝜃)Segmentation 𝑧
Egger, Schönborn, Schneider, Kortylewski, Morel-Forster, Blumer, Vetter (2018). Occlusion-aware Morphable Models and an Illumination Prior for Face Image Analysis, IJCV
![Page 28: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/28.jpg)
Application: Image Manipulation
28
Source: LFW Database
Egger, Schönborn, Blumer, Vetter (2017). Probabilistic Morphable Models in Statistical Shape and Deformation Analysis, Elsevier
![Page 29: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/29.jpg)
How to initialize?
29
Egger, Schönborn, Schneider, Kortylewski, Morel-Forster, Blumer, Vetter (2018). Occlusion-aware Morphable Models and an Illumination Prior for Face Image Analysis, IJCV
![Page 30: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/30.jpg)
Illumination Dominates Appearance
30
new illumination conditions
new pose new color new shape
Original
0.20 0.30 0.23
0.18 0.12 0.18
![Page 31: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/31.jpg)
Robust Illumination Estimation
31
Egger, Schönborn, Schneider, Kortylewski, Morel-Forster, Blumer, Vetter (2018). Occlusion-aware Morphable Models and an Illumination Prior for Face Image Analysis, IJCV
![Page 32: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/32.jpg)
Initialization: Robust Illumination Estimation
32
Init 𝜃𝑙𝑖𝑔ℎ𝑡
Estimate illumination
Init 𝜃𝑐𝑎𝑚𝑒𝑟𝑎
Consensus set: Init 𝑧Sampled points
Egger, Schönborn, Schneider, Kortylewski, Morel-Forster, Blumer, Vetter (2018). Occlusion-aware Morphable Models and an Illumination Prior for Face Image Analysis, IJCV
![Page 33: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/33.jpg)
Robust Illumination Estimation «in the Wild»
33
Egger, Schönborn, Schneider, Kortylewski, Morel-Forster, Blumer, Vetter (2018). Occlusion-aware Morphable Models and an Illumination Prior for Face Image Analysis, IJCV
Source: AFLW Database
![Page 34: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/34.jpg)
Illumination Prior from «in the Wild» data
34
Egger, Schönborn, Schneider, Kortylewski, Morel-Forster, Blumer, Vetter (2018). Occlusion-aware Morphable Models and an Illumination Prior for Face Image Analysis, IJCV
![Page 35: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/35.jpg)
Joint Color, Shape and Attributes Model
35
23.8 years, 182 cm, 77 kg, male
21.7 years, 165 cm, 54 kg, female
Egger*, Kaufmann*, Schönborn, Roth and Vetter (2016). (* equal contribution)Copula Eigenfaces with Attributes, CCIS
age, height, weight, sex
![Page 36: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/36.jpg)
Unnatural artifacts
36
![Page 37: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/37.jpg)
Marginal Distributions: Color Model
37
![Page 38: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/38.jpg)
0
Copula Morphable Model
39
sexageweightheight
𝑥1𝑦1𝑧1…𝑥𝑛𝑦𝑛𝑧𝑛𝑟1𝑔1𝑏1…𝑟𝑛𝑔𝑛𝑏𝑛𝑠𝑒𝑥𝑎𝑔𝑒
𝑤𝑒𝑖𝑔ℎ𝑡ℎ𝑒𝑖𝑔ℎ𝑡
…
Egger*, Kaufmann*, Schönborn, Roth and Vetter (2016). (* equal contribution)Copula Eigenfaces - Semiparametric Principal Component Analysis for Facial Appearance Modeling, GRAPP
![Page 39: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/39.jpg)
Copula Component Analysis benefits
• Dependency and marginals modeled seperately• Additional flexibility
• Non-Gaussian marginals can be modeled
• Scale-invariant• Combined shape, color and attribute models
• Robust to outliers
• Pre- and postprocessing
40
sexageweightheight
![Page 40: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/40.jpg)
Including Attributes
41
21.4 years, 172 cm, 62 kg 24.7 years, 175 cm 68 kg
23.8 years, 182 cm, 77 kg 21.7 years, 165 cm, 54 kg
+2𝜎−2𝜎
1st PC
2nd PC
![Page 41: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/41.jpg)
Random Samples
42
Egger*, Kaufmann*, Schönborn, Roth and Vetter (2016). (* equal contribution)Copula Eigenfaces with Attributes, CCIS
![Page 42: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/42.jpg)
Copula Posterior model
43
male
female
![Page 43: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/43.jpg)
Model / Feature PCA coefficients +Random forests
CopulaMorphableModel
Copula MorphableModel(only caucasian)
Prediction performance 76,2 % 82,5% 88,7%
Additional training data
Face Image Analysis: Sex prediction
44
male28 years74 kg175cm
![Page 44: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/44.jpg)
Semantic Morphable Model: Overview
45
male29 years70 kg175 cm
Input
Robust Illumination Estimation
3D ReconstructionAttribute-based-Description
Semantic Segmentation
age, height, weight, sex
Copula Morphable Model
Semantic Model Adaptation
![Page 45: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/45.jpg)
Semantic Image Description: Results
46
male23 years163 cm55 kg
Targets: LFW Face Database
female48 years172 cm63 kg
male47 years176 cm75 kg
female22 years170 cm65 kg
male48 years175 cm68 kg
female25 years180 cm67 kg
male29 years186 cm115 kg
male48 years190 cm84 kg
Target ሚ𝐼 Fit 𝐼(𝜃)Segmentation 𝑧 Fit A(𝜃) Target ሚ𝐼 Fit 𝐼(𝜃)Segmentation 𝑧 Fit A(𝜃)
![Page 46: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/46.jpg)
47
Photo by Pete Souza / The White House
female23 years173 cm68 kg
male61 years181 cm54 kg
male43 years176 cm60 kg
male30 years183 cm115 kg
female20 years165 cm50 kg
male26 years179 cm80 kg
male43 years173 cm75 kg
male56 years180 cm87 kg
male28 years180 cm89 kg
male48 years176 cm79 kg
male23 years168 cm58 kg
male28 years179 cm72 kg
male33 years179 cm77 kg
![Page 47: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/47.jpg)
Application: Image Manipulation
48Schönborn, Forster, Egger, Schneider (2014). Robust Image Analysis by fitting a 3DMM for Portrait Manipulation, Shape SymposiumWalker, Vetter (2009). Portraits made to measure: Manipulating social judgments about individuals with a statistical face model, JoV
https://gravis.dmi.unibas.ch/PMM/demo/face-manipulation/
![Page 48: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/48.jpg)
Efficient inverse graphics in biological face processing
49Ilker Yildirim, Mario Belledonne, Winrich Freiwald, Joshua Tenenbaum (2018) Efficient inverse graphics in biological face processing
![Page 49: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/49.jpg)
Study Limitations with Controlled Training Data
• Full control over training data
• Synthetic but similar challanges
• Generalization
• Dataset Bias
• Bootstrapping• Less real training data
• Data generator available
50Kortylewski, Egger, Schneider, Gerig, Morel-Forster and Vetter (2018), Empirically Analyzing the Effect of Dataset Biases on Deep Face Recognition Systems, CVPR Workshop
![Page 50: 3D Morphable Face Modelshelper.ipam.ucla.edu/publications/glws1/glws1_15712.pdf · 2019. 4. 8. · Probabilistic 3D Morphable Model 5 Probabilistic PCA Gerig et al (2018). Morphable](https://reader036.vdocuments.mx/reader036/viewer/2022081407/604c75b96945d311b470232a/html5/thumbnails/50.jpg)
Questions