3d face modeling michaël de smet. topics to discuss 3d morphable models 3d face reconstruction face...
TRANSCRIPT
![Page 1: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/1.jpg)
3D Face Modeling
Michaël De Smet
![Page 2: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/2.jpg)
Topics to Discuss
3D Morphable Models 3D face reconstruction Face recognition Lip synchronization
![Page 3: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/3.jpg)
Topics to Discuss
3D Morphable Models 3D face reconstruction Face recognition Lip synchronization
![Page 4: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/4.jpg)
3D Morphable Models
Statistical model of shape and texture Derived from laser scans
USF DARPA HumanID 3D Face Database Processing
Hole filling Surface smoothing Albedo estimation Dense correspondence
![Page 5: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/5.jpg)
3D Morphable Models
-2 +2
…
-2 +2
…
![Page 6: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/6.jpg)
Topics to Discuss
3D Morphable Models 3D face reconstruction Face recognition Lip synchronization
![Page 7: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/7.jpg)
3D Face Reconstruction
Fitting the 3DMM to one or more images of the same face
•Scale•Rotation•Translation•Illumination
•Shape•Texture
?
?
Optimization problem with > 100 parameters
![Page 8: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/8.jpg)
3D Face Reconstruction
Feature points Feature alignment Fitting result
![Page 9: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/9.jpg)
3D Face Reconstruction
![Page 10: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/10.jpg)
Dealing with Occlusions
![Page 11: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/11.jpg)
Dealing with Occlusions
Without occlusionhandling
With occlusionhandling
![Page 12: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/12.jpg)
3D Face Reconstruction
![Page 13: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/13.jpg)
3D Face Reconstruction
![Page 14: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/14.jpg)
3D Face Reconstruction
![Page 15: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/15.jpg)
3D Face Reconstruction
![Page 16: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/16.jpg)
3D Face Reconstruction
![Page 17: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/17.jpg)
3D Face Reconstruction
![Page 18: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/18.jpg)
Topics to Discuss
3D Morphable Models 3D face reconstruction Face recognition Lip synchronization
![Page 19: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/19.jpg)
Face RecognitionFit the 3DMM to an image of an unknown face
•Scale•Rotation•Translation•Illumination
•Shape•Texture
Compare to database Recognition
?
?
![Page 20: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/20.jpg)
Face Recognition
In controlled settings, almost perfect recognition is possible
Pose 1100.0%
Pose 2100.0%
Pose 3N/A
Pose 4100.0%
Pose 595.7%
Training view
![Page 21: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/21.jpg)
Face Recognition
Uncontrolled environments are challenging Face orientation unknown Difficult illumination Facial expressions Occlusions Low resolution Motion blur …
![Page 22: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/22.jpg)
Face Recognition
European parliament video: 21 persons, 86% correct
![Page 23: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/23.jpg)
Face Recognition
VRT news broadcasts: 12 persons
![Page 24: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/24.jpg)
Face Recognition
VRT news broadcasts: 12 persons
82.3% correctrecognition
![Page 25: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/25.jpg)
Topics to Discuss
3D Morphable Models 3D face reconstruction Face recognition Lip synchronization
![Page 26: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/26.jpg)
Lip Synchronization Speech driven animation Texture based, i.e. shape is fixed Strategy:
Extract 3D model of speaker’s face Track rigid motion of the face in video Extract texture for each frame Compute PCA model of texture Train ANN to link phonemes and PCA
coefficients (visemes)
![Page 27: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/27.jpg)
System Overview
AutomaticAutomaticPhonePhone
RecognitionRecognition
NeuralNeuralNetworkNetwork
FaceFaceSynthesisSynthesis
SpeechSpeechfeature vectorsfeature vectors
FacialFacialfeature vectorsfeature vectors
![Page 28: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/28.jpg)
Training Setup
AutomaticAutomaticPhonePhone
RecognitionRecognition
NeuralNeuralNetworkNetworkTrainingTraining
FaceFaceAnalysisAnalysis
SpeechSpeechfeature vectorsfeature vectors
FacialFacialfeature vectorsfeature vectors
![Page 29: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/29.jpg)
Video Processing
3D face model acquisition Rigid motion tracking Normalized texture extraction Texture feature extraction (PCA)
![Page 30: 3D Face Modeling Michaël De Smet. Topics to Discuss 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization](https://reader036.vdocuments.mx/reader036/viewer/2022062313/56649cf85503460f949c8ff3/html5/thumbnails/30.jpg)
Conclusion
3DMMs are a very powerful tool for face modeling
Many applications in computer vision and computer graphics