image-based shaving - computer...
TRANSCRIPT
![Page 1: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/1.jpg)
Image-based ShavingEurographics 2008
19 Apr 2008
Minh Hoai Nguyen, Jean-Francois Lalonde, Alexei A. Efros & Fernando de la Torre
Robotics Institute, Carnegie Mellon University
![Page 2: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/2.jpg)
Nguyen et al., 2008
A goal
synthesize
![Page 3: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/3.jpg)
Nguyen et al., 2008
Active Appearance Models (Cootes et al ECCV98)
• Issues:– Global model– No support for modification of local structure.
![Page 4: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/4.jpg)
Nguyen et al., 2008
Layered AAMs (Jones & Soatto ICCV05)
• Layers:– Defined manually– Require extensive set
of hand labeled landmarks.
• This work:– Attempt to extract
layers automatically.
![Page 5: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/5.jpg)
Nguyen et al., 2008
The idea
![Page 6: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/6.jpg)
Nguyen et al., 2008
The idea
Differences ???
Beard Layer
Model
+
… …
![Page 7: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/7.jpg)
Nguyen et al., 2008
Processing steps
68 landmarks
![Page 8: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/8.jpg)
Nguyen et al., 2008
A naïve approach• Reconstruct a bearded face by non-beard subspace
+…a1× +a2× +a3×≈
-a1× - a2× - a3× -…( )2
• This is equivalent to minimizing:
Reconstructed Image
![Page 9: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/9.jpg)
Nguyen et al., 2008
Naïve reconstruction
Problem: reconstruct what we don’t want to!
-a1× - a2× - a3× -…( )2
![Page 10: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/10.jpg)
Nguyen et al., 2008
Least square fitting
robust fitting
Iteratively Reweighted Least Square Fitting
![Page 11: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/11.jpg)
Nguyen et al., 2008
Robust Fitting
- a1× - a2× - a3× -…( )2
-…ρ( - a1× - a2× - a3× )Robustly reconstructed image
![Page 12: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/12.jpg)
Nguyen et al., 2008
Reconstruction Results
Original Naive reconstruction
Robust reconstruction
![Page 13: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/13.jpg)
Nguyen et al., 2008
There is a problem
• Beards are outliers of non-beard subspace.• But there are other outliers
Characteristic moles are also removed
![Page 14: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/14.jpg)
Nguyen et al., 2008
The idea
Differences ???
Beard Layer
Model
+
… …
![Page 15: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/15.jpg)
Nguyen et al., 2008
Subspace for beard layer
… …
Robust Fitting
Perform PCA on the residuals:
retaining 95% energy to get subspace for beard layers
![Page 16: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/16.jpg)
Nguyen et al., 2008
The first 6 principal components of B
super-imposed on the mean face
![Page 17: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/17.jpg)
Nguyen et al., 2008
Where we are
Differences ???
Beard Layer
Model
+
… …
B
![Page 18: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/18.jpg)
Nguyen et al., 2008
Factorizing beard layer
Given a face:Non-beard subspace
Beard-layersubspace
Non-beard layer Beard layerResidual (e.g. scar, mole)
![Page 19: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/19.jpg)
Nguyen et al., 2008
Using the beard layer
Non-beard layer Beard layerResidual (e.g. scar, mole)
Remove beard
Enhance beard
Reduce beard
![Page 20: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/20.jpg)
Nguyen et al., 2008
Changing contribution of the beard layer
Original Beard removed
Beard reduced
Beard enhanced
![Page 21: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/21.jpg)
Nguyen et al., 2008
Some results
![Page 22: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/22.jpg)
Nguyen et al., 2008
More results
![Page 23: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/23.jpg)
Nguyen et al., 2008
Failed cases
Too much beard!
Breakdown point of robust fitting is reached.
![Page 24: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/24.jpg)
Nguyen et al., 2008
Utilizing domain knowledge
• So far:– Generic technique
• For beard removal:– Additional cues
A pixel is likely to be beard pixel if most of its neighbors are.
![Page 25: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/25.jpg)
Nguyen et al., 2008
Beard Mask Segmentation• Formulate as graph labeling problem:
One vertex per pixel
Edges for neighboring
pixelsLabel the vertices
that minimize:
unary potential
binary potential
![Page 26: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/26.jpg)
Nguyen et al., 2008
Unary Potential
corresponds to ith pixel
Non-beard layer
Beard layerResidual (e.g. scar, mole)
Beard pixelNon-beard pixel
Unary potential: favors beard label if is big.
![Page 27: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/27.jpg)
Nguyen et al., 2008
Binary Potential
Binary potential: prefers same labels for neighboring pixels
![Page 28: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/28.jpg)
Nguyen et al., 2008
Optimization using graph-cuts(Boykov et al PAMI01)
Label the vertices
that minimize:
Exact global optimum solution can be found efficiently!
![Page 29: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/29.jpg)
Nguyen et al., 2008
Beard mask results
![Page 30: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/30.jpg)
Nguyen et al., 2008
Refinement
Original image
Beard removalUsing subspaces
Linear Feathering
Amount of blending
Beard mask
![Page 31: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/31.jpg)
Nguyen et al., 2008
Final results 1
![Page 32: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/32.jpg)
Nguyen et al., 2008
Final results 2
![Page 33: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/33.jpg)
Nguyen et al., 2008
Final results 3
![Page 34: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/34.jpg)
Nguyen et al., 2008
Final results 4
![Page 35: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/35.jpg)
Nguyen et al., 2008
Final results 5
![Page 36: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/36.jpg)
Nguyen et al., 2008
Beard removal, failure
Failure occurs at the robust fitting step
![Page 37: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/37.jpg)
Nguyen et al., 2008
Where we are
Differences ???
Beard Layer
Model
+
… …
![Page 38: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/38.jpg)
Nguyen et al., 2008
Beard Transfer
![Page 39: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/39.jpg)
Nguyen et al., 2008
So far, we talked about
Beards Beards Lots of Beards
But our method is generic!
![Page 40: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/40.jpg)
Nguyen et al., 2008
Glasses Removal
![Page 41: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/41.jpg)
Nguyen et al., 2008
… …
Glasses Removal
Differences ???
Glasses Layer
Model
![Page 42: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/42.jpg)
Nguyen et al., 2008
Preliminary results for glasses
![Page 43: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/43.jpg)
Nguyen et al., 2008
Preliminary results for glasses
![Page 44: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/44.jpg)
Nguyen et al., 2008
Glasses removal, failure
![Page 45: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/45.jpg)
Nguyen et al., 2008
Multi-PIE database• 1140 frontal, neutral faces
– 68 landmarks– From Multipie
• Female: 341– With glasses: 82 – Without glasses: 259
• Male: 799– With little or no facial hair: 480– With some facial hair: 319– With glasses: 340– Without glasses: 459
91 additional bearded faces
from the Internet.
![Page 46: Image-based Shaving - Computer Graphicsgraphics.cs.cmu.edu/projects/imageshaving/talk/nguyen_talk_08.pdfImage-based Shaving Eurographics 2008 19 Apr 2008 Minh Hoai Nguyen, Jean-Francois](https://reader034.vdocuments.mx/reader034/viewer/2022050303/5f6bde01f330f977db192462/html5/thumbnails/46.jpg)
Nguyen et al., 2008
References• Nguyen, M.H., Lalonde, J.F., Efros, A.A. & De la Torre, F. ‘Image-based
Shaving.’ Eurographics 08.• Cootes, T, Edwards, Taylor, G. ‘Active Appearance Models’, ECCV98.• Jones, E. & Soatto, S.(2005) ‘Layered Active Appearance Models.’
ICCV05.• Boykov, Y., Veksler, O. & Zabih, R. ‘Fast Approximate Energy
Minimization via Graph Cuts.’ PAMI01.• Gross, R., Matthews, I., Cohn, J., Kanade, T. & Baker, S. ‘The CMU
Multi-pose, Illumination, and Expression (Multi-PIE) Face Database.’ CMU TR-07-08.