template-based deformable shape inferenceadb/tutorial_2009/2011.11.tutorial... · 2011. 11. 13. ·...
TRANSCRIPT
![Page 1: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/1.jpg)
Template-Based Deformable Shape Inference
Adrien Bartoli
ALCoV – ISIT
Université d’Auvergne
Clermont-Ferrand, France
Fourth Tutorial on Computer Vision in a Nonrigid World ICCV’11, Barcelona, Spain – octobre 6, 2011
Lourdes Agapito, Adrien Bartoli, Alessio Del Bue
![Page 2: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/2.jpg)
The Equations of Shape Inference
2
2
![Page 3: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/3.jpg)
Template-Based Shape Inference
Ψ ∈ 𝐶0 is not enough: shape and deformation priors are needed
3D deformation Ψ ∈ 𝐶0
Warp 𝜑 ∈ 𝐶0
![Page 4: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/4.jpg)
Template-Based Developable Shape Inference
Template
Image Warp
Applicable to paper and some types of fabric
Isometries preserve geodesics
![Page 5: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/5.jpg)
Maximal Depth Heuristic: Principle
[Perriollat et al, IJCV’11]
Upper bound on depth
Template
Image
?
![Page 6: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/6.jpg)
Maximal Depth Heuristic: Convex SOCP Solution
As far as possible from the camera
Points must be in front of the camera
Upper bound on reprojection error
[Brunet et al., ACCV’10 ; Salzmann et al., ECCV’08]
![Page 7: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/7.jpg)
Analytical Developable Solution
• Non convex PDE
• Iterative estimation
• Initialization from the Maximum Depth Heuristic
[Brunet et al., ACCV’10]
![Page 8: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/8.jpg)
Quantitative Comparison with Stereo 78 point matches; average surface discrepancy : 3.3 mm
Monocular
Stereo
Discrepancy
![Page 9: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/9.jpg)
Quantitative Comparison with Stereo 72 point matches; average surface discrepancy : 1.6 mm
Monocular
Stereo
Discrepancy
![Page 10: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/10.jpg)
3D Reconstruction Results
Videos (with self-occlusion): visualization grid and new view synthesis
![Page 11: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/11.jpg)
3D Reconstruction Results
Video: retargetting Video: augmentation
![Page 12: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/12.jpg)
3D Reconstruction Results
Video: keypoint matches, camera and 3D surface
![Page 13: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/13.jpg)
3D Reconstruction Results
Video: point matches, camera and 3D surface
![Page 14: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/14.jpg)
Template-Based Isometric and Shading Constraints
Replacing isometry by shading constraints handles elastic objects
[Moreno-Noguer et al, CVPR’09]
![Page 15: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/15.jpg)
Template-Based Isometric and Shading Constraints
Shading constraints disambiguates template-based isometric solutions
[Moreno-Noguer et al, ECCV’10]
![Page 16: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/16.jpg)
Application in Laparoscopy
![Page 17: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/17.jpg)
A Simple Case Study
Issue 1: non-flat → 3D template
Issue 2: elastic → extensible deformations
![Page 18: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/18.jpg)
Reconstructing the Template
‘Exploratory’ phase Keypoint tracking
3D template
Template uv-map
1 - Rigid SfM
2 – Conformal flattening
![Page 19: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/19.jpg)
Results of Isometric 3D Reconstruction
Template uv-map Image with deformations
Registration
![Page 20: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/20.jpg)
Results of Conformal 3D Reconstruction
Image with deformations Template uv-map
Registration
[Malti et al, MIUA’11]
![Page 21: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/21.jpg)
Results of Conformal 3D Reconstruction
![Page 22: Template-Based Deformable Shape Inferenceadb/tutorial_2009/2011.11.Tutorial... · 2011. 11. 13. · Template-Based Deformable Shape Inference Adrien Bartoli ALCoV – ISIT Université](https://reader036.vdocuments.mx/reader036/viewer/2022070112/6058e08daad9b8748e2f4ed0/html5/thumbnails/22.jpg)
Template-Based Deformable Shape Inference
Adrien Bartoli
ALCoV – ISIT
Université d’Auvergne
Clermont-Ferrand, France
Fourth Tutorial on Computer Vision in a Nonrigid World ICCV’11, Barcelona, Spain – octobre 6, 2011
Lourdes Agapito, Adrien Bartoli, Alessio Del Bue