using strong shape priors for multiview reconstruction
DESCRIPTION
Using Strong Shape Priors for Multiview Reconstruction. Yunda SunPushmeet Kohli Mathieu BrayPhilip HS Torr. Department of Computing Oxford Brookes University. Objective. Images Silhouettes. Parametric Model. +. Pose Estimate Reconstruction. [Images Courtesy: M. Black, L. Sigal]. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/1.jpg)
Using Strong Shape Priors for Multiview Reconstruction
Yunda Sun Pushmeet Kohli
Mathieu Bray Philip HS Torr
Department of Computing
Oxford Brookes University
![Page 2: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/2.jpg)
Objective
+
[Images Courtesy: M. Black, L. Sigal]
Parametric Model
Images
Silhouettes
Pose
Estimate
Reconstruction
![Page 3: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/3.jpg)
Outline
Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results
![Page 4: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/4.jpg)
Outline
Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results
![Page 5: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/5.jpg)
Multiview Reconstruction
Need for Shape Priors
![Page 6: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/6.jpg)
Multiview Reconstruction No Priors
• Silhouette Intersection• Space Carving
Weak Priors• Surface smoothness
– Snow et al. CVPR ’00
• Photo consistency and smoothness
– Kolmogorov and Zabih [ECCV ’02]
– Vogiatzis et al. [CVPR ’05] [Image Courtesy: Vogiatzis et al.]
![Page 7: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/7.jpg)
Outline
Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results
![Page 8: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/8.jpg)
Shape-Priors for Segmentation
OBJ-CUT [Kumar et al., CVPR ’05]• Integrate Shape Priors in a MRF
POSE-CUT [Bray et al., ECCV ’06] • Efficient Inference of Model Parameters
![Page 9: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/9.jpg)
Parametric Object Models as Strong Priors
Layered Pictorial Structures
Articulated Models
Deformable Models
![Page 10: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/10.jpg)
Outline
Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation and Reconstruction Results
![Page 11: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/11.jpg)
Object-Specific MRF
![Page 12: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/12.jpg)
Object-Specific MRF
Energy Function
Shape Prior
Unary Likelihood
Smoothness Prior
x : Voxel label θ : Model Shape
![Page 13: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/13.jpg)
Object-Specific MRF
Shape Prior
x : Voxel label θ : Model Shape
: shortest distance of voxel i from the rendered model
![Page 14: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/14.jpg)
Object-Specific MRF
Smoothness Prior
x : Voxel label θ : Model Shape
Potts Model
![Page 15: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/15.jpg)
Object-Specific MRF
Unary Likelihood
x : Voxel label θ : Model Shape : Visual Hull
For a soft constraint we use a large constant K instead of infinity
![Page 16: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/16.jpg)
Object-Specific MRF
Energy Function
Shape Prior
Unary Likelihood
Smoothness Prior
Can be solved using Graph cuts
[Kolmogorov and Zabih, ECCV02 ]
![Page 17: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/17.jpg)
Object-Specific MRF
Energy Function
Shape Prior
Unary Likelihood
Smoothness Prior
How to find the optimal Pose?
![Page 18: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/18.jpg)
Outline
Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results
![Page 19: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/19.jpg)
Inference of Pose Parameters
Rotation and Translation of Torso in X axes
Rotation of left shoulder in X and Z axes
![Page 20: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/20.jpg)
Inference of Pose Parameters
Minimize F(ө) using Powell Minimization
Let F(ө) =
Computational Problem:
Each evaluation of F(ө) requires a graph cut to be computed. (computationally expensive!!) BUT..
Solution: Use the dynamic graph cut algorithm [Kohli&Torr, ICCV 2005]
![Page 21: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/21.jpg)
Outline
Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results
![Page 22: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/22.jpg)
Experiments
Deformable Models
Articulated Models• Reconstruction Results• Human Pose Estimation
![Page 23: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/23.jpg)
Deformable Models
Four Cameras 1.5 x 105 voxels DOF of Model: 5
Visual Hull
Our Reconstruction
Shape Model
![Page 24: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/24.jpg)
Articulated Models
![Page 25: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/25.jpg)
Articulated Models
Four Cameras 106 voxels DOF of Model: 26
Shape Model
Camera Setup
![Page 26: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/26.jpg)
Articulated Models
500 function evaluations of F(θ) required Time per evaluation: 0.15 sec Total time: 75 sec
Let F(ө) =
![Page 27: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/27.jpg)
Articulated Models
Visual Hull
Our Reconstruction
![Page 28: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/28.jpg)
Pose Estimation Results
Visual Hull
Reconstruction
Pose Estimate
![Page 29: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/29.jpg)
Pose Estimation Results
Quantitative Results• 6 uniformly distributed cameras• 12 degree (RMS) error over 21 joint angles
![Page 30: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/30.jpg)
Pose Estimation Results
Qualitative Results
![Page 31: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/31.jpg)
Pose Estimation Results
Video 1, Camera 1
![Page 32: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/32.jpg)
Pose Estimation Results
Video 1, Camera 2
![Page 33: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/33.jpg)
Pose Estimation Results
Video 2, Camera 1
![Page 34: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/34.jpg)
Pose Estimation Results
Video 2, Camera 2
![Page 35: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/35.jpg)
Future Work
• Use dimensionality reduction to reduce the number of pose parameters.
- results in less number of pose parameters to optimize- would speed up inference
• High resolution reconstruction by a coarse to fine strategy
• Parameter Learning in Object Specific MRF
![Page 36: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/36.jpg)
Thank You
![Page 37: Using Strong Shape Priors for Multiview Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081603/5681572e550346895dc4cabe/html5/thumbnails/37.jpg)
Object-Specific MRF
Energy Function
Shape Prior
Unary Likelihood
Smoothness Prior
+