i 3d: interactive planar reconstruction of objects and scenes adarsh kowdleyao-jen chang tsuhan chen...
TRANSCRIPT
![Page 1: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/1.jpg)
i3D: Interactive Planar Reconstruction of Objects and Scenes
Adarsh Kowdle Yao-Jen Chang Tsuhan Chen
School of Electrical and Computer Engineering
Cornell University
09/25/2009WNYIP 2009
![Page 2: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/2.jpg)
2
i3D: Interactive planar reconstruction
![Page 3: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/3.jpg)
3
i3D: Interactive planar reconstruction
![Page 4: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/4.jpg)
4
Introduction to i3D
Automatic 3D reconstruction algorithms
Snavely et. al., SIGGRAPH ‘06
Furukawa et. al., CVPR ‘09
Sinha et. al., ICCV ‘09
Furukawa et. al., CVPR ‘07
![Page 5: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/5.jpg)
5
Introduction to i3DInteractive 3D reconstruction algorithms
Hengel et. al., VideoTrace
SIGGRAPH ‘07
Debevec et. al., SIGGRAPH ‘96
Sinha et. al., SIGASIA ‘08
![Page 6: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/6.jpg)
6
Introduction to i3D
Scribbles as a form of interaction
Srivatsava et. al., VMV ‘09
Li et al. Lazy Snapping. SIGGRAPH ’04.
Boykov and Jolly, ICCV ’01.
Improvement to Make3D (3D from a single image)
![Page 7: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/7.jpg)
Algorithm
![Page 8: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/8.jpg)
8
Preprocessing
Structure from motion
Superpixel map – mean shift
SIFT-like daisy descriptor
![Page 9: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/9.jpg)
9
Interactions
![Page 10: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/10.jpg)
10
Segmentation What do we do the scribbles?
One-against-all classifier – Logistic regression Feature vector – mean daisy descriptor of pixels in
each labeled superpixel Test each image – obtain response for each class Maintain smoothness in segmentation – energy
minimization problem
Node energy
Edge energy
![Page 11: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/11.jpg)
11
Segmentation
![Page 12: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/12.jpg)
12
Plane co-segmentation
Transfer labels to 3D point cloud
Provide edge connectedness to help estimate a globally accurate plane
Estimate plane parameters – RANSAC
Estimate homography for each plane from scribbled image to other images
Warp these segments Also an energy minimization problem
![Page 13: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/13.jpg)
13
Plane co-segmentation
![Page 14: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/14.jpg)
14
Closing the loop
3D Model
![Page 15: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/15.jpg)
Visualizations
![Page 16: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/16.jpg)
16
Texture synthesis and rendering Texture preprocessing – segmentation
refinement #
View selection*
Incrementally build the model
Visualizations Dense point cloud Mesh visualization
*Debevec et. al., SIGGRAPH
’96
#Details in the paper
![Page 17: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/17.jpg)
17
Segmentation refinement
![Page 18: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/18.jpg)
18
Segmentation refinement
![Page 19: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/19.jpg)
19
Dense point cloud with texture Perform view selection and incrementally build the
model
Use camera projection matrix and back-project each pixel onto the corresponding plane
![Page 20: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/20.jpg)
20
Mesh visualization Modeling the 3D geometry using triangular
meshes Single mesh model – Single image texture
Multiple mesh model
![Page 21: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/21.jpg)
21
i3D: Interactive planar reconstruction More results…
![Page 22: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/22.jpg)
22
i3D: Interactive planar reconstruction More results…
![Page 23: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/23.jpg)
23
Conclusion i3D, a new interactive planar 3D
reconstruction algorithm using simple user interactions in the form of scribbles
Discuss multiple methods to refine segmentations and correct superpixel leaks
Complete and pleasing reconstructions – described two forms of visualizations
![Page 24: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/24.jpg)
Thank You
![Page 25: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/25.jpg)
![Page 26: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/26.jpg)
26
i3D: Interactive planar reconstruction More results…
![Page 27: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/27.jpg)
27
i3D: Interactive planar reconstruction More results…
![Page 28: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/28.jpg)
28
Closing the loop User provides scribbles on any image Perform segmentation of the image using the
daisy descriptors Provide edge connectedness scribbles Estimate plane parameters Perform plane co-segmentation Loop – Control with the user
![Page 29: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/29.jpg)
29
View selection Rank images based on angle between the
plane normal and viewing direction – lower the angle better the view
Incrementally build the model* Start with the best view for each plane – will have
holes due to occlusion Proceed with the next one in order of ranking and
incrementally fill up the holes
*Debevec et. al., SIGGRAPH ‘96
![Page 30: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/30.jpg)
30
Segmentation refinement Stage 1:
Warp segments from each image to every other image
For each image refine the segmentations of each plane by taking a per-pixel vote of all the warps
Stage 2: (mainly for scenes with polygon boundaries) Use the line of intersection of every pair
of planes Project this line in 3D onto images Use line to define plane boundary
![Page 31: I 3D: Interactive Planar Reconstruction of Objects and Scenes Adarsh KowdleYao-Jen Chang Tsuhan Chen School of Electrical and Computer Engineering Cornell](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56649ebd5503460f94bc6b06/html5/thumbnails/31.jpg)
31
3D Visualization Visualization chosen based on application
Full model required Dense point cloud with texture Multiple mesh model
Pleasing visualization – fly through Single mesh model