single-view 3d reconstruction
DESCRIPTION
10/12/10. Single-view 3D Reconstruction. Computational Photography Derek Hoiem, University of Illinois. Some slides from Alyosha Efros, Steve Seitz. Questions about Project 4?. Get me your labels asap if not already done (will count as late days after today) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/1.jpg)
Single-view 3D Reconstruction
Computational PhotographyDerek Hoiem, University of Illinois
10/12/10
Some slides from Alyosha Efros, Steve Seitz
![Page 2: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/2.jpg)
Questions about Project 4?• Get me your labels asap if not already done
(will count as late days after today)– Will post everything I’ve got later today
So far have labels from: Adair, Arash, Bret, Brian, Derek, Doheum, Lewis, Prateek, Raj, Russ, SteveD, WeiLou
![Page 3: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/3.jpg)
Project 3• Class favorites: would like more votes – vote
today if you haven’t– “Favorites” will be presented Thursday
![Page 4: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/4.jpg)
Take-home question
Suppose a 3D cube is facing the camera at a distance.
1. Draw the cube in perspective projection2. Now draw the cube in weak perspective projection
10/5/10
![Page 5: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/5.jpg)
Take-home questionSuppose you have estimated three vanishing points corresponding to orthogonal directions. How can you recover the rotation matrix that is aligned with the 3D axes defined by these points?– Assume that intrinsic matrix K has three parameters– Remember, in homogeneous coordinates, we can write a 3d point at
infinity as (X, Y, Z, 0)
10/7/2010
Photo from online Tate collection
VPx
VPz.
VPy
![Page 6: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/6.jpg)
Take-home questionAssume that the camera height is 5 ft. – What is the height of the man?– What is the height of the building?– How long is the right side of the building compared to the
small window on the right side of the building?
10/7/2010
![Page 7: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/7.jpg)
![Page 8: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/8.jpg)
Today’s class: 3D Reconstruction
![Page 9: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/9.jpg)
The challengeOne 2D image could be generated by an infinite number of 3D geometries
?
?
?
![Page 10: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/10.jpg)
The solution
Make simplifying assumptions about 3D geometry
Unlikely Likely
![Page 11: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/11.jpg)
Today’s class: Two Models
• Box + frontal billboards
• Ground plane + non-frontal billboards
![Page 12: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/12.jpg)
“Tour into the Picture” (Horry et al. SIGGRAPH ’97)
Create a 3D “theatre stage” of five billboards
Specify foreground objects through bounding polygons
Use camera transformations to navigate through the scene
Following slides modified from Efros
![Page 13: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/13.jpg)
The ideaMany scenes (especially paintings), can be represented as an axis-aligned box volume (i.e. a stage)
Key assumptions• All walls are orthogonal• Camera view plane is parallel to back of volume
How many vanishing points does the box have?• Three, but two at infinity• Single-point perspective
Can use the vanishing point to fit the box to the particular scene
![Page 14: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/14.jpg)
Fitting the box volume
• User controls the inner box and the vanishing point placement (# of DOF???)
• Q: What’s the significance of the vanishing point location?
• A: It’s at eye (camera) level: ray from COP to VP is perpendicular to image plane.
![Page 15: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/15.jpg)
High Camera
Example of user input: vanishing point and back face of view volume are defined
![Page 16: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/16.jpg)
High Camera
Example of user input: vanishing point and back face of view volume are defined
![Page 17: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/17.jpg)
Example of user input: vanishing point and back face of view volume are defined
Low Camera
![Page 18: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/18.jpg)
Low Camera
Example of user input: vanishing point and back face of view volume are defined
![Page 19: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/19.jpg)
High Camera Low Camera
Comparison of how image is subdivided based on two different camera positions. You should see how moving the box corresponds to moving the eyepoint in the 3D world.
![Page 20: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/20.jpg)
Left Camera
Another example of user input: vanishing point and back face of view volume are defined
![Page 21: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/21.jpg)
Left Camera
Another example of user input: vanishing point and back face of view volume are defined
![Page 22: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/22.jpg)
RightCamera
Another example of user input: vanishing point and back face of view volume are defined
![Page 23: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/23.jpg)
RightCamera
Another example of user input: vanishing point and back face of view volume are defined
![Page 24: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/24.jpg)
Left Camera Right Camera
Comparison of two camera placements – left and right. Corresponding subdivisions match view you would see if you looked down a hallway.
![Page 25: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/25.jpg)
Question• Think about the camera center and image
plane…– What happens when we move the box?– What happens when we move the vanishing
point?
![Page 26: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/26.jpg)
2D to 3D conversion
• First, we can get ratios
left right
top
bottom
vanishingpoint
backplane
![Page 27: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/27.jpg)
Size of user-defined back plane determines width/height throughout box (orthogonal sides)
Use top versus side ratio to determine relative height and width dimensions of box
Left/right and top/botratios determine part of 3D camera placement
left right
top
bottomcamerapos
2D to 3D conversion
![Page 28: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/28.jpg)
Depth of the box
• Can compute by similar triangles (CVA vs. CV’A’)• Need to know focal length f (or FOV)
• Note: can compute position on any object on the ground– Simple unprojection– What about things off the ground?
![Page 29: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/29.jpg)
Homography
A
B
C
D
A’B’
C’ D’
2d coordinates3d plane coordinates
![Page 30: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/30.jpg)
Image rectification
To unwarp (rectify) an image solve for homography H given p and p’: wp’=Hp
pp’
On board
![Page 31: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/31.jpg)
Solving for homographies
![Page 32: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/32.jpg)
Solving for homographies
A h 0
Defines a least squares problem:
2n × 9 9 2n
• Since h is only defined up to scale, solve for unit vector ĥ• Solution: ĥ = eigenvector of ATA with smallest eigenvalue• Works with 4 or more points
![Page 33: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/33.jpg)
Tour into the picture algorithm
1. Set the box corners
![Page 34: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/34.jpg)
Tour into the picture algorithm
1. Set the box corners2. Set the VP3. Get 3D coordinates
– Compute height, width, and depth of box
4. Get texture maps– homographies for
each face x
![Page 35: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/35.jpg)
Result
Render from new views
http://www.cs.cmu.edu/afs/cs.cmu.edu/academic/class/15463-f08/www/proj5/www/dmillett/
![Page 36: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/36.jpg)
Foreground ObjectsUse separate billboard for each
For this to work, three separate images used:
– Original image.– Mask to isolate desired
foreground images.– Background with
objects removed
![Page 37: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/37.jpg)
Foreground Objects
Add vertical rectangles for each foreground object
Can compute 3D coordinates P0, P1 since they are on known plane.
P2, P3 can be computed as before (similar triangles)
![Page 38: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/38.jpg)
Foreground Result
Video from CMU class: http://www.youtube.com/watch?v=dUAtdmGwcuM
![Page 39: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/39.jpg)
Automatic Photo Pop-up
Input
Ground
Vertical
Sky
Geometric Labels Cut’n’Fold 3D Model
Image
Learned Models
Hoiem et al. 2005
![Page 40: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/40.jpg)
Cutting and Folding
• Fit ground-vertical boundary– Iterative Hough transform
![Page 41: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/41.jpg)
Cutting and Folding
• Form polylines from boundary segments– Join segments that intersect at slight angles– Remove small overlapping polylines
• Estimate horizon position from perspective cues
![Page 42: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/42.jpg)
Cutting and Folding
• ``Fold’’ along polylines and at corners• ``Cut’’ at ends of polylines and along vertical-sky boundary
![Page 43: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/43.jpg)
Cutting and Folding
• Construct 3D model • Texture map
![Page 44: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/44.jpg)
Results
Automatic Photo Pop-up
Input Image
Cut and Fold
http://www.cs.illinois.edu/homes/dhoiem/projects/popup/
![Page 45: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/45.jpg)
Results
Automatic Photo Pop-upInput Image
![Page 46: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/46.jpg)
Comparison with Manual Method
Input Image
Automatic Photo Pop-up (30 sec)!
[Liebowitz et al. 1999]
![Page 47: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/47.jpg)
Failures
Labeling Errors
![Page 48: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/48.jpg)
Failures
Foreground Objects
![Page 49: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/49.jpg)
Adding Foreground Labels
Recovered Surface Labels + Ground-Vertical Boundary Fit
Object Boundaries + Horizon
![Page 50: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/50.jpg)
![Page 51: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/51.jpg)
Final project ideas• 1 Person
– Interactive program to make 3D model from an image (e.g., output in VRML, or draw path for animation)
• 2 Person– Add tools for cutting out foreground objects and
automatic hole-filling
![Page 52: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/52.jpg)
Summary• 2D3D is mathematically
impossible
• Need right assumptions about the world geometry
• Important tools– Vanishing points– Camera matrix– Homography
![Page 53: Single-view 3D Reconstruction](https://reader036.vdocuments.mx/reader036/viewer/2022081506/5681305c550346895d961fc0/html5/thumbnails/53.jpg)
The future….• 10/26: Kevin Karsch will show his work that
applies many ideas from this section (cutting, texture synthesis, projective geometry, etc.)
• Next class, we start a new section– Finding correspondences automatically– Image stitching– Various forms of recognition