3D photography

Marc Pollefeys

Fall 2008

http://www.inf.ethz.ch/personal/pomarc/courses/3dphoto/

3D Photography

• Obtaining 3D shape (and appearance) of real-world objects

Motivation

• Applications in many different areas

• A few examples …

Surgery - simulation

• simulate results of surgery• allows preoperative planning

Surgery - teaching

Capture models of surgery for interactive learning

Clothing

• Scan a person, custom-fit clothing

Forensics

• Crime scene recording and analysis

Forensics

                                

        

3D urban modeling

UNC/UKY UrbanScape project

Industrial inspection

• Verify specifications• Compare measured model with

CAD

       

                

Scanning industrial sites

as-build 3D model of off-shore oil platform

             

Robot navigation

small tethered rover

pan/tilt stereo head

ESA projectour task: Calibration + Terrain modelling + Visualization

Architecture

SurveyStability analysisPlan renovations

Architecture

SurveyStability analysisPlan renovations

Cultural heritageVirtual Monticello

Allow virtual visits

Cultural heritage

Stanford’s Digital Michelangelo

Digital archiveArt historic studies

Archaeology

accuracy ~1/500 from DV video (i.e. 140kb jpegs

576x720)

Record different excavation layers

Archaeology

Generate & verify construction hypothesis

Layer 1 Layer 2

Generate 4D excavation record

Computer games

Course objectives

• To understand the concepts that allow to recover 3D shape from images

• Explore the state of the art in 3D photography

• Implement a 3D photography system/algorithm

Material

Slides and more

http://www.inf.ethz.ch/personal/pomarc/courses/3dphoto/

Also check out on-line “shape-from-video” tutorial:http://www.cs.unc.edu/~marc/tutorial.pdfhttp://www.cs.unc.edu/~marc/tutorial/

Other interesting stuff:

• Book by Hartley & Zisserman, Multiple View Geometry

• Peter Allen’s 3D photography class webpage http://www1.cs.columbia.edu/~allen/F07/schedule.html

Learning approach

• Lectures: • cover main 3D photography concepts and

approaches.• Exercice sessions:

• Assignments (1st half semester) • Paper presentations and project updates (2nd

half semester) • 3D photography project:

• Choose topic, define scope (1st half semster)• Implement algorithm/system (2nd half semester)

Grade distribution

• Assignments & paper presentation: 50%• Course project: 50%

Content

• camera model and calibration• single-view metrology• triangulation• epipolar geometry, stereo and

rectification• structured-light, active techniques• feature tracking and matching• structure-from-motion• shape-from-silhouettes• space-carving• …

3D photography course schedule(tentative)

Lecture Exercise

Sept 17 Introduction -

Sept 24 Geometry & Camera model Camera calibration

Oct. 1 Single View Metrology Measuring in images

Oct. 8 Feature Tracking/matching(Jan-Michael Frahm?)

Correspondence computation

Oct. 15 Epipolar Geometry(David Gallup?)

F-matrix computation

Oct. 22 Shape-from-Silhouettes Visual-hull computation

Oct. 29 Stereo matching Project proposals

Nov. 5 Structured light and active range sensing

(TBD)

Papers

Nov. 12 Structure from motion Papers

Nov. 19 Multi-view geometry and self-calibration

Papers

Nov. 26 Shape-from-X Papers

Dec. 3 3D modeling and registration Papers

Dec. 10 Appearance modeling and image-based rendering

(TBD)

Papers

Dec. 17 Final project presentations

Fast Forward!

• Quick overview of what is coming…

Camera models and geometry

Pinhole camera

Geometric transformations in 2D and 3D

or

Camera Calibration

• Know 2D/3D correspondences, compute projection matrix

also radial distortion (non-linear)

Single View Metrology

Single View Metrology

Antonio Criminisi

Feature tracking and matching

Harris corners, KLT features, SIFT featureskey concepts: invariance of extraction, descriptors to viewpoint, exposure and illumination changes

l2

3D from images

C1m1 M?

L1

m2

L2

M

C2

Triangulation

- calibration

- correspondences

Epipolar Geometry

Fundamental matrix Essential matrix

Also how to robustly compute from images

Stereo and rectificationWarp images to simplify epipolar geometry

Compute correspondences for all pixels

Structured-light

• Projector = camera• Use specific patterns to obtain

correspondences

Initialize Motion (P1,P2 compatibel with F)

Initialize Structure (minimize reprojection error)

Extend motion(compute pose through matches seen in 2 or more previous views)

Extend structure(Initialize new structure, refine existing structure)

Structure from motion

*

*

projection

constraints

Auto-calibration

Shape-from-Silhouette

Space-carving

Only keep photo-consistent voxels

Shape-from-X

Shape-from-focus

Shape-from-texture

time-of-flight

3D modeling and texturing

Multiple depth images Surface model

Texture integration

patchwork texture map

Papers and discussion

• Will cover recent state of the art

Each student will present a paper, discussion

Select paper related to project

Course project: Build your own 3D scanner!

Example: Bouguet ICCV’98

Students can work in group or alone

3D photography course team

Marc PollefeysCAB F66marc.pollefeys@inf.ethz.eduTel. 044 63 23105

David GallupCAB F64.2gallup@cs.unc.eduTel. 044 23061