digital photography and geometry capture...reconstructing rome1 • “the advent of digital...

74
Digital Photography and Geometry Capture NBAY 6120 March 9, 2016 Donald P. Greenberg Lecture 4

Upload: others

Post on 04-Oct-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Digital Photography and

Geometry Capture

NBAY 6120

March 9, 2016

Donald P. GreenbergLecture 4

Page 2: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Required Reading

• Bilger, Burkhard. "Has the Self-Driving Car Arrived at Last?" The New

Yorker. N.p., 25 Nov. 2013. Web. 10 Sept. 2015.

– http://www.newyorker.com/magazine/2013/11/25/auto-correct

Page 3: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

William F. Strapp. “Technology of the Portrait, p. 108.

Page 4: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Lincoln’s Daguerreotype

Page 5: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Civil War Mathew Brady

Page 6: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Eadweard Muybridge - Galloping Horse 1878

Page 7: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

from The Story of Kodak, Douglas Collins

Page 8: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Eastman and box. - (from The Story of Kodak, Douglas

Page 9: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Color Film Paradigm Shift

From multiple lenses or multiple exposures

to multiple layered film

The transition from the optical approach to the

chemical approach formed the new basis for

color photography

Mannes & Godowsky

1920’s

Page 10: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

(The Story of Kodak, Douglas Collins

Page 11: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Protective Layer

Blue-sensitive Emulsion

Yellow Filter

Green-sensitive Emulsion

Interlayer

Red-sensitive emulsion

Foundation Layer

Acetate Base

Anti-halation Backing

(fig. 1.6, Color Photography, Robert Hirsch, p.

Page 12: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Old Days - You Dropped Off Your Roll And Got Prints Back

Page 13: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Polaroid Land Camera

Page 14: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

(Charge-Coupled Devices for Quantitative Electronic Imaging, 1991 Photo

Photo-detector Technology

Page 15: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

CMOS Technology

• Complementary metal oxide semiconductor

• Cheaper manufacturing technology than CCD’s

– Follows the semiconductor industry cost curves

– Reduces the number of chips/camera required

• Processing (which is “free”) can perform calculations on each pixel within

frame time (e.g. correct for lighting, motion blur, etc.).

Page 16: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

$14,000 approximately, June 1994

Bayer Pattern

Page 17: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Requirements For Pervasive Digital Photography

• High resolution, low cost image acquisition devices

• Sufficient computer processing power and memory systems for digital

manipulation

• Image enhancement software with easy-to-use interfaces

• High density, low-cost local storage systems

Page 18: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

• Cheap LCD displays for previewing

• Bandwidth! Bandwidth! Bandwidth!

– High network bandwidth (wired) for distant transmission

– Fast throughput (e.g. Firewire) for local transmission

– Wireless bandwidth (local) for ease of use

• High quality, low cost digital printers

Requirements For Pervasive Digital Photography

Page 19: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

CONSUMERDigital Cameras 2012

Sony CyberShot

20 MegaPixels

$80

Kodak EASYSHARE Touch

M5370

Cost: $129.95

16 Megapixels

Page 20: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

PROFESSIONALDigital Cameras 2014

Nikon Digital SLR

16.2 MegaPixels

$5,999

Canon EOS 5DSR

50.6 MegaPixels

$3,899

Page 21: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Samsung Galaxy S6

Rear Facing Camera

(16 megapixels)

Front Facing Camera

(5 Megapixels)

http://www.wireless.att.com/cell-phone-service/packages/packages-details.jsp?q_package=sku4710254

Page 22: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

iPhone 6S Camera – 12 MPixels

Page 23: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Nokia Lumia 1020 – 41 MPixels

Page 24: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Advanced Digital Photography

Page 25: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Eye of a Fly AWARE-2 Duke University

http://www.nanowerk.com/spotlight/spotid=3744.php

Page 26: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

AWARE-2

http://mosaic.disp.duke.edu:90/aware/image_list/image_list/public

Page 27: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

AWARE-2

http://mosaic.disp.duke.edu:90/aware/image_list/image_list/public

Page 28: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

• http://gigapan.com/galleries/11088/gigapans/146504

Page 29: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Creating Full View Panoramic Image Mosaics and Environment Maps

3D rotation registration of four

images taken with a hand-held camera

Szeliski, R., Shum, H.-Y. (1997) SIGGRAPH.

Page 30: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Gigapixel Images Prof. Pedro Sander HKUST

Page 31: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

• http://gigapan.com/gigapans/58857

Page 32: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Canon’s 250-megapixel camera sensor 09/08/15

• Can read letters 11 miles away!

Page 33: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Digital Geometry Capture

• Photographic methods

• Laser scanning

• Pattern projection methods

• Time of Flight

Page 34: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Capturing Geometry from Photographs

How can we extract geometric information from a set

of photographs when we only have image data?

Page 35: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Simple case

Known camera positions (xe, ye, ze) , camera optics,

known corresponding points each image.

Jeremiah Fairbank. View dependent perspective images. Master's thesis, Cornell University, August 2005.

Page 36: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Early Work - 1975

Page 37: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Sagan House

Page 38: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Sagan House

Page 39: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Can we reconstruct the 3D geometry from a set of

photographs from the same camera?

Capturing Geometry from Photographs

Page 40: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Autodesk 123 Catch

Page 41: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

1 2 3 Catch Autodesk

Page 42: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

1 2 3 Catch Autodesk

Page 43: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

MagicPlan

Page 44: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Can we reconstruct the 3D geometry from an arbitrary

set of photographs?

Capturing Geometry from Photographs

Page 45: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Reconstructing Rome1

• “The advent of digital photography and the recent growth of photo-sharing

websites ( ) have brought about the seismic change in photography and

the use of photo collections.”1

• A search for the word “Rome” on returns two million photos.

• This collection, or others like it, capture every popular site, facade, statue,

fountain, interior, café, etc.

Page 46: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Characteristics of Typical Photo Sets

• The photos are unstructured

– No particular order or distribution of camera viewpoints

• The photos are uncalibrated

– Nothing is known about the camera settings (exposure, focal length, etc.)

• The scale is enormous

– (millions, not thousands of photos)

and

We need to do this fast!

Page 47: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Correspondence and 3D Structure from Different Camera Positions

Note: The pictures are in correspondence

2D dots with same color correspond to the same 3D points.

Page 48: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

3D Structure from Different Camera Positions

Camera 1

Camera 3

Camera 2

Page 49: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

3D Structure from Different Camera Positions

Assuming the position of the red dot is known, there is reprojection error in Camera 3.

Camera 1

Camera 3

Camera 2

Page 50: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Change the Problem to an optimization problem

• Minimize the sum of the squares of the reprojection errors.

• This non-linear least squares problem is difficult to solve due to local

minima and maxima.

Page 51: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Feature Detection and Matching

Sameer Agarwal, Yasutaka Furukawa, Naoh Snavely, Brian Curless, Steve M. Seitz, Richard Szeliski. “Reconstructing Rome”, IEEE Computer, June 2010.

A track corresponding to a point on the face of the central statue of Oceanus at the Trevi

Fountain, the embodiment of a river encircling the world in Greek mythology.

Page 52: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Colosseum

The Colosseum (Rome)

Sameer Agarwal, Yasutaka Furukawa, Naoh Snavely, Brian Curless, Steve M. Seitz, Richard Szeliski. “Reconstructing Rome”, IEEE Computer, June 2010.

Reconstructed dense 3D point models. For places with many available

images, reconstruction quality is very high.

Page 53: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Cornell Campus, McGraw Hall - Noah Snavely

Page 54: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Light Field Photography

Page 55: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

What is a Light Field?

• Radiance is defined as the light energy coming from a specific direction.

• A light field is defined as the radiance at a position (x, y),

and a direction (θ, φ).

• Thus, the light field is a 4-dimensional space.

Page 56: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Light Field

Page 57: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Light Field

Page 58: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Multi-camera array and LF camera are duals

Page 59: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Key LF-camera advantage: a single lens

• (more familiar, reduces complexity, simplifies calibration, …)

Page 60: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the
Page 61: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the
Page 62: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the
Page 63: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the
Page 64: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the
Page 65: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

The Lytro Camera

Page 66: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Lytro Camera

Page 67: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Key LF-camera advantage: a single lens

• (more familiar, reduces complexity, simplifies calibration, …)

Page 68: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Digital Geometry Capture

• Photographic methods

• Laser scanning

• Pattern projection methods

• Time of Flight

Page 69: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Cyberware Scanner

Page 70: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Cyberware Scanner Diagram

Page 71: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Cyberware Scanner

Page 72: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Uncle Don

Page 73: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

Cyberware vs. 1 2 3 Catch

Page 74: Digital Photography and Geometry Capture...Reconstructing Rome1 • “The advent of digital photography and the recent growth of photo-sharing websites ( ) have brought about the

End