light field photography and videography marc levoy computer science department stanford university
TRANSCRIPT
![Page 1: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/1.jpg)
Light fieldphotography and videography
Marc Levoy
Computer Science DepartmentStanford University
![Page 2: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/2.jpg)
Marc Levoy
List of projects
• high performance imagingusing large camera arrays
• light field photographyusing a handheld plenoptic camera
• dual photography
![Page 3: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/3.jpg)
High performance imagingusing large camera arrays
Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez,Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy
(Proc. SIGGRAPH 2005)
![Page 4: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/4.jpg)
Marc Levoy
Stanford multi-camera array
• 640 × 480 pixels ×30 fps × 128 cameras
• synchronized timing
• continuous streaming
• flexible arrangement
![Page 5: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/5.jpg)
Marc Levoy
Ways to use large camera arrays
• widely spaced light field capture
• tightly packed high-performance imaging
• intermediate spacing synthetic aperture photography
![Page 6: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/6.jpg)
Marc Levoy
Intermediate camera spacing:synthetic aperture photography
![Page 7: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/7.jpg)
Marc Levoy
Example using 45 cameras[Vaish CVPR 2004]
![Page 8: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/8.jpg)
![Page 9: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/9.jpg)
Tiled camera array
• world’s largest video camera
• no parallax for distant objects
• poor lenses limit image quality
• seamless mosaicing isn’t hard
Can we match the image quality of a cinema camera?
![Page 10: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/10.jpg)
Tiled panoramic image(before geometric or color calibration)
![Page 11: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/11.jpg)
Tiled panoramic image(after calibration and blending)
![Page 12: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/12.jpg)
Tiled camera array
• world’s largest video camera
• no parallax for distant objects
• poor lenses limit image quality
• seamless mosaicing isn’t hard
• per-camera exposure metering
• HDR within and between tiles
Can we match the image quality of a cinema camera?
![Page 13: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/13.jpg)
same exposure in all cameras
individually metered
checkerboardof exposures
![Page 14: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/14.jpg)
Marc Levoy
High-performance photography as multi-dimensional sampling
• spatial resolution
• field of view
• frame rate
• dynamic range
• bits of precision
• depth of field
• focus setting
• color sensitivity
![Page 15: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/15.jpg)
Marc Levoy
Spacetime aperture shaping
• shorten exposure time to freeze motion → dark
• stretch contrast to restore level → noisy
• increase (synthetic) aperture to capture more light → decreases depth of field
![Page 16: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/16.jpg)
• center of aperture: few cameras, long exposure →high depth of field, low noise,but action is blurred
• periphery of aperture: many cameras, short exposure → freezes action, low noise,but low depth of field
![Page 17: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/17.jpg)
![Page 18: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/18.jpg)
![Page 19: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/19.jpg)
Marc Levoy
Light field photography using a handheld plenoptic camera
Ren Ng, Marc Levoy, Mathieu Brédif,Gene Duval, Mark Horowitz and Pat Hanrahan
(Proc. SIGGRAPH 2005and TR 2005-02)
![Page 20: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/20.jpg)
Marc Levoy
Conventional versus light field camera
![Page 21: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/21.jpg)
Marc Levoy
Conventional versus light field camera
uv-plane st-plane
![Page 22: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/22.jpg)
Marc Levoy
Conventional versus light field camera
uv-planest-plane
![Page 23: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/23.jpg)
Prototype camera
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens
Contax medium format camera Kodak 16-megapixel sensor
Adaptive Optics microlens array 125μ square-sided microlenses
![Page 24: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/24.jpg)
Marc Levoy
Mechanical design
• microlenses float 500μ above sensor
• focused using 3 precision screws
![Page 26: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/26.jpg)
Marc Levoy
Prior work
• integral photography– microlens array + film
– application is autostereoscopic effect
• [Adelson 1992]– proposed this camera
– built an optical bench prototype using relay lenses
– application was stereo vision, not photography
![Page 27: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/27.jpg)
Marc Levoy
Digitally stopping-down
• stopping down = summing only the central portion of each microlens
Σ
Σ
![Page 28: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/28.jpg)
Marc Levoy
Digital refocusing
• refocusing = summing windows extracted from several microlenses
Σ
Σ
![Page 29: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/29.jpg)
Marc Levoy
A digital refocusing theorem
• an f / N light field camera, with P × P pixels under each microlens, can produce views as sharp as an f / (N × P) conventional camera
– or –
• it can produce views with a shallow depth of field ( f / N ) focused anywhere within the depth of field of an f / (N × P) camera
![Page 30: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/30.jpg)
Marc Levoy
Example of digital refocusing
![Page 31: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/31.jpg)
Marc Levoy
Refocusing portraits
![Page 32: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/32.jpg)
Marc Levoy
Action photography
![Page 33: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/33.jpg)
Extending the depth of field
conventional photograph,main lens at f / 22
conventional photograph,main lens at f / 4
light field, main lens at f / 4,after all-focus algorithm
[Agarwala 2004]
![Page 34: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/34.jpg)
Marc Levoy
Macrophotography
![Page 35: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/35.jpg)
Marc Levoy
Digitally moving the observer
• moving the observer = moving the window we extract from the microlenses
Σ
Σ
![Page 36: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/36.jpg)
Marc Levoy
Example of moving the observer
![Page 37: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/37.jpg)
Marc Levoy
Moving backward and forward
![Page 38: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/38.jpg)
Marc Levoy
Implications
• cuts the unwanted link between exposure(due to the aperture) and depth of field
• trades off (excess) spatial resolution for ability to refocus and adjust the perspective
• sensor pixels should be made even smaller, subject to the diffraction limit
36mm × 24mm ÷ 2.5μ pixels = 266 megapixels
20K × 13K pixels
4000 × 2666 pixels × 20 × 20 rays per pixel
![Page 39: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/39.jpg)
Marc Levoy
Can we build a light field microscope?
• ability to photograph moving specimens
• digital refocusing → focal stack →deconvolution microscopy → volume data
![Page 40: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/40.jpg)
Dual Photography
Pradeep Sen, Billy Chen, Gaurav Garg, Steve Marschner,Mark Horowitz, Marc Levoy, Hendrik Lensch
(Proc. SIGGRAPH 2005)
![Page 41: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/41.jpg)
Helmholtz reciprocity
scene
light
camera
![Page 42: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/42.jpg)
Helmholtz reciprocity
scene
camera
light
![Page 43: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/43.jpg)
photocell
scene
Measuring transport along a set of paths
projector
![Page 44: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/44.jpg)
scene
point light
Reversing the pathscamera
![Page 45: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/45.jpg)
Forming a dual photograph
scene
photocellprojector “dual” light“dual” camera
![Page 46: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/46.jpg)
Forming a dual photograph
scene
image ofscene
“dual” light“dual” camera
![Page 47: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/47.jpg)
Physical demonstration
• light replaced with projector
• camera replaced with photocell
• projector scanned across the scene
conventional photograph,with light coming from right
dual photograph,as seen from projector’s position
and as illuminated from photocell’s position
![Page 48: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/48.jpg)
Marc Levoy
Related imaging methods
• time-of-flight scanner– if they return reflectance as well as range
– but their light source and sensor are typically coaxial
• scanning electron microscope
Velcro® at 35x magnification,Museum of Science, Boston
![Page 49: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/49.jpg)
The 4D transport matrix
scene
photocellprojector camera
![Page 50: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/50.jpg)
camera
The 4D transport matrix
scene
projector
P
pq x 1
C
mn x 1
T
mn x pq
![Page 51: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/51.jpg)
T PC =
The 4D transport matrix
pq x 1mn x 1
mn x pq
![Page 52: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/52.jpg)
TC =
The 4D transport matrix
10000
mn x pq
pq x 1mn x 1
![Page 53: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/53.jpg)
TC =
The 4D transport matrix
01000
mn x pq
pq x 1mn x 1
![Page 54: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/54.jpg)
TC =
The 4D transport matrix
00100
mn x pq
pq x 1mn x 1
![Page 55: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/55.jpg)
The 4D transport matrix
T PC =
pq x 1mn x 1
mn x pq
![Page 56: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/56.jpg)
The 4D transport matrix
applying Helmholtz reciprocity...
T PC =
pq x 1mn x 1
mn x pq
T P’C’ =
mn x 1pq x 1
pq x mn
T
![Page 57: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/57.jpg)
Marc Levoy
Example
conventional photographwith light coming from right
dual photographas seen from projector’s position
![Page 58: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/58.jpg)
Marc Levoy
Properties of the transport matrix
• little interreflection → sparse matrix
• many interreflections → dense matrix
• convex object → diagonal matrix
• concave object → full matrix
Can we create a dual photograph entirely from diffuse reflections?
![Page 59: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/59.jpg)
Marc Levoy
Dual photographyfrom diffuse reflections
the camera’s view
![Page 60: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/60.jpg)
Marc Levoy
The relighting problem
• subject captured under multiple lights
• one light at a time, so subject must hold still
• point lights are used, so can’t relight with cast shadows
Paul Debevec’sLight Stage 3
![Page 61: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/61.jpg)
The 6D transport matrix
![Page 62: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/62.jpg)
The 6D transport matrix
![Page 63: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/63.jpg)
Marc Levoy
The advantage of dual photography
• capture of a scene as illuminated by different lights cannot be parallelized
• capture of a scene as viewed by different cameras can be parallelized
![Page 64: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/64.jpg)
scene
Measuring the 6D transport matrix
projector
camera arraymirror arraycamera
![Page 65: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/65.jpg)
Marc Levoy
Relighting with complex illumination
• step 1: measure 6D transport matrix T
• step 2: capture a 4D light field
• step 3: relight scene using captured light field
scene
camera arrayprojector
TT P’C’ =
mn x uv x 1pq x 1
pq x mn x uv
![Page 66: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/66.jpg)
Marc Levoy
Running time
• the different rays within a projector can in fact be parallelized to some extent
• this parallelism can be discovered using a coarse-to-fine adaptive scan
• can measure a 6D transport matrix in 5 minutes
![Page 67: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/67.jpg)
Can we measure an 8D transport matrix?
scene
camera arrayprojector array
![Page 68: Light field photography and videography Marc Levoy Computer Science Department Stanford University](https://reader038.vdocuments.mx/reader038/viewer/2022102818/56649d035503460f949d6322/html5/thumbnails/68.jpg)
Marc Levoyhttp://graphics.stanford.edu
T PC =
pq x 1mn x 1 mn x pq