introduction to light fields

Slide 1

Introduction to Light Fields

1

The 5D Plenoptic FunctionQ: What is the set of all things that one can ever see?A: The Plenoptic Function [Adelson and Bergen 1991](from plenus, complete or full, and optic)

P(q, f, l, t)

The 5D Plenoptic FunctionQ: What is the set of all things that one can ever see?A: The Plenoptic Function [Adelson and Bergen 1991](from plenus, complete or full, and optic)

P(q, f, l, t)

Position-Angle Parameterization

2D position2D direction

s

q

Two-Plane Parameterization2D position2D position

s

u

Two-Plane Parameterization camera array2D position2D position

s

u

Two-Plane Parameterization SLR camera2D position2D position

s

u

Two-Plane Parameterization cell phone

s

u

Two-Plane Parameterization cell phone (pelican)

s

u

Application: Digital Image Refocusing[Ng 2005]

Kodak 16-megapixel sensor

125 square-sided microlenses

Application: 3D Displays

Parallax Panoramagram[Kanolt 1918]3DTV with Integral Imaging[Okano et al. 1999]MERL 3DTV[Matusik and Pfister 2004]

Multiple Sensors

12

Static Camera Arrays

Stanford Multi-Camera Array 125 cameras using custom hardware[Wilburn et al. 2002, Wilburn et al. 2005]

Distributed Light Field Camera64 cameras with distributed rendering[Yang et al. 2002]

Refocus

Temporal Multiplexing

15

Controlled Camera or Object MotionStanford Spherical Gantry[Levoy and Hanrahan 1996]

Relighting with 4D Incident Light Fields[Masselus et al. 2003]

Uncontrolled Camera or Object MotionUnstructured Lumigraph Rendering[Gortler et al. 1996; Buehler et al. 2001]

Spatial Multiplexing

18

Parallax Barriers (Pinhole Arrays)[Ives 1903]

sensorbarrierSpatially-multiplexed light field capture using masks (i.e., barriers):Cause severe attenuation long exposures or lower SNRImpose fixed trade-off between spatial and angular resolution (unless implemented with programmable masks, e.g. LCDs)

Integral Imaging (Flys Eye Lenslets)[Lippmann 1908]

sensorlensletf

Spatially-multiplexed light field capture using lenslets:Impose fixed trade-off between spatial and angular resolution

Light Field Photograph (Sensor)

Light Field Photograph (Decoded)[The (New) Stanford Light Field Archive]

looking uplooking to the rightSample Image

DEMO

lenstoys: light field camera

23

Modern, Digital ImplementationsDigital Light Field Photography Hand-held plenoptic camera [Ng et al. 2005] Heterodyne light field camera [Veeraraghavan et al. 2007]

Light Field = Array of (Virtual) CamerasSlide by Marc Levoy

2007 Marc LevoyTime = 25

Sub-apertureVirtual Camera = Sub-aperture ViewLight Field = Array of (Virtual) CamerasSlide by Marc Levoy


Sub-apertureVirtual Camera = Sub-aperture View

Light Field = Array of (Virtual) CamerasSlide by Marc Levoy


2007 Marc LevoyDEMO

http://lightfield.stanford.edu/aperture.swf?lightfield=data/chess_lf/preview.zip&zoom=1

The Lytro Light Field Camera

30

Stanford camera array

Microlens camera array

DigitalsensorMicrolens array (MLA)

Lytro: microlens array in camera

Main lensMicrolens array (MLA)DigitalsensorLens is not really thin, but can be treated as so.

330 330 hex array, 13.9 micron pitchOccluders are not required.14 Mpixel, square cropped to 11 MpixelsChoose pixel in same location behind each microlens

Microlens array (MLA)Main lensDigitalsensor# of sub-apertures=# of pixels behind each microlens(10 10)# of pixels persub-aperture image= # of microlenses(~ 120,000)

All rays pass through a sub-aperture

Sub-aperture captures on camera view

Choose pixel in same location behind each microlens

So why put the microlens array inside the camera?

Lytro camera has unusual shape

8x f/2 lens

Light field sensorBattery43-343 mm equivalent

Other Applications of Light Fields

36

Lens Glare Reduction [Raskar, Agrawal, Wilson, Veeraraghavan SIGGRAPH 2008]Glare/Flare due to camera lenses reduces contrast

Reducing Glare

Glare Reduced ImageAfter removing outliersConventional Photo

MERL, MIT Media LabGlare Aware Photography: 4D Ray Sampling for Reducing Glare

Raskar, Agrawal, Wilson & Veeraraghavan

Enhancing Glare

Glare Enhanced Image

Conventional Photo







Glare due to Lens Inter-Reflections

a

Sensor

b



Effects of Glare on ImageHard to model, Low Frequency in 2DBut reflection glare is outlier in 4D ray-spaceAngular Variation at pixel aLens Inter-reflections

a

Sensor

b



43

Key IdeaLens Glare manifests as low frequency in 2D Image

But Glare is highly view dependentmanifests as outliers in 4D ray-space

Reducing Glare == Remove outliers among rays

a

Sensor

b



Reducing Glare using a Light Field Camera



Single Shot Light Field Cameras

Using Mask, this paper

Using Lenslets, Ng et al. 2005

MaskAdelson and Wang, 1992, Ng et al. 2005 Kanolt 1933, Veeraraghavan et al. 2007



Put diagrams here46

Captured Photo: LED off



Captured Photo: LED On



Each Disk: Angular Samples at that Spatial Location

No Glare



With Glare



x

y

u

v



Sub aperture views51

Sequence of Sub-Aperture Views

Average of all the Light Field viewsOne of the Light Field viewsLow Res Traditional Camera PhotoGlare Reduced Image



Key Idea

Reducing Glare == Remove outlier among angular samples

a

Sensor

b



Light-sensitive DisplaysDepth CamerasMulti-touch Interaction

Multi-touch Interaction with Thin Displays

54

Hover InteractionBiDi Screen: Thin, Depth-sensing LCDs Seamless transition from multi-touch to gesture Thin form factor (LCD)

55

LCD Modifications

56

DiffuserCameraLCDLightsPrototype

57

introduction to light fields

Art & Photos