study dappled photography

Dappled Photography: Mask Enhanced Cameras forHeterodyned Light Fields and Coded Aperture Refocusing

Ashok Veeraraghavan, Ramesh Raskar, Amit AgrawalMitsubishi Electric Research Labs (MERL), Cambridge, MA

Ankit Mohan, Jack TumblinNorthwestern University, Evanston, IL

study

http://www.cfar.umd.edu/~vashok/

http://www.merl.com/

Abstract

• Heterodynes Light Field camera

• Coded Aperture camera

A theoretical framework for modulating 4D light fields using a mask between lens and sensors

Abstract


1. Add a high-frequency mask between Lens and Sensors

2. 4D light field - sense different rays from lens (u,v) in a sensor position (x,y)

3. Re-arrange light field

• Low resolution with different focus settings

• Full resolution in-focus


Abstract







• Coded Aperture camera 1. Replace the Aperture

with a coded mask

2. A broadband mask enhance refocus at full resolution for Lambertian scene


4D light field L(u,v,x,y)

Abstract




3. Rearrange light field




Abstract







• Coded Aperture camera 1. Replace the Aperture

with a coded mask

2. A broadband mask enhance refocus at full resolution for Lambertian scene

3. Refocusing partial


Heterodynes Light Field camera


• Based on modulation theorem in 4D frequency domain – mask carries rays

= rays x mask

F MaskRays

Modulation Theorem

• [Oppenheim et al. 99]http://en.wikipedia.org/wiki/Amplitude_modulation

http://en.wikipedia.org/wiki/Amplitude_modulation

http://en.wikipedia.org/wiki/Amplitude_modulation



= rays x mask

α depends on (d,v)

F MaskRays

good mask !

A poor mask blends the rays ! A good mask carriers the rays !

rays



= rays x mask

F MaskRays



• recover the light field by rearranging the tiles of 2D Fourier transfer into 4D plane to get the full resolution image information for the in-focus parts of the scene

F-1

= rays x mask

Rearrange

F MaskRays



• recover the light field by rearranging the tiles of 2D Fourier transfer into 4D plane to get the full resolution image information for the in-focus parts of the scene

• A raw sensor holds a modulated 4D light filed

= rays x mask

Raw sensor (modulate 4D light field

data)

In-focus at full resolution (demodulated)

Optical Heterodyning

Baseband Audio Signal

Receiver: DemodulationHigh Freq Carrier 100.1 MHz

ReferenceCarrier

Incoming Signal

Photographic Signal

(Light Field)

Carrier Incident Modulated

SignalReference

Carrier

Main LensObject Mask Sensor

RecoveredLight Field

Software Demodulation

99 MHz

Coded Aperture camera

• Base on Convolution• Aperture as a Modulator

▫ sinc function depends on θ

▫ Pinhole camera has a very very broadband modulator

▫ Design broadband mask

= rays x mask

Outline

•Introduction•Related Work•Theory & Framework•Heterodyne Light Field Camera•Encoded Blur Camera•Implements & Analysis•Contributions & Future Work

Introduction

Light Field

http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt

SensorLens

θ x

Imaginary film

x

θ

Sensed image (in-focus)

object

object

red : the in-focus lineyellow : sample



Lens Sensor

θ x

Imaginary film

x

θ

Sensed image (in-focus)



Lens Sensor

u x

Imaginary film

x

θ

Sensed image (in-focus)red : the in-focus lineyellow : sample



Lens Sensor

θ x

x

θ

Imaginary film

Sensed image (out of focus, far)red : the in-focus lineyellow : sample



Lens Sensor

θ x

x

θ

Imaginary film

Sensed image (out of focus, near)red : the in-focus lineyellow : sample


Light Field Acquisition

Integral Photography• Interal Photograpy

[Lippmann 1908]

• Integral camera [Okano et al. 99; Martnez-Corral et al. 04; Javidi and Okano 02]

• Light field Camera▫ Virtual viewpoint

[Levoy and Hanrahan 96]

[Gertler et al 96]

▫ Virtual aperture [Levoy and Hanrahan

96] [Isaksen et al. 00]

▫ Synthetic appearture photography (similar virtual aperture) [Levoy et al. 04] [Vaish et al. 04]

Light field Camera

Plenoptic cameraLight field rendering Dapped Photography

• [Levoy and Hanrahan Siggraph 96]

• [Gortler et al 96, 06]

• [Adelson et al, IEEE95]

• [Levoy and Hanrahan Siggraph 96]

• [Gortler et al 96]

• Hand-held light field camera [R Ng et al 05]

• Fourier slice photography [R Ng, SIGGRAPH05]

• The mask weights the rays

Related work

• Coded Imaging▫ Coded aperture imaging

Overcome the limit of pinhole camera [Skinner 98]

▫ Coded Exposure Camera [Raskar et al. 06]

• Wavefront Coding [Dowski and Cathey 95; Dowski and Johnson 99; van der

Gracht et al. 96] Traditional lens based [Farid and Simoncelli 98] Both wavefront and coded aperture [Jahnson et al. 00]

• Deblurring & deconvolution Include extended DOF images by refocusing a light field at

multiple depth and applying the digital photomontage tech. [Agarwala et al. 04]

Fusion of multiple blurred images [ Jaeberli 94]

Theory & Framework

For different focus settings, the obtained images correspond to slices at different angles, “Fourier Slice Photography ” [Ng, R. 05]

Assumption : simulate the aperture as mask placed at lens

Open Aperture

Assumption : a planar Lambertian object at the focus plane

Because no angular variations in the irradiance of rays from a Lambertian object, the content of light field is restricted to be along the fx axis

The sensed image is a slice of the modulated light field

Open Aperture

• In-focus sensor

▫ The in-focus image corresponds to a slice of LA(fx, f θ) along fx (f θ =0)

▫ No information lost

• Out of focus sensor▫ The sensor image is a slanted slice

▫ The slant angle depends on the degree of mis-focus

Open Aperture

Heterodyne Light Field Camera

Mask as Modulator

Mask as Modulator

• d = v (at aperture stop, θ plane)▫ Mask affects the all rays at an angle θ in a similar way !▫ m(x, θ) = c (y = θ)▫ α = 900

• d = 0 (at sensor, conjugate plane)▫ Mask attenuates all rays for the same x equally !▫ m(x, θ) = c (y = x)▫ α = 00

Mask as Modulator

• Optimal Mask Position

▫ In practice, since the spatial resolution is much larger than the angular resolution, is very small, and therefore the mask needs to be placed close to the sensor

• Optimal Mask Pattern

•Harmonic sine wave

•Boost

Notes

• 4D light field

• Aliasing▫ When band-limit assumption is not valid in the spatial

dimension, the energy in the higher spatial frequencies of the light field masquerade as energy in the lower angular dimension.

▫ Post-filter the recovered light field using a Kaiser-Bessel filter with a filer width of 1.5 [Ng 05]

Encoded Blur Camera

Mask as modulator

Assumption : layered Lambertian scene

Because no angular variations in the irradiance of rays from a Lambertian scene, the content of light field is restricted to be along the fx axis

∵

Optimal Mask for Encoding Defocus Blur

•Blurred image is linear convolution (circularly convolution with zero padded)▫Defocus by PSF (point spread function)

•Coded aperture remove SNR only special cases

+ Star , -Natural photography

- Optimal mask – continuous valued code by gradient decent optimization (Matlab, fmincon)- 7x7 Binary mask as initial guess

- 10 hours of search

Implementation & Analysis

Light Filed Camera

Mask & Sensor

Heterodyne Light Field Camera210 mm f/5.6Nikkor-W Lens

CanoScan LiDE 70scanner sensor

Mask80 dots/mm

Raw sensor image

Scene parts which are in-focus can be recovered at full resolution

Far Focused

Near Focused

In-focus – full resolution Out of focus Low resolution refocused image

Analysis

- Scanner sensor leading to pattern noise (horizontal /vertical lines)

+ Easy to cover over in a conventional digital camera with a finer mask placed inside in the future

• Computation

+ Computation burden is low because of computing light field and refocusing is done in Fourier domain

- Calibration of in-plane rotation and shift of the mask with respect to sensor

Failure Cases

• If Assumption of a band-limited light field is invalid, the aliasing artifacts in recovered light field

• 2D cosine mask needs to be moved away from the sensor because it results in diffraction

Encoded Blur Camera

Low resolution Mask

Encoded Blur Camera100 mm f/2.8 USM Macro Lens Mask Sensor

Canon Rebel XT SLR camera

ISO-12233 Chart

Modulation Transfer Function (MTF) of ISO-12233

MTF: low MTF: high

Full resolution digital refocusing using encoded blur camera

Captured photo

Refocused photo

In-focus fence + blurred person

Deblurring without taking the occluders

into account

Weighted deconvolution Eq.

Binary mask for the occluders

• In this case, we can recover the sharp image if the blur size is larger than the occluder size

• b is the vectorized blurred image

• A is the block-Toeplitz matrix representing 2D blur

• W is a weighting matrix which sets the weights corresponding to the occluded pixels in the blurred image to zero

Failure Cases

• Scenes with large variation in depths and those with view dependencies can not be handle

▫ Practice value

7x7 mask : blur size of about 20 pixels

• Finer resolution mask can handle large defocus blur but lead to diffraction blur

Contributions

• A theoretical framework of modulating 4D light fields camera working on frequency domain

• A new class of 4D light filed camera holds full resolution modulated 4D light field

• Don’t require additional optical elements such as lens arrays

• Analyze defocus blur as a special case of the frequency domain re-mapping and demonstrate that a broadband mask at aperture can preserve high spatial frequencies in defocused image

= rays x mask

Future Work

• Light Fields for Dynamic Scenes▫ Changing masks with time▫ Coding in time and space

• General Ray Modulators▫ Tilted/curved/multiple masks▫ Wavelength dependent masks▫ Angular/Spatial Resolution Tradeoff

• Applications▫ Estimating lens aberration▫ Microscopy▫ Light Field Applications

EndSource

http://www.umiacs.umd.edu/~aagrawal/sig07/MatlabCodeImages.html

study dappled photography

Technology