improving resolution and depth-of-field of light field cameras

Ashok Veeraraghavan, Ramesh Raskar, Amit Agrawal,

Ankit Mohan & Jack Tumblin

Mitsubishi Electric Research Labs (MERL), Cambridge, MA, USA

Northwestern University, IL

Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing

Coded Exposure [Raskar, Agrawal, Tumblin SIGGRAPH 2006]

Coded Exposure (Flutter Shutter) Camera Raskar, Agrawal, Tumblin [Siggraph2006]

Coding in Time: Shutter is opened and closed

Blurring == Convolution

Traditional Camera: Shutter is OPEN: Box Filter

PSF == Sinc Function

ω

Sharp Photo Blurred Photo

Flutter Shutter: Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo Blurred Photo

PSF == Broadband Function

Coded Exposure

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

How to handle focus blur?

Coded Exposure Coded Aperture

Temporal 1-D broadband code: Motion Deblurring

Spatial 2-D broadband mask: Focus Deblurring

In Focus Photo

Point light source (LED)

Out of Focus Photo: Open Aperture

Lens Camera

sensor

Point

spread

function

Image of a point

light source

Lens and defocus

Focal plane

Lens’ aperture

Slide Credit: Levin et. al

Lens Object Camera

sensor

Point

spread

function

Image of a

defocused point

light source

Lens and defocus

Lens’ aperture

Focal plane

Slide Credit: Levin et. al

Lens Camera

sensor

Point

spread

function

Image of a

defocused point

light source

Lens and defocus

Object

Lens’ aperture

Focal plane

Slide Credit: Levin et. al

Lens Camera

sensor

Point

spread

function

Image of a

defocused point

light source

Lens’ aperture

Lens and defocus

Object

Focal plane

Slide Credit: Levin et. al

Lens and defocus

Lens Camera

sensor

Point

spread

function

Image of a

defocused point

light source

Lens’ aperture

Object

Focal plane

Slide Credit: Levin et. al

Out of Focus Photo: Coded Aperture

Blurred Photos

Coded Aperture, 7 * 7 Mask Open Aperture

Deblurred Photos

Coded Aperture, 7 * 7 Mask Open Aperture

Captured Blurred Photo

Full Resolution Digital Refocusing

Blur Estimation & Segmentation

Captured Blurred Photo

Deblur at different blur sizes k

k = 10

k = 1

•Defocus blur dependent on depth •Assumptions

•Layered Lambertian Scene •Constant blur within each layer

k = 10

k = 1

Deblurred Images

Define Cost Function

Cost Function Images

Likelihood Error: (Blurred image - Sharp Image * PSFk)2

Gradient Error: Natural Image Statistics, Gradient Kurtosis

k = 1

k = 10

Blur Estimation & Segmentation == Labeling

• Graph cuts for labeling

Error Images K = 7

K = 1

k = 10

k = 1

Captured

Photo

Deblur, k = 7

Reblur

Fusion

Less is More Blocking Light == More Information

Coded Exposure Coding in Time

Coded Aperture Coding in Space

Flexible Depth of Field Photography

Nagahara, Kuthirammal, Zhou, and Nayar

ECCV 2008 Slide-deck credit: Nagahara et al.

Hardware Setup

Captured Image

Aperture f/1.4, Exposure 0.36 sec

Deblurred EDOF image

Single “traditional” Image

Aperture f/1.4, Exposure 0.36 sec

Single image with same EDOF

Aperture f/8, Exposure 0.36 sec

Captured Image

Aperture f/1.4, Exposure 0.36 sec

Deblurred EDOF image

Single “traditional” Image

Aperture f/1.4, Exposure 0.36 sec

Single image with same EDOF

Aperture f/8, Exposure 0.36 sec

Tunable focus ring

Discontinuous DOF

Discontinuous DOF

Aperture f/11

Discontinuous DOF

Aperture f/1.4

Tilted DOF

Image from normal camera

Aperture f/1.4

Tilted DOF

Aperture f/1.4

Non-planar DOF

Image from a normal camera

Aperture f/1.4

Non-planar DOF

Aperture f/1.4

Multi-Aperture Photography Paul Green – MIT CSAIL

Wenyang Sun – MERL

Wojciech Matusik – MERL

Frédo Durand – MIT CSAIL

Motivation

http://photographertips.net

Portrait Landscape

Small Aperture

Large Aperture

Depth of Field Control

Shallow Depth of Field

Large Depth of Field

plane of focus

Depth and Defocus Blur

sensor lens

defocus blur depends on distance from plane of focus

subject

rays from point in focus converge to single pixel

circle of confusion

Defocus Blur & Aperture

lens plane of focus

defocus blur depends on aperture size

aperture

http://photographertips.net

sensor

subject

circle of confusion

Goals

Aperture size is a critical parameter for photographers

■ post-exposure depth of field control

■ extrapolate shallow depth of field beyond physical aperture

Outline

Multi-Aperture Camera – New camera design

– Capture multiple aperture settings simultaneously

Applications – Depth of field control

– Depth of field extrapolation

– (Limited) refocusing

Related Work Computational Cameras

– Plenoptic Cameras • Adelson and Wang ‘92 • Ng et al ‘05 • Georgiev et al ‘06

– Split-Aperture Camera • Aggarwal and Ahuja ‘04

– Optical Splitting Trees • McGuire et al ‘07

– Coded Aperture • Levin et al ’07 • Veeraraghavan et al ’07

– Wavefront Coding • Dowski and Cathey ‘95

Depth from Defocus – Pentland ‘87

Georgiev et al‘06

Aggarwal and Ahuja ‘04 McGuire et al ‘07

Adelson and Wang ‘92

Levin et al ’07 Veeraraghavan et al ’07

Plenoptic Cameras

Capture 4D LightField – 2D Spatial (x,y)

– 2D Angular (u,v Aperture)

Trade resolution for flexibility after capture – Refocusing

– Depth of field control

– Improved Noise Characteristics

Lens Aperture

u

v

Sensor (x,y)

Lenslet Array

Subject

Lens (u,v)

1D vs 2D Aperture Sampling

u

v

Aperture

2D Grid

Sampling http://photographertips.net

4 Samples

u

v

Aperture

2D Grid

Sampling

1D vs. 2D Aperture Sampling

Aperture

1D “Ring”

Sampling

45 Samples

http://photographertips.net

Optical Design Principles

Aperture

3D sampling – 2D spatial

– 1D aperture size

– 1 image for each “ring”

Sensor

http://photographertips.net

Goal: Split aperture into 4 separate optical paths – concentric tilted mirrors

– at aperture plane

Aperture Splitting

Tilted Mirrors

Aperture Splitting

Incoming light

Sensor

Mirrors Focusing lenses

Tilted Mirrors

Aperture Splitting

Photographic Lens

Aperture Plane

Relay system Aperture splitting optics

New Aperture Plane

X

Ideally at aperture plane , but not physically possible! Solution: Relay Optics to create virtual aperture plane

Optical Prototype

Mirror Close-up

main lens relay optics

mirrors

tilted mirrors

lenses SLR Camera

Sample Data

Raw data from our camera

Ideally would be rings

Gaps are from occlusion

Point Spread Function Occlusion combined inner ring 1 ring 2 outer

Outline

Multi-Aperture Camera – New camera design

– Capture multiple aperture settings simultaneously

Applications – Depth of field control

– Depth of field extrapolation

– Refocusing

DOF Navigation

0I 2I

1I 3I

Approximate defocus blur as convolution

DOF Extrapolation?

0I

1I 2I 3I

? EI)(0 nn KII

)( nK - Circular aperture blurring kernel

Depends on depth and aperture size What is at each pixel in ? E

EI

Blur s

ize

Aperture Diameter

Largest physical aperture

DOF Extrapolation Roadmap

capture estimate blur fit model extrapolate blur

I E

I 1

I 2

I 0

I 3

Blur s

ize

Aperture Diameter D

I 1 I 2

I E

I 0

σ

I 3

Largest physical aperture

Defocus Gradient

Defocus blur

o

sos

fd

fddfdG

)(

o

sos

fd

fddfd )(DG

odsd

σ D

G is slope of this line

Defocus Gradient Map

Defocus Gradient

focal length

aperture diameter

sensor distance

object distance

Blur proportional to aperture diameter

Optimization

solve for discrete defocus gradient values G at each pixel

Data term

Graph Cuts with spatial regularization term

i

Ni iGKIIGD )()( 1

0

Defocus Gradient Map Smallest Aperture Image

Depth of Field Extrapolation

Synthetic Refocusing

Modify gradient labels and re-synthesize image

gradient map “refocused” map extrapolated f/1.8 “refocused” synthetic f/1.8

Discussion

■ Occlusion ■ Could help depth discrimination (coded aperture)

■ Difficult alignment process ■ Mostly because prototype

■ Refocusing limited by Depth of Field ■ helped by depth-guided deconvolution

■ Texture required for accurate defocus gradient map ■ Not critical for depth of field and refocus

74

4D Frequency Analysis of Computational Cameras for

Depth of Field Extension

Anat Levin1,2 Sam Hasinoff 1 Paul Green1

Frédo Durand 1 Bill Freeman1

1MIT CSAIL 2Weizmann Institute

75 Defocus blur in a standard lens

At focus depth,

sharp

Away from focus

depth, blurred

76 Small aperture – increased depth of field but noisy

Depth 1: sharp but

noisy

Depth 2: sharp but

noisy

77 Extended depth of field cameras

Modified optics

Extended DOF cameras: remove blur computationally

and design optics to make that easy

output input

Deconvolution

78 In this talk

• How much can depth of field be extended?

• New lens extending depth of field

Our design: assembly of subsquares with

different focal powers

each element focuses on a different depth

toy lattice-focal lens

with 4 elements

The lattice-focal lens 79

||)(E

,

3/13/4

3/82

,

yx

yxsS

A

Proof of concept

• 12 subsquares cut

from plano-convex

spherical lenses

• Attached to main lens

extra focal power

needed very low

• Modest DOF

extension with only

12 subsquares

Hardware construction 80

input depth map

• Defocus kernels vary with depth

• Depth estimation as for the coded aperture camera

[Levin et al. 07]

Depth estimation

defocus kernels at

different depths

81

Standard lens reference 82

Lattice-focal lens

Standard lens reference 84

Lattice-focal lens

Standard lens reference 86

Results Lattice-focal lens

Application: Refocusing from single captured image 88

89

Application: Refocusing from single captured image

90

Application: Refocusing from single captured image

The lattice-focal lens – limitations

• Depth estimation needed for deblurring

• Only capture part of the 4D light field

spectrum

• Subsquare size and focal power

optimized for a given focusing range

• Higher spectrum than previous designs,

but does not reach the upper bound

91