improving resolution and depth-of-field of light field cameras

91
Ashok Veeraraghavan, Ramesh Raskar, Amit Agrawal, Ankit Mohan & Jack Tumblin Mitsubishi Electric Research Labs (MERL), Cambridge, MA, USA Northwestern University, IL Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing

Upload: others

Post on 11-Sep-2021

7 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Improving Resolution and Depth-of-Field of Light Field Cameras

Ashok Veeraraghavan, Ramesh Raskar, Amit Agrawal,

Ankit Mohan & Jack Tumblin

Mitsubishi Electric Research Labs (MERL), Cambridge, MA, USA

Northwestern University, IL

Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing

Page 2: Improving Resolution and Depth-of-Field of Light Field Cameras

Coded Exposure [Raskar, Agrawal, Tumblin SIGGRAPH 2006]

Page 3: Improving Resolution and Depth-of-Field of Light Field Cameras

Coded Exposure (Flutter Shutter) Camera Raskar, Agrawal, Tumblin [Siggraph2006]

Coding in Time: Shutter is opened and closed

Page 4: Improving Resolution and Depth-of-Field of Light Field Cameras

Blurring == Convolution

Traditional Camera: Shutter is OPEN: Box Filter

PSF == Sinc Function

ω

Sharp Photo Blurred Photo

Page 5: Improving Resolution and Depth-of-Field of Light Field Cameras

Flutter Shutter: Shutter is OPEN and CLOSED

Preserves High Spatial Frequencies

Sharp Photo Blurred Photo

PSF == Broadband Function

Coded Exposure

Page 6: Improving Resolution and Depth-of-Field of Light Field Cameras

Traditional Coded Exposure

Image of Static Object

Deblurred Image

Deblurred Image

Page 7: Improving Resolution and Depth-of-Field of Light Field Cameras

How to handle focus blur?

Page 8: Improving Resolution and Depth-of-Field of Light Field Cameras

Coded Exposure Coded Aperture

Temporal 1-D broadband code: Motion Deblurring

Spatial 2-D broadband mask: Focus Deblurring

Page 9: Improving Resolution and Depth-of-Field of Light Field Cameras

In Focus Photo

Point light source (LED)

Page 10: Improving Resolution and Depth-of-Field of Light Field Cameras

Out of Focus Photo: Open Aperture

Page 11: Improving Resolution and Depth-of-Field of Light Field Cameras

Lens Camera

sensor

Point

spread

function

Image of a point

light source

Lens and defocus

Focal plane

Lens’ aperture

Slide Credit: Levin et. al

Page 12: Improving Resolution and Depth-of-Field of Light Field Cameras

Lens Object Camera

sensor

Point

spread

function

Image of a

defocused point

light source

Lens and defocus

Lens’ aperture

Focal plane

Slide Credit: Levin et. al

Page 13: Improving Resolution and Depth-of-Field of Light Field Cameras

Lens Camera

sensor

Point

spread

function

Image of a

defocused point

light source

Lens and defocus

Object

Lens’ aperture

Focal plane

Slide Credit: Levin et. al

Page 14: Improving Resolution and Depth-of-Field of Light Field Cameras

Lens Camera

sensor

Point

spread

function

Image of a

defocused point

light source

Lens’ aperture

Lens and defocus

Object

Focal plane

Slide Credit: Levin et. al

Page 15: Improving Resolution and Depth-of-Field of Light Field Cameras

Lens and defocus

Lens Camera

sensor

Point

spread

function

Image of a

defocused point

light source

Lens’ aperture

Object

Focal plane

Slide Credit: Levin et. al

Page 16: Improving Resolution and Depth-of-Field of Light Field Cameras

Out of Focus Photo: Coded Aperture

Page 17: Improving Resolution and Depth-of-Field of Light Field Cameras

Blurred Photos

Coded Aperture, 7 * 7 Mask Open Aperture

Page 18: Improving Resolution and Depth-of-Field of Light Field Cameras

Deblurred Photos

Coded Aperture, 7 * 7 Mask Open Aperture

Page 19: Improving Resolution and Depth-of-Field of Light Field Cameras

Captured Blurred Photo

Page 20: Improving Resolution and Depth-of-Field of Light Field Cameras

Full Resolution Digital Refocusing

Page 21: Improving Resolution and Depth-of-Field of Light Field Cameras

Blur Estimation & Segmentation

Captured Blurred Photo

Deblur at different blur sizes k

k = 10

k = 1

•Defocus blur dependent on depth •Assumptions

•Layered Lambertian Scene •Constant blur within each layer

Page 22: Improving Resolution and Depth-of-Field of Light Field Cameras

k = 10

k = 1

Deblurred Images

Define Cost Function

Cost Function Images

Likelihood Error: (Blurred image - Sharp Image * PSFk)2

Gradient Error: Natural Image Statistics, Gradient Kurtosis

k = 1

k = 10

Page 23: Improving Resolution and Depth-of-Field of Light Field Cameras

Blur Estimation & Segmentation == Labeling

• Graph cuts for labeling

Error Images K = 7

K = 1

k = 10

k = 1

Page 24: Improving Resolution and Depth-of-Field of Light Field Cameras

Captured

Photo

Deblur, k = 7

Reblur

Fusion

Page 25: Improving Resolution and Depth-of-Field of Light Field Cameras
Page 26: Improving Resolution and Depth-of-Field of Light Field Cameras

Less is More Blocking Light == More Information

Coded Exposure Coding in Time

Coded Aperture Coding in Space

Page 27: Improving Resolution and Depth-of-Field of Light Field Cameras

Flexible Depth of Field Photography

Nagahara, Kuthirammal, Zhou, and Nayar

ECCV 2008 Slide-deck credit: Nagahara et al.

Page 28: Improving Resolution and Depth-of-Field of Light Field Cameras
Page 29: Improving Resolution and Depth-of-Field of Light Field Cameras

Hardware Setup

Page 30: Improving Resolution and Depth-of-Field of Light Field Cameras

Captured Image

Aperture f/1.4, Exposure 0.36 sec

Page 31: Improving Resolution and Depth-of-Field of Light Field Cameras

Deblurred EDOF image

Page 32: Improving Resolution and Depth-of-Field of Light Field Cameras

Single “traditional” Image

Aperture f/1.4, Exposure 0.36 sec

Page 33: Improving Resolution and Depth-of-Field of Light Field Cameras

Single image with same EDOF

Aperture f/8, Exposure 0.36 sec

Page 34: Improving Resolution and Depth-of-Field of Light Field Cameras

Captured Image

Aperture f/1.4, Exposure 0.36 sec

Page 35: Improving Resolution and Depth-of-Field of Light Field Cameras

Deblurred EDOF image

Page 36: Improving Resolution and Depth-of-Field of Light Field Cameras

Single “traditional” Image

Aperture f/1.4, Exposure 0.36 sec

Page 37: Improving Resolution and Depth-of-Field of Light Field Cameras

Single image with same EDOF

Aperture f/8, Exposure 0.36 sec

Page 38: Improving Resolution and Depth-of-Field of Light Field Cameras

Tunable focus ring

Page 39: Improving Resolution and Depth-of-Field of Light Field Cameras

Discontinuous DOF

Page 40: Improving Resolution and Depth-of-Field of Light Field Cameras

Discontinuous DOF

Aperture f/11

Page 41: Improving Resolution and Depth-of-Field of Light Field Cameras

Discontinuous DOF

Aperture f/1.4

Page 42: Improving Resolution and Depth-of-Field of Light Field Cameras

Tilted DOF

Page 43: Improving Resolution and Depth-of-Field of Light Field Cameras

Image from normal camera

Aperture f/1.4

Page 44: Improving Resolution and Depth-of-Field of Light Field Cameras

Tilted DOF

Aperture f/1.4

Page 45: Improving Resolution and Depth-of-Field of Light Field Cameras

Non-planar DOF

Page 46: Improving Resolution and Depth-of-Field of Light Field Cameras

Image from a normal camera

Aperture f/1.4

Page 47: Improving Resolution and Depth-of-Field of Light Field Cameras

Non-planar DOF

Aperture f/1.4

Page 48: Improving Resolution and Depth-of-Field of Light Field Cameras

Multi-Aperture Photography Paul Green – MIT CSAIL

Wenyang Sun – MERL

Wojciech Matusik – MERL

Frédo Durand – MIT CSAIL

Page 49: Improving Resolution and Depth-of-Field of Light Field Cameras

Motivation

http://photographertips.net

Portrait Landscape

Small Aperture

Large Aperture

Depth of Field Control

Shallow Depth of Field

Large Depth of Field

Page 50: Improving Resolution and Depth-of-Field of Light Field Cameras

plane of focus

Depth and Defocus Blur

sensor lens

defocus blur depends on distance from plane of focus

subject

rays from point in focus converge to single pixel

circle of confusion

Page 51: Improving Resolution and Depth-of-Field of Light Field Cameras

Defocus Blur & Aperture

lens plane of focus

defocus blur depends on aperture size

aperture

http://photographertips.net

sensor

subject

circle of confusion

Page 52: Improving Resolution and Depth-of-Field of Light Field Cameras

Goals

Aperture size is a critical parameter for photographers

■ post-exposure depth of field control

■ extrapolate shallow depth of field beyond physical aperture

Page 53: Improving Resolution and Depth-of-Field of Light Field Cameras

Outline

Multi-Aperture Camera – New camera design

– Capture multiple aperture settings simultaneously

Applications – Depth of field control

– Depth of field extrapolation

– (Limited) refocusing

Page 54: Improving Resolution and Depth-of-Field of Light Field Cameras

Related Work Computational Cameras

– Plenoptic Cameras • Adelson and Wang ‘92 • Ng et al ‘05 • Georgiev et al ‘06

– Split-Aperture Camera • Aggarwal and Ahuja ‘04

– Optical Splitting Trees • McGuire et al ‘07

– Coded Aperture • Levin et al ’07 • Veeraraghavan et al ’07

– Wavefront Coding • Dowski and Cathey ‘95

Depth from Defocus – Pentland ‘87

Georgiev et al‘06

Aggarwal and Ahuja ‘04 McGuire et al ‘07

Adelson and Wang ‘92

Levin et al ’07 Veeraraghavan et al ’07

Page 55: Improving Resolution and Depth-of-Field of Light Field Cameras

Plenoptic Cameras

Capture 4D LightField – 2D Spatial (x,y)

– 2D Angular (u,v Aperture)

Trade resolution for flexibility after capture – Refocusing

– Depth of field control

– Improved Noise Characteristics

Lens Aperture

u

v

Sensor (x,y)

Lenslet Array

Subject

Lens (u,v)

Page 56: Improving Resolution and Depth-of-Field of Light Field Cameras

1D vs 2D Aperture Sampling

u

v

Aperture

2D Grid

Sampling http://photographertips.net

Page 57: Improving Resolution and Depth-of-Field of Light Field Cameras

4 Samples

u

v

Aperture

2D Grid

Sampling

1D vs. 2D Aperture Sampling

Aperture

1D “Ring”

Sampling

45 Samples

http://photographertips.net

Page 58: Improving Resolution and Depth-of-Field of Light Field Cameras

Optical Design Principles

Aperture

3D sampling – 2D spatial

– 1D aperture size

– 1 image for each “ring”

Sensor

http://photographertips.net

Page 59: Improving Resolution and Depth-of-Field of Light Field Cameras

Goal: Split aperture into 4 separate optical paths – concentric tilted mirrors

– at aperture plane

Aperture Splitting

Tilted Mirrors

Page 60: Improving Resolution and Depth-of-Field of Light Field Cameras

Aperture Splitting

Incoming light

Sensor

Mirrors Focusing lenses

Tilted Mirrors

Page 61: Improving Resolution and Depth-of-Field of Light Field Cameras

Aperture Splitting

Photographic Lens

Aperture Plane

Relay system Aperture splitting optics

New Aperture Plane

X

Ideally at aperture plane , but not physically possible! Solution: Relay Optics to create virtual aperture plane

Page 62: Improving Resolution and Depth-of-Field of Light Field Cameras

Optical Prototype

Mirror Close-up

main lens relay optics

mirrors

tilted mirrors

lenses SLR Camera

Page 63: Improving Resolution and Depth-of-Field of Light Field Cameras

Sample Data

Raw data from our camera

Page 64: Improving Resolution and Depth-of-Field of Light Field Cameras

Ideally would be rings

Gaps are from occlusion

Point Spread Function Occlusion combined inner ring 1 ring 2 outer

Page 65: Improving Resolution and Depth-of-Field of Light Field Cameras

Outline

Multi-Aperture Camera – New camera design

– Capture multiple aperture settings simultaneously

Applications – Depth of field control

– Depth of field extrapolation

– Refocusing

Page 66: Improving Resolution and Depth-of-Field of Light Field Cameras

DOF Navigation

0I 2I

1I 3I

Page 67: Improving Resolution and Depth-of-Field of Light Field Cameras

Approximate defocus blur as convolution

DOF Extrapolation?

0I

1I 2I 3I

? EI)(0 nn KII

)( nK - Circular aperture blurring kernel

Depends on depth and aperture size What is at each pixel in ? E

EI

Page 68: Improving Resolution and Depth-of-Field of Light Field Cameras

Blur s

ize

Aperture Diameter

Largest physical aperture

DOF Extrapolation Roadmap

capture estimate blur fit model extrapolate blur

I E

I 1

I 2

I 0

I 3

Page 69: Improving Resolution and Depth-of-Field of Light Field Cameras

Blur s

ize

Aperture Diameter D

I 1 I 2

I E

I 0

σ

I 3

Largest physical aperture

Defocus Gradient

Defocus blur

o

sos

fd

fddfdG

)(

o

sos

fd

fddfd )(DG

odsd

σ D

G is slope of this line

Defocus Gradient Map

Defocus Gradient

focal length

aperture diameter

sensor distance

object distance

Blur proportional to aperture diameter

Page 70: Improving Resolution and Depth-of-Field of Light Field Cameras

Optimization

solve for discrete defocus gradient values G at each pixel

Data term

Graph Cuts with spatial regularization term

i

Ni iGKIIGD )()( 1

0

Defocus Gradient Map Smallest Aperture Image

Page 71: Improving Resolution and Depth-of-Field of Light Field Cameras

Depth of Field Extrapolation

Page 72: Improving Resolution and Depth-of-Field of Light Field Cameras

Synthetic Refocusing

Modify gradient labels and re-synthesize image

gradient map “refocused” map extrapolated f/1.8 “refocused” synthetic f/1.8

Page 73: Improving Resolution and Depth-of-Field of Light Field Cameras

Discussion

■ Occlusion ■ Could help depth discrimination (coded aperture)

■ Difficult alignment process ■ Mostly because prototype

■ Refocusing limited by Depth of Field ■ helped by depth-guided deconvolution

■ Texture required for accurate defocus gradient map ■ Not critical for depth of field and refocus

Page 74: Improving Resolution and Depth-of-Field of Light Field Cameras

74

4D Frequency Analysis of Computational Cameras for

Depth of Field Extension

Anat Levin1,2 Sam Hasinoff 1 Paul Green1

Frédo Durand 1 Bill Freeman1

1MIT CSAIL 2Weizmann Institute

Page 75: Improving Resolution and Depth-of-Field of Light Field Cameras

75 Defocus blur in a standard lens

At focus depth,

sharp

Away from focus

depth, blurred

Page 76: Improving Resolution and Depth-of-Field of Light Field Cameras

76 Small aperture – increased depth of field but noisy

Depth 1: sharp but

noisy

Depth 2: sharp but

noisy

Page 77: Improving Resolution and Depth-of-Field of Light Field Cameras

77 Extended depth of field cameras

Modified optics

Extended DOF cameras: remove blur computationally

and design optics to make that easy

output input

Deconvolution

Page 78: Improving Resolution and Depth-of-Field of Light Field Cameras

78 In this talk

• How much can depth of field be extended?

• New lens extending depth of field

Page 79: Improving Resolution and Depth-of-Field of Light Field Cameras

Our design: assembly of subsquares with

different focal powers

each element focuses on a different depth

toy lattice-focal lens

with 4 elements

The lattice-focal lens 79

||)(E

,

3/13/4

3/82

,

yx

yxsS

A

Page 80: Improving Resolution and Depth-of-Field of Light Field Cameras

Proof of concept

• 12 subsquares cut

from plano-convex

spherical lenses

• Attached to main lens

extra focal power

needed very low

• Modest DOF

extension with only

12 subsquares

Hardware construction 80

Page 81: Improving Resolution and Depth-of-Field of Light Field Cameras

input depth map

• Defocus kernels vary with depth

• Depth estimation as for the coded aperture camera

[Levin et al. 07]

Depth estimation

defocus kernels at

different depths

81

Page 82: Improving Resolution and Depth-of-Field of Light Field Cameras

Standard lens reference 82

Page 83: Improving Resolution and Depth-of-Field of Light Field Cameras

Lattice-focal lens

Page 84: Improving Resolution and Depth-of-Field of Light Field Cameras

Standard lens reference 84

Page 85: Improving Resolution and Depth-of-Field of Light Field Cameras

Lattice-focal lens

Page 86: Improving Resolution and Depth-of-Field of Light Field Cameras

Standard lens reference 86

Page 87: Improving Resolution and Depth-of-Field of Light Field Cameras

Results Lattice-focal lens

Page 88: Improving Resolution and Depth-of-Field of Light Field Cameras

Application: Refocusing from single captured image 88

Page 89: Improving Resolution and Depth-of-Field of Light Field Cameras

89

Application: Refocusing from single captured image

Page 90: Improving Resolution and Depth-of-Field of Light Field Cameras

90

Application: Refocusing from single captured image

Page 91: Improving Resolution and Depth-of-Field of Light Field Cameras

The lattice-focal lens – limitations

• Depth estimation needed for deblurring

• Only capture part of the 4D light field

spectrum

• Subsquare size and focal power

optimized for a given focusing range

• Higher spectrum than previous designs,

but does not reach the upper bound

91