computational cameras - computer sciencelazebnik/research/fall08/rahul_raguram.pdfrahul raguram comp...
TRANSCRIPT
![Page 1: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/1.jpg)
Computational Cameras
Rahul RaguramCOMP 790-090
![Page 2: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/2.jpg)
What is a computational camera?Camera optics Camera
sensor
3D sceneFinal image
Traditional camera
Modified optics Camera sensor
3D sceneCoded image
Computational camera
Compute
Image
Additional information
![Page 3: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/3.jpg)
Computational cameras - examples
Catadioptric cameras
*
* Source: S. K. Nayar, 2006
![Page 4: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/4.jpg)
Computational cameras - examples
Catadioptric cameras HDR imaging with assorted pixels
* *
* Source: S. K. Nayar, 2006
![Page 5: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/5.jpg)
Computational cameras - examples
Catadioptric cameras HDR imaging with assorted pixels
Multiview radial cameras
Time-of-flight cameras
* *
* #
* Source: S. K. Nayar, 2006
# Source: L. Guan and M. Pollefeys, 2008
![Page 6: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/6.jpg)
The aperture
• Diameter of the lens opening (controlled by diaphragm)
• Expressed as a fraction of focal length (f-number)– f/2.0 with a 50mm lens: aperture is 25mm– f/2.0 with a 100mm lens: aperture is 50mm
• Typical f-numbers: f/1.4, f/2, f/2.8, f/4, f/5.6, f/8…– see a pattern?
Glossographia Anglicana Nova,1707
![Page 7: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/7.jpg)
Varying the aperture
Small aperture – large depth of field
![Page 8: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/8.jpg)
Varying the aperture
Large aperture – small depth of field
Bokeh (derived from Japanese boke ぼけ, a noun form of bokeru ぼける, "become blurred or fuzzy")
![Page 9: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/9.jpg)
Multi-Aperture Photography
Paul Green – MIT CSAILWenyang Sun – MERLWojciech Matusik – MERLFrédo Durand – MIT CSAIL
Slides by Green et al.
![Page 10: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/10.jpg)
Motivation
http://photographertips.net
Portrait Landscape
Small Aperture
Large Aperture
Depth of Field Control
Shallow Depth of Field
Large Depth of Field
![Page 11: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/11.jpg)
plane of focus
Depth and Defocus Blur
sensor lens
defocus blur depends on distance from plane of focus
subject
rays from point in focus converge to single pixel
circle of confusion
![Page 12: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/12.jpg)
Defocus Blur & Aperture
lens plane of focus
defocus blur depends on aperture size
aperture
http://photographertips.net
sensor
subject
circle of confusion
![Page 13: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/13.jpg)
GoalsAperture size is a critical
parameter for photographers
■ post-exposure depth of field control
■ extrapolate shallow depth of field beyond physical aperture
![Page 14: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/14.jpg)
OutlineMulti-Aperture Camera
■ New camera design■ Capture multiple aperture
settings simultaneously
Applications■ Depth of field control■ Depth of field extrapolation■ (Limited) refocusing
![Page 15: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/15.jpg)
Related WorkComputational Cameras
■ Plenoptic Cameras■ Adelson and Wang ‘92■ Ng et al ‘05■ Georgiev et al ‘06
■ Split-Aperture Camera■ Aggarwal and Ahuja ‘04
■ Optical Splitting Trees■ McGuire et al ‘07
■ Coded Aperture■ Levin et al ’07■ Veeraraghavan et al ’07
■ Wavefront Coding■ Dowski and Cathey ‘95
Depth from Defocus■ Pentland ‘87
Georgiev et al‘06
Aggarwal and Ahuja ‘04McGuire et al ‘07
Adelson and Wang ‘92
Levin et al ’07 Veeraraghavan et al ’07
![Page 16: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/16.jpg)
Plenoptic CamerasCapture 4D LightField
■ 2D Spatial (x,y)■ 2D Angular (u,v Aperture)
Trade resolution for flexibility after capture■ Refocusing■ Depth of field control■ Improved Noise
Characteristics
Lens Aperture
u
v
Sensor (x,y)
LensletArray
Subject
Lens (u,v)
![Page 17: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/17.jpg)
1D vs 2D Aperture Sampling
u
v
Aperture
2D Grid Sampling http://photographertips.net
![Page 18: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/18.jpg)
4 Samples4 Samples
u
v
Aperture
2D Grid Sampling
1D vs. 2D Aperture Sampling
Aperture
1D “Ring”Sampling
45 Samples45 Samples
http://photographertips.net
![Page 19: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/19.jpg)
Goals■ post-exposure
depth of field control■ extrapolate shallow
depth of field■ (limited) refocusing
■ 1d sampling■ no beamsplitters■ single sensor■ removable
![Page 20: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/20.jpg)
Optical Design Principles
Aperture
3D sampling■ 2D spatial■ 1D aperture size■ 1 image for
each “ring”
Sensor
http://photographertips.net
![Page 21: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/21.jpg)
Aperture Splitting
Incoming light
Sensor
MirrorsFocusing lenses
Tilted Mirrors
![Page 22: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/22.jpg)
Aperture Splitting
Photographic Lens
Aperture Plane
Relay system
Aperture splitting optics
New Aperture Plane
Ideally at aperture plane, but not physically possible!Solution: Relay Optics to create virtual aperture plane
![Page 23: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/23.jpg)
Optical Prototype
Mirror Close-up
main lens relay optics
mirrors
tilted mirrors
lenses SLR Camera
![Page 24: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/24.jpg)
Sample DataRaw data from our camera
![Page 25: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/25.jpg)
Ideally would be ringsGaps are from occlusion
Point Spread Function Occlusioncombinedinner ring 1 ring 2 outer
![Page 26: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/26.jpg)
OutlineMulti-Aperture Camera
■ New camera design■ Capture multiple aperture
settings simultaneously
Applications■ Depth of field control■ Depth of field extrapolation■ Refocusing
![Page 27: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/27.jpg)
DOF Navigation
0I 2I
1I 3I
![Page 28: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/28.jpg)
Approximate defocus blur as convolution
DOF Extrapolation?
0I 1I 2I 3I
?EI)(0 nn KII σ∗=
)( nK σ - Circular aperture blurring kernel
Depends on depth and aperture size
![Page 29: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/29.jpg)
Blu
r siz
e
Aperture Diameter
Largest physical aperture
DOF Extrapolation Roadmap
capture estimate blur fit model extrapolate
blur
IIEE
I1
I2I0
I3
![Page 30: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/30.jpg)
Summary■Multi-aperture camera
■1D sampling of aperture■Removable
■Post-Exposure depth of field control
■Depth of field extrapolation
![Page 31: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/31.jpg)
Image and Depth from a Conventional Camera with a Coded Aperture
Anat Levin, Rob Fergus, Frédo Durand, William Freeman
MIT CSAIL
Slides by Levin et al.
![Page 32: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/32.jpg)
Single input image:
Output #1: Depth map
![Page 33: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/33.jpg)
Single input image:
Output #1: Depth map
Output #2: All-focused image
![Page 34: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/34.jpg)
Single input image:
Output #1: Depth map
Output #2: All-focused image
![Page 35: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/35.jpg)
![Page 36: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/36.jpg)
![Page 37: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/37.jpg)
Lens Camera sensor
Point spread
function
Image of a point light source
Lens and defocus
Focal plane
Lens’ aperture
![Page 38: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/38.jpg)
LensObject Camera sensor
Point spread
function
Image of a defocused point
light source
Lens and defocus
Lens’ aperture
Focal plane
![Page 39: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/39.jpg)
Lens Camera sensor
Point spread
function
Image of a defocused point
light source
Lens and defocus
Object
Lens’ aperture
Focal plane
![Page 40: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/40.jpg)
Lens Camera sensor
Point spread
function
Image of a defocused point
light sourceLens’ aperture
Lens and defocus
Object
Focal plane
![Page 41: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/41.jpg)
Lens and defocus
Lens Camera sensor
Point spread
function
Image of a defocused point
light sourceLens’ aperture
Object
Focal plane
![Page 42: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/42.jpg)
Depth and defocus
Depth from defocus:Infer depth by analyzing local scale of defocus blur
Out of focus
In focus
![Page 43: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/43.jpg)
Challenges
• Hard to discriminate a smooth scene from defocus blur
• Hard to undo defocus blur
Input Ringing with conventional deblurring algorithm
Out of focus?
![Page 44: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/44.jpg)
Key contributions• Exploit prior on natural images
- Improve deconvolution
- Improve depth discrimination
• Coded aperture (mask inside lens)- make defocus patterns different from
natural images and easier to discriminate
Natural Unnatural
![Page 45: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/45.jpg)
Defocus as local convolution
Input defocused image
Calibrated blur kernels at different depths
![Page 46: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/46.jpg)
xfy k ⊗=
xfy k ⊗=
xfy k ⊗=
Defocus as local convolution
xfy k ⊗= xf k
Depth k=1:
Depth k=2:
Depth k=3:
Input defocused image
Local sub-window
Calibrated blur kernels
at depth
Sharp sub-window
k
y
![Page 47: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/47.jpg)
Overview
⊗= Correct scale
Smaller scale
Larger scale
⊗= ⊗=
Try deconvolving local input windows with different scaled filters:
Somehow: select best scale.
??????
![Page 48: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/48.jpg)
Challenges
• Hard to identify correct scale:
• Hard to deconvolve even when kernel is known
Input Ringing with the traditional Richardson-Lucy deconvolution
algorithm
⊗=
?? Correct scale
Smaller scale
? Larger scale
⊗= ⊗=
• Hard to deconvolve even when kernel is known
![Page 49: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/49.jpg)
?
yxf =⊗
=
Deconvolution is ill posed
⊗
![Page 50: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/50.jpg)
Deconvolution is ill posed
? =
=?
Solution 1:
Solution 2:
yxf =⊗
⊗
⊗
![Page 51: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/51.jpg)
Idea 1: Natural images prior
Image
gradient
put a penalty on gradients
Natural images have sparse gradients
Natural Unnatural
What makes images special?
![Page 52: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/52.jpg)
Deconvolution with prior
2|| minarg yxfx −⊗=
⊗ _
∑ ∇+i ix )(ρλ
2
+
⊗ _ +2
?
?
Convolution error Derivatives prior
High
Low Equal convolution error
![Page 53: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/53.jpg)
Comparing deconvolution algorithms
Input
Richardson-Lucy
(Non blind) deconvolution code available online:http://groups.csail.mit.edu/graphics/CodedAperture/
Gaussian prior
2)( xx ∇=∇ρ“spread” gradients
Sparse prior
“localizes” gradients
8.0)( xx ∇=∇ρ
![Page 54: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/54.jpg)
Comparing deconvolution algorithms
Input
Richardson-Lucy
(Non blind) deconvolution code available online:http://groups.csail.mit.edu/graphics/CodedAperture/
Gaussian prior
“spread” gradients
Sparse prior
“localizes” gradients
2)( xx ∇=∇ρ 8.0)( xx ∇=∇ρ
![Page 55: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/55.jpg)
⊗=
??Correct scale
Smaller scale
?Larger scale
⊗= ⊗=
Try deconvolving local input windows with different scaled filters:
Recall: Overview
Challenge: smaller scale not so different than correct
Somehow: select best scale.
![Page 56: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/56.jpg)
Idea 2: Coded Aperture
• Mask (code) in aperture plane- make defocus patterns different from
natural images and easier to discriminate
Conventional aperture
Our coded aperture
![Page 57: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/57.jpg)
Lens Camera sensor
Point spread
function
Object
Solution: lens with occluder
Focal plane
![Page 58: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/58.jpg)
Solution: lens with occluder
Lens with coded aperture
Camera sensor
Point spread
function
Image of a defocused point
light sourceAperture pattern
Object
Focal plane
![Page 59: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/59.jpg)
Lens with coded aperture
Camera sensor
Point spread
function
Image of a defocused point
light sourceAperture pattern
Solution: lens with occluder
Object
Focal plane
![Page 60: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/60.jpg)
Lens with coded aperture
Camera sensor
Point spread
function
Image of a defocused point
light sourceAperture pattern
Solution: lens with occluder
Object
Focal plane
![Page 61: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/61.jpg)
Lens with coded aperture
Camera sensor
Point spread
function
Image of a defocused point
light sourceAperture pattern
Solution: lens with occluder
Object
Focal plane
![Page 62: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/62.jpg)
Lens with coded aperture
Camera sensor
Point spread
function
Image of a defocused point
light sourceAperture pattern
Solution: lens with occluder
Object
Focal plane
![Page 63: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/63.jpg)
Why coded?
Conventional Coded
Coded aperture- reduce uncertainty in scale identification
Correct scale
Smaller scale
Larger scale
![Page 64: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/64.jpg)
Depth results
![Page 65: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/65.jpg)
Input Local depth estimation Regularized depth
Regularizing depth estimation
2|| minarg yxfx −⊗=
_
∑ ∇+i ix )(ρλ
2+
Convolution error Derivatives prior
⊗
Try deblurring with 10 different aperture scales
Keep minimal error scale in each local window + regularization
305
295
285
275
265
255
245
235
200
![Page 66: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/66.jpg)
Input
Local depth estimation
Regularized depth
Regularizing depth estimation
305
295
285
275
265
255
245
235
200
305
295
285
275
265
255
245
235
200
![Page 67: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/67.jpg)
Sometimes, manual intervention
305
295
285
275
265
255
245
235
200
305
295
285
275
265
255
245
235
305
295
285
275
265
255
245
235
Input
After user correctionsRegularized depth
Local depth estimation
![Page 68: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/68.jpg)
All focused results
![Page 69: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/69.jpg)
Input
![Page 70: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/70.jpg)
All-focused (deconvolved)
![Page 71: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/71.jpg)
Original image
All-focus image
Close-up
![Page 72: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/72.jpg)
Input
![Page 73: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/73.jpg)
All-focused (deconvolved)
![Page 74: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/74.jpg)
Original image All-focus image
Close-up
Naïve sharpening
![Page 75: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/75.jpg)
Comparison- conventional aperture result
Ringing due to wrong scale estimation
![Page 76: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/76.jpg)
Comparison- coded aperture result
![Page 77: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/77.jpg)
Application: Digital refocusing from a single image
![Page 78: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/78.jpg)
Application: Digital refocusing from a single image
![Page 79: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/79.jpg)
Application: Digital refocusing from a single image
![Page 80: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/80.jpg)
Application: Digital refocusing from a single image
![Page 81: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/81.jpg)
Application: Digital refocusing from a single image
![Page 82: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/82.jpg)
Application: Digital refocusing from a single image
![Page 83: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/83.jpg)
Application: Digital refocusing from a single image
![Page 84: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/84.jpg)
Image AND depth at a single shot
No loss of image resolution
Simple modification to lens
Depth is coarse
But depth is a pure bonus
Loss some light
But deconvolution increases depth of field
Coded aperture: pros and cons
unable to get depth at untextured areas, might need manual corrections.
-++
+-+
+
![Page 85: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/85.jpg)
http://groups.csail.mit.edu/graphics/CodedAperture/
Deconvolution code available
![Page 86: Computational Cameras - Computer Sciencelazebnik/research/fall08/rahul_raguram.pdfRahul Raguram COMP 790-090. What is a computational camera? Camera optics Camera sensor 3D scene Final](https://reader034.vdocuments.mx/reader034/viewer/2022043021/5f3d5d56158b3d73cd267f2f/html5/thumbnails/86.jpg)
Depth acquisition: priceless
$1Cardboard:$79.9550mm f/1.8:
Tape: $1