mit media lab camera culture image destabilization: programmable defocus using lens and sensor...
TRANSCRIPT
![Page 1: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/1.jpg)
MIT Media Lab Camera Culture
Image Destabilization:Programmable Defocus using
Lens and Sensor Motion
Ankit Mohan, Douglas Lanman,Shinsaku Hiura, Ramesh Raskar
MIT Media Lab
MIT Media Lab Camera Culture
![Page 2: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/2.jpg)
MIT Media Lab Camera Culture
Lots of glass; Heavy; Bulky; Expensive
Defocus Blur
![Page 3: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/3.jpg)
MIT Media Lab Camera Culture
Image Destabilization
Lens Sensor
Camera
Static Scene
![Page 4: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/4.jpg)
MIT Media Lab Camera Culture
Image Destabilization
Static Scene
Lens Motion Sensor Motion
Camera
![Page 5: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/5.jpg)
MIT Media Lab Camera Culture
Related Work
[Bae and Durand 2007]
[Hasinoff and Kutulakos 2007]
f8
f/4
f/2
f/1
extrapolated aperture
[Hiura et al. 2009][Vaish et al. 2004]
![Page 6: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/6.jpg)
MIT Media Lab Camera Culture
Laminography
X-Ray Source
X-Ray Sensor
Plane of focus
Related technique: Time Delay and Integration (TDI)
Motion direction
Motion direction
![Page 7: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/7.jpg)
MIT Media Lab Camera Culture
A
BA’
B’
Lens based Focusing
Lens Sensor
![Page 8: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/8.jpg)
MIT Media Lab Camera Culture
A
BA’
B’
Lens based Focusing
Lens Sensor
![Page 9: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/9.jpg)
MIT Media Lab Camera Culture
A
B
B’
A’
Lens
Smaller aperture Smaller defocus blur
Sensor
![Page 10: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/10.jpg)
MIT Media Lab Camera Culture
A
BA’
B’
Pinhole
Pinhole: All In-Focus
Sensor
![Page 11: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/11.jpg)
MIT Media Lab Camera Culture
A
B
A’
B’
Pinhole
Shifting Pinhole
Sensor
vp
![Page 12: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/12.jpg)
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
Shifting Pinhole
Sensor
vp
![Page 13: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/13.jpg)
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
Shifting Pinhole
Sensor
vp
![Page 14: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/14.jpg)
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
Shifting Pinhole
Sensor
vp
![Page 15: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/15.jpg)
MIT Media Lab Camera Culture
B
Pinhole Sensor
A’
B’
Shifting Pinhole
A
da
db ds
tp
vp
![Page 16: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/16.jpg)
MIT Media Lab Camera Culture
A
B
A’
B’
Pinhole
Shifting Pinhole and Sensor
vp
Sensor
vs
da
db ds
Focus Here
![Page 17: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/17.jpg)
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’vp
Sensor
vs
da
db ds
Focus Here
Shifting Pinhole and Sensor
![Page 18: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/18.jpg)
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
vp
Sensor
vs
da
db ds
Focus Here
Shifting Pinhole and Sensor
![Page 19: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/19.jpg)
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
vp
Sensor
vs
da
db ds
Focus Here
Shifting Pinhole and Sensor
![Page 20: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/20.jpg)
MIT Media Lab Camera Culture
A Lens in Time!
Analogous to shift and sum basedLight field re-focusing.
Lens Equation:
Virtual Focal Length:
Virtual F-Number:
![Page 21: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/21.jpg)
MIT Media Lab Camera Culture
Our Prototype
![Page 22: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/22.jpg)
MIT Media Lab Camera Culture
Adjusting the Focus Plane
all-in-focus pinhole image
![Page 23: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/23.jpg)
MIT Media Lab Camera Culture
Adjusting the Focus Plane
focused in the front using destabilization (10 second exposure)
![Page 24: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/24.jpg)
MIT Media Lab Camera Culture
Adjusting the Focus Plane
focused in the middle using destabilization (5 second exposure)
![Page 25: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/25.jpg)
MIT Media Lab Camera Culture
Adjusting the Focus Plane
focused in the back using destabilization (10 second exposure)
![Page 26: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/26.jpg)
MIT Media Lab Camera Culture
Adjusting the Virtual Aperture
focused in the middle using destabilization (5mm pinhole translation)
![Page 27: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/27.jpg)
MIT Media Lab Camera Culture
Adjusting the Virtual Aperture
focused in the middle using destabilization (30mm pinhole translation)
![Page 28: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/28.jpg)
MIT Media Lab Camera Culture
Shifting Lens and Sensor Defocus
• Physical vs. synthetic focus• Similar physical and synthetic focus
aperture
sensor
real focus
virtual focus
Defocus Exaggeration
![Page 29: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/29.jpg)
MIT Media Lab Camera Culture
Defocus Exaggeration
static lens with an f/2.8 aperture
![Page 30: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/30.jpg)
MIT Media Lab Camera Culture
Defocus Exaggeration
destabilization simulates a reduced f-number
![Page 31: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/31.jpg)
MIT Media Lab Camera Culture
Defocus Invariance
aperture
sensor
real focus
virtual focus
• Differing physical and synthetic focus
![Page 32: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/32.jpg)
MIT Media Lab Camera Culture
* =real PSF virtual PSF overall PSF
Defocus Invariance• Related work
– [Nagahara et al. 2008]– [Cathey and Dowski 1995]– [Levin et al. 2008]
• PSF not depth invariant– only size is depth invariant
• Gaussian special case– depth invariant PSF – inversion is ill-conditioned* =
real PSF virtual PSF overall PSF
[Nagahara et al. 2008]
![Page 33: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/33.jpg)
MIT Media Lab Camera Culture
Defocus Invariance
depth-invariant blur size (horizontal slit + destabilization)
![Page 34: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/34.jpg)
MIT Media Lab Camera Culture
Defocus Invariance
Richardson-Lucy deconvolution result
![Page 35: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/35.jpg)
MIT Media Lab Camera Culture
Tilted Sensor
C
DC′
D′
focus plane aperture plane sensor plane
Scheimpflug intersection
dC
dD
d′Cd′D
CC ddf 111
![Page 36: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/36.jpg)
MIT Media Lab Camera Culture
Tilted Sensor
C
D
vs?
focus plane aperture plane sensor plane
vp
CC ddf 111
dC
dD
![Page 37: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/37.jpg)
MIT Media Lab Camera Culture
Tilted Sensor
C
D
vp
C′ D′
vs
focus plane aperture plane sensor plane
dC
dD
d′Cd′D
s
p
DD
D
CC
C
v
v
dd
d
dd
d
![Page 38: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/38.jpg)
MIT Media Lab Camera Culture
Tuning the PSF
aperture
sensor
real focus
pinhole image (static f/22 aperture)
![Page 39: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/39.jpg)
MIT Media Lab Camera Culture
Tuning the PSF
real focus
aperture
sensor
large aperture image (static f/2.8 aperture)
![Page 40: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/40.jpg)
MIT Media Lab Camera Culture
Tuning the PSF
virtual focus
aperture
sensor
destabilized image using a pinhole (translated f/22 aperture)
![Page 41: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/41.jpg)
MIT Media Lab Camera Culture
Tuning the PSF
aperture
sensor
real focus
virtual focus
destabilized image using a large aperture (translated f/2.8 aperture)
![Page 42: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/42.jpg)
MIT Media Lab Camera Culture
Tuning the PSFreal focus
virtual focus
aperture
sensor
simulated aspheric lens using a vertical slit aperture and destabilization
![Page 43: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/43.jpg)
MIT Media Lab Camera Culture
Extension to 2D Displacements
ellipticallinear circular
“figure 8” hypocycloidal trispiral
![Page 44: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/44.jpg)
MIT Media Lab Camera Culture
Large apertures with tiny lenses?Benefits• No time or light inefficiency
wrt cheap cameras• Exploits unused area around
the lens• Compact design• With near-pinhole apertures
(mobile phones) many possibilities
Limitations• Coordinated mechanical
movement required• Diffraction (due to small aperture)
cannot be eliminated[Zhang and Levoy, tomorrow]
[Our group: augmented LF for wave analysis]
• Scene motion during exposure
![Page 45: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/45.jpg)
MIT Media Lab Camera Culture
Acknowledgements
Grace WooMIT CSAIL
Quinn SmithwickMIT Media Lab
Gabriel TaubinBrown University
Jaewon KimMIT Media Lab
MIT Media Lab: Camera Culture
![Page 46: MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh](https://reader036.vdocuments.mx/reader036/viewer/2022062619/5518a170550346881f8b48c2/html5/thumbnails/46.jpg)
MIT Media Lab Camera Culture
SLRs with tiny lenses?
• Analysis of space of relative lens/sensor displacement• Destabilization as virtual focusing mechanism• Shallower depth of field than physical aperture• Depth-independent defocus blur size