lecture 33: computational photography
DESCRIPTION
CS4670: Computer Vision. Noah Snavely. Lecture 33: Computational photography. Photometric stereo. Limitations. Big problems doesn’t work for shiny things, semi-translucent things shadows, inter-reflections Smaller problems camera and lights have to be distant calibration requirements - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/1.jpg)
Lecture 33: Computational photography
CS4670: Computer VisionNoah Snavely
![Page 2: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/2.jpg)
Photometric stereo
![Page 3: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/3.jpg)
LimitationsBig problems
• doesn’t work for shiny things, semi-translucent things• shadows, inter-reflections
Smaller problems• camera and lights have to be distant• calibration requirements
– measure light source directions, intensities
– camera response function
Newer work addresses some of these issues
Some pointers for further reading:• Zickler, Belhumeur, and Kriegman, "Helmholtz Stereopsis: Exploiting
Reciprocity for Surface Reconstruction." IJCV, Vol. 49 No. 2/3, pp 215-227. • Hertzmann & Seitz, “Example-Based Photometric Stereo: Shape
Reconstruction with General, Varying BRDFs.” IEEE Trans. PAMI 2005
![Page 4: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/4.jpg)
Finding the direction of the light source
P. Nillius and J.-O. Eklundh, “Automatic estimation of the projected light source direction,” CVPR 2001
![Page 5: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/5.jpg)
Application: Detecting composite photos
Fake photo
Real photo
Which is the real photo?
![Page 6: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/6.jpg)
The ultimate cameraWhat does it do?
![Page 7: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/7.jpg)
The ultimate cameraInfinite resolution
Infinite zoom control
Desired object(s) are in focus
No noise
No motion blur
Infinite dynamic range (can see dark and bright things)
...
![Page 8: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/8.jpg)
Creating the ultimate cameraThe “analog” camera has changed very little in >100 yrs
• we’re unlikely to get there following this path
More promising is to combine “analog” optics with computational techniques• “Computational cameras” or “Computational photography”
This lecture will survey techniques for producing higher quality images by combining optics and computation
Common themes:• take multiple photos• modify the camera
![Page 9: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/9.jpg)
Noise reductionTake several images and
average them
Why does this work?
Basic statistics: • variance of the mean
decreases with n:
![Page 10: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/10.jpg)
Field of viewWe can artificially increase the field of view by
compositing several photos together (project 2).
![Page 11: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/11.jpg)
Improving resolution: Gigapixel images
A few other notable examples:• Obama inauguration (gigapan.org)• HDView (Microsoft Research)
Max Lyons, 2003fused 196 telephoto shots
![Page 12: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/12.jpg)
Improving resolution: super resolutionWhat if you don’t have a zoom lens?
![Page 13: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/13.jpg)
D
For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (D), perfect reconstruction is possible.
D
Intuition (slides from Yossi Rubner & Miki Elad)
13
![Page 14: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/14.jpg)
Due to our limited camera resolution, we sample using an insufficient 2D grid
2D
2D
14
Intuition (slides from Yossi Rubner & Miki Elad)
![Page 15: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/15.jpg)
However, if we take a second picture, shifting the camera ‘slightly to the right’ we obtain:
2D
2D
15
Intuition (slides from Yossi Rubner & Miki Elad)
![Page 16: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/16.jpg)
Similarly, by shifting down we get a third image:
2D
2D
16
Intuition (slides from Yossi Rubner & Miki Elad)
![Page 17: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/17.jpg)
And finally, by shifting down and to the right we get the fourth image:
2D
2D
17
Intuition (slides from Yossi Rubner & Miki Elad)
![Page 18: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/18.jpg)
By combining all four images the desired resolution is obtained, and thus perfect reconstruction is guaranteed.
Intuition
18
![Page 19: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/19.jpg)
19
3:1 scale-up in each axis using 9 images, with pure global translation between them
Example
![Page 20: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/20.jpg)
Dynamic RangeTypical cameras have limited dynamic range
![Page 21: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/21.jpg)
HDR images — merge multiple inputs
Scene Radiance
Pixel count
![Page 22: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/22.jpg)
HDR images — merged
Radiance
Pixel count
![Page 23: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/23.jpg)
Camera is not a photometer!Limited dynamic range
• 8 bits captures only 2 orders of magnitude of light intensity• We can see ~10 orders of magnitude of light intensity
Unknown, nonlinear response • pixel intensity amount of light (# photons, or “radiance”)
Solution:• Recover response curve from multiple exposures, then
reconstruct the radiance map
![Page 24: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/24.jpg)
Camera response function
![Page 25: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/25.jpg)
Capture and composite several photosWorks for
• field of view• resolution• signal to noise• dynamic range• Focus
But sometimes you can do better by modifying the camera…
![Page 26: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/26.jpg)
FocusSuppose we want to produce images where the desired
object is guaranteed to be in focus?
Or suppose we want everything to be in focus?
![Page 27: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/27.jpg)
Light field camera [Ng et al., 2005]
http://www.refocusimaging.com/gallery/
![Page 28: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/28.jpg)
Conventional vs. light field camera
Conventional camera
Light field camera
![Page 29: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/29.jpg)
Light field camera
Rays are reorganized into many smaller images corresponding to subapertures of the main lens
![Page 30: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/30.jpg)
Prototype camera
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens
Contax medium format camera Kodak 16-megapixel sensor
Adaptive Optics microlens array 125μ square-sided microlenses
![Page 31: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/31.jpg)
![Page 32: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/32.jpg)
What can we do with the captured rays?Change viewpoint
![Page 33: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/33.jpg)
![Page 34: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/34.jpg)
Example of digital refocusing
![Page 35: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/35.jpg)
All-in-focus imagesCombines sharpest parts of all of the individual
refocused imagesUsing single pixel from each subimage
![Page 36: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/36.jpg)
All-in-focusIf you only want to produce an all-focus image, there
are simpler alternatives
E.g.,• Wavefront coding [Dowsky 1995]• Coded aperture [Levin SIGGRAPH 2007], [Raskar
SIGGRAPH 2007]– can also produce change in focus (ala Ng’s light field camera)
![Page 37: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/37.jpg)
Why are images blurry?
Camera focused at wrong distanceDepth of field
Motion blur
How can we remove the blur?
![Page 38: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/38.jpg)
Motion blurEspecially difficult to remove, because the blur kernel is
unknown
=
both unknown
![Page 39: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/39.jpg)
Multiple possible solutions
=
Blurry image
Sharp image Blur kernel
=
=
Slide courtesy Rob Fergus
![Page 40: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/40.jpg)
Priors can help
Priors on natural images
Image A is more “natural” than image B
![Page 41: Lecture 33: Computational photography](https://reader036.vdocuments.mx/reader036/viewer/2022062314/56814b85550346895db86a1f/html5/thumbnails/41.jpg)
Natural image statistics
Histogram of image gradients
Characteristic distribution with heavy tails