1

Computational Photography

Matthias ZwickerUniversity of Bern

Fall 2009

Today• Course organization

• Course overview

• Image formation

Course organizationInstructor

• Matthias Zwicker (zwicker@iam.unibe.ch)

Teaching Assistant

• To be announcedTo be announced

Course organizationLecture

• Mondays, 14:00-16:00, Engehaldenstr. 8, Hörsaal 001

Exercises

• Mondays, 16:00-17:00 , Engehaldenstr. 8, Hörsaal 001

Class web page• Schedule, slides, reading, project

descriptions, etc.http://www.cgg.unibe.ch/teaching/courses/herbstsemester-2009/computational-photography/

Web-based forum• On ILIAS

https://ilias.unibe.ch/ilias3/repository.php?cmd=frameset&ref_id=66993

• Use your campus account to log in

• Join group “IAM Computational Photography” with password Photography with password “cggcompphoto”

• Any questions and discussions related to class material and exercises

2

Exercises• Programming projects

– Matlab• Exercise series on paper

• Contribute 50% to final grade

• Late penalty• Late penalty– 50% of original score– Exceptions for military service, illness

• Collaboration– Discussion among student is encouraged– Each student must write up his/her own solution

Prerequisites• Familiarity with

– Linear algebra– Programming

Today• Course organization

• Course overview

• Image formation

PhotographyTraditionally

• „Measuring light“

• Optics focuses light on sensor

• Sensor records image

• Sensors

– Digital– Film

http://en.wikipedia.org/wiki/Single-lens_reflex_camera http://en.wikipedia.org/wiki/Digital_single-lens_reflex_camera

Computational photography• More than digital photography

• Arbitrary computation between light measurement and final image– Enhance and extend capabilities of digital

photographyh d f l – Light measured on sensor is not final image

• Two „types“ of computation– Post-process after traditional imaging– Design of new camera devices that require

computation to form an image• Overview of recent research

http://en.wikipedia.org/wiki/Computational_photography

Removing imaging artifacts• Denoising & deblurring

http://www.cs.ust.hk/~quan/publications/yuan-deblur-siggraph07.pdf

Blurry OutputNoisy

3

Removing imaging artifacts• High dynamic range images & tone mapping

Image manipulation• Panoramas

http://en.wikipedia.org/wiki/Image_stitching

Computational optics

Coded aperture

Captured image

Computational optics

Recovered depth

Refocused image

Focus of class• Algorithms and computational techniques

with potential applications in the consumer domain

– Mostly software, less hardware

• Recent research• Recent research

What you will learn• Basic understanding of photography, light, and

color

• Practical experience with implementation of algorithms for image processing & computational photography

• Cool and creative applications of mathematical Cool and creative applications of mathematical tools– Fourier transforms– Linear and non linear filtering– Optimization techniques (least squares, iteratively re-

weighted least squares, graph cuts)– Probabilistic models

• Lots of applications beyond processing images!

4

Related areas, not covered• Image processing for scientific applications

– Physics, biology, etc.

• Optics, lens design

• Sensors, sensor design

• Computational imaging

– Tomography, radar, microscopy

• 3D imaging

• Using photo processing tools, e.g. Photoshop

• Artistical aspects of photography

Syllabus1. Introduction, image formation

2. Color & color processing3. Dynamic range & contrast4. Sampling, reconstruction, & the frequency domain5. Image restoration: denoising & deblurring6. Image manipulation using optimization7. Gradient domain image manipulation7. Gradient domain image manipulation

8. Warping & morphing9. Panoramas10. Automatic alignment11. Probabilistic image models12. Light fields13. Capturing light transport

http://www.cgg.unibe.ch/teaching/courses/herbstsemester-2009/computational-photography

• Cameras, image artifacts

Image formation Color• Color perception, color spaces, color

measurement, color processing

Dynamic range & contrast• HDR imaging

http://en.wikipedia.org/wiki/High_dynamic_range_imaginghttp://en.wikipedia.org/wiki/Tone_mapping

Sampling, reconstruction• Sampling artifacts

• Frequency domainanalysis

Spatial Domain Frequency Domain

5

Image restoration• Denoising & deblurring

Blurry input Deblurred output

Estimated blur kernel (scaled)http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/

Image manipulation using optimization• Photomontage, matting, colorization

http://grail.cs.washington.edu/projects/photomontage/

http://www.cs.huji.ac.il/~yweiss/Colorization/http://grail.cs.washington.edu/projects/digital-matting/image-matting/

Gradient domain manipulation• Poisson equation

http://portal.acm.org/citation.cfm?id=882269

Warping & morphing

Panoramas• Automatic alignment, stitching

http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj4/www/wwedler/

Probabilistic models• Faces, textures

http://web4.cs.ucl.ac.uk/staff/j.kautz/publications/Visio_SIG09.pdf

6

• Beyond 2D images

Light fields

http://www-graphics.stanford.edu/papers/fourierphoto/

Capturing light transport• Dual photography

http://www-graphics.stanford.edu/papers/dual_photography/

Today• Course organization

• Course overview

• Image formation

Question• Why is there no image on a white piece of

paper?

Question• Why is there no image on a white piece of

paper?

• Receives all light rays

– Images from all viewpoints

• Need to select lightrays for specificeimage, viewpoint

• How?

• Invented by Alhazen, 10th centuryhttp://en.wikipedia.org/wiki/Pinhole_camera

Pinhole camera

7

Limitations• Small pinhole: sharper image, longer exposure

• Larger pinhole: blurrier image, shorter exposure

Camera model• Thin lens, aperture, shutter, film

Lenses• Gather more light

• Use refraction

• Need to be focused

Lens

http://en.wikipedia.org/wiki/Lens_(optics)

Scene point(emits or

reflects light)Image of

scene point

LensesPinhole Lens

6 sec. exposure 0.01 sec exposure

Thin lens model• Theoretical model for well-behaved lenses

• Properties

– All parallel rays converge at focal lengthlength

– Rays through the center are not deflectedSame perspective imageas pinhole at centerof lens

Thin lens model• How are arbitrary rays deflected when

passing through a thin lens?

8

Thin lens model Thin lens model• Similar triangles

Thin lens model• More similar triangles

Thin lens model• Thin lens formula

• All rays passing through a single point on a plane at distance in front of the lens will pass through a single point at distance behind the lensbehind the lens

Thin lens model• Focus at infinity:

Film plane

• Closest focusing distance:

Object

Thin lens model• Out of focus film plane results in spherical

blur

Out of focus film planes

Spherical blur

9

Properties of real lenses• Mostly undesired!

• Aberrations

– Spherical aberration– Chromatic aberration

• Distortion

– Barrel distortion– Pincushion distortion

• Etc.

Barrel & pincushion distortion

Camera model• Thin lens, aperture, shutter, film

Aperture• Blurriness of out of focus objects depends

on aperture size

• Aperture size determines depth of field:depth range that is sharp in image

Aperture

Depth of field

Circle of confusion• Also called „blur circle“

• Calculation of radius c

– Lens focused at S1

– Object at S2

– Aperture A– Focal length f sensor

http://en.wikipedia.org/wiki/Circle_of_confusion

Proportional to A

f-number• Fraction of focal length over diameter of

aperture

• Large aperture means small f-number

• Practice: f-stops increase by factors of√2

– f/2.0, f/2.8, f/4, f/5.6, f/8– Aperture area gets halved in each step

10

Camera model• Thin lens, aperture, shutter, film

Shutter• Determines time the film is exposed to

light

• Amount of light captured is proportional to exposure time

• Long exposure leads to motion blur• Long exposure leads to motion blur

Reciprocity• Amount of light captured stays

same if exposure is doubled and aperture area is halved (or vice versa)

Reciprocity• Which exposure/aperture combination?

Film• Film/sensor responds roughly linearly to light

– „Double the amount of light leads to double the recorded value“

• Film speed: sensitivity of film to light– Digital photography analog: sensor gain– Scaling factor

• Measured using ISO scale– Linear: sensitivity is proportional to ISO value– „Double ISO value, halve the exposure time“

Film• Trade-off: higher gain, more noise

ISO 100 ISO 3200

11

Film• Underexposure

– Not enough light, image too dark• Overexposure

– Film or sensor is saturated– Clipping of highlight details

Good exposure OverexposureUnderexposure

Conclusions• Simple camera model

– Thin lens, aperture, shutter, film

• Photographs often have undesired artifacts

– Distortions, color artifacts, blur, noise, d /under/overexposure

Goal

• Develop algorithms to remove artifacts after image is captured

References• „Photography“, by London, Upton, Stone

Next time• Color, color processing