advanced topics in computer vision” - york universitymbrown/eecs6323/... · “advanced topics in...

1

“Advanced Topics in Computer Vision”

Computational Photography

Prof. Michael S. BrownEECS – Lassonde School of Engineering

Introduction

Lecturer

• Dr. Michael S. Brown• Professor

EECS Department

Lassonde School of Engineering

• Office Location

– Lassonde 3022

• Office Hours

– Please arrange by email: mbrown@eecs.yorku.ca

(Subject: EECS 6323)

Brown 2

3

Welcome

• You probably have questions:

– What is computational photography?

– What was wrong with regular photography?

– What am I going to learn?

– How much work is this course?

– Will I get an A?

• Hope to answer these questions in today’s lecture

4

Tentative - Assessment

• 4-5 Assignments– Most assignments will have multiple subproblems from which you can select

• Tentative Assignments– Assignment 1: basic image processing routines

• Thresholding, edge detection, deblurring, etc. . .

– Assignment 2: Interactive computer vision/Computational Processing• Interactive Image Snapping

• Interactive Image Segmentation

• Gradient-domain Cut-and-pasting

– Assignment 3: Computational Illumination• HDR imaging + tone mapping

• Flash/No-Flash low-light photography

• Multi-flash camera (gradient camera)

– Assignment 4: Computational Optics• Coded exposure camera

• Focal Stacking (everywhere in-focus-image)

• Image pre-conditioning for projector blur

– Assignment 5: Misc/Vision Related• Seam Carving

• Texture Synthesis

Assignments will be marked

via individual one-on-one

sessions with me. I’ll likely

mark two assignments at a

time to minimize meetings.

What is Computational

Photography?

• Definition 1: the use of photographic imagery to create

graphics content

• Definition 2: The use of computational techniques to

overcome limitations of conventional photography

Definition from: A. Efros, Berkeley

Originally called Image-Based Rendering

(IBR), but calling it Computational

Photography is much cooler .

We will focus primarily on

topics/research related to

definition 2.

5

What is Computational

Photography?

• Definition 1: the use of photographic imagery to

create graphics content

• Definition 2: The use of computational techniques to

overcome limitations of conventional photography

6

Definition from: A. Efros, Berkeley

The Realism Spectrum

+ easy to create new worlds

+ easy to manipulate objects/ viewpoint

- very hard to look realistic

+ instantly realistic

+ easy to acquire

- very hard to manipulate

objects/viewpoint

Computer Graphics PhotographyComputationalPhotography

RealismManipulationEase of capture

Slide credit: A. Efros

Image-based rendering . . . Exploit

images to help improve the realism of

graphics. . .

Where have you seen IBR?

8

Texture maps Environment Maps

Mosaics

What is Computational

Photography?

• Definition 1: the use of photographic imagery to create

graphics content

• Definition 2: The use of computational techniques to

overcome limitations of conventional photography

9

Definition from: A. Efros, Berkeley

• Blur, camera shake, noise, damage

Limitations of Conventional Photography

10

• Limited resolution

Limitations of Conventional Photography

11

• Bad color / no color

Limitations of Conventional Photography

12

• Unwanted objects

Limitations of Conventional Photography

13

• Limited dynamic range

Limitations of Conventional Photography

14

• Single viewpoint, static 2D picture

Limitations of Conventional Photography

15

• Single depth of focus

Limitations of Conventional Photography

16

Where can we make changes?

17Shree Nayar’s (U. Columbia) vision for “Computational Cameras”...

What else can we do?

18

Put the “human in the loop” when editing photographs.

Three main topics covered

• Computational Processing

– Interactive Computer Vision

– Process an image, with the human in the loop

• Examples: Image segmentation, Image Colorization, Compositing . . .

• Computational Optics

– Modification of the optics

– Assume image is processed with prior knowledge of the

modification

• Examples: Coded exposure, coded aperture, translating imagery, hybrid

cameras . . .

• Computational Illumination

– Modification/control of the illumination and or exposure

– Assume image will be processed using the prior knowledge of

the illumination manipulation

• Examples: HDR imaging, dual photography, flash/no flash imaging . . .

19

20

Another opinion . . .

What is computational

photography?

Following notes (with some modifications) are from the SIGGRAPH’2007 course on Computational Photography, by Prof. Jack Tumblinfrom Northwestern University, USA.

Focus, Click, Print:

‘Film-Like Photography’

2D Image:

‘Instantaneous’

Intensity Map

Light + 3D Scene:

Illumination, shape, movement, surface BRDF,…

Ang

le(

,)

Positio

n(x

,y)

‘Center of

Projection’

(P3 or P2

Origin)

Ray Bundles

Ray Bundles

Display on a monitor(or print it).

Film-Like Photography

Thought Experiment:

Side-by-side digital camera & film camera.

• COMPARE:

– Digital Camera result.

– Film result.

Can we See more, Do more, Feel more?

Has photography really changed yet ?

scene

display

Scene

Light

Intensities

Display

Light

Intensities

‘Pixel values’(scene intensity? display intensity?

perceived intensity? ‘blackness/whiteness’ ?)

display

Digitally Perfected Photography?

‘Film-Like’ Photography

Film Camera design assumptions:

– ‘Instantaneous’ light measurement…

– Of focal plane image behind a lens.

– Reproduce those amounts of light.

Implied:

“What we see is

focal-plane intensities.”

well, no…we see much more!

(seeing is deeply cognitive)

Jack’s Definitions

• ‘Film-like’ Photography:

– Static ‘instantaneous’ record of

the 2D image formed by a lens

Display image sensor image

• ‘Computational’ Photography:

– displayed image sensor image

– A more expressive, controllable displayed result,

transformed, merged, decoded sensor data

What is Photography?

Safe answer:

A wholly new,expressive medium (ca. 1830s)

• Manipulated display of what we think, feel, want, …– Capture a memory, a visual experience in tangible form

– ‘painting with light’; express the subject’s visual essence

– “Exactitude is not the truth.” –Henri Matisse

• A ‘bucket’ word: a neat container for messy notions(e.g. aviation, music, comprehension)

• A record of what we see,or would like to see,in tangible form.

• Does ‘film’ photography

always capture it? No!

• What do we see?What is missing?

What is Photography?

Harold ‘Doc’ Edgerton 1936

DisplayRGB(x,y,tn)

ImageI(x,y,λ,t)

Light &

Optics3D Scenelight sources,

BRDFs,

shapes,

positions,

movements,

…

Eyepointposition,

movement,

projection,

…

PHYSICAL PERCEIVED

What is Photography?

Exposure

Control,

tone mapScenelight sources,

BRDFs,

shapes,

positions,

movements,

…

Eyepointposition,

movement,

projection,

…

Vis

ion

Photo: A Tangible Record

Editable, storable as

Film or Pixels

3D Scene?light sources,

BRDFs,

shapes,

positions,

movements,

…

Eyepoint?position,

movement,

projection,

…

Meaning…

Visual

Stimulus

3D Scenelight sources,

BRDFs,

shapes,

positions,

movements,

…

Eyepointposition,

movement,

projection,

…

PHYSICALPERCEIVED or UNDERSTOOD

Ultimate Photographic Goals

Vis

ion

Sen

sor(

s)

Com

pu

tin

g

Light &

Optics

Photo: A Tangible Record

Millions of delicate, fascinating treasures:

– < 1% of Smithsonian collection ever exhibited

– sparse $, displays; conservation limits access

A Driving Problem: Museum Artifacts

Current Archives:

Not rich enough

• Fixed, static viewpoint

• Fixed, static lighting

• Custom light: impractical

• Conflates shapes,

materials, shadows,

texture, highlights, …

Can you understand

this shape?

Current Archives:

Not rich enough

• Fixed, static viewpoint

• Fixed, static lighting

• Custom light: impractical

• Conflates shapes,

materials, shadows,

texture, highlights, …

Can you understand

this shape?

What ‘photo archive’

can best match in-hand,

direct examination ?

What is missing?

Missing:

Reliable Visual Boundaries5 ray sets explicit geometric occlusion boundaries

Ramesh Raskar, MERL, 2004

Rollout Photographs © Justin Kerr: Slide idea: Steve Seitz

http://research.famsi.org/kerrmaya.html

Missing: Occlusion Removal

BOTH capture visual appearance;

BOTH should be easy to make!

Missing:

Viewpoint Freedom “Multiple-Center-of-Projection Images” Rademacher, P, Bishop, G., SIGGRAPH '98

Missing:

Interaction…Adjust everything: lighting, pose, viewpoint, focus, FOV,…

Winnemoller EG 2005: after Malzbender, SIGG2001

Can I moved the light source?

Missing:

Expressive Time Manipulations

What other ways

better reveal

appearance to

human viewers?

(Without direct shape

measurement? )

Time for space wiggle. Gasparini, 1998.

Can you understand

this shape better?

Photographic Signal: Pixels Rays

• Core ideas are ancient, simple, seem obvious:

– Lighting: ray sources

– Optics: ray bending/folding devices

– Sensor: measure light

– Processing: assess it

– Display: reproduce it

• Ancient Greeks:

‘eye rays’ wipe the world

to feel its contents…

http://www.mlahanas.de/Greeks/Optics.htm

The Photographic Signal Path

Claim: Computing can improve every step

Light Sources Sensors Data Types,

Processing

DisplayRays

OpticsOptics

Scene

Rays

Eyes

Review: How many Rays in a 3-D Scene?

A 4-D set of infinitesimal members.

Imagine:

– Convex Enclosure of a 3D scene

– Inward-facing ray camera at every surface point

– Pick the rays you need for ANY camera outside.

2D surface of cameras,

2D ray set for each camera,

4D set of rays.

(Levoy et al. SIGG’96) (Gortler et al. ‘96)

+

The 4D Light Field

If you’re a bit confused, don’t worry, we will talk about this more. . .

4-D Light Field / Lumigraph

Measure all the outgoing light rays of the

scene.

And assumesa fixed illumination of the 3D scene.

4-D Illumination Field

Same Idea: Measure all the incoming light rays

coming into the scene.

Now thinkabout theilluminationof a 3D scene.It is also a 4Dbundle of rays.

4D x 4D = 8-D Reflectance Field

Ratio: Rij = (outgoing rayi) / (incoming rayj)

45[Debevec et al. 2002]

[Debevec et al. 2000] [Masselus et al. 2002]

[Masselus et al. 2003] [Malzbender et al. 2002]

[Matusik et al. 2002]

Is a 4-D Light Source Required?

Is A 4D Camera Required? e.g. MIT Dynamic Light Field Camera 2002

• Multiple dynamic Virtual Viewpoints

• Efficient Bandwidth usage:‘send only what you see’

• 64 tightly packed commodity CMOS webcams

• 30 Hz, Scaleable, Real-time:

or is it just “more film-like cameras, but now with computers!” ?

Is this the whole answer?

Or do Ray Changes Convey Appearance?

5 ray sets explicit geometric occlusion boundaries

Ramesh Raskar, MERL, 2004

Or do Ray Changes Convey Appearance?

• These rays + all these rays give me…

• MANY more useful

details I can examine…

Mild Viewing & Lighting Changes;

Are these Enough?

Convicing visual appearance:

Is Accurate Depth really necessary?

a few good 2-D images may be enough…

“Image jets, Level Sets,

and Silhouettes“

Lance Williams,

talk at Stanford, 1998.

‘The Ideal Photographic Signal’

I (Jack) CLAIM IT IS:

All Rays? Some Rays? Changes in Some Rays

Photographic ray space is vast and redundant>8 dimensions: 4D view, 4D light, time, ,

? Gather only ‘visually significant’ ray changes ?

? What rays should we measure ?

? How should we combine them ?

? How should we display them ?

Future PhotographyNovel Illuminators

Novel Cameras

Scene: 8D Ray Modulator

Generalized

Sensors

Generalized

Processing4D Ray

Sampler

Ray Reconstructor

General Optics:4D Ray Benders

Recreated 4D Light field

Lights

Modulators

4D Incident Lighting

Ge

ne

ral O

ptics:

4D

Ray B

en

ders

Generalized Display

Novel Displays

Beyond ‘Film-Like’ Photography

Call it ‘Computational Photography’:

To make ‘meaningful ray changes’ tangible,

• Optics can do more…

• Sensors can do more…

• Light Sources can do more…

• Processing can do more…

by applying low-cost storage,

computation, and control.

Introduction Summary

• Photography is changing

– Already, most images are “touched up” via software

– Cameras have built-in algorithms that modify the

picture directly (sometimes without you knowing)

• Camera hardware is changing too

– Since we plan to process the image, we can now

explore ways to modify the optics and illumination

with post-processing

– This may change the way lens, flashes are used in

the future

53