northeastern university, fall 2005 csg242: computational photography ramesh raskar mitsubishi...

Northeastern University, Fall 2005Northeastern University, Fall 2005 CSG242: Computational PhotographyCSG242: Computational Photography

Ramesh RaskarMitsubishi Electric Research Labs

Northeastern UniversityOct 19th, 2005

Course WebPage :

http://www.merl.com/people/raskar/photo/course/

Plan for TodayPlan for Today

• ““The Eye As a Camera” Michael SandbergThe Eye As a Camera” Michael Sandberg

• Computational Computational IlluminationIllumination

• Second Programming AssignmentSecond Programming Assignment

• Mid TermMid Term– Oct 26Oct 26thth

• Project Proposals DueProject Proposals Due– November 2November 2ndnd

• Paper reading Paper reading • 2 per student, 15 mins each, Reading list on the web2 per student, 15 mins each, Reading list on the web• Starts Nov 2Starts Nov 2ndnd

CreditsCredits• Assignments:Assignments:• Five project-oriented assignmentsFive project-oriented assignments• Requires programming in MatlabRequires programming in Matlab• 8 points each (Last assignment format is flexible)8 points each (Last assignment format is flexible)

• Mid-term ExamMid-term Exam• 20 points20 points•• Paper reading (two papers per student, 15 min presentation, 5pts each)Paper reading (two papers per student, 15 min presentation, 5pts each)• 10 points (Was Term Paper, 15 points)10 points (Was Term Paper, 15 points)

• Final ProjectFinal Project• Individual or in a group of 2Individual or in a group of 2• 20 points20 points

• Discretionary creditDiscretionary credit• 10 points (Was 5)10 points (Was 5)

Tentative ScheduleTentative Schedule

• Oct 26: Midterm examOct 26: Midterm exam

• Nov 2Nov 2ndnd Project Proposals Due Project Proposals Due

• Nov 9Nov 9thth Class ? Likely on 10 Class ? Likely on 10thth

• Nov 16Nov 16thth Class ? Class ?

• Nov 23Nov 23rdrd -> Likely on Nov 22 -> Likely on Nov 22ndnd

• Nov 30Nov 30thth

• Dec 7Dec 7thth

• Dec 15Dec 15thth (Exam week) Projects (Exam week) Projects

Mid-TermMid-Term

• Oct 26Oct 26thth at 6pm, at 6pm, • Duration: 90 minutesDuration: 90 minutes• Questions: Think, Explore, SolveQuestions: Think, Explore, Solve

– No need to remember all the formulas in detailNo need to remember all the formulas in detail– More concepts than math problemsMore concepts than math problems– Drawing diagrams to explain conceptsDrawing diagrams to explain concepts

• 20 points20 points• TopicsTopics

• All material covered till Oct 19All material covered till Oct 19thth

• Slides, assignments and in-class discussionsSlides, assignments and in-class discussions• Basics, Dynamic Range, Focus, IlluminationBasics, Dynamic Range, Focus, Illumination

FocusFocus

Computational Illumination

Synthetic LightingSynthetic LightingPaul Haeberli, Jan 1992Paul Haeberli, Jan 1992

Computational Computational PhotographyPhotography

Novel Illumination

Novel Cameras

Scene: 8D Ray Modulator

Display

GeneralizedSensor

Generalized Optics

Processing

Recreate 4D Lightfield

Light Sources

Photography Artifacts: Photography Artifacts: Flash HotspotFlash Hotspot

Ambient Flash

Flash Hotspot

Underexposed Reflections

Ambient Flash

Reflections due to FlashReflections due to Flash

Flash Brightness Falloff with DistanceFlash Brightness Falloff with Distance

Flash

Distant people underexposed

Combining Flash/No-flash Images Combining Flash/No-flash Images forfor High Dynamic Range (HDR) Imaging High Dynamic Range (HDR) Imaging

Need Both Ambient and Flash!! FlashAmbient

HDR Scene:

Underexposed

Well-lit in Flash

Well-lit in Ambient

Exposure Time

1/100 1/20 1/5 1 41/250

Conventional Exposure HDR:Conventional Exposure HDR:Varying Exposure TimeVarying Exposure Time

Flash

Bri

gh

tness

42

7

1

0

Flash HDR:Flash HDR:Varying Flash BrightnessVarying Flash Brightness

Scene distance dependence

Exposure Time

Flash

Bri

gh

tness

Flash-Exposure Flash-Exposure SamplingSampling

Flash-Exposure Flash-Exposure HDR:HDR:

Varying bothVarying both

Varying Exposure time Varying Flash brightness Varying both

Capturing HDR Image

Do We Need All Images ?Do We Need All Images ?

Regular 2D Sampling24 Pictures

Adaptive Sampling5 pictures

Next Best Picture ? Next Best Picture ?

Exposure Time

Flash

Bri

ghtn

ess

Exposure Time

Flash

Bri

ghtn

ess

• Based on all previous picturesBased on all previous pictures• Maximize well-lit pixels over the imageMaximize well-lit pixels over the image• Exclude pixels already captured as well-exposedExclude pixels already captured as well-exposed

HDR Image using N images

HDR Image using N+1 images

Underexposed Still UnderexposedWell-exposedExposure Time

Flash

Bri

ghtn

ess

?

?

?

N+1th picture ?

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, TurkMultiFlash NPR Camera

Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew Turk

Mitsubishi Electric Research Labs (MERL), Cambridge, MAU of California at Santa Barbara

U of North Carolina at Chapel Hill

Non-photorealistic Camera: Non-photorealistic Camera: Depth Edge Detection Depth Edge Detection andand Stylized Stylized

Rendering Rendering usingusing Multi-Flash ImagingMulti-Flash Imaging


Car Manuals


What are the problems with ‘real’ photo in conveying information ?

Why do we hire artists to draw what can be photographed ?


Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors


Shadows

Clutter

Many Colors

Highlight Edges

Mark moving parts

Basic colors

A New ProblemA New Problem


Why Non-photorealistic (NPR) Why Non-photorealistic (NPR) Images ?Images ?

• Easy to Understand• Easy to Display• Require not-so-rich (3D) data

Can we directly capture using a camera ?– Quick comprehensible images for the masses– Tools for the artists


Depth Edge Camera


Depth Discontinuities

Internal and externalShape boundaries, Occluding contour, Silhouettes


Depth Edges


Sigma = 9 Sigma = 5

Sigma = 1

Canny Intensity Edge Detection

Our method captures shape edges


Our MethodCanny


Our Method

Photo



Our Method

Photo Result



Our Method


Imaging Geometry

Shadow lies along epipolar ray


Shadow lies along epipolar ray,

Epipole and Shadow are on opposite sides of the edge

Imaging Geometry

m


Shadow lies along epipolar ray,

Shadow and epipole are on opposite sides of the edge

Imaging Geometry

m


Depth Edge Camera

Light epipolar rays are horizontal or vertical


Normalized

Left / Max

Right / Max

Left Flash

Right Flash

Input U{depth edges}


Normalized

Left / Max

Right / Max

Left Flash

Right Flash

Input U{depth edges}

Negative transition along epipolar ray is depth edge

Plot


Normalized

Left / Max

Right / Max

Left Flash

Right Flash

Input

Negative transition along epipolar ray is depth edge

Plot U{depth edges}


% Max composite maximg = max( left, right, top, bottom);

% Normalize by computing ratio imagesr1 = left./ maximg; r2 = top ./ maximg;r3 = right ./ maximg; r4 = bottom ./ maximg;

% Compute confidence mapv = fspecial( 'sobel' ); h = v';d1 = imfilter( r1, v ); d3 = imfilter( r3, v ); % vertical sobeld2 = imfilter( r2, h ); d4 = imfilter( r4, h ); % horizontal sobel

%Keep only negative transitions silhouette1 = d1 .* (d1>0); silhouette2 = abs( d2 .* (d2<0) );silhouette3 = abs( d3 .* (d3<0) );silhouette4 = d4 .* (d4>0);

%Pick max confidence in eachconfidence = max(silhouette1, silhouette2, silhouette3, silhouette4);imwrite( confidence, 'confidence.bmp');

No magicparameters

!

northeastern university, fall 2005 csg242: computational photography ramesh raskar mitsubishi...

Documents

flash slide

nd slide

focus slide

ambient slide

th class

flash hdr

th slides

flash welllit