advanced topics in computer vision” - york universitymbrown/eecs6323/... · “advanced topics in...
TRANSCRIPT
![Page 1: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/1.jpg)
1
“Advanced Topics in Computer Vision”
Computational Photography
Prof. Michael S. BrownEECS – Lassonde School of Engineering
Introduction
![Page 2: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/2.jpg)
Lecturer
• Dr. Michael S. Brown• Professor
EECS Department
Lassonde School of Engineering
• Office Location
– Lassonde 3022
• Office Hours
– Please arrange by email: [email protected]
(Subject: EECS 6323)
Brown 2
![Page 3: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/3.jpg)
3
Welcome
• You probably have questions:
– What is computational photography?
– What was wrong with regular photography?
– What am I going to learn?
– How much work is this course?
– Will I get an A?
• Hope to answer these questions in today’s lecture
![Page 4: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/4.jpg)
4
Tentative - Assessment
• 4-5 Assignments– Most assignments will have multiple subproblems from which you can select
• Tentative Assignments– Assignment 1: basic image processing routines
• Thresholding, edge detection, deblurring, etc. . .
– Assignment 2: Interactive computer vision/Computational Processing• Interactive Image Snapping
• Interactive Image Segmentation
• Gradient-domain Cut-and-pasting
– Assignment 3: Computational Illumination• HDR imaging + tone mapping
• Flash/No-Flash low-light photography
• Multi-flash camera (gradient camera)
– Assignment 4: Computational Optics• Coded exposure camera
• Focal Stacking (everywhere in-focus-image)
• Image pre-conditioning for projector blur
– Assignment 5: Misc/Vision Related• Seam Carving
• Texture Synthesis
Assignments will be marked
via individual one-on-one
sessions with me. I’ll likely
mark two assignments at a
time to minimize meetings.
![Page 5: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/5.jpg)
What is Computational
Photography?
• Definition 1: the use of photographic imagery to create
graphics content
• Definition 2: The use of computational techniques to
overcome limitations of conventional photography
Definition from: A. Efros, Berkeley
Originally called Image-Based Rendering
(IBR), but calling it Computational
Photography is much cooler .
We will focus primarily on
topics/research related to
definition 2.
5
![Page 6: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/6.jpg)
What is Computational
Photography?
• Definition 1: the use of photographic imagery to
create graphics content
• Definition 2: The use of computational techniques to
overcome limitations of conventional photography
6
Definition from: A. Efros, Berkeley
![Page 7: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/7.jpg)
The Realism Spectrum
+ easy to create new worlds
+ easy to manipulate objects/ viewpoint
- very hard to look realistic
+ instantly realistic
+ easy to acquire
- very hard to manipulate
objects/viewpoint
Computer Graphics PhotographyComputationalPhotography
RealismManipulationEase of capture
Slide credit: A. Efros
Image-based rendering . . . Exploit
images to help improve the realism of
graphics. . .
![Page 8: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/8.jpg)
Where have you seen IBR?
8
Texture maps Environment Maps
Mosaics
![Page 9: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/9.jpg)
What is Computational
Photography?
• Definition 1: the use of photographic imagery to create
graphics content
• Definition 2: The use of computational techniques to
overcome limitations of conventional photography
9
Definition from: A. Efros, Berkeley
![Page 10: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/10.jpg)
• Blur, camera shake, noise, damage
Limitations of Conventional Photography
10
![Page 11: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/11.jpg)
• Limited resolution
Limitations of Conventional Photography
11
![Page 12: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/12.jpg)
• Bad color / no color
Limitations of Conventional Photography
12
![Page 13: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/13.jpg)
• Unwanted objects
Limitations of Conventional Photography
13
![Page 14: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/14.jpg)
• Limited dynamic range
Limitations of Conventional Photography
14
![Page 15: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/15.jpg)
• Single viewpoint, static 2D picture
Limitations of Conventional Photography
15
![Page 16: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/16.jpg)
• Single depth of focus
Limitations of Conventional Photography
16
![Page 17: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/17.jpg)
Where can we make changes?
17Shree Nayar’s (U. Columbia) vision for “Computational Cameras”...
![Page 18: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/18.jpg)
What else can we do?
18
Put the “human in the loop” when editing photographs.
![Page 19: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/19.jpg)
Three main topics covered
• Computational Processing
– Interactive Computer Vision
– Process an image, with the human in the loop
• Examples: Image segmentation, Image Colorization, Compositing . . .
• Computational Optics
– Modification of the optics
– Assume image is processed with prior knowledge of the
modification
• Examples: Coded exposure, coded aperture, translating imagery, hybrid
cameras . . .
• Computational Illumination
– Modification/control of the illumination and or exposure
– Assume image will be processed using the prior knowledge of
the illumination manipulation
• Examples: HDR imaging, dual photography, flash/no flash imaging . . .
19
![Page 20: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/20.jpg)
20
Another opinion . . .
What is computational
photography?
Following notes (with some modifications) are from the SIGGRAPH’2007 course on Computational Photography, by Prof. Jack Tumblinfrom Northwestern University, USA.
![Page 21: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/21.jpg)
Focus, Click, Print:
‘Film-Like Photography’
2D Image:
‘Instantaneous’
Intensity Map
Light + 3D Scene:
Illumination, shape, movement, surface BRDF,…
Ang
le(
,)
Positio
n(x
,y)
‘Center of
Projection’
(P3 or P2
Origin)
Ray Bundles
Ray Bundles
Display on a monitor(or print it).
![Page 22: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/22.jpg)
Film-Like Photography
Thought Experiment:
Side-by-side digital camera & film camera.
• COMPARE:
– Digital Camera result.
– Film result.
Can we See more, Do more, Feel more?
Has photography really changed yet ?
![Page 23: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/23.jpg)
scene
display
Scene
Light
Intensities
Display
Light
Intensities
‘Pixel values’(scene intensity? display intensity?
perceived intensity? ‘blackness/whiteness’ ?)
display
Digitally Perfected Photography?
![Page 24: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/24.jpg)
‘Film-Like’ Photography
Film Camera design assumptions:
– ‘Instantaneous’ light measurement…
– Of focal plane image behind a lens.
– Reproduce those amounts of light.
Implied:
“What we see is
focal-plane intensities.”
well, no…we see much more!
(seeing is deeply cognitive)
![Page 25: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/25.jpg)
Jack’s Definitions
• ‘Film-like’ Photography:
– Static ‘instantaneous’ record of
the 2D image formed by a lens
Display image sensor image
• ‘Computational’ Photography:
– displayed image sensor image
– A more expressive, controllable displayed result,
transformed, merged, decoded sensor data
![Page 26: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/26.jpg)
What is Photography?
Safe answer:
A wholly new,expressive medium (ca. 1830s)
• Manipulated display of what we think, feel, want, …– Capture a memory, a visual experience in tangible form
– ‘painting with light’; express the subject’s visual essence
– “Exactitude is not the truth.” –Henri Matisse
![Page 27: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/27.jpg)
• A ‘bucket’ word: a neat container for messy notions(e.g. aviation, music, comprehension)
• A record of what we see,or would like to see,in tangible form.
• Does ‘film’ photography
always capture it? No!
• What do we see?What is missing?
What is Photography?
Harold ‘Doc’ Edgerton 1936
![Page 28: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/28.jpg)
DisplayRGB(x,y,tn)
ImageI(x,y,λ,t)
Light &
Optics3D Scenelight sources,
BRDFs,
shapes,
positions,
movements,
…
Eyepointposition,
movement,
projection,
…
PHYSICAL PERCEIVED
What is Photography?
Exposure
Control,
tone mapScenelight sources,
BRDFs,
shapes,
positions,
movements,
…
Eyepointposition,
movement,
projection,
…
Vis
ion
Photo: A Tangible Record
Editable, storable as
Film or Pixels
![Page 29: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/29.jpg)
3D Scene?light sources,
BRDFs,
shapes,
positions,
movements,
…
Eyepoint?position,
movement,
projection,
…
Meaning…
Visual
Stimulus
3D Scenelight sources,
BRDFs,
shapes,
positions,
movements,
…
Eyepointposition,
movement,
projection,
…
PHYSICALPERCEIVED or UNDERSTOOD
Ultimate Photographic Goals
Vis
ion
Sen
sor(
s)
Com
pu
tin
g
Light &
Optics
Photo: A Tangible Record
![Page 30: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/30.jpg)
Millions of delicate, fascinating treasures:
– < 1% of Smithsonian collection ever exhibited
– sparse $, displays; conservation limits access
A Driving Problem: Museum Artifacts
![Page 31: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/31.jpg)
Current Archives:
Not rich enough
• Fixed, static viewpoint
• Fixed, static lighting
• Custom light: impractical
• Conflates shapes,
materials, shadows,
texture, highlights, …
Can you understand
this shape?
![Page 32: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/32.jpg)
Current Archives:
Not rich enough
• Fixed, static viewpoint
• Fixed, static lighting
• Custom light: impractical
• Conflates shapes,
materials, shadows,
texture, highlights, …
Can you understand
this shape?
What ‘photo archive’
can best match in-hand,
direct examination ?
What is missing?
![Page 33: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/33.jpg)
Missing:
Reliable Visual Boundaries5 ray sets explicit geometric occlusion boundaries
Ramesh Raskar, MERL, 2004
![Page 34: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/34.jpg)
Rollout Photographs © Justin Kerr: Slide idea: Steve Seitz
http://research.famsi.org/kerrmaya.html
Missing: Occlusion Removal
BOTH capture visual appearance;
BOTH should be easy to make!
![Page 35: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/35.jpg)
Missing:
Viewpoint Freedom “Multiple-Center-of-Projection Images” Rademacher, P, Bishop, G., SIGGRAPH '98
![Page 36: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/36.jpg)
Missing:
Interaction…Adjust everything: lighting, pose, viewpoint, focus, FOV,…
Winnemoller EG 2005: after Malzbender, SIGG2001
Can I moved the light source?
![Page 37: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/37.jpg)
Missing:
Expressive Time Manipulations
What other ways
better reveal
appearance to
human viewers?
(Without direct shape
measurement? )
Time for space wiggle. Gasparini, 1998.
Can you understand
this shape better?
![Page 38: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/38.jpg)
Photographic Signal: Pixels Rays
• Core ideas are ancient, simple, seem obvious:
– Lighting: ray sources
– Optics: ray bending/folding devices
– Sensor: measure light
– Processing: assess it
– Display: reproduce it
• Ancient Greeks:
‘eye rays’ wipe the world
to feel its contents…
http://www.mlahanas.de/Greeks/Optics.htm
![Page 39: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/39.jpg)
The Photographic Signal Path
Claim: Computing can improve every step
Light Sources Sensors Data Types,
Processing
DisplayRays
OpticsOptics
Scene
Rays
Eyes
![Page 40: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/40.jpg)
Review: How many Rays in a 3-D Scene?
A 4-D set of infinitesimal members.
Imagine:
– Convex Enclosure of a 3D scene
– Inward-facing ray camera at every surface point
– Pick the rays you need for ANY camera outside.
2D surface of cameras,
2D ray set for each camera,
4D set of rays.
(Levoy et al. SIGG’96) (Gortler et al. ‘96)
+
![Page 41: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/41.jpg)
The 4D Light Field
If you’re a bit confused, don’t worry, we will talk about this more. . .
![Page 42: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/42.jpg)
4-D Light Field / Lumigraph
Measure all the outgoing light rays of the
scene.
And assumesa fixed illumination of the 3D scene.
![Page 43: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/43.jpg)
4-D Illumination Field
Same Idea: Measure all the incoming light rays
coming into the scene.
Now thinkabout theilluminationof a 3D scene.It is also a 4Dbundle of rays.
![Page 44: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/44.jpg)
4D x 4D = 8-D Reflectance Field
Ratio: Rij = (outgoing rayi) / (incoming rayj)
![Page 45: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/45.jpg)
45[Debevec et al. 2002]
[Debevec et al. 2000] [Masselus et al. 2002]
[Masselus et al. 2003] [Malzbender et al. 2002]
[Matusik et al. 2002]
Is a 4-D Light Source Required?
![Page 46: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/46.jpg)
Is A 4D Camera Required? e.g. MIT Dynamic Light Field Camera 2002
• Multiple dynamic Virtual Viewpoints
• Efficient Bandwidth usage:‘send only what you see’
• 64 tightly packed commodity CMOS webcams
• 30 Hz, Scaleable, Real-time:
or is it just “more film-like cameras, but now with computers!” ?
Is this the whole answer?
![Page 47: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/47.jpg)
Or do Ray Changes Convey Appearance?
5 ray sets explicit geometric occlusion boundaries
Ramesh Raskar, MERL, 2004
![Page 48: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/48.jpg)
Or do Ray Changes Convey Appearance?
• These rays + all these rays give me…
• MANY more useful
details I can examine…
![Page 49: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/49.jpg)
Mild Viewing & Lighting Changes;
Are these Enough?
Convicing visual appearance:
Is Accurate Depth really necessary?
a few good 2-D images may be enough…
“Image jets, Level Sets,
and Silhouettes“
Lance Williams,
talk at Stanford, 1998.
![Page 50: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/50.jpg)
‘The Ideal Photographic Signal’
I (Jack) CLAIM IT IS:
All Rays? Some Rays? Changes in Some Rays
Photographic ray space is vast and redundant>8 dimensions: 4D view, 4D light, time, ,
? Gather only ‘visually significant’ ray changes ?
? What rays should we measure ?
? How should we combine them ?
? How should we display them ?
![Page 51: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/51.jpg)
Future PhotographyNovel Illuminators
Novel Cameras
Scene: 8D Ray Modulator
Generalized
Sensors
Generalized
Processing4D Ray
Sampler
Ray Reconstructor
General Optics:4D Ray Benders
Recreated 4D Light field
Lights
Modulators
4D Incident Lighting
Ge
ne
ral O
ptics:
4D
Ray B
en
ders
Generalized Display
Novel Displays
![Page 52: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/52.jpg)
Beyond ‘Film-Like’ Photography
Call it ‘Computational Photography’:
To make ‘meaningful ray changes’ tangible,
• Optics can do more…
• Sensors can do more…
• Light Sources can do more…
• Processing can do more…
by applying low-cost storage,
computation, and control.
![Page 53: Advanced Topics in Computer Vision” - York Universitymbrown/EECS6323/... · “Advanced Topics in Computer Vision” ... • 4-5 Assignments – Most assignments will have multiple](https://reader034.vdocuments.mx/reader034/viewer/2022042216/5ebdaa9e74911f3abe278a10/html5/thumbnails/53.jpg)
Introduction Summary
• Photography is changing
– Already, most images are “touched up” via software
– Cameras have built-in algorithms that modify the
picture directly (sometimes without you knowing)
• Camera hardware is changing too
– Since we plan to process the image, we can now
explore ways to modify the optics and illumination
with post-processing
– This may change the way lens, flashes are used in
the future
53