lightfields and lumigraphs
DESCRIPTION
Lightfields and Lumigraphs. Vision for Graphics CSE 590SS, Winter 2001 Richard Szeliski (with lots of slides from Michael Cohen). Modeling light. How do we generate new scenes and animations from existing ones? Classic “3D Vision + Graphics”: take (lots of) pictures recover camera pose - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/1.jpg)
Lightfields and Lumigraphs
Vision for GraphicsCSE 590SS, Winter 2001
Richard Szeliski
(with lots of slides from Michael Cohen)
![Page 2: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/2.jpg)
3/7/2001 Vision for Graphics 2
Modeling light
How do we generate new scenes and animations from existing ones?
Classic “3D Vision + Graphics”:• take (lots of) pictures• recover camera pose• build 3D model• extract texture maps / BRDFs• synthesize new views
![Page 3: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/3.jpg)
3/7/2001 Vision for Graphics 3
Computer Graphics
Image
Output
ModelSyntheticCamera
![Page 4: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/4.jpg)
3/7/2001 Vision for Graphics 4
Real Scene
Computer Vision
Real Cameras
Model
Output
![Page 5: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/5.jpg)
3/7/2001 Vision for Graphics 5
Combined
Model Real Scene
Real Cameras
Image
Output
SyntheticCamera
![Page 6: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/6.jpg)
3/7/2001 Vision for Graphics 6
But, vision technology falls short
ModelReal Scene
Real Cameras
Image
Output
SyntheticCamera
![Page 7: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/7.jpg)
3/7/2001 Vision for Graphics 7
… and so does graphics.
ModelReal Scene
Real Cameras
Image
Output
SyntheticCamera
![Page 8: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/8.jpg)
3/7/2001 Vision for Graphics 8
Image Based Rendering
Real Scene
Real Cameras-or-
Expensive Image Synthesis
Images+Model
Image
Output
SyntheticCamera
![Page 9: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/9.jpg)
3/7/2001 Vision for Graphics 9
Ray
Constant radiance• time is fixed
5D• 3D position• 2D direction
![Page 10: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/10.jpg)
3/7/2001 Vision for Graphics 10
All Rays
Plenoptic Function:• all possible images• too much stuff!
![Page 11: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/11.jpg)
3/7/2001 Vision for Graphics 11
Line
Infinite line
4D• 2D direction• 2D position• non-dispersive medium
![Page 12: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/12.jpg)
3/7/2001 Vision for Graphics 12
Ray
Discretize, then interpolate
Distance between 2 rays• Which is closer together?
![Page 13: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/13.jpg)
3/7/2001 Vision for Graphics 13
Image
What is an image?
All rays through a point• Panorama?
![Page 14: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/14.jpg)
3/7/2001 Vision for Graphics 14
Image
Image plane
2D• position
![Page 15: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/15.jpg)
3/7/2001 Vision for Graphics 15
Light leaving towards “eye”
2D• just dual of image
Object
![Page 16: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/16.jpg)
3/7/2001 Vision for Graphics 16
Object
All light leaving object
![Page 17: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/17.jpg)
3/7/2001 Vision for Graphics 17
Object
4D• 2D position• 2D direction
![Page 18: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/18.jpg)
3/7/2001 Vision for Graphics 18
Object
All images
![Page 19: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/19.jpg)
3/7/2001 Vision for Graphics 19
Lumigraph / Lightfield
Outside convex space
4D
StuffEmpty
![Page 20: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/20.jpg)
3/7/2001 Vision for Graphics 20
Lumigraph
Inside convex space
4D
EmptyStuff
![Page 21: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/21.jpg)
3/7/2001 Vision for Graphics 21
Lumigraph
How to • organize• capture• render
![Page 22: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/22.jpg)
3/7/2001 Vision for Graphics 22
Lumigraph - Organization
2D position
2D direction
s
![Page 23: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/23.jpg)
3/7/2001 Vision for Graphics 23
Lumigraph - Organization
2D position
2D position
2 plane parameterization
su
![Page 24: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/24.jpg)
3/7/2001 Vision for Graphics 24
Lumigraph - Organization
2D position
2D position
2 plane parameterization
us
t s,tu,v
v
s,t
u,v
![Page 25: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/25.jpg)
3/7/2001 Vision for Graphics 25
Lumigraph - Organization
Hold s,t constant
Let u,v vary
An image
s,t u,v
![Page 26: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/26.jpg)
3/7/2001 Vision for Graphics 26
Lumigraph - Organization
Discretization• higher res near object
– if diffuse– captures texture
• lower res away– captures directions
s,t u,v
![Page 27: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/27.jpg)
3/7/2001 Vision for Graphics 27
Lumigraph - Capture
Idea 1• Move camera carefully over
s,t plane• Gantry
– see Lightfield paper
s,t u,v
![Page 28: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/28.jpg)
3/7/2001 Vision for Graphics 28
Lumigraph - Capture
Idea 2• Move camera anywhere• Rebinning
– see Lumigraph paper
s,t u,v
![Page 29: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/29.jpg)
3/7/2001 Vision for Graphics 29
Lumigraph - Rendering
For each output pixel• determine s,t,u,v• either
– find closest discrete RGB– interpolate near values s,t u,v
![Page 30: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/30.jpg)
3/7/2001 Vision for Graphics 30
Lumigraph - Rendering
For each output pixel• determine s,t,u,v
• either• use closest discrete RGB• interpolate near values
s u
![Page 31: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/31.jpg)
3/7/2001 Vision for Graphics 31
Lumigraph - Rendering
Nearest• closest s• closest u• draw it
Blend 16 nearest• quadrilinear interpolation
s u
![Page 32: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/32.jpg)
3/7/2001 Vision for Graphics 32
Lumigraph - Rendering
Depth Correction• closest s• intersection with “object”• best u• closest u
s u
![Page 33: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/33.jpg)
3/7/2001 Vision for Graphics 33
Lumigraph - Rendering
Depth Correction • quadralinear interpolation• new “closest”• like focus
s u
![Page 34: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/34.jpg)
3/7/2001 Vision for Graphics 34
Lumigraph - Rendering
Fast s,t,u,v finding• scanline interpolate• texture mapping• shear warp
s u
![Page 35: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/35.jpg)
3/7/2001 Vision for Graphics 35
Lumigraph - Ray Space
![Page 36: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/36.jpg)
3/7/2001 Vision for Graphics 36
Lumigraph - Ray Space
Image effects:• parallax• occlusion• transparency• highlights
![Page 37: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/37.jpg)
3/7/2001 Vision for Graphics 37
Lumigraph - Demo
Lumigraph• Lion, Fruit Bowl, Visible Woman, Path Tracing
![Page 38: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/38.jpg)
3/7/2001 Vision for Graphics 38
Lightfield - DemoDigital Michelangelo Project
– Marc Levoy, Stanford University– Lightfield (“night”) assembled by Jon Shade
![Page 39: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/39.jpg)
3/7/2001 Vision for Graphics 39
Surface Lightfields
Turn 4D parameterization around
Leverage coherence
![Page 40: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/40.jpg)
3/7/2001 Vision for Graphics 40
Surface Lightfields
Wood et al, SIGGRAPH 2000
![Page 41: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/41.jpg)
3/7/2001 Vision for Graphics 41
3D Representations
Image is 2D
Lumigraph is 4D
What happened to 3D?• 3D Lumigraph subset• Concentric mosaics
![Page 42: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/42.jpg)
3/7/2001 Vision for Graphics 42
3D Lumigraph
One row of s,t plane• i.e., hold t constant
s,t u,v
![Page 43: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/43.jpg)
3/7/2001 Vision for Graphics 43
3D Lumigraph
One row of s,t plane• i.e., hold t constant• thus s,u,v• a “row of images”
s
u,v
![Page 44: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/44.jpg)
3/7/2001 Vision for Graphics 44
Concentric Mosaics
Replace “row” with “circle” of images
![Page 45: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/45.jpg)
3/7/2001 Vision for Graphics 45
Concentric Mosaics
![Page 46: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/46.jpg)
3/7/2001 Vision for Graphics 46
Concentric Mosaics
From above
![Page 47: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/47.jpg)
3/7/2001 Vision for Graphics 47
Concentric Mosaics
Depth correction
![Page 48: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/48.jpg)
3/7/2001 Vision for Graphics 48
Concentric Mosaics
Panorama
![Page 49: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/49.jpg)
3/7/2001 Vision for Graphics 49
2.5D Representations
Image is 2DLumigraph is 4D3D
• 3D Lumigraph subset• Concentric mosaics
2.5D• Layered Depth Images• View Dependent Surfaces
![Page 50: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/50.jpg)
3/7/2001 Vision for Graphics 50
Layered Depth Image
ImageDepthLayered
2.5 D ?
![Page 51: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/51.jpg)
3/7/2001 Vision for Graphics 51
Layered Depth Image
Rendering from LDI
• Incremental in LDI X and Y • Guaranteed to be in back-to-front order
![Page 52: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/52.jpg)
3/7/2001 Vision for Graphics 52
Layered Depth Image
Rendering from LDI
• Incremental in LDI X and Y • Guaranteed to be in back-to-front order
![Page 53: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/53.jpg)
3/7/2001 Vision for Graphics 53
Layered Depth Image (LDI)
![Page 54: Lightfields and Lumigraphs](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814556550346895db2248a/html5/thumbnails/54.jpg)
3/7/2001 Vision for Graphics 58
Summary
5D: Plenoptic Function (Ray)
4D: Lumigraph / Lightfield
3D: Lumigraph Subset
3D: Concentric Mosaics
2.5D: Layered Depth Image
2.5D: Image Based Models
2D: Image