image-based rendering. © 2002 james k. hahn2 image-based rendering usually based on 2-d...

Image-based Rendering

© 2002 James K. Hahn2

Image-based Rendering• Usually based on 2-D imagesUsually based on 2-D images

• Pre-calculationPre-calculation

– Pre-rendering (speed)Pre-rendering (speed)

– From real photographs (speed and realism)From real photographs (speed and realism)

• Usually for static scene and moving viewpointUsually for static scene and moving viewpoint

• Rendering time decoupled from scene complexityRendering time decoupled from scene complexity

• Usually based on 2-D imagesUsually based on 2-D images

• Pre-calculationPre-calculation

– Pre-rendering (speed)Pre-rendering (speed)

– From real photographs (speed and realism)From real photographs (speed and realism)

• Usually for static scene and moving viewpointUsually for static scene and moving viewpoint

• Rendering time decoupled from scene complexityRendering time decoupled from scene complexity


2-D Techniques• Warp reference image(s) to generate required imageWarp reference image(s) to generate required image

• Consider images as texture mapsConsider images as texture maps

• Use hardware for handling texturesUse hardware for handling textures

• Warp reference image(s) to generate required imageWarp reference image(s) to generate required image

• Consider images as texture mapsConsider images as texture maps

• Use hardware for handling texturesUse hardware for handling textures


Sprites• Billboard: 2-D image that is handled as a 3-D objectBillboard: 2-D image that is handled as a 3-D object

– E.g. image of a tree kept perpendicular to direction of viewE.g. image of a tree kept perpendicular to direction of view

• Imposters: generalization of billboardImposters: generalization of billboard

– May be pre-calculated to correspond to each of its May be pre-calculated to correspond to each of its bounding box sidesbounding box sides

– Imposter corresponding to the side which face the Imposter corresponding to the side which face the viewpoint is usedviewpoint is used

– Rendered as texture mapRendered as texture map

– Warped as the viewpoint movesWarped as the viewpoint moves

• Billboard: 2-D image that is handled as a 3-D objectBillboard: 2-D image that is handled as a 3-D object

– E.g. image of a tree kept perpendicular to direction of viewE.g. image of a tree kept perpendicular to direction of view

• Imposters: generalization of billboardImposters: generalization of billboard

– May be pre-calculated to correspond to each of its May be pre-calculated to correspond to each of its bounding box sidesbounding box sides

– Imposter corresponding to the side which face the Imposter corresponding to the side which face the viewpoint is usedviewpoint is used

– Rendered as texture mapRendered as texture map

– Warped as the viewpoint movesWarped as the viewpoint moves


Error of planar imposters

V0V1

ImpostorX’

x1

x2

• As viewpoint moves from As viewpoint moves from V0 to to V1

X’ no longer represents no longer representsboth both x1 and and x2

• If angle is less than thatsubtended by pixel, acceptableerror

• Amount of warp constrained

• No motion parallax

• As viewpoint moves from As viewpoint moves from V0 to to V1

X’ no longer represents no longer representsboth both x1 and and x2

• If angle is less than thatsubtended by pixel, acceptableerror

• Amount of warp constrained

• No motion parallax


Image layering• “2 ½ D” rendering

• 3D scene is segmented into different layers

– Objects assigned to different layers roughly according to distance to viewer

• Rendering resources allocated to different “display memory” by spatial and/or temporal sampling rates

– Can be prioritized by distance or speed

• Each layer results in sprites that are then warped according to viewing direction

• Sprites are then composited into final output image

• “2 ½ D” rendering

• 3D scene is segmented into different layers

– Objects assigned to different layers roughly according to distance to viewer

• Rendering resources allocated to different “display memory” by spatial and/or temporal sampling rates

– Can be prioritized by distance or speed

• Each layer results in sprites that are then warped according to viewing direction

• Sprites are then composited into final output image


Using depth information• Layers or sprites with depth information (not per-pixel depth)

• Images with z-buffer (per-pixel depth)

• Layered depth images (LDI)

– Single view of scene with multiple pixels along each line of sight

– Complexity a function of depth complexity (average number of surfaces that project onto a pixel)

• Layers or sprites with depth information (not per-pixel depth)

• Images with z-buffer (per-pixel depth)

• Layered depth images (LDI)

– Single view of scene with multiple pixels along each line of sight

– Complexity a function of depth complexity (average number of surfaces that project onto a pixel)


Images with z-buffer• For each I(x,y) warp to I(x’,y’) as viewpoint moves to a new

location

– x’, y’ a function of x, y, z, and transformation of viewpoint

• Image folding problem: more than one pixel in the reference image maps into a single pixel in extrapolated view

• Holes due to occluded point in reference image becoming visible in extrapolated view

• Holes due to “stretching”

• For each I(x,y) warp to I(x’,y’) as viewpoint moves to a new location

– x’, y’ a function of x, y, z, and transformation of viewpoint

• Image folding problem: more than one pixel in the reference image maps into a single pixel in extrapolated view

• Holes due to occluded point in reference image becoming visible in extrapolated view

• Holes due to “stretching”


Layered depth images (LDI)• 3-D structure for a particular viewpoint

• For each pixel, store information for all surfaces that it intersects

– Color, surface normal, depth

• Generated by ray-tracing or warping n images (with depth information) from different viewpoints

• During rendering, incremental warp of each layer in back to front order

• 3-D structure for a particular viewpoint

• For each pixel, store information for all surfaces that it intersects

– Color, surface normal, depth

• Generated by ray-tracing or warping n images (with depth information) from different viewpoints

• During rendering, incremental warp of each layer in back to front order


View interpolation• Frames required for walkthrough from

– set of reference images

– warp script that describe corresponding pixels (pixel motion)

• View morphing

– Generate interpolated view from reference images

– Interpolated transformation that preserve object shape

– Need to know camera parameters

• Frames required for walkthrough from

– set of reference images

– warp script that describe corresponding pixels (pixel motion)

• View morphing

– Generate interpolated view from reference images

– Interpolated transformation that preserve object shape

– Need to know camera parameters


Lumigraph(light field rendering)• For each point in the scene, pre-calculate and store

the radiance in every direction at that point

• Assume occluder-free space (along a ray, radiance is constant)

• Parameterized by 4-D function: two parallel planes with (s, t) and (u, v) parameterization

• Can be generated from photography by taking a picture at discrete points in (s, t)

• For each point in the scene, pre-calculate and store the radiance in every direction at that point

• Assume occluder-free space (along a ray, radiance is constant)

• Parameterized by 4-D function: two parallel planes with (s, t) and (u, v) parameterization

• Can be generated from photography by taking a picture at discrete points in (s, t)


Photo-modeling• Generation of 3-D model from photography

• Allow rich textures to be used from real world

• Much user intervention by specifying correspondence with known geometry

• Generation of 3-D model from photography

• Allow rich textures to be used from real world

• Much user intervention by specifying correspondence with known geometry


Photographic panorama• E.g. Apple Computer’s QuickTime VR

• Individual images stitched into cylindrical panorama

• Given a viewpoint, can pan in any direction in real-time

• E.g. Apple Computer’s QuickTime VR

• Individual images stitched into cylindrical panorama

• Given a viewpoint, can pan in any direction in real-time

image-based rendering. © 2002 james k. hahn2 image-based rendering usually based on 2-d...

Documents