View Morphingby

Steven M. Seitz Charles R. Dyer

Irwin Chiu HauComputer ScienceMcGill University

Winter 2004

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Overview

• What is view morphing?• How to do view morphing?• Results• Conclusion

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Mona Lisa view morphsSource: http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

View Morphing

What is view morphing?– What is it?

– Why do we care about it?

– Where do we see them?

– Image Morphing vs View interpolation vs View Morphing

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Virtual Cameras

Source:View Morphing;Steven M. Seitz,Charles R. Dyer

View Morphing: Key Idea

Beier-Neely morph is NOT shape-preserving!– distortions– un-natural

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

A Shape-Distorting MorphSource: Steven M. Seitz, Charles R. Dyer

View Morphing: Key IdeaView morphing uses 3D shape preserving morph!

– no distortions

– natural

A morph is 3D shape preserving if the results of two

different views represent new views of the same object

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

A Shape-Distorting MorphSource: Steven M. Seitz, Charles R. Dyer

View morphing fixesthese intermediate

steps!

Why do we care?

View morphing is efficient

Produces new views without – 3D modelling

– Taking additional photos

View morphing creates impressive effects– Camera motion

– Image morphing

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

trueSpaceSource: www.caligari.com

Image Morphing vs View interpolation vs View Morphing

View Morphing is an extension toImage Morphing (Beier and Neely, 1992)

– Produces physically plausible new views of a scene

View Morphing (Seitz and Dyer, 1996) is an improvement over View Interpolation (Chen and Williams, 1993)

– Addresses non-rigid transformations problems– Does not require depth values– Creates realistic image transitions

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Overview

• What is view morphing?• How to do view morphing?• Results• Conclusion

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Mona Lisa view morphsSource: http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

How to do View Morphing?

View morphing in three steps1. Prewarp two images

2. Compute a morph between the prewarped images

3. Postwarp each in between images produced by the morph

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

View Morphing ProcedureSource: Steven M. Seitz, Charles R. Dyer

How to do View Morphing?

View morphing in three steps1. Prewarp two images

2. Compute a morph between the prewarped images

3. Postwarp each in between images produced by the morph

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

View morphing in 1 steps: Assume parallel views!1. Compute a morph between the parallel images

Parallel Views

Basic Theory

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

image point p0 = (x0,y0)scene point P = (X,Y,Z)p0 = Π0 Pwhere Π0 is a projection matrix

Morphing Parallel ViewsSource: Steven M. Seitz, Charles R. Dyer

image point p1 = (x1,y1)scene point P = (X,Y,Z)p1 = Π1 Pwhere Π1 is a projection matrix

Parallel Views

Basic Theory

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

image point p = (x,y)scene point P = (X,Y,Z)p = Π Pwhere Π is a homogenous projection matrix

Π = [ H | - HC]H: position and orientation of image planeC: euclidean position of the cameraMorphing Parallel Views

Source: Steven M. Seitz, Charles R. Dyer

Parallel Views

Mathematics for View Interpolation

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Πs: Linear interpolation of Π0 and Π1

Πs = (1 – s) Π0 + s Π1

s = [0,1]

Cs = (sCx, sCy, 0)fs = (1 – f) f0 + s f1

f : focal lenghC : center of a camera

View Interpolation Recap

View Interpolation vs View Morphing

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Morphing Parallel ViewsSource: Steven M. Seitz, Charles R. Dyer

View InterpolationSource: 3D Games byAlan Watt and Fabio Policarpo

How to do View Morphing?

View morphing in three steps1. Prewarp two images

2. Compute a morph between the prewarped images

3. Postwarp each in between images produced by the morph

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

View morphing in 1 steps: Assume parallel views!1. Compute a morph between the parallel images

Non-Parallel Views

The General Case

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

This is where the 3-Step Algorithmcomes into play1. Prewarping: I0 to Î0 and I1 to Î1 2. Morphing: Î0 and Î1 into Îs

3. Postwarping: Îs into Is

Morphing in Three StepsSource: Steven M. Seitz, Charles R. Dyer

Non-Parallel Views

Mathematics for Image Reprojection

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

ĤH-1p = pH and Ĥ are 3x3 matrices that represent the position and the orientation of their image planes

The resulting 3x3 matrix, ĤH-1 is a projective transformation that reprojects the image plane I into Î

Morphing in Three StepsSource: Steven M. Seitz, Charles R. Dyer

^

Limitations

• Singular Views cannot be reprojected to form parallel views

• Singular configurations are settings where one of the camera resides in the field of view of another camera

• Still works, just kind of conceptually hacky

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Parallel viewSource: Steven M. Seitz, Charles R. Dyer

Singular viewSource: Steven M. Seitz, Charles R. Dyer

Traditional Problems

Change in visibility creates1. Folds

– Occurs when a visible area becomes occluded

2. Holes– Occurs when an occluded area becomes visible

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Penumbra, umbra and hole regionsSource: Chen and Williams

FoldSource: Irwin Chiu Hau

Area

Point

Producing The Morph

We have talked about theory behind algorithm1. How to project/unproject images to parallel planes

2. How to warp between parallel planes

3. Theoretical problems

Now, let’s do an actual View Dependent Morph!

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Producing The Morph

We need:1. Two images I0 and I1

2. Two perspective projection matrices Π0 and Π1

3. Correspondence between pixels

Note that a sequence of projection matrices Πs is required to

control the entire animation, but Πs can be computed automatically

if we know Hs .

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Controlling The Morph

Hs can be obtained indirectly by establishing constraints– Recall Hs is position and orientation of image plane – One way is to specify four control points

Note: Control points implicitly determine the postwarpingtransformation

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

View Morphing ProcedureSource: Steven M. Seitz, Charles R. Dyer

Four control points form the red bounding box to determine the postwarping stage

Yellow lines are set of features to determine the prewarping stage

View Morphing sans Prewarping

Prewarping is not necessary for:– Objects that aren’t closely related

• Prewarping is less effective

• Computation is unstable

– Images that are approx. orthographic (eg. telephoto)

However, postwarping should not be left out to:– Reduce image plane distorsions

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Overview

• What is view morphing?• How to do view morphing?• Results• Conclusion

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Mona Lisa view morphsSource: http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

Results

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Facial view morphsSource: Steven M. Seitz, Charles R. Dyer

Results

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Facial view morphsSource: http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

Results

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Mona Lisa view morphsSource: http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

Results

Image Morphing vs View Morphing

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Image Morphing vs View MorphingSource: Steven M. Seitz, Charles R. Dyer

Conclusions

Things to remember

View Morphing• Powerful extension to image morphing• Produces new views of a scene• 3D shape preserving

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

References

• View Morphing - Seitz and Dyer, 1996• View Interpolation - Chen and Williams, 1993• Image Morphing - Beier and Neely, 1992• 3D Games: Realtime rendering and Sofware Technology

- Alan Watt and Fabio Policarpo

Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Questions?

Comp 767: Advanced Topics in Graphics: Image-Based Rendering