3d motion estimation. 3d model construction video manipulation

48
3D Motion Estimation

Post on 22-Dec-2015

231 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: 3D Motion Estimation. 3D model construction Video Manipulation

3D Motion Estimation

Page 2: 3D Motion Estimation. 3D model construction Video Manipulation

3D model construction

Page 3: 3D Motion Estimation. 3D model construction Video Manipulation

3D model construction

Page 4: 3D Motion Estimation. 3D model construction Video Manipulation

Video Manipulation

Page 5: 3D Motion Estimation. 3D model construction Video Manipulation
Page 6: 3D Motion Estimation. 3D model construction Video Manipulation

Visual Motion

• Allows us to compute useful properties of the 3D world, with very little knowledge.

• Example: Time to collision

Page 7: 3D Motion Estimation. 3D model construction Video Manipulation

Time to Collision

ff

LL

vv

LL

DDoo

l(t)l(t)

An object of heightAn object of height L L moves with moves with

constant velocity constant velocity v:v:•At time At time t=0t=0 the object is at: the object is at:

• D(0) = DD(0) = Doo

•At time At time tt it is at it is at

•D(t) = DD(t) = Doo – vt – vt

•It will crash with the camera at It will crash with the camera at

time:time:

• D(D() = D) = Doo – v – v = 0 = 0

• = D= Doo/v/v

t=0t=0tt

D(t)D(t)

Page 8: 3D Motion Estimation. 3D model construction Video Manipulation

Time to Collision

ff

LL

vv

LL

DDoo

l(t)l(t)

t=0t=0tt

D(t)D(t)

The image of the object has size The image of the object has size

l(t):l(t):

Taking derivative wrt time:Taking derivative wrt time:

Page 9: 3D Motion Estimation. 3D model construction Video Manipulation

Time to Collision

ff

LL

vv

LL

DDoo

l(t)l(t)

t=0t=0tt

D(t)D(t)

And their ratio is:And their ratio is:

Page 10: 3D Motion Estimation. 3D model construction Video Manipulation

Time to Collision

ff

LL

vv

LL

DDoo

l(t)l(t)

t=0t=0tt

D(t)D(t)

And And time to collisiontime to collision::

Can be Can be directly directly measured measured from from imageimage

Can be found, without knowing Can be found, without knowing LL or or DDoo or or vv !! !!

Page 11: 3D Motion Estimation. 3D model construction Video Manipulation
Page 12: 3D Motion Estimation. 3D model construction Video Manipulation
Page 13: 3D Motion Estimation. 3D model construction Video Manipulation
Page 14: 3D Motion Estimation. 3D model construction Video Manipulation

Structure from Motion

rtzu ˆ

1tr Z

rrzu ωˆ1

rot Ft,

r

Zrottr uuu

u

Page 15: 3D Motion Estimation. 3D model construction Video Manipulation

Passive Navigation and Structure from Motion

. velocity rotational and ,

velocity ionalh translatmotion wit rigid a with moves system The

,T

zyxT

zyx ωωωttt t

fyxZYX T ,, points image ontoproject ,, points Scene rR

image in the observed ispoint scene a of ,, velocity 3D theand zyx VVVR

.0., velocity as vur

Page 16: 3D Motion Estimation. 3D model construction Video Manipulation

Consider a 3D point P and its image:

ff

PP

pp

ZZ zz

Using pinhole camera equation:Using pinhole camera equation:

Page 17: 3D Motion Estimation. 3D model construction Video Manipulation

Let things move:

ff

PPVV

ppvv

ZZ zz

The The relative velocityrelative velocity of P wrt camera: of P wrt camera:

TranslatioTranslation velocityn velocity Rotation Rotation

angular angular velocityvelocity

Page 18: 3D Motion Estimation. 3D model construction Video Manipulation

3D Relative Velocity:

ff

PPVV

ppvv

ZZ zz

The relative velocity of P wrt camera:The relative velocity of P wrt camera:

Page 19: 3D Motion Estimation. 3D model construction Video Manipulation

Motion Field: the velocity of p

ff

PPVV

ppvv

ZZ zz

Taking derivative wrt time:Taking derivative wrt time:

Page 20: 3D Motion Estimation. 3D model construction Video Manipulation

Motion Field: the velocity of p

ff

PPVV

ppvv

ZZ zz

Page 21: 3D Motion Estimation. 3D model construction Video Manipulation

Motion Field: the velocity of p

ff

PPVV

ppvv

ZZ zz

Page 22: 3D Motion Estimation. 3D model construction Video Manipulation

Motion Field: the velocity of p

ff

PPVV

ppvv

ZZ zz

TranslationaTranslational componentl component

Scaling ambiguity(t and Z can onlybe derived up to a scale Factor)

Page 23: 3D Motion Estimation. 3D model construction Video Manipulation

Motion Field: the velocity of p

ff

PPVV

ppvv

ZZ zz

Rotational Rotational componentcomponent

NOTE: The rotational component is independent of depth Z !NOTE: The rotational component is independent of depth Z !

Page 24: 3D Motion Estimation. 3D model construction Video Manipulation

Translational flow field

(FOC).n contractio of focusor

(FOE)expansion of focus theis ,, p where 00 o

f

t

tf

t

tyx

z

y

z

x

Page 25: 3D Motion Estimation. 3D model construction Video Manipulation

Rotational flow field

(AOR). plane image thepierces axisrotation theepoint wher theis ,

ff

z

y

z

x

Page 26: 3D Motion Estimation. 3D model construction Video Manipulation

Pure Translation

What if tWhat if tzz 0 ? 0 ?

All motion field vectors are parallel to each other and All motion field vectors are parallel to each other and inversely proportional to depth !inversely proportional to depth !

Page 27: 3D Motion Estimation. 3D model construction Video Manipulation

Pure Translation: Properties of the MF

• If tz 0 the MF is RADIAL with all vectors pointing towards (or away from) a single point po. If tz = 0 the MF is PARALLEL.

• The length of the MF vectors is inversely proportional to depth Z. If tz 0 it is also directly proportional to the distance between p and po.

Page 28: 3D Motion Estimation. 3D model construction Video Manipulation

Pure Translation: Properties of the MF

• po is the vanishing point of the direction of translation.

• po is the intersection of the ray parallel to the translation vector and the image plane.

Page 29: 3D Motion Estimation. 3D model construction Video Manipulation

Special Case: Moving Plane

ff

PPVV

ppvv

ZZ zz

nnPlanar surfaces are Planar surfaces are common in man-made common in man-made environmentsenvironments

Question: How does the MF of a moving plane look like?Question: How does the MF of a moving plane look like?

Page 30: 3D Motion Estimation. 3D model construction Video Manipulation

Special Case: Moving PlanePoints on the plane must satisfy Points on the plane must satisfy the equation describing the plane. the equation describing the plane.

Let Let •nn be the unit vector normal to the be the unit vector normal to the plane.plane.•dd be the distance from the plane to be the distance from the plane to the origin.the origin.•NOTE:NOTE: If the plane is moving wrt If the plane is moving wrt camera, camera, nn and and dd are functions of timeare functions of time..

Then:Then:

dd PP

ZZ

nn

OOYY

XX

wherewhere

Page 31: 3D Motion Estimation. 3D model construction Video Manipulation

Special Case: Moving Plane

dd PP

ZZ

nn

OOYY

XX

Using the pinhole Using the pinhole projection projection equation:equation:pp

LetLet be the image of Pbe the image of P

Using the plane equation:Using the plane equation:

Solving for Z:Solving for Z:

Page 32: 3D Motion Estimation. 3D model construction Video Manipulation

Special Case: Moving Plane

And Plug in:And Plug in:

dd PP

ZZ

nn

OOYY

XX

pp

Now consider the MF equations:Now consider the MF equations:

Page 33: 3D Motion Estimation. 3D model construction Video Manipulation

Special Case: Moving Plane

dd PP

ZZ

nn

OOYY

XX

pp

The MF equations become:The MF equations become:

wherewhere

Page 34: 3D Motion Estimation. 3D model construction Video Manipulation

Special Case: Moving Plane

dd PP

ZZ

nn

OOYY

XX

pp

MF equations:MF equations:

Q: What is the significance of this?Q: What is the significance of this?

A: The MF vectors are given by low order (second) A: The MF vectors are given by low order (second) polynomials. polynomials.

•Their coeffs. aTheir coeffs. a11 to a to a8 8 (only 8 !) are functions of n, d, t and (only 8 !) are functions of n, d, t and ..•The same coeffs. can be obtained with a different plane The same coeffs. can be obtained with a different plane and relative velocity.and relative velocity.

Page 35: 3D Motion Estimation. 3D model construction Video Manipulation

Moving Plane: Properties of the MF

• The MF of a planar surface is at any time a quadratic function in the image coordinates.

• A plane nTP=d moving with velocity V=-t-£ P has the same MF than a plane with normal n’=t/|t| , distance d and moving with velocity

V=|t|n –( + n£ t/d)£ P

Page 36: 3D Motion Estimation. 3D model construction Video Manipulation

Classical Structure from Motion

• Established approach is the epipolar minimization: The “derotated flow” should be parallel to the translational flow.

uu

-urot

-urot

EE

utr

t

utr

Page 37: 3D Motion Estimation. 3D model construction Video Manipulation

The Translational Case(a least squares formulation)

Substitute back

min

Substitute

)),(( ||),(||

min

22

2

22

Zv

Zu

yttxtt

dydxyxfyxf

dydxZ

tytv

Z

txtu

zyzx

yzxz

min22

2

dydxvu

)minimized. be would

for which oflength the(Find . respect to with Minimize :1 Step tr dZ u

vu

ZZZ

vZZ

u

22

220

2 minimize We d

Page 38: 3D Motion Estimation. 3D model construction Video Manipulation

Step 2: Differentiate with respect to tx, ty, tz, set expression to zero.

222

Let

vuvuK

0III

0II

0I

dydxKxtyt

dydxKxtt

dydxKytt

yx

zx

zy

0IIIIII equationsdependent linearly 3 zyx ttt

Equations are nonlinear in tx, ty, tz

Page 39: 3D Motion Estimation. 3D model construction Video Manipulation

Using a different norm

dydxyxfyxf

dydxZ

tytv

Z

txtu yzxz

)(),(||),(||

min )(

222,

22

22

vu

ZZZ

vZZ

au

22

220

First, differentiate integrand with respect to Z and set to zero

),,(

min2

zyx tttg

dydxvu

Page 40: 3D Motion Estimation. 3D model construction Video Manipulation

zxzyyxzyxzyx tfttettdtctbtattttg 222),,( 222

where

, )(

, )(

,

, )(

,

,

2

2

dydxyuxvvf

dydxyuxvue

dydxuvd

dydxyuxvc

dydxub

dydxva

Page 41: 3D Motion Estimation. 3D model construction Video Manipulation

Differentiate g( tx, ty, tz ) with respect to tx, ty, tz and set to zero

cef

ebd

fda

G

||t||Gt 1 with 0

Solution is singular vector corresponding to smallest singular value

Page 42: 3D Motion Estimation. 3D model construction Video Manipulation

The Rotational Case min2

rot2

rot vvuu

01

012

2

xxyyv

yxxyu

zyx

zyx

v

u

xxyfy

yfxxy

z

y

x

2

2

In matrix form

u

u

TT AAA

A1

Let f=1

Page 43: 3D Motion Estimation. 3D model construction Video Manipulation

The General Case

Minimization of epipolar distance

or, in vector notation

min2

tr

tr

rot

rot

dydxu

v

vv

uu

min2 rrrrt d

Page 44: 3D Motion Estimation. 3D model construction Video Manipulation

Motion ParallaxThe difference in motion between two very close points does not depend on rotation.Can be used at depth discontinuities to obtain the direction of translation.

FOE

Page 45: 3D Motion Estimation. 3D model construction Video Manipulation

Motion Parallax

0

0

21,2,1

21,2,1

21

2 1

)11

)((

)11

)((

, ,

have we),(p andp pointsAt

),(

xx

yy

u

v

ZZyyvvv

ZZxxuuu

yx

vu

otrtrtr

otrtrtr

rotrot

rottr

uu

uuu

Page 46: 3D Motion Estimation. 3D model construction Video Manipulation

Vectors perpendicular to translational component

)|| ,||() ,(

00

00

xxyyxxyy

tr u

rotrotxxyytr vxxuyy )()( 00)|| ,||(1

00

uu

Vector component perpendicular to translational componentis only due to rotation rotation can be estimated from it.

Page 47: 3D Motion Estimation. 3D model construction Video Manipulation

Motion Estimation Techniques• Prazdny (1981), Burger Bhanu (1990), Nelson Aloimonos (1988), Heeger

Jepson (1992):

Decomposition of flow field into translational and rotational components. Translational flow field is radial (all vectors are emanating from (or pouring into) one point), rotational flow field is quadratic in image coordinates.

Either search in the space of rotations: remainig flow field

should be translational. Translational flow field

is evaluated by minimizing deviation from radial field:

.

Or search in the space of directions of translation: Vectors

perpendicular to translation are due to rotation only

0),(),( 00 yyxxuv

Page 48: 3D Motion Estimation. 3D model construction Video Manipulation

Motion Estimation Techniques

• Longuet-Higgins Prazdny (1980), Waxman (1987): Parametric model for local surface patches (planes or quadrics) solve locally for motion parameters and structure, because flow is linear in the motion parameters

(quadratic or higher order in the image coordinates)