ComputerVision

Linear Tracking

Jan-Michael FrahmCOMP 256

Some slides from Welch & Bishop

ComputerVision

2

Tracking

Tracking is the problem of generating an inference about the motion of an object given a sequence of images.

The key technical difficulty is maintaining an accurate representation of the posterior on object position given measurements, and doing so efficiently.

ComputerVision

3

Examples of tracking

ComputerVision

4

Model for tracking

• Object has internal state – Capital indicates random variable– Small represents particular value

• Obtained measurements in frame i are– Value of the measurement

iX

iYiy

iXix

ComputerVision

5

General Steps of Tracking

1. Prediction: What is the next state of the object given past measurements

2. Data association: Which measures are relevant for the state?

3. Correction: Compute representation of the state from prediction and measurements.

1100 ,, iii yYyYXP

iiiii yYyYyYXP ,,, 1100

ComputerVision

6

Tracking

predictpredict correctcorrect

ComputerVision

7

• Only immediate past matters

• Measurements depend only on current state

Important simplifications

Fortunately it doesn’t limit to much!

Independence Assumptions

111 ,, iiii XXPXXXP

ikjiiikji XYYPXYPXYYYP ,,,,,

ComputerVision

8

Linear Dynamic Models

• State is linear transformed plus Gaussian noise

• Relevant measures are linearly obtained from state plus Gaussian noise

• Sufficient to maintain mean and standard deviation

idiii xDNx ,~ 1

imiii xMNy ,~

ComputerVision

9

A really simple exampleWe are on a boat at night and lost our position

We know:

• star position

ComputerVision

10

Constant Velocity

p is position of boat, v is velocity of boat

state is

We only measure position so

1 ii pp

IDi

IM i

ii pX

1 iii XDX

ComputerVision

11

Marc makes a measurement

14121086420-2

,

Conditional Density Function

0y 0m

),(00 myN 00 yx

00 m

ComputerVision

12

Jan makes a measurement

Conditional Density Function

14121086420-2

,1y 1m

?1 x

?1

),(11 myN

ComputerVision

13

Combine measurements & variances

14121086420-2

Conditional Density Function

Online weighted average!

),( 22 xN

)( 12212 xyKxx

221

21

2

2y

K

221

22 2

111

y

ComputerVision

14

Rudolf Emil Kalman

• Born 1930 in Hungary• BS and MS from MIT• PhD 1957 from Columbia• Filter developed in 1960-61• Now retired

ComputerVision

15

Kalman filter

• Just some applied math.

• A linear dynamic system: f(a+b) = f(a) + f(b)

• Noisy data in hopefully less noisy out.

• But delay is the price for filtering...

ComputerVision

16

Predict Correct

KF operates by 1. Predicting the new state and its

uncertainty2. Correcting with the new measurement

predictpredict correctcorrect

ComputerVision

17

What is it used for?

• Tracking missiles• Tracking heads/hands/drumsticks• Extracting lip motion from video• Fitting Bezier patches to point data• Lots of computer vision applications• Economics• Navigation

ComputerVision

18

A really simple exampleWe are on a boat at night and lost our position

We know:

• move with constant velocity

• star position

ComputerVision

19

But suppose we’re moving

• Not all the difference is error. Some may be motion

• KF can include a motion model

• Estimate velocity and position

14121086420-2

ComputerVision

20

Process Model

• Describes how the state changes over time

• The state for the first example was scalar

• The process model was “nothing changes”

• A better model might be constant velocity motion

11 )( iii vtpp

1 ii vv

vpX

ComputerVision

21

Measurement Model

“What you see from where you are” not

“Where you are from what you see”

ComputerVision

22

Constant Velocity

p is position of boat, v is velocity of boat

state is

We only measure position so

11 )( iii vtpp

I

tIIDi 0

0IM i

vpX

1 iii XDX

ComputerVision

23

State and Error Covariance

• First two moments of Gaussian process

Error Covariance

Process State (Mean)

ix

id

ComputerVision

24

The Process Model

Uncertainty over intervalUncertainty over interval

State transitionState transition

Process dynamics

iiii wxDx 1

idi Nw ,0~

Difficult to determine

ComputerVision

25

Measurement Model

Measurement uncertainty

Measurement uncertainty

Measurementmatrix

Measurementmatrix

Measurement relationship to state

imi Nv ,0~

iiii vxMy

ComputerVision

26

Predict (Time Update)

a priori state, error covariance, measurement

1iii xDx

idTiiii DD

1 iii xMy

1ix

ix

i

1i

ComputerVision

27

Measurement Update (Correct)

Kalman gainKalman gain

a posteriori state and error covariance

iiiiii xMyKxx

iiii MKI

ix

ix

i

iMinimizes posteriori

error covariance

ComputerVision

28

The Kalman Gain

1 im

Tiii

Tiii MMMK

Weights between prediction and measurements to posteriori error covariance

For no measurement uncertainty 0im 111 iii

Ti

Tiii MMMMK

iiiiiiii yMxMyMxx 11

State is deduced only from measurement

ComputerVision

29

The Kalman Gain

Simple univariate (scalar) example

m

K

a posteriori state and error covariance

iiii xyKxx

ii K 1

ComputerVision

30

Summary

PREDICT CORRECT

1iii xDx

idTiiii DD

1

iiiiii xMyKxx

iiii MKI

1 im

Tiii

Tiii MMMK

ComputerVision

31

Estimating a Constant

The state transition matrix ID

iiiii wxwDxx

11

iiiii vxvMxy

The measurement matrix IM

Prediction

1ii xx

idii

1

ComputerVision

32

Measurement Update

imi

iiK

iiiii xyKxx

iii K1

ComputerVision

33

Setup/Initialization

510id

00 x

10

ComputerVision

34

State and Measurements = 0.1im

1.0

im

ComputerVision

35

Error Covariance

2)( i

ComputerVision

36

State and Measurements = 1im

ComputerVision

37

State and Measurements = 0.01

im

ComputerVision

38

Example camera pose estimation

ComputerVision

39

Kalman Filter Web Site

http://www.cs.unc.edu/~welch/kalman/

• Electronic and printed references– Book lists and recommendations– Research papers– Links to other sites– Some software

• News

ComputerVision

40

Java-Based KF Learning Tool• On-line 1D simulation• Linear and non-linear• Variable dynamics

http://www.cs.unc.edu/~welch/kalman/

ComputerVision

41

KF Course Web Page

http://www.cs.unc.edu/~tracker/ref/s2001/kalman/index.html

( http://www.cs.unc.edu/~tracker/ )

• Java-Based KF Learning Tool• KF web page

ComputerVision

42

Closing Remarks

• Try it!– Not too hard to understand or program

• Start simple– Experiment in 1D– Make your own filter in Matlab, etc.

• Note: the Kalman filter “wants to work”– Debugging can be difficult– Errors can go un-noticed

ComputerVision

43

Relevant References

• Azarbayejani, Ali, and Alex Pentland (1995). “Recursive Estimation of Motion, Structure, and Focal Length,” IEEE Trans. Pattern Analysis and Machine Intelligence 17(6): 562-575.

• Dellaert, Frank, Sebastian Thrun, and Charles Thorpe (1998). “Jacobian Images of Super-Resolved Texture Maps for Model-Based Motion Estimation and Tracking,” IEEE Workshop on Applications of Computer Vision (WACV'98), October, Princeton, NJ, IEEE Computer Society.

• http://mac-welch.cs.unc.edu/~welch/COMP256/