lecture 11 tracking - davide scaramuzzarpg.ifi.uzh.ch/docs/teaching/2020/11_tracking.pdf11 ðŒðŒ...
TRANSCRIPT
Vision Algorithms for Mobile Robotics
Lecture 11Tracking
Davide Scaramuzzahttp://rpg.ifi.uzh.ch
1
Lab Exercise â This afternoon
Implement the Kanade-Lucas-Tomasi (KLT) tracker
2
Outline
⢠Point tracking⢠Template tracking⢠Tracking by detection of local image features
3
Point Tracking
⢠Problem: given two images, estimate the motion of a pixel point from image ðŒðŒ0 to image ðŒðŒ1
4
ðŒðŒ0(ð¥ð¥, ðŠðŠ)
Point Tracking
⢠Problem: given two images, estimate the motion of a pixel point from image ðŒðŒ0 to image ðŒðŒ1
5
ðŒðŒ1(ð¥ð¥,ðŠðŠ)
Point Tracking
⢠Problem: given two images, estimate the motion of a pixel point from image ðŒðŒ0 to image ðŒðŒ1
⢠Two approaches exist, depending on the amount of motion between the frames⢠Block-based methods⢠Differential methods
6
ðŒðŒ0(ð¥ð¥, ðŠðŠ)
(ð¢ð¢, ð£ð£): optical flow vector
Point Tracking
⢠Consider the motion of the following corner
7
Point Tracking
⢠Consider the motion of the following corner
8
Point Tracking with Block Matching
⢠Search for the corresponding patch in a ð·ð· à ð·ð· region around the point to track.⢠Use SSD, SAD, or NCC
9
Search region
Patch to track
Pros and Cons of Block Matching
⢠Pros:⢠Works well if the motion is large
⢠Cons⢠Can become computationally demanding if the motion is large
⢠Can the âsearchâ be implemented in a smart way if the motion is âsmallâ?⢠Yes, use Differential methods
10
Point Tracking with Differential Methods
Looks at the local brightness changes at the same location. No patch shift is performed!
11
ðŒðŒ0(ð¥ð¥, ðŠðŠ)
Point Tracking with Differential Methods
Looks at the local brightness changes at the same location. No patch shift is performed!
12
ðŒðŒ0(ð¥ð¥, ðŠðŠ)
Point Tracking with Differential Methods
Looks at the local brightness changes at the same location. No patch shift is performed!
13
ðŒðŒ1(ð¥ð¥,ðŠðŠ)
Point Tracking with Differential Methods
Assumptions:⢠Brightness constancy
⢠The intensity of the pixels around the point to track does not change much between the two frames
⢠Temporal consistency⢠The motion displacement is small (1-2 pixels); however, this
can be addressed using multi-scale implementations (see later)
⢠Spatial coherency⢠Neighboring pixels undergo similar motion (i.e., they all lay
on the same 3D surface, i.e., no depth discontinuity)
14
The Kanade-Lucas-Tomasi (KLT) tracker
Consider the reference patch centered at (ð¥ð¥, ðŠðŠ) in image ðŒðŒ0 and the shifted patch centered at (ð¥ð¥ + ð¢ð¢, ðŠðŠ + ð£ð£)in image ðŒðŒ1. The patch has size Ω. We want to find the motion vector (ð¢ð¢, ð£ð£) that minimizes the Sum of Squared Differences (SSD):
This is a simple quadratic function in two variables (ð¢ð¢, ð£ð£)
15Lucas, Kanade, An iterative image registration technique with an application to stereo vision. Proceedings of Imaging Understanding Workshop, 1981. PDF.
Tomasi, Kanade, Detection and Tracking of Point Features, Carnegie Mellon University Technical Report CMU-CS-91-132, 1991. PDF.
ðððððð ð¢ð¢, ð£ð£ = ï¿œð¥ð¥,ðŠðŠâΩ
(ðŒðŒ0 ð¥ð¥,ðŠðŠ â ðŒðŒ1 ð¥ð¥ + ð¢ð¢,ðŠðŠ + ð£ð£ )2
â ï¿œ(ðŒðŒ0 ð¥ð¥,ðŠðŠ â ðŒðŒ1 ð¥ð¥,ðŠðŠ â ðŒðŒð¥ð¥ð¢ð¢ â ðŒðŒðŠðŠð£ð£)2
â ðððððð(ð¢ð¢, ð£ð£) = ï¿œ(âðŒðŒ â ðŒðŒð¥ð¥ð¢ð¢ â ðŒðŒðŠðŠð£ð£)2
ðŒðŒ0
ðŒðŒ1
(ð¥ð¥, ðŠðŠ)
(ð¥ð¥, ðŠðŠ)(ð¥ð¥ + ð¢ð¢, ðŠðŠ + ð£ð£)
The Kanade-Lucas-Tomasi (KLT) tracker
16
To minimize it, we differentiate it with respect to (ð¢ð¢, ð£ð£) and equate it to zero:
ðððððð(ð¢ð¢, ð£ð£) = ï¿œ(âðŒðŒ â ðŒðŒð¥ð¥ð¢ð¢ â ðŒðŒðŠðŠð£ð£)2
ðððððððððððð
= 0 , ðððððððððððð
= 0
â2ï¿œðŒðŒð¥ð¥(âðŒðŒ â ðŒðŒð¥ð¥ð¢ð¢ â ðŒðŒðŠðŠð£ð£) = 0
â2ï¿œðŒðŒðŠðŠ âðŒðŒ â ðŒðŒð¥ð¥ð¢ð¢ â ðŒðŒðŠðŠð£ð£ = 0
ððððððððððð¢ð¢
= 0 â
ððððððððððð£ð£
= 0 â
The Kanade-Lucas-Tomasi (KLT) tracker
⢠Linear system of two equations in two unknowns
⢠We can write them in matrix form:
17
ï¿œðŒðŒð¥ð¥(âðŒðŒ â ðŒðŒð¥ð¥ð¢ð¢ â ðŒðŒðŠðŠð£ð£) = 0
ï¿œðŒðŒðŠðŠ âðŒðŒ â ðŒðŒð¥ð¥ð¢ð¢ â ðŒðŒðŠðŠð£ð£ = 0
ï¿œðŒðŒð¥ð¥ðŒðŒð¥ð¥ ï¿œðŒðŒð¥ð¥ðŒðŒðŠðŠ
ï¿œðŒðŒð¥ð¥ðŒðŒðŠðŠ ï¿œðŒðŒðŠðŠðŒðŒðŠðŠ
ð¢ð¢ð£ð£ =
ï¿œðŒðŒð¥ð¥âðŒðŒ
ï¿œðŒðŒðŠðŠâðŒðŒâ ð¢ð¢
ð£ð£ =
ï¿œðŒðŒð¥ð¥ðŒðŒð¥ð¥ ï¿œðŒðŒð¥ð¥ðŒðŒðŠðŠ
ï¿œðŒðŒð¥ð¥ðŒðŒðŠðŠ ï¿œðŒðŒðŠðŠðŒðŒðŠðŠ
â1
ï¿œðŒðŒð¥ð¥âðŒðŒ
ï¿œðŒðŒðŠðŠâðŒðŒ
Havenât we seen this matrix already? Recall Harris detector!
Notice that these are NOT matrix products but pixel-wise products!
The Kanade-Lucas-Tomasi (KLT) tracker
In practice, det(ðð) should be non zero, which means that its eigenvalues should be large (i.e., not a flat region, not an edge) â in practice, it should be a corner or more generally contain any textured region!
18
Edge â det(M) is low
Flat â det(M) is low
Texture â det(M) is high
ðð =
ï¿œðŒðŒð¥ð¥ðŒðŒð¥ð¥ ï¿œðŒðŒð¥ð¥ðŒðŒðŠðŠ
ï¿œðŒðŒð¥ð¥ðŒðŒðŠðŠ ï¿œðŒðŒðŠðŠðŒðŒðŠðŠRR
= â
2
11
00λ
λ
Application to Corner Tracking
Color encodes motion direction
19
Application to Optical Flow
What if you track every single pixel in the image?
20
Application to Optical Flow
21
Application to Optical Flow
22
Optical Flow example
23
Aperture Problem
⢠Consider the motion of the following corner
24
Aperture Problem
⢠Consider the motion of the following corner
25
Aperture Problem
⢠Now, look at the local brightness changes through a small aperture
26
Aperture Problem
⢠Now, look at the local brightness changes through a small aperture
27
Aperture Problem
⢠Now, look at the local brightness changes through a small aperture
⢠We cannot always determine the motion direction â Infinite motion solutions may exist!⢠Solution?
28
Aperture Problem
⢠Now, look at the local brightness changes through a small aperture
⢠We cannot always determine the motion direction â Infinite motion solutions may exist!⢠Solution?
⢠Increase aperture size!
29
Block-based vs Differential methods
⢠Block-based methods:⢠Robust to large motions⢠Can be computationally expensive (ð·ð·Ãð·ð· validations need to be made for a single point to track)
⢠Differential methods: ⢠Works only for small motions (e.g., high frame rate). For larger motion, multi-scale implementations
are used but are more expensive (see later)⢠Much more efficient than block-based methods. Thus, can be used to track the motion of every pixel
in the image (i.e., optical flow). It avoids searching in the neighborhood of the point by analyzing the local intensity changes (i.e., differences) of an image patch at a specific location (i.e., no search is performed)
30
Outline
⢠Point tracking⢠Template tracking⢠Tracking by detection of local image features
31
Template tracking
Goal: follow a template image in a video sequence by estimating the warp
32
Template tracking
Goal: follow a template image in a video sequence by estimating the warp
33
Template image
Template Warping
⢠Given the template image ðð(ð±ð±)⢠Take all pixels from the template image ðð(ð±ð±) and warp them using the function ðð ð±ð±,ð©ð©
parameterized in terms of parameters ð©ð©
34
Template image
ðð(ð±ð±)
ðð ð±ð±,ð©ð©
warp
ðŒðŒ(ðð ð±ð±,ð©ð© )
Current image
Common 2D Transformations
⢠Translation
⢠Euclidean
⢠Affine
⢠Projective(homography)
35
ð¥ð¥â² = ð¥ð¥ + ðð1ðŠðŠâ² = ðŠðŠ + ðð2
ð¥ð¥â² = ð¥ð¥ð¥ð¥ð¥ð¥ð¥ð¥(ðð3) â ðŠðŠð¥ð¥ðŠðŠðŠðŠ(ðð3) + ðð1ðŠðŠâ² = ð¥ð¥ð¥ð¥ðŠðŠðŠðŠ(ðð3) + ðŠðŠð¥ð¥ð¥ð¥ð¥ð¥(ðð3) + ðð2
ð¥ð¥â² = ðð1ð¥ð¥ + ðð3ðŠðŠ + ðð5ðŠðŠâ² = ðð2ð¥ð¥ + ðð4ðŠðŠ + ðð6
ð¥ð¥â² =ðð1ð¥ð¥ + ðð2ðŠðŠ + ðð3ðð7ð¥ð¥ + ðð8ðŠðŠ + 1
ðŠðŠâ² =ðð4ð¥ð¥ + ðð5ðŠðŠ + ðð6ðð7ð¥ð¥ + ðð8ðŠðŠ + 1
Common 2D Transformations in Matrix form
We denote the transformation W ð±ð±,ð©ð© and p the set of parameters ðð = (ðð1,ðð2, ⊠,ðððð)
⢠Translation
⢠Euclidean
⢠Affine
⢠Projective(homography)
36
ðð ð±ð±,ð©ð© =ð¥ð¥ + ðð1ðŠðŠ + ðð2
= 1 0 ðð10 1 ðð2
ð¥ð¥ðŠðŠ1
ðð ð±ð±,ð©ð© = ð¥ð¥ð¥ð¥ð¥ð¥ð¥ð¥(ðð3) â ðŠðŠð¥ð¥ðŠðŠðŠðŠ(ðð3) + ðð1ð¥ð¥ð¥ð¥ðŠðŠðŠðŠ(ðð3) + ðŠðŠð¥ð¥ð¥ð¥ð¥ð¥(ðð3) + ðð2
= cos(ðð3) âsin(ðð3) ðð1sin(ðð3) cos(ðð3) ðð2
ð¥ð¥ðŠðŠ1
ðð ð±ð±,ð©ð© =ðð1ð¥ð¥ + ðð3ðŠðŠ + ðð5ðð2ð¥ð¥ + ðð4ðŠðŠ + ðð6
=ðð1 ðð3 ðð5ðð2 ðð4 ðð6
ð¥ð¥ðŠðŠ1
Homogeneous coordinates
ðð ï¿œðð,ð©ð© =ðð1 ðð2 ðð3ðð4 ðð5 ðð6ðð7 ðð8 1
ð¥ð¥ðŠðŠ1
Common 2D Transformations in Matrix form
37
ðð ð±ð±,ð©ð© = 1 0 ðð10 1 ðð2
ð¥ð¥ðŠðŠ1
ðð ð±ð±,ð©ð© = cos(ðð3) âsin(ðð3) ðð1sin(ðð3) cos(ðð3) ðð2
ð¥ð¥ðŠðŠ1
ðð ð±ð±,ð©ð© =ðð1 ðð3 ðð5ðð2 ðð4 ðð6
ð¥ð¥ðŠðŠ1
ðð ï¿œðð,ð©ð© =ðð1 ðð2 ðð3ðð4 ðð5 ðð6ðð7 ðð8 1
ð¥ð¥ðŠðŠ1
ðð ð±ð±,ð©ð© = ðð4cos(ðð3) âsin(ðð3) ðð1ð¥ð¥ðŠðŠðŠðŠ(ðð3) cos(ðð3) ðð2
ð¥ð¥ðŠðŠ1
Derivative and gradient
⢠Function: ðð ð¥ð¥
⢠Derivative: ððâ² ð¥ð¥ = ðððððððð
, where ð¥ð¥ is a scalar
⢠Function: ðð(ð¥ð¥1, ð¥ð¥2, ⊠, ð¥ð¥ðð )
⢠Gradient: âðð(ð¥ð¥1, ð¥ð¥2, ⊠, ð¥ð¥ðð )= ðððððððð1
, ðððððððð2
, ⊠, ðððððððððð
38
Jacobian
⢠ð¹ð¹(ð¥ð¥1, ð¥ð¥2, ⊠, ð¥ð¥ðð ) =ðð1(ð¥ð¥1, ð¥ð¥2, ⊠, ð¥ð¥ðð )
â®ðððð(ð¥ð¥1, ð¥ð¥2, ⊠, ð¥ð¥ðð )
is a vector-valued function
⢠The derivative in this case is called Jacobian ððð¹ð¹ððð±ð±
:
39
Carl Gustav Jacob (1804-1851)
ððð¹ð¹ððð±ð±
=
ðððð1ððð¥ð¥1
, ⊠,ðððð1ððð¥ð¥ðð
â®ððððððððð¥ð¥1
, ⊠,ððððððððð¥ð¥ðð
Displacement-model Jacobians âðððð
ðð = (ðð1,ðð2, ⊠,ðððð)
⢠Translation:
⢠Euclidean:
⢠Affine:
40
ðð ð±ð±,ð©ð© =ð¥ð¥ + ðð1ðŠðŠ + ðð2
ðð ð±ð±,ð©ð© = ð¥ð¥ð¥ð¥ð¥ð¥ð¥ð¥(ðð3) â ðŠðŠð¥ð¥ðŠðŠðŠðŠ(ðð3) + ðð1ð¥ð¥ð¥ð¥ðŠðŠðŠðŠ(ðð3) + ðŠðŠð¥ð¥ð¥ð¥ð¥ð¥(ðð3) + ðð2
ðð ð±ð±,ð©ð© =ðð1ð¥ð¥ + ðð3ðŠðŠ + ðð5ðð2ð¥ð¥ + ðð4ðŠðŠ + ðð6
ððððððð©ð©
=
ðððð1
ðððð1ðððð1
ðððð2ðððð2
ðððð1ðððð2
ðððð2
= 1 00 1
ððððððð©ð©
= 1 0 âð¥ð¥ð¥ð¥ðŠðŠðŠðŠ(ðð3) â ðŠðŠð¥ð¥ð¥ð¥ð¥ð¥(ðð3)0 1 ð¥ð¥ð¥ð¥ð¥ð¥ð¥ð¥(ðð3) â ðŠðŠð¥ð¥ðŠðŠðŠðŠ(ðð3)
ððððððð©ð©
= ð¥ð¥ 0 ðŠðŠ0 ð¥ð¥ 0
0 1 0ðŠðŠ 0 1
Template Warping
⢠Given the template image ðð(ð±ð±)⢠Take all pixels from the template image ðð(ð±ð±) and warp them using the function ðð ð±ð±,ð©ð©
parameterized in terms of parameters ð©ð©
41
Template image
ðð(ð±ð±)
ðð ð±ð±,ð©ð©
warp
ðŒðŒ(ðð ð±ð±,ð©ð© )
Current image
Template Tracking: Problem Formulation
⢠The goal of template-based tracking is to find the set of warp parameters p such that:
⢠This is solved by determining p that minimizes the Sum of Squared Differences:
42
ðŒðŒ ðð ð±ð±,ð©ð© = ðð(ð±ð±)
ððððð·ð· = ï¿œð±ð±âðð
ðŒðŒ ðð ð±ð±,ð©ð© â ðð(ð±ð±) ðð
Assumptions
⢠No errors in the template image boundaries: only the object to track appears in the template image
⢠No occlusion: the entire template is visible in the input image
⢠Brightness constancy, ⢠Temporal consistency, ⢠Spatial coherency
43
KLT tracker applied to template tracking
⢠Uses the Gauss-Newton method for minimization, that is:⢠Applies a first-order approximation of the warp⢠Attempts to minimize the SSD iteratively
44Lucas, Kanade, An iterative image registration technique with an application to stereo vision. Proceedings of Imaging Understanding Workshop, 1981. PDF.
Tomasi, Kanade, Detection and Tracking of Point Features, Carnegie Mellon University Technical Report CMU-CS-91-132, 1991. PDF.
Derivation of the KLT algorithm
⢠Assume that an initial estimate of p is known. Then, we want to find the increment âð©ð©that minimizes
⢠First-order Taylor approximation of ðŒðŒ ðð ð±ð±,ð©ð© + âð©ð© yelds to:
45
ðððððð = ï¿œð±ð±âðð
ðŒðŒ ðð ð±ð±,ð©ð© â ðð(ð±ð±)ðð
ðððððð = ï¿œð±ð±âðð
ðŒðŒ ðð ð±ð±,ð©ð© + âð©ð© â ðð(ð±ð±)ðð
ðŒðŒ ðð ð±ð±,ð©ð© + âð©ð© â ðŒðŒ ðð ð±ð±,ð©ð© +ð»ð»ðŒðŒ ððððððð©ð©âð©ð©
ð»ð»ðŒðŒ = ðŒðŒð¥ð¥ , ðŒðŒðŠðŠ = Image gradient evaluated at ðð(ð±ð±,ð©ð©) Jacobian of the warp ðð(ð±ð±,ð©ð©)
Derivation of the KLT algorithm
⢠By replacing ðŒðŒ ðð ð±ð±,ð©ð© + âð©ð© with its 1st order approximation, we get
⢠How do we minimize it?⢠We differentiate SSD with respect to âð©ð© and we equate it to zero, i.e.,
46
ðððððð = ï¿œð±ð±âðð
ðŒðŒ ðð ð±ð±,ð©ð© + âð©ð© â ðð(ð±ð±)ðð
ðððððð = ï¿œð±ð±âðð
ðŒðŒ ðð ð±ð±,ð©ð© +ð»ð»ðŒðŒððððððð©ð©
âð©ð© â ðð(ð±ð±)ðð
ððððððððððâð©ð©
= 0
Derivation of the KLT algorithm
47
ððððððððððâð©ð©
= 2ï¿œð±ð±âðð
ð»ð»ðŒðŒððððððð©ð©
T
ðŒðŒ ðð ð±ð±,ð©ð© +ð»ð»ðŒðŒððððððð©ð©
âð©ð© â ðð(ð±ð±)
ððððððððððâð©ð©
= 0
2ï¿œð±ð±âðð
ð»ð»ðŒðŒððððððð©ð©
T
ðŒðŒ ðð ð±ð±,ð©ð© +ð»ð»ðŒðŒððððððð©ð©
âð©ð© â ðð(ð±ð±) = 0 â
ðððððð = ï¿œð±ð±âðð
ðŒðŒ ðð ð±ð±,ð©ð© +ð»ð»ðŒðŒððððððð©ð©
âð©ð© â ðð(ð±ð±)ðð
Derivation of the KLT algorithm
48
â âð©ð© = ð»ð»â1ï¿œð±ð±âðð
ð»ð»ðŒðŒððððððð©ð©
T
ðð ð±ð± â ðŒðŒ ðð ð±ð±,ð©ð© =
ð»ð» = ï¿œð±ð±âðð
ð»ð»ðŒðŒððððððð©ð©
T
ð»ð»ðŒðŒððððððð©ð©
Second moment matrix (Hessian) of the warped image
What does H look like when the warp is a pure translation?
Notice that these are NOT matrix products but pixel-wise products!
KLT algorithm
1. Warp ðŒðŒ(ð±ð±) with ðð(ð±ð±,ð©ð©) âðŒðŒ ðð ð±ð±,ð©ð©
2. Compute the error: subtract ðŒðŒ ðð ð±ð±,ð©ð© from ðð(ð±ð±)
3. Compute warped gradients: ð»ð»ðŒðŒ = ðŒðŒð¥ð¥ , ðŒðŒðŠðŠ , evaluated at ðð(ð±ð±,ð©ð©)
4. Evaluate the Jacobian of the warping: ððððððð©ð©
5. Compute steepest descent: ð»ð»ðŒðŒ ððððððð©ð©
6. Compute Inverse Hessian: ð»ð»â1 = âð±ð±âðð ð»ð»ðŒðŒ ððððððð©ð©
Tð»ð»ðŒðŒ ðððð
ððð©ð©
â1
7. Multiply steepest descend with error: âð±ð±âðð ð»ð»ðŒðŒ ððððððð©ð©
Tðð ð±ð± â ðŒðŒ ðð ð±ð±,ð©ð©
8. Compute âð©ð©
9. Update parameters: ð©ð©âð©ð© + âð©ð©10. Repeat until âð©ð© < ðºðº
49
â âð©ð© = ð»ð»â1ï¿œð±ð±âðð
ð»ð»ðŒðŒððððððð©ð©
T
ðð ð±ð± â ðŒðŒ ðð ð±ð±,ð©ð©
KLT algorithm: computing âð©ð© = ð»ð»â1 âð±ð±âðð ð»ð»ðŒðŒ ððððððð©ð©
Tðð ð±ð± â ðŒðŒ ðð ð±ð±,ð©ð©
6x1
6x6
50
KLT algorithm: computing âð©ð© = ð»ð»â1 âð±ð±âðð ð»ð»ðŒðŒ ððððððð©ð©
Tðð ð±ð± â ðŒðŒ ðð ð±ð±,ð©ð©
6x1
6x6
51
KLT algorithm: computing âð©ð© = ð»ð»â1 âð±ð±âðð ð»ð»ðŒðŒ ððððððð©ð©
Tðð ð±ð± â ðŒðŒ ðð ð±ð±,ð©ð©
6x1
6x6
What is the size?ðððð Ã ðððð
ðð Ã ðððð
Why does it look like that?ððððððð©ð©
=ð¥ð¥ 0 ðŠðŠ
0 ð¥ð¥ 0
0 1 0
ðŠðŠ 0 1
52
KLT algorithm: computing âð©ð© = ð»ð»â1 âð±ð±âðð ð»ð»ðŒðŒ ððððððð©ð©
Tðð ð±ð± â ðŒðŒ ðð ð±ð±,ð©ð©
6x1
6x6
Why does it look like that?(slide 48)
What is the size?ðððð Ã ðððð
ðð Ã ðððð
Whatâs its size?
ðð Ã ðððð
ððððððð©ð©
=ð¥ð¥ 0 ðŠðŠ
0 ð¥ð¥ 0
0 1 0
ðŠðŠ 0 1
KLT algorithm: computing âð©ð© = ð»ð»â1 âð±ð±âðð ð»ð»ðŒðŒ ððððððð©ð©
Tðð ð±ð± â ðŒðŒ ðð ð±ð±,ð©ð©
6x1
6x6
What is the size?ðððð Ã ðððð
ðð Ã ðððð
Whatâs its size?
ðð Ã ðððð
ððððððð©ð©
=ð¥ð¥ 0 ðŠðŠ
0 ð¥ð¥ 0
0 1 0
ðŠðŠ 0 1
54
Why does it look like that?(slide 48)
KLT algorithm: computing âð©ð© = ð»ð»â1 âð±ð±âðð ð»ð»ðŒðŒ ððððððð©ð©
Tðð ð±ð± â ðŒðŒ ðð ð±ð±,ð©ð©
ððððððð©ð©
=ð¥ð¥ 0 ðŠðŠ
0 ð¥ð¥ 0
0 1 0
ðŠðŠ 0 1
What is the size?ðððð Ã ðððð
Whatâs its size?
ðð Ã ðððð
ðð Ã ðððð
6x1
6x66x1
55
Why does it look like that?(slide 48)
KLT algorithm: Discussion
Lucas-Kanade follows a predict-correct cycle
⢠A prediction ðŒðŒ ðð ð±ð±,ð©ð© of the warped image is computed from an initial estimate
⢠The correction parameter âð©ð© is computed as a function of the error ðð ð±ð± â ðŒðŒ ðð ð±ð±,ð©ð©between the prediction and the template
⢠The larger this error, the larger the correction applied
predict correct
56
KLT algorithm: Discussion
⢠How to get the initial estimate p?⢠When does the Lucas-Kanade fail?
⢠If the initial estimate is too far, then the linear approximation does not longer hold -> solution?
⢠Pyramidal implementations (see next slide)
⢠Other problems:⢠Deviations from the mathematical model: object deformations, illumination changes,
etc.⢠Occlusions⢠Due to these reasons, tracking may drift -> solution?
⢠Update the template with the last image
57
Coarse-to-fine estimation
image I
pI(W)warp refine
p Îp+
Pyramid of image I Pyramid of image T
image Tu=10 pixels
u=5 pixels
u=1.25 pixels
u=2.5 pixels
image T
58
Coarse-to-fine estimation
I I(W) Twarp refine
inp
pâ+
I I(W) Twarp refine
p
pâ+
I
pyramid construction
I I(W) Twarp refine
pâ+
T
pyramid construction
outp 59
Generalization of KLT
⢠The same concept (predict/correct) can be applied to tracking of 3D object (in this case, what is the transformation to etimate? What is the template?)
60
Generalization of KLT
⢠The same concept (predict/correct) can be applied to tracking of 3D object (in this case, what is the transformation to etimate? What is the template?)
⢠In order to deal with wrong prediction, it can be implemented in a Particle-Filter fashion (using multiple hipotheses that need to be validated)
61
Outline
⢠Point tracking⢠Template tracking⢠Tracking by detection of local image features
62
Tracking by detection of local image features
⢠Step 1: Keypoint detection and matching⢠invariant to scale, rotation, or perspective
63Template image with the object to detect Current test image
Tracking by detection of local image features
⢠Step 1: Keypoint detection and matching⢠invariant to scale, rotation, or perspective
64Template image with the object to detect Current test image
Tracking by detection of local image features
⢠Step 1: Keypoint detection and matching⢠invariant to scale, rotation, or perspective
⢠Step 2: Geometric verification (RANSAC) (e.g., 4-point RANSAC for planar objects, or 5 or 8-point RANSAC for 3D objects)
65Template image with the object to detect Current test image
Tracking by detection of local image features
66
Tracking issues
⢠How to segment the object to track from background?⢠How to initialize the warping?⢠How to handle occlusions⢠How to handle illumination changes and non modeled effects?
67
Readings
⢠Chapter 8 of Szeliskiâs book, 1st edition
68
Understanding Check
Are you able to answer the following questions?⢠Are you able to illustrate tracking with block matching?⢠Are you able to explain the underlying assumptions behind differential methods, derive their mathematical expression
and the meaning of the M matrix?⢠When is this matrix invertible and when not?⢠What is the aperture problem and how can we overcome it?⢠What is optical flow?⢠Can you list pros and cons of block-based vs. differential methods for tracking?⢠Are you able to describe the working principle of KLT?⢠What functional does KLT minimize?⢠What is the Hessian matrix and for which warping function does it coincide to that used for point tracking?⢠Can you list Lukas-Kanade failure cases and how to overcome them?⢠How do we get the initial guess?⢠Can you illustrate the coarse-to-fine Lucas-Kanade implementation?⢠Can you illustrate alternative tracking procedures using point features?
69