2011-07-19 robust tracking using constrains and context
TRANSCRIPT
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
1/147
Helmut Grabner
Ftan, 2011/07/19
Robust Tracking using VisualConstrains and Context
ETH-Zurich, Computer Vision Lab 1H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
2/147
Put cards on the table: I will NOT talk about
ETH-Zurich, Computer Vision Lab 2H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
3/147
BUT, I will talk about
ETH-Zurich, Computer Vision Lab 3H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
4/147
We all know what tracking is, right?
[Grabner et al., VideoProc CVPR06]
ETH-Zurich, Computer Vision Lab 4H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
5/147
We all know what tracking is, right?
from time t to t+1Time t find again
ETH-Zurich, Computer Vision Lab 5H. Grabner, Robust Tracking using Visual Constrains and Context
ac ua o ec pos on
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
6/147
What to track?
ETH-Zurich, Computer Vision Lab 6H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
7/147
What to track?
centerpoint
ETH-Zurich, Computer Vision Lab 7H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
8/147
What to track?
multiplepoints
ETH-Zurich, Computer Vision Lab 8H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
9/147
Example application: Structure From Motion
[Pollefeyes et al., IJCV 2004]
ETH-Zurich, Computer Vision Lab 9H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
10/147
What to track?
(body)parts
ETH-Zurich, Computer Vision Lab 10H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
11/147
Example application: Game interface
[Shotton et al., CVPR 2011]
ETH-Zurich, Computer Vision Lab 11H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
12/147
What to track?
region
ETH-Zurich, Computer Vision Lab 12H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
13/147
What to track?
outline
ETH-Zurich, Computer Vision Lab 13H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
14/147
What to track?
structure
ETH-Zurich, Computer Vision Lab 14H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
15/147
Example application: Augmented reality
[Wagner et al., ISMAR09]
ETH-Zurich, Computer Vision Lab 15H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
16/147
In this talk
ETH-Zurich, Computer Vision Lab 16H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
17/147
OBJECT
ETH-Zurich, Computer Vision Lab 17H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
18/147
Why is it hard?
initialization
ETH-Zurich, Computer Vision Lab 18H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
19/147
Why is it hard?
initialization
fast movement
ETH-Zurich, Computer Vision Lab 19H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
20/147
Why is it hard?
initialization
fast movement
nonlinear dynamics
ETH-Zurich, Computer Vision Lab 20H. Grabner, Robust Tracking using Visual Constrains and Context
Wrong prediction
Correctprediction
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
21/147
Why is it hard?
initialization
fast movement
nonlinear dynamics
ETH-Zurich, Computer Vision Lab 21H. Grabner, Robust Tracking using Visual Constrains and Context
mu t p e s m arobjects
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
22/147
Why is it hard?
initialization
fast movement
nonlinear dynamics
ETH-Zurich, Computer Vision Lab 22H. Grabner, Robust Tracking using Visual Constrains and Context
mu t p e s m ar o ects background clutter
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
23/147
Why is it hard?
initialization
fast movement
nonlinear dynamics
ETH-Zurich, Computer Vision Lab 23H. Grabner, Robust Tracking using Visual Constrains and Context
mu t p e s m ar o ects background clutter
appearance changes
(model drift)
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
24/147
Why is it hard?
initialization
fast movement
nonlinear dynamics
ETH-Zurich, Computer Vision Lab 24H. Grabner, Robust Tracking using Visual Constrains and Context
mu t p e s m ar o ects background clutter
appearance changes
(model drift) occlusions/ self
occlusions
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
25/147
In this talk
ETH-Zurich, Computer Vision Lab 25H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
26/147
OBJECT
CONSTRAINS
ETH-Zurich, Computer Vision Lab 26H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
27/147
Before we actually start:Two Tracking paradigm
ETH-Zurich, Computer Vision Lab 27H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
28/147
(i) Traditional tracking looppredict to t+1
time t
measure at t+1
update location
update model
ETH-Zurich, Computer Vision Lab 28H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
29/147
(ii) Tracking-by-detection
time t time t+1 time t+n
(i) detect object(s)
independently in each frame
(ii) associate detections overtime into tracks
ETH-Zurich, Computer Vision Lab 29H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
30/147
Using constrains and context
Prediction/Correction(model-free-tracking)
Tracking-by-detection
predict to t+1 measure at t+1
information about the object dynamics
scene context (e.g., geometry)
ETH-Zurich, Computer Vision Lab H. Grabner, Robust Tracking using Visual Constrains andContext
30
time t
update locationupdate model
time t time t+1time t+n
build and adapt the detector
learning (refining) a model during tracking
exploit relations to other objects
object and its surrounding
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
31/147
Tracking-by-detection
model of the object (object category) is given
ETH-Zurich, Computer Vision Lab 31H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
32/147
SCENE CONTEXT
[Grabner et al., Photogrammetry & Remote Sensing07]
[Klucker et al., ICCV07 WS on 3dRR]
time t time t+1 time t+n
build and adapt the detector
ETH-Zurich, Computer Vision Lab 32H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
33/147
How to get the detections?
Nemos Background
Supervised LearningETH-Zurich, Computer Vision Lab 33H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
34/147
Fewer Clicks - Less Frustration
Image
Detections
On-linelearning algorithm
updated classifier
Labels
ETH-Zurich, Computer Vision Lab 34H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
35/147
ETH-Zurich, Computer Vision Lab 35H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
36/147
Minimize the human labeling effort
Detections
Image
Labels
updated classifier
On-linelearning algorithm
ETH-Zurich, Computer Vision Lab 36H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
37/147
Exploiting the 3d structure
Mutliple Images
Depth map
ETH-Zurich, Computer Vision Lab 37H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
38/147
A 3d teacher
Detections
Image
Labels
updated classifier
On-linelearning algorithm
verify detections
(no big depth discontinuities)
ETH-Zurich, Computer Vision Lab 38H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
39/147
Results
ETH-Zurich, Computer Vision Lab 39H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
40/147
Results
ETH-Zurich, Computer Vision Lab 40H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
41/147
Results
ETH-Zurich, Computer Vision Lab 41H. Grabner, Robust Tracking using Visual Constrains and Context
L 1
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
42/147
LESSON LEARNED 1
Visual constrains saves
you time!
ETH-Zurich, Computer Vision Lab 42H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
43/147
SCENE CONTEXT
information about the object dynamics
[Stalder et al., ECCV10]
time t time t+1 time t+n
scene context (e.g., geometry)
ETH-Zurich, Computer Vision Lab 43H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
44/147
ETH-Zurich, Computer Vision Lab 44H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
45/147
Detection
Tracking
Input image
Behaviours
Workflow
StatisticsReasoning
ETH-Zurich, Computer Vision Lab 45H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
46/147
Challenges of an industrial environment
ETH-Zurich, Computer Vision Lab 46H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
47/147
Challenges of an industrial environment
Occlusions
Self Occlusions
Other movin Ob ects
Similar Objects
Industrial working
conditions
Real-time
ETH-Zurich, Computer Vision Lab 47H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
48/147
State-of-the-art person detector
i-Lids Scovis-NISSAN
P(p
erson)
ETH-Zurich, Computer Vision Lab 48H. Grabner, Context in Tracking - ETH-Zurich, Computer Vision Lab
H. Grabner, Robust Tracking using Visual Constrains andContext
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
49/147
reasonablygood detector
Use unlabled datato specialize!
ETH-Zurich, Computer Vision Lab 49H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
50/147
How to interpret unlabeled data?
Machine Learning?
Transfer learning?
Learning to Learn?
ETH-Zurich, Computer Vision Lab 50H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
51/147
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
52/147
Unlabeled data
Thomas Bayes1702-1761
ETH-Zurich, Computer Vision Lab 52H. Grabner, Robust Tracking using Visual Constrains and Context
Further Assumptions!
(e.g., manifold assumption,cluster assumption,)
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
53/147
Additional information/constraints
Geometry and structure Objects are embedded intocontext
Temporal consistency Typical behavior
ETH-Zurich, Computer Vision Lab 53H. Grabner, Robust Tracking using Visual Constrains and Context
LESSON LEARNED 2
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
54/147
LESSON LEARNED 2
Vision is more than pure
machine learning!
ETH-Zurich, Computer Vision Lab 54H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
55/147
Input to our System
Input image Object detector Confidence map
ETH-Zurich, Computer Vision Lab 55H. Grabner, Robust Tracking using Visual Constrains and Context
[Dalal and Triggs, CVPR05]
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
56/147
Geometric Filter
Confidence map Geometric constrains Confidence map
ETH-Zurich, Computer Vision Lab 56H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
57/147
Background Filter
Confidence volume Background modeling Confidence volume
ETH-Zurich, Computer Vision Lab 57H. Grabner, Robust Tracking using Visual Constrains and Context
[Stauffer and Grimson, CVPR99]
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
58/147
Trajectory Filter
Confidence volume Vessel-Filtering Confidence volume
ETH-Zurich, Computer Vision Lab 58H. Grabner, Robust Tracking using Visual Constrains and Context
[Frangi et al., 98]
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
59/147
Particle Filter (cont. Trajectories)
Confidence volume Particle filter Confidence volume
ETH-Zurich, Computer Vision Lab 59H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
60/147
Input
Cascaded ConfidenceFiltering
Geometry Filter
Background Filter
Trajectory Filter
ETH-Zurich, Computer Vision Lab 60H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
61/147
Results
ETH-Zurich, Computer Vision Lab 61H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
62/147
ETH-Zurich, Computer Vision Lab 62H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
63/147
ETH-Zurich, Computer Vision Lab 63H. Grabner, Robust Tracking using Visual Constrains and Context
T ki b D t ti
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
64/147
Input Sequence
Object Detector
Object DetectionsData Assocciation
Tracking-by-Detection
Tracking-by-Detection
[Huang et al., ECCV08]ETH-Zurich, Computer Vision Lab 64H. Grabner, Robust Tracking using Visual Constrains and Context
Object Detections
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
65/147
Input Sequence
CCF
Object Detections
Data AssocciationTracking-by-Detection
Confidence Map
ETH-Zurich, Computer Vision Lab 65H. Grabner, Robust Tracking using Visual Constrains and Context
Improved tracking by detection
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
66/147
Improved tracking-by-detection
ETH-Zurich, Computer Vision Lab 66H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
67/147
ETH-Zurich, Computer Vision Lab 67H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
68/147
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
69/147
ETH-Zurich, Computer Vision Lab 69H. Grabner, Robust Tracking using Visual Constrains and Context
[Leibe et al., PAMI08]
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
70/147
Model-free-Object Tracking
No object model is given
ETH-Zurich, Computer Vision Lab 70H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
71/147
LOCAL CONTEXT
predict to t+1 measure at t+1
time t
update locationupdate model
object and its surrounding
[Grabner et al., CVPR06, BMVC06]ETH-Zurich, Computer Vision Lab 71H. Grabner, Robust Tracking using Visual Constrains and Context
Objects are not isolated!
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
72/147
Objects are not isolated!
Learning current object appearance vs. local background.
currentac groun
currentobj. appearance
ETH-Zurich, Computer Vision Lab 72H. Grabner, Robust Tracking using Visual Constrains and Context
Tracking as binary classification probelm
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
73/147
Tracking as binary classification probelm
object
background
vs.
ETH-Zurich, Computer Vision Lab 73H. Grabner, Robust Tracking using Visual Constrains and Context
Tracking as binary classification probelm
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
74/147
Tracking as binary classification probelm
object
background
vs.
ETH-Zurich, Computer Vision Lab 74H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
75/147
search Region
actual object position
from time t to t+1 evaluate classifier on sub-patches
-
+
- -
-
create confidence mapanalyze map and set new
object positionupdate classifier (tracker)
ETH-Zurich, Computer Vision Lab 75H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
76/147
ETH-Zurich, Computer Vision Lab 76H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
77/147
ackin
g
Simple
t
ETH-Zurich, Computer Vision Lab 77H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
78/147
invis
ibvle
Tr
ackingth
ETH-Zurich, Computer Vision Lab 78H. Grabner, Robust Tracking using Visual Constrains and Context
LESSON LEARNED 4
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
79/147
Foreground/ background
discrimination reducesambiguities.
ETH-Zurich, Computer Vision Lab 79H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
80/147
Tracking Solved
ETH-Zurich, Computer Vision Lab 80H. Grabner, Robust Tracking using Visual Constrains and Context
Does it fail?
If yes, when?
When does it fail
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
81/147
ETH-Zurich, Computer Vision Lab 81H. Grabner, Robust Tracking using Visual Constrains and Context
When does it fail
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
82/147
ETH-Zurich, Computer Vision Lab 82H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
83/147
search Region
actual object position
from time t to t+1 evaluate classifier on sub-patches
-
+
- -
-
create confidence mapanalyze map andset new objectposition
update classifier (tracker)
Self-learning
ETH-Zurich, Computer Vision Lab 83H. Grabner, Robust Tracking using Visual Constrains and Context
Drifting
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
84/147
Tracked Patches Confidence
ETH-Zurich, Computer Vision Lab 84H. Grabner, Robust Tracking using Visual Constrains and Context
LESSON LEARNED 5
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
85/147
Self-learning drifting!
ETH-Zurich, Computer Vision Lab 85H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
86/147
TRACKING AND DETECTION
predict to t+1 measure at t+1
time t
update locationupdate model
learning (refining) a model during tracking
[Grabner et al., ECCV08]ETH-Zurich, Computer Vision Lab 86H. Grabner, Robust Tracking using Visual Constrains and Context
Review: Self-learning
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
87/147
ETH-Zurich, Computer Vision Lab 87H. Grabner, Robust Tracking using Visual Constrains and Context
Tracking is a semi-supervised problem!
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
88/147
Un-labeled
data
ETH-Zurich, Computer Vision Lab 88H. Grabner, Robust Tracking using Visual Constrains and Context
Labeled data
Tracking is a semi-supervised problem!
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
89/147
Current Model
Fix (initial) ModelPrior
ETH-Zurich, Computer Vision Lab 89H. Grabner, Robust Tracking using Visual Constrains and Context
Semi-Supervised Tracking
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
90/147
Prior
ETH-Zurich, Computer Vision Lab 90H. Grabner, Robust Tracking using Visual Constrains and Context
Labeled data
Un-labeleddata
Tracking Loop evaluate classifier on
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
91/147
Tracking Loop
search Region
actual object position
evaluate classifier onsub-patches
Prior
create confidence mapupdate classifier (tracker)
??
??
ETH-Zurich, Computer Vision Lab 91H. Grabner, Robust Tracking using Visual Constrains and Context
Robust Tacking
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
92/147
ETH-Zurich, Computer Vision Lab 92H. Grabner, Robust Tracking using Visual Constrains and Context
Robust Tracking
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
93/147
ETH-Zurich, Computer Vision Lab 93H. Grabner, Robust Tracking using Visual Constrains and Context
Drifting
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
94/147
CLICK HERE TO START
ETH-Zurich, Computer Vision Lab 94H. Grabner, Robust Tracking using Visual Constrains and Context
LESSON LEARNED 6
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
95/147
Tracking is a semi-
ETH-Zurich, Computer Vision Lab 95H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
96/147
Tracking Solved
ETH-Zurich, Computer Vision Lab 96H. Grabner, Robust Tracking using Visual Constrains and Context
Does it fail?
If yes, when?
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
97/147
ETH-Zurich, Computer Vision Lab 97H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
98/147
Prior is toon ri . irf
ETH-Zurich, Computer Vision Lab 98H. Grabner, Robust Tracking using Visual Constrains and Context
to similar objects)
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
99/147
ETH-Zurich, Computer Vision Lab 99H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
100/147
Prior is toor ri iv
ETH-Zurich, Computer Vision Lab 100H. Grabner, Robust Tracking using Visual Constrains and Context
(e.g., partialocclusions)
LESSON LEARNED 7
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
101/147
The prior (detector) is-
supervisedmachine learning.
ETH-Zurich, Computer Vision Lab 101H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
102/147
SCENE CONTEXT
predict to t+1 measure at t+1
time t
update locationupdate model
learning (refining) a model during tracking II
[Kalal et al. CVPR10]
[Stalder et al. ICCV09 WS on OLCV]ETH-Zurich, Computer Vision Lab 102H. Grabner, Robust Tracking using Visual Constrains and Context
Review: Self-Learning
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
103/147
Too adaptive
ETH-Zurich, Computer Vision Lab 103H. Grabner, Robust Tracking using Visual Constrains and Context
Review: Semi-Supervised Tracking
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
104/147
Limit drifting but maybe too
restrictive/general
ETH-Zurich, Computer Vision Lab 104H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
105/147
Unlabeled data
Thomas Bayes1702-1761
ETH-Zurich, Computer Vision Lab 105H. Grabner, Robust Tracking using Visual Constrains and Context
Vision is more than puremachine learning!
LESSON LEARNED 2
Prior Model = Detector
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
106/147
Non adaptive at all!
Obj. Det.
ETH-Zurich, Computer Vision Lab 106H. Grabner, Robust Tracking using Visual Constrains and Context
cf. Tracking-by-DetectionObj. Det.
Adapt the Prior using Visual Constrains!
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
107/147
new object samples
Obj. Det.
ETH-Zurich, Computer Vision Lab 107H. Grabner, Robust Tracking using Visual Constrains and Context
new non-objectsamples
Obj. Det.Background
Object Detectornon adaptive
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
108/147
Visual ConstrainsBuilding/ Refining amodel usin labeled
ETH-Zurich, Computer Vision Lab 108H. Grabner, Robust Tracking using Visual Constrains and Context
-Prior (Detector) with
unlabelled updates
data
Object Trackeradaptive (self-learning)
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
109/147
ETH-Zurich, Computer Vision Lab 109H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
110/147
ETH-Zurich, Computer Vision Lab 110H. Grabner, Robust Tracking using Visual Constrains and Context
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
111/147
ETH-Zurich, Computer Vision Lab 111H. Grabner, Robust Tracking using Visual Constrains and Context
LESSON LEARNED 8
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
112/147
Use visual constrains for
detector.
ETH-Zurich, Computer Vision Lab 112H. Grabner, Robust Tracking using Visual Constrains and Context
OTHER OBJECTS
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
113/147
OTHER OBJECTS
redict to t+1 measure at t+1
exploit relations to other objects
[Grabner et al., CVPR11]
time t time t+1 time t+n
time t
update locationupdate model
ETH-Zurich, Computer Vision Lab 113H. Grabner, Robust Tracking using Visual Constrains and Context
Im Carl Track me
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
114/147
ETH-Zurich, Computer Vision Lab 114H. Grabner, Robust Tracking using Visual Constrains and Context
Tracking Carl
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
115/147
ETH-Zurich, Computer Vision Lab 115H. Grabner, Robust Tracking using Visual Constrains and Context
Goal: Estimate the Position of an Object
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
116/147
Track the Object with occlusions outside the image
ETH-Zurich, Computer Vision Lab 116H. Grabner, Robust Tracking using Visual Constrains and Context
Many temporary, but potential verystrong links exists between a tracked
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
117/147
strong links exists between a trackedobject and parts of the images.
We discover the dynamic elements called SUPPORTERSthat predict the position of the target.
ETH-Zurich, Computer Vision Lab 117H. Grabner, Robust Tracking using Visual Constrains and Context
SUPPORTERS
came with different strength.
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
118/147
change over time.
ETH-Zurich, Computer Vision Lab 118H. Grabner, Robust Tracking using Visual Constrains and Context
SUPPORTERS
came with different strength.
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
119/147
change over time.
ETH-Zurich, Computer Vision Lab 119H. Grabner, Robust Tracking using Visual Constrains and Context
SUPPORTERS help Tracking of
bj t hi h h
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
120/147
objects which changethere appearance very
quickly.
occluded objects or
object outside the image.
small and/or low
textured objects or evenvirtual points.
ETH-Zurich, Computer Vision Lab 120H. Grabner, Robust Tracking using Visual Constrains and Context
Local Image Features as Supporters
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
121/147
ETH-Zurich, Computer Vision Lab 121H. Grabner, Robust Tracking using Visual Constrains and Context
Local Image Features as Supporters
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
122/147
ETH-Zurich, Computer Vision Lab 122H. Grabner, Robust Tracking using Visual Constrains and Context
Local Image Features as Supporters
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
123/147
ETH-Zurich, Computer Vision Lab 123H. Grabner, Robust Tracking using Visual Constrains and Context
Local Image Features as Supporters
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
124/147
Object FeaturesBackgroundFeatures
ETH-Zurich, Computer Vision Lab 124H. Grabner, Robust Tracking using Visual Constrains and Context
Local Image Features as Supporters
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
125/147
Supporters
ETH-Zurich, Computer Vision Lab 125H. Grabner, Robust Tracking using Visual Constrains and Context
Local Image Features as Supporters
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
126/147
Supporters
ETH-Zurich, Computer Vision Lab 126H. Grabner, Robust Tracking using Visual Constrains and Context
SUPPORTERS are features which
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
127/147
contribute to the prediction of the
target position.
They at least temporarily move in away which is statistically related to the
motion of the object.
ETH-Zurich, Computer Vision Lab 127H. Grabner, Robust Tracking using Visual Constrains and Context
Discovering the
Supporters
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
128/147
Predictedpostion
Reliablemeasurments
Model &learning
ETH-Zurich, Computer Vision Lab 128H. Grabner, Robust Tracking using Visual Constrains and Context
Reliable Information & Motion Coupling
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
129/147
x(i)t MotionCoupling
x*
t
ETH-Zurich, Computer Vision Lab 129H. Grabner, Robust Tracking using Visual Constrains and Context
Supporter
Strong Supporter
Strong motion coupling
Weak Supporter
Weak motion couplingAlmost Unrelated
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
130/147
Strong motion coupling
Peaky vote
Weak motion coupling
Blurred voteFeatures
ETH-Zurich, Computer Vision Lab 130H. Grabner, Robust Tracking using Visual Constrains and Context
Experimental Results: ETH-CupSequenze
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
131/147
ETH-Zurich, Computer Vision Lab 131H. Grabner, Robust Tracking using Visual Constrains and Context
ETH-Cup: Humans
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
132/147
ETH-Zurich, Computer Vision Lab 132H. Grabner, Robust Tracking using Visual Constrains and Context
ETH-Cup: Of the Web Tracker
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
133/147
ETH-Zurich, Computer Vision Lab 133H. Grabner, Robust Tracking using Visual Constrains and Context
ETH-Cup: Our Result Voting Space
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
134/147
ETH-Zurich, Computer Vision Lab 134H. Grabner, Robust Tracking using Visual Constrains and Context
ETH-Cup: Improving Object Tracking
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
135/147
10 Humans
( 2 )2 )2 )2 )
Tracker[Stalder et al, OLCV09]
45 % 100 %
+ supporter 89 % 97 %
ETH-Zurich, Computer Vision Lab 135H. Grabner, Robust Tracking using Visual Constrains and Context
Please note, a simple interpolation would not work.
ETH-Cup: Supportes
Supporter
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
136/147
Reliable
Information
ETH-Zurich, Computer Vision Lab 136H. Grabner, Robust Tracking using Visual Constrains and Context
Predicted Position(Mode of the Voting Space)
ETH-Cup: Supporters
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
137/147
ETH-Zurich, Computer Vision Lab 137H. Grabner, Robust Tracking using Visual Constrains and Context
Beyond the Image
Voting Space Supporters
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
138/147
ETH-Zurich, Computer Vision Lab 138H. Grabner, Robust Tracking using Visual Constrains and Context
Changing Supporter
Voting Space Supporters
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
139/147
ETH-Zurich, Computer Vision Lab 139H. Grabner, Robust Tracking using Visual Constrains and Context
Appearance Change
Voting Space Supporters
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
140/147
Zurich, 2010/06/03 140H. Grabner, Tracking the Invisible - ETH-Zurich, Computer Vision LabETH-Zurich, Computer Vision Lab 140H. Grabner, Robust Tracking using Visual Constrains and Context
Coupled Motion
Voting Space Supporters
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
141/147
ETH-Zurich, Computer Vision Lab 141H. Grabner, Robust Tracking using Visual Constrains and Context
Changing Supporters
Voting Space Supporters
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
142/147
ETH-Zurich, Computer Vision Lab 142H. Grabner, Robust Tracking using Visual Constrains and Context
Obviously, there are failure cases
Voting Space Supporters
. and magician knows that.
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
143/147
ETH-Zurich, Computer Vision Lab 143H. Grabner, Robust Tracking using Visual Constrains and Context
LESSON LEARNED 9
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
144/147
Look for supporters!
They are common and
take many forms.
ETH-Zurich, Computer Vision Lab 144H. Grabner, Robust Tracking using Visual Constrains and Context
OBJECT
Summary & Conclusion
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
145/147
CONSTRAINS
ETH-Zurich, Computer Vision Lab 145H. Grabner, Robust Tracking using Visual Constrains and Context
OBJECT
Summary & Conclusion
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
146/147
CONSTRAINS
ETH-Zurich, Computer Vision Lab 146H. Grabner, Robust Tracking using Visual Constrains and Context
Acknowledgments
Centere of MachineP ti
Computer VisionLaboratory
-
8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context
147/147
Severin StalderLuc Van GoolJiri Matas
PreceptionCzech Technical
University
LaboratoryETH Zurich
Michael Grabner Christian LeistnerHorst Bischof
Institute for Computer Graphics and VisionGraz, University of Technology, Austria
ETH-Zurich, Computer Vision Lab 147H. Grabner, Robust Tracking using Visual Constrains and Context
Philippe Cattin
Medical ImageAnalysis Center
University of Basel