fusion of multiple cues from color and depth domains using occlusion aware bayesian tracker
DESCRIPTION
Fusion of Multiple Cues from Color and Depth Domains using Occlusion Aware Bayesian Tracker. Kourosh MESHGI Shin- ichi MAEDA Shigeyuki OBA Shin ISHII 18 MAR 2014. Integrated System Biology Lab (Ishii Lab) Graduate School of Informatics Kyoto University [email protected] - PowerPoint PPT PresentationTRANSCRIPT
Fusion of Multiple Cues from Color and Depth Domains using
Occlusion Aware Bayesian Tracker
Kourosh MESHGIShin-ichi MAEDAShigeyuki OBAShin ISHII
18 MAR 2014
Integrated System Biology Lab(Ishii Lab)Graduate School of InformaticsKyoto [email protected]
IEICE NC Tamagawa’14
TRACKING APPLICATIONS
K O U R O S H M E S H G I – I S H I I L A B - D E C 2 0 1 3 - S L I D E 2
MAIN APPLICATIONS
Surveillance Public Entertainment
Robotics Video Indexing
Action Recog.
TRACKING CHALLENGES
K O U R O S H M E S H G I – I S H I I L A B - D E C 2 0 1 3 - S L I D E 3
MAIN CHALLENGES
Varying ScaleClutterNon-Rigid
OcclusionIlluminationAbrupt Motion
[Mihaylova et al., 07] RGB Color + Texture + Motion + Edge, Two PF
[Spinello & Arras,11] HOG on RGB + HOG on Depth, SVM Classification
[Shotton et al, 11] Skeleton from Depth, Random Forrest
[Choi et al, 11] Ensemble of Detectors (upper body, face, skin, shape from depth, motion from depth), RJ-MCMC
LITERATURE REVIW(Channel Fusion & Occlusion)
[Song et al., 13] 2.5D Shape + Motion + HOG on Color and Depth, Occlusion Indicator, SVM
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 5
OBSERVATION MODELFrame: t
Observation
, ,{ , }t rgb t d tI I I
Image Patch
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 6
STATE REPRESENTATIONFrame: t
State
{ , , , }t t t t tB x y w h{ }t tX B
( ; )t t tY g I B
w
h
(x,y)
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 7
FEATURES
Feature Set1{ ,..., }nF f f
Color
Shape Edge
Texture
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 8
TEMPLATE INIT.Frame: 1
Template1 1,1 ,1{ ,..., }n
f1 fj fn
1 ,1 1{ }ni i
1 1{ ( )}if Y
… …
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 9
PARTICLES INITIALIZATIONFrame: 1
Particles, ,{ }k t k tX B1,2, ,k N
Initialized Overlapped
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 0
MOTION MODELFrame: t
Motion Model, , ,k t k t k tB B
→ t + 1
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 1
FEATURE EXTRACTIONFrame: t + 1
Feature Vectors , 1( )i k tf Y
f1 f2 fn
X1,t+1
X2,t+1
XN,t+1
…
…
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 1 2
FEATURE FUSIONFrame: t
Probability of Observation( | )t tp Y X ,1
( | , )ni t t i ti
p Y B ( | , )t t tp Y B 1, 2, ,( | , , , , )t t t t n tp Y B 1 1, ,( | , ) ( | , )t t t n t t n tp Y B p Y B ,1
( ),ni i i t i ti
p D f Y ,1
exp ( ),ni i i t i tiD f Y
,1
exp ( ),ni i i t i tiD f Y
Each Feature(.)if(.)iD
i Indepen
dence
Assumptio
n
!
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 3
PROB. CALCULATIONFrame: t + 1
Particles
Brighter = More Probable
,k tp
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 4
TARGET ESTIMATIONFrame: t + 1
Expectation 1 ,
ˆt k tB B E
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 5
MODEL UPDATEFrame: t + 1
New Model
Model Update
1ˆ ˆ( ; )t t tY g I B
1ˆ ˆ( )t i tf Y
, 1 , 1
,
ˆ
(1 )i t i i t
i i t
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 6
RESAMPLINGFrame: t + 1
Proportional to Probability
1( | )t tp X X1
2
345
67
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 7
APPEARANCE CHANGES
Same Color ObjectsBackground ClutterIllumination ChangeShadows, Shades
Use Depth!
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 8
MODEL DRIFT PROBLEM
Templates Corrupted! t
Handle Occlusion!
PERSISTENT OCCLUSION
Particles Converge to Local Optima / Remains The Same Region
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 9
Advanced Motion Models(not always feasible)
Restart Tracking(slow occlusion recovery)
Expand Search Area!
OCCLUSIONdo not address occlusion explicitly
maintain a large set of hypotheses
computationally expensive
direct occlusion detection robust against partial & temp occ. persistent occ. hinder tracking
GENERATIVE MODELS DISCRIMINATIVE MODELS
Dynamic Occlusion: Pixels of other object close to camera
Scene Occlusion: Still objects are closer to camera than the target object
Apparent Occlusion: Due to shape change, silhouette motion, shadows, self-occ
UPDATE MODEL FOR TARGET TYPE OF OCCLUSION IS IMPORTANT KEEP MEMORY VS. KEEP FOCUS ON THE TARGET
Combine
Them!
DYNAMICS…
* The Search is not Directed* Neither of the Channels have Useful Information* Particles Should Scatter Away from Last Known Position
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 1
Occlusion!
Occlusion Flag (for each particle)
Observation Model
No-Occlusion Particles Same as Before
Occlusion-Flagged Particles Uniform Distribution
OCCLUSION AWAREPARTICLE FILTER FRAMEWORK
( | ) ( | , , )t t t t t tp Y X p Y B Z ( | ) (1 ) ( | , 0, ) ( | , 1, )t t t t t t t t t t t tp Y X Z p Y B Z Z p Y B Z
,k tZ
( | , 1, ) 1t t t tp Y B Z
,1( | , 0, ) exp ( ),n
t t t t i i i t i tip Y B Z D f Y
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 2
Probability of Occlusion for the Next Box
Modified Dynamics Model of Particle
PARTICLE FILTER DYNAMICS
1 , , , , ,1 1ˆ ( ) ( ) ( | )N N
t t i t i t i t i t i ti iZ Z p Z Z p X Y Z
E
1 1 1 1 1( | ) ( , | , ) ( | ) ( | )t t t t t t t t t tp X X p B Z B Z p B B p Z Z
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 3
Model Update
Separately for each Feature
UPDATE RULE
11
1 1
ˆ( ) ,( )
ˆ ˆ( ) (1 ) ( ) ,t t occ
t
t t t occ
f Zf
f Y f Z
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 4
OA-PF DYNAMICS
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 5
Occlusion!
OA-PF DYNAMICS
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 6
Occlusion!
GOTCHA!
OA-PF DYNAMICS
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 7
Quick Occlusion Recovery Low CPE
No Template Corruption
No Attraction to other Object/ Background
CO
LO
R
(HO
C)
TE
XT
UR
E
(LB
P)
ED
GE
(L
OG
)
DE
PTH
(H
OD
)
3D SH
APE
(PC
L Σ)FEATURES
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 8
Princeton Tracking Dataset
DATASET( )
5 Validation Video with Ground Truth95 Evaluation Video
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 9
EXPERIMENT
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 0
A. Edge PF
B. Edge + Color PF
C. Edge + Color + Depth PF
D. Edge + Color + Depth + Texture PF
E. Edge + Color + Depth + Texture + 3D Shape PF
F. Occlusion Aware PF
DEMONSTRATION(Yellow Dashed Line is Ground Truth)
PASCAL VOC
CRITERIA I
1
1
*1
* *1 1 1
*1 1
*1 1
ˆ
ˆ ˆ, 0ˆ1 , 1ˆ1 ,
t
t
t
t t t
t t t
t t
B B
B B Z Z
S Z Z
Z Z
0 1ott oS t AUC
toSu
cces
sOverlap Threshold
0
1
1
Area Under Curve
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 2
RESULTS
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 3 3
Success Plot A D
B EC F
1
1Overlap Threshold
Succ
ess
Rat
e
Mean Central Point Error: Localization Success
Mean Scale Adaption Error
CRITERIA II
* 2 * 21
ˆˆ( ) ( )Tt t t tt
w w h hSAE
T
* 2 * 21
ˆ ˆ( ) ( )Tt t t tt
x x y yCPE
T
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 4
ˆˆ ˆ ˆ ˆ{ , , , }t t t t tB x y w h * * * * *{ , , , }t t t t t
B x y w h
Estimated Ground Truth
RESULTS
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 3 5
Center Positioning ErrorA D
B EC F
100
50Frames
CPE
(pix
els)
RESULTS
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 3 6
Scale Adaptation Error
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 3 6
140
SAE
(pix
els)
50Frames
A D
B EC F
RESULTS
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 7
Tracker AUC CPE SAE
A (edg)15.7
2192.27
41.62
B (edg+hoc)29.8
893.4
746.7
1
C (edg+hoc+hod)46.7
434.6
240.8
8D (edg+hoc+hod+tex)
48.49
30.18
46.27
E (edg+hoc+hod+tex+shp)
58.03
23.84
29.62
F (all + occlusion handling)
63.58
17.46
25.07
FUTURE WORKS
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 8
More Resilient Features + Scale
Adaptation
Active Occlusion Handling
Measure the Confidence of
each Data Channel
Adaptive Model Update
QUESTIONS?Thank you for your time…
Image Credit: http://www.engg.uaeu.ac.ae/