Rita Cucchiara, Simone Calderara, Francesco Solera
Imagelab
DIPARTIMENTO DI INGEGNERIA «Enzo Ferrari»
Università di Modena e Reggio Emilia, Italia
http://www.imagelab.ing.unimore.it
The challenge of tracking Social Groups in Crowd
UNIMORE University of Modena and Reggio Emilia
Post-CVPR AC Meeting Workshop on Recent trends in computer vision University of Maryland , Feb 2014
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
ImageLab: current projects..
Computer Vision Pattern Recognition and machine learning
Multimedia
Surveillance (NATO, EU projects, collaboration with companies, SMART TOURISM project Emilia R. Region with S.Calderara)
Ego-Vision (collaboration with ETHZ with G.Serra)
Animal behavior (collaboration
with Italian Health Ministry with S.Calderara)
Document analysis (collaboration with Treccani Italian Enciclopedy and Miniature Libs with C.Grana)
Medical imaging (EU projects in dermatology , C.Grana)
Sensing floors (collaboration
with FLORIM spa with R.Vezzani)
Web image retrieval for cultural heritage (ITALIAN PON
Project EU-FESR DICET)
Natural interaction for children ( Cluster Project smart city CITYEDU)
Current PROJECTS
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
In Smart City Projects..
Goal Understand what the people want/their intentions in the city while they are:
ALONE IN GROUPS IN THE CROWD
TOURISTIC TOURS – CULTURE ENTERTAINMENT – CHILDREN IN SCHOOLS
From surveillance to human behavior analysis….
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
Reasoning about CROWDS
What is a CROWD? a large number of persons gathered closely together
What does LARGE mean?
We are working on crowds where single person and
groups can be recognized.
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
Before understanding groups..
Many challenges:
What we are doing at Imagelab:
• Detecting single people
• Tracking single people
• Tracking multiple people
• Working on trajectories (or tracklets)
• Recognizing (socially consistent) groups in crowd
• By shape classification
• By trajectory analysis
ENVI-VISION EGO-VISION
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
Detecting people ..
Pedestrian detectors a long story… • Detectors: Dalal, Triggs CVPR05, Felzenszwalb, CVPR08, Gavrila et al PAMI09……….
• Benchmarks: Dollar et al CVPR09
• Search modes : Lampert et al CVPR08
• Detection in crowd: Ge Collins PETS09, Li et al. CVPR13
• Detection and tracking in crowd: Rodriguez et al. ICCV11
• Survey: Dollar et al TPAMI11…
Improving speed and accuracy “Multi-Stage Particle Windows for
Fast and Accurate Object Detection”
[Gualdi, Prati, Cucchiara TPAMI12]
form sliding windows to particle windows
search for people (and other targets)
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
..and tracking single (people) target
Is tracking a solved problem? Another long story from L.Davis W4 CVPR98 ICIAP99…..
We tried to answer this questions in an “experimental evaluation”
Even in case of single target tracking*
- a very large dataset
of 14 categories of challenges
- a large set of performance measures
- a large experimentation
(with code available over 3 clusters in 3 labs)
* D.Chu, A.Smeulders, S.Calderara, R.Cucchiara, A. Dehghan, M.Shah Visual Tracking: an Experimental Survey [TPAMI 2013]
MOTA; OTA; Deviaton…. F-Measure SURVIVAL CURVES..
19 trackers BASELINES STATE OF THE ART
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
19 Trackers
A. Tracking by Matching
B. Tracking by Matching with extended model (ST memory)
C. Tracking by Matching with constraints
D. Tracking by Discriminant Classification
E. Tracking by discriminant Classification
with constraints
• [NCC] Normalized Cross-Correlation K. Briechle and U. Hanebeck, SPIE 2001
• [KLT] Lucas-Kanade Tracker S. Baker and I. Matthews, IJCV2004
• [KAT] Kalman Appearance Tracker H. Nguyen and A. Smeulders, TPAMI 2004
• [FRT] Fragments-based Robust Tracking A. Adam, E. Rivlin, and I. Shimshoni, CVPR2006
[MST] Mean Shift Tracking D. Comaniciu, V. Ramesh, and P. Meer, CVPR2000
• [LOT] Locally Orderless Tracking S. Oron, A. Bar-Hillel, D. Levi, S. Avidan, CVPR2012
• [IVT] Incremental Visual Tracking D. Ross, J. Lim, and R.S.Lin, IJCV2008
• [TAG] Tracking on the Affine Group J. Kwon and F.C. Park, CVPR2009
• [TST] Tracking by Sampling Trackers J. Kwon, K.M. Lee, ICCV 2011
• [TMC] Tracking by Monte Carlo sampling J. Kwon, K.M. Lee,CVPR 2009
• [ACT] Adaptive Coupled-layer Tracking L. Cehovin, M. Kristan, A. Leonardis, ICCV2011
• [L1T] L1-minimization Tracker X. Mei and H. Ling, ICCV2009
• [L1O] L1 Tracker with Occlusion detection X. Mei, H. Ling, Y. Wu, E. Blasch, L. Bai, CVPR2011
• [MIT] Multiple Instance learning Tracking B. Babenko, M.H. Yang, and S. Belongie, CVPR2009
• [TLD] Tracking, Learning and Detection Z. Kalal, J. Matas, and K. Mikolajczyk, CVPR2010
• [FBT] Foreground-Background Tracker H. Nguyen and A. Smeulders, 2006, IJCV2010
• [HBT] Hough-Based Tracking M. Godec, P.M. Roth, H.Bischof, ICCV2011
[SPT] Super Pixel tracking S. Wang, H. Lu, F. Yang, M.H. Yang, ICCV2011
• [STR] STRuck S. Hare, A. Saffari, P. Torr, ICCV2011
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
14 tracking challenges in 313 videos
01-LIGHT
02-SURFACECOVER
03-SPECULARITY
04-TRANSPARENCY
05-SHAPE
06-MOTIONSMOOTHNESS
07-MOTIONCOHERENCE
08-CLUTTER
09-CONFUSION
10-LOWCONTRAST
11-OCCLUSION
12-MOVINGCAMERA
13-ZOOMINGCAMERA
14-LONGDURATION
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
http://www.alov300.org or http://imagelab.ing.unimo.it/dsm
The dataset: an example
email to [email protected]
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
A comprehensive view Survival curve
A B C D E
[NCC]
[STR]
[L1O]
[TST]
[TLD]
[FBT]
The upper bound, taking the best of all trackers at each frame 10%
The lower bound, what all trackers can do 7%
About the 30%, correctly tracked only
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
Confusion challenge: trackers comparison
[FBT][NCC][STR] [TLD][TST] [L1O]
CONFUSION.. CROWD short term tracking
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
Long term challenge: trackers comparison
[FBT][NCC][STR] [TLD][TST] [L1O]
We need more effort Welcome to “Long term tracking workshop” at CVPR2014
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
What we learned?
Many observations…
• In cluttered and confusion scenes, Tracking-by-detection methods that use data association , based on discriminative classifiers seem to be promising…..
• Moving from single target to multiple targets in long term cannot be done with multiple instances of a good single-target tracker
• State of the art papers
• Discrete –continue optimization Andriyenko et al CVPR2012
• Continue energy minimization Milan and Roth PAMI2014
• Generalized minim clique Zamire et al ECCV2012
• K-shortest path optimization Berclaz et al PAMI 2011
What do they all have in common?
They are data association techniques that work on already detected pedestrians
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
Work in progress…
Cognitive Visual Tracking with latent structural svms • From neuroscience : two (connected but different) areas for detection ( people,
faces..) and spatio temporal localization (independently by their shapes) • From perceptual psychology : the “object file” theory (Kahnemann, Treisman, Gibbs 1995)
• We use distance only when is possible • Motion prediction and appearance is a plus when useful
Thus?
1. Split the crowd in influence zones (latent knowledge)
2. Decide whether those zones are ambiguous (also latent)
3. Solve unambiguous associations with distance only
4. Employ different level features in ambiguous cases ( ask for shapes, color.. edges.. motion)
http://imagelab.ing.unimore.it/files2/RitaWashington/video/influencezones_tracking.avi
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
Detection and tracklets
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
Survival curve
[KSP] Multiple Object Tracking using K-Shortest Paths Optimization J. Berclaz, F. Fleuret, E. Türetken and P. Fua, PAMI 2011
With a perfect detector
With a detector with errors
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
Groups of People
If tracking was solved…
If we were given the trajectories of every pedestrian in the scene (more or less). would we be able to discern the presence of groups?
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
Detecting social groups in crowds
Group detection: learn to partition into groups the pedestrians being part of a crowd observing pairwise relations and transitivities.*
Integrating two cues:
* Structured learning for detection of social groups in crowd Solera, Calderara, Cucchiara, AVSS 2013
2. GRANGER CAUSALITY • Intuition: two pedestrian belonging to the
same group will probably influence each other position and direction!2
• The Granger causality test is a statistical hypothesis test for determining whether one time series is useful in forecasting another
1. HALL’S PROXEMICS • Hall’s proxemics theory 1 defines reaction
bubbles around every individual and
• the interaction between pairs of individuals can be classified according to a quantization of their mutual distance
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
Results
Features: Proxemics and Granger causality Structure function: pair-wise correlation clustering Group detection: Structured SVM [groups]
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
Conclusions and Open Problems
• Single target Detection & Tracking
• tracking is (still) an open problem
• computer visionaries are working a lot.. ( also in the weekend )
• Multiple target tracking
• more and more challenging ( more if real-time is required)
• tracking-by-detection
• Cognitive assumptions are useful
• Detection Social groups and interactions
• interesting and growing topic
• Many many many applications
• Social hypotheses Must be considered
.
People/ group Detection
People/ group tracking
People/ group Detection
People/ group tracking
People Detection
People/ group Tracking
Social group Detection
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia
Thanks
PEOPLE @ http://imagelab.ing.unimore.it
Rita Cucchiara Giuseppe Serra Marco Manfredi
Costantino Grana Paolo Santinelli Francesco Solera
Roberto Vezzani Martino Lombardi Simone Pistocchi
Simone Calderara Michele Fornaciari Fabio Battilani
Dalia Coppi Patrizia Varini Augusto Pieracci
THANKS!
UN
IMO
RE
Un
iversity o
f Mo
den
a and
Reggio
Emilia