multi-camera multi-human tracking system (oral presentation)
TRANSCRIPT
![Page 1: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/1.jpg)
Multi-Camera Multi-Human
Tracking System
Copyright © Yu-Sheng Chen [Yosen]
![Page 2: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/2.jpg)
Outline
Top view of the entire tracking system
What local trackers do
Identify people depth order
Handle body occlusion
What central server does
Compare observations across cameras
Classify to different people
Recognize people when re-entering
Yosen Chen, Copyright Reserved 2
![Page 3: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/3.jpg)
Images
Local camera
#A tracker
,
1
A
tz 2
A
tz
2
B
tz1
B
tzLocal camera
#B trackerImages
Central Server
multi-camera correspondence
Person#1 Person#2
Identify how many people
in the observed space
Shared data on Inter-process platform
, 2
A
tz 1
B
tz1
A
tz 2
B
tz
Listen/
Broadcast
![Page 4: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/4.jpg)
What local trackers do?
Multi-human
tracking &
detection
Central
Server
Operation flow of local trackers
Identify
people depth
order
If detect new,
broadcastGet people labels
& people heights.
Handle body
occlusionStart
Keep updating body info. to server databases
Extract
people body
info.
Local trackerCentral server
![Page 5: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/5.jpg)
What local trackers do: Identify depth order
A. By appearances
B. By standing locations
Image source
1:Front
2:Rear
![Page 6: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/6.jpg)
Depth order estimation by head shape completeness is a bad idea!
Yosen Chen, Copyright Reserved 6
![Page 7: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/7.jpg)
What local trackers do: Identify depth order
A. By appearances
B. By standing locations
Image source
1:Front
2:Rear
![Page 8: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/8.jpg)
Yosen Chen, Copyright Reserved 8
Tell the depth order by people’s
standing locations
How to know the standing locations?
To be discussed in the central server part!!
Bd Ground plane
Ad
Camera image
Cd
BdAd Cd depth order: A=1, B=2, C=3
A
BC
![Page 9: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/9.jpg)
Depth order estimation by 3D standing locations is accurate.
Yosen Chen, Copyright Reserved 9
![Page 10: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/10.jpg)
What local trackers do: Handle body occlusion Occluded parts can be estimated by
3D geometry with body modeling
![Page 11: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/11.jpg)
Occlusion Estimation by 3D Body
Modeling
Yosen Chen, Copyright Reserved 11
Ground plane
Camera image
B
AAB
Camera visible parts
Camera invisible parts
![Page 12: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/12.jpg)
Body Occlusion Handling
Yosen Chen, Copyright Reserved 12
Depth order
check
Body occlusion
handling
Target info
storage
Body appearance is available for
color extraction
Body appearance is not available
![Page 13: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/13.jpg)
What central server does?
Compare
observations
across cameras
Local
trackers
Operation flow of central server
Get observation
correspondence
between camerasIf detect new,
broadcast
Return people labels & people heights.Start
Keep updating body info. to server database
People
database
Compare with
database to recognize
/create people
Local trackerCentral server
![Page 14: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/14.jpg)
What central server does: Compare observations across
cameras Geometric-based color comparison
![Page 15: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/15.jpg)
2D position on image plane 3D line in real space
3D standing point in real space 2D feet position on image plane
observation r
Optic center Image plane of
camera j
Ground plane
Optic center
observation q
Image plane of
camerak
3D Geometric Correspondence across
Cameras
Calibrated camera system
Head Point
height
Estimated Standing Location
Yosen Chen, Copyright Reserved 15
![Page 16: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/16.jpg)
What central server does: Classify to different people Use decisions in higher confidence to
determine lower ones.
![Page 17: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/17.jpg)
Observation Correspondence
across Cameras
Object Conf = 3 Object Conf = 2
Matching body color (Block-based body color comparison)
Yosen Chen, Copyright Reserved 17
Different colors mean different labels
![Page 18: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/18.jpg)
Multi-People Correspondence
Yosen Chen, Copyright Reserved 18
1. Multi-people depth order √ 2. body occlusion handling √ 3. Multi-people correspondence √
![Page 19: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/19.jpg)
What central server does: Identify people if re-enter Construct people database once
they got tracked.
![Page 20: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/20.jpg)
Yosen Chen, Copyright Reserved 20
Check height
difference
How to recognize people when they re-enterAre they all from the same person?
Body Color
comparison
by block
Label history
consistency
No, if diff > threshold No, if color unmatched
Yes, they are from
the same person!!
Camera
observations
People model
in database
![Page 21: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/21.jpg)
Future Works…
Extend to more cameras
Challenge of
Communication load & algorithm complexity
Tracking/labeling stability
More challenges:
Outdoors Surveillance (human body detection)
Uncalibrated multi-camera system (Training)
Motive cameras (rotation 3D-2D mapping)
Yosen Chen, Copyright Reserved 21
![Page 22: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/22.jpg)
Thanks for Pf. Li-Chen Fu
Advanced Control Lab, NTU
Intelligent Robot Lab, NTU
![Page 23: Multi-Camera Multi-Human Tracking System (oral presentation)](https://reader033.vdocuments.mx/reader033/viewer/2022060111/5564ae57d8b42a366c8b54eb/html5/thumbnails/23.jpg)
Copyright © Yu-Sheng Chen [Yosen]