![Page 1: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/1.jpg)
Cristian Sminchisescu (University of Toronto)
Bill Triggs (INRIA Rhone-Alpes)
Kinematic Jump Processes for Monocular 3D Human Tracking
![Page 2: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/2.jpg)
Goal: track human body motion in monocular video and estimate 3D joint motion
Why Monocular ?• Movies, archival footage
• Resynthesis, e.g. change point of view or actor• Tracking / interpretation of actions & gestures (HCI)• How do humans do this so well?
![Page 3: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/3.jpg)
Overall Modeling Approach1. Generative Human Model
– Kinematics, geometry, photometry– Predicts images or descriptors– Priors and anatomical constraints
2. Model-image matching cost function– Robust, probabilistically motivated– Contour and intensity based
3. Tracking by search / optimization– Discovers well supported
configurations of matching cost
![Page 4: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/4.jpg)
Why is 3D-from-monocular hard?
Image matching ambiguities
Depth ambiguities
Violations of physical constraints
![Page 5: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/5.jpg)
How many local minima are there?
Thousands ! – even without image matching ambiguities …
![Page 6: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/6.jpg)
Examples of Kinematic Ambiguities
• Minima are separated by large distances in parameter space
![Page 7: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/7.jpg)
Monocular 3D Tracking Methods• CONDENSATION (discrete, motion models)
– Deutscher et al.’00: annealing, walking– Sidenbladh et al.’00,02: importance sampling (walking + snippets)
• CSS, ET/HS/Hyperdynamics (continuous, cost-sensitive)– Sminchisescu&Triggs’01,02
Covariance Scaled Sampling (CSS)
HyperdynamicsHypersurface Sweeping (HS)
![Page 8: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/8.jpg)
Search Globality and Adaption• Cost sensitive continuous search methods are
– Efficient - avoid large wastage factors with random sampling– Generic - no assumptions on known motions
• Focus on locating transition states and nearby minima
• But– Still local (i.e. sometimes myopic)
• Minima are typically far in parameter space
– No knowledge of global long-range minimum structure
• Want to search quasi-globally, yet preserve generality– Can we find other minima more efficiently by exploiting
intrinsic problem structure?
![Page 9: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/9.jpg)
Kinematic Jump Sampling
• For any given model configuration, we can explicitly build the interpretation tree of alternative kinematic solutions with identical joint projections– work outwards from root of kinematic tree, recursively
evaluating forward/backward ‘flips’ for each body part• Alternatively, sample by generating flips randomly • … or, for tracking, sample shallowly and treat each limb quasi-
independently
![Page 10: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/10.jpg)
Efficient Inverse Kinematics• The inverse kinematics is
simple, efficient to solve– Constrained by many
observations (3D articulation centers)
– The quasi-spherical articulation of the body
– Mostly in closed form
• The iterative solution is also very competitive • Optimize over model-hypothesized 3D joint assignments • 1 local optimization work per new minimum found
An adaptive diffusion method (CSS) is necessary for correspondence ambiguities
![Page 11: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/11.jpg)
Candidate Sampling Chains
s=CovarianceScaledSampling(mi)
S=BuildInterpretationTree (s,C)
E=InverseKinematics(S)
Prune and locally optimize E
1tp
M
i
N
jijj CC
1 1
][v)(vote
tp
),( iiim
C=SelectSamplingChain(mi)
E
C1 CMC
The KJS Algorithm
![Page 12: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/12.jpg)
Tracking Experiments
• 4s agile dancing sequence, 25 frames per second
• Cluttered background, self-occlusion, motion in depth
• Automatically select kinematic jump samples (KJS) from short 3-link chains (rooted at hips, shoulders, neck)
• 8 modes, CSS diffusion with scaling 4
![Page 13: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/13.jpg)
Jump Sampling in Action
![Page 14: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/14.jpg)
Quantitative Search Statistics
• Initialize in one minimum, different sampling regimes• Improved minima localization by KJS
– Local optimization often not necessary
![Page 15: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/15.jpg)
Summary• Kinematic Jump Sampling Algorithm
– Construct interpretation trees of 3D joint positions corresponding to monocular kinematic ambiguities
– Solve efficiently using closed-form inverse kinematics
• Highly accurate hypothesis generator for long-range search
• Local optimization polishing often un-necessary
– Explicit kinematic jumps + cost-sensitive sampling
• Address both depth and image matching ambiguities
• Future work– Scene constraints (ground plane, equilibrium)
– Jump strategies for image matching
– Prior knowledge (Sminchisescu&Jepson03 upcoming)
![Page 16: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/16.jpg)
The End
![Page 17: Kinematic Jump Processes for Monocular 3D Human Tracking](https://reader031.vdocuments.mx/reader031/viewer/2022012922/56813513550346895d9c6753/html5/thumbnails/17.jpg)
The End