depth-based real time head motion tracking using 3d template matching
TRANSCRIPT
![Page 1: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/1.jpg)
Depth-Based Real Time Head Motion Tracking Using 3D Template Matching
以深度影像及三維樣板比對為基礎之即時頭部運動軌跡追蹤
學生:屠愚
指導教授:歐陽明博士
1
![Page 2: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/2.jpg)
Outline
• Introduction
• System Overview
• Algorithm
• Results & System Demo
• Conclusion & Future Work
2
![Page 3: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/3.jpg)
3
Introduction
![Page 4: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/4.jpg)
4
Y-a
xis
Yaw
Pitch
Roll
Y translatio
n
Real-time reconstruction of this 6-DoF motion vectorgiven a stream of video input.
What’s Head Motion Tracking
![Page 5: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/5.jpg)
5
AnimationVideo Game Gaze OrientationDancing
Why Head Motion Tracking
![Page 6: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/6.jpg)
• Color Image Based Methods“Head Pose Estimation in Computer Vision : A Survey”,
E. Murphy-Chutorian and M. M. Trivedi., PAMI 09
• Appearance Template Methods
• Feature Tracking Methods
• Detector Arrays
• Nonlinear Regression Methods
• Manifold Embedding Methods
• Flexible Models
• Geometric Methods
• Hybrid Methods
• Too sensitive to illumination variations! 6
Related Work
![Page 7: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/7.jpg)
• SwissRanger SR4000
– Messa Image (2006)
– $9000
• CameCube 2.0
– PMD Technology (2002)
– $12000
7
Depth Camera
![Page 8: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/8.jpg)
• Kinect by Microsoft– $149.99
• Xtion Pro by Asus– $189 .
– $300 .
8
Depth Camera
![Page 9: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/9.jpg)
9
“Real-time performance-based facial animation”T. Weise et al., SIGGRAPH 2011
“Real Time Head Pose Estimation with Random Regression Forests”, G. Fanelli et al., CVPR 2011
Related Work
![Page 10: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/10.jpg)
10
System Overview
![Page 11: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/11.jpg)
11
![Page 12: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/12.jpg)
12
![Page 13: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/13.jpg)
Avatar ControlDepth Data
Acquisition Real-time Head Pose Estimation
Least Square Error Method
Inverse Rotation
Nose Detection
Sampling
Iterative Optimization Method
13
User Acting Avatar ControlDepth Data
Acquisition Real-time Head Pose Estimation
Least Square Error Method
Inverse Rotation
Nose Detection
Sampling
Iterative Optimization Method
Flow Chart
![Page 14: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/14.jpg)
AlgorithmMethod 1 :Least square
error method
14
![Page 15: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/15.jpg)
15
Nose Detection
![Page 16: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/16.jpg)
16
Z-axis
Zmin+5
Zmin
Nose Detection
![Page 17: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/17.jpg)
17
Nose Detection
![Page 18: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/18.jpg)
Nose Detection
18
![Page 19: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/19.jpg)
Inverse Rotation
19
![Page 20: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/20.jpg)
20
Nose DetectionNose𝑡 𝑥, 𝑦
Nose𝑡−1 𝑥, 𝑦 ,
𝜃𝑡−1, 𝜓𝑡−1, 𝜙𝑡−1,Depth Map
𝑡Searching
Window
Setting
Inverse
Rotation
z-min
Searching
Nose Tracking Flow Chart
![Page 21: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/21.jpg)
21
Ax+By+Cz=1
Least Square Plane Fitting
Yaw & Pitch !
![Page 22: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/22.jpg)
23
Least Square Ellipse Fitting
Roll !
![Page 23: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/23.jpg)
Dynamic weighted average filter
24“Real-time performance-based facial animation” T. Weise et al., Siggraph 2011
Flickering Issue
𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑𝑤𝑒𝑖𝑔ℎ𝑡 𝜔
𝑓𝑟𝑎𝑚𝑒𝑖 𝑖-1 𝑖-2 𝑖-3 𝑖-4
0.010.1110
𝐻 ∙ 𝑚𝑎𝑥𝑙∈[1,𝑘] 𝑡𝑖 − 𝑡𝑖−𝑙
𝑤𝑗 = 𝑒−𝑗∙𝐻∙𝑚𝑎𝑥𝑙∈[1,𝑘] 𝑡𝑖−𝑡𝑖−𝑙𝑡𝑖
∗ = 𝑗=0𝑘 𝑤𝑗𝑡𝑖−𝑗
𝑗=0𝑘 𝑤𝑗
![Page 24: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/24.jpg)
AlgorithmMethod 2 :Optimization
method
25
![Page 25: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/25.jpg)
Avatar ControlDepth Data
Acquisition
26
User ActingDepth Data
Acquisition Real-time Head Pose Estimation
Least Square Error Method
Inverse Rotation
Nose Detection
Sampling
Iterative Optimization Method
Flow Chart
Avatar Control
![Page 26: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/26.jpg)
27
Model Point Cloud
![Page 27: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/27.jpg)
28
Sample Point Cloud
![Page 28: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/28.jpg)
• 𝑚𝑜𝑡𝑖𝑜𝑛 𝑣𝑒𝑐𝑡𝑜𝑟 𝑎 Θ,𝜓, 𝜙, 𝑡𝑥 , 𝑡𝑦, 𝑡𝑧
• 𝐸 𝑎, 𝑃𝑆, 𝑃𝑀 = 𝑝𝑝∈𝑃𝑆min(𝑑 𝑝′, 𝑃𝑀 )
• 𝑅𝑒𝑠𝑢𝑙𝑡 = 𝑎∗ = arg𝑚𝑖𝑛 (𝐸)
29
Energy Function
![Page 29: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/29.jpg)
30
• Gradient Decent Algorithm
Non-differentiable
Optimization
𝑎𝑘+1 = 𝑎𝑘 − 𝛼𝑘𝛻𝐸(𝑎𝑘)
• 12 Possible Gradients
𝛻1 =
100000
, 𝛻2 =
−100000
, 𝛻3 =
010000
, 𝛻4 =
0−10000
, 𝛻5 =
001000
, 𝛻6 =
00−1000
,
𝛻7 =
000100
, 𝛻8 =
000−100
, 𝛻9 =
000010
, 𝛻10 =
0000−10
, 𝛻11 =
000001
, 𝛻12 =
00000−1
![Page 30: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/30.jpg)
31
Optimization
![Page 31: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/31.jpg)
ResultsOf Method 1
32
![Page 32: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/32.jpg)
33
![Page 33: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/33.jpg)
34
![Page 34: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/34.jpg)
35
![Page 35: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/35.jpg)
ResultsOf Method 2
36
![Page 36: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/36.jpg)
37
![Page 37: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/37.jpg)
38
![Page 38: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/38.jpg)
39
![Page 39: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/39.jpg)
40
Our Method with Whole Database
Our Method with 70% Database (Without Crash)
Our Method with 50% Database (Without Bad Initialization)
![Page 40: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/40.jpg)
41
![Page 41: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/41.jpg)
System Demo
42
![Page 42: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/42.jpg)
Conclusion
43
![Page 43: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/43.jpg)
• Contribution:
– Two novel algorithms.
– Not affected by varying lighting conditions.
– Real-time responses without GPU acceleration.
– Outperforms the state-of-the-art approach (G. Fanelliet al., CVPR 2011).
• Future Work:
– Tracking without nose.
– Combine with NITE skeleton tracking.
– Facial Expression Recognition / Retargeting.
44
![Page 44: Depth-Based Real Time Head Motion Tracking Using 3D Template Matching](https://reader038.vdocuments.mx/reader038/viewer/2022102706/55a23d7b1a28ab146e8b4604/html5/thumbnails/44.jpg)
Thank You Very Much
45