3d fingertip and palm tracking in depth image sequences
DESCRIPTION
3D Fingertip and Palm Tracking in Depth Image Sequences. Hui Liang , Junsong Yuan and Daniel Thalmann. Proceedings of the 20th ACM international conference on Multimedia 2012. Outline. Introduction Related Work Proposed Method Experimental Results Conclusion. Introduction. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/1.jpg)
3D Fingertip and Palm Tracking in Depth Image SequencesHui Liang, Junsong Yuan and Daniel Thalmann
Proceedings of the 20th ACM international conference on Multimedia 2012
![Page 2: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/2.jpg)
2
Outline• Introduction • Related Work• Proposed Method• Experimental Results• Conclusion
![Page 3: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/3.jpg)
3
Introduction
![Page 4: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/4.jpg)
4
Introduction• Human hand is an essential body part for human-computer
interaction.
• The positions of tracked fingertips: gesture estimation
• Difficulty in fingertip tracking:
Side-by-side Bending Nearby
![Page 5: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/5.jpg)
5
Introduction• Many previous methods:
• Only focus on extracting 2D fingertips• Cannot track fingertips robustly for a freely moving hand
• In this paper:
• Present a robust fingertip and palm tracking scheme• With the input of depth images (KINECT)• Track the 3D fingertip positions quite accurately
![Page 6: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/6.jpg)
6
Related Work
![Page 7: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/7.jpg)
Related work• Focus only on 2D fingertips:[4][5][6][9]
• Based on contour analysis of the extracted hand region:[2][4][5][6]
• Usually can track the fingertips for only stretched fingers.
![Page 8: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/8.jpg)
Related work• In [6],
• Fingertips are tracked for infrared image sequences.• It utilizes a template matching strategy • Fingertip tracking : Kalman filter
• In [2],
• Stereoscopic vision is adopted• Maximize the distance center of gravity of the hand & the boundary
![Page 9: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/9.jpg)
Related work• In [9] (Kinect),
Depth < Threshold Circular filter
Minimum depth
[9] J. L. Raheja, A. Chaudhary, and K. Singal. Tracking of fingertips and centres of palm using kinect, 2011. Proc. Of the 3rd Int’l Conf. on Computational Intelligence, Modelling and Simulation.
![Page 10: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/10.jpg)
Related work• [2] S. Consei1, S. Bourennane, and L. Martin. Three dimensional fingertip tracking in
stereovision, 2005. Proc. of the 7th Int’l Conf. on Advanced Concepts for Intelligent Vision Systems.
• [4] K. Hsiao, T. Chen, and S. Chien. Fast fingertip positioning by combining particle filtering with particle random diffusion, 2008. Proc. IEEE Int’l Conf. on Multimedia and Expo.
• [5] I. Katz, K. Gabayan, and H. Aghajan. A multi-touch surface using multiple cameras, 2007. Proc. of the 9th Int’l Conf. on Advanced concepts for intelligent vision systems.
• [6] K. Oka, Y. Sato, and H. Koike. Real-time tracking of multiple fingertips and gesture recognition for augmented desk interface systems, 2002. Proc. IEEE Int’l Conf. on Automatic Face and Gesture Recognition.
• [9] J. L. Raheja, A. Chaudhary, and K. Singal. Tracking of fingertips and centres of palm using kinect, 2011. Proc. Of the 3rd Int’l Conf. on Computational Intelligence, Modelling and Simulation.
![Page 11: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/11.jpg)
11
ProposedMethod
![Page 12: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/12.jpg)
12
OverviewForeground
Segmentation
PalmLocalization
HandSegmentation
Fingertip Detection
Fingertip Tracking
Stereo images(Depth map)
Fingertips
![Page 13: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/13.jpg)
13
Hand and Palm Detection• 1) Assume the hand is the nearest object
• 2) Constrain global hand rotation by:
• : global rotation angle of the hand
ForegroundSegmentation
PalmLocalization
HandSegmentation
![Page 14: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/14.jpg)
14
• Threshold the depth frame to obtain the foreground F :
• p : a pixel coordinate • z(p) : depth value (of point p )• z0 : the minimum depth value
• zD : depth threshold
Hand and Palm DetectionForeground
SegmentationPalm
LocalizationHand
Segmentation
foreground F
0.2m
![Page 15: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/15.jpg)
15
• The palm region is approximated with a circle:
• pp : the palm center (of point p )
• rp : the radius
• Assume that hand palm forms a globally largest blob• Cp equals to the largest inscribed circle of the contour of F .• 2D Kalman filter : reduce computational complexity
Hand and Palm DetectionForeground
SegmentationPalm
LocalizationHand
Segmentation
![Page 16: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/16.jpg)
16
• Separate hand and forearm by a line:
• 1) Tangent to Cp
• 2) Perpendicular to the orientation of the forearm
• Orientation of the forearm :• The Eigenvector that corresponds to the largest
Eigenvalue of the covariance matrix of the contour pixel coordinates of F
Hand and Palm DetectionForeground
SegmentationPalm
LocalizationHand
Segmentation
Hand region : FV(2D) → FD(3D)
![Page 17: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/17.jpg)
17
Fingertip Detection and Tracking• Constraints on possible fingertip locations:
• 1) Only in depth discontinuous region ( in contour Fv)• 2) | Depth(one point) – Depth(neighborhoods) | are important.• 3) Utilize the 3D geodesic shortest path (GSP)
Fingertip vs. Non-fingertip
Nearby Fingertips
FingertipDetection
Fingertip Tracking
![Page 18: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/18.jpg)
18
Fingertip detectionFingertip tracking
• Goal: detect all five fingertips in the depth image• Based on three depth-based features
• Build a graph G :
• Vh : contains of all points within FV (hand contour)
• Eh : for each pair of vertices(p,q), 1) they are in the 8-neighborhood of each other 2) their 3D distance is within threshold τ
Fingertip Detection
Edge weight : 3D Euclidean distance
![Page 19: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/19.jpg)
19
• Calculate Geodesic distance dg(p):• From palm center pp for each vertex Vh
• Dijkstra graph search on Gh
• GSP point set Ug(p):• The set of vertices on the shortest path from pp to p
• Rectangle local feature RL(p):• Describe the neighborhood of a point p in FV
•
• : ratio of 1s
Fingertip Detection
0 0 11 1 1 1 11 1 X 1 01 1 1 1
0 1 0
Fingertip detectionFingertip tracking
1cm
![Page 20: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/20.jpg)
20
• Calculate Geodesic distance dg(p):
Fingertip Detection
0.4
Fingertip detectionFingertip tracking
dg(p)
𝜂(𝑝)
fingertips
![Page 21: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/21.jpg)
21
• Fingertip labeling:
Fingertip Detection
:estimate the probability that has the label lj
number of GSP points kth GSP point
1 2 3 4 5
1
2
3
4
5i : fingertip
j : label
Fingertip detectionFingertip tracking
Nmax=6
![Page 22: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/22.jpg)
22
• Fingertip labeling:
Fingertip DetectionFingertip detectionFingertip tracking
![Page 23: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/23.jpg)
23
Fingertip Tracking• Build a particle filter for each fingertip
• (x, ω) denote a particle• x : 2D position in FV • ω : the particle weight
• Constrain the positions of each particle to the border point set UB to reduce the search space
Fingertip detectionFingertip tracking
![Page 24: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/24.jpg)
24
• Likelihood function :
•
Metric parameters
difference
Hausdorff difference
feature difference
Fingertip Tracking Fingertip detectionFingertip tracking
/
/
Geodesicdistance
GSP points
Neighbordepth
pre now
dg(p)
𝑹𝑳(𝒑)
![Page 25: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/25.jpg)
25
Fingertip Tracking• Likelihood function :
Fingertip detectionFingertip tracking
![Page 26: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/26.jpg)
26
ExperimentalResults
![Page 27: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/27.jpg)
27
Experimental Results• Quantitative results on synthetic sequences:
Seq. No. Motion Seq.
No. Motion
Seq. 1 grasping Seq. 4 flexion
Seq. 2 adduction/abduction Seq. 5 global rotation
Seq. 3 successive single finger Seq. 6 combination of grasping and global rotation
‧Error : Euclidean distance‧Ground truth : phalanx end point
![Page 28: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/28.jpg)
28
Experimental Results
![Page 29: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/29.jpg)
29
Experimental Results• Virtual object grasping:
![Page 30: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/30.jpg)
30
Conclusion
![Page 31: 3D Fingertip and Palm Tracking in Depth Image Sequences](https://reader036.vdocuments.mx/reader036/viewer/2022081502/56815f45550346895dce20dc/html5/thumbnails/31.jpg)
31
Conclusion• Using multiple depth-based features for accurate fingertip
localization
• Adopting a particle filter to track the fingertips over successive frames
• Track the 3D positions of fingertips robustly
• Great potential for extension to other HCI applications