1

A Robust Method of Detecting Hand Gestures Using

Depth Sensors

Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang

Haptic Audio Visual Environments and Games (HAVE), 2012 IEEE International Workshop on

2

Outline

IntroductionRelated WorksThe Proposed MethodExperimental ResultsConclusion

3

Introduction

4

Introduction

In human-computer interaction(HCI) system, recognizing hand and finger gestures are significant. Medical system, computer games, and human-robot

Depth-sensing camera(Kinect, Xtion) add a dimension to increase accuracy.

Goal: detect hand gestures with color and depth information

5

Related Works

6

Related Works

Body [4][5] V.S. HandHand

Superiority: simple Inferiority: small scale, low resolution Strict condition: cluttered background, lighting

variation

7

Related Works

Hand gesture recognition Only color[12]

Data glove[7]

Training process[9][10]

Earth Mover’s Distance(EMD)[11]

8

References [7] R. M. Satava, “Virtual reality surgical simulator,” Surgical Endoscopy, vol. 7, pp.

203–205, 1993. [9] C. Keskin, F. Kirac, Y. Kara, and L. Akarun, “Real time hand pose estimation

using depth sensors,” in Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, nov. 2011.

[10] P. Doliotis, A. Stefan, C. McMurrough, D. Eckhard, and V. Athitsos, “Comparing gesture recognition accuracy using color and depth information,” in Proceedings of the 4th International Conference on Pervasive Technologies Related to Assistive Environments, ser. PETRA ’11. New York, NY, USA: ACM, 2011, pp. 20:1–20:7.

[11] Z. Ren, J. Yuan, and Z. Zhang, “Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera,” in Proceedings of the 19th ACM international conference on Multimedia, ser. MM ’11. New York, NY, USA: ACM, 2011, pp. 1093–1096.

[12] A. Argyros and M. Lourakis, “Real-time tracking of multiple skincolored objects with a possibly moving camera,” in Computer Vision -ECCV 2004, ser. Lecture Notes in Computer Science. Springer Berlin/ Heidelberg, 2004, vol. 3023, pp. 368–379.

9

The Proposed Method

10

The Proposed Method

Gesture Representation

Finger RecognitionFind convex hull Detect fingertip and direction

Hand SegmentationFind hands through color

Separate hands by k-means

Find palm center

11

Hand Segmentation (I)

Find hands through color Train skin-color[12], detect face[15], image filtering[16],

color threshold L*a*b color space

b Operate AND on the two images Minimum depth (10cm)

12

Hand Segmentation (I)--Find hands through color

RGB image Depth image L*a*b color space where b = 2

L*a*b color space where b = 3

Skin color images after ANDoperation

Binary image of hand segmentation

13

Hand Segmentation (II)

Separate hands by k-means k=2 Assignment:

Update:

Threshold of distance between 2 clusters

14

Hand Segmentation (II)--Separate Hands By K-means

15

Hand Segmentation (III)

Find palm center Inscribed circle Minimum inner distance

Maximum element of inner distances set

16

Finger Recognition (I)

Find convex hull Graham’s scan algorithm

P: the lowest y-coordinate Sort in increasing order of angle Point to point is left/right turn Left-turn: O ; right-turn: X

17

Finger Recognition (II)

Detect fingertip and direction Fingers are long and narrow Find an isosceles triangle with V

V: Every vertex on the convex hull Set a maximum threshold to the vertex angle The direction vector is paralleled with the median

length of an isosceles triangle

18

Finger RecognitionBlue point : cluster centroid Green point : palm centerRed points : fingertipsYellow curves : hand contourLong lines : finger directions Structures around the hand : convex hull

19

The Proposed Method

Gesture Representation

Finger RecognitionFind convex hull Detect fingertip and direction

Hand SegmentationFind hands through color

Separate hands by k-means

Find palm center

20

Gesture Representation

All information about hands Palm center location Finger number Fingertips location Finger direction vectors

Gestures Rock-paper-scissors game Drag images Grasping, releasing

21

Experimental Results

22

Experimental Results

Use Kinect as input of depth and color images The detection successful rate can reach 95%.No matter the hand is horizontally or vertically

placed.

23

24

Experimental Results

25

Experimental Results--Shadow Puppetry

26

Conclusion

27

Conclusion

Present a new method to detect hands’ positions and gestures

NO training, NO database Future works

Set a threshold to the distance between the palm center and the fingers

Add additional sensor devices to overcome no palm detection

Shadow Puppetry project