gesture recognition on smart camera - csic · vision based gesture recognition ... sign language...

&

www.cea.fr

Aziz Dziri, Stephane Chevobbe, Mehdi Darouich

CEA LIST – Embedded Computing Laboratory – France

[email protected]

Gesture recognition on smart camera

Cliquez pour modifier le style du titre

WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 2 &

Vision based gesture recognition

Targeted applications Smart buildings

Device control

Sign language recognition

Advantages Natural interaction

No additional expensive devices

Hitachi TV

Targeted systems Compact vision system

Low power < 10 W

Small size < 10 x 10 cm



Problematic

Constraints of embedded systems Input image : VGA grey level - QVGA RGB

Computing power : 50 – 700 MHz

Memory < 1 MByte

Existing gesture recognition algorithms Execution on PC platforms

Active sensor for some algorithms

Purpose : Study of the existing gesture recognition algorithms

Construction of gesture recognition pipelines

Evaluation of the pipelines on embedded processors

IcyCam by CSEM (44 mm2 – 80 mW)

Kinect camera

ToF camera



Overview

Context

State of art of gesture recognition algorithms

Implementation

Methodology and Results

Conclusions and future works



Overview

Context


Implementation





State of art

Hand gesture recognition pipeline Hand detection step

Gesture recognition step

Non-learning based methods

Learning based methods

Tracking step (not addressed)

Gesture recognition

Identified

gesture Features

extraction Classification

Hand detection

Input

image

Hand model

Segmentation localization

Learning

Hand

Region of Interest Gesture model

Learning

Hand tracking



State of art

Synthesis of the studied related work

Ref Detection method

Recognition method

(Classifier)

Recognition rate

(Number of gesture)

Image resolution

(Frame rate)

Size of feature vector

Computer

[1] Horimoto et al

2003

Background subtraction + skin color

threshold (Hue)

PCA (Nearest

neighbor)

83.7% (20)

160x120 (18 fps)

40 Celeron 1.3 GHz

[2] Zhao et al

2011

skin color thresholding (CbCr)+ Background

subtraction

HOG + PCA-LDA (Euclidean distance)

91% (10)

320x240 (15 fps)

9

2.67 GHz desktop

computer

[3] Liu et al

2009

Adaboost (learning machine)

invariant moments

(Multi-class SVM)

96.7% (7)

48x40 (-)

7

-

[4] Ankit et al

2009

Background subtraction

Finger tips detection

- (5)

- - 1.67GHz Intel Centrino Duo

processor

[5]Jae-Ho Shin et al 2006

Motion detection + entropy analysis

Hand edge in polar coordinate

95% (6)

320x240 (15 fps)

- Pentium PC



State of art

Complexity estimation 42x42 ROI size

HoG parameters [2]

Method ADD MUL DIV MOD ln sqrt arctan

With learning

HOG 11664 1296 5200 0 0 16 5184

Invariant Moment 49392 35321 8 0 6 1 0

Without learning

Finger tips detection 11 x m 12 x m m 2 x m 0 0 0

Hand edge in polar coordinate

3 x m + 5292

2 x m + 3528

m 0 0 m m

m = edge size



Gesture recognition pipelines

Learning based pipeline: Invariant Moment

Non-learning based pipeline: Finger tips detection

Color-based hand detection

Invariant Moment Recognition

Hand model

Gesture model

Finger Tips Detection

Gesture

Number of fingers


Hand model



Overview

Context


Implementation





Detection

Color

conversion

Skin pixel

selection

Skin

histogram

Non-Skin

histogram

Histograms

normalization

Maximum

likelihood

Skin model

Segmentation Labeling ROI selection ROI

Skin pixels

Input image

Detection

Learning



Filter

Off-line

On-line



Invariant Moments

Normalized

Central

Moments

Flusser

Invariant

Moments

Logarithm

transform ROI

gesture models

Manhattan

distance

Learning

Classification

Identified

gesture





Fingertips

Chain code Peaks detection ROI Number of finger tips

K contour points K contour points


Finger Tips detection



Overview

Context


Implementation





Methodology

Code transformations Floating point data coding not supported by some embedded processors

Floating point coding conversion to Fixed point coding

Maximum dynamic 64 bits

Operator transformation

Maximum resolution of Region of Interest

Invariant Moment : QQVGA (160x120)

Finger tips detection: QVGA (320x240)

Gesture recognition

Hand detection

Input

image

Identified

gesture

Gesture model

Hand model

Segmentation localization Features

extraction Classification

Learning

Hand

Region of Interest

Learning



Methodology

Methodology Quality vs execution time evaluation

C code

Camera

OpenCV

Visualization

Display

Intermediate

results

Processors

ARM - ANTX Tools

Execution

time

Fixed-point

PC Tools Quality

- Reference

- Degradation

Floating-point

ARM Cortex A9 : One core embedded processor without FPU ANTX : Low footprint control oriented processor without HW multiplier

OpenCV

acquisition

Hand

gesture

database

Fixed-point



Database

10 gestures - 40 occurrences



Results : Invariant Moments

Quality results : QQVGA (160x120) images ; 1 to 9 gestures

Execution performances 250 ms (4 fps) on ANTX @ 400 MHz

33ms (30 fps) on ARM Cortex A9 @ 400 MHz

Memory capacity < 150 kB



Results : Finger tips detection

Quality results QVGA images (320x240)

0 to 5 fingers to count

Reference (Floating point) : 80 % recognition rate

Embedded version (Fixed-point) : 79 % recognition rate

Execution performances 500ms (2 fps) on ANTX @ 400 MHz

100ms (10 fps) on ARM Cortex A9 @ 400 MHz

Memory capacity < 200 kB



Overview

Context


Implementation






Two pipelines suitable for embedded context Learning based pipeline

Hand detection: Color based method

Gesture recognition: invariant moments

Non-learning based pipeline


Gesture recognition: finger tips detection

Evaluation of both quality level and performances Invariant moments pipeline :

70% recognition rate for 4 gestures

4 to 30 fps

150 kB

Finger Tips pipeline :

79% recognition rate (0 to 5 fingers)

2 to 10 fps

200 kB

Quality : - Better classification step: Manhattan distance may be too simple - Use of peaks and valleys for finger tips detection




Two pipelines suitable for embedded context Learning based pipeline


Gesture recognition: invariant moments

Non-learning based pipeline


Gesture recognition: finger tips detection

Evaluation of both quality level and performances Invariant moments pipeline :

70% recognition rate for 4 gestures

4 to 30 fps

150 kB

Finger Tips pipeline :

79% recognition rate (0 to 5 fingers)

2 to 10 fps

200 kB

Performances : - Optimization and parallelization of the code

-Tracking step



Next step …

Implementation of the pipelines on single chip vision system

IcyCam by CSEM (44 mm2 – 80 mW)



Hand model

Gesture model

Finger Tips Detection

Gesture

Number of fingers


Hand model

Centre de Grenoble 17 rue des Martyrs

38054 Grenoble Cedex

Centre de Saclay Nano-Innov PC 172

91191 Gif sur Yvette Cedex

Thank you Questions ?



References

[1] Horimoto, S., Arita, D., and Taniguchi, R., “Real-time hand shape recognition for human interface," in [Image Analysis and Processing, 2003.Proceedings. 12th International Conference on], 20-25 (sept. 2003).

[2] Zhao, Y., Wang, W., and Wang, Y., “A real-time hand gesture recognition method," in [Electronics, Communications and Control (ICECC), 2011 International Conference on], 2475-2478 (sept. 2011).

[3] Liu, Y. and Zhang, P., “Vision-based human-computer system using hand gestures," in [Computational Intelligence and Security, 2009. CIS '09. International Conference on], 2, 529-532 (dec. 2009).

[4] Ankit, G. and Kumar, A. P., “Finger tips detection and gesture recognition," tech. rep., Indian Institute of Technology, Kanpur (2009).

[5] Jae-Ho Shin, J.-S. L., “Hand region extraction and gesture recognition using entropy analysis," A 6, 216-222, IJCSNS (February 2006).

gesture recognition on smart camera - csic · vision based gesture recognition ... sign language...

Documents