knowledge systems lab jn 8/24/2015 a method for temporal hand gesture recognition joshua r. new...
TRANSCRIPT
![Page 1: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/1.jpg)
JN 04/19/23
Knowledge Systems Lab
A Method for Temporal Hand Gesture Recognition
Joshua R. NewKnowledge Systems Laboratory
Jacksonville State University
![Page 2: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/2.jpg)
JN 04/19/23
Knowledge Systems Lab
Outline
• Terminology
• Motivation
• Current Research and Applications
• System Overview
• Implementation Approach
• Demonstration
![Page 3: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/3.jpg)
JN 04/19/23
Knowledge Systems Lab
Terminology
Image Processing - Computer manipulation of images. Some of the many algorithms used in image processing include convolution (on which many others are based), edge detection, and contrast enhancement.
Computer Vision - A branch of artificial intelligence and image processing concerned with computer processing of images from the real world. Computer vision typically requires a combination of low level image processing to enhance the image quality (e.g. remove noise, increase contrast) and higher level pattern recognition and image understanding to recognize features present in the image.
![Page 4: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/4.jpg)
JN 04/19/23
Knowledge Systems Lab
Motivation
![Page 5: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/5.jpg)
JN 04/19/23
Knowledge Systems Lab
Motivation
• Gesturing is a natural form of communication– Gesture naturally while talking– Babies gesture before they can talk
• Interaction problems with the mouse– Have to locate cursor– Hard for some to control (Parkinsons or
people on a train)– Limited forms of input from the mouse
![Page 6: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/6.jpg)
JN 04/19/23
Knowledge Systems Lab
Motivation (2)
• Interaction Problems with the Virtual Reality Glove– Reliability– Always connected – Encumbrance– Sanitation
![Page 7: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/7.jpg)
JN 04/19/23
Knowledge Systems Lab
System Overview
![Page 8: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/8.jpg)
JN 04/19/23
Knowledge Systems Lab
System Overview
StandardWeb Camera
Rendering
User InterfaceDisplay
Han
d
Movem
ent
User
Gesture Recognition
System
ImageCapture
Update Object
Image Input
![Page 9: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/9.jpg)
JN 04/19/23
Knowledge Systems Lab
System Overview (2)
• System:• OpenCV and IPL libraries (from Intel)
• Input:• 640x480 video image• Hand calibration measure
• Output:• Rough estimate of centroid• Refined estimate of centroid• Number of fingers being held up• Manipulation of 3D skull in QT interface in response to
gesturing
![Page 10: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/10.jpg)
JN 04/19/23
Knowledge Systems Lab
System Overview (3)
• Hand Calibration Measure:• Max hand size in x and y orientations in # of pixels
![Page 11: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/11.jpg)
JN 04/19/23
Knowledge Systems Lab
System Overview (4)
Saturation Channel Extraction (HSL space):
Original ImageOriginal Image Hue
Lightness
Saturation
![Page 12: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/12.jpg)
JN 04/19/23
Knowledge Systems Lab
Proposed Approach
![Page 13: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/13.jpg)
JN 04/19/23
Knowledge Systems Lab
Proposed Approach
![Page 14: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/14.jpg)
JN 04/19/23
Knowledge Systems Lab
Proposed Approach (2)
![Page 15: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/15.jpg)
JN 04/19/23
Knowledge Systems Lab
Proposed Approach (3)
)(*19.0 HandSizeYHandSizeXRadius
• The finger-finding function sweeps out a circle around the rCoM, counting the number of white and black pixels as it progresses
• A finger is defined to be any 10+ white pixels separated by 17+ black pixels (salt/pepper tolerance)
• Total fingers is number of fingers minus 1 for the hand itself
![Page 16: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/16.jpg)
JN 04/19/23
Knowledge Systems Lab
Proposed Approach (4)
• Temporal Recognition System for hand movement patterns
• Input feature-vector creation: based on calculations from 12 centroid locations (hand movement during 3 seconds – average length of temporal gesture)
• Learning system training and recognition: SFAM classification used for ability to interactively add new, user-defined gestures
![Page 17: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/17.jpg)
JN 04/19/23
Knowledge Systems Lab
Proposed Approach (5)
• Temporal Gesture Recognition System
![Page 18: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/18.jpg)
JN 04/19/23
Knowledge Systems Lab
Proposed Approach (6)
Input Feature-Vector Creation• Twelve centroids collected (x,y)• Find top-left centroid so that gestures are not start-point
dependent (a square is a square whether you start at the top left or the bottom right)
• Compute centroid differences to recognize movement, not position (a square whether the hand is at the precise 12 points or in between those points)
• Normalize using the perimeter to recognize percentage of total movement since users are inaccurate in repeating a gesture (a square whether large or small)
Note: System still sensitive to clockwise vs. counter-clockwise square (undo-like feature)
![Page 19: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/19.jpg)
JN 04/19/23
Knowledge Systems Lab
Proposed Approach (7)
Learning System Training and Classification1. Feature vector is formatted for use by SFAM2. Movements of the hand were recorded and assembled into
one file for training the SFAM system- 25 examples each of circle, square, left/right, up/down
3. System classification of hand gesture every 3 seconds4. Train New Gesture button provided, stores gesture under the
label entered in the box (5 is the default since 1-4 are already taken)
![Page 20: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/20.jpg)
JN 04/19/23
Knowledge Systems Lab
Proposed Approach
![Page 21: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/21.jpg)
JN 04/19/23
Knowledge Systems Lab
Demonstration
SystemConfiguration
SystemGUI Layout
![Page 22: Knowledge Systems Lab JN 8/24/2015 A Method for Temporal Hand Gesture Recognition Joshua R. New Knowledge Systems Laboratory Jacksonville State University](https://reader031.vdocuments.mx/reader031/viewer/2022032705/56649ddb5503460f94ad183b/html5/thumbnails/22.jpg)
JN 04/19/23
Knowledge Systems Lab
Demonstration (2)
Gesture to Interaction Mapping
Number of Fingers:
2 – Roll Left3 – Roll Right
4 – Zoom In5 – Zoom Out