GESTURAL COMMUNICATIONS WITH ACCELEROMETER-BASED INPUT DEVICES AND MULTI-MODAL DISPLAYS

Paul D. VarcholikACTIVE LaboratoryInstitute for Simulation and TrainingUniversity of Central Floridapvarchol@ist.ucf.edu

James L. MerloLTC, US ArmyUS Military AcademyWest Pointjames.merlo@usma.edu

Gestural Communication

Problem Statement Reliable communications between

military personnel is critical

Hand gestures are used when vocal means are inadequate

Line of sight issues may cause visual signals to be unreliable

Our Research Purpose

To determine the usefulness of a computer mediated gesturing recognition system for non-visual communication

ScopeProvide a proof of concept which would lay the groundwork for future research

Research Questions Have we developed a recognition system capable of

accurately converting and transmitting a visual communication mode into a non-visual form?

Do computer-mediated gestures provide a viable form of non-visual communication?

Input Device

Nintendo Wiimote

Nintendo Wiimote 3-axis accelerometer Wireless (Bluetooth) Inexpensive COTS 100Hz Sampling Rate

Output Devices

Tactile Belt

Auditory (headphones) Tactile Display

Wireless (Bluetooth) 1.2 lbs (w/o battery) Elastic belt 8 tactors at 45-degree

increments

Tactile Patterns Emulating Standard Army Hand Signals (FM 21-60)

TACTONS

Gesture Recognition Machine Learning Algorithms (3 implemented for

evaluation) Linear Classifier AdaBoost Artificial Neural Network (evolved w/ NEAT)

29 Features Based on work by Rubine (1991) on 2D symbol

recognition Example features:

○ Bounding Volume Length○ Min, Max, Median, Mean (X, Y, Z)○ Starting Angle, Total Angle Traversed, Total Gesture Distance

Training & Visualization UI Arbitrary gesture set Left-hand, right-hand,

both-hands 3D animated soldier Text label display Sound display Tactile display Wiimote visualization Data serialization UI Independent of

recognition API

Experiments & Results

Several experiments run to dateDifferent algorithms and gesture setsAccuracy > 94%Classification time < 10ms / gestureLinear classifier best performer (for training

time and classification considered together)AdaBoost (highest accuracy, but slower

training time than linear classifier)ANN w/ NEAT (worst performer – requires

more training data)

Discussion Proved Concept

System capable of accurately converting and transmitting a visual communication mode into a non-visual form.

Wiimote is a convenient and inexpensive device for experimentation. Technology transfers to more robust hardware (e.g. instrumented glove).

Wiimote produces some ambiguous data (e.g. static poses). Additional attachment (e.g. gyroscopes) required for more accuracy.

Experiments indicate promising form of communication – more experiments are needed.

Future Work Determine the maximum number of

gestures that can be accurately recognized Gesture rejection Dynamic mapping between gesture, sound,

and tactile sequence Scenario development for realistic

experimentation (establishing context) Transmitting signal data via RF (currently

sent to local device or via UDP/IP)

USMA Collaboration

CDT Robert Darket, CDT Zachary Schaeffer (Principal Investigators)

Application: TrainingCollect exemplar gestures from SMEsValidate less-experienced soldier’s gestures

against exemplars

Video Demonstration

Questions?

Gestural communications with accelerometer-based input devices and multi-modal displaysPaul D. Varcholik

ACTIVE LaboratoryInstitute for Simulation and TrainingUniversity of Central Floridapvarchol@ist.ucf.edu

James L. MerloLTC, US ArmyUS Military AcademyWest Pointjames.merlo@usma.edu