Closing conference of SYSIASS – June 17th 2014

Multimodal Bio-signal based Control of Intelligent Wheelchair

Professor Huosheng HuLeader of Activity 3

University of Essex, U.K.

Closing conference of SYSIASSJune 17th 2014 2

SYSIASS: Autonomous and Intelligent Healthcare System

Closing conference of SYSIASSJune 17th 2014 3

Aim and Objectives Multimodal Human-Machine Interface aims to make wheelchairs more

user-friendly.

The key challenge is the understanding of

the user who interacts with the wheelchair

the system (the computer and sensor technology & their usability)

the interaction between the user and the wheelchair.

A proper balance between

Functionality that is defined by the set of actions or services that it provides to its users, and

Usability that is the range and degree by which the system can be used efficiently and adequately to accomplish certain goals for certain users.

Closing conference of SYSIASSJune 17th 2014 4

System Configuration

Multimodal HMI

Voice, gesture, EMG, EEG

Closing conference of SYSIASSJune 17th 2014 5

Detection of Human Intension Inertial Sensing: to sense the position and motion of the human

hand and other body parts for use in HMI.

Audio Sensing: Using microphones to sense the sound is to interpret speech, which is the most natural modality.

Visual Sensing: Using cameras to detect the human motion, such as gestures, lip motion, gaze, facial expressions, head & other body movements.

Touch and force: This is especially important for building a proper feel of “realism” in human intension.

Muscle and Brainwave: These bio-signals are obtained noninvasively from the surface of the scalp and skin; used for wheelchair control.

Closing conference of SYSIASSJune 17th 2014 6

Wireless Voice based Control To use human voice commands; Pre-defined control phrases:

“forward”, “turn left”, “turn right”, “backwards”, “stop”;

Wireless wearable headphone device; Fast voice recognition and no pre-

training Plug and play wireless and comfort

control Obstacle avoidance with laser scanner Unlimited voice control commands

available Video show

Closing conference of SYSIASSJune 17th 2014 7

Visual Head Gesture based Control To help disabled and elderly who cannot

use joystick To realise hands-free control of the

wheelchair Obstacle avoidance to ensure user safety Functionalities:Visual detection of head gestures of users

Pre-defined commands (left, right, forward, backward, stop) for controlling a wheelchair

Laser based obstacle detection & avoidance

Video show

Closing conference of SYSIASSJune 17th 2014 8

Facial Expression & Head Movement Facial expressions are detected using

the cognitiv suite of Emotiv.

Head movements are detected by the gyroscope of Emotiv.

Pre-defined commands (left, right, forward, backward, and stop)

A simple and safe hands-free control with flexible configurations for users.

An ideal alternative to the joystick control for users with severe disability.

2F+1H 3F+1H

Closing conference of SYSIASSJune 17th 2014 9

Wireless IMU in a Baseball Cap Detect head motion for wheelchair control Minimum user head motion required Adapt to people with weak neck abilities Obstacle avoidance with laser scanner Totally replace joystick control interface

Video show

Closing conference of SYSIASSJune 17th 2014 10

Minimally Invasive Intra-Oral Palate for wheelchair User

A minimally invasive way to control wheelchair

It is comfortable for users and allows prolonged use.

It is based on an intra-oral dental retainer clip with9 force sensitive resistors10 Bit 200Hz data capture Opto-isolation is deployed.Connectivity to GPSB

Closing conference of SYSIASSJune 17th 2014 11

Conformal RFID Sensing for wheelchair control

Strain gauge RFID tag is designed to sense the stretch on skin (neck/eyebrow movement)

A tongue controlled RFID tag is designed to sense the tongue position in the mouth

To provide an on/off switch and control mechanism for controlling the wheelchair.

Closing conference of SYSIASSJune 17th 2014 12

Conclusion Human-Machine Interaction (HMI)

aims to achieve information full accessibility.

Human-Machine Interaction plays an key role in assistive technology.

Hands-free control of wheelchairs are extremely useful to general public without programming skills.

Multimodal HMI is necessary to overcome the limitations of single modality HMI.

Activity 3 of SYSIASS has been conducted successfully.