Department of Electrical and Computer Engineering

He ZhouHui ZhengWilliam MaiXiang Guo

Advisor: Professor Patrick Kelly

ASLLENGE

Department of Electrical and Computer Engineering

Outline Background System Requirements Design Challenges System Overview Hardware and Software Team Roles Timeline Q/A

2

Department of Electrical and Computer Engineering

Background Leverage Kinect technology to recognize and

offer feedback on American Sign Language ASL is the one of most commonly used language

in US(.5 to 2 million population use ASL as their primary form of communication)

Educational learning tool for ASL

3

Department of Electrical and Computer Engineering

Motivation Helping hearing impaired people to learn ASL. Facilitate communication between ASL speaker

and non-speaker

4

Department of Electrical and Computer Engineering

System Requirements Create a user-friendly interface that recognize

sign language and offer feedback• Software must be easy to use

Display translated word or phase being demonstrated by user• Provide quality feedback to user concerning

the accuracy of ASL Cost should be within the budget

5

Department of Electrical and Computer Engineering

Design Challenges Human Factors Background Interferences Finger Gestures Hardware limitation• Kinect only has 320x240 resolution and 16 bit

depth

6

Department of Electrical and Computer Engineering

Kinect Has a depth and regular Camera Associated with software development kit

• Able to track a human being.• Use computer generated algorithms

to form human skeleton data points Track upper body movements

7

Department of Electrical and Computer Engineering

System Overview

8

Department of Electrical and Computer Engineering

Video demonstration(Hand Rotation)

9

Department of Electrical and Computer Engineering

Video demonstration(Hand Crossing)

10

Department of Electrical and Computer Engineering

Video demonstration(Hand behind Head)

11

Department of Electrical and Computer Engineering

Video demonstration(Hand behind Back)

12

Department of Electrical and Computer Engineering

Hardware Implementation 1 Increasing the flex sensor

output voltage with bend Voltage output goes into

an AD converter board AD converter digitizes the

output signal PC captures the signal via

USB or Bluetooth

13

Department of Electrical and Computer Engineering

Hardware Implementation 2 Kinect Camera captures the raw image and the

depth image data The data send to PC to be processed into skeleton

tracker Combining the signal from flex sensor and search

in database for matching motions

14

Department of Electrical and Computer Engineering

Software Design Flow

15

Department of Electrical and Computer Engineering

Development Model

16

Department of Electrical and Computer Engineering

Cost of Major Components

Name of Component

Flex Sensor

A/D Converter

Board

A Pair of Gloves

Kinect Sensor

Bluetooth Module

Quantities 10 1 1 1 1

Unit Price $12 $50 $10 $150 $20

Total Price $350

17

Department of Electrical and Computer Engineering

Team Roles He - test and integrate software prototype

Hui- image processing and programming

William - build microcontroller to detect finger and palm movements

Xiang- website, overall testing and building ASL database

18

Department of Electrical and Computer Engineering

MDR deliverablesThe goal for our MDR is to be able to detect and track arm and hand positions, possibly be able to translate simple hand gestures

19

Department of Electrical and Computer Engineering

ConclusionOur project is good for SDP:

• Contains both hardware and software• Expandable: database, audio recognition, braile

Our project is a good product:• Aid the disabled people• Personal/commercial use• Relatively inexpensive

20

Department of Electrical and Computer Engineering

TimelinePDR

Purchase PartsPlanning

GUIInterface with the Sensors

Build Finger SensorsBuild Test Database

Prototype 1MileStone MDR Deliverables

System IntegrationWinter Break

Expand DatabasePrototype 2

MileStone CDR DeliverablesAlpha Testing

UpdateBeta TestingSpring Break

Finalize ProjectMileStone FRP Deliverables

10/16/2011 12/5/2011 1/24/2012 3/14/2012 5/3/2012

21

Department of Electrical and Computer Engineering

Q&A

22