Programming by Voice with Scratch:Teaching the Cat to Obey a Bird

Ramaraju Rudraraju, Srinivasa Datla, Avishek Banerjee, Mandar Sudame Univ. of Alabama at Birmingham, Department of Computer and Information Sciences

Jeff Gray University of Alabama, Department of Computer Science

Gary Edwards United Cerebral Palsy of Greater Birmingham

This research sponsored by a Google Research Award

Overview of Presentation

Scratch Script Video

Motivation

Architecture & Working of Myna

Dynamic Grammars

Myna Features

Scratch Script

Get your popcorn, its Movie Time!!

The WIMP Metaphor

Motivation

Window

Menu

Icon

s

Pointing Device

Motivation

The WIMP assumes dexterity

The MYNA Solution for Scratch

Drag and Drop Navigation Continuous Navigation Navigation using Transparent Frames

Features

Types of Navigation

7

Features

(An extension of Transparent Frames)

Drop After Drop In Drop Before

Macro Commands

Myna Architecture

ComponentMappings

Sprite State Storage

Component Hierarchy

Scratch UI

Transparent Frames

Speech Recognizer

Command Executor

Grammar

Java Robot

MYNA

Model

View Controller

Cloud Garden

Grammar

Speech Recognizer

Under the Hood of Myna

Speech Recognizer

GrammarCommand Executor

MappingsMappings

Move

Right Controller Model

Command Executor

Java RobotJava

RobotComponent Hierarchy

Component Hierarchy

Dynamic Grammar

Default Grammar

Navigation Grammar

Speech Pause

Grammar

File Grammar

File Grammar

Control Grammar

Looks Grammar

Default Grammar

Navigation Grammar

FilePauseResume

Limitations Scratch needs to be in maximized mode Not compatible with different screen resolutions Myna is not scroll-enabled Large vocabulary and still growing!

Future Work Integration with UCP - Deep human subjects testing with

United Cerebral Palsy of Greater Birmingham Portability on other platforms Configurable screen resolutions

Looking Ahead

Myna is our preliminary attempt towards voice-driven programming applied to Initial Learning Environments (ILEs) to overcome the physical challenges of using the traditional WIMP metaphor

Our long-term goal is to assist children with disabilities by giving them a learning tool that enables them to encounter the joys and excitement of programming, while also improving their logic skills

Summary

Thank you

Questions and

comments?

Additional questions?teammyna@googlegroups.

com

https://sites.google.com/site/projectmyna2010/