fname=“Yevgen”lname=“Borodin”zip=“11790”…

user=“Yevgen”session=“browse”…

title=“circuit city”mode=“formFill”…

Cached Files

main.vxmlallLinks.vxml history.vxmlfavorites.vxmlkeyList.vxml commands.vxml…

Processor

VariableSpace

Manager

InputManager

Event-HandlingManager

OutputManager

VXML SubmittedVariables

Input Queue

KeyboardInterface

ConsoleSUN Audio

PlayerSpeech

Recognition EngineText-To-Speech

Engine

KeyboardInput

VoiceInput

VoiceOutput

AudioOutput

Output Buffer

JSAPI

SGRSGrammarInterpreter

JavaScriptInterpreter

File Manager

VXMLSurfer in ActionVXMLSurfer in Action

A Flexible VXML (Voice XML) InterpreterYevgen Borodin

Computer Science Department, Stony Brook University

Back-End ProcessingBack-End Processing

HELP

…<rule id="phone_number"> <item repeat="0-1">d</item> <tag><![CDATA[$.phone="ddd-ddd-dddd";]]></tag> </item></rule> …

<?xml version='1.0'?><vxml> … <form id=‘shipping'>  <field name="fname">   <prompt>Enter first name</prompt>   <grammar src="customer.xml#fname“ type="application/grammar+xml"/> </field> <field name="lname">   <prompt>Enter last name</prompt>   <grammar src="customer.xml#lname" type="application/grammar+xml"/> </field> …   </form></vxml>

Interface Manager

Browser Object

Context Analyzer

Frame Tree Processor

VXML

Dialog Generator

Frame Tree

HTTPrequest

WEB

HTTPrequest

HTML

Frame Tree

Frame Tree

HTTPrequest

Big Picture: HearSay BrowserBig Picture: HearSay Browser

VXML

Interpreter

Mozilla

Engine

<catch event="Insert+F1 help"> <prompt> You are at Circuit City check out </prompt></catch>

<catch event=“Ctrl+S submit"> <submit namelist=“fname lname …”></catch> …

INS + F1

SGRS GrammarSGRS Grammar

Variables | Files | Return | EventsVariables | Files | Return | Events

fname=“Yevgen”lname=“Borodin”zip=“11790” …

Features of VXMLSurferFeatures of VXMLSurfer

Application of VXMLSurferApplication of VXMLSurfer

Future WorkFuture Work

Inadequacy of screen-readers in Web browsing Development of interface Manager for HearSay voice-browser Absence of fully-implemented open-source VXML interpreters Need for specialization in Web browsing applications Need for a multiplatform, extensible, modular, flexible system

Compliant with VoiceXML 2.0 specifications Geared to Web-browsing as opposed to telephony Modular, Extensible, Multi-Platform (Java) Extends VoiceXML 2.0 to give more control over dialog flow Loaded with add-ons: Spell Check, TTS, SR, etc.

VXMLSurfer is the interface of the HearSay voice-browser Users interact with VXMLSurfer through keyboard and mic HTTP request is forwarded to the Mozilla browser engine The loaded Web page is converted into a frame tree The frame tree is processed and converted to VXML dialogs

Complete VoiceXML 2.0 Specifications Messaging between VXMLSurfer and calling application Speech Recognition (CMU Sphinx) Java Script Interpreter and Grammar Interpreter Modules Multilingual TTS, etc.

VoiceXML dialog files are sent to the interpreter for processing Variables are returned to the calling application (HearSay) HearSay invokes form-filling module to fill and submit the form

MotivationMotivation

Uses of VoiceXMLUses of VoiceXML VXML is typically used in telephony applications Computer games use VXML to program interactive dialogs VXML dialogs disseminate information through public terminals VoiceXML can be used in voice browsing!

This material is based upon work supported by the National Science Foundation - Awards: IIS-0534419, CNS-0751083, IIS-0808678and National Institute on Disability and Rehabilitation Research (NIDRR) - Award: H133S090065.