International Journal of Social RoboticsDOI 10.1007/s12369-009-0032-4

Development and Validation of a Robust SpeechInterface for Improved Human-Robot Interaction

Amin Atrash · Robert Kaplow · Julien Villemure · RobertWest · Hiba Yamani · Joelle Pineau

Accepted: 17 September 2009

Abstract Robotics technology has made progresson a number of important issues in the last decade.However many challenges remain when it comes tothe development of systems for human-robot inter-action. This paper presents a case study featuringa robust dialogue interface for human-robot com-munication onboard an intelligent wheelchair. Un-derlying this interface is a sophisticated softwarearchitecture which allows the chair to perform real-time, robust tracking of the dialogue state, as wellas select appropriate responses using rich proba-bilistic representations. The paper also examinesthe question of rigorous validation of complex human-robot interfaces by evaluating the proposed inter-face in the context of a standardized rehabilitationtask domain.

Keywords Intelligent wheelchair · Dialoguemanagement · Service robotics · Human-robotinteraction

1 Introduction

For many people suffering from chronic mobilityimpairments, such as spinal cord injuries or multi-ple sclerosis, using a powered wheelchair to movearound their environment can be difficult. Accord-ing to a recent survey, 40% of patients found dailysteering and maneuvering tasks to be difficult orimpossible [7], and clinicians believe that between61% and 91% of all wheelchair users would benefit

Corresponding author: J. PineauSchool of Computer Science, McGill University

McConnell Engineering Building, Rm 318

Montreal, PQ, CANADAE-mail: jpineau@cs.mcgill.ca

from a smart wheelchair [29]. Such numbers sug-gest that the deployment of intelligent wheelchairscatering to those patients’ needs could have a deepsocietal impact.

Over the last decade, robotics technology hasmade progress on a number of important issuespertaining to mobility. Many of these developmentscan be transfered to the design of intelligent wheelchairs.Yet many challenges remain—both technical andpractical—when it comes to the development ofthe human-robot interaction components. A recentsurvey of the literature on smart wheelchairs sug-gests that while voice control has often been usedto control smart wheelchairs, it remains difficult toimplement successfully [28].

Our work addresses two main challenges per-taining to the development of voice-controlled as-sistive robots. First, we tackle the problem of ro-bust processing of speech commands. In support ofthis goal, we propose a complete architecture forhandling speech signals, which includes not onlysignal processing, but also syntactic and semanticprocessing, as well as probabilistic decision-makingfor response production. Many of these compo-nents have been exploited separately in human-machine speech interfaces, but to the best of ourknowledge, this is the first system to incorporateall components in a coherent architecture for speech-based robot control.

Second, the paper tackles the issue of devel-oping tools and standards for the formal testingof assistive robots. The use of standard testinghas been common currency in some sub-tasks per-taining to human-robot interaction, most notablyspeech recognition. However few tools are avail-able for the rigorous and standardized testing of

fully integrated systems. Here we propose a novelmethodology and environment for the standard-ized testing of smart wheelchairs. The procedure isinspired from one commonly used in the evaluationof conventional (non-intelligent) wheelchairs. Wedemonstrate, through careful user experiments, howit can be adapted to formally evaluate voice-controlledsmart wheelchairs. Results described below showthat the coupling of speech input, grammaticalinference, and probabilistic decision-making pro-vides a robust architecture to process natural in-teractions from untrained users.

The cognitive architecture described in this pa-per for handling speech interaction is appropriatefor a wide spectrum of human-robot interactiontasks. As such, the work should be of interest toresearchers working on a variety of social robotplatforms, not only smart wheelchairs. The test-ing procedure we describe however is specificallytargeted at the evaluation of smart wheelchairs.Nonetheless this case study may provide ideas andmotivation for the development of standardizedevaluation tools for other social robot interactiondomains.

2 The SmartWheeler Platform

The SmartWheeler project aims at developing—in collaboration with engineers and rehabilitationclinicians—a prototype of a multi-functional intel-ligent wheelchair to assist individuals with mobil-ity impairments in their daily locomotion, whileminimizing physical and cognitive loads [23] .

Figure 1 shows a picture of the SmartWheelerplatform. The intelligent wheelchair is built on topof a commercially available Sunrise Quickie Freestylewheelchair. The intelligent sensing and computingcomponents were designed and installed in-houseat McGill University’s Centre for Intelligent Ma-chines. These include two (one forward-facing, onebackward-facing) SICK laser range-finders, custom-made wheel odometers, a Lilliput 8” touch-sensitiveLCD, a two-way voice interface, and an onboard1.4 GHz Pentium M Processor. The laser range-finders and odometers are used for mapping, nav-igation and obstacle avoidance. The display, voiceinterface, and wheelchair joystick are the main modesof communication with the user. The onboard com-puter interfaces with the wheelchair’s motor con-trol board to provide autonomous navigational com-mands.

Fig. 1 The robotic wheelchair platform.

3 Cognitive Architecture

Figure 2 presents an overview of the software ar-chitecture underlying the communication modules.This includes the modules handling the variouscommunication modalities (Speech Recognizer, SpeechSynthesis, Visuo-Tactile Unit), a module handlinggrammatical parsing of the speech signal (Seman-tic Grammar), a core decision-making unit (Inter-action Manager), a module to translate the de-cision into low-level output functions (BehaviorManager), and a module to handle the robot’s low-level navigation components (Robot Control Sys-tem).

Integration of the software architecture is doneusing the Acropolis framework [40]. In this frame-work, components are integrated as plug-ins thatexecute within the architecture. Plug-ins are de-fined in terms of what inputs they require andwhat outputs they produce; a so-called circuit filespecifies which plug-ins are connected. This “looselycoupled” scheme allows for rapid prototyping throughthe switching of components on the fly. To con-nect components, Acropolis provides its own com-munication channel, making it easy and efficientto transfer data between components. Our cur-rent implementation includes only the componentsshown in Figure 2, however Acropolis is well suitedto a variety of other robotic components, and isparticularly suitable for quick testing of differentconfigurations, since new components can be easilyadded, removed, or replaced.

Acropolis usually includes a plug-in to a stan-dard Robot Control System. Good options for thismodule include the popular Player application [5],or the Carmen navigation toolkit [22]. The SmartWheeler’s

2

Fig. 2 Interaction Architecture

sensors and actuators can then be accessed di-rectly through these navigation tools. As a result,the interaction architecture we present in this pa-per is relatively independent of the physical robotplatform, and could be used for a wide variety ofhuman-robot interaction systems.

3.1 Speech Recognition

A speech interface provides a comfortable and nat-ural input modality for users with limited mobil-ity. In general, speech requires little training, andis relatively high-bandwidth, thus allowing for richcommunication between the robot and human. Theperformance of speech recognition systems are in-fluenced by many aspects, including the vocabu-lary, language and acoustic models, speaking mode(isolated words vs. continuous speech), etc. Someof these aspects have to be taken into account whendesigning the speech interface.

Selecting a speech recognizer that performs wellfor the task at hand is important. To preserveflexibility in the development process, we consid-ered two open source speech recognition systems:HTK [12] and CMU’s Sphinx-4 [4]. Both of thesesystems are speaker-independent, continuous speech,recognition systems, which typically require lesscustomization than commercial systems. Becausecustomization is minimal, it is important that thesystem be pre-trained on a large speech corpussuch that appropriate acoustic models can be pre-computed. Such corpora usually falls under one oftwo categories: those developed for acoustic pho-netic research and those developed for very spe-cific tasks. Since SmartWheeler is still at an earlystage of development, and domain-specific data is

not available, we use a general purpose acousticmodel, such as the Wall Street Journal AcousticModel [36].

A small vocabulary makes speech recognitionmore accurate but requires the user to learn whichwords or phrases are allowed. And while our cur-rent focus is on building and validating an inter-action platform for a specific set of tasks (definedbelow), we also want the user to be able to interactwith the system in the same way s/he would, wheninteracting with any caregiver, and with very littleprior training. Thus a fixed set of tasks is con-sidered, but several possible commands for eachtask are allowed. For example, if a user wants todrive forward two meters, possible commands in-clude (amongst others):

– Roll two meters forward

– Drive forward two meters.

– Roll forward two meters fast.

– Drive fast two meters forward.

The specific vocabulary allowed by the speech rec-ognizer is extracted from a set of preliminary in-teractions recorded between users and the chair,based on the specific tasks selected. We providemore detail on our current implementation below.

Both Sphinx-4 and HTK allow the designerto specify a set of syntactic rules (or grammar)which specifies constraints on the ordering of wordswithin a sentence. This grammar can be useful toenhance speech recognition quality by constrainingthe hypothesis space. In the SmartWheeler archi-tecture, we have a separate module to handle bothsemantic and syntactic grammatical constraints (seethe next section), therefore it suffices to use a veryloose grammar in the speech recognition system.

3

The dictionary and grammar files used for bothSphinx-4 and HTK are available online [31].

We conducted a series of preliminary tests us-ing both Sphinx-4 and HTK. Three male and onefemale university students in their early twentiesrecorded 179-183 commands each (723 speech com-mands in total). Subjects were presented with ascript of commands, corresponding to each task,and instructed to read them in order.

We analyzed the results in terms of substitu-tions (when the speech recognizer fails to recognizea word and substitutes another incorrect word),deletions (words that were spoken but missed bythe recognizer), insertions (words that were notspoken but added by the recognizer), word errorrate (proportion of incorrectly recognized words,including substitutions, deletions and insertions),and sentence error rate (proportion of sentencesin which one or more words are incorrectly recog-nized).

Both speech recognition packages showed equiv-alent performance across the board. Note that theresults indicated rather high error rates. The meansentence error rate was 45.2% when using Sphinx-4and 46.7% when using HTK, while the mean worderror rate was 16.1% with Sphinx-4 and 16.6%for HTK. This analysis suggests that performanceof the two candidate speech recognition systemsis equivalent. We chose to integrate Sphinx-4 on-board the SmartWheeler simply because it is slightlyeasier to handle in terms of software integrationwithin Acropolis.

While the error rates are quite high, upon closeranalysis the situation is not as discouraging as itwould seem. Observed errors can be classified intoone of three types: design errors, syntax errors, andsemantic errors.

Design errors

Errors of this type are introduced because of errorsin the design of the task vocabulary. For example,a number of substitutions occurred when Sphinxrecognized meter as meters. The task grammarwithin the speech recognizer was modified to avoidsuch minor errors.

Syntax errors

Errors of this type are introduced because the taskgrammar contains many ways to say the same thing.For example, when subject 1 said, roll back one

meter, HTK recognized it as roll backward

one meter. This is counted as one substitution

error. However, even though the commands do notmatch, they do have the same semantic meaningin this particular task domain. The natural lan-guage processing module described in Section 3.2is specifically designed to handle this type of error.

Semantic errors

Errors of this type are introduced when the speechrecognition system fails to recognize the correctcommand. For example, when subject 3 said, de-

scend the curb, Sphinx recognized it as ascend

the curb. In order to handle this type of er-ror, it is necessary to reason about the environ-ment and the user’s intents. A more severe errorof this type occurred when subject 3 said drive

slowly two meters back and Sphinx recog-nized ascend ridge. This error is harder to de-tect, but not impossible. Handling this type of er-ror involves reasoning about the user’s intentionsand the robot’s physical context. This is the mainmotivation for adopting a rich probabilistic frame-work for the Interaction Manager, as described inSection 3.7.

Evidence from these preliminary experimentsstrongly supports the integration of the diversecomponents included in the cognitive architecture.

3.2 Semantic Grammar

The output of the speech recognizer is processedby a natural language parser, which extracts syn-tactic and semantic information from the stringof words. The primary role of the grammar is toconstrain the output space of the speech interface.The framework we use for natural language pars-ing is that of Combinatory Categorical Grammars(CCG) [33], for which an open-source implemen-tation is freely available [37].

It is worth noting that the grammatical anal-ysis carried out in the NLP parser replaces thegrammatical analysis that could be performed withinthe speech recognition system. The main advan-tage of using a separate NLP processing unit isthat the CCG parser considers both syntactic andsemantic constraints (as opposed to only consid-ering syntactic constraints). Thus different wordswith identical semantic meaning can be mapped toeach other. This can reduce the per-sentence errorrate.

The semantic component of the parser is ex-pressed in logical form, with logical predicates rep-resenting the meaning of the input sentence. The

4

set of these outputs forms the basis for the statespace of the Interaction Manager. The set of pos-sible outputs can grow quickly, depending on thespace of logical predicates. The NLP parser keepsthis to a manageable size by abstracting away allsyntactical components for the later processing.The semantic logical form, which could take a hier-archical form (e.g. two meters is the distance ar-gument of roll two meters forward, and two

is the number argument of the distance argument),is flattened into a fixed-size feature vector withpre-specified slots (e.g. 〈Action: roll, DistanceU-nit: meter, DistanceValue: 2, Direction: undefined,...〉). One can think of this representation as lyingbetween the overly complex logical form commonto full NLP systems, and the overly simple bag-of-words assumption used by many machine learningapplications.

To conclude this section, it is worth emphasiz-ing the advantages of using the CCG grammar asa separate module, over using the grammar offeredin the Sphinx (or HTK) speech recognition system.First, the CCG grammar is more expressive thanthe simple Backus-Naur-Form context-free gram-mar framework used in Sphinx. Another importantadvantage of CCG is that it provides support forthe construction of a logical-form semantic repre-sentation during the parsing process, which thenforms the basic state representation for the Inter-action Manager. Since the Interaction Manager’sstate space is based directly on the output space ofthe grammatical unit, substantial compression inthe state space (and decision space) of the Interac-tion Manager is achieved through the applicationof the CCG parser’s syntactic and semantic con-straints. This significantly improves the scalabilityof the interaction system.

In cases where the CCG parser does not returna valid parse, we can still recuperate the set ofwords produced by the speech recognition systemand pass them directly to the Interaction Man-ager to be processed as a bag-of-words, withoutany syntactic or semantic structure. This is thestandard approach in dialogue management sys-tems that do not include grammatical parsing [6,27]. In some cases, this can help guide the selectionof appropriate clarification queries can be prefer-able to having the speech recognition system goto unnecessary lengths to fit the observed speechsignal into a valid (but incorrect) parse.

3.3 Speech Synthesis

In order to provide speech feedback, the Festival [9]speech synthesis system is used. This system isfreely available, and does not require customiza-tion, therefore we do not discuss it further. Thelist of speech output sentences is set by hand andstored in the Behavior Manager.

3.4 Visuo-Tactile Unit

A graphical user interface (GUI) complements thespeech interface by providing immediate visual feed-back to the user, as well as a secondary mode ofinput through the tactile interface. Note that thescreen-based interaction and speech-based interac-tion modes are thoroughly complementary. At anypoint during the interaction, the user can entercommands either through the speech interface, orthrough the touch-sensitive display. Additionally,there is in fact a third mode of input, which is to se-lect the buttons of the visuo-tactile display using ajoystick (or equivalent device). The screen changesreactively whenever instructions are issued usingany of the three input modalities.

The primary motivation behind the GUI de-sign is to provide an alternative to the speech in-terface for users who are not able to reliably con-trol the wheelchair using voice. Some people mayhave disabilities which interfere with their vocalabilities; others may find the speech interface tooerror-prone and prefer more reliable alternatives.The relative level of reliability of course will beaffected by the user’s fine motor control.

Another motivation for the GUI design is toprovide context and support to facilitate the useof the speech interface. The display can be used toinform of the current state of the wheelchair (e.g.moving, stopped), as well as the set of commandsthat can be used in that particular situation. Theuser is always free to voice commands that do notappear on the screen (this can lead to “short-cuts”through the menu system).

We do not formally evaluate the GUI design inexperiments reported below, therefore we do notdescribe this module in detail. Design and valida-tion of this component will be the subject of futurework, since they require a substantial involvementfrom the disabled user population.

5

3.5 Behavior Manager

The role of the behavior manager is to providean interface layer between the high-level decision-making and the low-level robot control. As ex-plained below, the Interaction Manager chooseshigh-level action in response to the received pro-cessed sensory input. These actions are either robotcontrol actions, such as movement or hardwareconfiguration, or response actions to be communi-cated to the user. The Interaction Manager passesthe selected action to the Behavior Manager, whichthen translates the high-level action into lower-level robot specific instructions. This allows thehardware specific aspects to be abstracted awayfrom the interaction and decision-making modules.The Behavior Manager is implemented as a sim-ple look-up table, in which the decomposition ofa high-level action is detailed as a sequence oflow-level commands for the appropriate modules(speech generation and/or robot navigation).

3.6 Robot Control System

The Robot Control System can be implementedusing different publicly available software packages,for example the popular Player application [5] orthe Carmen robot navigation toolkit [22]. We haveused both throughout our preliminary experiments.The goal of this component is to handle tasks suchas mapping, localization and path planning. We donot discuss this component further as it is some-what orthogonal to the main focus of this paper.

3.7 Interaction Manager

The Interaction Manager acts as the core decision-making unit in the robot architecture. This moduleis ultimately responsible for selecting the behaviorof the robot throughout the interaction with theuser.

In this context, the Interaction Manager canbe seen as an Input-Ouput device, where informa-tion about the world is received via the grammarsystem and the low-level robot navigation system.The unit then outputs actions in the form of speechand display responses, or issuing of control com-mands to the navigation unit. These actions areprocessed through the Behavior manager, to ex-tract a pre-set sequence of low-level operations, be-fore being sent to the respective modules (speechsynthesis, visuo-tactile unit, robot control system).

The goal of the Interaction Manager is to pro-vide a robust decision-making mechanism capableof handling the complexity of the environment.This is a challenging goal due to the high degreeof noise in the environment. While the semanticgrammar can help handle some of the noise, evenproperly transcribed speech can contain ambigui-ties. Accounting for the set of possible outcomescan be crucial in providing robust action selec-tion. Therefore we favor a probabilistic approachto decision-making.

The Partially Observable Markov Decision Pro-cess (POMDP) paradigm has been shown to be apowerful tool for modeling a wide range of robot-related applications featuring uncertainty, includ-ing robot navigation [18], dialogue management [30,27,38,6], and behavior tracking [11]. One of the ad-vantages of the POMDP model is its ability to cap-ture the idea of partial observability, namely thatthe state of the world cannot be directly observed,but instead must be inferred through noisy obser-vations [32]. In the case of our Interaction Man-ager, the user’s intention must be inferred thoughobservations coming from verbal commands, as wellas contextual information inferred through the phys-ical state of the robot in its environment.

Each state in the POMDP represents a discreteuser intention, such as moving forward, turning thechair, or reaching a specific target. The states aredefined using a pre-defined set of semantic slots,and a finite domain for each slot.

The output of the Interaction Manager is de-fined by a set of actions. Because each state repre-sents a different user intention, there is one “cor-rect” action for each state. This includes robotcontrol actions and interaction response actions. Ifthe agent selects and executes the correct action,a positive reward is given. If the incorrect action isselected, a high cost is inflicted. In addition, thereis a set of query actions, in the form of clarificationquestions, which can be emitted in situations thatwarrant additional information. All query actionsincur a small cost. The goal of the decision-makingengine is to select actions so as to maximize thisreward function.

Observations occur when there is input fromthe sensor-processing modules. Most of the time,these appear as an output of the grammatical parser,and consist of assignments to the various semanticslots. Given the observation, the goal is to deter-mine the likelihood of each state, si, given this as-signment, z = (CommandV alue = v1, CommandType =v2,

6

DirectionV alue = v3, ...), written as z = (v1, v2, v3, ...)for readability. We assume that assignments of val-ues to the slots is independent:

P (si|v1, v2, v3, ...) =P (v1, v2, v3, ...|si)P (si)

P (v1, v2, v3, ...)(1)

=P (v1|si)P (v2|si)P (v3|si)...P (si)

P (v1)P (v2)P (v3)...These values, P (vj |si), P (vj), and P (si) can becalculated from collected data.

In cases where there was no valid parse, thephrase is sent directly from the Speech Recognizerto the Interaction Manager as a bag-of-words. Thestate probability update is done in similar fashion,but considering the probability of each word, giventhe state. Again, these can be learned from data.

Ideally, the transition from state-to-state shouldbe estimated from training data, to reflect patternsin daily activities. In cases where such data is notavailable, one can assume a uniform transition toany other state following execution of a non-queryaction (i.e. we don’t know what task the user is go-ing to tackle next) and assume the system remainsin the same state (with slight noise) following ex-ecution of a query action.

Once the model is created, a POMDP solvingalgorithm is applied to determine a proper pol-icy. A policy is a mapping from distributions overstates (as defined in Equation 1) to actions. A typ-ical policy might be as follows: When the systemis confident about the user’s intention, the systemexecutes the corresponding action. If there is somedegree of ambiguity, such as the system believingthe action is a move command, but lacking theparameters, the system can then execute a specificquery, such as request move parameters. Ifthere is sufficient noise and the system is unsure ofthe state, it can execute the more general repeat

query. This policy is optimized using dynamic pro-gramming methods based on the transition, ob-servation, and reward parameters. As the cost ofqueries decreases, or the penalty for an incorrectaction increases, the system will become more cau-tious and inclined to present multiple queries be-fore committing to an action. Several techniquesexist for determining a policy from a model. Weuse a standard point-based approximation tech-nique [24].

4 A Standardized Evaluation Tool forSmart Wheelchairs

In his survey of smart wheelchairs, Simpson [28]comments that: ”While there has been a signifi-

cant amount of effort devoted to the developmentof smart wheelchairs, scant attention has been paidto evaluating their performance. (...) Furthermore,no smart wheelchair has been subjected to a rig-orous, controlled evaluation”. His comments aremade mostly in the context of highlighting theneed for better experiments with the target popu-lation, in real-world settings. However, before we—as a community—are in a position to achieve thisfull goal, a necessary step is to develop the toolsand methods for achieving such rigorous, controlledevaluation. This section describes one such tool.

4.1 Target population

The target population for our smart wheelchairplatform is those users who require powered wheelchairto get around their environment on a daily ba-sis, yet find the use of a wheelchair problematic,whether due to fatigue, limited motor skills, or sen-sory impairment. This can include individuals suf-fering from spinal cord injuries, multiple sclerosis,or other mobility disorders.

The experiment described in the latter sectionof this paper do not include disabled users; all re-sults reported below were obtained with healthysubjects. The primary goal of the experiment pre-sented below is to serve as a pilot study for theproject. This is an important step towards acquir-ing ethical approval for the experiments with thedisabled population.

Before conducting the pilot study, we consultedat length with clinical experts to define the selec-tion criteria for the target population which will beincluded in later phases of experimentation. Thisselection criteria includes: a reduced mobility com-ponent (i.e. verifying the subject’s need for a mo-torized wheelchair), a communication fluency com-ponent (i.e. verifying the subject’s ability to use atwo-way speech interface), as well as other criteriaensuring the safety and well-being of the test sub-jects. In the context of the pilot study, the reducedmobility criterion was ignored, but all other crite-ria were applied without modification. Given thatour primary goal in this paper is to evaluate theeffectiveness and robustness of the human-robotinterface, and in particular robust speech interac-tion, we believe that the results obtained in thepilot study with healthy subjects will be reason-ably indicative of results for the target population,assuming similar communication abilities.

7

4.2 A Standardized Rehabilitation Environment

The long-term goal of this project is to increasethe autonomy and safety of individuals with se-vere mobility impairments by developing a roboticwheelchair that is adapted to their needs. This is areasonably ambitious goal, which can entail a widespectrum of tasks and activities.

While the long-term goal is to deploy our plat-form in natural indoor/outdoor living environmentsadapted for wheelchair users, there is a need inthe earlier stages of the project to formally as-sess the effectiveness and safety of the platform ina more controlled task domain and environment.In seeking such a testing methodology, we havecome across a standard wheelchair training proto-col called the Wheelchair Skills Test (WST). TheWST was developed to assess and train wheelchairusers through a representative set of wheelchairskills [15]. Extensive information about the testcan be found on the WST website1. The test iscurrently being used in clinical settings to identifyskills that should be addressed during training, aswell as to compare a subject’s performance beforeand after rehabilitation.

There are many reasons why we believe thistesting methodology is useful for the validation ofsmart wheelchairs. The WST includes 39 skills, di-vided into 18 skills groups, and representing mul-tiple levels of difficulty ranging from simple tasks(such as moving forward short distances) to morecomplex tasks (such as performing turning ma-noeuvres in constrained spaces). The set of skillscan be thought of as an “obstacle-course” for wheelchairs,and is considered representative for general wheelchairperformance. The assumption is that a person do-ing well on the 39 tasks included in the WST canbe considered a skilled wheelchair user becausethe abilities tested are sufficiently rich to resem-ble situations encountered on a daily basis. Thechoice of tasks is based on realistic scenarios, butis still standardized enough to allow for preciseperformance measurements. As one would expect,most tasks test navigation skills (e.g. Roll for-

ward 10 m in 30 s, Get over 15-cm pot-

hole), but there are some other actions as well,e.g. those concerning the wheelchair configuration,like Controls recline function. Accomplish-ing all tasks takes roughly 30 minutes [16]. Thetest does require some environment infrastructure(e.g. a ramp, a few walls) but the space require-

1 http://www.wheelchairskillsprogram.ca

ments are not excessive and typical of the spacerequired for standard mobile robotics research.

Other wheelchair skills tests have been pro-posed in the occupational therapy literature. See [14]and [26] for comprehensive overviews of existingtests. Kilkens et al. concluded that out of the 24tests they reviewed, only the Wheelchair Skills Testhas been “adequately tested on both validity and re-liability”. It is also one of the few tests which wasdesigned for powered wheelchairs. Furthermore, ithas not been designed for a specific target group(e.g. stroke patients), but for wheelchair users ingeneral. This is a useful aspect since our aim in de-signing the SmartWheeler platform is to improvemobility for a large spectrum of mobility-impairedindividuals.

For our use of the WST, it is particularly inter-esting to note that a version of this test was explic-itly conceived to provide a means of evaluating thewheelchair+user+caregiver as a team. The goal isnot to rate the performance of the wheelchair user,but rather that of this team. In the case of an intel-ligent system, we view the added robot technologyas playing a role similar to that of a caregiver andwe use the WST in the version ‘powered wheelchairwith caregiver’ (WST-P/CG in [15]). The WSTmanual specifies that a caregiver need not be ahuman: “An animal (e.g. a service dog) that as-sists with the performance of a skill is considereda caregiver, not an aid.” [15] We are extending thisnotion even further by introducing the intelligentsystem as a caregiver. Although the WST manualdoes not explicitly define a caregiver, this is justi-fied because the purpose of our intelligent softwaresystem is in fact to cooperate with the wheelchairuser in order to accomplish the target activities.

Finally, it is worth emphasizing that the WSTevaluates skill proficiency, as well as safety. TheWST includes a formal evaluation protocol, wherebyperformance on each of the tasks is graded in termsof these two criteria in a binary manner: a personeither passes or fails a task, and he/she does soeither in a safe or in an unsafe way. The pass/failgrading method makes the evaluation simple andas objective as possible. This is reflected in thehigh test-retest, intra- and inter-rater reliabilitiesachieved by the WST [16,17]. Because the set oftasks and the testing environment is precisely de-fined, it is easier to provide strong guarantees re-garding the safety and security of the wheelchairin this constrained domain, compared to naturalliving environments. Thus it is also easier to en-

8

sure we meet ethical requirements for testing withthe target population.

4.3 Customization of the Cognitive Architecturefor the WST

The description of the cognitive architecture inSection 3 was done in reasonably general terms toemphasize that it is appropriate for a large spec-trum of human-robot interaction tasks. In this sec-tion we outline a few of the design decisions thatwere made to deploy this architecture onboard ourrobot platform and in the context of the WheelchairSkills Test.

The vocabulary used by the speech recognizerwas designed to include 61 words, which were de-termined by going through sample interactions be-tween users and the chair, based on the specifictasks of the WST. The dictionary and grammarfiles used for both Sphinx and HTK are availableonline [31]. Throughout our experiments, the speechrecognition system was used in press-to-speak mode,requiring the subjects to press a button while speak-ing to the robot. This could be an obstacle forseverely disabled individuals, however given thatone of our selection criteria is that the person cur-rently uses a powered wheelchair (which s/he con-trols through a joystick or other pressure sensor),this poses no problem at this time in our investi-gations.

Regarding the design of the logical rules in thesemantic grammar, obviously the goal is not toequip the robot with a full-fledged English gram-mar at this stage of the project. The current gram-mar is based on the tasks of the Wheelchair SkillsTest. Our CCG grammar file is also available on-line [31]. In cases where an input sentence has mul-tiple correct parses, we have to make a choice onwhich parse to pass on to the Interaction Man-ager. Currently we pick the first parse from thelist. In the future we will investigate methods tolearn from experience which parse is most informa-tive, or consider extending the Interaction Man-ager such that it accepts multiple parses.

To realize the interface for the WST, the setof semantic slots produced by the grammar andused as the state representation for the interactionmanager are presented in Table 1. In reality, we usea slightly larger set, generated based on data. Eachstate represents one set of assignments to each ofthe semantic slots. Typically, a state has 3 to 6semantic slots assigned, with the remainder beingset to null.

Semantic Slot Possible Value Assignments

CommandValue move, switch, activate, deactivate,select, set, drive, turn, NULL

CommandType move-action, hardware-action,

config-action, NULLDirectionValue away, back, forward, backward, right,

left, NULL

DirectionType direction, vector-direction, angle-direction, NULL

PatientType controller, speed, drive-mode, seat,motor, NULL

PatientValue on, off, fast, medium, slow, cautious,

indoor, outdoor, NULL

Table 1 Semantic Slots and Possible Value Assignments

The POMDP domain file implementing the In-teraction Manager is available online [31]. Statis-tics for the observation probabilities in the case ofbag-of-words were calculated from data gatheredduring our preliminary study of the speech recog-nition system (see Section 3.1). We assume uni-form transition from state-to-state after executingany non-query action, and stationary transitions(with slight noise) after executing query actions.This is adequate to capture the Wheelchair SkillsTest, where the sequence of tasks is not meaning-ful.

In addition to the evaluation metrics includedin the WST, to better characterize our interface,we consider other metrics such as: the speech recog-nition accuracy, the proportion of sentences thatare correctly parsed, and the number of correctaction choices by the Interaction Manager.

5 Empirical Evaluation

We conducted experiments to evaluate the perfor-mance of our cognitive architecture in the contextof the WST task domain. The evaluation was con-ducted in our laboratory. Seven healthy subjectswere involved. Five male and two female universitystudents in their early twenties were recruited forthe evaluation. All were healthy subjects, withoutphysical disabilities. None of the subjects were di-rectly involved in the development of the SmartWheelerrobot.

Subjects were given a brief (5 minutes) intro-duction to the robot, including a high-level de-scription of the Wheelchair Skills Test task do-main, and interface modalities. Unlike in the pre-liminary evaluation (Section 3.1), where subjectswere given a script of commands, in this latterevaluation subjects were only give a descriptionof each task (as precisely scripted in the WST),

9

and chose themselves the words to communicatewith the robot. While the wheelchair did not moveduring these experiments (such that the perfor-mance of the interface could be studied indepen-dently from any navigation issue), the users wereprovided with feedback about the robot’s (simu-lated) current state through the screen-based in-terface.

Throughout the interactions, the time requiredto compute the robot’s response, for each interac-tion, was on the order of 1 second (and in somecases much less). In general, most of the compu-tation time is taken by the speech recognition; thegrammatical analysis, POMDP decision-making,and behavior selection are comparatively much faster.

Most of the subjects tackled 21 of the tasks inthe Wheelchair Skills Test (version WST-P/CG),except subject 3, who tackled 22 tasks (due to aprotocol error on the part of the tester) and sub-ject 4, who did 19 (due to recording issues). Taskswere presented to the test subjects using instruc-tions specified in the WST manual. E.g. Navigate

your chair over the finish line one meter

ahead of you, Put your chair into the fast

speed setting, and Turn the wheelchair to

your right about 90 degrees. The phrasingused to present tasks to the test subjects was suchas to allow flexibility in the actual phrases used toissue the commands to the chair, as opposed to thesubjects simply repeating the task description.

Table 2 shows results for this set of test sub-jects. First, we notice that the speech recognitionrates are very poor (much worse than the prelim-inary study described in Section 3.1). This is ex-plained by the fact that in this case, subjects werefree to formulate sentences as they wished, ratherthan reading off a script. Furthermore, they werenot even informed of the vocabulary accepted bythe speech recognizer. The number of sentencesparsed by the CCG module is also quite low, thoughthis is not necessarily a problem, as we demon-strate through an example below. The number ofcorrectly identified tasks shows substantial vari-ance between subjects, with five of the subjectsachieving high performance on the overall task, de-spite the poor recognition accuracy.

The number of speech commands per tasks cap-tures the frequency of queries issued to help resolvethe task at hand. If we consider subject 7, 2.10commands were required for each task on average,meaning one command for the task itself and withan additional 1.10 queries (on average) for each

task. Subject 3 required significantly fewer queriesper tasks, with only one query for every third ac-tion, due to a higher recognition accuracy. In mostcases, whenever the task was incorrectly identified,a very similar task was selected. Prior to startingthe experiments, subjects were instructed that ifat any point in time they were unsatisfied withthe interface’s behavior (e.g. wrong task was se-lected, or repetitive queries), they could use a spe-cific command (Stop) to end the task and moveon to the next one. This rarely happened, and re-sults for these trials are included as incorrect tasksin column 5 of Table 2. More flexible methods oferror recovery will be investigated in future phasesof the project.

We now consider a few illustrative interactionstranscribed from the experiment.

Example 1: Figure 3 shows a recorded dia-logue sequence. This example shows the flexibilitythat is afforded by the modularity of our archi-tecture. In the first steps of the interaction, thesentence is correctly parsed by the NLP gram-mar. The Interaction Manager nonetheless choosesa clarification query because the information pro-vided is insufficient to select a task. In the lat-ter steps, the user provides very succinct answers,which cannot be parsed by the grammar, yet theInteraction Manager is able to use these (underthe bag-of-word assumption) to correctly updateits state distribution and eventually produce thecorrect action.

Example 2: Looking at the subjects who hadlow task correctness rates yields useful insight intoweaknesses of the current system. Test subject 5was extremely verbose while issuing commands,saying for example (Figure 4) Let’s move to-

wards the wall and end up parallel to

the wall in an attempt to align the wheelchairto the wall, whereas most other subjects simplysaid align the chair to the wall. The systemwas often able to infer the task based on informa-tive words such as wall, though in some casesthe system was unable to compensate. The op-posite effect was observed in test subject 2, whoused extremely terse language, issuing commandssuch as back and fast to capture tasks such asmove backwards one meter and move for-

ward quickly.For some subjects, the task recognition rate is

high, but the number of queries used is also rel-atively high. This generally indicates poor speechrecognition. In our study, subjects 1 and 6 are non-native English speakers, with residual accents, for

10

Subject Word Sentence % Sentence % Task # CommandsId Err. Rate Err. Rate Parsed Correct per Task1 99.2 95.8 3 76 1.812 51.3 74.5 29 43 1.753 39.4 67.5 38 95 1.384 51.2 63.6 34 79 1.685 93.4 97.3 0 29 1.626 62.0 71.4 14 76 2.437 48.9 54.9 41 76 2.10

Mean 63.6 75.0 22 67 1.82Table 2 Results from Test Subject Interaction.

Command 1:

<SentenceRecognized value=‘‘true" />

<Phrase value=‘‘move backward" />

<AllPhrases value=‘‘[move backward]" />

Actual: ‘‘Move backward"

<Action value=‘‘move" type=‘‘move-action" />

<SpeechAct value=‘‘command" />

<Direction value=‘‘backward"

type=‘‘vector-direction" />

Best Action: MOVE_QUERY

Command 2:

<SentenceRecognized value=‘‘false" />

<Phrase value=‘‘backward" />

<AllPhrases value=‘‘[backward]" />

Actual: ‘‘Backward"

Best Action: MOVE_QUERY

Command 3:

<SentenceRecognized value=‘‘false" />

<Phrase value=‘‘ "/>

<AllPhrases value=‘‘[backward set]" />

Actual: ‘‘Backwards"

Best Action: drive backward

Fig. 3 A typical sequence including queries and goal task.

For each speech command, lines 2 and 3 indicate whether

the sentence could be parsed. Line 4 is the output of thespeech recognizer. Line 5 is a hand-labeled transcription.

The last line shows the action selected by the Interaction

Manager.

Command 1:

<SentenceRecognized value=‘‘false" />

<Phrase value=‘‘ " />

<AllPhrases value=‘‘[the move to wall to wall and

and and up veer overcome wall on]" />

Actual: ‘‘Let’s move towards the wall and end up

parallel to the wall"

Best Action: 21 : align to wall

Fig. 4 Example of verbose user input. Lines 2 and 3 in-dicate that the sentence could not be parsed by the gram-

mar. Line 4 is the output of the speech recognizer. Line 5

(“Actual:”) is a hand-labeled transcription added for clari-fication. Line 6 shows the action selected by the InteractionManager.

which the speech recognition system is not trained.Subject 7 also had poor recognition accuracy, pos-sibly due to a high speech volume. In each case,speaker-customization could substantially reducethe number of queries. It is nonetheless encour-aging to see that even though the recognition ac-curacy was low, the number of correct tasks wassubstantially better.

6 Related Work on Intelligent WheelchairPlatforms

Wheelchairs are a challenging platform for intelli-gent agents and are being explored by many groupsas a way to help disabled and elderly individu-als. There is an excellent, and fairly recent, reviewof intelligent wheelchair platforms [28], which pro-vides a summary of research in this area over thelast 20 years. It classifies existing systems in termsof their form factor, input methods, onboard sen-sors, and control software.

Much of the most recent work on intelligentwheelchairs focuses on the navigation behavior ofthe chair, for instance providing smooth motion toimprove user comfort and investigating sophisti-cated representations of the physical environment [10,2]. We do not review this work in detail as it tack-les issues that are orthogonal to the focus of thispaper.

Many of the earlier intelligent wheelchairs pro-totypes used a voice interface, as does the SmartWheelerplatform, to allow the user to provide commandsto the robot [25,19,13,20,1,21,3]. Earlier systemswere often prone to errors in the speech recogni-tion, which was an impediment to robust inter-action. It is worth noting that most of these sys-tems preceded the latest statistical methods forlanguage understanding and dialogue management.It is an important contribution of this paper to

11

show how this recent technology can be used toimprove the robustness of the interaction.

We are aware of one instance of an autonomouswheelchair which has a focus on human-robot in-teraction and uses probabilistic decision-makingin the dialogue manager [6], similar to the ap-proach proposed here. However, this system is lim-ited to speech-to-speech interactions, and does notinclude grammatical inference. It also has not yetbeen validated in the context of clinically-relevanttask domains.

More recently, researchers have investigated thedevelopment of methods of simplified interfaces forhuman-robot communication. This includes for ex-ample mounting a robotic arm on a wheelchair toprovide assistance for patients who have difficultymanipulating objects in their environment [35]. An-other interesting example is the work on provid-ing a concise communication protocol for enteringtext commands into the wheelchair using joysticksor touch-pads [39]. Finally, other researchers haveinvestigated the use of EEG signals to control apowered wheelchair [8]. Such work is useful for pa-tients with limited vocal abilities, but may be lesspreferred due to the lower communication band-width and the longer training time.

There is a large body of work on rehabilitationrobotics, whereby robots aim to provide instruc-tions and/or physical assistance to help individualswith disabilities to regain some of their mobility.Some of this work considers ideas of robustness andadaptation, by providing learning mechanisms toimprove the robot’s performance [34]. Such meth-ods may be useful at a later stage in our work, eventhough the application is fundamentally different.

7 Conclusion

The aims of this paper are two-fold. First, we presenta cognitive architecture for voice-driven human-robot interaction. The emphasis is on achievingrobust interaction through a number of key compo-nents: Speech Recognition, Natural Language Pro-cessing, Interaction Manager, and Behavior Man-ager. All of these components have been used inhuman-robot interfaces previously, but not alto-gether. To the best of our knowledge, this is thefirst system to integrate both grammatical parsingand probabilistic decision-making. This is an im-portant step towards achieving robust, yet flexible,human-robot interaction in complex task domain.The primary interface modality has been speech,but the architecture is well equipped to handle in-

put from other modalities, such as a joystick ortouch-sensitive display. This is the topic of ongo-ing work.

The second important aspect of this work is theemphasis on evaluating the robot platform in thecontext of a standardized, clinically relevant, do-main. The lack of formal evaluation has been iden-tified as one of the key challenges for researchersdeveloping intelligent wheelchair platforms. In thispaper, we argue that the Wheelchair Skills Test do-main is a useful and well-specified instrument forsuch a purpose. While its applicability is limited towheelchair-type devices, many of its characteristicsshould provide inspiration for developing similartools for other assistive robotic platforms. It is im-portant to remember that evaluation on standardenvironments is an important step towards demon-strating safety and security of robot platforms, andthis is especially crucial for human-robot applica-tions as we prepare to move towards more naturalliving environments.

One of the risks of focusing evaluation on highlystandardized test environment is that it may leadto over-specialization of the robot interface andplatform to this precise environment. It is certainlythe case that certain components (e.g. the vocab-ulary of the Speech Recognizer, the logical rules inthe grammar, and the state and action sets usedin the Interaction Manager) were designed to betask specific. Nonetheless the overall cognitive ar-chitecture we proposed could be re-used withoutmodification for a wide array of task domains. Infact, we have used the architecture and platformfor a number of experiments unrelated to those de-scribed in this paper without much effort.

In conclusion, the work presented here offersuseful insights into the design and evaluation ofvoice-controlled interfaces for intelligent robots. Themain focus is on the integration of componentssuch as grammatical parsing and probabilistic decision-making, which offer robust processing of the inter-face data. While the success rate is still far fromperfect, we show that the integration of these com-ponents contributes to improving the robustness ofthe system, compared to using only speech recogni-tion as an input to select behaviors. An importantnext step is to validate these results with the tar-get population. We anticipate that the results willbe similar, especially for subjects whose communi-cation abilities are intact, despite having mobilityimpairments.

In the longer term, we also aim to improve thetask success rate, as well as tackle a larger set of

12

tasks, such that the system can be effective in arich set of situations. One important direction forimproving the rate of success is in customizing theinterface for each user. This can be achieved in anumber of ways, for example by exploiting a user-specific acoustic model for the speech recognition,or by learning the grammatical inference rules au-tomatically from user data, as well as by adaptingthe reward function of the POMDP-based inter-action manager based on user preferences. Thereexist good algorithmic methods for realizing theseimprovements, however these are often data-expensive.There are important research opportunities in find-ing ways to achieve user customization without re-quiring excessive amounts of data.

Acknowledgements The authors would like to thank itsresearch partners, in particular Paul Cohen from Ecole Poly-

technique de Montreal, Robert Forget of the Universite de

Montreal, and the clinicians at the Centres de readaptationLucie-Bruneau and Constance-Lethbridge. The authors also

acknowledge financial support by the Natural Sciences and

Engineering Council Canada (NSERC), as well as the FondsQuebecois de la recherche sur la nature et les technologies

(FQRNT).

References

1. Amori, R.D.: Vocomotionan intelligent voice-controlsys-tem for powered wheelchairs. In: Proceedings of the

RESNA 1992 Annual Conference, pp. 421–423 (1992)

2. Beeson, P., MacMahon, M., Modayil, J., Murarka, A.,

Kuipers, B., Stankiewicz, B.: Integrating multiple rep-

resentations of spaitla knowledge for mapping, naviga-tion, and communcation. In: AAAI Spring Symposium

Series 2007, Interaction Challenges for Intelligent As-

sistants, AAAI Technical Report SS-07-04 (2007)

3. Clark, J.A., Roemer, R.B.: Voice controlled wheelchair.Arch Phys Med Rehabil 58(4), 169–175. (1977)

4. CMU: Sphinx-4 (2004). http://cmusphinx.

sourceforge.net/sphinx4/

5. Collett, T.H.J., MacDonald, B.A., Gerkey, B.P.: Player2.0: Toward a practical robot programming frame-

work. In: Proceedings of the Australasian Conference

on Robotics and Automation (2005)

6. Doshi, F., Roy, N.: Efficient model learning for dialog

management. In: International Conference on Human-Robot Interaction, pp. 65–72 (2007). DOI http://doi.

acm.org/10.1145/1228716.1228726

7. Fehr, L., Langbein, E., Skaar, S.B.: Adequacy of powerwheelchair control interfaces for persons with severedisabilities: A clinical survey. J. of Rehabilitation Re-search & Development 37, 353–360 (2000)

8. Felzer, T., Freisleben, B.: Hawcos: the ”hands-free”

wheelchair control system. In: Proceedings of the fifth

international ACM conference on Assistive technologies(2002)

9. Festival: Festival speech synthesis system (2004). http://www.cstr.ed.ac.uk/projects/festival/

10. Gulati, S., Kuipers, B.: High performance control forgraceful motion of an intelligent wheelchair. In: Pro-

ceedings of the IEEE International Conference on

Robotics and Automation (ICRA) (2008)11. Hoey, J., Poupart, P., Boutilier, C., Mihailidis, A.:

POMDP models for assistive technology. In: AAAIFall Symposium on Caring Machines: AI in Eldercare

(2005)12. HTK: HTK 3.4 (2006). http://htk.eng.cam.ac.uk/

13. Katevas, N.I., Sgouros, N.M., Tzafestas, S.G.,

Papakonstantinou, G., Beattie, P., Bishop, J.M.,Tsanakas, P., Koutsouris, D.: The autonomous mobile

robot senario: A sensor-aided intelligent navigation

system for powered wheelchairs. In: IEEE Roboticsand Automation Magazine, pp. 60–70 (1997)

14. Kilkens, O.J.E., Post, M.W.M., Dallmeijer, A.J., See-

len, H.A.M., van der Woude, L.H.V.: Wheelchair skillstests: A systematic review. Clinical Rehabilitation 17,

418–430 (2003)15. Kirby, R.L.: Wheelchair Skills Program (WSP), Ver-

sion 4.1. Wheelchair Skills Test (WST) Manual (2007).

http://www.wheelchairskillsprogram.ca/eng/4.1/

WSTManual Version 4.1.pdf (accessed 04/27/2008)16. Kirby, R.L., Dupuis, D.J., MacPhee, A.H., Coolen,

A.L., Smith, C., Best, K.L., Newton, A.M., Mountain,

A.D., MacLeod, D.A., Bonaparte, J.P.: The Wheelchair

Skills Test (version 2.4): Measurement properties.Arch. Phys. Med. Rehabil. 85, 794–804 (2004)

17. Kirby, R.L., Swuste, J., Dupuis, D.J., MacLeod, D.A.,Monroe, R.: The Wheelchair Skills Test: A pilot study

of a new outcome measure. Arch. Phys. Med. Rehabil.

83, 10–18 (2002)18. Koenig, S., Simmons, R.: Unsupervised learning of

probabilistic models for robot navigation. In: Interna-

tional Conference on Robotics and Automation (1996)19. Levine, S.P., Bell, D.A., Jaros, L.A., Simpson, R.C.,

Koren, Y., Borenstein, J.: The navchair assistivewheelchair navigation system. IEEE Transactions on

Rehabilitation Engineering 7(4), 443–451 (1999)20. McGuire, W.R.: Voice operated wheelchair using dig-

ital signal processing technology. In: Proceedings of

the 22nd Annual International Conference on Assistive

Technology for People with Disabilities (RESNA), pp.364–366 (1999)

21. Miller, G.E., Brown, T.E., Randolph, W.R.: Voice con-troller for wheelchairs. Med Biol Eng Comput. 23(6),597–600 (1985)

22. Montemerlo, M., Roy, N., Thrun, S.: Perspectives onstandardization in mobile robot programming: The

Carnegie Mellon navigation (CARMEN) toolkit. In:Proceedings of the IEEE/RSJ International Conference

on Intelligent Robots and Systems (IROS), vol. 3, pp.

pp 2436–2441 (2003)23. Pineau, J., Atrash, A.: SmartWheeler: A robotic

wheelchair test-bed for investigating new models ofhuman-robot interaction. In: AAAI Spring Symposiumon Multidisciplinary Collaboration for Socially Assis-

tive Robotics, pp. 59–64 (2007)24. Pineau, J., Gordon, G., Thrun, S.: Point-based value

iteration: An anytime algorithm for POMDPs. In: Pro-

ceedings of the 18th International Joint Conference onArtificial Intelligence (IJCAI), pp. 1025–1032 (2003)

25. Pires, G., Nunes, U.: A wheelchair steered through

voice commands and assisted by a reactive fuzzy-logiccontroller. Journal of Intelligent Robotic Systems

34(3), 301–314 (2002)26. Routhier, F., Vincent, C., Desrosiers, J., Nadeau, S.:

Mobility of wheelchair users: A proposed performance

assessment framework. Disability & Rehabilitation 25,19–34 (2003)

13

27. Roy, N., Pineau, J., Thrun, S.: Spoken dialog manage-

ment using probabilistic reasoning. In: Proceedings ofthe 38th Annual Meeting of the Association for Com-

putational Linguistics (ACL) (2000)28. Simpson, R.C.: Smart wheelchairs: A literature re-

view. Journal of Rehabilitation Research & Develop-ment 42(4), 423–436 (2005)

29. Simpson, R.C., LoPresti, E.F., Cooper, R.A.: Howmany people would benefit from a smart wheelchair?

J. of Rehabilitation Research & Development 45, 53–

72 (2008)30. Singh, S., Litman, D., Kearns, M., Walker, M.: Opti-

mizing dialogue management with reinforcement learn-

ing: Experiments with the NJFun system. Journal

of Artificial Intelligence Research (JAIR) 16, 105–133(2002)

31. SmartWheeler: (2008). http://www.cs.mcgill.ca/

~smartwheeler/data.php

32. Sondik, E.: The optimal control of partially observable

Markov decision processes. Ph.D. thesis, Stanford Uni-

versity (1971)33. Steedman, M., Baldridge, J.: Combinatory Categorial

Grammar. Blackwell (2005). ftp://ftp.cogsci.ed.

ac.uk/pub/steedman/ccg/manifesto.pdf

34. Tapus, A., Tapus, C., Mataric, M.: Hands-off therapist

robot behavior adaptation to user personality for post-

stroke rehabilitation therapy. In: International Con-ference on Robotics and Automation, pp. 1547–1553

(2007)35. Tsui, K., Yanco, H.: Simplifying wheelchair mounted

robotic arm control with a visual interface. In: AAAI

Spring Symposium on Multidisciplinary Collaborationfor Socially Assistive Robots, pp. 97–102 (2007)

36. Vertanen, K.: HTK wall street journal acoustic models

(2006). URL http://www.inference.phy.cam.ac.uk/

kv227/htk/acoustic_models.html

37. White, M.: OpenCCG: The OpenNLP CCG Library

(2001). http://openccg.sourceforge.net

38. Williams, J., Poupart, P., Young, S.: Partially observ-able Markov decision processes with continuous obser-

vations for dialogue management. In: SigDial Work-

shop on Discourse and Dialogue, pp. 393–422 (2005)39. Wobbrock, J.O., Myers, B.A., Aung, H.H., LoPresti,

E.F.: Text entry from power wheelchairs: EdgeWritefor joysticks and touchpads. In: Proceedings of the 6th

international ACM SIGACCESS conference on Com-

puters and accessibility (2004)40. Zalzal, V.: Acropolis: Une architecture logicielle

evolutive pour plate-formes mobiles autonomes:

Principes de fonctionnement. Tech. rep., Ecole

Polytechnique de Montreal (2004)

Amin Atrash is a Ph.D. candidate in ComputerScience at McGill University. He received his B.Sc.and M.Sc. in Computer Science at the Georgia In-stitute of Technology. His research interests focuson planning under uncertainty and human-robotinteraction.Robert Kaplow is an M.Sc. candidate in Com-puter Science at McGill University, where he previ-ously completed is B.Sc. His research interests in-clude mobile robot navigation, probabilistic plan-ning and reinforcement learning.Julien Villemure received his B.Sc. in ComputerScience from McGill University in 2008, where he

is now an M.Sc. candidate. His research focuses ondeveloping and validating software architecturesfor human-robot interaction systems.Robert West received received a Diplom in Com-puter Science from the Technische Universitat Munchen,then joined McGill’s School of Computer Science,where he is an M.Sc. candidate. His research inter-ests are in natural language processing, and in de-veloping novel methods for inferring semantic dis-tances between concepts.Hiba Yamani completed her M.Sc. in ComputerScience at McGill University in 2009. She is now anapplication developer at Nuance Communications.Joelle Pineau is Assistant Professor of ComputerScience at McGill University. She received her Ph.D.from Carnegie Mellon University in 2004. Her re-search interests are in artificial intelligence, prob-abilistic machine learning, and robotics.

14