MATCH: An Architecture for Multimodal Dialogue Systems

Michael Johnston, Srinivas Bangalore, Gunaranjan Vasireddy, Amanda Stent�Patrick Ehlen, Marilyn Walker, Steve Whittaker, Preetam MaloorAT&T Labs - Research, 180 Park Ave, Florham Park, NJ 07932, USA

johnston,srini,guna,ehlen,walker,stevew,pmaloor@research.att.com�Now at SUNY Stonybrook, stent@cs.sunysb.edu

Abstract

Mobile interfaces need to allow the userand system to adapt their choice of com-munication modes according to user pref-erences, the task at hand, and the physi-cal and social environment. We describe amultimodal application architecture whichcombines finite-state multimodal languageprocessing, a speech-act based multimodaldialogue manager, dynamic multimodaloutput generation, and user-tailored textplanning to enable rapid prototyping ofmultimodal interfaces with flexible inputand adaptive output. Our testbed appli-cation MATCH (Multimodal Access ToCity Help) provides a mobile multimodalspeech-pen interface to restaurant and sub-way information for New York City.

1 Multimodal Mobile Information Access

In urban environments tourists and residents alikeneed access to a complex and constantly changingbody of information regarding restaurants, theatreschedules, transportation topology and timetables.This information is most valuable if it can be de-livered effectively while mobile, since places closeand plans change. Mobile information access devices(PDAs, tablet PCs, next-generation phones) offerlimited screen real estate and no keyboard or mouse,making complex graphical interfaces cumbersome.Multimodal interfaces can address this problem byenabling speech and pen input and output combiningspeech and graphics (See (Andre, 2002) for a detailedoverview of previous work on multimodal input andoutput). Since mobile devices are used in differentphysical and social environments, for different tasks,by different users, they need to be both flexible in in-put and adaptive in output. Users need to be able to

provide input in whichever mode or combination ofmodes is most appropriate, and system output shouldbe dynamically tailored so that it is maximally effec-tive given the situation and the user’s preferences.

We present our testbed multimodal applicationMATCH (Multimodal Access To City Help) and thegeneral purpose multimodal architecture underlyingit, that: is designed for highly mobile applications;enables flexible multimodal input; and provides flex-ible user-tailored multimodal output.

Figure 1: MATCH running on Fujitsu PDA

Highly mobile MATCH is a working city guideand navigation system that currently enables mobileusers to access restaurant and subway information forNew York City (NYC). MATCH runs standalone ona Fujitsu pen computer (Figure 1), and can also runin client-server mode across a wireless network.

Flexible multimodal input Users interact with agraphical interface displaying restaurant listings anda dynamic map showing locations and street infor-mation. They are free to provide input using speech,by drawing on the display with a stylus, or by us-ing synchronous multimodal combinations of the twomodes. For example, a user might ask to see cheap

Computational Linguistics (ACL), Philadelphia, July 2002, pp. 376-383. Proceedings of the 40th Annual Meeting of the Association for

Italian restaurants in Chelsea by saying show cheapitalian restaurants in chelsea, by circling an area onthe map and saying show cheap italian restaurantsin this neighborhood; or, in a noisy or public envi-ronment, by circling an area and writing cheap anditalian (Figure 2). The system will then zoom to theappropriate map location and show the locations ofrestaurants on the map. Users can ask for informationabout restaurants, such as phone numbers, addresses,and reviews. For example, a user might circle threerestaurants as in Figure 3 and say phone numbers forthese three restaurants (or write phone). Users canalso manipulate the map interface directly. For exam-ple, a user might say show upper west side or circlean area and write zoom.

Figure 2: Unimodal pen command

Flexible multimodal output MATCH providesflexible, synchronized multimodal generation andcan take initiative to engage in information-seekingsubdialogues. If a user circles the three restaurants inFigure 3 and writes phone, the system responds witha graphical callout on the display, synchronized witha text-to-speech (TTS) prompt of the phone number,for each restaurant in turn (Figure 4).

Figure 3: Two area gestures

Figure 4: Phone query callouts

The system also provides subway directions. If theuser says How do I get to this place? and circles one

of the restaurants displayed on the map, the systemwill ask Where do you want to go from? The usercan then respond with speech (e.g., 25th Street and3rd Avenue), with pen by writing (e.g., 25th St & 3rdAve), or multimodally ( e.g, from here with a circlegesture indicating location). The system then calcu-lates the optimal subway route and dynamically gen-erates a multimodal presentation of instructions. Itstarts by zooming in on the first station and then grad-ually zooms out, graphically presenting each stage ofthe route along with a series of synchronized TTSprompts. Figure 5 shows the final display of a sub-way route heading downtown on the 6 train and trans-ferring to the L train Brooklyn bound.

Figure 5: Multimodal subway route

User-tailored generation MATCH can also pro-vide a user-tailored summary, comparison, or rec-ommendation for an arbitrary set of restaurants, us-ing a quantitative model of user preferences (Walkeret al., 2002). The system will only discuss restau-rants that rank highly according to the user’s diningpreferences, and will only describe attributes of thoserestaurants the user considers important. This per-mits concise, targeted system responses. For exam-ple, the user could say compare these restaurants andcircle a large set of restaurants (Figure 6). If the userconsiders inexpensiveness and food quality to be themost important attributes of a restaurant, the systemresponse might be:

Compare-A: Among the selected restaurants, the followingoffer exceptional overall value. Uguale’s price is 33 dollars. Ithas excellent food quality and good decor. Da Andrea’s price is28 dollars. It has very good food quality and good decor. John’sPizzeria’s price is 20 dollars. It has very good food quality andmediocre decor.

Figure 6: Comparing a large set of restaurants

2 Multimodal Application Architecture

The multimodal architecture supporting MATCHconsists of a series of agents which communicatethrough a facilitator MCUBE (Figure 7).

Figure 7: Multimodal Architecture

MCUBE is a Java-based facilitator which enablesagents to pass messages either to single agents orgroups of agents. It serves a similar function to sys-tems such as OAA (Martin et al., 1999), the use ofKQML for messaging in Allen et al (2000), and theCommunicator hub (Seneff et al., 1998). Agents mayreside either on the client device or elsewhere on thenetwork and can be implemented in multiple differ-ent languages. MCUBE messages are encoded inXML, providing a general mechanism for messageparsing and facilitating logging.

Multimodal User Interface Users interact withthe system through the Multimodal UI, which isbrowser-based and runs in Internet Explorer. Thisgreatly facilitates rapid prototyping, authoring, andreuse of the system for different applications sinceanything that can appear on a webpage (dynamicHTML, ActiveX controls, etc.) can be used in

the visual component of a multimodal user inter-face. A TCP/IP control enables communication withMCUBE.

MATCH uses a control that provides a dynamicpan-able, zoomable map display. The control has inkhandling capability. This enables both pen-based in-teraction (on the map) and normal GUI interaction(on the rest of the page) without requiring the user toovertly switch ‘modes’. When the user draws on themap their ink is captured and any objects potentiallyselected, such as currently displayed restaurants, areidentified. The electronic ink is broken into a lat-tice of strokes and sent to the gesture recognitionand handwriting recognition components which en-rich this stroke lattice with possible classifications ofstrokes and stroke combinations. The UI then trans-lates this stroke lattice into an ink meaning latticerepresenting all of the possible interpretations of theuser’s ink and sends it to MMFST.

In order to provide spoken input the user must tapa click-to-speak button on the Multimodal UI. Wefound that in an application such as MATCH whichprovides extensive unimodal pen-based interaction, itis preferable to use click-to-speak rather than pen-to-speak or open-mike. With pen-to-speak, spuriousspeech results received in noisy environments candisrupt unimodal pen commands.

The Multimodal UI also provides graphical outputcapabilities and performs synchronization of multi-modal output. For example, it synchronizes the dis-play actions and TTS prompts in the answer to theroute query mentioned in Section 1.

Speech Recognition MATCH uses AT&T’s Wat-son speech recognition engine. A speech managerrunning on the device gathers audio and communi-cates with a recognition server running either on thedevice or on the network. The recognition server pro-vides word lattice output which is passed to MMFST.

Gesture and handwriting recognition Gestureand handwriting recognition agents provide possibleclassifications of electronic ink for the UI. Recogni-tions are performed both on individual strokes andcombinations of strokes in the input ink lattice. Thehandwriting recognizer supports a vocabulary of 285words, including attributes of restaurants (e.g. ‘chi-nese’,‘cheap’) and zones and points of interest (e.g.‘soho’,‘empire’,‘state’,‘building’). The gesture rec-ognizer recognizes a set of 10 basic gestures, includ-ing lines, arrows, areas, points, and question marks.It uses a variant of Rubine’s classic template-basedgesture recognition algorithm (Rubine, 1991) trainedon a corpus of sample gestures. In addition to classi-

fying gestures the gesture recognition agent also ex-tracts features such as the base and head of arrows.Combinations of this basic set of gestures and hand-written words provide a rich visual vocabulary formultimodal and pen-based commands.

Gestures are represented in the ink meaning lat-tice as symbol complexes of the following form: GFORM MEANING (NUMBER TYPE) SEM. FORMindicates the physical form of the gesture and has val-ues such as area, point, line, arrow. MEANING indi-cates the meaning of that form; for example an areacan be either a loc(ation) or a sel(ection). NUMBERand TYPE indicate the number of entities in a selec-tion (1,2,3, many) and their type (rest(aurant), the-atre). SEM is a place holder for the specific contentof the gesture, such as the points that make up an areaor the identifiers of objects in a selection.

When multiple selection gestures are presentan aggregation technique (Johnston and Bangalore,2001) is employed to overcome the problems withdeictic plurals and numerals described in John-ston (2000). Aggregation augments the ink meaninglattice with aggregate gestures that result from com-bining adjacent selection gestures. This allows a de-ictic expression like these three restaurants to com-bine with two area gestures, one which selects onerestaurant and the other two, as long as their sum isthree. For example, if the user makes two area ges-tures, one around a single restaurant and the otheraround two restaurants (Figure 3), the resulting inkmeaning lattice will be as in Figure 8. The first ges-ture (node numbers 0-7) is either a reference to alocation (loc.) (0-3,7) or a reference to a restaurant(sel.) (0-2,4-7). The second (nodes 7-13,16) is eithera reference to a location (7-10,16) or to a set of tworestaurants (7-9,11-13,16). The aggregation processapplies to the two adjacent selections and adds a se-lection of three restaurants (0-2,4,14-16). If the usersays show chinese restaurants in this neighborhoodand this neighborhood, the path containing the twolocations (0-3,7-10,16) will be taken when this lat-tice is combined with speech in MMFST. If the usersays tell me about this place and these places, thenthe path with the adjacent selections is taken (0-2,4-9,11-13,16). If the speech is tell me about these orphone numbers for these three restaurants then theaggregate path (0-2,4,14-16) will be chosen.

Multimodal Integrator (MMFST) MMFST re-ceives the speech lattice (from the Speech Manager)and the ink meaning lattice (from the UI) and buildsa multimodal meaning lattice which captures the po-tential joint interpretations of the speech and ink in-

puts. MMFST is able to provide rapid response timesby making unimodal timeouts conditional on activityin the other input mode. MMFST is notified when theuser has hit the click-to-speak button, when a speechresult arrives, and whether or not the user is inking onthe display. When a speech lattice arrives, if inkingis in progress MMFST waits for the ink meaning lat-tice, otherwise it applies a short timeout (1 sec.) andtreats the speech as unimodal. When an ink meaninglattice arrives, if the user has tapped click-to-speakMMFST waits for the speech lattice to arrive, other-wise it applies a short timeout (1 sec.) and treats theink as unimodal.

MMFST uses the finite-state approach to multi-modal integration and understanding proposed byJohnston and Bangalore (2000). Possibilities formultimodal integration and understanding are cap-tured in a three tape device in which the first taperepresents the speech stream (words), the second theink stream (gesture symbols) and the third their com-bined meaning (meaning symbols). In essence, thisdevice takes the speech and ink meaning lattices asinputs, consumes them using the first two tapes, andwrites out a multimodal meaning lattice using thethird tape. The three tape finite-state device is sim-ulated using two transducers: G:W which is used toalign speech and ink and G W:M which takes a com-posite alphabet of speech and gesture symbols as in-put and outputs meaning. The ink meaning latticeG and speech lattice W are composed with G:W andthe result is factored into an FSA G W which is com-posed with G W:M to derive the meaning lattice M.

In order to capture multimodal integration usingfinite-state methods, it is necessary to abstract overspecific aspects of gestural content (Johnston andBangalore, 2000). For example, all possible se-quences of coordinates that could occur in an areagesture cannot be encoded in the finite-state device.We employ the approach proposed in (Johnston andBangalore, 2001) in which the ink meaning lattice isconverted to a transducer I:G, where G are gesturesymbols (including SEM) and I contains both gesturesymbols and the specific contents. I and G differ onlyin cases where the gesture symbol on G is SEM, inwhich case the corresponding I symbol is the specificinterpretation. After multimodal integration a pro-jection G:M is taken from the result G W:M machineand composed with the original I:G in order to rein-corporate the specific contents that were left out ofthe finite-state process (I:G o G:M = I:M).

The multimodal finite-state transducers used atruntime are compiled from a declarative multimodalcontext-free grammar which captures the structure

Figure 8: Ink Meaning Lattice

and interpretation of multimodal and unimodal com-mands, approximated where necessary using stan-dard approximation techniques (Nederhof, 1997).This grammar captures not just multimodal integra-tion patterns but also the parsing of speech and ges-ture, and the assignment of meaning. In Figure 9 wepresent a small simplified fragment capable of han-dling MATCH commands such as phone numbers forthese three restaurants. A multimodal CFG differsfrom a normal CFG in that the terminals are triples:W:G:M, where W is the speech stream (words), Gthe ink stream (gesture symbols) and M the meaningstream (meaning symbols). An XML representationfor meaning is used to facilate parsing and loggingby other system components. The meaning tape sym-bols concatenate to form coherent XML expressions.The epsilon symbol (eps) indicates that a stream isempty in a given terminal.

When the user says phone numbers for thesethree restaurants and circles two groups of restau-rants (Figure 3). The gesture lattice (Figure 8) isturned into a transducer I:G with the same sym-bol on each side except for the SEM arcs which aresplit. For example, path 15-16 SEM([id1,id2,id3])becomes [id1,id2,id3]:SEM. After G and the speechW are integrated using G:W and G W:M. The G pathin the result is used to re-establish the connectionbetween SEM symbols and their specific contentsin I:G (I:G o G:M = I:M). The meaning read offI:M is<cmd><phone><restaurant> [id1,id2,id3]</restaurant> </phone> </cmd>. This is passedto the multimodal dialog manager (MDM) and fromthere to the Multimodal UI resulting in a display likeFigure 4 with coordinated TTS output. Since thespeech input is a lattice and there is also potentialfor ambiguity in the multimodal grammar, the outputfrom MMFST to MDM is an N-best list of potentialmultimodal interpretations.

Multimodal Dialog Manager (MDM) The MDMis based on previous work on speech-act based mod-els of dialog (Stent et al., 1999; Rich and Sidner,1998). It uses a Java-based toolkit for writing dialogmanagers that is similar in philosophy to TrindiKit(Larsson et al., 1999). It includes several rule-based

S ! eps:eps:<cmd> CMD eps:eps:</cmd>CMD ! phone:eps:<phone> numbers:eps:eps

for:eps:eps DEICTICNPeps:eps:</phone>

DEICTICNP ! DDETPL eps:area:eps eps:selection:epsNUM RESTPL eps:eps:<restaurant>eps:SEM:SEM eps:eps:</restaurant>

DDETPL ! these:G:epsRESTPL ! restaurants:restaurant:epsNUM ! three:3:eps

Figure 9: Multimodal grammar fragment

processes that operate on a shared state. The stateincludes system and user intentions and beliefs, a di-alog history and focus space, and information aboutthe speaker, the domain and the available modalities.The processes include interpretation, update, selec-tion and generation processes.

The interpretation process takes as input an N-bestlist of possible multimodal interpretations for a userinput from MMFST. It rescores them according to aset of rules that encode the most likely next speechact given the current dialogue context, and picks themost likely interpretation from the result. The updateprocess updates the dialogue context according to thesystem’s interpretation of user input. It augments thedialogue history, focus space, models of user and sys-tem beliefs, and model of user intentions. It also al-ters the list of current modalities to reflect those mostrecently used by the user.

The selection process determines the system’s nextmove(s). In the case of a command, request or ques-tion, it first checks that the input is fully specified(using the domain ontology, which contains informa-tion about required and optional roles for differenttypes of actions); if it is not, then the system’s nextmove is to take the initiative and start an information-gathering subdialogue. If the input is fully specified,the system’s next move is to perform the command oranswer the question; to do this, MDM communicateswith the UI. Since MDM is aware of the current setof preferred modalities, it can provide feedback andresponses tailored to the user’s modality preferences.

The generation process performs template-basedgeneration for simple responses and updates the sys-tem’s model of the user’s intentions after generation.The text planner is used for more complex genera-

tion, such as the generation of comparisons.In the route query example in Section 1, MDM first

receives a route query in which only the destinationis specified How do I get to this place? In the se-lection phase it consults the domain model and de-termines that a source is also required for a route.It adds a request to query the user for the source tothe system’s next moves. This move is selected andthe generation process selects a prompt and sends itto the TTS component. The system asks Where doyou want to go from? If the user says or writes 25thStreet and 3rd Avenue then MMFST will assign thisinput two possible interpretations. Either this is a re-quest to zoom the display to the specified location orit is an assertion of a location. Since the MDM dia-logue state indicates that it is waiting for an answerof the type location, MDM reranks the assertion asthe most likely interpretation. A generalized overlayprocess (Alexandersson and Becker, 2001) is used totake the content of the assertion (a location) and addit into the partial route request. The result is deter-mined to be complete. The UI resolves the locationto map coordinates and passes on a route request tothe SUBWAY component.

We found this traditional speech-act based dia-logue manager worked well for our multimodal inter-face. Critical in this was our use of a common seman-tic representation across spoken, gestured, and multi-modal commands. The majority of the dialogue rulesoperate in a mode-independent fashion, giving usersflexibility in the mode they choose to advance the di-alogue. On the other hand, mode sensitivity is alsoimportant since user modality choice can be used todetermine system mode choice for confirmation andother responses.

Subway Route Constraint Solver (SUBWAY)This component has access to an exhaustive databaseof the NYC subway system. When it receives a routerequest with the desired source and destination pointsfrom the Multimodal UI, it explores the search spaceof possible routes to identify the optimal one, using acost function based on the number of transfers, over-all number of stops, and the walking distance fromthe station at each end. It builds a list of actions re-quired to reach the destination and passes them to themultimodal generator.

Multimodal Generator and Text-to-speech Themultimodal generator processes action lists fromSUBWAY and other components and assigns appro-priate prompts for each action using a template-basedgenerator. The result is a ‘score’ of prompts and ac-tions which is passed to the Multimodal UI. The Mul-

timodal UI plays this ‘score’ by coordinating changesin the interface with the corresponding TTS prompts.AT&T’s Natural Voices TTS engine is used to pro-vide the spoken output. When the UI receives a mul-timodal score, it builds a stack of graphical actionssuch as zooming the display to a particular locationor putting up a graphical callout. It then sends theprompts to be rendered by the TTS server. As eachprompt is synthesized the TTS server sends progressnotifications to the Multimodal UI, which pops thenext graphical action off the stack and executes it.

Text Planner and User Model The text plan-ner receives instructions from MDM for executionof ‘compare’, ‘summarize’, and ‘recommend’ com-mands. It employs a user model based on multi-attribute decision theory (Carenini and Moore, 2001).For example, in order to make a comparison betweenthe set of restaurants shown in Figure 6, the textplanner first ranks the restaurants within the set ac-cording to the predicted ranking of the user model.Then, after selecting a small set of the highest rankedrestaurants, it utilizes the user model to decide whichrestaurant attributes are important to mention. Theresulting text plan is converted to text and sent to TTS(Walker et al., 2002). A user model for someone whocares most highly about cost and secondly about foodquality and decor leads to a system response such asthat in Compare-A above. A user model for someonewhose selections are driven by food quality and foodtype first, and cost only second, results in a systemresponse such as that shown in Compare-B.

Compare-B: Among the selected restaurants, the following of-fer exceptional overall value. Babbo’s price is 60 dollars. It hassuperb food quality. Il Mulino’s price is 65 dollars. It has superbfood quality. Uguale’s price is 33 dollars. It has excellent food.

Note that the restaurants selected for the user whois not concerned about cost includes two rather moreexpensive restaurants that are not selected by the textplanner for the cost-oriented user.

Multimodal Logger User studies, multimodal datacollection, and debugging were accomplished by in-strumenting MATCH agents to send details of userinputs, system processes, and system outputs to a log-ger agent that maintains an XML log designed formultimodal interactions. Our critical objective wasto collect data continually throughout system devel-opment, and to be able to do so in mobile settings.While this rendered the common practice of video-taping user interactions impractical, we still requiredhigh fidelity records of each multimodal interaction.To address this problem, MATCH logs the state ofthe UI and the user’s ink, along with detailed data

from other components. These components can inturn dynamically replay the user’s speech and ink asthey were originally received, and show how the sys-tem responded. The browser- and component-basedarchitecture of the Multimodal UI facilitated its reusein a Log Viewer that reads multimodal log files, re-plays interactions between the user and system, andallows analysis and annotation of the data. MATCH’slogging system is similar in function to STAMP (Ovi-att and Clow, 1998), but does not require multimodalinteractions to be videotaped and allows rapid re-configuration for different annotation tasks since itis browser-based. The ability of the system to logdata standalone is important, since it enables testingand collection of multimodal data in realistic mobileenvironments without relying on external equipment.

3 Experimental Evaluation

Our multimodal logging infrastructure enabledMATCH to undergo continual user trials and evalu-ation throughout development. Repeated evaluationswith small numbers of test users both in the lab andin mobile settings (Figure 10) have guided the designand iterative development of the system.

Figure 10: Testing MATCH in NYC

This iterative development approach highlightedseveral important problems early on. For example,while it was originally thought that users would for-mulate queries and navigation commands primarilyby specifying the names of New York neighborhoods,as in show italian restaurants in chelsea, early fieldtest studies in the city revealed that the need forneighborhood names in the grammar was minimalcompared to the need for cross-streets and points ofinterest; hence, cross-streets and a sizable list of land-marks were added. Other early tests revealed theneed for easily accessible ‘cancel’ and ‘undo’ fea-tures that allow users to make quick corrections. We

also discovered that speech recognition performancewas initially hindered by placement of the ‘click-to-speak’ button and the recognition feedback box onthe bottom-right side of the device, leading manyusers to speak ‘to’ this area, rather than toward themicrophone on the upper left side. This placementalso led left-handed users to block the microphonewith their arms when they spoke. Moving the but-ton and the feedback box to the top-left of the deviceresolved both of these problems.

After initial open-ended piloting trials, more struc-tured user tests were conducted, for which we devel-oped a set of six scenarios ordered by increasing levelof difficulty. These required the test user to solveproblems using the system. These scenarios were leftas open-ended as possible to elicit natural responses.

Sample scenario:You have plans to meet your aunt for dinnerlater this evening at a Thai restaurant on the Upper West Sidenear her apartment on 95th St. and Broadway. Unfortunately,you forgot what time you’re supposed to meet her, and you can’treach her by phone. Use MATCH to find the restaurant and writedown the restaurant’s telephone number so you can check on thereservation time.

Test users received a brief tutorial that was inten-tionally vague and broad in scope so the users mightoverestimate the system’s capabilities and approachproblems in new ways. Figure 11 summarizes re-sults from our last scenario-based data collection fora fixed version of the system. There were five sub-jects (2 male, 3 female) none of whom had been in-volved in system development. All of these five testswere conducted indoors in offices.

exchanges 338 asr word accuracy 59.6%speech only 171 51% asr sent. accuracy 36.1%multimodal 93 28% handwritten sent. acc. 64%pen only 66 19% task completion rate 85%GUI actions 8 2% average time/scenario 6.25m

Figure 11: MATCH study

There were an average of 12.75 multimodal ex-changes (pairs of user input and system response) perscenario. The overall time per scenario varied from1.5 to to 15 minutes. The longer completion timesresulted from poor ASR performance for some of theusers. Although ASR accuracy was low, overall taskcompletion was high, suggesting that the multimodalaspects of the system helped users to complete tasks.Unimodal pen commands were recognized more suc-cessfully than spoken commands; however, only 19%of commands were pen only. In ongoing work, weare exploring strategies to increase users’ adoption ofmore robust pen-based and multimodal input.

MATCH has a very fast system response time.Benchmarking a set of speech, pen, and multimodalcommands, the average response time is approxi-mately 3 seconds (time from end of user input to sys-tem response). We are currently completing a largerscale scenario-based evaluation and an independentevaluation of the functionality of the text planner.

In addition to MATCH, the same multimodal ar-chitecture has been used for two other applications:a multimodal interface to corporate directory infor-mation and messaging and a medical application toassist emergency room doctors. The medical proto-type is the most recent and demonstrates the utility ofthe architecture for rapid prototyping. System devel-opment took under two days for two people.

4 Conclusion

The MATCH architecture enables rapid develop-ment of mobile multimodal applications. Combin-ing finite-state multimodal integration with a speech-act based dialogue manager enables users to interactflexibly using speech, pen, or synchronized combina-tions of the two depending on their preferences, task,and physical and social environment. The systemresponds by generating coordinated multimodal pre-sentations adapted to the multimodal dialog contextand user preferences. Features of the system suchas the browser-based UI and general purpose finite-state architecture for multimodal integration facili-tate rapid prototyping and reuse of the technology fordifferent applications. The lattice-based finite-stateapproach to multimodal understanding enables bothmultimodal integration and dialogue context to com-pensate for recognition errors. The multimodal log-ging infrastructure has enabled an iterative processof pro-active evaluation and data collection through-out system development. Since we can replay multi-modal interactions without video we have been ableto log and annotate subjects both in the lab and inNYC throughout the development process and usetheir input to drive system development.

Acknowledgements

Thanks to AT&T Labs and DARPA (contract MDA972-99-3-0003) for financial support. We would also like to thank NoemieElhadad, Candace Kamm, Elliot Pinson, Mazin Rahim, OwenRambow, and Nika Smith.

References

J. Alexandersson and T. Becker. 2001. Overlay as the ba-sic operation for discourse processing in a multimodal

dialogue system. In 2nd IJCAI Workshop on Knowl-edge and Reasoning in Practical Dialogue Systems.

J. Allen, D. Byron, M. Dzikovska, G. Ferguson,L. Galescu, and A. Stent. 2000. An architecture fora generic dialogue shell. JNLE, 6(3).

E. Andre. 2002. Natural language in multime-dia/multimodal systems. In Ruslan Mitkov, editor,Handbook of Computational Linguistics. OUP.

G. Carenini and J. D. Moore. 2001. An empirical study ofthe influence of user tailoring on evaluative argumenteffectiveness. In IJCAI, pages 1307–1314.

M. Johnston and S. Bangalore. 2000. Finite-state mul-timodal parsing and understanding. In Proceedings ofCOLING 2000, Saarbrucken, Germany.

M. Johnston and S. Bangalore. 2001. Finite-state meth-ods for multimodal parsing and integration. In ESSLLIWorkshop on Finite-state Methods, Helsinki, Finland.

M. Johnston. 2000. Deixis and conjunction in mul-timodal systems. In Proceedings of COLING 2000,Saarbrucken, Germany.

S. Larsson, P. Bohlin, J. Bos, and D. Traum. 1999.TrindiKit manual. Technical report, TRINDI Deliver-able D2.2.

D. Martin, A. Cheyer, and D. Moran. 1999. The OpenAgent Architecture: A framework for building dis-tributed software systems. Applied Artificial Intelli-gence, 13(1–2):91–128.

M-J. Nederhof. 1997. Regular approximations of CFLs:A grammatical view. In Proceedings of the Interna-tional Workshop on Parsing Technology, Boston.

S. L. Oviatt and J. Clow. 1998. An automated tool foranalysis of multimodal system performance. In Pro-ceedings of ICSLP.

C. Rich and C. Sidner. 1998. COLLAGEN: A collabora-tion manager for software interface agents. User Mod-eling and User-Adapted Interaction, 8(3–4):315–350.

D. Rubine. 1991. Specifying gestures by example. Com-puter graphics, 25(4):329–337.

S. Seneff, E. Hurley, R. Lau, C. Pao, P. Schmid, andV. Zue. 1998. Galaxy-II: A reference architecture forconversational system development. In ICSLP-98.

A. Stent, J. Dowding, J. Gawron, E. Bratt, and R. Moore.1999. The CommandTalk spoken dialogue system. InProceedings of ACL’99.

M. A. Walker, S. J. Whittaker, P. Maloor, J. D. Moore,M. Johnston, and G. Vasireddy. 2002. Speech-Plans:Generating evaluative responses in spoken dialogue. InIn Proceedings of INLG-02.