■ DiamondHelp is a generic collaborative task guid-ance system motivated by the current usability crisisin high-tech home products. It combines an applica-tion-independent conversational interface (adaptedfrom online chat programs) with an application-spe-cific direct-manipulation interface. DiamondHelp isimplemented in Java and uses Collagen for repre-senting and using task models.

Our diagnosis of the current usability cri-sis in high-tech home products (seesidebar this page) identifies two funda-

mental underlying causes: the exhaustion ofconventional interaction paradigms and thelack of consistency in user interface design.This article addresses both of these causes byintroducing a new framework for building col-laborative task guidance systems, called Dia-mondHelp.

The dominant current paradigm for human-computer interaction is direct manipulation(Shneiderman and Plaisant 2005), which canbe applied with great effectiveness and ele-gance to those aspects of an interface thatafford natural and intuitive analogies to phys-ical actions, such as pointing, dragging, sliding,and so on. In practice, however, most inter-faces of any complexity also include an ad hocand bewildering collection of other mecha-nisms, such as tool bars, pop-up windows,menus, command lines, and so on.

To make matters worse, once one goesbeyond the most basic functions (such as open,save, and so on), there is very little design con-sistency between interfaces to different prod-ucts, even from the same manufacturer. Theresult is that it requires too large an investment

of the typical user’s time to learn all the intri-cacies of each new product.

The DiamondHelp framework consists ofthree components: (1) an interaction paradigmbased on task-oriented human collaboration,(2) a graphical user interface (GUI) design thatcombines conversational and direct-manipula-tion interfaces, and (3) a software architectureof reusable JavaBeans.

The first two of these components are ori-ented towards the user; the third componentaddresses the needs of the software developer.Each of these components is described in detaillater on.

Figure 1 illustrates the DiamondHelp userinterface design. The top half of the screen isthe generic conversational interface, while thebottom half is a direct-manipulation interfacefor a combination washer-dryer. Although ourimmediate motivation has been the usabilitycrisis in DVD recorders, programmable ther-mostats, combination washer-dryers, refrigera-tor-ovens, and so on, DiamondHelp is not lim-ited to home appliances; it can be applied toany software interface.

DiamondHelp grew out of a longstandingresearch thread on human-computer collabo-ration, organized around the Collagen system(see the Collagen sidebar). Our work on Colla-gen, however, has been primarily concernedwith the underlying semantic and pragmaticstructures for modeling collaboration, whereasDiamondHelp is focused on the appearance(the “look and feel”) that a collaborative sys-tem presents to the user. Although Collagenplays a key role in our implementations of Dia-mondHelp applications (see the SoftwareArchitecture Section), the DiamondHelp

Articles

SUMMER 2007 33Copyright © 2007, Association for the Advancement of Artificial Intelligence. All rights reserved. ISSN 0738-4602

DiamondHelp: A Generic Collaborative Task Guidance System

Charles Rich and Candace L. Sidner

AI Magazine Volume 28 Number 2 (2007) (© AAAI)

framework can also be used independently ofCollagen.

Finally, in relation to this special issue of AIMagazine, we would say that mixed initiative isat the heart of DiamondHelp since collabora-tion and mixed initiative are virtually coexten-sive concepts: all collaborative systems aremixed initiative, and most interesting mixed-initiative systems are collaborative (see furtherdiscussion in the Mixed Initiative section).

Interaction ParadigmAlthough an “interaction paradigm” is some-what abstract and difficult to define formally, itis crucial to the overall development of inter-active systems. In particular, the consistentexpression and reinforcement of the interac-tion paradigm in the user interface design(what the user sees) leads to systems that areeasier to learn and use. The interaction para-digm also provides organizing principles for

the underlying software architecture, whichmakes such systems easier to build.

The collaborative paradigm for human-com-puter interaction, illustrated in figure 2, mim-ics the relationships that typically hold whentwo humans collaborate on a task involving ashared artifact, such as two mechanics workingon a car engine together or two computer usersworking on a spreadsheet together. This para-digm differs from the conventional view ofinterfaces in two key respects.

First, notice that the diagram in figure 2 issymmetric between the user and the system.Collaboration in general involves both actionand communication by both participants, andin the case of co-present collaboration (whichis the focus of DiamondHelp) the actions ofeach participant are directly observed by theother participant.

One consequence of this symmetry is thatthe collaborative paradigm spans, in a princi-pled fashion, a very broad range of interactionmodes, depending on the relative knowledgeand initiative of the system versus the user. Forexample, “tutoring” (also known as intelligentcomputer-aided instruction) is a kind of col-laboration in which the system has most of theknowledge and initiative. At the other end ofthe range is “intelligent assistance,” whereinthe user has most of the knowledge and initia-tive. Furthermore, a collaborative system caneasily and incrementally shift within this rangedepending on the current task context.

Second, and at a deeper level, the primaryrole of communication in collaboration is notfor the user to tell the system what to do (thetraditional “commands”), but rather to estab-lish and negotiate about goals and how toachieve them. J. C. R. Licklider observed thisfundamental distinction more than 40 yearsago in a classic article, which is well worthrevisiting (see the Licklider sidebar and Lesh etal. 2004).

The problem with the command view ofuser interfaces is that it demands too muchknowledge on the part of the user. Conven-tional systems attempt to compensate for thisproblem by adding various kinds of ad hochelp facilities, tool tips, wizards, and so on,which are typically not integrated into anyclear paradigm. (See Related Work for specificdiscussion of the relation of DiamondHelp towizards and Microsoft’s Clippy.) In contrast,collaboration encompasses all of these capabil-ities within the paradigm.

Finally, notice that figure 2 does provide aplace to include direct manipulation as part ofthe collaborative paradigm; that is, one canchoose to implement the application’s GUI

Articles

34 AI MAGAZINE

Figure 1. DiamondHelp for Combination Washer-Dryer.

The Usability Crisis in High-Tech Home Products... technology remains far too hard for most folks to use and mostpeople can only utilize a tiny fraction of the power at their dispos-al. (Business Week, 2004)

Half of all “malfunctioning products” returned to stores by con-sumers are in full working order, but customers can’t figure outhow to operate the device. (Reuters, 2006)

using direct manipulation. This is, in fact,exactly what we have done in the user interfacedesign for DiamondHelp, described in the nextsection.

User Interface DesignOur overarching goal for the DiamondHelpuser interface design was to signal, as stronglyas possible, a break from the conventionalinteraction style to the new collaborative para-digm. A second major challenge was to acco-modate what is inescapably different abouteach particular application of DiamondHelp(the constructs needed to program a thermo-stat are very different from those needed toprogram a washing machine) while preservingas much consistency as possible in the collabo-rative aspects of the interaction. If someoneknows how to use one DiamondHelp applica-tion, he or she should know how to use anyDiamondHelp application.

In order to address these concerns, we divid-ed the screen into two areas, as shown in theexample DiamondHelp interfaces of figures 1,3, and 4. The washer-dryer has been imple-

mented in Java; the other two appliances areFlash simulations. The top half of each of thesescreens is the same distinctive DiamondHelpconversational interface. The bottom half ofeach screen is an application-specific direct-manipulation interface. Dividing the screeninto two areas is, of course, not new; our con-tributions are the specific graphical interfacedesign and the reusable software architecturedescribed later.

Articles

SUMMER 2007 35

GOALS(etc.)

SYSTEMcommunication

observe observe

action action

USER

ApplicationGUI

Figure 2. The Collaborative Paradigm.

J. C. R. Licklider in 1960 on Human-Computer Collaboration

[Compare] instructions ordinarily addressed to intelligent humanbeings with instructions ordinarily used with computers. The lat-ter specify precisely the individual steps to take and the sequencein which to take them. The former present or imply somethingabout incentive or motivation, and they supply a criterion bywhich the human executor of the instructions will know when hehas accomplished his task. In short: instructions directed to com-puters specify courses; instructions directed to human beings spec-ify goals. (Licklider 1960)

Conversational InterfaceTo express and reinforce the human-computercollaboration paradigm, which is based onhuman-human communication, we adoptedthe scrolling speech bubble metaphor oftenused in online chat programs, which supporthuman-human communication. The bubblegraphics nicely reflect the collaborative para-digm’s symmetry between the system and theuser, discussed previously. This graphicalmetaphor also naturally extends to the use of

speech synthesis and recognition technology(see the conclusion).

The top half of every DiamondHelp interfaceis thus a conversation (“chat”) between Dia-mondHelp (the system), represented by theMitsubishi three-diamond logo on the left, andthe user, represented by the human profile onthe right. All communication between the userand system takes place in these bubbles; thereare no extra toolbars, pop-up menus, and soon.

The basic operation of the conversationalpart of the DiamondHelp interface is as fol-lows. Let’s start with the opening of the con-versation, which is the same for all Diamond-Help applications:

After the system says its welcome, a userbubble appears with several options for theuser to choose from. We call this the user’s“things-to-say” bubble. At the opening of theconversation, there are only four choices:“What next?,” “oops,” “done,” and “help.”Notice that the last three of these options arespecially laid out with icons to the right of thevertical line; this is because these three optionsare always present (see Predefined Things toSay).

At this point, the user is free either to replyto the system’s utterance by selecting one ofthe four things to say or to interact with thedirect manipulation part of the interface.Unlike the case with traditional “dialog boxes,”the application GUI is never locked. This flexi-bility is a key aspect of how we have combinedconversational and direct-manipulation inter-faces in DiamondHelp.

In this scenario, the user decides to requesttask guidance from the system by selecting“What next?” in the things-to-say bubble. Aswe have argued elsewhere (see Rich, Sidner,and Lesh [2001]), every interactive systemshould be able to answer a “What (can I do)next?” question.

As part of the interface animation, whenev-er the user makes a things-to-say selection (byclicking or touching the desired word orphrase), the not-selected items are erased andthe enclosing bubble is shrunk to fit only theselected word or phrase. Furthermore, in prepa-

Articles

36 AI MAGAZINE

Figure 3. DiamondHelp for DVD Recorder.

Figure 4. DiamondHelp for Programmable Thermostat.

ration for the next step of the conversation, thecompleted system and user bubbles are “grayedout” and scrolled upward (if necessary) to makeroom for the system reply bubble and the user’snext things-to-say bubble:

DiamondHelp applications always reply to“What next?” by a “You can...” utterance withthe actual options for the possible next taskgoals or actions presented inside the user’sthings-to-say bubble:

In this scenario, from DiamondHelp for aprogrammable thermostat (see figure 4), thedirect-manipulation interface shows the fami-ly’s schedule for the week. The entries in thisschedule determine the thermostat’s tempera-ture settings throughout the day. The userchooses the goal of removing a schedule entry:

The system replies by asking the user to indi-cate, by selection in the direct-manipulationinterface, which schedule entry is to beremoved. Notice that the user’s things-to-saybubble includes only three choices. “Whatnext?” is not included because the system hasjust told the user what to do next:

Although the system is expecting the usernext to click (or touch) in the direct-manipula-tion interface (see figure 5), the user is still freeto use the conversational interface, such as toask for help.

Scrolling HistoryAs in chat windows, a DiamondHelp user canat any time scroll back to view parts of the con-versation that have moved off the screen. Thisis particularly useful for viewing explanatorytext, which can be quite long (for example,suppose the system utterance in figure 3 wasseveral lines longer).

The oval beads on each side of the upper half

of the screen in figures 1, 3, and 4 are standardscroll bar sliders. We also support scrolling bysimply dragging anywhere on the lined back-ground of the upper window.

An interesting DiamondHelp extension toexplore is allowing the user to “restart” theconversation at an earlier point in the historyand move forward again with different choices.This could be a good way to support the back-tracking paradigm for problem solving and isalso closely related to our earlier work on his-tory-based transformations in Collagen; seeRich and Sidner (1998).

Things to SayThe user’s things-to-say bubble is partly anengineering compromise to compensate forthe lack of natural language understanding (seethe conclusion). It also, however, partly servesthe function of suggesting to the user what shecan do at the current point.

In the underlying software architecture, eachchoice in the things-to-say bubble is associatedwith the semantic representation of an utter-ance. When the user selects the desired word orphrase, the architecture treats the choice as ifthe missing natural language-understandingsystem produced the associated semantics.

From a design perspective, this means thatthe wording of the text displayed for eachthings-to-say choice should read naturally asan utterance by the user in the context of the

Articles

SUMMER 2007 37

Figure 5. Selection in the Direct-Manipulation Area.

(See the small triangular cursor in the first column of the schedule.)

ongoing conversation. For example, in figure 1,in reply to the system’s question “How can Ihelp you?,” one of the things-to-say choices is“Explain wash agitation,” not “wash agita-tion.”

At a deeper level, to be true to the spirit ofthe collaborative paradigm, the content of theconversations in DiamondHelp, that is, boththe system and user utterances, should concernnot only primitive actions (“Remove a sched-ule entry”) but also higher-level goals (“Sched-ule a vacation”) and motivation (“Why?”).

When Collagen is used in the implementa-tion of a DiamondHelp application, all systemutterances and the user’s things-to-say choicesare automatically generated from the task mod-el given for the application (see the CollagenCollaboration Plug-in section). Without Colla-gen, the appropriate collaborative content isprovided by the application developer, guidedby the principles elucidated here.

An obvious limitation of the things-to-sayapproach is that there is only room for a rela-tively small number of choices inside the userbubble—six or eight without some kind ofnested scrolling, which we would like to avoid.Furthermore, since we would also like to avoidthe visual clutter of drop-down choices withina single user utterance, each thing-to-say is afixed phrase without variables or parameters.

Given these limitations, our design strategyis to use the direct-manipulation interface toenter variable data. For example, in Diamond-Help for a washer-dryer, instead of the systemasking “How long is the dry cycle?” and gener-ating a things-to-say bubble containing “Thedry cycle is 10 minutes,” “... 15 minutes,” andso on, the system says “Please set the dry cycletime,” points to the appropriate area of thedirect-manipulation interface, and expects theuser to enter the time through the appropriategraphical widget. Figure 5 is a similar exampleof using the direct-manipulation interface toprovide variable data, in this case to select thethermostat schedule entry to be removed.

Use of a textual things-to-say list togetherwith Collagen and speech recognition has beenstudied by Sidner and Forlines (2002) for a per-sonal video recorder interface and by DeKoven(2004) for a programmable thermostat.

Predefined Things to SayMost of the things to say are specific to anapplication. However, a key part of the genericDiamondHelp design is the following set ofuser utterances, which should have the samemeaning in all applications: “What next?,”“Never mind,” “Oops,” “Done,” and “Help.” Inaddition, to save space and enhance appear-

ance, we have adopted a special layout for thelast three of these utterances, which are alwayspresent.

“What next?” has been discussed previously.“Never mind” is a way to end a question with-out answering it (see figure 1). “Oops,” “Done,”and “Help” each initiate a subdialogue inwhich the system tries to determine, respec-tively, how to recover from a mistake, what lev-el of task goal has been completed, or whatform of help the user desires.

When Collagen is used in the implementa-tion of DiamondHelp, the semantics of the pre-defined things to say are automatically imple-mented correctly; otherwise this is theresponsibility of the collaboration plug-inimplementor (see the Software Architecturesection).

Task BarAn additional component of the DiamondHelpconversational interface is the task bar, whichis the single line located immediately above thescrollable history and below the DiamondHelplogo in figures 1 and 3 (the task bar in figure 4happens to be blank at the moment of thescreen shot). For example, in figure 1, the con-tents of the task bar reads:

Make a new cycle > Adjust wash agitation > Help

The task bar, like the scrollable history and thepredefined “What next?” utterance, is a mech-anism aimed towards helping users when theylose track of their context, which is a commonproblem in complex applications. When Colla-gen is used in the implementation of Dia-mondHelp, the task bar is automatically updat-ed with the path to the currently active node inthe task model tree (see figure 8 and the Colla-gen sidebar). Otherwise, it is the responsibilityof the collaboration plug-in implementor toupdate the task bar with appropriate task con-text information.

Currently, the task bar is for display only.However, we have considered extending thedesign to allow users to click (touch) elementsof the displayed path and thereby cause thetask focus to move. This possible extension isclosely related to the scrollable history restartextension discussed previously.

Application GUIThe bottom half of each screen in figures 1, 3,and 4 is an application-specific direct-manipu-lation interface. The details of these GUIs arenot important to the main point of this article.If fact, there is nothing in the DiamondHelpframework that guarantees or relies upon theapplication GUI being well designed or even

Articles

38 AI MAGAZINE

that it follow the direct-manipulation para-digm, though this is recommended. (The onlyway to guarantee this would be for Diamond-Help to automatically generate the applicationGUI from a task model. We have done someresearch on this approach [Eisenstein and Rich2002], but it is still far from practical.)

Furthermore, to achieve our overall goal ofconsistency across applications, an industrial-grade DiamondHelp would, like conventionalUI toolkits, provide standard color palettes,widgets, skins, and so on. However, this isbeyond the scope of a research prototype.

Two design constraints that are relied uponby the DiamondHelp framework are (1) theuser should be able to use the application GUIat any time, and (2) every user action on theapplication GUI is reported to the system (seethe Manipulation Plug-in section).

Finally, it is worth noting that, for the threeapplications presented here, it is possible toperform every function the application sup-ports entirely using the direct-manipulationinterface, that is, totally ignoring the conversa-tional window. While this is a pleasant fact, wealso believe that in more complex applications,there may be some overall advantage in relax-ing this constraint. This is a design issue we arestill exploring.

Software ArchitectureIn contrast to most of the previous discussion,which focuses on the the user’s view of Dia-mondHelp, this section addresses the needs ofthe software developer. The overarching issuein DiamondHelp’s software architecture isreuse, that is, factoring the code so that thedeveloper of a new DiamondHelp applicationhas to write only what is idiosyncratic to thatapplication while reusing as much generic Dia-mondHelp framework code as possible.

Figure 6 shows our solution to this chal-lenge, which we have implemented in Jav-aBeans and Swing. (Dotted lines indicateoptional, but recommended, components.) Allthe application-specific code is contained intwo “plug-ins,” which are closely related to thetwo halves of the user interface. The rest of thecode (shaded region) is generic DiamondHelpframework code.

In addition to the issues discussed in this sec-tion, there are a number of other standardfunctions of plug-and-play architectures (ofwhich DiamondHelp is an instance), such asdiscovering new devices connected to a net-work, loading the appropriate plug-ins, and soon, which are beyond the scope of this article.

Manipulation Plug-inNotice that the manipulation plug-in “sticksout” of the DiamondHelp framework box infigure 6. This is because it is directly responsi-ble for managing the application-specific por-tion of the DiamondHelp interface. Diamond-Help simply gives this plug-in a Swingcontainer object corresponding to the bottomhalf of the screen.

We recommend that the manipulation plug-in provide a direct-manipulation style of inter-face implemented using the model-view archi-tecture, as shown by the dotted lines in figure6. In this architecture, all of the “semantic”state of the interface is stored in the model sub-component; the view subcomponent handlesthe application GUI.

Regardless of how the manipulation plug-inis implemented internally, it must provide anapplication programming interface (API) withthe DiamondHelp framework that includestwo event types: outgoing events (user actionobservation), which report state changes result-ing from user GUI actions, and incomingevents (system action), which specify desiredstate changes. Furthermore, in order to pre-serve the symmetry of the collaborative para-digm (see figure 2), it is the responsibility of theplug-in to update the GUI in response toincoming events so that the user may observesystem actions. As an optional but recom-mended feature, the manipulation plug-in mayalso provide the DiamondHelp framework withthe graphical location of each incoming statechange event, so that DiamondHelp can movea cursor or pointer to that location to help theuser observe system actions.

Note that the content of both incoming andoutgoing state change events is specified insemantic terms, for example, “change dry cycletime to 20 min,” not “button click at pixel100,200.” Lieberman (1998) further discussesthis and other issues related to interfacingbetween application GUIs and intelligent sys-tems.

Collaboration Plug-inBasically, the responsibility of the collabora-tion plug-in is to generate the system’s respons-es (actions and utterances) to the user’s actionsand utterances. Among other things, this plug-in is therefore responsible for implementingthe semantics of the predefined utterances (seenext section).

The collaboration plug-in has two inputs:observations of user actions (received from themanipulation plug-in), and user utterances(resulting from user choices in the things-to-say bubble). It also has four outputs: system

Articles

SUMMER 2007 39

actions (sent to the manipulation plug-in), sys-tem utterances (which go into system bubbles),things to say (which go into user bubbles), andthe task bar contents.

Notice that the collaboration plug-in isresponsible for providing only the content ofthe system and user bubbles and the task bar.All of the graphical aspects of the conversa-tional window are managed by DiamondHelpframework code. It is also an important featureof the DiamondHelp architecture that the col-laboration plug-in developer does not need tobe concerned with the graphical details of theapplication interface. The collaboration plug-in developer and the manipulation plug-indeveloper need only to agree on a semanticmodel of the application state.

For a very simple application, the collabora-tion plug-in may be implemented by a statemachine or other ad hoc mechanisms. Howev-er, in general, we expect to use the Collagenversion described in the next section.

Collagen Collaboration Plug-inThe Collagen version of the collaboration plug-in includes an instance of Collagen, with a lit-tle bit of wrapping code to match the collabo-ration plug-in API. Collagen has already beendescribed in an earlier article (see Collagensidebar). We will only highlight certain aspectshere which relate to DiamondHelp.

The most important reason to use the Colla-

gen collaboration plug-in is that the applica-tion developer needs to provide only onething: a task model. All four outputs of the col-laboration plug-in described previously arethen automatically generated by Collagen asshown in figure 7.

System utterances and actions are producedby Collagen’s usual response generation mech-anisms (Rich et al. 2002). The semantics ofsome of the predefined DiamondHelp userutterances, such as “Oops,” “Done,” and“Help,” are captured in a generic task model,which is used by Collagen in addition to theapplication-specific task model provided. Eachof these predefined utterances introduces ageneric subgoal with an associated subdialogue(using Collagen’s discourse stack). Other pre-defined utterances, such as “What next?” and“Never mind,” are built into Collagen.

The third collaborative plug-in output pro-duced by Collagen is the user’s things to say.Basically, the things to say are the result of fil-tering the output of Collagen’s existing dis-course (agenda) generation algorithm. The firststep of filtering is remove everything exceptexpected user utterances. Then, if there are stilltoo many choices, a set of application-indepen-dent heuristics are applied based on the type ofcommunicative act (proposal, question, and soon). Lastly, predefined utterances are added asappropriate. The further details of this algo-rithm need to be the topic of a separate article.

Articles

40 AI MAGAZINE

Model

View

ManipulationPlug-in

user action

userutterance

observation

Collagen

Task Model

utterancesystem

application GUI update

user bubble

system bubble

thingsto say

task bar

user manipulation

system action

userchoice

CollaborationPlug-in

DiamondHelp

Figure 6. DiamondHelp Software Architecture.

Mixed InitiativeIn this section, we review DiamondHelp withrespect to the shared themes of this specialissue of AI Magazine on mixed-initiative assis-tants.

To begin, let us follow up on our earlier com-ment that mixed initiative and collaborationare virtually coextensive concepts. Horvitz pro-vides as good a definition as any of mixed ini-tiative:

... an efficient, natural interleaving of contribu-tions by users and automated services aimed atconverging on solutions to problems. (Horvitz1999)

Compare this with the definition of collabo-

ration upon which Collagen (and thereforeDiamondHelp) is based:

... a process in which two or more participantscoordinate their actions toward achievingshared goals. (Rich, Sidner, and Lesh 2001)

If the two participants in a collaboration arethe user and an automated system, then such acollaboration is generally mixed initiative;only in those rare or simple circumstanceswhere one participant provides no contribu-tion to any goal (including clarification, suc-cess/failure report, and so on) is mixed initativelacking. Conversely, the only part of the defi-nition of collaboration that is missing from thedefinition of mixed initiative is the explicitmention of “shared goals” (and it is perhaps

Articles

SUMMER 2007 41

Collagen (for collaborative agent) is Java middleware forbuilding collaborative interface agents based on Groszand Sidner’s SharedPlan theory of collaborative discourse(Grosz and Sidner 1986, 1990; Grosz and Kraus 1996;Lochbaum 1998). Collagen has been used to build morethan a dozen prototypes, including an air travel planningassistant (Rich and Sidner 1998); e-mail assistant (Gruenet al 1999); VCR programming assistant (Sidner and For-lines 2002); power system operation assistant (Rickel etal. 2001); gas turbine engine operation tutor (Davies et al.2001); flight path planning assistant (Cheikes and Gert-ner 2001); recycling resource allocation assistant; soft-ware design tool assistant; programmable thermostathelper (DeKoven, Keyson, and Freudenthal 2001);mixed-initiative multimodal form filling, and robot host-ing system (Sidner et al. 2005).

The two key data structures in Collagen’s architecture(see figure 7) are the task model and the discourse state.The two key algorithms are discourse interpretation andgeneration.

Task ModelA task model is an abstract, hierarchical, partially orderedrepresentation of the actions typically performed toachieve goals in the application domain. Figure 8 showsa fragment of the task model for the programmable ther-mostat application in figure 4. Although this exampledoes not, task models in general also contain alternativedecomposition choices. Collagen currently provides aJava extension language for defining task models, but wewill adopt the new CEA standard (see the conclusion) assoon as it is completed.

Discourse StateThe discourse state tracks the beliefs and intentions ofthe participants in the current collaboration and pro-vides a focus of attention mechanism for tracking shiftsin the task and conversational context. Collagen’s dis-

course state is implemented as a goal decomposition(plan) tree plus a stack of focus spaces (each of whichincludes a focus goal). The top goal on the focus stack isthe “current purpose” of the discourse.

Discourse InterpretationThe basic job of discourse interpretation is to update thediscourse state incrementally by explaining how eachoccurring event (action or utterance by the user or thesystem) contributes to the current purpose. An eventcontributes to a purpose if it either (1) directly achievesthe purpose, or (2) is a step in the task model for achiev-ing the purpose, or (3) selects one of alternative decom-position choices for achieving the purpose, or (4) identi-fies which participant should achieve the purpose, or (5)identifies a parameter of the purpose.

If the current event contributes, either directly or indi-rectly, to the current purpose, then the plan tree is appro-priately extended to include the event; the focus stackmay also be pushed or popped. Our discourse-interpreta-tion algorithm extends Lochbaum’s (1998) original for-mulation by incorporating plan recognition (Lesh, Rich,and Sidner 1999, 2001) and handling interruptions.

Discourse Generation.The discourse-generation algorithm in Collagen is essen-tially the inverse of discourse interpretation. Based onthe current discourse state, it produces a prioritized list,called the agenda (see figure 7), of partially or totallyspecified utterances and actions that would contribute tothe current discourse purpose according to the five casesitemized previously.

In general, an agent may use any application-specificlogic it wants to decide on its next action or utterance. Inmost cases, however, an agent can simply choose the firstitem on the agenda generated by Collagen.

*This sidebar is a summary of Rich, Sidner, and Lesh (2001).

Collagen: Applying Collaborative Discourse Theory to Human-Computer Interaction

implicit in “converging on solutions”). Onecould imagine some sort of interaction withinterleaved contributions without sharedgoals, but it is hard to believe it would be veryefficient or natural.

Division of Labor and ControlIn general in a collaboration, decisions aboutwho does what and when are matters for nego-tiation between the participants. In many con-crete settings, however, many of these deci-sions are conventionalized by the context.Consider, for example, the collaborationbetween the customer and clerk at a retailcounter. Similarly, DiamondHelp has beendesigned with a certain expected, thoughadjustable (see the Personalization subsectionthat follows), division of labor and control.

Basically, DiamondHelp assumes that the

user knows what she wants to do at a high lev-el but needs help carrying out the necessarydetails. Furthermore, DiamondHelp normallyreturns control to the user as quickly as possible(by putting “What next?” in the user bubble).

ArchitectureThe key to DiamondHelp’s power and flexibil-ity as a mixed-initiative system is the highdegree of symmetry between the user and thesystem in all aspects of the architecture, fromthe user interface, to discourse interpretation,to the task model. Many interactive systems,even mixed-initiative ones, have built-in asym-metries that limit their flexibility.

The symmetry of the chat-based user inter-face has already been discussed previously. Infigure 7, notice that the discourse-interpreta-tion process is applied to both user and system

Articles

42 AI MAGAZINE

ChooseDiscourse State

DiscourseGeneration

Interpretation

user utterance

user action

system action

system utterance

observation

Task Model

Discourse

things to say

task bar

Agenda

Figure 7. Collagen Architecture.

(Arrows on the left of this figure correspond to arrows with the same labels in figure 6).

utterances and actions. This is a reflection ofthe fact that the collaborative discourse staterepresents the mutual beliefs of the partici-pants (see Sidner [1994] for more details).

Finally, and most fundamentally, the taskmodel that drives a DiamondHelp interactiongenerally specifies only what needs to be done,not who should do it. (The only exception iswhen one participant is fundamentally inca-pable of performing some action.) For example,consider the two steps to achieve the schedulevacation goal in figure 8. This task modelaccounts for an interaction in which the userselects both the first day and the last daythrough the direct-manipulation area, and alsoan interaction in which the system selects bothdays (perhaps based on other knowledge), orinteractions in which one day is selected by theuser and one by the system. Which interactionactually occurs depends on the actions andknowledge of the user and the system in theparticular situation.

Communication and Shared AwarenessAs discussed earlier, DiamondHelp is based ona model of co-present collaboration involvinga shared artifact/application (see figure 2). Ouruser interface design attempts to give the feel-ing of natural communication without actual-ly requiring natural language or speech pro-cessing. For shared awareness of theapplication state, DiamondHelp relies totallyon the graphical user interface providedthrough the manipulation plug-in. Diamond-Help itself provides two mechanisms to sup-port shared awareness of the task/conversationstate: the task bar and the scrollable chat win-dow. Collagen also provides a much more com-

plete representation of task/conversation state,called the segmented interaction history (seeRich, Sidner, and Lesh [2001]), which is notcurrently incorporated into the design of Dia-mondHelp.

PersonalizationIn addition to the significant personalizationand adaptation inherent in all collaborativeinteractions, there are at least two specific per-sonalization mechanisms in DiamondHelpworth mentioning here. Both of these mecha-nisms take advantage of facilities in Collagen,such as the student model, developed for intel-ligent tutoring (Rickel et al. 2002), which is notsurprising, since personalization is a key aspectof intelligent tutoring.

First, as discussed previously, DiamondHelpnormally returns control to the user as quicklyas possible. However, based on simple observa-tions of the user’s behavior, such as timing anderrors, DiamondHelp can switch into a modewhere it takes control and guides the userthrough an entire task or subtask, without theuser having to ask “What next?” at each step.

A second personalization has to do withwhether the system asks the user to performcertain manipulations on the application GUIor simply performs them itself. For example, inthe washer-dryer application, the system caneither say “Please click here to pop up the tem-perature dial” or simply pop up the tempera-ture dial window at the appropriate time itself.The former case has the benefit that the userlearns where to click in the future; the lattercase has the benefit of being faster. Diamond-Help can switch between these modes on a peraction basis, depending on whether the userhas already performed the action once or twiceherself.

Articles

SUMMER 2007 43

select last day

add entry remove entry modify entry schedule vacation

program thermostat

select first day

Figure 8. Task Model for Programmable Thermostat.

EvaluationFigure 9 shows the three conditions in a userstudy we are currently planning to evaluateDiamondHelp using the washer-dryer exam-ple. The printed manual in condition B willcontain the same information (literally thesame text) that is communicated dynamicallyby DiamondHelp in condition C. Users in eachcondition will be assigned a set of tasks thatwill require them to use the advanced pro-grammability features of the washer-dryer. Weplan to obtain both objective measures, such astime and quality of task completion, and sub-jective evaluations of experience.

Related WorkThe most closely related work to Diamond-Help, in terms of both application andapproach, is the Roadie system (Lieberman andEspinosa 2006), which is also directed towardsconsumer electronics and shares the task-mod-el approach. Roadie differs from DiamondHelpprimarily in not using a conversational or col-laborative model for its interface. Roadie alsoincorporates natural language understandingand commensense knowledge components forguessing the user’s desires,

Also closely related is the Writer’s Aid system(Babaian, Grosz, and Shieber 2002), whichapplies the same collaborative theory uponwhich DiamondHelp is based to a single specif-ic task, rather than developing a general tool.

Of the articles in this special issue of AI Mag-azine, the most closely related is Ferguson andAllen (2007). The collaborative model andimplementation architecture used in that workare very similar to those described here. The

scope of their work, however, is broader thanours, since they also build natural language-understanding components. Furthermore,their architecture includes a beliefs, desires,and intentions (BDI) component that is pres-ent only in vestigial form in DiamondHelp/Collagen.

BDI architecture is at the heart of the mixed-initiative personal assistant of Myers et al.(2007). They describe the interaction paradigmused in their work as “delegative,” and contrastit with the collaborative paradigm. We wouldview delegation as being included within therange of collaboration; see Grosz and Kraus(1996).

Another category of related work is genericsoftware architectures for connecting arbitraryapplications with help facilities. The best-known example of this category is Microsoft’sClippy. Clippy’s interaction paradigm might bedescribed as “watching over the user’s shoulderand jumping in when you have some advice”and is, in our opinion, the main reason forClippy’s well-known unpopularity amongusers. In contrast, the collaborative paradigmunderlying DiamondHelp emphasizes ongoingcommunication between the system and userto maintain shared context.

Also in this category are so-called wizards,such as software installation wizards, and soon, which come in many different forms frommany different manufacturers. Our feelingabout wizards is that wizards embody the rightparadigm, that is, interactively guiding theuser, but in too rigid a form. DiamondHelpsubsumes the capabilities of wizards, but alsoallows users to take more initiative when theywant to.

Articles

44 AI MAGAZINE

(C) DiamondHelp(A) No Guidance (B) Printed Manual

Figure 9. Three Conditions in Planned User Study.

ConclusionThe contributions of this work are twofold.First, we have explicated a novel user interfacedesign that expresses and reinforces thehuman-computer collaboration paradigm bycombining conversational and direct-manipu-lation interfaces. Second, and more concretely,we have produced the DiamondHelp software,which can be used by others to easily constructsuch interfaces for new applications.

A number of small extensions to Diamond-Help have already been discussed in the bodyof this article. Another general set of exten-sions have to do with adding speech and natu-ral language-understanding technology.

Adding text-to-speech generation to the sys-tem bubble is a very easy extension, which wehave already done (it’s an optional feature ofthe software). Furthermore, we have found thatusing a system with synthesized speech is sur-prisingly more pleasant than without, evenwhen user input is still by touch or click.

Adding speech recognition can be done intwo ways. The first, more limited way, is to usespeech recognition to choose one of the dis-played things to say. We have already done this(another optional feature) and have found thatthe speech recognition is in general quite reli-able, because of the small number of very dif-ferent utterances being recognized. Further-more, because of the way that the things-to-saymechanism is implemented, this extensionrequires no changes in the collaboration ormanipulation plug-ins for an application.

A second, and much more ambitiousapproach, is to dispense with the things-to-saybubble and try to recognize anything the usersays, which, of course, requires broad underly-ing speech and natural language-understand-ing capabilities. If one has broad natural lan-guage understanding, another variation is tosupport unrestricted typed input from the userinstead of speech.

Alternatively, it would be trivial (at leastfrom the user interface point of view) to revertthe conversational interface to its originalfunction as a chat between two humans, theuser and a remote person. This could be useful,for example, for remote instruction or trou-bleshooting. The system could even automati-cally shift between normal and “live chat”mode.

Finally, returning to the original motivationof this work, namely the usability crisis inhigh-tech home products, the Consumer Elec-tronics Association (www.ce.org) has recentlycreated a new Task-Based User Interface work-ing group, which will develop a standard (CEA-2018) for representing task models. This stan-

dard has the potential of helping systems likeDiamondHelp to have a significant impact onthe usability problem.

AcknowledgementsThe work described here took place at Mit-subishi Electric Research Laboratories betweenapproximately January 2004 and June 2006.The basic DiamondHelp concept was devel-oped in collaboration with Neal Lesh andAndrew Garland. Shane Booth provided graph-ic design input. Markus Chimani implementedkey parts of the graphical interface in Java.

ReferencesBabaian, T.; Grosz, B.; and Shieber, M. 2002. AWriter’s Collaborative Aid. In Proceedings of the Inter-national Conference on Intelligent User Interfaces, 7–14.New York: Association for Computing Machinery.

Cheikes, B., and Gertner, A. 2001. Teaching to Planand Planning to Teach in an Embedded Training Sys-tem. Paper presented at the Tenth International Con-ference on Artificial Intelligence in Education, 19–23May, San Antonio, Texas.

Davies, J.; Lesh, N.; Rich, C.; Sidner, C.; Gertner, A.;and Rickel, J. 2001. Incorporating Tutorial Strategiesinto an Intelligent Assistant. In Proceedings of theInternational Conference on Intelligent User Interfaces,53–56. New York: Association for ComputingMachinery.

DeKoven, E. 2004. Help Me Help You: Designing Sup-port for Person-Product Collaboration. Ph.D. Disser-tation, Delft Institute of Technology, Delft, theNetherlands.

DeKoven, E.; Keyson, D.; and Freudenthal, A. 2001.Designing Collaboration in Consumer Products. InProceedings of the ACM Conference on Computer HumanInteraction, Extended Abstracts, 195–196. New York:Association for Computing Machinery.

Eisenstein, J., and Rich, C. 2002. Agents and GUI’sfrom Task Models. In Proceedings of the InternationalConference on Intelligent User Interfaces, 47–54. NewYork: Association for Computing Machinery.

Ferguson, G., and Allen, J. 2007. Mixed-Initiative Sys-tems for Collaborative Problem Solving. AI Magazine28(2).

Grosz, B. J., and Kraus, S. 1996. Collaborative Plansfor Complex Group Action. Artificial Intelligence86(2): 269–357.

Grosz, B. J., and Sidner, C. L. 1986. Attention, Inten-tions, and the Structure of Discourse. ComputationalLinguistics 12(3): 175–204.

Grosz, B. J., and Sidner, C. L. 1990. Plans for Dis-course. In Intentions and Communication, ed. P.Cohen, J. Morgan, and M. Pollack, 417–444. Cam-bridge, MA: MIT Press.

Gruen, D.; Sidner, C.; Boettner, C.; and Rich, C. 1999.A Collaborative Assistant for E-Mail. In Proceedings ofthe ACM SIGCHI Conference on Human Factors in Com-puting Systems, Extended Abstracts, 196–197. NewYork: Association of Computing Machinery.

Horvitz, E. 1999. Uncertainty, Action and Interac-

Articles

SUMMER 2007 45

tion: In Pursuit of Mixed-Initiative Computing. IEEEInteligent Systems 14(5): 17–20.

Lesh, N.; Marks, J.; Rich, C.; and Sidner, C. 2004.“Man-Computer Symbiosis” Revisited: AchievingNatural Communication and Collaboration withComputers. IEICE Transactions Information and Sys-tems E87–D(6): 1290–1298.

Lesh, N.; Rich, C.; and Sidner, C. 1999. Using PlanRecognition in Human-Computer Collaboration. InProceedings of the Seventh International Conference onUser Modeling, 23–32. New York: Springer-Verlag.

Lesh, N.; Rich, C.; and Sidner, C. 2001. Collaboratingwith Focused and Unfocused Users under ImperfectCommunication. In Proceedings of the Ninth Interna-tional Conference on User Modeling, 64–73. New York:Springer-Verlag.

Licklider, J. C. R. 1960. Man-Computer Symbiosis.IRE Transactions on Human Factors in Electronics HFE-1: 4–11.

Lieberman, H. 1998. Integrating User InterfaceAgents with Conventional Applications. In Proceed-ings of the International Conference on Intelligent UserInterfaces, 39–46. New York: Association for Comput-ing Machinery.

Lieberman, H., and Espinosa, J. 2006. A Goal-Orient-ed Interface to Consumer Electronics using Planningand Commonsense Reasoning. In Proceedings of theInternational Conference on Intelligent User Interfaces,226–233. New York: Association for ComputingMachinery.

Lochbaum, K. E. 1998. A Collaborative PlanningModel of Intentional Structure. Computational Lin-guistics 24(4): 525–572.

Myers, K.; Berry, P.; Blyth, J.; Conley, K.; Gervasio, M.;McGuinness, D.; Morley, D.; Pfeffer, A.; Pollack, M.;and Tambe, M. 2007. An Intelligent Personal Assis-tant for Task and Time Management. AI Magazine28(2).

Rich, C., and Sidner, C. 1998. COLLAGEN: A Collab-oration Manager for Software Interface Agents. UserModeling and User-Adapted Interaction 8(3–4): 315–350.

Rich, C.; Lesh, N.; Rickel, J.; and Garland, A. 2002. APlug-In Architecture for Generating CollaborativeAgent Responses. In Proceedings of the First Interna-tional Joint Conference on Autonomous Agents and Mul-tiagent Systems. New York: Association for Comput-ing Machinery.

Rich, C.; Sidner, C.; and Lesh, N. 2001. Collagen:Applying Collaborative Discourse Theory to Human-Computer Interaction. AI Magazine 22(4): 15–25.

Rickel, J.; Lesh, N.; Rich, C.; Sidner, C.; and Gertner,A. 2001. Using a Model of Collaborative Dialogue toTeach Procedural Tasks. Paper presented at the Work-shop on Tutorial Dialogue Systems, Tenth Interna-tional Conference on Artificial Intelligence in Educa-tion, 19–23 May, San Antonio, Texas.

Rickel, J.; Lesh, N.; Rich, C.; Sidner, C.; and Gertner,A. 2002. Collaborative Discourse Theory as a Foun-dation for Tutorial Dialogue. In Proceedings of the 6thInternational Conference on Intelligent Tutoring Systems,542–551. Berlin: Springer.

Shneiderman, B., and Plaisant, C. 2005. Designing the

User Interface: Strategies for Effective Human-ComputerInteraction. Reading, MA: Addison-Wesley.

Sidner, C. L. 1994. An Artificial Discourse Languagefor Collaborative Negotiation. In Proceedings of theTwelfth National Conference on Artificial Intelligence,814–819. Menlo Park, CA: AAAI Press.

Sidner, C. L.; Lee, C.; Kidd, C.; Lesh, N.; and Rich, C.2005. Explorations in Engagement for Humans andRobots. Artificial Intelligence 166 (1–2): 104–164.

Sidner, C. L., and Forlines, C. 2002. Subset Languagesfor Conversing with Collaborative Interface Agents.Paper presented at the International Conference onSpoken Language Processing, Denver, CO., 16–20September.

Charles Rich is a professor ofcomputer science at WorcesterPolytechnic Institute. He was pre-viously a distinguished researchscientist at Mitsubishi ElectricResearch Laboratories. He earnedhis Ph.D. at the MIT ArtificialIntelligence Laboratory, where hewas a founder and director of the

Programmer’s Apprentice project. Rich is a fellow andpast councilor of AAAI, as well as having served aschair of the 1992 International Conference on Prin-ciples of Knowledge Representation and Reasoning,cochair of the 1998 National Conference on ArtificialIntelligence, and program cochair of the 2004 Inter-national Conference on Intelligent User Interfaces.His e-mail address is rich@cs.wpi.edu.

Candace L. Sidner is a divisionscientist at BAE Systems AdvancedInformation Technologies. Shewas previously a senior researchscientist at Mitsubishi ElectricResearch Laboratories. She earnedher Ph.D. at the MIT ArtificialIntelligence Laboratory. Sidner is afellow and past councilor of AAAI,

as well as having served as president of the Associa-tion for Computational Linguistics, chair of the 2001and program cochair of the 2006 International Con-ference on Intelligent User Interfaces, and cochair ofthe 2004 SIGdial Workshop on Discourse and Dia-logue, and general chair of the 2007 NAACL HumanLanguage Technology conference. Her e-mail addressis candy.sidner@baesystems.com.

Articles

46 AI MAGAZINE