slot: a research platform for investigating multimodal

Copyright 2003 Psychonomic Society Inc 408

Behavior Research Methods Instruments amp Computers2003 35 (3) 408-419

When two humans communicate with each other in aface-to-face situation they do not only exchange speechOther so-called channels or modalities such as gaze (egeye contact) facial expression intonation voice qualityand gestures (eg pointing) also play an important role inboth the semantic and the socio-emotional aspects of com-munication (see eg Ellsworth amp Ludwig 1972) Someauthors have even claimed that nonverbal communicationcontributes more to the interpretation of communicativeacts than does verbal information (Archer amp Akert 1977Mehrabian amp Wiener 1967) Archer and Akert (1977)used recorded stimuli in which both the nonverbal infor-mation (voice quality) and the verbal information (seman-tic content) were systematically varied in emotional qual-ity (hostile neutral or friendly) In judging the emotionalquality of the stimuli in which the nonverbal and verbalqualities did not match participants indeed attributed

more importance to the nonverbal signal Brown (1986) ar-gued that when judges of emotional quality are presentedwith incongruous information the nonverbal aspects ofthe communication are usually assumed to be under lessconscious control by the speaker than the verbal content isand therefore the nonverbal information is perceived tobe a more reliable source than the verbal content (p 498)

Some authors have put forth the weaker claim that non-verbal communication is more suited for expressing feel-ings and affective information whereas the content ofspeech is more often employed to refer to speaker-externalevents (Argyle Salter Nicolson Williams amp Burgess1970) Later however Krauss Apple Morency Wenzeland Winton (1981) using naturalistic stimulus materialdemonstrated that in judgments of the emotional quality ofan exchange judgments of the verbal content (presented inthe form of transcripts) were a much more reliable sourceof information than those of video-only or content-filteredspeech (presenting voice quality only) when each wascompared with judgments of the full video and audio datawhich served as a baseline

Independent of the contribution of nonverbal commu-nication to the overall interpretation of communicativeacts there is no doubt that for the transmission of certaintypes of abstract information (eg conditionals counter-

This work was supported by Grant IST-2001-32311 from the Euro-pean Union Project Conversational Multimodal Interaction with Com-puters (COMIC) The authors thank Jolanda Bource Paula Dings andHanneke Ribberink for their meticulous transcription work Correspon-dence concerning this article should be addressed to J P de Ruiter MaxPlanck Institute for Psycholinguistics P O Box 310 NL-6500 AH Nij-megen The Netherlands (e-mail janpeterderuitermpinl)

SLOTA research platform

for investigating multimodal communication

JAN PETER DE RUITERMax Planck Institute for Psycholinguistics Nijmegen The Netherlands

STEacutePHANE ROSSIGNOL and LOUIS VUURPIJLCatholic University of NijmegenNICI Nijmegen The Netherlands

DOUGLAS W CUNNINGHAMMax Planck Institute for Biological Cybernetics Tuumlbingen Germany

and

WILLEM JM LEVELTMax Planck Institute for Psycholinguistics Nijmegen The Netherlands

In this article we present the spatial logistics task (SLOT) platform for investigating multimodal com-munication between 2 human participants Presented are the SLOT communication task and the soft-ware and hardware that has been developed to run SLOT experiments and record the participantsrsquo mul-timodal behavior SLOT offers a high level of flexibility in varying the context of the communicationand is particularly useful in studies of the relationship between pen gestures and speech We illustrate theuse of the SLOT platform by discussing the results of some early experiments The first is an experi-ment on negotiation with a one-way mirror between the participants and the second is an exploratorystudy of automatic recognition of spontaneous pen gestures The results of these studies demonstratethe usefulness of the SLOT platform for conducting multimodal communication research in bothhumanndash human and humanndashcomputer interactions

INVESTIGATING MULTIMODAL COMMUNICATION 409

factuals and tense information) linguistic communicationhas a clear advantage over nonverbal communication

The different channels through which people communi-cate often complement each other For instance a spokenstatement accompanied by raised eyebrows might be in-terpreted as a question by the interlocutor A spoken state-ment containing the phrase ldquoover thererdquo is usually accom-panied by a pointing gesture to indicate where ldquothererdquo isIn sum face-to-face interaction is a form of multimodalcommunication

The word multimodalitysuggests that the concept refersto more than one perceptual modality (eg auditory per-ception and vision) However this interpretation of theconcept is too narrow to capture the phenomena in whichmultimodal communication researchers are generally in-terested Although both hand gestures and facial expres-sions are perceived in the visual modality they clearly rep-resent distinct aspects of communication The same holdsfor voice quality and content of speech both of which relyon the auditory modality To remedy this definitionalproblem we propose the notion of a semiotic channel Asemiotic channel is a set of identifiable behavioral unitsthat (1) cannot be performed simultaneously with eachother and (2) can be performed simultaneously with (al-most) all behavioral elements in other semiotic channelsFor example facial expression and voice quality consti-tute two different semiotic channels for it is not possibleto have two different facial expressions or two differentvoice qualities at the same time whereas all voice quali-ties can in principle be combined with all facial expres-sions The proposed definition is admittedly not entirelyairtight Although we believe eye gaze and facial expres-sion to be separate semiotic channels certain facial ex-pressions such as the ldquothinkingrdquo expression (Cunning-ham Breidt Kleiner Wallraven amp Buumllthoff 2003) oftencontain specific eye-gaze patterns as integral parts How-ever given that eye gaze and facial expression can in mostcases operate independently of each other we neverthe-less consider them to be separate semiotic channels Dueto the widespread use in the literature of the term multi-modal communication we will continue to use this termto refer to the general field of study but what we mean byit is communication involving the simultaneous use ofmore than one semiotic channel We will reserve the wordmodality to refer to the perceptual modality (eg visualauditory) and will use the word channel to refer to semi-otic channels as defined above

In order to gain systematic knowledge about how mul-timodal communication between human participantsworks the collection of behavioral data is essential Astraightforward way to obtain data is to record interactionsbetween people and analyze them carefully (see egGoodwin 1981 Kendon 1972) A disadvantage of thisnaturalistic approach is that one has very little control overthe content of the communication which makes it ratherhard to predict what participants are going to communi-cate about and how they are going to do it This in turnleads to an unacceptable level of variance in both the con-

tent of the communication and the use of the differentmodalities in which the humanndashhuman interaction re-searcher is interested

A higher level of control over the content of the com-munication is achieved by Clark and Krych (in press) Intheir task one participant had to instruct another in build-ing a prespecified Lego model An important advantageof this task is that it provides an objective measure of boththe efficiency and the quality of the communication (asdemonstrated by the duration of the interaction and thenumber of errors in the resulting Lego model respec-tively) Although these tasks yield results that are impor-tant for the field of multimodal communication they arelimited to the study of collaborative communicationAlso the two participants have different predefined rolesOne is ldquodirectorrdquo and the other is ldquobuilderrdquo

Another interesting multimodal task is the map task(Anderson et al 1991) In this task the 2 participants eachhave a map in front of them and one of them (the routegiver) instructs the other (the route follower) to follow acertain route through the map The maps used by route giverand route follower are not identical which makes the taskmore difficult and therefore potentially more interestingAs with the directorbuilder task of Clark and Krych (inpress) this task assigns different roles to the 2 participants

An important reason for us to develop a new researchplatform instead of using the map task or the directorbuilder task is that we wanted to use a communication taskthat is highly structured but does not assign different rolesto the 2 participants Also since an important goal of thehumanndashhuman data that we want to generate is to provideinformation that could produce guidelines for (natural-looking) humanndashcomputer interaction (HCI) we wantedour task to involve the use of both electronic pen data andspeechmdashmodalities often used in multimodal HCI (Ovi-att 1999) Finally we wanted to be able to have control overthe competitiveness of the interaction to be able to dis-tinguish between multimodal communication at differentlevels of competitiveness of the task

THE COMMUNICATIVE TASK

CriteriaAn earlier version of the communicative task in the spa-

tial logistics task (SLOT) was originally developed (seeLevelt 2001) with the following criteria in mind

1 The experimenter should have a high degree of con-trol over the communicative goals of the participants dur-ing the experiment This should be the case for general(overall) goals as well as for possible subgoals at any timeduring the course of the experiment

2 It should be possible to assess objectively the qualityof the end result of the communication

3 Although any task involving naturalistic uncon-strained communication will result in data with high inter-and intrasubject variability this variability should be keptas low as possible to facilitate analyses with sufficient sta-tistical power

410 DE RUITER ROSSIGNOL VUURPIJL CUNNINGHAM AND LEVELT

4 In the task it should be natural for the participants touse all the channels that are available to them especiallythe electronic pen

5 The task should allow for the manipulation of thepresence or absence of certain channels (with the excep-tion of speech which will always be available) withoutchanging the essence of the communication task or mak-ing the task impossible to fulfill

6 The task should be the same for both participants7 The level of competitiveness of the task should be

variable

Task DefinitionWith these criteria in mind the SLOT communication

task has been defined as a route negotiation task The gen-eral idea is as follows Two participants are shown a map(formally corresponding to a mathematical graph) con-sisting of cities (vertices) that are connected to each otherby roads (arcs) Some of the cities are marked as targetsThe participants have to negotiate a route that begins andends at a specified start city and passes through all of thespecified targets while minimizing personal andor globaltravel costs The subjects are given the opportunity to usean electronic pen to draw on the presented maps (imple-menting a shared whiteboard metaphor) to facilitate thenegotiation process

There are two variants of this taskmdasha cooperative vari-ant and a competitive variantmdashwhich will be describedseparately below The two modes of SLOT enable thestudy of potential differences in communicational behav-ior in cooperative versus competitive social contexts

Cooperative ModeIn the cooperative mode all the specified targets are

presented in the same color (purple we have used gray inFigure 1) See Figure 1 for an example of a cooperative

SLOT map The participants have to negotiate a route thatpasses through all of the targets while keeping the travelcosts at a minimum (see below) The participants have tofigure out collectively which route will result in the low-est cost This is a variant of the well-known traveling sales-man problem (TSP see Burkard 1979 for an overview)the difference from the canonical version of the TSP beingthat the travel costs for a certain step are not independentof the route that has been traveled up to that point After aroute has been negotiated one of the participants is askedto submit it to the computer using the electronic pen tohave it checked for legality and to compute and store theresulting costs which are also displayed on the tablets

Competitive ModeIn the competitive mode there are different targets for

each of the participants marked by the use of a differentcolor for each participant (red and blue light gray anddark gray in Figure 2) See Figure 2 for an example of a com-petitive SLOT map In addition to having his or her own per-sonal targets each participant also has his or her own costaccounting and these personal costs are a function of thenumber of personal targets that have been reached at anypoint during the route Note that by different placementsof the red and blue target nodes the degree of competi-tiveness can be varied For maps on which the optimalroute for the red targets is identical to that for the blueones a situation approximately equal to that of the coop-erative mode is realized The larger the difference betweenthe respective optimal routes the higher the degree ofcompetition

The Cost RuleThe general cost rule in SLOT is that every step from

node to node over a single arc has a cost of 1 1 the num-ber of designated targets that still must be reached As

Figure 1 A cooperative SLOT map


soon as a target has been reached the cost per step dropsby 1 After all the targets have been reached the cost ofany subsequent step (ie returning to the start node) isequal to 1 In the cooperative mode this holds for any tar-get that has been reached whereas in the competitivemode it holds only for those targets assigned to the par-ticipant whose score is being calculated For example inthe competitive map presented in Figure 2 a first stepfrom the start node (7) to Node 8 would have a cost of 4for both participants because no target has been reachedyet (1 1 number targets not yet reached [3] 5 4) How-ever a subsequent step from Node 8 to Node 1 would havecost 3 for the participant assigned the red targets (becauseNode 8 was a red target) and cost 4 for the participant as-signed the blue targets (because no blue target has beenreached yet) The motivation for the use of this specificcost rule is to give participants an incentive to try to reachtheir targets as soon as possible in the competitive modeand to reach any target as soon as possible in the cooper-ative mode

Summary of Task MotivationThe use of a route negotiation task enables the manip-

ulation of the communicational context (competitive vscooperative) and the tracking of the overall goal and sub-goals at any time during the negotiation The highly visualnature of the task makes the use of the pen natural (likedrawing with a pencil on a real map) but not essential forthe participants and the quality of the result of the nego-tiation (ie a certain route) can be computed objectively

One critical feature of SLOT is that it allows for asym-metric manipulation of the available channels (with theexception of the speech channel) in the sense that a cer-tain channel or modality can be available to one partici-

pant but not to the other This possibility is interesting inits own right but it is also important for research support-ing work in humanndashmachine interaction The availablechannels in humanndashmachine interaction are likely to beasymmetrically distributed due to the simple fact that theautomatic (computer-based) recognition of most relevantchannels is far less reliable than the corresponding recog-nition capabilities of humans

THE HARDWARE PLATFORM

General Description of the Current Laboratory Setup

The 2 participants in a SLOT experiment sit at a smalltable (with a surface of 1 3 1 m) opposite and facing eachother In front of each of them is a graphical tablet that ispositioned almost vertically at an 80ordm angle relative to thetable surface The height of the tablets does not prevent ei-ther participant from seeing the face of his or her interlocu-tor The graphical tablets are both controlled by a singlehigh-performance Pentium-IV personal computer Thiscomputer is operated by the experimenter who has accessto the keyboard mouse and screen The computer and theexperimenter operating it are located in the corner of thelaboratory at a distance of at least 25 m from the 2 par-ticipants in order to minimize the influence they exert onthe ongoing interaction between the participants

Three video cameras are used to record the partici-pantsrsquo behavior Two cameras are positioned opposite theparticipants (behind and 05 m above their respective in-terlocutors) to record their facial behavior whereas a thirdcamera is located perpendicular to the participants to pro-vide an overview of both participants simultaneously andfrom the side An omnidirectional stand-alone micro-phone on a tripod is positioned to the side of the partici-pantsrsquo desk to record the speech of both participants

The signals of the three video cameras and the infor-mation represented on the graphical tablets are capturedby a video processor and compressed into a 4-fold (2 3 2)split screen image The split screen image and the signalfrom the microphone are recorded using a professionalquality S-VHS video recorder This setup allows for the si-multaneous and time-locked recording of the speech non-verbal behavior and pen behavior of both subjects on onevideo tape For additional fine-grained analysis of the pengestures or handwriting behavior the pen data stored onthe hard disk can be used

Graphical TabletsA central feature of the hardware platform for SLOT is

the use of WACOM PL-550-02B0 graphical tablets col-loquially known as the ldquoWacom Cintiq 15Xrdquo These areflat desktop LCD devices that function as a replacementfor a standard color VGA computer screen In additionthey allow for drawing and erasing directly on the screensurface with an electronic pen Tablet coordinates aresampled at 200 Hz with a spatial resolution of 50 pointsper mm The pen is pressure sensitive with a resolution of

Figure 2 A competitive SLOT map


512 quantification levels The display allows for viewingangles of up to 160ordm The shared whiteboard is implementedby sending an identical VGA signal from the SLOT com-puter to both tablets (and to the computer monitor of theexperimenter) at the same time using a VGA splitterHowever the pen data are recorded separately for eachparticipant using two independent USB connections

An overview of the laboratory setup is given in Fig-ure 3 and a snapshot from a video recording of a SLOTexperiment is shown in Figure 4

THE SOFTWARE PLATFORM

General EnvironmentThe general software environment for which SLOT has

been developed is the GNULinux operating systemwhich is a noncommercial variant of the Unix operatingsystem The particular distribution of Linux that is used isthe SuSE 73 distribution although SLOT should run onany GNULinux distribution (we know of one SLOT en-vironment running without problems on a RedHat 72Linux system) The programming languages used inSLOT are C (for the SLOT program) C11 (for the analy-sis tools) and Icon (for the graphical map editor) SLOTis programmed in plain Xlib using the Athena widget setIn order to capture simultaneous data streams from thetwo USB tablets with minimal delays from operating sys-tem overhead we patched the USB drivers and rebuilt theLinux kernel

The SLOT ApplicationIn programming SLOT special care was taken to pro-

vide a writing environment that is as similar as possible toa real whiteboard The rendered ink traces take into ac-count pressure data If a participant exerts more pressurethe ink trace becomes thicker Furthermore the partici-

pants can use the back of the pen to erase their (and theother participantrsquos) virtual ink but not the displayed map

The SLOT application performs the following steps1 It draws a SLOT map that is specified in an experi-

mental design file (made with the editor described earlier)on both WACOM tablets and enters negotiation mode inwhich it (1) registers the participantsrsquo pen gestures duringthe negotiation phase and displays them in real time onboth tablets as virtual ink in blue for 1 participant and inred for the other (2) stores all pen activity on disk eachsample containing the xy coordinates and the corre-sponding pressure values and time stamp in milliseconds(3) registers the participantsrsquo erasure activity and processesthe virtual ink accordingly and (4) stores all erasure ac-tivity (time stamped) on disk

2 After the negotiation phase has been completed it al-lows the experimenter to put the program into submissionmode in which it (1) allows one of the participants (whichone is designated in the experimental design file) to enterthe negotiated route using the pen (2) stores the enteredroute (time stamped) on disk and (3) evaluates the enteredroute for legality If the route is illegal it provides feed-back to the participant about the nature of the error(s) andrestarts Step 2

3 It computes the costs incurred by the participants anddisplays them

4 It goes back to Step 1 until there are no more maps tobe displayed

Additional Computational ToolsIn addition to the SLOT program to run experiments

some other tools have been developed for the preparationand evaluation of SLOT maps

Graphical map editor In order to interactively designSLOT maps an interactive graphical map editor was writ-ten The program was written in the Icon programming

Figure 3 Schematic diagram of interconnected hardware in the SLOT laboratory


language which is portable among almost all existingcomputer platforms (eg Mac Linux and Windows NT)The program allows for the creation of maps by using themouse for selection placement and movement aroundnodes vertices and the keyboard for elementary opera-tions such as storing maps and assigning labels (eg ldquotar-getrdquo ldquostartrdquo) to nodes Map definitions are stored in xmlformat for easy editing and compatibility with existingmultimodal annotation tools

Map analyzer The map analyzer is a program writtenin C11 for the Linux platform It performs several cal-culations on SLOT maps that are relevant to the evaluationof certain dependent and independent variables in SLOTexperiments For any given map the program computes thefollowing properties (1) the optimal (ie lowest cost) routefor reaching all targets1 (2) the optimal route for the the par-ticipant assigned the red targets (competitive maps) andthe associated cost (3) the cost of the above route for theparticipants assigned the blue targets (4) the optimal routefor the the participant assigned the blue targets (competi-tive maps) and the associated cost (5) the cost for the aboveroute for the participants assigned the red targets (6) a mea-sure of how non-Euclidian a map is defined as the stan-dard deviation (SD) of the physical (screen) distances be-tween every two connected nodes after these distances arestandardized to an average of 1 (if all of the actual screendistances are the same this measure is equal to zero) and(7) a measure of the intrinsic unfairness of a competitivemap This is defined as the difference between the costs ofthe optimal route to reach all targets for the participant as-signed the red targets and the one assigned the blue tar-gets respectively

SLOT pen data analyzer and transformer The pendata collected through SLOT is stored in an efficient data

format in order to avoid significant disk access delays Aprogram was developed to examine these data

The program displays the collected pen data in real timeand can be used to analyze pen movements produced bythe participants during a SLOT experiment In additionthe stored pen data can be transformed to the UNIPENformat (see httpwwwunipenorg) which is the de factostandard for storing dynamic handwriting data For thisdata format many tools are available that can be used toannotate and analyze the data

Preprocessing SLOT DataBefore the data generated in SLOT experiments can be

used to address specific research questions they need tobe preprocessed by human coders The participantsrsquo speechneeds to be transcribed into a textual format whereas theother channels under investigation are to be coded intodiscrete categories with their corresponding time inter-vals For this a multimodal transcription tool such as EU-DICO (Brugman Russel Broeder amp Wittenburg 2000)can be used on the digitized video recordings See Table 1for an overview of a number of channels that can be an-notated in SLOT data

METHOD

In this section we present some initial results generated in our SLOTlaboratory to illustrate the type of research question that can be in-vestigated using SLOT First we will describe the experiment (moreinformation can be obtained from de Ruiter 2003) and then we willpresent and discuss two illustrative results from this experiment

ParticipantsSixteen Dutch-speaking participants from the participant pool of

the Max Planck Institute for Psycholinguistics were paid to take part

Figure 4 Snapshot from SLOT video recording


in the experiment Eight participants were female and 8 were maleThe participants were randomly paired to create eight mixed-genderdyads None of the participants knew his or her experimental partner

MaterialsFor the experiment 10 competitive SLOT maps were generated with

the map editor described earlier Two of these maps were used forpractice purposes and the other 8 were designed for the actual exper-iment Each map contained three red targets and three blue ones Usingthe map analyzer four of the experimental maps were created suchthat the intrinsic unfairness was high (ranging from 10 to 24 M 5 17)whereas for the other four it was low (each of them rating 1) Theorder of presentation of the maps was randomized and was the samefor all dyads

DesignThe principal between-dyads manipulation in this experiment is

that for four of the dyads there was a one-way (half-silvered) mir-ror standing on the table between the two tablets Lighting condi-tions were arranged so that 1 of the participants could clearly see hisor her partner through the glass whereas the participant on the otherside could not see his or her partner due to the mirror effect The aimof this between-dyads manipulation was to study the effect of anasymmetric distribution of the visual modality between the partici-pants in a dyad The principal within-dyad manipulation was the in-trinsic unfairness of the used maps which was high for four exper-imental maps and low for the other four The independent variablesof gender color played in SLOT (red or blue) and side of the mir-ror on which the participants in the mirror condition sat were coun-terbalanced

ProcedureThe participants were handed written instructions in which the

rules of the SLOT task were explained They were encouraged tominimize both their personal costs and the global (summed) costsThere was no mention of a strict time limit but the participants weretold that the experiment was expected to last about 45 min to 1 hDuring a session with the two practice maps the participants couldget used to the shared whiteboard and the electronic pen and to sub-mitting the negotiated route into the computer using the electronicpen After the practice session the participants had an opportunityto ask questions after which the eight experimental maps were ne-gotiated During negotiation the participants were allowed to freelydraw on the tablets as much as they wanted After reaching an agree-ment about the route to follow the participants would signal this tothe experiment leader who would put SLOT in route-submissionmode The participants took turns entering the negotiated routes intothe computer

TranscriptionFor detailed information about the transcription procedure and the

used maps we refer the reader to de Ruiter (2003) For the purposes

of the analysis below we will mention a few relevant aspects of thetranscription process Time markers were created that allowed us totemporally locate the beginning and end of the negotiation for everymap Pen gestures were transcribed initially by marking their begin-nings and ends in time The beginning end and transcript of eachspeech turn produced by each of the participants were coded Thecost scores were stored automatically by the SLOT computer

Results and DiscussionIn this section we will present and discuss two analyses

of data collected in the SLOT experiment The first focuseson the effect of blocking the visual modality for only oneof the participants in a SLOT session on efficiency and per-formance in the negotiation task The second analysis con-cerns the use of the pen-gesture data generated in SLOTto investigate the possible use of multimodal informationto improve automatic pen-gesture recognition and inter-pretation

Modality Effects on Performance and EfficiencySince the main focus of this study is the possible effects

of asymmetrical availability of the visual modality on ne-gotiation performance and efficiency the main dependentvariables of interest in this study are the duration of thenegotiation and the cost of the negotiated route A generallinear model analysis was performed to investigate a pos-sible main effect of the presence of the mirror (the fixedfactor mirror) and of the fairness of the maps (the fixed fac-tor fairness) The data entered into the analysis were thedurations of the eight experimental maps for each of theeight dyads leading to a total of 64 data points See Ta-ble 2 for a summary of the results As can be seen both thefactors mirror and fairness have a substantial effect on thenegotiation times

The main effect of fairness was significant [F(163) 5613 p 5 016] as was the main effect of mirror [F(163) 5430 p 5 042] The interaction of fairness and mirror wasnot significant [F(163) lt 1] Both the presence of the mir-ror and the (un)fairness of the maps led to an increase of

Table 1Channels Typically Annotated in SLOT Data

Channel Registration Medium Focus of Analysis

Speech Video soundtrack Speech acts (content analysis)Turn-taking behavior

Intonation Video soundtrack Intonational contoursphrasesGaze Video Eye contact between participantsPosture Video Body position shiftsHead orientation Video Use of head-tilt signalsFacial expression Video Finding set of used facial expressionsPen gesture Video 1 digital registration Classes of pen gesture relation of pen

gesture to speech (timing content)Handwriting Video 1 digital registration Use of labels words exclamation

marks etc

Table 2Mean Negotiation Durations (in Seconds) by Condition

Factor No Mirror Mirror

Fair 818 1508Unfair 1632 2219


the duration of the negotiation This increased durationcould have been caused either by the participantsrsquo pro-ducing more speech or by their being silent for a largerproportion of the time Additional analyses of both totalspeech time (the sum of the duration of the speech of bothparticipants in the negotiation) and silence (the time dur-ing which neither of the participants spoke) revealed thatthe prolonged duration for the factor mirror was due tolonger silences whereas that for the factor fairness wasdue to both more speech and more silence That there wasmore speech and more silence with the unfair maps is notsurprising because these maps are harder to negotiatewhich could be expected to lead to more discussion andalso to longer periods of pondering over proposals SeeTable 3 for the average total speech time and silence timeper negotiation in the four conditions Note that speech timeand silence time do not add up to negotiation time due tothe fact that the participants often spoke simultaneously

The effect of mirror on speech was not significant[F(163) lt 1] whereas the effect of mirror on silence wassignificant [F(163) 5 73 p 5 009] indicating that theeffect of the mirror was indeed to increase not the total du-ration of speech but rather the total duration of silenceThe main effects of fairness on both speech and silencewere significant [speech F(163) 5 723 p 5 009 si-lence F(163) 5 50 p 5 03]

We now analyze the effect of the presence of the mir-ror on the cost scores of the negotiation This analysis willbe performed not on dyads as the previous analysis wasbut on individual participants Three groups of participantscan be identified those who had no mirror in front of them(duplex) those who were on the side of the mirror wherethey could not see their partners (nonvisual) and those whowere on the side of the mirror where they could see theirpartners (visual) The average cost over all eight maps was4656 (SD 5 697) for the duplex participants 4541 (SD 51003) for the nonvisual participants and 4566 (SD 5899) for the visual participants None of these differencesreached conventional levels of significance (all Fs lt 1) sup-porting the conclusion that the presence of the mirror did nothave an effect on the final outcome of the negotiation

Discussion of Modality Effects on Performanceand Efficiency

To summarize the previous analysis revealed the fol-lowing

1 Neither the presence of the one-way mirror nor theside on which the participants in the mirror condition wereseated had any effect on the outcome of the negotiation

2 The presence of the one-way mirror made negotia-tions last longer and this was due to the participantsrsquobeing silent for longer periods of time

3 The more intrinsically unfair the map the morespeech was produced and the longer was the silence time

To start with the last finding it shows that one impor-tant criterion of SLOTmdashnamely the possibility of varyingthe level of competitiveness of the negotiationmdashwas metWith respect to the first two findings it is interesting tocompare them with the results by Drolet and Morris (2000)which are remarkably similar In their study they com-pletely blocked visual contact between the negotiators andcompared negotiation times and results with those of an-other group for which visual contact was possible Theyfound that negotiation times were 25 longer in the con-dition without visual contact (in our study they wereabout 50 longer in the mirror condition) and that the ne-gotiation results were unaffected They attribute theirfindings to a higher level of rapport (see Tickle-Degnen ampRosenthal 1990) between participants who could see eachother and we have no reason to disagree with their expla-nation The study reported here in combination with thestudy by Drolet and Morris suggests that the availabilityof the visual modality affects negotiation times but not theoutcome and that these effects on negotiation time andoutcome are independent of whether the visual modalitywas blocked for both or just one of the negotiators Ap-parently in order for the visual modality to facilitate thedevelopment of rapport the participantsrsquo possibility of ex-changing visual signals must be mutually manifest (Sper-ber amp Wilson 1995) This finding also has important con-sequences for HCI research in which 3-D animations of ahuman-like head are used but in which the computer can-not perceive the visual signals of the human user If thegoal is to increase efficiency in HCI by creating rapportthis will work only if users have at least the illusion thatthe computer can see them as well How this illusion canbe created without giving the computer access to the vis-ible behavior of the user is an important question that willbe addressed in future research

Analysis and Automated Recognition of Multimodal Pen Gestures

The combination of pen and speech input is widelyused in multimodal humanndashcomputer interaction (Oviatt1999) In general the use of two or more modalities is re-quired in natural interaction dialogs with the computer tomake it possible to disambiguate user utterances in eithermodality In this section we describe how the manualanalysis and annotation of the data acquired throughSLOT is used to develop automated systems for the recog-nition of pen gestures produced in the SLOT task

In any pattern recognition task such as the present ex-ample knowing about the participantsrsquo speech and ges-ture repertoires and the availability of properly annotateddata is of paramount importance These are needed to pro-vide insight into how users generate and use speech andgesture to train and test the required pattern recognition

Table 3Mean Duration of Speech

and Silence (in Seconds) by Condition


Fair Speech 482 606Silence 577 1206

Unfair Speech 1084 889Silence 1090 1774


algorithms and also for the design of user interfaces (Ovi-att et al 2000 Potamianos Kuo Pargellis Saad amp Zhou1999 Rossignol et al 2003) We have annotated the pengestures and speech utterances in order to identify possi-ble classes of pen gestures It was found that the interpre-tation of pen gestures is generally not possible without theaccompanying speech and the visual context in which theyoccurred (ie the SLOT map on the whiteboard) This isa phenomenon that has also been observed in manual ges-tures and speech (de Ruiter 2000 McNeill 1992) Notethat the gestures recorded in the SLOT environment wereproduced for the benefit of the understanding of anotherhuman and that the recipient of the pen gestures under-stood them without any apparent problems

Two annotators marked and classified a total of 454 pengestures in the SLOT data using the recorded image ofthe pen gesture and the accompanying speech of the pro-ducer of the pen gesture Four main classes of pen ges-tures where identified In the data the most frequent typewas the trajectory (N 5 313) A trajectory is an uninter-rupted line connecting two or more cities correspondingto a (partial) route through the negotiated map The sec-ond class that we identified was the marker class (N 5113) A marker is used to indicate a certain point on themap usually a city The third class is called directionalpointing (N 5 18) which consists of a line or arrow thatindicates a certain direction on the map The final class iscalled area (N 5 10) This is an encircling type of pen ges-ture indicating an area on the map that contains more thanone city We refer to Figure 5 for examples of recordedpen gestures from each of the four classes

For the development of automated pen-gesture recog-nition systems we used the even-numbered gestures in thedata set for the training of a classification method and weused the odd-numbered gestures for testing For the sub-sequent computational analysis the high-resolution pendata from the tablets that are stored on the SLOT computerwere used As is apparent from the second row in Figure 5samples from different classes can have highly similarshapes

Two well-known classification methods were used forthe automatic recognition the multilayered perceptron(MLP see Rumelhart Hinton amp Williams 1986) and thek-nearest neighbor method (KNN see Duda amp Hart1973) From each pen gesture seven features were extractedto be used as feature vectors for training and testing theclassifiers The features we used were (1) the number ofpoints with high curvature (curvature being defined as an-gular velocity divided by absolute velocity) (2) the totalnumber of samples (sampling rate 5 200 Hz) (3) the spa-tial distance between the first and last samples (4) thespatial distance between the first sample and the sampleoccurring after half the duration of the pen gesture (whichwe call the halfway sample) (5) the spatial distance be-tween the halfway sample and the last sample (6) the sur-face area of the bounding box enclosing the entire gestureand (7) the ratio of the width and the height of the bound-ing box enclosing the gesture

For details about the type of features used in pen ges-ture recognition see Rubine (1991)

Both the KNN and the MLP used the same training dataand testing data The recognition result of the MLP was

Figure 5 Examples of recorded pen gestures of the four different classes


881 which is much better than the 718 we obtainedwith the KNN (using k 5 5) The MLP used seven inputnodes two hidden layers with eight nodes each and fouroutput nodes In Table 4 the confusion matrix for theMLP analysis is shown

As can be seen from the confusion matrix the MLP isunable to distinguish between area and marker Every areais recognized as a marker Also directional pointings aremore often than not recognized as trajectories This is be-cause the pen gestures of markers and areas and alsothose of trajectories and directional pointings often havesimilar shapes (see the middle row of Figure 5) Further-more the total number of area gestures and directionalpointings in our data set is low making correct recognitionmore difficult due to limited training data

Clearly more information is required if similar gesturesof different classes are to be distinguished Within theSLOT domain another potential source of information isthe location of the cities on the map This informationcould be used to disambiguate between circular markers(which enclose only one city) and areas (which enclosetwo or more cities) The distinction between trajectoriesand directional pointings would be improved by use of theinformation that trajectories always pass through cities onthe map whereas directional pointings rarely do

The approach that is explored here is to consider dis-tinctive fragments of the accompanying speech as new or-thogonal features For each pen gesture we located thespeech turn of the producer of that pen gesture that wastemporally closest to the occurrence of the pen gestureFrom the transcription of that turn we counted the occur-rences in speech of five words that often accompany pengestures The relative frequencies of these words couldperhaps be used to disambiguate conflicting cases Thewords that we counted were the Dutch words die deze zohier and daar [English ldquothatrdquoldquothoserdquo ldquothisrdquoldquotheserdquoldquothis wayrdquo (manner or path) ldquohererdquo and ldquothererdquo respec-

tively] These words were chosen because they often occurin utterances in which gesture and speech are combinedIn Table 5 the relative frequency with which these wordsco-occurred with the pen gestures from the four differentclasses is presented

It can be concluded that the occurrence of the word hier(English ldquohererdquo) indicates that it is likely that a marker orarea was generated by the user Furthermore if the wordzo (English ldquothis wayrdquo) is used it is likely that a trajectoryor directional pointing was generated

The use of multiple classifiers for difficult pattern recog-nition problems is a well studied technique The approachfollowed here is described in Vuurpijl Schomaker andVan Erp (in press) in which the comparison of outputsfrom multiple classifiers is used to resolve conflicting sit-uations To distinguish between the confusing gestureclasses the following two-staged classification schemewas used During the first classification stage the resultsfrom both classifiers were considered on the basis ofshape information only In cases in which both the KNNand the MLP yielded the same output this was consideredas the f inal output of the combined classifier If eachyielded a different output the speech signal was consid-ered to rule between outcomes If the speech featurescould not be used to disambiguate between classes theoutcome of the MLP was used The latter decision wasbased on the fact that the MLP yields the highest recogni-tion results Note that with this method if both classifiersare wrong there is no way to correct the output of the clas-sification

In all cases in which both classifiers agreed during thefirst classification stage the resulting classification wascorrect In 63 cases a conflict between the two classifierswas observed In 13 of these 63 confusions the two clas-sifiers yielded different outcomes but both were wrongIn 22 cases the speech signal did not contain extra infor-mation (eg there were no occurrences of words) From

Table 4Confusion Matrix for Pen Gesture Classification Using MLP

Gesture Is Recognized as

Gesture Is of Type Marker Area Trajectory Directional pointing N

Marker 7937 0 1746 317 63Area 10000 0 0 0 6Trajectory 201 0 9799 0 149Directional pointing 0 0 5556 4444 9

Table 5Probability of Occurrence of Words in Concurrent Speech

for Each Pen Gesture Class

Pen Gesture Class

Concurrent Speech Marker Area Trajectory Directional pointing N

die (ldquothatrdquoldquothoserdquo) 30 22 24 15 84deze (ldquothisrdquoldquotheserdquo) 59 78 24 08 125zo (ldquothis wayrdquo) 29 67 100 100 234hier (ldquohererdquo) 100 100 13 00 161daar (ldquothererdquo) 05 11 07 23 24


these 22 cases the MLP outcome resulted in four errorsFor the remaining 28 cases the speech features were con-sidered to determine the final outcome In 16 cases thisresulted in an error amounting to a total of 33 (13 1 4 1 16)errors yielding a recognition performance of only 855

Although our present automatic gesture recognizerdoes not perform at the desired near-100 level using thenatural SLOT data has given us some valuable insights intothe nature of humanndashhuman pen gestures Researchers inmultimodal HCI often discuss and try out multichannelintegration using pen gestures and speech but our data(both the qualitative impressions from the manual anno-tations and the results from automatic recognizers) sug-gest that the visual context in which pen gestures occurand possibly also a deeper level of semantic analysis ofthe speech are indispensable for accurate automatic ges-ture recognition The SLOT platform is well suited forgenerating accurately recorded natural pen gestures in-tended for other humans together with coexpressivespeech which occur in well-defined 2-D contexts

DISCUSSION

The SLOT route negotiation task is a communication taskthat imposes some structure on the communication withoutdisrupting the natural flow of conversation Its structure issymmetrical in that both participants in an experimenthave the same task and role Nevertheless the platform al-lows for the systematic research of asymmetric modalityand channel distributions without changing the basicstructure of the task Furthermore it enables the efficientrecording and analysis of speech pen gesture and severalchannels of nonverbal communication

In this article we have presented two illustrative re-search projects using SLOT First we showed an exampleof an asymmetric modality manipulation (in this case thevisual modality) that yielded nontrivial results regardingefficiency and performance in negotiation The findingthat efficiency in interaction is served by mutually mani-fest visual communication is relevant both for fundamen-tal communication research and for HCI applications in-volving an animated head In our second study we tried totrain automatic recognizers on pen gestures that were notperformed for the computer but used in humanndashhumancommunication in the same way as two people might usea real whiteboard together The results of this study serveto focus and guide further research in human pen gesturesand their relationship to the accompanying speech and thevisual context

These projects are by no means the only ones that arecurrently performed with SLOT Studies are in prepara-tion that compare the pen gestures from SLOT with spon-taneously produced 3-D hand gestures (see de Ruiter ampWilkins 1998) Furthermore the speech data from SLOTexperiments are being used to develop quantitative mod-els of turn-taking and to investigate the use of visual sig-nals to regulate turn-taking behavior Another study isaimed at comparing efficiency in the SLOT task with andwithout the use of the pen Finally the negotiation moves

in SLOT are annotated and analyzed in detail for study ofthe structure of negotiation under different levels of com-petitiveness and with varying channel availability

The SLOT platform has not only served us well to ad-dress the issues we had in mind during its design but it hasalso inspired many new research directions We inviteother researchers in the field of multimodal communica-tion to consider using this platform as well

AvailabilityResearchers or institutions interested in using the SLOT

platform are advised to contact the first author for furtherinformation on how to obtain copies of our software andmanuals All the hardware components used in SLOT areavailable on the market

REFERENCES

Anderson A H Bader M Bard E G Boyle E Doherty GGarrod S Isard S Kowtko J McAllister J Miller JSotillo C amp Thompson H S (1991) The HCRC Map Task Cor-pus Language amp Speech 34 351-366

Archer D amp Akert R M (1977) Words and everything else Ver-bal and nonverbal cues to social interpretation Journal of Personalityamp Social Psychology 35 443-449

Argyle M Salter V Nicolson H Williams N amp Burgess P(1970) The communication of inferior and superior attitudes by ver-bal and nonverbal signals British Journal of Social amp Clinical Psy-chology 9 222-231

Brown R (1986) Social psychology (2nd ed) New York Free PressBrugman H Russel A Broeder D amp Wittenburg P (2000 May)

EUDICO Annotation and exploitation of multi media corpora Paperpresented at the International Conference on Language Resources andEvaluation Athens

Burkard R E (1979) Traveling salesman and assignment problem Asurvey In P L Hammer E L Johnson amp B H Korte (Eds) Discreteoptimization (Vol 1 pp 193-215) Amsterdam North-Holland

Clark H H amp Krych M A (in press) Speaking while monitoringaddressees for understanding Journal of Memory amp Language

Cunningham D Breidt M Kleiner M Wallraven C ampBuumllthoff H (2003 May) How believable are real faces Towardsa perceptual basis for conversational animation Paper presented atthe Conference on Computer Animation and Social Agents (CASA)New Brunswick NJ

de Ruiter J P (2000) The production of gesture and speech In D Mc-Neill (Ed) Language and gesture (pp 284-311) Cambridge Cam-bridge University Press

de Ruiter J P (2003) Research on modality effects on performancequality and efficiency (IST COMIC Public Report ) Available athttpwwwhcrcedacukcomicdocuments

de Ruiter J P amp Wilkins D P (1998 December) The synchronisa-tion of gesture and speech in Dutch and Arrernte (an Australian Abo-riginal language) A cross-cultural comparison Paper presented atthe Confeacuterence Oraliteacute et Gestualiteacute Besanccedilon France

Drolet A L amp Morris M W (2000) Rapport in conflict resolutionAccounting for how face-to-face contact fosters mutual cooperation inmixed-motive conflicts Journal of Experimental Social Psychology36 26-50

Duda R amp Hart P (1973) Pattern classification and scene analysisNew York Wiley

Ellsworth P C amp Ludwig L M (1972) Visual behavior in socialinteraction Journal of Communications 22 375-403

Goodwin C (1981)Conversational organization Interaction betweenspeakers and hearers New York Academic Press

Kendon A (1972) Some relationships between body motion andspeech In A W Sigman amp B Pope (Eds) Studies in dyadic commu-nication New York Pergamon

Krauss R M Apple W Morency N Wenzel C amp Winton W


(1981) Verbal vocal and visible factors in judgments of anotherrsquos af-fect Journal of Personality amp Social Psychology 40 312-319

Levelt W J M (2001) SLOT A spatial logistics planning task forCOMIC (Unpublished report) Nijmegen Max Planck Institute forPsycholinguistics

McNeill D (1992) Hand and mind Chicago Chicago University PressMehrabian A amp Wiener M (1967) Decoding of inconsistent com-

munications Journal of Personality amp Social Psychology 6 109-114Oviatt S [L] (1999) Ten myths of multimodal interaction Commu-

nications of the ACM 42 75-81Oviatt S L Cohen P R Wu L Vergo J Duncan L Suhm B

Bers J Holzman T Winograd T Landay J Larson J ampFerro D (2000) Designing the user interface for multimodal speechand gesture applications State-of-the-art systems and research direc-tions HumanndashComputer Interaction 15 263-322

Potamianos A Kuo H Pargellis A Saad A amp Zhou Q (1999June) Design principles and tools for multimodal dialog systemsPaper presented at the ESCA Workshop on Interactive Dialog andMulti-modal Systems Koster Irsee Germany

Rossignol S Ten Bosch L Vuurpijl L Neumann A Boves LDen Os E amp de Ruiter J P (2003 June) Multi-modal interactionfor bathroom design The role of human factors Paper presented at theHCI International 2003 Crete Greece

Rubine D (1991) Specifying gestures by example Journal of Com-puter Graphics 25 329-337

Rumelhart D E Hinton G E amp Williams R J (1986) Learninginternal representations by error propagation In D E Rumelhart ampJ L McClelland (Eds) Parallel distributed processing Explorationsin the microstructure of cognition (Vol 1 pp 318-362) CambridgeMA MIT Press

Sperber D amp Wilson D (1995)Relevance Communication and cog-nition (2nd ed) Oxford Blackwell

Tickle-Degnen L amp Rosenthal R (1990) The nature of rapportand its nonverbal correlates Psychological Inquiry 1 285-293

Vuurpijl L Schomaker L amp Van Erp M (in press) Architecturesfor detecting and solving conflicts Two-stage classification and sup-port vector classif iers International Journal of Document Analysisand Recognition

NOTE

1 The complexity of this computation increases exponentially withthe number of target nodes which means that in practice it is feasibleonly for maps having a maximum of 10 targets

(Manuscript received September 19 2002 revision accepted for publication May 14 2003)


factuals and tense information) linguistic communicationhas a clear advantage over nonverbal communication

The different channels through which people communi-cate often complement each other For instance a spokenstatement accompanied by raised eyebrows might be in-terpreted as a question by the interlocutor A spoken state-ment containing the phrase ldquoover thererdquo is usually accom-panied by a pointing gesture to indicate where ldquothererdquo isIn sum face-to-face interaction is a form of multimodalcommunication

The word multimodalitysuggests that the concept refersto more than one perceptual modality (eg auditory per-ception and vision) However this interpretation of theconcept is too narrow to capture the phenomena in whichmultimodal communication researchers are generally in-terested Although both hand gestures and facial expres-sions are perceived in the visual modality they clearly rep-resent distinct aspects of communication The same holdsfor voice quality and content of speech both of which relyon the auditory modality To remedy this definitionalproblem we propose the notion of a semiotic channel Asemiotic channel is a set of identifiable behavioral unitsthat (1) cannot be performed simultaneously with eachother and (2) can be performed simultaneously with (al-most) all behavioral elements in other semiotic channelsFor example facial expression and voice quality consti-tute two different semiotic channels for it is not possibleto have two different facial expressions or two differentvoice qualities at the same time whereas all voice quali-ties can in principle be combined with all facial expres-sions The proposed definition is admittedly not entirelyairtight Although we believe eye gaze and facial expres-sion to be separate semiotic channels certain facial ex-pressions such as the ldquothinkingrdquo expression (Cunning-ham Breidt Kleiner Wallraven amp Buumllthoff 2003) oftencontain specific eye-gaze patterns as integral parts How-ever given that eye gaze and facial expression can in mostcases operate independently of each other we neverthe-less consider them to be separate semiotic channels Dueto the widespread use in the literature of the term multi-modal communication we will continue to use this termto refer to the general field of study but what we mean byit is communication involving the simultaneous use ofmore than one semiotic channel We will reserve the wordmodality to refer to the perceptual modality (eg visualauditory) and will use the word channel to refer to semi-otic channels as defined above

In order to gain systematic knowledge about how mul-timodal communication between human participantsworks the collection of behavioral data is essential Astraightforward way to obtain data is to record interactionsbetween people and analyze them carefully (see egGoodwin 1981 Kendon 1972) A disadvantage of thisnaturalistic approach is that one has very little control overthe content of the communication which makes it ratherhard to predict what participants are going to communi-cate about and how they are going to do it This in turnleads to an unacceptable level of variance in both the con-

tent of the communication and the use of the differentmodalities in which the humanndashhuman interaction re-searcher is interested

A higher level of control over the content of the com-munication is achieved by Clark and Krych (in press) Intheir task one participant had to instruct another in build-ing a prespecified Lego model An important advantageof this task is that it provides an objective measure of boththe efficiency and the quality of the communication (asdemonstrated by the duration of the interaction and thenumber of errors in the resulting Lego model respec-tively) Although these tasks yield results that are impor-tant for the field of multimodal communication they arelimited to the study of collaborative communicationAlso the two participants have different predefined rolesOne is ldquodirectorrdquo and the other is ldquobuilderrdquo

Another interesting multimodal task is the map task(Anderson et al 1991) In this task the 2 participants eachhave a map in front of them and one of them (the routegiver) instructs the other (the route follower) to follow acertain route through the map The maps used by route giverand route follower are not identical which makes the taskmore difficult and therefore potentially more interestingAs with the directorbuilder task of Clark and Krych (inpress) this task assigns different roles to the 2 participants

An important reason for us to develop a new researchplatform instead of using the map task or the directorbuilder task is that we wanted to use a communication taskthat is highly structured but does not assign different rolesto the 2 participants Also since an important goal of thehumanndashhuman data that we want to generate is to provideinformation that could produce guidelines for (natural-looking) humanndashcomputer interaction (HCI) we wantedour task to involve the use of both electronic pen data andspeechmdashmodalities often used in multimodal HCI (Ovi-att 1999) Finally we wanted to be able to have control overthe competitiveness of the interaction to be able to dis-tinguish between multimodal communication at differentlevels of competitiveness of the task

THE COMMUNICATIVE TASK

CriteriaAn earlier version of the communicative task in the spa-

tial logistics task (SLOT) was originally developed (seeLevelt 2001) with the following criteria in mind

1 The experimenter should have a high degree of con-trol over the communicative goals of the participants dur-ing the experiment This should be the case for general(overall) goals as well as for possible subgoals at any timeduring the course of the experiment

2 It should be possible to assess objectively the qualityof the end result of the communication

3 Although any task involving naturalistic uncon-strained communication will result in data with high inter-and intrasubject variability this variability should be keptas low as possible to facilitate analyses with sufficient sta-tistical power





variable




















































METHOD


























marks etc



Fair 818 1508Unfair 1632 2219










































Pen Gesture Class






DISCUSSION








REFERENCES
































NOTE







variable




















































METHOD


























marks etc



Fair 818 1508Unfair 1632 2219










































Pen Gesture Class






DISCUSSION








REFERENCES
































NOTE











































METHOD


























marks etc



Fair 818 1508Unfair 1632 2219










































Pen Gesture Class






DISCUSSION








REFERENCES
































NOTE
























marks etc



Fair 818 1508Unfair 1632 2219










































Pen Gesture Class






DISCUSSION








REFERENCES
































NOTE












































Pen Gesture Class






DISCUSSION








REFERENCES
































NOTE






DISCUSSION








REFERENCES
































NOTE

















NOTE



slot: a research platform for investigating multimodal

Documents