Robotic Partners’ Bodies and Minds:An Embodied Approach to Fluid Human-Robot Collaboration

Guy Hoffman and Cynthia BreazealMIT Media Laboratory

20 Ames Street E15-468Cambridge, MA 02139

{guy,cynthiab}@media.mit.edu

Abstract

A mounting body of evidence in psychology and neuro-science points towards an embodied model of cognition,in which the mechanisms governing perception and ac-tion are strongly interconnected, and also play a centralrole in higher cognitive functions, traditionally modeledas amodal symbol systems.We argue that robots designed to interact fluidly withhumans must adopt a similar approach, and shed tra-ditional distinctions between cognition, perception, andaction. In particular, embodiment is crucial to fluid jointaction, in which the robot’s performance must tightlyintegrate with that of a human counterpart, taking ad-vantage of rapid sub-cognitive processes.We thus propose a model for embodied robotic cogni-tion that is built upon three propositions: (a) modal, per-ceptual models of knowledge; (b) integration of percep-tion and action; (c) top-down bias in perceptual process-ing. We then discuss implications and derivatives of ourapproach.

“[T]he human being is a unity, an indivisible whole. [...] ideas,emotions and sensations are all indissolubly interwoven. A bodilymovement ‘is’ a thought and a thought expresses itself in corporealform.”

— Augusto Boal (2002)

IntroductionWe aim to build robots that work fluidly with humans in ashared location. These robots can be teammates in a jointhuman-robot team; household, office, or nursing robots as-sisting everyday people in their tasks; or robotic entertainersperforming in conjunction with a human actor. In each ofthese cases, the robot is expected to maintain continuous in-teraction with a person, tightly meshing its actions with thatof the human.

Drawing from a growing body of knowledge in psychol-ogy and neuroscience, we propose an architecture that di-verges from much of the traditional work in artificial intel-ligence in that it emphasizes the embodied aspects of cog-nition (Wilson 2002). It is becoming increasingly clear

Copyright c© 2006, American Association for Artificial Intelli-gence (www.aaai.org). All rights reserved.

that human (as well as animal) perception and action arenot mere input and output channels to an abstract sym-bol processor or rule-generating engine, but that insteadthought, memory, concepts, and language are inherentlygrounded in our physical presence (Pecher & Zwaan 2005;Barsalou 1999; Wilson 2001). The same principles alsoplay a substantial role in social cognition and joint action,when it becomes crucial to coordinate one agent’s activ-ity with another’s (Sebanz, Bekkering, & Knoblich 2006;Barsalou et al. 2003). It is our belief that these lessons canand should be transferred to robotic cognition as well, andin particular if we are to design robots that act in natural andfluid dialog with a human partner.

A central goal of this work is to steer away from the stop-and-go turntaking rigidity present in virtually all human-robot interaction to date. Based on many of the same under-lying assumptions of symbolic AI, robotic dialog systems,robotic teammates, and robotic stage actors take little or noadvantage of the social and non-verbal behaviors that enablehumans to easily display an impressive level of coordinationand flexibility. The premise of this work is that a physicallygrounded cognitive architecture can give rise to appropriatebehavior enabling a much more fluid meshing of a robot’sactions with those of its human counterpart.

In this paper we propose an architecture built on three in-terrelated principles backed by psychological and neurolog-ical data:

Modal, perceptual models of knowledge Much of AI isfocused on amodal theories of knowledge, which as-sert that information is translated from perceptual stim-uli into nonperceptual symbols, later used for informationretrieval, decision making, and action production. Thisview also corresponds to much of the work currently donein robotics, where sensory input is translated into seman-tic symbols, which are then operated upon for the pro-duction of motor control. In contrast, a growing num-ber of psychologists and neuroscientists support a percep-tual model of cognition, in which perceptual symbols arestored through modus-specific mechanisms, subsequentlyused by ways of “simulation” or “imagery” (Barsalou1999; Kosslyn 1995). Perceptual symbols are organizedin cross-modal networks of activation which are used toreconstruct and produce knowledge.

Integration of perception and action Evidence alsopoints to a close integration between perceptual mecha-nisms and action production. A large body of work pointsto an isomorphic representation between perception andaction, leading to mutual and often involuntary influencebetween the two (Wilson 2001). These mechanismscould also underlie the rapid and effortless adaptation toa partner that is needed to perform a joint task (Sebanz,Bekkering, & Knoblich 2006). For robots that are towork in a similarly fluid manner, it makes sense tobreak from the traditional modular approach separatingperception, action, and higher-level cognition. Instead,intentions, goals, actions, perceptions, concepts, andobject representations should interact in a complexnetwork of activation, giving rise to behaviors that areappropriate in timing and context.

Top-down bias in perceptual processing Finally, the twoprinciples outlined above, as well as a large body of re-lated experimental data, give rise to the following insight:perceptual processing is not a strictly bottom-up analysisof raw available data, as it is often modeled in robotic sys-tems. Instead, simulations of perceptual processes primethe acquisition of new perceptual data, motor knowledgeis used in sensory parsing, and intentions, goals, and ex-pectations all play a role in the ability to parse the worldinto meaningful objects. This seems to be particularly truefor the parsing of human behavior in a goal-oriented man-ner, a vital component of joint action.

In this paper, we will briefly present evidence for the prin-ciples outlined above and discuss their pertinence to fluidjoint action. We will describe how we use these principlesin a cognitive framework for a robot designed to act jointlywith a human counterpart. We then discuss a number of in-terrelated concepts as they stem from this approach, notably:(a) attention, (b) routine action, (c) practice, (d) flexible res-olution, (e) anticipatory action, and (f) mutual responsive-ness.

Embodied Cognition: Evidence fromNeuropsychology

During the second half of the twentieth century artificialintelligence has not only drawn from theories of cognitivepsychology, but also shaped notions of amodal, symbolicinformation processing. According to this view, perceptualinput is hierarchically processed in isolated modules, andeventually gives rise to a non-perceptual representation ofconcepts, which are processed symbolically and may subse-quently give rise to physical behaviors.

An increasing body of recent findings challenges this viewand suggests instead that concepts and memory are phenom-ena which are grounded in modal representations utilizingmany of the same mechanisms used during the perceptualprocess. Moreover, action production and perception are notseparated by supervisory, symbolic rule operators, but in-stead intimately linked. Models of action production affectperceptual acquisition and retention, and give rise to a num-ber of intermodal effects.

Perceptual Models of KnowledgeIn recent years, many authors in language, psychology, andneuroscience have supported perceptual theories of cog-nition (Spivey, Richardson, & Gonzalez-Marquez 2005;Lakoff & Johnson 1999; Stanfield & Zwaan 2001). A promi-nent theory explaining these findings is one of “simulators”,siting long-term memory and recall in the very neural mod-ules that govern perception itself (Barsalou 1999). By thistheory, concepts are formed as activation patterns of expe-riences related to the pattern, stored in associative networksrelating them to other modalities, and re-invoked using per-ceptual simulators of such experiences. This view is sup-ported by a large number inter-modal behavioral influences,as well as by the detection of perceptual neural activationwhen a subject is engaging in cognitive tasks. Thus, whenmemory—and even language—is invoked to produce behav-ior, the underlying perceptual processes elicit many of thesame behaviors normally used to regulate perception.

A similar case has been made with regard to hand signals,which are viewed as instrumental to lexical lookup duringlanguage generation (Krauss, Chen, & Chawla 1996), andis supported by findings of redundancy in head-movements(McClave 2000) and facial expression (Chovil 1992) duringspeech generation.

A number of experiments evaluating perceptual taskshave shown that task time is related to the mental simula-tion of kinematic configuration (Parsons 1994). Similarly, astudy shows that children who mispronounce certain lettersconfuse the same letters in word recall, even when the mem-ory cue is purely visual (Locke & Kutz 1975). In visualprocessing, imagery is a conscious simulation mechanism(Kosslyn 1995), and is believed by some to use the sameneural mechanisms as perception (Kreiman, Koch, & Fried2000; Kosslyn 1995).

Perception-Action IntegrationIn parallel to a perception-based theory of cognition liesan understanding that cognitive processes are equally inter-woven with motor activity. Evidence in human develop-mental psychology shows that motor and cognitive devel-opment are not parallel but highly interdependent. For ex-ample, artificially enhancing 3-month old infant’s graspingabilities (through the wearing of a ‘sticky’ mitten), equatedsome of their cognitive capabilities to the level of older, al-ready grasping1, infants (Somerville, Woodward, & Need-ham 2004).

Additionally, neurological findings indicate that—insome primates—observing an action and performing itcauses activation in the same cerebral areas (Gallese et al.1996; Gallese & Goldman 1996). This common coding isthought to play a role in imitation, and the relation of thebehavior of others to our own, which is considered a centralprocess in the development of a Theory of Mind (Meltzoff& Moore 1997). For a review of these so-called mirror neu-rons and the connection between perception and action, as itrelates to imitation in robots, see (Matarić 2002).

1In the physical sense.

The capability to parse the actions and intentions of oth-ers is a cornerstone of joint action. Mirror mechanisms havebeen found to play a role in experiments of joint activity,where it has been suggested that the goals and tasks of ateam member are not only represented, but coded in func-tionally similar ways to one’s own (Sebanz, Bekkering, &Knoblich 2006).

Another important outcome of the common mechanismof motor simulators and perception is that there seem to be“priviledged loops” between perception and action, whichare faster and more robust than supervised activity. Humanscan “shadow” speech with little effort or interference fromother modalities, and this effect is dampened when even thesimplest translation is required (McLeod & Posner 1984).

Much of our critical cognitive capabilities occur in whatis sometimes called a “cognitive unconscious” (Lakoff &Johnson 1999), and it seems that these are more prominentin routine activity than those governed by executive control.Routine actions are faster and seem to operate in an unsuper-vised fashion, leading to a number of action lapses (Cooper& Shallice 2000). From introspection we know that super-visory control is often utilized when the direct pathways failto achieve the expected results. These unsupervised capa-bilities seem to be highly subject to practice by repetition,and are central to the coordination of actions when workingon a joint task, a fact that can be witnessed whenever weenjoy the performance of a highly trained sports team or awell-rehearsed performance ensemble.

Top-down Perceptual ProcessingAn important conclusion of the approach laid out herein isthat intelligence is neither purely bottom-up nor strictly top-down. Instead, higher-level cognition plays an importantrole in the mechanisms of lower-level processes such as per-ception and action, and vice versa. This view is supportedby neurological findings, as stated in (Spivey, Richardson,& Gonzalez-Marquez 2005): “the vast and recurrent inter-connectedness between anatomically and functionally segre-gated cortical areas unavoidably compromises any assump-tions of information encapsulation, and can even wind upblurring the distinction between feedback and feedforwardsignals.”

Rather than intelligence and behavior emerging from ahierarchical analysis of perception into higher-level con-cepts, a continuous stream of information must flow in bothdirections—from the perception and motor systems to thehigher-level concepts, intentions and goals, and back fromconcepts, intentions, goals, and object features towards thephysically grounded aspects of cognition, shaping and bias-ing those for successful operation.

Experimental data supports this hypothesis, finding per-ception to be predictive (for a review, see (Wilson &Knoblich 2005)). In vision, information is sent both up-stream and downstream, and object priming triggers top-down processing, biasing lower-level mechanisms in sen-sitivity and criterion. Similarly, visual lip-reading affectsthe perception of auditory syllables indicating that the soundsignal is not processed as a raw unknown piece of data (Mas-

saro & Cohen 1983)2. High-level visual processing is alsoinvolved in the perception of human figures from point lightdisplays, enabling subjects to identify gender and identityfrom very sparse visual information (Wilson 2001).

This evidence leads to an integrated view of human cogni-tion, which can collectively be called “embodied”, viewingmental processes not as amodal semantic symbol processorswith perceptual inputs and motor outputs, but as integratedpsycho-physical systems acting as indivisible wholes.

Our work proposes to take a similar approach to designinga cognitive architecture for robots acting with human coun-terparts, be it teammates, helpers, or scene partners. It isfounded on the hope that grounding cognition in perceptionand action can hold a key to socially appropriate behavior ina robotic agent, as well as to context and temporally precisehuman-robot collaboration, enabling hitherto unattained flu-idity in this setting.

ArchitectureInspired by the principles laid out above, we are developinga cognitive architecture for robots that follows an embod-ied view. The following sections describe some of the maincomponents of the proposed architecture.

Perception-Based MemoryGrounded in perceptual symbol based theories of cognition,and in particular that of simulators (Barsalou 1999), we pro-pose that long-term memory and concepts of objects and ac-tions are based on perceptual snapshots and reside in the var-ious perceptual systems rather than in an amodal semanticnetwork (Figure 1).

These percept activation are heavily interconnected bothdirectly and through an associative memory mechanism.Perceptions of different modalities are connected based onconcurrency with other perceptual snapshots, as well as con-currency with perceptions generated by simulators duringactivation.

Bidirectional Percept Trees Incoming perceptions are fil-tered through percept trees for each modality, organizingthem on a gradient of specificity (Blumberg et al. 2002). Atthe same time, predictions stemming from more schematicperceptions bias perceptual processes closer to the sensorylevel. An example of how higher level parsing is both con-structed from and influencing lower level perception is pro-posed in a separate technical report (Hoffman 2005). In thatwork, intentions are build up gradually from motor trajec-tories, actions, and goals. Conversely, probability distribu-tions on intentions bias goal identification, which in turn setthe priors for action detection, and subsequently motion es-timation.

Retention Perceptions are retained in memory, but decayover time. Instead of subscribing to a firm division of short-and long-term memory, the proposed architecture advocates

2As reported in (Wilson 2001).

Associative Memory

Visual Perception

Visual Sensors

Visual Perception Memory

Auditory Perception

Auditory Sensors

Auditory Perception Memory

Proprioception

Motor Sensors

Proprioception Memory

. . .

. . .

weighted activation connectiondecaying perceptual memory

percept tree

Actions Concepts

Figure 1: Perception-Based Memory

a gradient information-based decay in which more specificperceptual memory decays faster, and more general percep-tual information is retained longer. Decay is governed by theamount of storage needed to keep perceptual memory at var-ious levels of specificity, retaining more specific memoriesfor a shorter period of time, and more schematic perceptionsfor a longer time span.

Retention is also influenced by the relevance of the per-ceptual information to the current task and their attentivesaliency: perceptual elements that are specifically attendedto at time of acquisition will be retained longer than onesthat were peripheral at that time. In a similar vein, affec-tive and motivational states of the system can also influencememory retention.3

For example in the visual processing pathway, raw im-ages are retained for a short period of time, while line ori-entations, colors, blob location, overall rotation, and fullyrecognized objects remain in memory for a longer time span(Figure 2).Weighted Connections Memories are organized in activa-tion networks, connecting them to each other, as well as toconcepts and actions in associative memory (the dotted linesin Figure 1). However, it is important to note that connec-tions between perceptual, conceptual, and action memoryare not binary. Instead, a single perceptual memory—forexample the sound of a cat meowing—can be more stronglyconnected to one memory—like the proprioceptive memoryof tugging at a cat’s tail—and more weakly connected to an-

3Such an approach could also explain why certain marginal per-ceptions are retained for a long time if they occurred in a traumaticsetting.

specificity

timenow

Figure 2: Perceptual memory is filtered and remains inmemory for a variable amount of time, based—amongothers—on the information at each specificity level.

other memory, such as that of one’s childhood living roomscent.

Weights of connectivity are learned and refined over time.Their value is influenced by a number of factors, such asfrequency of co-ocurrance, attention while creating the per-ceptual memory, and affective states during perception.

Simulators and Production A key capability of percep-tual memory is the production of new perceptual patternsand concepts. Using simulation, the activation of a per-ceptual symbol can evoke the construction of both an ex-perienced and a fictional situation. Citing (Barsalou 1999):“Productivity in perceptual symbol systems is approxi-mately the symbol formation process run in reverse. During

symbol formation, large amounts of information are filteredout of perceptual representations to form a schematic rep-resentation of a selected aspect. During productivity, smallamounts of the information filtered out are added back.” Inthe proposed system, the production of such memories ismade possible by running a constructive process on the tem-porally information-based filtered perceptive memories de-scribed above.

The produced perceptual activations can be used in rea-soning as well as guiding motor activity. In addition, ac-cording to the principle of top-down processing, they alsobias current perception by both affecting the filtering in thepercept tree and the parameters of the sensory modules.

Action-Perception Activation NetworksAssociative memory not only governs and interconnects per-ceptual memory, but also establishes weighted connectionsbetween concepts and actions (see: Figure 1). Inspiredby architectures such as Contention Scheduling (Cooper &Shallice 2000), activities, concepts, and perceptions occur ina perpetually updating activation relationship.

Action-Perception Activation Networks operate as fol-lows: currently acquired perceptions exert a weighted in-fluence on concepts and activities, leading to (a) potentialaction selection; (b) the simulation of other, related percep-tions; and (c) the activation of an attention mechanism whichin turn impacts the sensory layer and the perceptual task,aiding in its pertinent operation.

Thus, for example, the presentation of a screwdriver mayactivate a grasping action, as well as the activity of apply-ing a screwdriver to a screw. This could in turn activatea perceptual simulator guiding the visual search towards ascrew-like object, and the motor memory related to a clock-wise rotation of the wrist. A key point to notice regardingthese networks is that they don’t follow a single line of ac-tion production, but instead activate a number of interrelatedmechanisms which can — if exceeding a certain threshold— activate a particular motor pattern.

Bregler demonstrated a convincing application of thiskind of top-bottom feedback loop in the realm of MachineVision action recognition (Bregler 1997), and others arguedfor a symbolic task-level integration of schemas, objects,and resources (Cooper & Shallice 2000). In this work, weaim to span this approach across both action selection andperception.

Intentions, Motivations, and SupervisionThe above architecture is predominantly suited for govern-ing automatic or routine activity. While this model can beadequate for well-established behavior, a robot acting jointlywith a human must also behave in concordance with internalmotivations and intentions, as well as higher-level supervi-sion. This is particularly important since humans naturallyassign internal states and intentions to animate and eveninanimate objects (Baldwin & Baird 2001; Dennett 1987;Malle, Moses, & Baldwin 2001). Robots acting with peoplemust therefore behave according to internal drives as wellas clearly communicate these drives to their human counter-part.

Action-Perception Activation Network

Pose Graph

Goal Recipes

Drives / Motivation

Goal-Action Planner

Action Space

Motor Graph

Learned Recipes

Learned Actions

Learned Motion Trajectories

Motor Control

Goal Arbitration

Planning

Perception

Figure 3: Motivational drives are used to form plans over-riding or swaying action selection in the Action-PerceptionActivation Network.

It thus makes sense to embellish the perception-actionsubsystem with supervisory intention- and motivation-basedcontrol that affects the autonomic processing scheme out-lined above (Figure 3). At the base of this supervisorysystem lie core drives, such as hunger, boredom, attention-seeking, and domain-specific drives, modeled as scalar flu-ents, which the agent seeks to maintain at an optimal level(similar to (Breazeal 2002)). If any of those fluents fallsabove or below the defined range, they trigger a goal re-quest. The number and identity of the agent’s motivationaldrives are fixed. Goals are represented as a vector indexedto the various motivational drives, and encoding the effectof achieving a goal on each of the drives. This representa-tion can be used by the Goal Arbitration Module to decidewhat goal to pursue. The relative deviance of the motiva-tional drive from its optimal value in combination with theexpected effect can be used to compute a utility function forselecting each goal. The most useful goal according to thisfunction is thus selected.

Once a goal is selected, a planner is used to con-struct a plan that satisfies the goal. The planner usesrecipes for achieving a goal in a STRIPS-like pre-condition/action/effect representation. If the planner has arecipe that achieves a certain goal (i.e. a hierarchical set ofactions from its action space with the overall effect of sat-isfying the goal), it will attempt to use this recipe, whilemonitoring the perceptual input to evaluate goal satisfaction(in a similar manner as described in (Hoffman & Breazeal2004)). Some recipes may be innate, although none need to

be. Given a set of actions and a set of goals, the planner canbuild up new recipes by searching the action space using, forexample, regression planning (Russell & Norvig 2002).

The action space is modeled as a set of STRIPS-likeprecondition/action/effect actions, and their translation intopaths into motor graph(s). The action space-motor graphrelationship is modeled as described in (Blumberg et al.2002).

Actions that are selected based on the above approach in-fluence, and in some cases override, the automatic action se-lection mechanisms in the action-perception activation net-work. Once an action is activated in such a way, it also trig-gers simulators and guides perception as described above.

This motivation-goal-action model holds an additionalbenefit for joint action, as it can be used as a basis for anexperienced-based intention reading framework, as we de-scribed in a separate technical report (Hoffman 2005).

Joint Action DerivativesThe approach presented in this paper aims to allow fora number of important cognitive mechanisms that underlyfluid human-robot co-acting. We plan to investigate thesederivatives under the proposed approach:Attention The generic “perception problem” is quite in-

tractable. Humans (and probably other animals) use at-tention systems to filter incoming data, and attention hasalso proved useful for robotic agents (Breazeal & Scas-sellati 1999). In addition, mutual attention is a key fac-tor in the successful behavior of a human-robot team. Inthe framework advocated herein, attention emerges fromthe interaction between actions, concepts, and perceptualprocesses, and in a top-down fashion biases sensation andperception. Attention also determines learning in the sys-tem, by affecting the weighted ties between elements inperceptual memory as well as between perceptional mem-ory and concepts.

Routine Action Performing a well-known task is likely toinvolve non-conscious and sub-planning action selectionmechanisms. In humans, routine action performance isoften found to be faster and more robust than consciousor planned activity. Teams and ensembles working to-gether increasingly rely on routine action for action coor-dination, and these become more emphasized when tightmeshing of physical activity is required. We believe thattrue fluidity in human-robot collaboration can only beachieved through reliance on these “privileged loops” be-tween perception and action.

Practice In direct relation to routine activity, practice is animportant facet of joint action, and of any kind of per-formance art. The mechanisms underlying repeated prac-tice of a task, and in particular joint practice to achievefluid collaboration are not well understood. We believethat the above architecture allows for a novel investiga-tion into concepts of practice and refinement of routinejoint activities.

Flexible Resolution Cognition often appears to conduct it-self on a ‘need-to-know’ or ‘need-to-do’ basis. It can be

argued that planning and adjustment of trained conceptsand action schemas only occurs when well-practiced rou-tine activation models fail. Similarly, planning, while in-formed by lower-level perceptual phenomena, does notneed to delve into percept-level resolution unless more in-formation than available is needed. Perception and actioncan rely on existing patterns and routine simulations un-less there is a significant mismatch between the expectedand the observed. An embodied approach spanning bothlow-level processing and high-level goal-directed actioncould allow for the flexibility of cognition needed forhuman-robot joint action.

Anticipatory Action Among other factors, successful co-ordinated action has been linked to the formation of ex-pectations of each partner’s actions by the other (Flanagan& Johansson 2003), and the subsequent acting on theseexpectations (Knoblich & Jordan 2003). We have pre-sented initial work aimed at understanding possible mech-anisms to achieve this behavior in a collaborative robot(Hoffman & Breazeal 2006). A further research goal ofthis work to evaluate whether more fluid anticipatory ac-tion can emerge from an approach that uses perception-action activation as its underlying principle.

Mutual Responsiveness Finally, a central goal of thiswork is to achieve appropriate mutual responsiveness be-tween a robot and a human. Mutual responsiveness hasbeen strongly tied to joint action (Bratman 1992). By re-acting to external stimuli in a combined autonomic andsupervised fashion, we hope to be able to demonstratetruly fluid mutual responsiveness as a result of the workproposed herein.

DiscussionSurprisingly, while the existence and complexity of jointaction has been acknowledged for decades, and its behav-ioral operation has been investigated extensively, the neuro-cognitive mechanisms underlying shared activity have onlyreceived sparse attention, predominantly over the last fewyears (for a review, see (Sebanz, Bekkering, & Knoblich2006)). Much of the work in this field exploring the cogni-tive activity of two humans sharing a task builds on embod-ied structures of cognition, and emphasizes the non-verbalaspects of collaboration. It is our belief that this approach isthe most promising for robots working together with people,as well.

Notions of embodiment are not new in robotics research.It has been fifteen years since Brooks postulated that the“fundamental decomposition of the intelligent system is notinto independent information processing units which mustinterface with each other via representations. Instead, theintelligent system is decomposed into independent and par-allel activity producers which all interface directly to theworld through perception and action, rather than interfaceto each other particularly much.” (Brooks 1991) In this con-text, Brooks advocated incremental design of simple reac-tive robots. Today, it seems that embodied approaches arestill mostly confined to simple robots interacting with aninanimate environment.

Designers of robots interacting with humans, however,have taken little note of theories of embodied cognition,with most systems translating perception into amodal sym-bols for further processing. While active perception hasbeen investigated (Aloimonos 1993; Breazeal & Scassel-lati 1999), and top-down influences in object recognitionhave been explored on low-level vision tasks (Bregler 1997;Hamdan, Heitz, & Thoraval 1999), a full human-robot inter-action system taking advantage of the ideas brought forth inthis paper has yet to be implemented.

ConclusionTo date, most robots interacting with humans still operatein an non-fluid command-and-response manner. In addi-tion, most are based on an amodal model of cognition, em-ploying relatively little embodied intelligence. In the mean-time, neuroscience and psychology have gathered impres-sive evidence for a view of cognition that is based on ourphysical experience, showing that concept representation isperception- and action-based; that perception and action arecoupled not only through supervisory control; and that intel-ligence is top-down as much as it is bottom-up.

We believe that these mechanisms are at the core of jointaction. To build robots that display a continuous fluid mesh-ing of their actions with those of a human, designers ofrobots for human interaction need therefore take these in-sights to heart. We have presented a framework that im-plements the above ideas and discussed possible behavioralderivatives crucial to fluid joint activity emerging from theproposed architecture.

AcknowledgmentsThis work is funded in part by the Office for Naval ResearchYIP Grant N000140510623.

ReferencesAloimonos, Y., ed. 1993. Active Perception. LawrenceErlbaum Associates, Inc.Baldwin, D., and Baird, J. 2001. Discerning intentionsin dynamic human action. Trends in Cognitive Sciences5(4):171–178.Barsalou, L. W.; Niedenthal, P. M.; Barbey, A.; and Rup-pert, J. 2003. Social embodiment. The Psychology ofLearning and Motivation 43:43–92.Barsalou, L. 1999. Perceptual symbol systems. Behavioraland Brain Sciences 22:577–660.Blumberg, B.; Downie, M.; Ivanov, Y.; Berlin, M.; John-son, M. P.; and Tomlinson, B. 2002. Integrated learningfor interactive synthetic characters. In SIGGRAPH ’02:Proceedings of the 29th annual conference on Computergraphics and interactive techniques, 417–426. ACM Press.Boal, A. 2002. Games for Actors and Non-Actors. Rout-ledge, 2nd edition.Bratman, M. 1992. Shared cooperative activity. The Philo-sophical Review 101(2):327–341.

Breazeal, C., and Scassellati, B. 1999. A context-dependent attention system for a social robot. In IJCAI ’99:Proceedings of the Sixteenth International Joint Confer-ence on Artificial Intelligence, 1146–1153. Morgan Kauf-mann Publishers Inc.Breazeal, C. 2002. Designing Sociable Robots. MIT Press.Bregler, C. 1997. Learning and recognizing human dynam-ics in video sequences. In CVPR ’97: Proceedings of the1997 Conference on Computer Vision and Pattern Recog-nition, 568. IEEE Computer Society.Brooks, R. A. 1991. Intelligence without representation.Artificial Intelligence 47:139–159.Chovil, N. 1992. Discourse-oriented facial displays in con-versation. Research on Language and Social Interaction25:163–194.Cooper, R., and Shallice, T. 2000. Contention schedulingand the control of routing activities. Cognitive Neuropsy-chology 17:297–338.Dennett, D. C. 1987. Three kinds of intentional psy-chology. In The Intentional Stance. Cambridge, MA: MITPress. chapter 3.Flanagan, J. R., and Johansson, R. S. 2003. Action plansused in action observation. Nature 424(6950):769–771.Gallese, V., and Goldman, A. 1996. Mirror neurons and thesimulation theory of mind-reading. Brain 2(12):493–501.Gallese, V.; Fadiga, L.; Fogassi, L.; and Rizzolatti, G.1996. Action recognition in the premotor cortex. Brain119:593–609.Hamdan, R.; Heitz, F.; and Thoraval, L. 1999. Gesture lo-calization and recognition using probabilistic visual learn-ing. In Proceedings of the 1999 Conference on ComputerVision and Pattern Recognition (CVPR ’99), 2098–2103.Hoffman, G., and Breazeal, C. 2004. Collaboration inhuman-robot teams. In Proc. of the AIAA 1st IntelligentSystems Technical Conference. Chicago, IL, USA: AIAA.Hoffman, G., and Breazeal, C. 2006. What lies ahead?expectation management in human-robot collaboration. InWorking notes of the AAAI Spring Symposium.Hoffman, G. 2005. An experience-based framework forintention-reading. Technical report, MIT Media Labora-tory, Cambridge, MA, USA.Knoblich, G., and Jordan, J. S. 2003. Action coordinationin groups and individuals: learning anticipatory control.Journal of Experimental Psychology: Learning, Memory,and Cognition 29(5):1006–1016.Kosslyn, S. 1995. Mental imagery. In Kosslyn, S., andD.N.Osherson., eds., Invitation to Cognitive Science: Vi-sual Cognition, volume 2. Cambridge, MA: MIT Press,2nd edition. chapter 7, 276–296.Krauss, R. M.; Chen, Y.; and Chawla, P. 1996. Non-verbal behavior and nonverbal communication: What doconversational hand gestures tell us? In Zanna, M., ed.,Advances in experimental social psychology. Tampa: Aca-demic Press. 389–450.

Kreiman, G.; Koch, C.; and Fried, I. 2000. Imagery neu-rons in the human brain. Nature 408:357–361.Lakoff, G., and Johnson, M. 1999. Philosophy in the flesh: the embodied mind and its challenge to Western thought.New York: Basic Books.Locke, J. L., and Kutz, K. J. 1975. Memory for speechand speech for memory. Journal of Speech and HearingResearch 18:176–191.Malle, B.; Moses, L.; and Baldwin, D., eds. 2001. Inten-tions and Intentionality. MIT Press.Massaro, D., and Cohen, M. M. 1983. Evaluation and in-tegration of visual and auditory information in speech per-ception. Journal of Experimental Psychology: Human Per-ception and Performance 9(5):753–771.Matarić, M. J. 2002. Sensory-motor primitives as a ba-sis for imitation: linking perception to action and biologyto robotics. In Dautenhahn, K., and Nehaniv, C. L., eds.,Imitation in animals and artifacts. MIT Press. chapter 15,391–422.McClave, E. 2000. Linguistic functions of head move-ments in the context of speech. Journal of Pragmatics32:855–878.McLeod, P., and Posner, M. I. 1984. Privileged loops frompercept to act. In Bouma, H., and Bouwhuis, D. G., eds.,Attention and performance, volume 10. Hillsdale, NJ: Erl-baum. 55–66.Meltzoff, A. N., and Moore, M. K. 1997. Explaining fa-cial imitation: a theoretical model. Early Development andParenting 6:179–192.Parsons, L. M. 1994. Temporal and kinematic propertiesof motor behavior reflected in mentally simulated action.Journal of Experimental Psychology: Human Perceptionand Performance 20(4):709–730.Pecher, D., and Zwaan, R. A., eds. 2005. Grounding cog-nition: the role of perception and action in memory, lan-guage, and thinking. Cambridge, UK: Cambridge Univ.Press.Russell, S., and Norvig, P. 2002. Artificial Intelligence: AModern Approach (2nd edition). Prentice Hall.Sebanz, N.; Bekkering, H.; and Knoblich, G. 2006. Jo-ing action: bodies and minds moving together. Trend inCognitive Sciences 10(2):70–76.Somerville, J. A.; Woodward, A. L.; and Needham, A.2004. Action experience alters 3-month-old infants’ per-ception of others actions. Cognition.Spivey, M. J.; Richardson, D. C.; and Gonzalez-Marquez,M. 2005. On the perceptual-motor and image-schematicinfrastructure of language. In Pecher, D., and Zwaan, R. A.,eds., Grounding cognition: the role of perception and ac-tion in memory, language, and thinking. Cambridge, UK:Cambridge Univ. Press.Stanfield, R., and Zwaan, R. 2001. The effect of impliedorientation derived from verbal context on picture recogni-tion. Psychological Science 12:153–156.

Wilson, M., and Knoblich, G. 2005. The case for mo-tor involvement in perceiving conspecifics. PsychologicalBulletin 131:460–473.Wilson, M. 2001. Perceiving imitable stimuli: conse-quences of isomorphism between input and output. Psy-chological Bulletin 127(4):543–553.Wilson, M. 2002. Six views of embodied cognition. Psy-chonomic Bulletin & Review 9(4):625–636.