1

Direct and Indirect Communicationin Multi-Human Multi-Robot Interaction

Jayam Patel Student Member, IEEE, Tyagaraja Ramaswamy, Zhi Li, and Carlo Pinciroli Member, IEEE

Abstract—How can multiple humans interact with multiplerobots? The goal of our research is to create an effective interfacethat allows multiple operators to collaboratively control teams ofrobots in complex tasks. In this paper, we focus on a key aspectthat affects our exploration of the design space of human-robotinterfaces — inter-human communication. More specifically,we study the impact of direct and indirect communication onseveral metrics, such as awareness, workload, trust, and interfaceusability. In our experiments, the participants can engage directlythrough verbal communication, or indirectly by representingtheir actions and intentions through our interface. We reportthe results of a user study based on a collective transport taskinvolving 18 human subjects and 9 robots. Our study suggeststhat combining both direct and indirect communication is the bestapproach for effective multi-human / multi-robot interaction.

Index Terms—Multi-human multi-robot interaction, multi-robot systems

I. INTRODUCTION

Multi-robot systems are envisioned in scenarios in whichparallelism, scalability, and resilience are key features forsuccess. Scenarios such as firefighting, search-and-rescue,construction, and planetary exploration are typically imaginedwith teams of robots acting intelligently and autonomously[1]. However, autonomy is only part of the picture — humansupervision remains necessary to assess the correct progress ofthe mission, modify goals, and intervene in case of unexpectedfaults that cannot be handled autonomously [2].

Significant work has been devoted to interfaces that allowsingle human operators to interact with multiple robots. Theseinterfaces typically enable intuitive interaction for specifictasks, such as navigation [3], dispersion [4], and foraging [5].As the complexity of the tasks involved increases, however, theamount of information, the number of robots, and the natureof the interactions is likely to exceed the span of apprehensionof any individual operator [6]. For this reason, it is reasonableto envision that, in future missions with multi-robot systems,multiple operators will be involved.

One immediate advantage of multiple operators is the oppor-tunity to partition a large autonomous system in smaller, moremanageable parts, each assigned to a dedicated operator. Amore interesting insight is that cooperative supervisory controlcan exploit individual differences among operators. Theseinclude diversity in cognitive abilities and in the way operatorsinteract with automation. In this context, the main factors are(i) the ability to focus and shift attention flexibly, (ii) the ability

The authors are with the Department of Robotics Engineering, Worces-ter Polytechnic Institute, MA, USA. Email: {jupatel, tramaswamy,zli11, cpinciroli}@wpi.edu

to process spatial information, and (iii) prior experience withgaming interfaces [7]. Cooperative supervision that embracesthese factors will be extremely effective.

However, the presence of multiple operators introduces newand interesting challenges. These include ineffective coopera-tion among the operators [8], unbalanced workload [9], [10],and inhomogeneous awareness [11], [12], [13]. Neglectingthese challenges produces an undesirable phenomenon: theout-of-the-loop (OOTL) performance problem, caused by alack of engagement in the task at hand, of awareness of itsstate, and of trust in the system and other operators [14], [15].

In this paper, we study the impact of inter-human communi-cation on several metrics, such as awareness, workload, trust,and interface usability. Although inter-human communicationhas been studied extensively [16], it has sparsely been investi-gated in the context of multi-robot systems, and, to the best ofour knowledge, no study exists that focuses on mobile teamsof robots.

For the purposes of our work, we categorize communica-tion in two broad types: direct and indirect. In our exper-iments, direct communication includes verbal and gesture-based modalities, and we define it as an explicit exchangeof information among human operators. In contrast, we defineindirect communication as an exchange of information that oc-curs implicitly, through the mediation of, e.g., a graphical userinterface. Which elements of the graphical user interface fosterefficient indirect communication is a key research question westudy in this paper.

This work offers two main contributions:

1) From the technological point of view, we present agraphical user interface for multi-robot control, withfeatures designed to support information-rich indirectcommunication among human operators. To the best ofour knowledge, this interface is the first of its kind, in thatit enables effective interaction between multiple humansand multiple mobile robots in a complex scenario;

2) From the scientific point of view, we study the impactthat direct and indirect communication have both in thedesign of novel interfaces and in the performance of theinteraction these interfaces produce.

Our experimental evaluation is based on a study that involved18 operators, grouped in teams of 2, and 9 real mobile robotsinvolved in an object transport scenario. This paper takesadvantage of our previous work on multi-granularity [17] inter-face design, in which we showed which interaction modalities[18] and which graphical elements of a user interface [19]provide the best performance in a multi-operator setting.

arX

iv:2

102.

0067

2v1

[cs

.RO

] 1

Feb

202

1

2

The paper is organized as follows. In Sec. II we discussrelevant literature on human-robot communication. In Sec. IIIwe present the design of our interface. In Sec. IV we introducethe user study, and discuss our main findings in Sec. V. Weconclude the paper in Sec. VI.

II. BACKGROUND

Granularity of control. Over the last two decades, HRI re-search on mobile multi-robot systems has focused on identify-ing suitable interfaces and investigating methods to effectivelyinteract the robots. One of the key aspects of these interfacesis control granularity [2], [20], and it generally includesrobot-oriented, team-oriented, and environment-oriented con-trol. In robot-oriented control, the operator interacts with asingle robot, either statically predetermined or dynamicallyselected [21], [22], [23], [24], [25]. Team-oriented controlallows an operator to assign a goal to a groups of robots as ifthe group were a single entity [26], [27], [28]. Environment-oriented control occurs when the operators modifies the en-vironment, typically using augmented or mixed reality, toinfluence the behavior of the robots or to indicate goals [29],[2], [17], [18], [19]. These control modalities are typicallystudied with a single operator in mind.

Human-robot communication. Research on communica-tion has mostly focused on the relationship between a humanoperator and one or more robots. The distinction we considerin this paper between direct and indirect communication isa common aspect of human-robot communication modali-ties [30], [31]. Notable works include direct human-robot com-munication through natural language conveyed verbally [32]and non-verbal communication through social cues [33], facialexpressions [34] and eye gaze [35], [36]. Lakhmani et al.[37] presented a method for direct communication betweenan operator and a robot in which communication can beeither directional or bidirectional. In directional communi-cation, only the robot can send information to the operator.In bidirectional communication, the operator and the robotcan send information to each other. The authors reported thatthe operator performed better with directional communicationas compared to bidirectional communication. Indirect com-munication has been studied in several contexts, including:(i) a human operator attempting to infer the behavior ofthe robots [38], [39], and (ii) robots attempting to predictthe actions of the human-in-the-loop [40], [41], [42], [43].Che et al. [44], compare the effects of direct and indirectcommunication between a human and a robot in a navigationtask. In this work, the human acts as a bystander and therobot must navigate around the human. The robot can eitherindirectly predict the human’s direction of movement andnavigate around, or can directly notify the human about itsintentions of moving in a direction. The authors reportedthat the combination of indirect and direct communicationpositively impacts the humans performance and trust.

Conflicts among multiple operators. To the best of ourknowledge, analogous studies in communication that focuson multiple human operators are currently missing. The mostnotable works in multi-human multi-robot interaction concern

inter-operator conflict in scenarios in which the operators donot communicate. A conflict might arise when multiple oper-ators wish to control the same robot or specify incompatiblegoals. If communication is not possible, pre-assigning robotsamong operators is a possible solution. Lee et al. [45] comparethe performance of two scenarios: in one, the operators controlrobots from a shared pool of robots; and the other, the oper-ators have disjoint assigned pools. The operators can engagewith robots in a robot-oriented fashion and manipulate onerobot at a time. Lee et al.’s findings show that the performanceof the operators is better when they can manipulate robots fromthe assigned pools. Their performance drops when they controlrobots from the shared pool, due to the risk of specifyingconflicting goals for the same robots. In a similar vein, Youet al. [46] study a scenario in which two operators controlseparate robots, and with them push physical objects from oneplace to another. You et al. divide the environment into tworegions, with a operator-robot pair assigned to each region andlimited the operator from moving from one region to another.No conflicts are possible in this case.

Novelty. In this paper, we take inspiration from the threeresearch strands discussed above, and combine them in acoherent study. We investigate whether an interface that offersmultiple granularity of control can enable the operators todecide when and how to control the robots, and study therole that communication has in the design of such interfaceand in the way the operators collaborate. To the best ofour knowledge, this study is the first to study these researchquestions together.

III. MULTI-HUMAN MULTI-ROBOT INTERACTION SYSTEM

A. System Overview

Our overall system, schematized in Fig. 1, consists of amixed-reality (MR) interface, a team of 9 Khepera IV robots,1

a Vicon motion tracking system,2 and ARGoS [47], a fastand modular multi-robot simulator. The MR interface enablesan operator to interact with the robots, displaying usefulinformation on actions and intentions of robots and otheroperators. The Khepera IV are differential-drive robots thatwe programmed to perform behaviors such as navigation andcollective object transport. ARGoS acts as the software gluebetween the MR interfaces, the robots, and the Vicon.

To this aim, we modified ARGoS by replacing its simulatedphysics engine with a new plug-in that receives positional datafrom the motion capture systems. In addition, we created plug-ins that allow ARGoS to communicate directly with the robotsfor control and debugging purposes. These plug-ins create afeedback loop between the robots and the environment whichallows us to programmatically control what the robots do andsense as an experiment is running. Thanks to this featureof our setup, the MR interfaces, once connected to ARGoS,can both retrieve data and convey modifications made usingthe control modalities offered by the interfaces. To make thispossible, ARGoS also translates the coordinates seen by theMR interface with those seen by the Vicon and viceversa.

1https://www.k-team.com/khepera-iv2http://vicon.com

3

Fig. 1: System overview.

When an operator defines a new goal position for the robotsor the objects present in the system, the MR interface transmitsthe goal to ARGoS. The latter converts the request into motionprimitives and transmits them to the robots. Finally, the robotsexecute the primitives. At any time during the execution, theoperators can interfere with the system by defining new goalsfor the robots. This feature is particularly important whenrobots temporarily misbehave due to faults or noise.

B. Operator Interface and Control Modalities

The interaction between the operators and the robots hap-pens through the MR interface installed on an Apple iPad. Weimplemented the MR interface with Vuforia,3 a software de-velopment kit for mixed-reality applications. Vuforia providesa functionality to recognize and track physical objects withfiducial markers. We integrated Vuforia with the Unity GameEngine4 for visualization. Fig. 2 shows a screenshot of theview through the app running on an iPad.

The MR interface recognizes objects and robots throughunique fiducial markers applied to the each entity of interest.The interface overlays virtual objects on the recognized fidu-cial markers. The operator can move these virtual objects usinga one-finger swipe and rotate them with a two-finger twist. Theoperator can also select multiple robots (for, e.g., collectivemotion) by drawing a closed contour. The manipulation ofrobots and objects occurs in three phase: start, move, andend. The start phase is initiated when the operator touches thehandheld device. Once the touch is detected, the move phase

3http://vuforia.com4http://unity3d.com

Fig. 2: Screenshot of the MR interface running on an iPad. Theblack arrow was added in this picture to indicate the positionof the origin marker. This marker is used by Vuforia to identifythe origin of the environment.

(a) Object recognition (b) New Goal Defined

(c) Robots approach and push (d) Transport complete

Fig. 3: Object manipulation by interaction with the virtualobject through the interface. The dotted black arrow indicatesthe one-finger swipe gesture used to move the virtual objectand the red dotted arrow indicates the two-finger rotationgesture.

is executed as long as the operator is performing a continuousmotion gesture on the device. The end phase is triggeredwhen the operator releases the touch. After the completionof the end phase, the interface parses the gesture and actsaccordingly, e.g., sends the final pose of a selected object toARGoS. The interface enables three interaction modalities:object-oriented (a special case of the environment-orientedmodality), robot-oriented, and team-oriented, explained in therest of this section.

Object-oriented Interaction. The interface overlays a vir-tual cuboid on objects equipped with a fiducial marker (see

4

(a) Robot recognition (b) New robot position

Fig. 4: Robot manipulation by interaction with the virtualrobots through the interface. The dotted black arrow indicatesthe one-finger swipe gesture to move the virtual robot and thearrowhead color indicates the moved virtual robot.

Fig. 3a). The dimensions and textures of the virtual objectsmatch the dimensions and textures of the fiducial markers. Theoperators can differentiate between virtual cuboids using thesesimilarities while simultaneously moving multiple objects. Theoperators can move these virtual objects using a one-fingerswipe and rotate it with a two-finger twist (see Fig. 3b).If two or more operators simultaneously want to move thesame virtual object, then the robots transport the object to thelast received position. Fig. 3c and Fig. 3d show the robotstransporting the object.

Robot-oriented Interaction. The interface overlays a vir-tual cylinder on the recognized robot tags (see Fig. 4a). Thedimensions and colors of the virtual cylinder are identicalto the dimensions and colors of the fiducial marker on therobot. Operators can differentiate between different virtualcylinders using these colors while controlling multiple robotsat once. The operators can use a one-finger swipe to move thevirtual cylinders to denote a desired position for the robots(see Fig. 4b). If multiple operators simultaneously want tomove the virtual cylinder, then the robot considers the lastreceived request. If the robot is performing collective objecttransport, then other robots in the team pause their operationuntil the selected robot reaches the desired position. If therobot is part of an operator-defined team, then the other robotsare not affected by the position change. The robot control canalso be used to manually push the object.

Robot Team Selection and Manipulation. Operators candraw a closed contour with a one-finger continuous swipe toselect robots enclosed in the shape (see Fig. 5a). The interfacedraws the closed contour in red color to show the area ofselection (see Fig. 5b). The interface displays a cube, hoveringat the centroid of the created contour, to represent the teamof robots (see Fig. 5b). The operators can move this cubewith a one-finger swipe to define a desired location for theteam to navigate (see Fig. 5d and Fig. 5d). Each operatorcan create only one custom team at a time and the interfacedeletes all previous selections. If a robot is part of multipleteams defined by multiple operators, then the robot moves tothe latest location.

C. Collective TransportWe used a finite state machine to implement our collective

transport behaviour. The finite state machine is executed

(a) Robot team selection (b) Robot team creation

(c) Robot team manipulation (d) Robot team re-positioned

Fig. 5: Robot team creation and manipulation. The dottedblack arrow indicates the one-finger swipe gesture to movethe virtual cube for re-positioning the team of robots.

Fig. 6: Collective transport state machine.

independently by each robot, using information from the Viconand the MR mediated by ARGoS. Fig. 6 shows the statemachine, along with the conditions for state transition. Wedefine the states as follows:

Reach Object. The robots navigate to the object uponreceiving a desired position from the operator. The robotsarrange themselves around the object. The state ends whenall the robots reach their chosen positions.

Approach Object. The robots face the centroid of the objectand start moving towards it. The state ends when all the robotsare in contact with the object.

Push Object. The robots orient themselves in the directionof the goal position and start moving when all robots are facingthat direction. The robot in the front of the formation maintainsa fixed distance from the object allowing all the robots to stayin the formation. The state transitions to Approach Object if

5

Fig. 7: Transparency features in the mixed-reality interface.The black arrow indicates the text-based log. The red arrowindicates the Robot Panel. The green arrow indicates the on-robot status. The orange arrow indicates the object panel.

Fig. 8: Shared awareness demonstrated using four mixed-reality interfaces.

the formation breaks. The state ends when the object reachesthe desired position.

Rotate Object. The robots arrange themselves around theobject and move in a circular direction to rotate the object.The state transitions to Approach Object if the formationbreaks. The state ends when the object achieves the desiredorientation.

D. Information Transparency

We designed our interface to provide a rich set of data aboutthe robots, the task progress, and other operators’ actions.We studied the effectiveness of these features in previouswork [19], assessing how they contribute to making the systemmore transparent [48] for the operators. The interface is shownin Fig. 7.

To foster shared awareness, when an operator manipulates avirtual object, the action is broadcast to the other interfaces in

real-time, making it possible for every operator to see what isbeing done by other operators. Fig. 8 shows the image of fourmixed-reality interfaces demonstrating this shared awarenessfeature.

The information is organized into a set of panels, updatedin real time whenever robots and operators perform actionsthat modify the state of the system and progress of the tasksin execution. The interface offers the following informationwidgets:

• Small ‘on-robot’ panels that follow each robot in view toconvey their current state and actions. These panels areupdated when operators assign new tasks to the robots. Incase of faults, this panel displays an error message thathelps to identify which robot has malfunctioned.

• A robot panel (on the left side of the screen) thatgraphically indicates functional and faulty robots. Therobots currently engaged in a task are displayed withspecific color shades. The panel also displays a blinkingred exclamation point in case a robot acts in a faultymanner.

• A text-based log (on the top-left of the screen) thatnotifies the operators about other operators’ interactionswith robots and objects. The interface updates the logevery time other operators manipulate any virtual object.The log stores the last three actions and discards the rest.

• An object panel (on the right side of the screen) thatprovides information regarding the objects that are beingmanipulated the interface user and the other operators.The panel also highlights the object icon correspondingto the object being moved by the robots. Additionally, theinterface offers the option of selecting an object icon tolock it for future use. An operator can lock the object bytapping the lock icon. This changes the color of the lockto blue, signifying the lock. An operator can select onlyone object at a time. The interface of the other operatorshighlights the lock with a red icon.

IV. USER STUDY

A. Communication Modes and Hypotheses

This study aims to investigate the effects of direct andindirect communication on multi-human multi-robot interac-tion. The consider three modalities of communication: direct,indirect, and mixed (the combination of direct and indirect).We based our experiments on three main hypotheses.H1: Mixed communication (MC) has the best outcome com-

pared to the other communication modalities in termsof situational awareness, trust, interaction score and taskload.

H2: Operators prefer mixed communication (MC) to the othermodes.

H3: Operators prefer direct communication (DC) over indirectcommunication (IC).

B. Task Description

We designed a gamified scenario in which the operatorsmust instruct the robots to transport objects to target areas.

6

Fig. 9: User study experiment setup.

The environment had 6 objects (2 big objects and 4 smallobjects), and the operators received points according to whichobjects were successfully transported to their target. The bigobjects were worth 2 points each and the small objects were1 point each, for a total of 8 points.

The operators, in teams of 2, had a maximum of 8 minutesto accrue as many points as possible. The operators couldmanipulate big objects with any kind of interaction modality(object-oriented, team-oriented, robot-oriented); small objects,in contrast, could only be manipulated with the robot- andteam-oriented modalities. The operators were given 9 robotsto complete the game.

Fig. 9 shows the initial positions of the objects and therobots. All the participants had to perform the task four timeswith a different communication modalities as follows:

• No Communication (NC): The operators are not allowedto communicate in any way;

• Direct Communication (DC): The operators can com-municate verbally and non-verbally, but the interfaces donot broadcast information.

• Indirect Communication (IC): The operators can com-municate indirectly through the transparency features ofthe interface, but are not allowed to talk, use gestures, orinfer from body language what other operators are doing;

• Mixed Communication (MC): The complete set ofcommunication modalities is allowed for both operators.

C. Participant SampleWe recruited 18 students from our university (11 male, 7

females) with ages ranging from 20 to 30 (M = 24.17, SD =2.68). No participant had prior experience with our interface,our robots, or even the laboratory layout.

D. ProceduresEach session lasted approximately 105 minutes. We ex-

plained the study to the participants after signing a consentform, and gave the participants 10 minutes to play with theinterface. We randomized the order of the tasks in an attemptto reduce the influence of learning effects.

• Did you understand your teammate’s intentions? Wereyou able to understand why your teammate was taking acertain action?• Could you understand your teammate’s actions? Couldyou understand what your teammate was doing at anyparticular time?• Could you follow the progress of the task? Whileperforming the tasks, were you able to gauge how muchof it was pending?• Did you understand what the robots were doing? Atall times, were you sure how and why the robots werebehaving the way they did?• Was the information provided by the interface clear tounderstand?

Fig. 10: Questionnaire on the effect of the inter-operatorcommunication modalities we considered in our user study.

E. Metrics

We recorded subjective measures from the operators andobjective measures using ARGoS for each game. We used thefollowing scales as metrics:

• Situational Awareness: We measured situational aware-ness using the Situational Awareness Rating Technique(SART) [49] on a 4-point Likert scale [50];

• Task Workload: We used the NASA TLX [51] scale ona 4-point Likert scale to compare the perceived workloadin each task;

• Trust: We employed the trust questionnaire [52] on a4-point Likert scale to assess trust in the system;

• Interaction: We used a custom questionnaire on a 5-pointLikert scale to quantify the effects of communication.The interaction questionnaire had the questions reportedin Fig. 10

• Performance: We measured the performance achievedwith each communication modality by using the pointsearned in each game.

• Usability: We asked participants to select the features(text log, robot panel, object panel, and on-robot status)they used during the study. Additionally, we asked themto rank the communication modalities from 1 to 4, 1 beingthe highest rank.

F. Results

We analyzed the data using the Friedman test [53] andsummarized the results based on the significance and themean ranks. Table I shows the summarized results along withrelationship ranking between the communication modes. Weformed the rankings for each scale using the mean ranks of theFriedman test. We categorized the relationship as significantfor p < 0.05 and marginally significant for p < 0.10.

Fig. 11 shows the usability results, i.e., the percentageof operators that used a particular feature to complete aspecific task. Fig. 12 shows how the participants ranked thecommunication modalities.

7

TABLE I: Results with relationships between communicationmodes. The relationship are based on mean ranks obtainedthrough Friedman’s Test. The symbol ∗ denotes significantdifference (p < 0.05) and the symbol ∗∗ denotes marginallysignificant difference (p < 0.10). The symbol − denotesnegative scales and lower ranking is a good ranking.

Attributes Relationship χ2(3) p-valueSART SUBJECTIVE SCALE

Instability of Situation− NC>IC>DC>MC∗ 9.000 0.029

Complexity of Situation− not significant 2.324 0.508

Variability of Situation− IC>NC>MC>DC∗ 9.303 0.026

Arousal IC>NC>MC>DC∗ 6.371 0.095

Concentration of Attention IC>NC>DC>MC∗ 17.149 0.001

Spare Mental Capacity not significant 5.858 0.119

Information Quantity MC>DC>IC>NC∗ 15.075 0.002

Information Quality MC>DC>IC>NC∗ 15.005 0.002

Familiarity with Situation not significant 6.468 0.101

NASA TLX SUBJECTIVE SCALE

Mental Demand− not significant 2.226 0.527

Physical Demand− not significant 2.165 0.539

Temporal Demand− not significant 3.432 0.330

Performance− not significant 0.412 0.938

Effort− not significant 1.450 0.694

Frustration− not significant 4.454 0.216

TRUST SUBJECTIVE SCALE

Competence not significant 4.740 0.192

Predictability MC>IC>DC>NC∗ 10.626 0.014

Reliability MC>IC>DC>NC∗ 8.443 0.038

Faith MC>IC>DC>NC∗ 9.451 0.024

Overall Trust MC>IC>DC>NC∗∗ 6.633 0.085

Accuracy not significant 1.891 0.595

INTERACTION SUBJECTIVE SCALE

Teammate’s Intent DC>MC>IC>NC∗ 19.610 0.000

Teammate’s Action MC>DC>IC>NC∗ 13.810 0.003

Task Progress MC>DC>IC>NC∗ 9.686 0.021

Robot Status not significant 0.811 0.847

Information Clarity not significant 5.625 0.131

PERFORMANCE OBJECTIVE SCALE

Points Scored not significant 0.808 0.848

TABLE II: Ranking scores based on the Borda count. The graycells indicate the leading scenario for each type of ranking.

Borda Count NC DC IC MC

Based on Collected Data Ranking(Table I)

18 34 33 45

Based on Preference Data Ranking(Fig. 12)

18 49 41 72

We used the Borda count [54] to quantify all the derivedrankings based on data and preference to find an overallwinner. We also inverted the ranking of the negative scalesfor consistency. Table II shows the results.

Fig. 11: Feature Usability.

Fig. 12: Task Preference.

V. DISCUSSION

Table I shows that mixed communication (MC) has the bestinformation quality and quantity, leading to the best awarenessand trust. However, with more information, the operatorsexperienced greater instability of situation and variability ofsituation (see the SART Subjective Scale section of Table I).The participants indicated mixed communication as the bestchoice in both the data and their expressed preference (seeTable II), confirming the hypotheses H1 and H2.

Direct communication (DC), compared to indirect commu-nication (IC), had better information quality and quantity, lead-

8

Fig. 13: Task performance for each communication mode.

Fig. 14: Learning effect in the user study.

ing to better awareness. However, similar to mixed communi-cation, the operators experienced greater instability of situationand variability of situation. Although trust is higher in indirectcommunication, operators prefer direct communication. TheBorda count for preference data shows direct communicationranks better than indirect communication, supporting hypoth-esis H3.

However, in the absence of direct communication, the oper-ators concentrated more on the task, leading to a higher levelof arousal and better trust (see Trust Subjective Scale sectionof Table I). Although indirect communication provides morediverse and more visually augmented information, operatorsrelied on direct communication when working as a team.

We also observed operators directly communicating eitherat the start of the task, to define a strategy, or near theend of the task, to coordinate the last part of the task. Onereason can be familiarity of information. Humans are morefamiliar with direct communication. The participants were newto the interface and were unable to use it as effectively ascommunicating directly. This raises another research questionand a potential future study on comparing the effects of mixedcommunication on novice operators and on expert operators.

Our experiments did not detect a substantial differencein performance across communication modes. However, wehypothesize that this lack of difference is because of thelearning effect across the trials each team had to perform.Fig. 13 represent the points earned for each communicationmode and Fig. 14 shows the learning effects as the increasein objective performance in order of the performed task.

VI. CONCLUSION AND FUTURE WORK

We presented a study on the effects of inter-operatorcommunication in multi-human multi-robot interaction. Weconcentrated on two broad types of communication: direct,whereby two operators engage in an exchange of informationverbally or non-verbally (e.g., with gestures, body language);and indirect, whereby information is exchanged through asuitably designed interface. We presented the design of ourinterface, which offers multi-granularity control of teams ofrobots.

In a user study involving 18 participants and 9 robots,we assessed the effect of inter-operator communication bycomparing direct, indirect, mixed, and no communication.We considered metrics such as awareness, trust, workloadand usability. Our results suggest that allowing for mixedcommunication is the best approach, because it fosters flexibleand intuitive collaboration among operators.

In future work, we will study how training affects commu-nication. In comparing direct and indirect communication, wenoticed that our novice participants preferred direct to indi-rect communication. Our experience as online videogamersleads us to hypothesize that novice users heavily lean onverbal communication due to a lack of experience with theinformation conveyed by the interface. As the experience ofthe operators increases, we expect direct communication tobecome more sporadic and high-level. At the same time, moreexpert users might request advanced features to further makeindirect communication more efficient.

ACKNOWLEDGEMENTS

This work was funded by an Amazon Research Award.

REFERENCES

[1] M. Brambilla, E. Ferrante, M. Birattari, and M. Dorigo, “Swarmrobotics: A review from the swarm engineering perspective,” SwarmIntelligence, vol. 7, no. 1, pp. 1–41, 2013.

[2] A. Kolling, P. Walker, N. Chakraborty, K. Sycara, and M. Lewis,“Human Interaction With Robot Swarms: A Survey,” IEEE Transactionson Human-Machine Systems, vol. 46, no. 1, pp. 9–26, Feb. 2016.[Online]. Available: http://ieeexplore.ieee.org/document/7299280/

9

[3] N. Ayanian, A. Spielberg, M. Arbesfeld, J. Strauss, and D. Rus,“Controlling a team of robots with a single input,” in Robotics andAutomation (ICRA), 2014 IEEE International Conference On. IEEE,2014, pp. 1755–1762.

[4] G. Kapellmann-Zafra, N. Salomons, A. Kolling, and R. Groß, “Human-Robot Swarm Interaction with Limited Situational Awareness,” in In-ternational Conference on Swarm Intelligence. Springer, 2016, pp.125–136.

[5] A. Kolling, K. Sycara, S. Nunnally, and M. Lewis, “Human SwarmInteraction: An Experimental Study of Two Types of Interactionwith Foraging Swarms,” Journal of Human-Robot Interaction, vol. 2,no. 2, Jun. 2013. [Online]. Available: http://dl.acm.org/citation.cfm?id=3109714

[6] M. Lewis, H. Wang, S. Y. Chien, P. Velagapudi, P. Scerri, and K. Sycara,“Choosing autonomy modes for multirobot search,” Human Factors,vol. 52, no. 2, pp. 225–233, 2010.

[7] J. Y. C. Chen and M. J. Barnes, “Human–Agent Teaming for MultirobotControl: A Review of Human Factors Issues,” IEEE Transactions onHuman-Machine Systems, vol. 44, no. 1, pp. 13–29, Feb. 2014.

[8] K. Allen, R. Bergin, and K. Pickar, “Exploring trust, group satisfaction,and performance in geographically dispersed and co-located universitytechnology commercialization teams,” in VentureWell. Proceedings ofOpen, the Annual Conference. National Collegiate Inventors &Innovators Alliance, 2004, p. 201.

[9] S. E. McBride, W. A. Rogers, and A. D. Fisk, “Understanding the effectof workload on automation use for younger and older adults,” Humanfactors, vol. 53, no. 6, pp. 672–686, 2011.

[10] J. Y. Chen and M. J. Barnes, “Human–agent teaming for multirobotcontrol: A review of human factors issues,” IEEE Transactions onHuman-Machine Systems, vol. 44, no. 1, pp. 13–29, 2014.

[11] J. M. Riley and M. R. Endsley, “Situation awareness in hri withcollaborating remotely piloted vehicles,” in proceedings of the HumanFactors and Ergonomics Society Annual Meeting, vol. 49, no. 3. SAGEPublications Sage CA: Los Angeles, CA, 2005, pp. 407–411.

[12] J. D. Lee, “Review of a pivotal human factors article:“humans andautomation: use, misuse, disuse, abuse”,” Human Factors, vol. 50, no. 3,pp. 404–410, 2008.

[13] R. Parasuraman and V. Riley, “Humans and automation: Use, misuse,disuse, abuse,” Human factors, vol. 39, no. 2, pp. 230–253, 1997.

[14] M. R. Endsley and E. O. Kiris, “The out-of-the-loop performanceproblem and level of control in automation,” Human factors, vol. 37,no. 2, pp. 381–394, 1995.

[15] J. Gouraud, A. Delorme, and B. Berberian, “Autopilot, mind wandering,and the out of the loop performance problem,” Frontiers in neuroscience,vol. 11, p. 541, 2017.

[16] M. Tomasello, Origins of human communication. MIT press, 2010.[17] J. Patel, Y. Xu, and C. Pinciroli, “Mixed-granularity human-swarm inter-

action,” in Robotics and Automation (ICRA), 2019 IEEE InternationalConference on. IEEE, 2019.

[18] J. Patel and C. Pinciroli, “Improving human performance using mixedgranularity of control in multi-human multi-robot interaction,” in IEEEInternational Conference on Robot and Human Interactive Communica-tion (Ro-Man 2020). IEEE Press, 2020.

[19] J. Patel, T. Ramaswamy, Z. Li, and C. Pinciroli, “Transparency in multi-human multi-robot interaction,” arXiv preprint arXiv:2101.10495, 2021.

[20] M. R. Endsley, “From here to autonomy: lessons learned from human–automation research,” Human factors, vol. 59, no. 1, pp. 5–27, 2017.

[21] T. Setter, A. Fouraker, H. Kawashima, and M. Egerstedt, “HapticInteractions With Multi-Robot Swarms Using Manipulability,” Journalof Human-Robot Interaction, vol. 4, no. 1, p. 60, Jan. 2015. [Online].Available: http://dl.acm.org/citation.cfm?id=3109839

[22] G. Kapellmann-Zafra, N. Salomons, A. Kolling, and R. Groß, “Human-Robot Swarm Interaction with Limited Situational Awareness,” inInternational Conference on Swarm Intelligence. Springer, 2016, pp.125–136. [Online]. Available: http://link.springer.com/chapter/10.1007/978-3-319-44427-7 11

[23] J. Nagi, A. Giusti, L. M. Gambardella, and G. A. Di Caro,“Human-swarm interaction using spatial gestures,” in 2014 IEEE/RSJInternational Conference on Intelligent Robots and Systems. Chicago,IL, USA: IEEE, Sep. 2014, pp. 3834–3841. [Online]. Available:http://ieeexplore.ieee.org/document/6943101/

[24] J. Alonso-Mora, S. Haegeli Lohaus, P. Leemann, R. Siegwart,and P. Beardsley, “Gesture based human - Multi-robot swarminteraction and its application to an interactive display,” in 2015 IEEEInternational Conference on Robotics and Automation (ICRA). Seattle,WA, USA: IEEE, May 2015, pp. 5948–5953. [Online]. Available:http://ieeexplore.ieee.org/document/7140033/

[25] B. Gromov, L. M. Gambardella, and G. A. Di Caro, “Wearablemulti-modal interface for human multi-robot interaction.” IEEE,Oct. 2016, pp. 240–245. [Online]. Available: http://ieeexplore.ieee.org/document/7784305/

[26] G. Podevijn, R. O’Grady, Y. S. G. Nashed, and M. Dorigo, “Gesturingat Subswarms: Towards Direct Human Control of Robot Swarms,”in Towards Autonomous Robotic Systems, A. Natraj, S. Cameron,C. Melhuish, and M. Witkowski, Eds. Berlin, Heidelberg: SpringerBerlin Heidelberg, 2014, vol. 8069, pp. 390–403. [Online]. Available:http://link.springer.com/10.1007/978-3-662-43645-5 41

[27] N. Ayanian, A. Spielberg, M. Arbesfeld, J. Strauss, and D. Rus,“Controlling a team of robots with a single input,” in Robotics andAutomation (ICRA), 2014 IEEE International Conference on. IEEE,2014, pp. 1755–1762. [Online]. Available: http://ieeexplore.ieee.org/abstract/document/6907088/

[28] Y. Diaz-Mercado, S. G. Lee, and M. Egerstedt, “Distributed dynamicdensity coverage for human-swarm interactions,” in American ControlConference (ACC), 2015. IEEE, 2015, pp. 353–358.

[29] S. Bashyal and G. K. Venayagamoorthy, “Human swarm interactionfor radiation source search and localization,” in 2008 IEEE SwarmIntelligence Symposium. IEEE, 2008, pp. 1–8.

[30] N. Mavridis, “A review of verbal and non-verbal human–robot interac-tive communication,” Robotics and Autonomous Systems, vol. 63, pp.22–35, 2015.

[31] S. Saunderson and G. Nejat, “How robots influence humans: A surveyof nonverbal communication in social human–robot interaction,” Inter-national Journal of Social Robotics, vol. 11, no. 4, pp. 575–608, 2019.

[32] Y. Bisk, D. Yuret, and D. Marcu, “Natural language communicationwith robots,” in Proceedings of the 2016 Conference of the NorthAmerican Chapter of the Association for Computational Linguistics:Human Language Technologies, 2016, pp. 751–761.

[33] F. Sartorato, L. Przybylowski, and D. K. Sarko, “Improving therapeuticoutcomes in autism spectrum disorders: Enhancing social communi-cation and sensory processing through the use of interactive robots,”Journal of psychiatric research, vol. 90, pp. 1–11, 2017.

[34] Z. Liu, M. Wu, W. Cao, L. Chen, J. Xu, R. Zhang, M. Zhou, and J. Mao,“A facial expression emotion recognition based human-robot interactionsystem,” IEEE/CAA Journal of Automatica Sinica, 2017.

[35] H. Admoni and B. Scassellati, “Social eye gaze in human-robot interac-tion: a review,” Journal of Human-Robot Interaction, vol. 6, no. 1, pp.25–63, 2017.

[36] C. Breazeal, C. D. Kidd, A. L. Thomaz, G. Hoffman, and M. Berlin,“Effects of nonverbal communication on efficiency and robustness inhuman-robot teamwork,” in 2005 IEEE/RSJ international conference onintelligent robots and systems. IEEE, 2005, pp. 708–713.

[37] S. G. Lakhmani, J. L. Wright, M. R. Schwartz, and D. Barber,“Exploring the Effect of Communication Patterns and Transparencyon Performance in a Human-Robot Team,” Proceedings of the HumanFactors and Ergonomics Society Annual Meeting, vol. 63, no. 1, pp.160–164, Nov. 2019. [Online]. Available: http://journals.sagepub.com/doi/10.1177/1071181319631054

[38] H. Knight, R. Thielstrom, and R. Simmons, “Expressive path shape(swagger): Simple features that illustrate a robot’s attitude toward its goalin real time,” in Intelligent Robots and Systems (IROS), 2016 IEEE/RSJInternational Conference on. IEEE, 2016, pp. 1475–1482. [Online].Available: http://ieeexplore.ieee.org/abstract/document/7759240/

[39] B. Capelli, C. Secchi, and L. Sabattini, “Communication ThroughMotion: Legibility of Multi-Robot Systems,” in 2019 InternationalSymposium on Multi-Robot and Multi-Agent Systems (MRS). NewBrunswick, NJ, USA: IEEE, Aug. 2019, pp. 126–132. [Online].Available: https://ieeexplore.ieee.org/document/8901100/

[40] D. Claes and K. Tuyls, “Multi robot collision avoidance in ashared workspace,” Autonomous Robots, vol. 42, no. 8, pp. 1749–1770, Dec. 2018. [Online]. Available: http://link.springer.com/10.1007/s10514-018-9726-5

[41] Y. Wang, L. R. Humphrey, Z. Liao, and H. Zheng, “Trust-basedMulti-Robot Symbolic Motion Planning with a Human-in-the-Loop,”ACM Transactions on Interactive Intelligent Systems, vol. 8, no. 4,pp. 1–33, Nov. 2018, arXiv: 1808.05120. [Online]. Available:http://arxiv.org/abs/1808.05120

[42] A. Bajcsy, S. L. Herbert, D. Fridovich-Keil, J. F. Fisac, S. Deglurkar,A. D. Dragan, and C. J. Tomlin, “A Scalable Framework For Real-TimeMulti-Robot, Multi-Human Collision Avoidance,” arXiv:1811.05929[cs], Nov. 2018, arXiv: 1811.05929. [Online]. Available: http://arxiv.org/abs/1811.05929

[43] L. Zhang and R. Vaughan, “Optimal robot selection by gaze direction inmulti-human multi-robot interaction,” in 2016 IEEE/RSJ International

10

Conference on Intelligent Robots and Systems (IROS). Daejeon,South Korea: IEEE, Oct. 2016, pp. 5077–5083. [Online]. Available:http://ieeexplore.ieee.org/document/7759745/

[44] Y. Che, A. M. Okamura, and D. Sadigh, “Efficient and trustworthy socialnavigation via explicit and implicit robot–human communication,” IEEETransactions on Robotics, 2020.

[45] P.-J. Lee, H. Wang, S.-Y. Chien, M. Lewis, P. Scerri, P. Velagapudi,K. Sycara, and B. Kane, “Teams for Teams Performance in Multi-Human/Multi-Robot Teams,” th ANNUAL MEETING, p. 5, 2010.

[46] S. You and L. E. Robert Jr., “Curiosity vs. Control: Impacts of Trainingon Performance of Teams Working with Robots,” in Proceedings ofthe 19th ACM Conference on Computer Supported Cooperative Workand Social Computing Companion - CSCW ’16 Companion. SanFrancisco, California, USA: ACM Press, 2016, pp. 449–452. [Online].Available: http://dl.acm.org/citation.cfm?doid=2818052.2869121

[47] C. Pinciroli, V. Trianni, R. O’Grady, G. Pini, A. Brutschy, M. Brambilla,N. Mathews, E. Ferrante, G. Di Caro, F. Ducatelle, M. Birattari, L. M.Gambardella, and M. Dorigo, “ARGoS: a modular, parallel, multi-enginesimulator for multi-robot systems,” Swarm Intelligence, vol. 6, no. 4, pp.271–295, 2012.

[48] A. Bhaskara, M. Skinner, and S. Loft, “Agent Transparency:A Review of Current Theory and Evidence,” IEEE Transactionson Human-Machine Systems, pp. 1–10, 2020. [Online]. Available:https://ieeexplore.ieee.org/document/8982042/

[49] R. M. Taylor, “Situational awareness rating technique (sart): The devel-opment of a tool for aircrew systems design,” in Situational awareness.Routledge, 1990, pp. 111–128.

[50] R. LIKERT, “A technique for the measurement of attitudes,”Arch Psych, vol. 140, p. 55, 1932. [Online]. Available: https://ci.nii.ac.jp/naid/10024177101/en/

[51] S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task loadindex): Results of empirical and theoretical research,” in Advances inpsychology. Elsevier, 1988, vol. 52, pp. 139–183.

[52] A. Uggirala, A. K. Gramopadhye, B. J. Melloy, and J. E. Toler, “Mea-surement of trust in complex and dynamic systems using a quantitativeapproach,” International Journal of Industrial Ergonomics, vol. 34,no. 3, pp. 175–186, 2004.

[53] M. Friedman, “The use of ranks to avoid the assumption of normalityimplicit in the analysis of variance,” Journal of the american statisticalassociation, vol. 32, no. 200, pp. 675–701, 1937.

[54] D. Black, “Partial justification of the borda count,” Public Choice, pp.1–15, 1976.