ASCARI: a component based simulator for distributedmobile robot systems

Mirko Ferrati, Centro di Ricerca ”E.Piaggio”, Dipartimento di Ingegneria dell’Informazione,Universita di Pisa, mirko.ferrati@gmail.com, Italy

Alessandro Settimi, Centro di Ricerca ”E.Piaggio”, Dipartimento di Ingegneria dell’Informazione,Universita di Pisa, alessandro.settimi@for.unipi.it, Italy

Lucia Pallottino, Centro di Ricerca ”E.Piaggio”, Dipartimento di Ingegneria dell’Informazione,Universita di Pisa, l.pallottino@centropiaggio.unipi.it, Italy

Abstract

ASCARI is a simulator dedicated to distributed and cooperative mobile robotics systems. ASCARI has beendesigned to be a generic framework for implementing and testing multi-agent collaborative algorithms, especiallysuited to evaluate algorithms performances with a non-perfect communication channel (e.g. delayed, limitedbandwidth, limited range). The design process has taken into account state-of-art robotics simulators but wasmainly driven by a complex new requirement: inter-agent communication has to be integrated in the simulationloop.The core of the project is a server with a simple dynamic engine, a synchronization facility and an API(Application Programming Interface) to control simulated communication which is used by server plugins toprovide implementations for the characteristics of the communication channel. Different channel requirementscan be added by users through the dedicated plugin. Beside the server, there are the agents involved in thesimulation.The only custom code required from the user is the agent control law.The control law has to be written as a plugin which receives sensors data and control actuators using an APIas an abstraction layer from hardware. As in every mature simulator, the abstraction layer quickly enables thedeveloper to use the desired control law directly on a real robot by changing only hardware drivers.Finally, the inter-agent communication is provided in a transparent way to the user by some templatecommunication classes. The user can exchange information between agents by simply creating sendersand receivers classes with their custom data type. Simulated communications and filters are all handledautomatically. The simulator is completed by a 2D viewer and a simple GUI (Graphical User Interface) thatallow the user to intuitively follow the simulation evolution and to start the various processes (simulator, agents,viewer), controlling the number of agents involved in and the simulation speed.In this paper we first describe the ASCARI simulator and we then validate it on a distributed traffic controland a distributed task assignment algorithm.

1 Introduction

Simulations are widely used in multi-robot systems asa validation method, yet most of the software simu-lators are usually developed to target a specific ap-plication/algorithm, thus they lack the capabilities tobecome a general and stardardized tool for researchersin this field. While, for generic robotics applications,there exist some simulators that are recognized asstandard tools, these softwares are not very suitedto multi-robot systems. Here we will briefly exam-ine the features of some of the most known roboticsimulators from the point of view of multi-agent ap-plications. Specifically, our aim is to build a systemthat allows easy testing of communication-based dis-

tributed algorithms, where the communication chan-nel and equipment can be specified by the user. Themost inspiring simulator for our work was Stage, fromthe project Player/Stage/Gazebo. The Player/Stage(P/S) [1] project is an open-source software for rapiddevelopment of robot control code. Player [3, 2] isbased on a set of simple interfaces for communicatingand controlling sensors and actuators, while Stage isa 2D simulator for multiple robots that includes asimple physical simulations. Stage uses a set of simu-lated devices that are compatible with standard Playerinterfaces. Thus the interface that is presented to theuser by Player can be used unchanged from simulationto real hardware.

For example, a program that drives a simulated laser-equipped robot in Stage will also drive a real laser-equipped robot, with no changes to the control code.

Stage software architecture was designed with multi-robot systems in mind, the resulting software is capa-ble of handling thousands of mobile agents with simpledynamics and an arbitrary control code. The main lim-itation of P/S is the communication between robots: itwas not designed for simulating a communication, andthe Wifi interface has been only declared as a possiblesensor, but it was not implemented.

We believe that not taking into account the capabilityof simulating communications among agents duringthe design of Stage makes it very hard to add thatfeature now. Anyway, the project is no longer devel-oped since some years, maybe due to the focus ofopen source robotics community on Gazebo. Gazebohas recently evolved as a standalone project integratedwith ROS, and lost its old Player interface coupling.The main difference between Gazebo and Stage is thequality and the precision of the dynamic integration:Gazebo is a 3D environment with accurate body colli-sions and joint simulations, focused on a single robothardly interacting with the environment. Since oneyear Gazebo provides an 802 wireless simulated sensorwith Hata-Okumara propagation model. This sensorcan be used as a starting point for developing othertype of wireless communication sensors. Gazebo is notsuited for thousands of robots because of the com-putational complexity of the simulation: simulatingthousands of robots in Gazebo would require too muchtime and usually researchers in distributed controlalgorithms for multi-robot systems are more interestedin fast approximated simulations than in realistic ones.The same considerations stand for similar simulatorssuch as V-rep [9], USARSim [8] and Microsoft RoboticsStudio [7].

MatSim [4] uses a totally different approach to multi-agent simulations: by ignoring agent dynamics it iscapable of simulating more than 1 millions of agentsin the same environment. It is a grown and stablesoftware easy to use, but as such, it appears complexto understand and edit in its simulating capabilities.

Webots [5] is a commercial closed-source 3D mobilerobot simulator. The robot controllers can be pro-grammed with the built-in IDE (Integrated Develop-ment Environment) or with third party developmentenvironments. The robot behavior can be tested inphysically realistic worlds. The controller programscan optionally be transferred to commercially availablereal robots. A simulated wifi-communication module isimplemented inside the simulator but, since Webots isnot expandable by the user, the communication modelcannot be changed or improved.

Argos [6] was inspired by Stage. The main differencescan be found in the simulation speed (Argos is faster)

and in the decoupling of the physics engine from thesimulated world. Argos suffers the same design prob-lem of Stage related to the possibility of simulatingcommunication channels. The user cannot influencethe simulator side, meaning that he/she can’t write acommunication module integrated with the simulatedworld. As in Stage, the wifi sensor exists but it has anempty behavior, meaning that it was not implemented.

Mason [10] is a multi-agent simulation suite writtenin Java to be portable on different operating systems.It was not built as a complete simulator, but assomething to be easily adapted by other researchersinto a custom simulator. Mason is focused on agentsin a wider sense, so it does not provide any mobilerobotics facility such as communications, sensors or arealistic dynamic.

It is worth noting Swarm [11], which is probably oneof the oldest multi-agent simulators and served asinspiration for Mason, Argos and Stage.

Finally, two new simulators have been growing inthe last years: RoboRobo! [12] and Morse [13]. Theyare both inspired by the need of new fast simulatorsdedicated to multi-agent research and developed withstate-of-the-art technologies. Roborobo! can be prettyuseful as a base for developing a custom simulator fora particular application (e.g. a specific swarm mobilerobotics environment and simulation), but not as sim-ulator capable of providing a platform for testing manydifferent algorithms. Morse is built with interoperabil-ity as a priority: it supports most known middlewares(ROS, YARP) and uses a script to set up simulations.Robot models inside Morse are controlled throughinterprocess communication, new models can be addedby writing sensors and actuators simulation code aspython scripts. At the moment of writing, there areno simulated communication sensors in the simulatorsensor library.

Although the state of art of simulators is mature andfulfills almost every research requirements, we believethat a generic communication system integrated in thesimulation loop is a missing feature. We will proposea design capable of providing this new feature whilekeeping the best practices from state of art simulators.We believe that our communication system design cansuccessfully be added to newborn simulators such asRoboRobo and Morse.

Our work aims to have a completely application–independent simulator suited for distributed mo-bile robotics with simulated communication network,which can be used together with simulated robots andreal ones, see figure 1.

This paper is organized as follows. In section 2 wedescribe the requirements which our simulator wasdesigned for. The global software architecture is ad-dressed in section 3, here the communication facilities

Figure 1: The ASCARI simulated communication.

are introduced, a more detailed description is in section4. The hardware integration is addressed in section 5.We describe two application that use our simulator insection 6.

2 Design requirements

ASCARI has been developed with a set of require-ments that were identified as main features from stateof art simulators. In fact, we believe that building asimulator primarly around novel features would leadto a custom non-generic software, our target is insteadto set a new standard for multi–robot systems simula-tions (as Stage did in 2008).

One of the major features of modern simulators is thepossibility to use the same code for both simulatedand real agents. The only changes required should bethe input/output drivers, not the control law or itsimplementation.

A difference between custom simulators and thewidespread ones is their modular and expandable de-sign. A simulator should never have hardcoded as-sumptions on the system that will be simulated, newmodules with new features should be easily integratedinto the simulator, even core features should be builtas modules. Argos and Gazebo fully represents thisapproach by using interfaces and plugins.

Usually, simulators support multi–process systemswhere each robot controller runs in its own processor computer, while the simulation handles dynamicintegration and model collisions. This approach is alimitation when simulating thousands of robots, andthe solution is the capability of integrating the robotcontrollers directly into the same simulation process,as Stage does. Designing ASCARI, we chose a mixedapproach where controllers can be wrapped in a stan-dalone process or run in a different thread of the sim-ulator process itself. In the next section we will showhow this capability emerged naturally from ASCARIsoftware architecture design.

Following the Gazebo plugin approach, ASCARI sup-ports any kind of dynamic law required by the user.Dynamic laws can be expressed directly as code or as aset of equations: x = f(x, u) where x ∈ Rn is the stateof the agent, u ∈ Rm is the controller output and f is,in general, a non-linear function.

Finally, our main target is the design of a communi-cation system integrated in both the simulation loopand the real robots. The implementation of the com-munication between agents is based on communicationprimitives. Such primitives manage any kind of mes-sage data and hide the simulator to the user providingthe same API for real and simulated agents. Thecommunication module inside the simulator has beendesigned as a plugin that may be changed to modelvarious communication channels with different char-acteristics such as ranges, signal noises and visibility.

3 Software Architecture

In this section we describe some of the main ASCARIsoftware components, focusing in particular on thedesign choices related to the communication facilities.From an high level point of view, ASCARI has aserver that acts as a world simulator, a set of clients(simulated agents) and a simple 2D/3D visualizer.

Agents StateAgents State

Communicationmodule interface

Filteredsensors state

Filteredsensors state

Worldfile

Worldfile

SimulatorSimulatorDynamicIntegrationModule Pa

rserHardware

CommunicationModule

HardwareCommunicationModule

Inter-AgentCommunicationModule

Inter-AgentCommunicationModule

Figure 2: The ASCARI server architecture.

The important server sub-modules are shown in figure2. The server parses a configuration file where theworld and the agents are described. A database witheach agent state is then created and updated by adynamic integration module at each simulation stepafter receiving commands from clients. A user codeis loaded as a communication module, it has readingaccess to the agent state database, and updates anagent communication filter database, which is usedto allow or block inter-agent communications. At themoment, two dynamic modules are available, a simplefixed step forward integration (default) and an exter-nal robot state receiver, which can receive updatedstates from any external dynamic integration systemor real hardware sensors, allowing hardware in the loopsimulations (see details in section 5). The server basicloop flowchart is shown in figure 3.

It sends the updated state to each agent, receives newactuator controls from them, integrates the dynamic,executes the communication module update function,forwards all the inter-agent communications, based onthe agent communication filter database, and startsagain.

The agent structure is represented in figure 4. Asalready said, ASCARI is based on many little building

Figure 3: The ASCARI server flowchart.

blocks, so many client sub-modules are similar totheir respective ones in the server. Hardware andinter–agent communication module are connected tothe server modules. The configuration file is read bythe parser and it is used to provide initializationparameters to the agent control loop.

AgentAgent

Custom loopInterface

C++Control loop

Pars

er

AgentAgentfilefile

Sensors

Actuators Customcontrol loops

HardwareCommunicationModule

HardwareCommunicationModule

Inter-AgentCommunicationModule

Inter-AgentCommunicationModule

Figure 4: The ASCARI client architecture.

The control loop handles sensors and actuators, pro-viding the data to the custom control loop as it waslocal and not remotely simulated. The custom loopinterface is where user code is running. It has access tothe agent state and to the inter-agent communicationprimitives that allows the user to create distributedalgorithms. The parser results are given also to thecustom loop, so that user can specify additional pa-rameters in the same agent configuration file. Theclient basic loop flowchart is shown in figure 5.

Figure 5: The ASCARI client flowchart.

It receives the updated state from the server, executesthe user custom code, and sends the resulting com-mands to the simulator.

Configuration file structure and options

An example of a configuration file is reported. Thestructure is the classic one of the yaml files. The usercan use the provided parser to get the information fromthe file in his/hers custom code.

Inside the tag WORLD (where we can define addi-tional variables) there are the two main tags: BEHAV-IORS and AGENTS.

- WORLD:

BEHAVIORS:

- name: unicycle

states: [x,y,theta]

control_commands: [v,w]

DYNAMIC_MAP:

- x: ’v*cos(theta)’

y: ’v*sin(theta)’

theta: ’w’

AGENTS:

- agent: Agent1

COMMUNICATION_AREA: circle(50)

INITIAL:

- x: ’3’

y: ’3’

theta: ’0’

BEHAVIOR: unicycle

SIMULATED: 1

Inside the BEHAVIORS tag the user can define manyagent types with different states and dynamics.

Inside the AGENTS tag the user can define as manydesired agents with their own parameters and be-haviours. In the example, the variable COMMUNICA-TION AREA is used by a communication filter mod-ule to filter all the messages outside a circle centeredin the robot and with a radius of 50 meters. TheSIMULATED tag allows to use the same configurationfile for real and simulated agents, which can also bemixed together to obtain an hardware in the loopsimulation.

Finally, a 2D/3D visualizer and a simple GUI tostart both the server and all the agents specified ina configuration file complete the ASCARI softwarearchitecture. The visualizer reads the network trafficgenerated between the server and the clients in atransparent way, so that zero or many viewers can beattached to the same simulation.

4 Communication Facility

The core of our system is a set of communication mod-ules and interfaces organized as generic basic buildingblocks, usually a sender and a receiver. Both the com-munication between clients and server and inter-agentcommunication modules are implemented using theseblocks. Messages are strongly typed and can be anycomplex c++ structure, including any STL container.Receiver and sender are c++ classes template–ized onthe type of message they work with, they are initializedwith an IP address, as 1-to-1 or 1-to-N connections,and they can have an optional topic name similar toROS topics used to filter messages. Usually a developeruses both classes and creates a transceiver made witha pair of receiver/sender. The agent code can use thetransceiver to communicate with other agents withoutcaring about the presence of a simulator. In fact, if asimulation is running, communications are automati-cally synchronized with the simulation loop among allthe agents, while in the case of real agents, the sametransceiver creates direct inter–agent connections, thusthe agent code does not require any changes in thecommunication part when switching from simulatedto real hardware. The only changes required are insidethe configuration file, by editing the SIMULATED tag.

Agents StateAgents State

Filteredsensors state

Filteredsensors state

Custom Comm. ModuleCustom Comm. Module

Inter-AgentComm. ModuleInter-AgentComm. Module

- default: receive()- default: receive()- default: forward()- default: forward()

updateFilter()

optional: receive()

optional: forward()

Figure 6: : The ASCARI custom CommunicationModule.

From the simulator point of view, the user needs toinherit a simple filterModule that represents a com-munication channel and to create a custom commu-nication module (figure 6). We chose to give the usermaximum freedom on the channel behaviour but, in itssimpler form, the inherited class has just to implementan updateFilter(...) method. The updateFilter inputsare the simulation states of all the agents, defined asstates variables (e.g. planar coordinates of agents).This method updates a filtering table that tells thesimulator if two agents are able to communicate toeach other at each simulation step. If the user wantsmore control, he can also override the receive(...) andforward(...) methods. These two methods are the onesused by the server in order to receive and transmitinter–agent messages by acting as a gateway. If theyare reimplemented, advanced behavior such as delayson the channel, random packets drop, bandwidth lim-

itations and so on can be obtained. Note that receiveand forward methods have access to the messagescontent, while the updateFilter has, as input, only theagents state.

5 Hardware Integration

All the code written in ASCARI can be reused on thereal hardware thanks to the communication facility.During the execution, if in the configuration file anagent has been declared as not simulated, instead ofsending the commands to the simulator, these aresent via serial communication directly to the hard-ware through a serial communication block that fullycomplies with the standard messages that the clien-t/server exchanges during the simulation. Basicallyinstead of using the actuation commands to simulatethe dynamics of the robot, these are used to move thereal robot. With this configuration the only thing auser, that want to pass to the hardware, has to careabout is to have something that can receive via serialbus the actuation commands and execute them. Thecommands for the robots currently used are linearacceleration and angular acceleration.

An example of the hardware we used is shown in figure7.

Figure 7: A mobile robot, equipped with a RaspberryPi and an Arduino Uno

In this robot there is a Raspberry Pi, which is thecore of the robot. On board there are a wireless keyto communicate with the other agents and a Linuxsystem on which the various algorithms run. Pluggedto it via serial bus there is an Arduino Uno whichreceive the actuation commands from the Raspberryand send them to the motors via a shield board (wireshave been omitted for the sake of clarity).

One advantage to have this architecture is that wecan work with real and simulated robot as well justdefining the type of the robots in the configuration file.For example we can run a collision avoidance algorithmon two real robot and a simulated one. In the real

scenario we will see the real robots steering ‘near’ thesimulated one, even if there is nothing in front of them.

6 Applications

Many applications have been developed thanks to AS-CARI, both on the software and on the hardwareside. These applications concern for example: colli-sion avoidance, agents localization using webcams andmarkers and classical distributed robotics algorithmsuch as rendez-vous and distributed consensus.

Two of the most interesting applications are a dis-tributed task assignment algorithm and a distributedtraffic control one.

In the task assignment application a subgradientmethod based algorithm allows to assign various tasksto heterogeneous robots optimally, [15]. ConcerningASCARI, it has been used to simulated the variousrobots’ dynamics and to coordinate them during thescenario evolution. Thanks to the communication facil-ity the agents exchange through the network their localsubgradient, allowing the other robots to compute thetotal subgradient until the algorithm converges. In thesimulated scenario, the simulator takes into accountthe dynamicity of the tasks, making them appearing,evolving and expiring during time. The tasks’ stateis sent through the network by the simulator, so thevarious agents can know if a task is already taken,or expired, or if some new tasks are available. Inthe real scenario the tasks are the classical industrialones, such as putting material on a conveyor belt, ormove pallets across a warehouse. In this applicationthe robot recharging is seen has a high priority task,depending on the amount of charge of the robot, whichappears if the amount of a robot’s charge goes belowa certain value. In Figure 8 a frame of a simulation ofthe algorithm in an industrial scenario is depicted.

Figure 8: Distributed task assignment algorithm inASCARI.

In [14], a distributed traffic control protocol has beenproposed. The algorithm uses communication to fore-see and avoid agents collisions on a topological maprepresented with a graph (see Figure 9). The messagesexchanged by agents are a set of future nodes and arcs

that each agent would like to cross, a resource lockingalgorithm ensures that agents will not try to cross thesame node in the same future time and hence ensurethe safety of the system. The algorithm was testedwith different communication ranges and with randompackets drop to assess its robustness. In the single-process version of ASCARI 30 agents with a simpleunicycle dynamics were simulated at around 50x realtime with an integration step of 0.001 s.

Figure 9: Distributed traffic control algorithm inASCARI.

Notice that ASCARI can be used to implement andtest any robot control algorithm, but it is especiallysuited for multi–agent collaborative algorithms alsowith a non–perfect communication channel. A usermust define the agents’ control law and a dedicatedcommunicator if particular data exchange between theagents (or between the agents and the simulator) isneeded.

To view the original videos please go to the AS-CARISimulator channel on:

http://www.youtube.com/user/ASCARIsimulator.

7 Conclusions

In this paper the ASCARI distributed mobile roboticssimulator with simulated communication network hasbeen presented. The simulator is suited to developmany applications, and in particular it has been de-signed to be scalable and hence it is able to handlemany robots. The hardware integration is straight-forward and the user can use real robots and simu-lated ones as well. Future works include: export thecommunication facilities to Argos and RoboRobo! andreal object simulation (as done in the task assignmentapplication).

References

[1] Vaughan, Richard. ”Massively multi-robot simu-lation in stage.” Swarm Intelligence 2.2-4 (2008):189-208.

[2] Vaughan, Richard T., Brian P. Gerkey, andAndrew Howard. ”On device abstractions forportable, reusable robot code.” Intelligent Robots

and Systems, 2003.(IROS 2003). Proceedings. 2003IEEE/RSJ International Conference on. Vol. 3.IEEE, 2003.

[3] Gerkey, Brian, Richard T. Vaughan, and An-drew Howard. ”The player/stage project: Toolsfor multi-robot and distributed sensor systems.”Proceedings of the 11th international conferenceon advanced robotics. Vol. 1. 2003.

[4] Balmer, Michael, et al. ”MATSim-T: Architectureand simulation times.” Multi-agent systems fortraffic and transportation engineering (2009): 57-78.

[5] Michel, Olivier. ”Webots: Symbiosis between vir-tual and real mobile robots.” Virtual Worlds.Springer Berlin Heidelberg, 1998.

[6] Pinciroli, Carlo, et al. ”ARGoS: a modular, par-allel, multi-engine simulator for multi-robot sys-tems.” Swarm intelligence 6.4 (2012): 271-295.

[7] Jackson, Jared. ”Microsoft robotics studio: A tech-nical introduction.” Robotics & Automation Mag-azine, IEEE 14.4 (2007): 82-87.

[8] Carpin, Stefano, et al. ”USARSim: a robot sim-ulator for research and education.” Robotics andAutomation, 2007 IEEE International Conferenceon. IEEE, 2007.

[9] Freese, Marc, et al. ”Virtual robot experimenta-tion platform V-REP: a versatile 3D robot simula-tor.” Simulation, Modeling, and Programming for

Autonomous Robots. Springer Berlin Heidelberg,2010. 51-62.

[10] Luke, Sean, et al. ”MASON: A Java multi-agentsimulation library.” Proceedings of Agent 2003Conference on Challenges in Social Simulation.Vol. 9. 2003.

[11] Minar, Nelson, et al. ”The swarm simulationsystem: a toolkit for building multi-agent systems.”Santa Fe NM: Santa Fe Institute Working Paper(1996): 96-06.

[12] Bredeche, Nicolas, et al. ”Roborobo! a fast robotsimulator for swarm and collective robotics.” arXivpreprint arXiv:1304.2888 (2013).

[13] Echeverria, Gilberto, et al. ”Modular open robotssimulation engine: Morse.” Robotics and Automa-tion (ICRA), 2011 IEEE International Conferenceon. IEEE, 2011.

[14] M. Ferrati and Pallottino, L., ”A time expandednetwork based algorithm for safe and efficient dis-tributed multi-agent coordination”, in IEEE Con-ference on Decision and Control, Florence, Italy,2013, pp. 2805 - 2810.

[15] A. Settimi and Pallottino, L., ”A SubgradientBased Algorithm for Distributed Task Assignmentfor Heterogeneous Mobile Robots”, in IEEE Con-ference on Decision and Control, Florence, Italy,2013, pp. 3665 - 3670.