Mobile Ad Hoc Networking (Cutting Edge Directions) || Experimental Work Versus Simulation in the Study of Mobile Ad Hoc Networks
Post on 08-Dec-2016
6EXPERIMENTAL WORK VERSUSSIMULATION IN THE STUDY OFMOBILE AD HOC NETWORKS
Carlo Vallati, Victor Omwando, and Prasant Mohapatra
Simulation and testbed implementations have been variously adopted in the last yearsby the mobile ad hoc network research community in the development and assessmentof new complex systems and protocols in place of analytical models. Since eachapproach has its advantages and drawbacks, it is important to understand what eachapproach offers in order to adopt the appropriate solution in the validation of ideas.In this chapter we first review existing testbeds and simulation tools and then discussissues that cause gaps between their results. In order to help the reader in makinginformed decisions in their evaluations and obtaining sound results, we focus onissues which jeopardize the credibility of their experiments and present strategieswhich can help to improve reliability.
Simulators and physical testbeds have been widely embraced by the mobile ad hocnetwork research community in the development of new and improved systems andprotocols. While analytical models provide much insight to the performance of pro-posed designs, they do not have enough detail to handle the complexity of recentprotocols and device implementations in mobile ad hoc networks.
Mobile Ad Hoc Networking: Cutting Edge Directions, Second Edition. Edited by Stefano Basagni,Marco Conti, Silvia Giordano, and Ivan Stojmenovic. 2013 by The Institute of Electrical and Electronics Engineers, Inc. Published 2013 by John Wiley & Sons, Inc.
192 EXPERIMENTAL WORK VERSUS SIMULATION
Each of the evaluation approaches has its merits and demerits. The benefits of usingsimulations include easy and fast deployment, controllable and flexible environment,better scalability, and repeatable results. However, it is quite easy to produce simu-lations that lack credibility; in fact, opinion is spreading that some published resultssuffer from this problem . The reliability of simulations results can be jeopardizedmainly in two ways: bad simulation practices (mistakes in the setting or design ofsimulations) or poor model assumptions . On the other hand, testbeds largely avoidissues related to imperfect modeling and can provide something close to a target op-erating environment. Even so, they are far harder to prototype, configure, and deploy,especially if different operating conditions need to be considered. Thus, it is importantto understand what each option offers in order to make the appropriate choices duringthe validation of ideas.
In this chapter we will review existing testbeds and simulation tools and thendiscuss issues that cause gaps between their results. We will also opine on possi-ble avenues for the improvement of simulation tools and testbed environments. Wehope that, as a result, readers will be able to make informed decisions on a suitableevaluation strategy for their solutions.
6.2 OVERVIEW OF MOBILE AD HOC NETWORK SIMULATIONTOOLS AND EXPERIMENTAL PLATFORMS
In this section we shall provide an overview of existing simulators and testbeds,focusing on their major features and characteristics in order to provide the reader agood picture of existing solutions for MANET assessment. Simulators and testbedsare the two tools that researchers can exploit to assess the performance of MANETsolutions. So far, the use of simulators has been predominant as demonstrated inthe statistics presented in reference 3. Indeed, modules and extensions for mobile adhoc networking have been included in almost all the available simulators, both opensource and commercial.
Recently, the proliferation of cheap WLAN hardware has boosted the use oftestbeds that are recommended for performance evaluation due to their accuracyand reliability. Nevertheless, simulators are still preferred, especially when a rapiddeployment is needed.
6.2.1 Simulation Tools
By providing an overview of the existing network simulators supporting mobile ad hocnetworks, we aim to illustrate advantages and disadvantages commonly attributed tothe general structure of each simulator before going into the details of their respectivewireless module implementations in Section 6.3.5.
This survey mainly focuses on the following simulators that have enjoyed themost usage by researchers, according to the usage statistics between 2000 and 2005presented in reference 3: ns2 , GloMoSim , QualNet  and Opnet . In orderto take into account simulators that have recently gained popularity, we additionally
OVERVIEW OF MOBILE AD HOC NETWORK SIMULATION TOOLS 193
consider the following: ns3 , OMNeT++  and Jist . Several in-depth surveyson network simulators can be found in references 1113.
ns-2  is the most popular discrete event network simulator used by researchers.It has been developed as an open-source project since 1989 and is written in C++and OTcl. While the former is used for programming the simulation core, the latter isadopted to dynamically compose simulation scenarios without having to recompilethe whole source code every time. As the most widely used open source networksimulator, its main advantage is the large number of external projects deployed to addnew functionality and support new networks. Its behavior is highly trusted within thenetwork community, by virtue of it being the oldest and most used project. Althoughit was initially designed for wired networks, wireless modules have been deployed asexternal contributions and then included into the official version. Its main downsideis the inherent complexity caused by the lack of modularity, which makes the imple-mentation of new features in the original core nontrivial . Another drawback is thelack of official tools for statistical collection and analysis; even if external solutionscan be found, the official version only offers the possibility to save traces that haveto be postprocessed to collect statistics.
ns3  is the new major revision of ns2. Like its predecessor, ns3 relies on C++ forsimulation model implementation. Since frequently recompiling the source code is notan issue nowadays, the scenarios are defined in pure C++ or optionally in Python. Thecore has been designed with scalability in mind in order to support future development.Even though ns3 it is not compatible with much of the ns2 codebase, several projectshave already been ported. Additionally, several new features have been developed.One of the most interesting is the support for integration with real devices. Althoughattention and development efforts on ns3 are increasing, the project is still consideredyoung and some important functionality such as a module for statistics collection isstill to be implemented.
GloMoSim  is the second most popular wireless network simulator. Written inParsec, it benefits from the ability of this language to run in parallel environments. Incontrast to ns2, parallelization enables GloMoSim to run more complex scenarios; thesimulated network is partitioned into different subnetworks, each run by distinct pro-cessors. This feature was the main reason why GloMoSim was used as the base for thecommercial QualNet simulator. The lack of documentation and project discontinuityare the main disadvantages.
QualNet  is a commercial simulator based on the core of GloMoSim. The basecore has been largely extended and a new set of protocols and models are supported.The project is currently maintained, and updated documentation is available. A set ofgraphical tools are provided in order to help users define simulation scenarios, andcollect and analyze measurements.
Opnet  is a well-established commercial discrete event simulator mainly usedby companies to simulate the organization of their networks. It offers a vast numberof models for both wired and wireless networks via its Wireless Suite. A graphicaluser interface aids the user in defining simulation scenarios and analyzing results. Allthe models are officially developed by the company itself which is responsible fortheir validation. The user has the option to define new modules by means of graphical
194 EXPERIMENTAL WORK VERSUS SIMULATION
interface but the procedure can be complicated. The main drawback is that every newmodule has to be defined as a finite state machine that is difficult to debug, extend, andvalidate. Moreover, the actions behind every state are described through a nonstandardlanguage, the Proto-C.
OMNeT++  is a simulator platform written in C++ for general-purpose discreteevent simulation. However, it is mainly applied to network simulations due to its INETpackage which provides a collection of Internet protocols. In addition, other modelpackages like Mobility Package and Castalia allow the simulation of mobile ad-hocnetworks. OMNeT++ has a modular structure; each atomic module (simple module)can be used to produce more complex entities (compound modules). Besides its exten-sible and flexible structure, another merit is the large and well-written documentationavailable to go along with a set of tutorials on the project website. Simulation scenar-ios are described through a high-level language called Network Description (NED).More complex scenarios can be assembled with the aid of a graphical user interfaceprovided in the standard package. The only minor disadvantage is the lack of supportfor statistics collection and data analysis.
Jist  is a general-purpose simulator written in Java. The authors provide apackage, Swans , which implements wireless mobile ad hoc network simulationcapabilities. As with OMNeT++, Jist has a modular structure made up of entities: eachentity represents a network element whose behavior is described by a Java class. Avery simple parallelization mechanism is available: simulation load can be distributednot only on multiple CPU inside the same machine but also onto different servers. Nodefault interface for configuring simulation scenarios is provided; either Java code ora configuration file parsed at run time can be adopted. Although Swans is a relativelynew project, Jist core development is no longer pursued by its original author.
6.2.2 Experimental Platforms
Experimental platforms can be broadly categorized into real world deployments andemulations. As the term suggests, the former refers to testbeds that have been set up toreplicate target deployment characteristics like network size and unmodified wirelesschannels to produce observations that closely mirror what would be expected in alive deployment. However, setting up these environments involves a great degreeof complexity, and builders are faced with problems related to costs, scalability, man-agement, experimental control, repeatability, and applicability to multiple scenarios.In addition, they may have limitations in the number of possible topologies that can beexplored. Thus, most real-world testbeds do not provide support for mobility testing.In fact, many real-world experiments are conducted in short-lived, proof of conceptnetworks that are devised as needed by researchers. A comprehensive treatment ofthese experiments as well as static ad hoc, sensor and mesh network testbeds can befound in references 1518. We instead focus on the few persistent real-world testbedsthat support mobile ad hoc networking.
APE (Ad Hoc Protocol Evaluation Testbed)  is designed achieve test re-peatability and result reproducibility. It is distributed as a software package
OVERVIEW OF MOBILE AD HOC NETWORK SIMULATION TOOLS 195
consisting of build scripts and source code. The software is installed on lap-tops that are carried around by test participants who can be either on foot or invehicles. Experiments are choreographed via movement scenario files, whichinform participants as to when and where they shall move during the experi-ment. Furthermore, scenario files contain commands and instructions that are tobe executed during the duration of the experiment.
DOME (Diverse Outdoor Mobile Environment)  is a large-scale, highlydiverse mobile systems testbed that provides considerable technological andspatial diversity in addition to temporal diversity by virtue of it being operationalfor at least 5 years. It consists of three major hardware components: DieselNetvehicular nodes, half a dozen throwboxes that can serve as relays, and a municipalWiFi mesh network with 26 stationary access points. The major constituent,DieselNet, covers 150 square miles and consists of 40 transit buses equippedwith 802.11abg capable nodes. Each node has a 802.11g AP, a wireless 3G USBmodem, and a 900-MHz USB RF modem. Bus riders can thus connect to theInternet via the 3G modem, after associating with the AP. The WiFi interfacecan also connect to the APs on other buses. The throwboxes use solar-chargedbatteries that allow them to be nomadic. They can be attached in front of bicyclesand can also be allowed to remain stationary for several hours or days. TheWiFi APs are mounted on different buildings and light posts within the urbanarea, but only the building-mounted APs have a link to the local fiber network.The system also has software modules for link management, remote softwareupdates, logging, and maintenance monitoring. At the moment, DOME is notavailable to the public, but plans are underway to allow remote access via theGENI project . Also, traces of experiments run on DOME can be found athttp://traces.cs.umass.edu.
Mobile Emulab  was conceived to provide a remotely accessible sensor andmobile network experimentation platform in an indoor environment. Experi-ments are easy to deploy via the Emulab  web-based front-end by supply-ing ns-2 scripts. The testbed is in an L-shaped area of 60 m2 by 2.5 m high.There are 6 Acroname Garcia  robots to provide mobility. Each robot hasa WiFi-equipped computers and sensor mote. Robot positioning is performedvia medium-cost video camera equipment. Unfortunately, as of 2011, mobileEmulab is not supported and is no longer publicly accessible.
QuRiNet(Quail Ridge Wireless Mesh Network)  is set in a 2000-acre naturalreserve and provides an experimental setting free of unwanted electromagneticinterference due to the remote setting of the reserve. It has 38 static nodes runningthe OLSR ad hoc routing protocol, with ongoing efforts to permanently installmobile nodes on six all-terrain vehicles. Mobility is also supported by havingtest participants walk or ride on all-terrain vehicles while carrying laptops tocreate mobility scenarios. External researchers can make use of QuRiNet byrequesting access to the testbed manager.
Figure 6.1 shows the static node placement of QuRiNet, along with the accessroads used by test participants and all-terrain vehicles.
196 EXPERIMENTAL WORK VERSUS SIMULATION
Figure 6.1 QuRiNet Layout.
Emulators bridge the gap between real-world testbeds and full-fledged simulators.As with real wireless networks, they utilize real network stacks and hardware, butthey additionally int...