mobile ad hoc networking (cutting edge directions) || experimental work versus simulation in the...
Post on 08-Dec-2016
Embed Size (px)
6EXPERIMENTAL WORK VERSUSSIMULATION IN THE STUDY OFMOBILE AD HOC NETWORKS
Carlo Vallati, Victor Omwando, and Prasant Mohapatra
Simulation and testbed implementations have been variously adopted in the last yearsby the mobile ad hoc network research community in the development and assessmentof new complex systems and protocols in place of analytical models. Since eachapproach has its advantages and drawbacks, it is important to understand what eachapproach offers in order to adopt the appropriate solution in the validation of ideas.In this chapter we first review existing testbeds and simulation tools and then discussissues that cause gaps between their results. In order to help the reader in makinginformed decisions in their evaluations and obtaining sound results, we focus onissues which jeopardize the credibility of their experiments and present strategieswhich can help to improve reliability.
Simulators and physical testbeds have been widely embraced by the mobile ad hocnetwork research community in the development of new and improved systems andprotocols. While analytical models provide much insight to the performance of pro-posed designs, they do not have enough detail to handle the complexity of recentprotocols and device implementations in mobile ad hoc networks.
Mobile Ad Hoc Networking: Cutting Edge Directions, Second Edition. Edited by Stefano Basagni,Marco Conti, Silvia Giordano, and Ivan Stojmenovic. 2013 by The Institute of Electrical and Electronics Engineers, Inc. Published 2013 by John Wiley & Sons, Inc.
192 EXPERIMENTAL WORK VERSUS SIMULATION
Each of the evaluation approaches has its merits and demerits. The benefits of usingsimulations include easy and fast deployment, controllable and flexible environment,better scalability, and repeatable results. However, it is quite easy to produce simu-lations that lack credibility; in fact, opinion is spreading that some published resultssuffer from this problem . The reliability of simulations results can be jeopardizedmainly in two ways: bad simulation practices (mistakes in the setting or design ofsimulations) or poor model assumptions . On the other hand, testbeds largely avoidissues related to imperfect modeling and can provide something close to a target op-erating environment. Even so, they are far harder to prototype, configure, and deploy,especially if different operating conditions need to be considered. Thus, it is importantto understand what each option offers in order to make the appropriate choices duringthe validation of ideas.
In this chapter we will review existing testbeds and simulation tools and thendiscuss issues that cause gaps between their results. We will also opine on possi-ble avenues for the improvement of simulation tools and testbed environments. Wehope that, as a result, readers will be able to make informed decisions on a suitableevaluation strategy for their solutions.
6.2 OVERVIEW OF MOBILE AD HOC NETWORK SIMULATIONTOOLS AND EXPERIMENTAL PLATFORMS
In this section we shall provide an overview of existing simulators and testbeds,focusing on their major features and characteristics in order to provide the reader agood picture of existing solutions for MANET assessment. Simulators and testbedsare the two tools that researchers can exploit to assess the performance of MANETsolutions. So far, the use of simulators has been predominant as demonstrated inthe statistics presented in reference 3. Indeed, modules and extensions for mobile adhoc networking have been included in almost all the available simulators, both opensource and commercial.
Recently, the proliferation of cheap WLAN hardware has boosted the use oftestbeds that are recommended for performance evaluation due to their accuracyand reliability. Nevertheless, simulators are still preferred, especially when a rapiddeployment is needed.
6.2.1 Simulation Tools
By providing an overview of the existing network simulators supporting mobile ad hocnetworks, we aim to illustrate advantages and disadvantages commonly attributed tothe general structure of each simulator before going into the details of their respectivewireless module implementations in Section 6.3.5.
This survey mainly focuses on the following simulators that have enjoyed themost usage by researchers, according to the usage statistics between 2000 and 2005presented in reference 3: ns2 , GloMoSim , QualNet  and Opnet . In orderto take into account simulators that have recently gained popularity, we additionally
OVERVIEW OF MOBILE AD HOC NETWORK SIMULATION TOOLS 193
consider the following: ns3 , OMNeT++  and Jist . Several in-depth surveyson network simulators can be found in references 1113.
ns-2  is the most popular discrete event network simulator used by researchers.It has been developed as an open-source project since 1989 and is written in C++and OTcl. While the former is used for programming the simulation core, the latter isadopted to dynamically compose simulation scenarios without having to recompilethe whole source code every time. As the most widely used open source networksimulator, its main advantage is the large number of external projects deployed to addnew functionality and support new networks. Its behavior is highly trusted within thenetwork community, by virtue of it being the oldest and most used project. Althoughit was initially designed for wired networks, wireless modules have been deployed asexternal contributions and then included into the official version. Its main downsideis the inherent complexity caused by the lack of modularity, which makes the imple-mentation of new features in the original core nontrivial . Another drawback is thelack of official tools for statistical collection and analysis; even if external solutionscan be found, the official version only offers the possibility to save traces that haveto be postprocessed to collect statistics.
ns3  is the new major revision of ns2. Like its predecessor, ns3 relies on C++ forsimulation model implementation. Since frequently recompiling the source code is notan issue nowadays, the scenarios are defined in pure C++ or optionally in Python. Thecore has been designed with scalability in mind in order to support future development.Even though ns3 it is not compatible with much of the ns2 codebase, several projectshave already been ported. Additionally, several new features have been developed.One of the most interesting is the support for integration with real devices. Althoughattention and development efforts on ns3 are increasing, the project is still consideredyoung and some important functionality such as a module for statistics collection isstill to be implemented.
GloMoSim  is the second most popular wireless network simulator. Written inParsec, it benefits from the ability of this language to run in parallel environments. Incontrast to ns2, parallelization enables GloMoSim to run more complex scenarios; thesimulated network is partitioned into different subnetworks, each run by distinct pro-cessors. This feature was the main reason why GloMoSim was used as the base for thecommercial QualNet simulator. The lack of documentation and project discontinuityare the main disadvantages.
QualNet  is a commercial simulator based on the core of GloMoSim. The basecore has been largely extended and a new set of protocols and models are supported.The project is currently maintained, and updated documentation is available. A set ofgraphical tools are provided in order to help users define simulation scenarios, andcollect and analyze measurements.
Opnet  is a well-established commercial discrete event simulator mainly usedby companies to simulate the organization of their networks. It offers a vast numberof models for both wired and wireless networks via its Wireless Suite. A graphicaluser interface aids the user in defining simulation scenarios and analyzing results. Allthe models are officially developed by the company itself which is responsible fortheir validation. The user has the option to define new modules by means of graphical
194 EXPERIMENTAL WORK VERSUS SIMULATION
interface but the procedure can be complicated. The main drawback is that every newmodule has to be defined as a finite state machine that is difficult to debug, extend, andvalidate. Moreover, the actions behind every state are described through a nonstandardlanguage, the Proto-C.
OMNeT++  is a simulator platform written in C++ for general-purpose discreteevent simulation. However, it is mainly applied to network simulations due to its INETpackage which provides a collection of Internet protocols. In addition, other modelpackages like Mobility Package and Castalia allow the simulation of mobile ad-hocnetworks. OMNeT++ has a modular structure; each atomic module (simple module)can be used to produce more complex entities (compound modules). Besides its exten-sible and flexible structure, another merit is the large and well-written documentationavailable to go along with a set of tutorials on the project website. Simulation scenar-ios are described through a high-level language called Network Description (NED).More complex scenarios can be assembled with the aid of a graphical user interfaceprovided in the standard package. The only minor disadvantage is the lack of supportfor statistics collection and data analysis.
Jist  is a general-purpose simulator written in Java. The authors provide apackage, Swans , which implements wireless mobile ad hoc network simulationcapabilities. As with OMNeT++, Jist has a modular structure made up of entities: eachentity represents a network element whose behavior is described by a Java class. Avery simple parallelization mechanism is available: simulat