[ op_ research @ im.ntu ]

[[OP_ ResearchOP_ Research @@ IM.NTU IM.NTU]]

Agenda:Agenda:

IntroductionIntroduction Survey ResultsSurvey Results Common Simulation PitfallsCommon Simulation Pitfalls

Simulation SetupSimulation Setup Simulation ExecutionSimulation Execution Output AnalysisOutput Analysis PublishingPublishing

Community ResourcesCommunity Resources ConclusionConclusion

MANET Simulation Studies : The IncredibilesMANET Simulation Studies : The Incredibiles2005 Mobile Computing and Communications Review, Volume 1, Number 22005 Mobile Computing and Communications Review, Volume 1, Number 2

指導教授：林永松2008/06/09

D95725009陳仁偉

OP@ IM. NTU 2008 Eric Chen .

[ 2 ]

AuthorAuthor

Education Ph.D. Computer Science, The College of William & Mary, 1993M.S. Computer Science, Michigan State University, 1989B.A. Mathematics, Kalamazoo College, 1987

Stuart Kurkowski

Michael Colagrosso


[ 3 ]

Mobile Ad Hoc Networks (MANETs) are wireless Mobile Ad Hoc Networks (MANETs) are wireless

mobile nodes that cooperatively form a network mobile nodes that cooperatively form a network

without Infrastructurewithout Infrastructure

Challenges:Challenges: Routing, wireless communications issues, Routing, wireless communications issues,

resource issues, resource issues, The leading is simulation

Simulation Study: ACM International Symposium on ACM International Symposium on

Mobile Ad Hoc Networking and Computing (MobiHoc) Mobile Ad Hoc Networking and Computing (MobiHoc)

from 2000-2005from 2000-2005

MANET simulation studies lack believability

Mobile Ad Hoc Networks (MANETs) are wireless Mobile Ad Hoc Networks (MANETs) are wireless

mobile nodes that cooperatively form a network mobile nodes that cooperatively form a network

without Infrastructurewithout Infrastructure

Challenges:Challenges: Routing, wireless communications issues, Routing, wireless communications issues,

resource issues, resource issues, The leading is simulation

Simulation Study: ACM International Symposium on ACM International Symposium on

Mobile Ad Hoc Networking and Computing (MobiHoc) Mobile Ad Hoc Networking and Computing (MobiHoc)

from 2000-2005from 2000-2005

MANET simulation studies lack believability

I. IntroductionI. Introduction


[ 4 ]

This study focused on the following four areas of credibility in research

1.1. RepeatableRepeatable

2.2. UnbiasedUnbiased

3.3. RigorousRigorous

4.4. Statistically soundStatistically sound

This study focused on the following four areas of credibility in research

1.1. RepeatableRepeatable

2.2. UnbiasedUnbiased

3.3. RigorousRigorous

4.4. Statistically soundStatistically sound

FocusFocus


[ 5 ]

MANET research MANET research full papers full papers published in published in MobiHocMobiHoc 114 out of the 151 MobiHoc papers published 114 out of the 151 MobiHoc papers published

(75.5%) used (75.5%) used simulationsimulation Unfortunately, 34 of the 114 published MobiHoc Unfortunately, 34 of the 114 published MobiHoc

simulation papers (29.8%) did not identify the simulation papers (29.8%) did not identify the

simulatorsimulator used in the research. used in the research. Network Simulator-2 (Network Simulator-2 (NS-2)NS-2) [34] is the most used [34] is the most used

simulator in MANET research; 35 of the 80 simulation simulator in MANET research; 35 of the 80 simulation

papers that state the simulator used in the papers that state the simulator used in the

simulation study used NS-2 (43.8%).simulation study used NS-2 (43.8%). No paper made a statement about No paper made a statement about code availability code availability

MANET research MANET research full papers full papers published in published in MobiHocMobiHoc 114 out of the 151 MobiHoc papers published 114 out of the 151 MobiHoc papers published

(75.5%) used (75.5%) used simulationsimulation Unfortunately, 34 of the 114 published MobiHoc Unfortunately, 34 of the 114 published MobiHoc

simulation papers (29.8%) did not identify the simulation papers (29.8%) did not identify the

simulatorsimulator used in the research. used in the research. Network Simulator-2 (Network Simulator-2 (NS-2)NS-2) [34] is the most used [34] is the most used

simulator in MANET research; 35 of the 80 simulation simulator in MANET research; 35 of the 80 simulation

papers that state the simulator used in the papers that state the simulator used in the

simulation study used NS-2 (43.8%).simulation study used NS-2 (43.8%). No paper made a statement about No paper made a statement about code availability code availability

I.A. The Current State of MANETI.A. The Current State of MANET Simulation Studies Simulation Studies

Repeatability - 1Repeatability - 1


[ 6 ]


[ 7 ]

The researcher must identify the The researcher must identify the simulatorsimulator and and

version,version, the the operating systemoperating system, and all , and all variable variable

settingssettings.. Repeatability is also based on the scenarios Repeatability is also based on the scenarios

evaluated, the techniques used to avoid evaluated, the techniques used to avoid initialization initialization

biasbias (influence of empty queues, etc., at the start), (influence of empty queues, etc., at the start),

and the techniques used to and the techniques used to analyze the resultsanalyze the results.. Thus, a Thus, a published paper published paper must discuss or reference all must discuss or reference all

of these details to meet the of these details to meet the repeatability criteriarepeatability criteria

The researcher must identify the The researcher must identify the simulatorsimulator and and

version,version, the the operating systemoperating system, and all , and all variable variable

settingssettings.. Repeatability is also based on the scenarios Repeatability is also based on the scenarios

evaluated, the techniques used to avoid evaluated, the techniques used to avoid initialization initialization

biasbias (influence of empty queues, etc., at the start), (influence of empty queues, etc., at the start),

and the techniques used to and the techniques used to analyze the resultsanalyze the results.. Thus, a Thus, a published paper published paper must discuss or reference all must discuss or reference all

of these details to meet the of these details to meet the repeatability criteriarepeatability criteria

I.A. The Current State of MANETI.A. The Current State of MANET Simulation Studies Simulation Studies

Repeatability -2Repeatability -2


[ 8 ]

Unbiased study: Unbiased study:

A project must address A project must address initialization biasinitialization bias, , random random

numbernumber issues, and issues, and use a variety of scenariosuse a variety of scenarios. .

Rigorous study:Rigorous study:

Factors such as Factors such as node densitynode density, , node footprintnode footprint, ,

coveragecoverage, , speedspeed, and , and transmission range transmission range must be must be

set to exercise the protocol under test. set to exercise the protocol under test.

Unbiased study: Unbiased study:

A project must address A project must address initialization biasinitialization bias, , random random

numbernumber issues, and issues, and use a variety of scenariosuse a variety of scenarios. .

Rigorous study:Rigorous study:

Factors such as Factors such as node densitynode density, , node footprintnode footprint, ,

coveragecoverage, , speedspeed, and , and transmission range transmission range must be must be

set to exercise the protocol under test. set to exercise the protocol under test.

I.A. The Current State of MANETI.A. The Current State of MANET Simulation StudiesSimulation Studies


[ 9 ]

Statistically sound study:Statistically sound study:

A project must account for A project must account for initialization biasinitialization bias, , executeexecute

a number of simulation iterations, provide the a number of simulation iterations, provide the

confidence levels confidence levels that exist in the results, and list that exist in the results, and list

any statistical assumptions made.any statistical assumptions made.

Statistically sound study:Statistically sound study:

A project must account for A project must account for initialization biasinitialization bias, , executeexecute

a number of simulation iterations, provide the a number of simulation iterations, provide the

confidence levels confidence levels that exist in the results, and list that exist in the results, and list

any statistical assumptions made.any statistical assumptions made.

I.A. The Current State of MANETI.A. The Current State of MANET Simulation StudiesSimulation Studies


[ 10 ]

Simulation type: - Simulation type: - terminating or steady-state

66 of the 114 simulation papers (57.9%) did not 66 of the 114 simulation papers (57.9%) did not

mention the type of simulation used in the study.mention the type of simulation used in the study.

Pseudo random number generator (PRNG)Pseudo random number generator (PRNG)

The goal is to raise awareness of the issues and to The goal is to raise awareness of the issues and to

introduce tools to aid MANET researchers in introduce tools to aid MANET researchers in

conducting and reporting credible simulation resultsconducting and reporting credible simulation results

Simulation type: - Simulation type: - terminating or steady-state

66 of the 114 simulation papers (57.9%) did not 66 of the 114 simulation papers (57.9%) did not

mention the type of simulation used in the study.mention the type of simulation used in the study.

Pseudo random number generator (PRNG)Pseudo random number generator (PRNG)

The goal is to raise awareness of the issues and to The goal is to raise awareness of the issues and to

introduce tools to aid MANET researchers in introduce tools to aid MANET researchers in

conducting and reporting credible simulation resultsconducting and reporting credible simulation results

I.B. Survey MotivationI.B. Survey Motivation


[ 11 ]

Simulation study questions.Simulation study questions.

Reviewed each paper individuallyReviewed each paper individually

For consistency, the same person reviewed all of the For consistency, the same person reviewed all of the

papers; to validate the results, we had a second papers; to validate the results, we had a second

person review all of the papers with a subset of the person review all of the papers with a subset of the

questions and a third person to correct the few questions and a third person to correct the few

inconsistencies found.inconsistencies found.

114 114 out of the 151 (75.5%) published MobiHoc out of the 151 (75.5%) published MobiHoc

papers used simulation as the basis for the study.papers used simulation as the basis for the study.

Simulation study questions.Simulation study questions.

Reviewed each paper individuallyReviewed each paper individually

For consistency, the same person reviewed all of the For consistency, the same person reviewed all of the

papers; to validate the results, we had a second papers; to validate the results, we had a second

person review all of the papers with a subset of the person review all of the papers with a subset of the

questions and a third person to correct the few questions and a third person to correct the few

inconsistencies found.inconsistencies found.

114 114 out of the 151 (75.5%) published MobiHoc out of the 151 (75.5%) published MobiHoc

papers used simulation as the basis for the study.papers used simulation as the basis for the study.

II. Survey ResultsII. Survey ResultsResearch DesignResearch Design


[ 12 ]


[ 13 ]


[ 14 ]


[ 15 ]

III.A. Simulation SetupIII.A. Simulation SetupIII.A.1. Simulation TypeIII.A.1. Simulation Type

III.A.2. Model Validation & VerificationIII.A.2. Model Validation & Verification

III.A.3. PRNG Validation & VerificationIII.A.3. PRNG Validation & Verification

III.A.4. Variable DefinitionIII.A.4. Variable Definition

III.A.5. Scenario DevelopmentIII.A.5. Scenario Development

III.B. Simulation ExecutionIII.B. Simulation ExecutionIII.B.1. Setting the PRNG SeedIII.B.1. Setting the PRNG Seed

III.B.2. Scenario InitializationIII.B.2. Scenario Initialization

III.B.3. Metric CollectionIII.B.3. Metric Collection

III.C. Output AnalysisIII.C. Output AnalysisIII.C.1. Single Set of DataIII.C.1. Single Set of Data

III.C.2. Statistical AnalysisIII.C.2. Statistical Analysis

III.C.3. Confidence IntervalsIII.C.3. Confidence Intervals

III.D. PublishingIII.D. Publishing

III.A. Simulation SetupIII.A. Simulation SetupIII.A.1. Simulation TypeIII.A.1. Simulation Type

III.A.2. Model Validation & VerificationIII.A.2. Model Validation & Verification

III.A.3. PRNG Validation & VerificationIII.A.3. PRNG Validation & Verification

III.A.4. Variable DefinitionIII.A.4. Variable Definition

III.A.5. Scenario DevelopmentIII.A.5. Scenario Development

III.B. Simulation ExecutionIII.B. Simulation ExecutionIII.B.1. Setting the PRNG SeedIII.B.1. Setting the PRNG Seed

III.B.2. Scenario InitializationIII.B.2. Scenario Initialization

III.B.3. Metric CollectionIII.B.3. Metric Collection

III.C. Output AnalysisIII.C. Output AnalysisIII.C.1. Single Set of DataIII.C.1. Single Set of Data

III.C.2. Statistical AnalysisIII.C.2. Statistical Analysis

III.C.3. Confidence IntervalsIII.C.3. Confidence Intervals

III.D. PublishingIII.D. Publishing

III. Common Simulation PitfallsIII. Common Simulation Pitfalls


[ 16 ]

III.A.1. Simulation Type - III.A.1. Simulation Type - terminating vs. steady-

state 66 out of the 114 simulation papers (57.9%) did 66 out of the 114 simulation papers (57.9%) did

not statenot state MANET : steady-state MANET : steady-state statistically unsound resultsstatistically unsound results

III.A.2. Model Validation & VerificationIII.A.2. Model Validation & Verification Simulation modelSimulation model The model must be validated as a baseline to The model must be validated as a baseline to

start any experimentationstart any experimentation Not validating the model or verifying code is a Not validating the model or verifying code is a

common pitfallcommon pitfall

III.A.1. Simulation Type - III.A.1. Simulation Type - terminating vs. steady-

state 66 out of the 114 simulation papers (57.9%) did 66 out of the 114 simulation papers (57.9%) did

not statenot state MANET : steady-state MANET : steady-state statistically unsound resultsstatistically unsound results

III.A.2. Model Validation & VerificationIII.A.2. Model Validation & Verification Simulation modelSimulation model The model must be validated as a baseline to The model must be validated as a baseline to

start any experimentationstart any experimentation Not validating the model or verifying code is a Not validating the model or verifying code is a

common pitfallcommon pitfall

III.A. Simulation SetupIII.A. Simulation Setup


[ 17 ]

III.A.3. PRNG Validation & VerificationIII.A.3. PRNG Validation & Verification NS-2 Cycle length is 2NS-2 Cycle length is 23131-1 -1 Akaroa-2 [11] offers an NS-2 compatible PRNG Akaroa-2 [11] offers an NS-2 compatible PRNG

with a cycle of 2with a cycle of 2191 191 − 1.− 1. 2-dimensional request on a PRNG is valid for 2-dimensional request on a PRNG is valid for

pproximately 8p3 L, where L is the cycle length.pproximately 8p3 L, where L is the cycle length. III.A.4. Variable DefinitionIII.A.4. Variable Definition

There are 538 variables defined in the ns-There are 538 variables defined in the ns-

default.tcl file of NS-2.1b7a and there are 674 default.tcl file of NS-2.1b7a and there are 674

variables defined in the ns-default.tcl file of NS-variables defined in the ns-default.tcl file of NS-

2.27.2.27. Our review of the Tcl driver files show that many Our review of the Tcl driver files show that many

simulation driver files leave key parameters simulation driver files leave key parameters

undefined.undefined.

III.A.3. PRNG Validation & VerificationIII.A.3. PRNG Validation & Verification NS-2 Cycle length is 2NS-2 Cycle length is 23131-1 -1 Akaroa-2 [11] offers an NS-2 compatible PRNG Akaroa-2 [11] offers an NS-2 compatible PRNG

with a cycle of 2with a cycle of 2191 191 − 1.− 1. 2-dimensional request on a PRNG is valid for 2-dimensional request on a PRNG is valid for

pproximately 8p3 L, where L is the cycle length.pproximately 8p3 L, where L is the cycle length. III.A.4. Variable DefinitionIII.A.4. Variable Definition

There are 538 variables defined in the ns-There are 538 variables defined in the ns-

default.tcl file of NS-2.1b7a and there are 674 default.tcl file of NS-2.1b7a and there are 674

variables defined in the ns-default.tcl file of NS-variables defined in the ns-default.tcl file of NS-

2.27.2.27. Our review of the Tcl driver files show that many Our review of the Tcl driver files show that many

simulation driver files leave key parameters simulation driver files leave key parameters

undefined.undefined.



[ 18 ]

III.A.5. Scenario DevelopmentIII.A.5. Scenario Development Only 48 of the 109 MANET protocol simulation Only 48 of the 109 MANET protocol simulation

papers in our survey of published MobiHoc papers papers in our survey of published MobiHoc papers

provided all three of these input parameters, provided all three of these input parameters,

detailing detailing 6161 simulation scenarios. simulation scenarios. #36 & #37#36 & #37 number of nodes : 10 – 30000 Simulation area : 25m x 25m - 5000m x 5000 m. Transmission ranges: 3m - 1061 m.Transmission ranges: 3m - 1061 m. The survey results highlight the wide range of The survey results highlight the wide range of

simulation scenarios used to conduct MANET simulation scenarios used to conduct MANET

research and the lack of uniform rigorous testing research and the lack of uniform rigorous testing

of MANET protocols.of MANET protocols.

III.A.5. Scenario DevelopmentIII.A.5. Scenario Development Only 48 of the 109 MANET protocol simulation Only 48 of the 109 MANET protocol simulation

papers in our survey of published MobiHoc papers papers in our survey of published MobiHoc papers

provided all three of these input parameters, provided all three of these input parameters,

detailing detailing 6161 simulation scenarios. simulation scenarios. #36 & #37#36 & #37 number of nodes : 10 – 30000 Simulation area : 25m x 25m - 5000m x 5000 m. Transmission ranges: 3m - 1061 m.Transmission ranges: 3m - 1061 m. The survey results highlight the wide range of The survey results highlight the wide range of

simulation scenarios used to conduct MANET simulation scenarios used to conduct MANET

research and the lack of uniform rigorous testing research and the lack of uniform rigorous testing

of MANET protocols.of MANET protocols.



[ 19 ]


[ 20 ]

III.A.5. Scenario DevelopmentIII.A.5. Scenario Development We validated the wide range of input parameters We validated the wide range of input parameters

by comparing the by comparing the derived parameters derived parameters of each of each

scenario.scenario. Figure 2 is a scatter plot of all the derived Figure 2 is a scatter plot of all the derived

parameters for the 61 sets of input parameters.parameters for the 61 sets of input parameters. The scatter plot reflects the wide range of The scatter plot reflects the wide range of

scenarios and the scenarios and the lack of correlation lack of correlation between between

parameters. parameters. Figure 2 also shows the Figure 2 also shows the lack of independence lack of independence

between parameters, such as between parameters, such as node density node density and and

node coverage node coverage .. the lack of the lack of multiple groupings multiple groupings in each plot in each plot

illustrates that the community is not covering the illustrates that the community is not covering the

range of values in a consistent organized range of values in a consistent organized

manner.manner.

III.A.5. Scenario DevelopmentIII.A.5. Scenario Development We validated the wide range of input parameters We validated the wide range of input parameters

by comparing the by comparing the derived parameters derived parameters of each of each

scenario.scenario. Figure 2 is a scatter plot of all the derived Figure 2 is a scatter plot of all the derived

parameters for the 61 sets of input parameters.parameters for the 61 sets of input parameters. The scatter plot reflects the wide range of The scatter plot reflects the wide range of

scenarios and the scenarios and the lack of correlation lack of correlation between between

parameters. parameters. Figure 2 also shows the Figure 2 also shows the lack of independence lack of independence

between parameters, such as between parameters, such as node density node density and and

node coverage node coverage .. the lack of the lack of multiple groupings multiple groupings in each plot in each plot

illustrates that the community is not covering the illustrates that the community is not covering the

range of values in a consistent organized range of values in a consistent organized

manner.manner.



[ 21 ]


[ 22 ]


[ 23 ]

III.A.5. Scenario DevelopmentIII.A.5. Scenario Development The MANET community needs a way to The MANET community needs a way to

characterize characterize simulation scenarios simulation scenarios in order to in order to

evaluate and compare protocols and performance evaluate and compare protocols and performance

and ensure protocols are rigorously tested.and ensure protocols are rigorously tested. For example, from Table 2, scenario #8, the For example, from Table 2, scenario #8, the

simulation area is 3000mx 3000 m, but the simulation area is 3000mx 3000 m, but the

transmission range of 1061m lowers the average transmission range of 1061m lowers the average

hop count to only 1.67 hops. This hop count hop count to only 1.67 hops. This hop count

means most source and destination pairs are means most source and destination pairs are

direct neighbors and the rest have only one direct neighbors and the rest have only one

intermediate node.intermediate node.

III.A.5. Scenario DevelopmentIII.A.5. Scenario Development The MANET community needs a way to The MANET community needs a way to

characterize characterize simulation scenarios simulation scenarios in order to in order to

evaluate and compare protocols and performance evaluate and compare protocols and performance

and ensure protocols are rigorously tested.and ensure protocols are rigorously tested. For example, from Table 2, scenario #8, the For example, from Table 2, scenario #8, the

simulation area is 3000mx 3000 m, but the simulation area is 3000mx 3000 m, but the

transmission range of 1061m lowers the average transmission range of 1061m lowers the average

hop count to only 1.67 hops. This hop count hop count to only 1.67 hops. This hop count

means most source and destination pairs are means most source and destination pairs are

direct neighbors and the rest have only one direct neighbors and the rest have only one

intermediate node.intermediate node.



[ 24 ]

III.B.1. Setting the PRNG SeedIII.B.1. Setting the PRNG Seed NS-2 uses a default seed of 12345 for each NS-2 uses a default seed of 12345 for each

simulation runsimulation run Produce identical resultsProduce identical results Negate the independent replication methodNegate the independent replication method None of the 84 simulation papers addressed None of the 84 simulation papers addressed

PRNG issues.PRNG issues.

III.B.1. Setting the PRNG SeedIII.B.1. Setting the PRNG Seed NS-2 uses a default seed of 12345 for each NS-2 uses a default seed of 12345 for each

simulation runsimulation run Produce identical resultsProduce identical results Negate the independent replication methodNegate the independent replication method None of the 84 simulation papers addressed None of the 84 simulation papers addressed

PRNG issues.PRNG issues.

III.B. Simulation ExecutionIII.B. Simulation Execution


[ 25 ]

III.B.2. Scenario InitializationIII.B.2. Scenario Initialization This pitfall usually occurs from a lack of This pitfall usually occurs from a lack of

understanding of the two types of simulation. understanding of the two types of simulation.

terminating or steady-state Only Only eight eight of the 114 simulation papers in our of the 114 simulation papers in our

MobiHoc survey (7.0%) addressed initialization MobiHoc survey (7.0%) addressed initialization

bias, and bias, and all eight use the unreliable method of all eight use the unreliable method of

arbitrarily deleting dataarbitrarily deleting data..

III.B.2. Scenario InitializationIII.B.2. Scenario Initialization This pitfall usually occurs from a lack of This pitfall usually occurs from a lack of

understanding of the two types of simulation. understanding of the two types of simulation.

terminating or steady-state Only Only eight eight of the 114 simulation papers in our of the 114 simulation papers in our

MobiHoc survey (7.0%) addressed initialization MobiHoc survey (7.0%) addressed initialization

bias, and bias, and all eight use the unreliable method of all eight use the unreliable method of

arbitrarily deleting dataarbitrarily deleting data..

III.B. Simulation ExecutionIII.B. Simulation Execution


[ 26 ]

III.C.1. Single Set of DataIII.C.1. Single Set of Data This pitfall is taking the This pitfall is taking the first set of results first set of results from a from a

simulation and accepting the results as “truth”.simulation and accepting the results as “truth”. Only 39 of the 109 MANET protocol simulation Only 39 of the 109 MANET protocol simulation

papers (35.8%) stated the papers (35.8%) stated the number of simulation number of simulation

runs executed.runs executed.

III.C.2. Statistical AnalysisIII.C.2. Statistical Analysis This pitfall concerns not using the This pitfall concerns not using the correct correct

statistical formulasstatistical formulas with the different forms of with the different forms of

output.output.

III.C.1. Single Set of DataIII.C.1. Single Set of Data This pitfall is taking the This pitfall is taking the first set of results first set of results from a from a

simulation and accepting the results as “truth”.simulation and accepting the results as “truth”. Only 39 of the 109 MANET protocol simulation Only 39 of the 109 MANET protocol simulation

papers (35.8%) stated the papers (35.8%) stated the number of simulation number of simulation

runs executed.runs executed.

III.C.2. Statistical AnalysisIII.C.2. Statistical Analysis This pitfall concerns not using the This pitfall concerns not using the correct correct

statistical formulasstatistical formulas with the different forms of with the different forms of

output.output.

III.C. Output AnalysisIII.C. Output Analysis


[ 27 ]

III.C.3. Confidence IntervalsIII.C.3. Confidence Intervals Confidence intervals are a tool to provide a range Confidence intervals are a tool to provide a range

where we think the population mean is located where we think the population mean is located

relative to the point estimaterelative to the point estimate

9898 of the 112 simulation papers using plots of the 112 simulation papers using plots

(87.5%) did not show confidence intervals on the (87.5%) did not show confidence intervals on the

plots.plots.

III.C.3. Confidence IntervalsIII.C.3. Confidence Intervals Confidence intervals are a tool to provide a range Confidence intervals are a tool to provide a range

where we think the population mean is located where we think the population mean is located

relative to the point estimaterelative to the point estimate

9898 of the 112 simulation papers using plots of the 112 simulation papers using plots

(87.5%) did not show confidence intervals on the (87.5%) did not show confidence intervals on the

plots.plots.

III.C. Output AnalysisIII.C. Output Analysis


[ 28 ]

The inconsistency prevents the direct comparison of The inconsistency prevents the direct comparison of

results, limiting research advancements.results, limiting research advancements. A new researcher cannot repeat the studies to start A new researcher cannot repeat the studies to start

his or her own follow-on research.his or her own follow-on research. When publishing, the authors need to state if the When publishing, the authors need to state if the

code is available and how to obtain the codecode is available and how to obtain the code..

The inconsistency prevents the direct comparison of The inconsistency prevents the direct comparison of

results, limiting research advancements.results, limiting research advancements. A new researcher cannot repeat the studies to start A new researcher cannot repeat the studies to start

his or her own follow-on research.his or her own follow-on research. When publishing, the authors need to state if the When publishing, the authors need to state if the

code is available and how to obtain the codecode is available and how to obtain the code..

IV.D. PublishingIV.D. Publishing


[ 29 ]

The The Akaroa-2Akaroa-2 suite, which help a researcher suite, which help a researcher

monitor simulation execution to determine steady-monitor simulation execution to determine steady-

state and prevent correlation among multiple state and prevent correlation among multiple

replications of a simulation.replications of a simulation. The interactive NS-2 protocol and environment The interactive NS-2 protocol and environment

confirmation tool (confirmation tool (iNSpectiNSpect) , which ) , which visualizesvisualizes the the

trace file of an NS-2 simulation. The visualizations trace file of an NS-2 simulation. The visualizations

can be used for scenario development, model can be used for scenario development, model

validation, protocol verification, and results analysis.validation, protocol verification, and results analysis. The Simulator for Wireless Ad Hoc Networks The Simulator for Wireless Ad Hoc Networks

(SWAN) , which enables a researcher to create a (SWAN) , which enables a researcher to create a

virtual environment for conducting experiments with virtual environment for conducting experiments with

MANETs.MANETs.

The The Akaroa-2Akaroa-2 suite, which help a researcher suite, which help a researcher

monitor simulation execution to determine steady-monitor simulation execution to determine steady-

state and prevent correlation among multiple state and prevent correlation among multiple

replications of a simulation.replications of a simulation. The interactive NS-2 protocol and environment The interactive NS-2 protocol and environment

confirmation tool (confirmation tool (iNSpectiNSpect) , which ) , which visualizesvisualizes the the

trace file of an NS-2 simulation. The visualizations trace file of an NS-2 simulation. The visualizations

can be used for scenario development, model can be used for scenario development, model

validation, protocol verification, and results analysis.validation, protocol verification, and results analysis. The Simulator for Wireless Ad Hoc Networks The Simulator for Wireless Ad Hoc Networks

(SWAN) , which enables a researcher to create a (SWAN) , which enables a researcher to create a

virtual environment for conducting experiments with virtual environment for conducting experiments with

MANETs.MANETs.

IV. Community ResourcesIV. Community Resources


[ 30 ]

Summarizing the four areas of credibility, we found Summarizing the four areas of credibility, we found

less than less than 15%15% of the published MobiHoc papers are of the published MobiHoc papers are

repeatable.repeatable... Only seven of the 58 MobiHoc simulation papers that Only seven of the 58 MobiHoc simulation papers that

use a public simulator (12.1%) mention the simulator use a public simulator (12.1%) mention the simulator

versionversion used. used. Only Only eight eight of the 114 simulation papers (7.0%) of the 114 simulation papers (7.0%)

addressed addressed initialization bias initialization bias and none of the 84 and none of the 84

simulation papers addressed random number simulation papers addressed random number

generator issues.generator issues. statistical soundness, statistical soundness, 70 70 of the 109 MANET protocol of the 109 MANET protocol

simulations papers (64.2%) did not identify the simulations papers (64.2%) did not identify the

number of simulation iterations number of simulation iterations used, and 98 of the used, and 98 of the

112 papers that used plots to present simulation 112 papers that used plots to present simulation

results (87.5%) did not include results (87.5%) did not include confidence intervals.confidence intervals.

Summarizing the four areas of credibility, we found Summarizing the four areas of credibility, we found

less than less than 15%15% of the published MobiHoc papers are of the published MobiHoc papers are

repeatable.repeatable... Only seven of the 58 MobiHoc simulation papers that Only seven of the 58 MobiHoc simulation papers that

use a public simulator (12.1%) mention the simulator use a public simulator (12.1%) mention the simulator

versionversion used. used. Only Only eight eight of the 114 simulation papers (7.0%) of the 114 simulation papers (7.0%)

addressed addressed initialization bias initialization bias and none of the 84 and none of the 84

simulation papers addressed random number simulation papers addressed random number

generator issues.generator issues. statistical soundness, statistical soundness, 70 70 of the 109 MANET protocol of the 109 MANET protocol

simulations papers (64.2%) did not identify the simulations papers (64.2%) did not identify the

number of simulation iterations number of simulation iterations used, and 98 of the used, and 98 of the

112 papers that used plots to present simulation 112 papers that used plots to present simulation

results (87.5%) did not include results (87.5%) did not include confidence intervals.confidence intervals.

V. ConclusionsV. Conclusions


[ 31 ]

Hence, only approximately Hence, only approximately 12%12% of the MobiHoc of the MobiHoc

simulation results appear to be based on simulation results appear to be based on sound sound

statistical statistical techniques.techniques. This survey of MobiHoc papers showed the current This survey of MobiHoc papers showed the current

state of MANET research and the state of MANET research and the lack of consistencylack of consistency, ,

re-enforcing the need for simulation study guidance.re-enforcing the need for simulation study guidance. The goals is to make the research repeatable, The goals is to make the research repeatable,

unbiased, rigorous, and statistically soundunbiased, rigorous, and statistically sound

Hence, only approximately Hence, only approximately 12%12% of the MobiHoc of the MobiHoc

simulation results appear to be based on simulation results appear to be based on sound sound

statistical statistical techniques.techniques. This survey of MobiHoc papers showed the current This survey of MobiHoc papers showed the current

state of MANET research and the state of MANET research and the lack of consistencylack of consistency, ,

re-enforcing the need for simulation study guidance.re-enforcing the need for simulation study guidance. The goals is to make the research repeatable, The goals is to make the research repeatable,

unbiased, rigorous, and statistically soundunbiased, rigorous, and statistically sound

V. ConclusionsV. Conclusions

[ op_ research @ im.ntu ]

Documents