dr paul hubbard drdc ottawa presented by: nacer abdellaoui october 13, 2011 for nato msg-087:...

Dr Paul HubbardDRDC Ottawa

Presented by:Nacer Abdellaoui

October 13, 2011

For NATO MSG-087: Enhance or Replace – Finding the Right Live vs. Synthetic Balance

BERN - SWITZERLAND

Measuring the Appropriateness of Live and Simulation

2

1. Why is this hard? Complicating Factors

2. Emergence of LVC

3. Indicators for when to use Live vs Simulation

4. Implications and Conclusions

Outline

3

Why is it hard to decide between L, V and C?

• More options than before on how to do experimentation and training

• Lack of knowledge about capability of virtual and constructive

• Evolving simulation technology (sophistication)

• Changing cost estimates, initial investments and maintenance costs

• Cultural reliance on live exercises

V

L

C

4

Emergence of LVC

• LVC offers a taxonomy of M&S (see the excellent wikipedia page)

• Traditionally, simulation supports live, e.g. mission rehearsals, Hardware-in-the-loop

• Appears to be a good marriage for an experimental campaign:

• Live has high validity

• Simulation has high reproducibility

• Offers interesting possibilities such as augmented reality

V or C?

5

Philosophy

• Along the lines of Donald Rumsfeld*:

simulation is good at turning known unknowns into known knowns,

simulation is not good at turning unknown unknowns into known unknowns,

*Rumsfeld, D. Press conference by US Secretary of Defence at NATO HQ, 6 June 2002, http://www.nato.int/docu/speech/2002/s020606g.htm.

L or V?

http://www.nato.int/docu/speech/2002/s020606g.htm

6

Indicators to support selection of LVC

• A workshop at DRDC Ottawa sought to answer:

How should live and SE experimentation be coordinated?

• Developed 32 indicators with name, relevant question, and suitability

Indicator of Utility

ID # Questions/Issues with respect to the Measure

Comments upon Dominant SuitabilityVery suitable for SE (VSE), Suitable to SE (SE), Equal suitability between SE and Live (B), Suitable to Live (L), Very suitable to Live (VL)

Iteration Ease 1 Are multiple iterations required for the experiment?

SE experiments are ideally suited, subject to computational complexity, to multiple iterations. For Live experiments this is challenging due to uncontrollable events. – VSE

7

How to use the following table of indicators

If indicator and question are of high value to your experiment, then the assessment indicates a recommended use, i.e. VSE implies simulation more appropriate.

Caveat: assessments were made in context of a large trial, rather than a tightly controlled experiment. The table itself needs to be validated.

8


Indicator of Utility ID # Questions/Issues with respect to the Measure


Iteration Ease 1 Are multiple iterations required for the experiment?

SE experiments are ideally suited, subject to computational complexity, to multiple iterations. For Live experiments this is challenging due to uncontrollable events. – VSE

Controllability 2 To what extent do external variables or events need to be controlled?

SE trials can be completely scripted, but when there are humans in the loop, i.e. virtual rather than constructive simulation, branching in the scenario execution still occurs. Live trials attempt to follow a script, but are much more susceptible to uncontrollable events such as weather can affect scripts. – B

Credibility 3 Is it important for the results and conclusions to appear credible by decision-makers?

If both experiments are successful, results of Live will be seen as more credible. If both experiments fail then failure for the SE experiment will generally be viewed as a problem in design/implementation or fidelity; whereas failure in a Live experiment is generally attributed to issues outside of experimental control (e.g., weather) and results will not be viewed as pessimistically. Staff may choose live experiments for this reason. – L

9




Ground Truth 5 Is ground truth data required to support evaluations or calculation of metrics in the experiment? (note this is not a validity question, but simply the availability of the data deemed “ground truth”.)

Ground truth is generally considered available for SE, when relative measures are needed. All data is considered available in principle either through data recording or playback. Ground truth data may not be available as readily for complicated Live trials that include multiple platforms. – SE

Fidelity 6 Does the experiment need to match reality?

Considered variable for SE – though not always necessary, if, for instance, the experiment is based on a fictitious future world with conceptual systems. Increasing the fidelity requires more modelling, cost and effort. Live experiments are assumed to be real and therefore maximum fidelity. – VL

Cost 7 What is the cost comparison of an SE vs. Live trial?

For total costs of a single experiment, these are assessed as equal for SE and Live, due to the potentially high development cost for SE experiments. – B. However, for subsequent repetitions and when re-use in another trial is considered, there is an advantage to SE. – VSE

10




Repeatability 8 Is it important that a repeated version experiment give identical results? Or slight variations (as in Monte Carlo simulation)?

Computer components can be repeated deterministically if identical results are needed, which is almost impossible in Live experiments – VSE, However, when controlling variables, live trials can result in only slight variations trial-to-trial. – SE

Safety 10 Is human safety and risk a key requirement for the experiment?

Assessed as distinct advantage to SE, both in safety to participants and the ability to test risky operational scenarios. – VSE

Ethics 11 Which of live experiments or simulation enables assessment of a broader ethical spectrum?

The live trial spectrum is limited, e.g. nuclear effects or explosive echo ranging cannot be tested easily. Ethical dilemmas can be posed to participants as easily in an SE experiment. – VSE

Environmental Impact

12 Which form of experiment has lower environmental impact?

For SE, impact comes from power consumption for manufacturing and at run-time, as well as obsolescent equipment waste. This is considered minimal in comparison to Live trials, that may damage ecosystems and generate industrial-scale debris. – VSE

11




Collateral Training

15 Is there ancillary training that was not designed in experiment?

Assumed unlikely in a SE, apart from machine learning, whereas highly likely mission-relevant training occurs within Live trial. – L

Time Compress-

ibility

16 Is it important to compress time in this experiment? i.e. skip long transits or operational delays.

Not possible in Live trial – SE is well suited. – VSE

Validation 22 To what extent is the experiment valid, or valid to a given specification?

For SE, validation can only be done based on an input specification. Matching to reality requires SME assessment, or computational comparison to live experimental data. Live trials are considered “real” so deemed valid for that moment in time and experimental circumstances. – VL

Obsolescence of Result

26 How long will the results be of utility?

More dependent on the content of the experiment. – B

12




Observability 29 How hard is it to observe (key) events?

Due to availability of ground truth data, observation of key events may be done with simple code additions. In live experiments, key events may be obscured unless explicitly accounted for. VSE

Data Quality 30 How good is the data in terms of format, persistence, and coverage

May be easier to plan for and conceptualize in an SE trial. Also, there is the restart potential to restart SE experiments with additional data recording. In live trials, you “get what you get”, but if well planned, can lead to the ideal data set. – B

Sensitivity to Fraud

32 To what extent can the experiment be faked or misinterpreted?

Assumed easier to manufacture results in SE trial. Also inadvertent manipulation of SE and results possible. – VL

13

Discussion and Implications

1. Cultural preference for live experiments / training still exists - perceived credibility / validity / acceptability all assess in favour of LIVE

2. Live experiments / training already perceived as more expensive.

3. Validity and reproducibility are polarities

4. Collateral training in a live military exercise not to be undervalued.

14

Conclusion

• A score-card approach to support the decision of simulation vs live was presented

• This will remain a tricky question for some time because the decision space is evolving:

• Sims are more sophisticated,

• Fuel and manufacturing costs increasing

• Parting Thought:

• Should LVC really be LVCA? A=Autonomous = “M&S involving simulated people working real equipment.”

dr paul hubbard drdc ottawa presented by: nacer abdellaoui october 13, 2011 for nato msg-087:...

Documents

live experiments

live b

se experiments

results of live

se vse

appropriateness of live

se trials

live vs simulation