babes in the woods: how naïve analysts clashed with trained killers to the mutual benefit of all

One Army, One Team, One Vision Operational Research

Une Armée, une équipe, une vision Recherche opérationnelle

Babes in the Woods:Babes in the Woods:How Naïve Analysts Clashed with Trained KillersHow Naïve Analysts Clashed with Trained Killers

to the Mutual Benefit of Allto the Mutual Benefit of All

Babes in the Woods:Babes in the Woods:How Naïve Analysts Clashed with Trained KillersHow Naïve Analysts Clashed with Trained Killers

to the Mutual Benefit of Allto the Mutual Benefit of All

Fred CameronFred CameronOperational Research AdvisorOperational Research Advisor

to the Director General Land Capability Developmentto the Director General Land Capability Development



Defence Research and Development Canada

Kingston, Ontario

Directorate ofLandSyntheticEnvironments



Land Capability Development Operational Research Team

Land Capability Development Operational Research Team

• Mr Fred Cameron (Team Leader)• Mr Roger Roy• Ms Eugenia (Jenny) Kalantzis• Mr Ian Chapman • Mr François Cazzolato• Maj Bruce Chapman (to join July 2006)

• The Army Experimentation Centre– Part of the Directorate of Land Synthetic Environments (Army’s home for

Computer-based Wargames, Models, and Simulations for Training and Experimentation)

• Strategic Analyst – Mr Peter Gizewski• Science Advisor – Mr Regan Reshke



Deciding Which Method to Use – A Meta-Decision

Deciding Which Method to Use – A Meta-Decision

• A meta-model of the meta-decision process• An analysis:

– Description• What happened?

– Ascription• Were there causes and effects?

– Prescription• What should we do in future?

– Proscription• What should we avoid in future?



Cameron’s Rubbish Bin Model of Decision Making

Cameron’s Rubbish Bin Model of Decision Making

• Born of many years of observation• Employed for organizations that

are “organized anarchies”• Basis for workshop on “Ambiguity

and Command” in 1986• Used provocatively:

– Is it really that bad?

Sources: Cohen, March, and Olsen (1972), March and Weissenger-Baylon (1986), and March (1994)

March’s Garbage Can ModelMarch’s Garbage Can Model



March’s Garbage Can Model of Decision Making


Complex interactions between:

• Actors (decision makers)

• Problems

• Choice opportunities

• Potential solutions





• Some ways to “improve” garbage-can decision making– Increase the heat and pressure

• Force more frequent interactions

– Put more garbage in the can• Add more potential solutions• Add more problems, more actors,

more decision opportunities too?

– Make actors “stickier” • Hope that they will carry problems or

potential solutions with them long enough to get a better fit



DescriptionDescription

• “Deciding Which Method to Use”– OR analysts make great critics of the decision-

making of others– But, turn the lens into a mirror

• The Analyst’s Toolbox… Our Experience

• Canadian Army Modelling and Simulation

• Specific Examples and Lessons



The Operational Research Tool BoxSome Examples

The Operational Research Tool BoxSome Examples

• Structured brain storming• Scenarios• Analysis of historical data • Options analysis, Decision analysis• Group ranking• Seminar war games• Analytical tools – mathematical analysis• Computer-based war games (JANUS, OneSAF)• Trials, exercises, and evaluations

– Source: Future Army Development Plan, 1998

More:• time• resources• credibility

Less:• time• resources• credibility

Soft

OR/OAMethods

Hard



Our Early Meta-decision Rules for “Deciding Which Method to Use”

Our Early Meta-decision Rules for “Deciding Which Method to Use”

• If time and resources are short, and credibility is not an issue, then use soft OR methods

• If credibility is paramount, and time and resources are available, then use hard OR methods

• Our experience supports: “hard and soft methods can be used together at appropriate stages of a study (usually soft for problem formulation, hard for resolution)”



Simulation TypesSimulation TypesPeople

Equipment

Real

LiveReal

Simulated

Virtual ConstructiveSimulated



Goal Synthetic EnvironmentGoal Synthetic Environment

Live

Constructive

Virtual

Joint War Game

Simulators Simulators Simulators

Army Navy Air Force Operational

Army Navy Air Force Tactical

Link

Link

Link

Link

C2

Sys

tem

s

Rea

l Wor

ld Link Link

Link Link Link

Link Link Link



Constructive SimulationsConstructive Simulations

• Command and Staff Trainer• UK’s ABACUS



Constructive SimulationsConstructive Simulations



Lesson 1: AE-7B* Armed Griffon HelicopterLesson 1: AE-7B* Armed Griffon Helicopter• Aim. To provide insights into differences in effectiveness

and survivability between an armed and an unarmed helicopter

• Purpose. To validate the incorporation of aerial firepower requirements in the CH-146 Griffon Helicopter mid-life upgrade

• Simulation Structure. Constructive and virtual computer-based simulation supported by OneSAF Testbed Baseline (OTB), with virtual workstations for Griffon helicopters and Pointer-type Unmanned Air Vehicles (UAV)

* AE = Army Experiment



Lesson 1: AE-7B Armed Griffon HelicopterLesson 1: AE-7B Armed Griffon Helicopter

Constructive

Virtual

Tactical

Griffon

DIS

OTBSAF

Pointer

DIS



AE-7B – The Textbook CaseAE-7B – The Textbook CaseJu

n 02

Jul 0

2

Aug

02

Sep

02

Oct

02

Nov

02

Jan

03

Feb

03

Mar

03

Apr

03

Dec

02

Define

Develop

Conduct

Analyze

Report

Weak Synthetic Environment SpecificationLate Terrain Generation

Late GriffonSimulation Delivery

Late Scenario Build

Strong Sponsor Involvement

Excellent Participant Base

Automated Small Sample Statistics

Corrective Action• Implemented Synthetic Environment Statement of Requirement



Lesson 2: LOE* 0301 – TUAV in Controlled AirspaceLesson 2: LOE* 0301 – TUAV in Controlled Airspace

• Aim. To investigate possible reduced airspace restrictions for UAVs if Air Traffic Controllers have better situational awareness

• Objective. To identify any differences in situational awareness enabled by various improvements to ATC situational awareness (in the vicinity of Kabul)

• Simulation Structure. Constructive and virtual computer-based simulation supported by OneSAF Testbed Baseline (OTB), with virtual workstations for ATC Tower, Air Defence and ATC radars, and UAV

* LOE = Limited Objective Experiment



Lesson 2: LOE 0301 – TUAV in Controlled AirspaceLesson 2: LOE 0301 – TUAV in Controlled Airspace

Constructive

Virtual

Tactical

Tower Ops

DIS

OTBSAF

TUAV

DIS

AD Rdrs/EOs

DIS

ATC Rdr

DIS



LOE 0301 – The Short CircuitLOE 0301 – The Short CircuitM

ay 0

3

Jun

03

Jul 0

3

Aug

03

Sep

03

Oct

03

Nov

03

Define

Develop

Conduct

Analyze

Report

Staff Check

Sponsor Identification

Problem Identification

Terrain Incomplete

Iterative Prototyping

Sponsor Participant Access

Automated Situation Awareness Analysis Tools

Report Pre-Preparation

Corrective Action• Create Terrain/Visualization Cell



AscriptionAscription

• Does the Garbage Can Model fit? Is it really that bad?

• Decision outcomes driven by– Temporal Confluence: Decision makers,

Choice opportunities, Problems, Potential solutions

• Is the method we use for “deciding which method to use” any better that that?



Toolbox• Structured brain storming• Scenarios• Analysis of historical data • Options analysis, Decision

analysis• Group ranking• Seminar war games• Analytical tools –

mathematical analysis• Computer-based war games

(JANUS, OneSAF)• Trials, exercises, and

evaluations

The Operational Research Skill SetThe Operational Research Skill SetSkill set• Facilitator• Scribe• Co-author of scenarios

• Enumerator of votes• Analyst of preferences

• Modeller• Mathematician/Statistician

• Experimenter in simulation space

• Experimenter in live trials

Soft

OR/OAMethods

Hard



Roles of Soft OR MethodsRoles of Soft OR Methods

• To support thinking and planning by an individual analyst or decision maker – problem articulation

• To support discussion between consultant and decision maker – problem negotiation

• To support debate and conclusion among decision makers – group decision support

• To initiate or strengthen organisational capabilities – organisational development

– Source: Steve Cropper presentation in Holt and Pickburn (2001)



PrescriptionPrescription

• Consider the full spectrum in the tool box

• Prepare for the full range of required skills

• Implement a “lessons” process, and strive to improve



Application of the Spectrum of MethodsApplication of the Spectrum of Methods• Soft and hard OA should be viewed as equally useful and

part of the analyst’s toolkit. These should be viewed as a spectrum of options within the analyst’s toolkit which should be selected and used as appropriate.

• It was recognised that hard and soft methods can be used together at appropriate stages of a study (usually soft for problem formulation, hard for resolution).

• Analysts need to be aware of all OA techniques and, broadly where they should and should not be employed. Better education may help to achieve this.

• An expert practitioner (or team) is needed with expertise in the range of methods appropriate to the problem.

– Extracts from Holt and Pickburn (2001), pp. 15, 17 and 19



ProscriptionProscription

• Woe, woe, and thrice woe

• Beware of complexity



Err

or

Complexity

SpecificationError

Minimize Model Error

Err

or

Complexity

MeasurementError

Models, Complexity and ErrorModels, Complexity and Error

Err

or

Complexity

Model Error



Err

or

Complexity

SpecificationError

Err

or

Complexity

Model Error

Err

or

Complexity

MeasurementError

If we can increase the accuracy of performance

characteristics, we can accommodate greater complexity.


Overall error drops.



Err

or

Complexity

SpecificationError

Err

or

Complexity

Model Error

Err

or

Complexity

MeasurementError

Minimize Model Error by

Decreasing Complexity




Main RecommendationMain Recommendation

• Make the actors “stickier”:– Analysts need to be aware of all OA techniques

and, broadly where they should and should not be employed. Better education may help to achieve this

– An expert practitioner (or team) is needed with expertise in the range of methods appropriate to the problem

Source: Holt and Pickburn (2001)



ReferencesReferences

• Michael D. Cohen, James G. March, Johan P. Olsen “A Garbage Can Model of Organizational Choice” Administrative Science Quarterly, Vol. 17, No. 1. (Mar., 1972), pp. 1-25

• James G. March and Roger Weissenger-Baylon eds. (1986) Ambiguity and Command: Organizational Perspectives on Military Decision-Making. Marshfield, MA Pitman Publishing Inc.

• James G. March (1994) A Primer on Decision Making: How Decisions Happen. New York: The Free Press

• Future Army Development Plan (1998) Kingston, Ontario: Directorate of Land Strategic Concepts

• John Holt and George Pickburn (2001) OA Techniques for the Future. Farnborough, UK: Defence Evaluation and Research Agency. 30 March 2001



Questions – DiscussionQuestions – Discussion

Fred Cameron

Tel: +1.613.541.5010 ext 2470

Email: [email protected]

babes in the woods: how naïve analysts clashed with trained killers to the mutual benefit of all

Documents