1 15 ismor - 01/09/2014 centre for defence analysis british crown copyright 1998/dera rigorous...

1

15 ISMOR - 11/04/23 Centre for Defence AnalysisBritish Crown Copyright 1998/DERA

Rigorous Subjectivity How will OA survive the “soft” revolution?

Graham L MathiesonCentre for Defence Analysis

Defence Evaluation and Research AgencyPortsdown West, Fareham, Hants., UK

e-mail: [email protected]

15th International Symposium on Military Operational ResearchRMCS Shrivenham, September 1998

2


Agenda

What’s the problem? Problem space and analysis methods Objectivity and quantification Adding rigour to multi-criteria analysis

3


What’s the problem?

Complexityof

problemspace

Breadth of problem space

Hard systemsmodelling

4


Complexity and Scope

Jointness and Combinedness Integration and digitization Spectrum of conflict Duration of operations Multiplicity of dimensions

5


Methods and Problem Spaces

Breadth of problem space

Complexityof

problemspace

Hard systemsmodelling

Soft systemsanalysis

Multi-criteriaanalysis

Exploratorymodelling

6


So, what’s the real problem?

7


Facilitative decision support

Partial modelreflecting DM’s mind

8


“Off-line” decision analysis

Complete modelincorporating

DM understanding

9


“Off-line” decision analysis

Complete modelincorporating

DM understanding

Verification&

Validation

10


Operational Analysis = Rigour

Repeatability

Independence

Grounding in reality

Objectivity of process

Uncertainty in data

Robustness of results

11


Objectivity Quantification

12



SubjectiveQualification

ObjectiveQuantification

13



SubjectiveQualification

ObjectiveQuantification

SubjectiveQuantification

ObjectiveQualification

15


Multi-criteria analysis (MCA)

16


Common problems with MCA

Measures of value which are not “real” Bias in subjective preference scores Unrepeatable, “one-shot” studies Inability to represent dynamic interdepency

Methodological improvements to rigour of MCA Causal benefits modelling Development of “real” scales Explicit uncertainty modelling Compensating for interdependencies

17


Benefits modelling

Investments

Value metrics

How do we benefit fromwhat we invest in?

19


Generic Benefits Model Structure

Investments

Value metrics

System functions and attributes

Organisation activities and attributes

Military capabilities and attributes

“...and attributes” is important!

Causal modeling is essential!

21


faster &

better informed

decision

efficient

allocation

of forces

efficient

allocation

of assets

responsive

logistics

situational

awareness

mission

rehearsal aids

ability to

send/receive

information

accurate, timely

& appropriate

recognised picture

asset

tracking

system

interoperability

minimise

data interchange

requirements

complete, timely

& accurate

information

Causal mapping (a fragment from a C3I study)

22


Valuing a digitised system of systems

ORGANISATION ACTIVITIES/ATTRIBUTES

Information captureInformation exchangeInformation processingDecision makingProcess interoperabilityProcess flexibility

MILITARY CAPABILITIES

SimultaneityConcentration of force

- situation awarenessConcentration of force

- synchronisationFlexibility of

dispositionFlexibility of capabilitySpeed of decisionInteroperabilitySustainabilityEfficiency / economy

of effortReach

VALUE SYSTEM

ManoeuvreFirepower /

attritionCoercionSurvivabilityPresence

23


FOR ALL EQUIPMENTS

Improve– Stretch potential– Reliability– Robustness /– Survivability– Time to service– Value for money– Interopability

Reduce– Cost– Slippage– Maintenance– Mass– Vulnerability– Need for support in

the field

From a presentation by Lt General Burton, DCDS(S), July 1997

25


Using “real” scales

“The process of determining values for criteria weights calls for a lot of hard thinking on the part of the decision-maker” (R.L.Pratt, V.Belton).

Off-line analysis do not have this luxury. Therefore, scales at each stage in the model

must be meaningful so that scores are open to independent verification.

26


Using “real” scales

“The process of determining values for criteria weights calls for a lot of hard thinking on the part of the decision-maker” (R.L.Pratt, V.Belton).

Off-line analysis do not have this luxury. Therefore, scales at each stage in the model

must be meaningful so that scores are open to independent verification.

0.0 1.0

27


Uncertainty and sensitivity

Study related to a particular HQ C2 system Benefits modelling used to establish the overall

requirement and to compare options in a COEIA Multi-stage benefits model...

28


Scenario MOEs

Dependency andvalue

measures

Dependencyand

valuemeasures

C2I System

Non-CIS

non

-C2I

MoP

s(t

ext

base

d)

C2I

syst

em

MoP

s

Mis

sion

MO

Es Mission

Weightings

C2I fns

Other CISinvestments

Stage 1: Systemfunctions and attributes

Stage 2: Organisationalfunctions and attributes

Stage 3: MilitaryCapabilities

Stage 4: Value System

Benefits model used for a C2 system

29


Maths from a typical Benefits Model

OEMoE

ScenkScen

kkScen

kScen

k

30


Maths from a typical Benefits Model

OE

kScen

k

kScen

MoEk

ScenkScen

22

2 21

14

2

2

kScen

k

kScen

kScen

kkScen

kScen

kk

MoE MoEkScen

OEMoE

ScenkScen

kkScen

kScen

k

32


Interdependence of criteria

Benefits modelling, like other MCA methods, is basically a set of linear filters.

Non-linearities in individual causal chains can be included.

Non-linearities due to criterion interdependence are more problematic.

A partial solution is the additional of first order corrections via interdependency matrices.

33


Generalised BM transition(based on Quality Function Deployment)

Input categories

Ou

tpu

t ca

teg

ori

es

Dependencymatrix

Interdependency matrix

34


Linking to dynamic modelling

Causal mapping leads naturally on to dynamic network modelling techniques such as System Dynamics or Petri-Nets.

Benefits modelling can be used to identify reduce the scope of the problem by identifying driving parameters.

In the C2 system study, it was concluded from the benefits analysis that timeliness was not a driver and, thus, dynamic modelling would not be cost-effective.

35


Summary

“Soft” analysis methods are firmly established in OA because they can cope with the breadth and complexity of the problem space.

Many methods have arisen from facilitative decision support and have limited rigour.

Without rigour, OA is nothing but consultancy. The rigour of “soft” methods can be improved.

OA can survive the “soft” revolution...

...but only if it adopts rigorous subjectivity.

36


Questions?

1 15 ismor - 01/09/2014 centre for defence analysis british crown copyright 1998/dera rigorous...

Documents

dera operational analysis

multicriteria analysis

analysis methods zobjectivity

dera complexity

dera methods

dera whats

mathieson centre

dera rigorous subjectivity