0 0 exploring a range of decision making techniques nov 18,2009 john bates ( [email protected] )...
TRANSCRIPT
22
The Range of Decision Making Techniques
Considerations when selecting a technique…
How important is the Decision; who will review decision? How much data/information do you have or can you get? How much time and $ do you have?
Discussion
Group (BOGSAT
)
Poorly Done
Decision Analysis
Decision Analysis
Incorporating Mod & Sim
Decision Analysis
Incorporating SMEs
Coin Tossor
Guess
Least Desirable - Most DesirableAdvantages/Disadvantages to each technique.
Fielded system;
record data throughout lifecycle.
OperationalExercise and
Testing
33
Symptoms of poorly done decision analysis
Lack of creative and significant Alternatives
Solving the Wrong Problem/Objective
Not involving the real decision makers
Competing Objectives
Avoiding Uncertainty
Success measured on weight and thickness of final brief
Not sure about the definition of the attributes
Provided the solution at the beginning
44
How important is your decision?
55
Recommend a structured process that includes all stakeholders and is traceable/defendable
• Scoring scales can be nominal, ordinal, interval, or ratios
• Utility curves transform scoring scales into a common measurable output (Utility)
• Weighting of the tree structure determines relative importance
• There are various methods to determine importance
• AHP
• Relative Weighting
• Overall Goal: Determine best solution to fill need
• Sub-Goals:
• Maximize Technical Performance, Measures of Effectiveness
• Maximize Operational Suitability
• Minimize Risks, etc.
• Criteria must be
independent, discriminatory, meaningful, and measurable
• Assess each alternative relative to the criteria in the decision tree
• Utility curves then transform the scoring scales into measurable data ranging from 0 to 1 with 1 being the most preferred
• Rank ordering of alternatives can be further decomposed to determine major contributors to the overall decision
• Conduct estimates for each alternative
• Can use cost objectives and thresholds
• Cost estimates and be treated separate or added into the evaluation model
• Sensitivity analysis can be conducted to analyze the results due to:
• Changes in weights
• Changes in scores
• Utility scores can be can be graphed versus cost to determine the "Best Value" alternative
• Determine Requirements
• Identify/Clarify Goal
• Define Criteria
1 • Determine Priorities
• Develop Utility Curves
• Develop Scoring Scales
2• Conduct
Sensitivity Analysis
• Determine “Best Value”
5
• Assess Alternatives
• Rank Order Alternatives
4• Generate Alternatives
• Develop Estimates
• Incorporate Estimates into Model
3
Understand Problem/Vision Modeling Decide and Communicate
66
Prioritization (Weighting) TechniquesAnalytic Hierarchy Process
Speed Reliability Length Range sum row avgSpeed 1 3 1 3 0.375 0.705 0.107 0.188 1.375 0.344
Reliability 1/3 1 7 9 0.125 0.235 0.750 0.563 1.673 0.418Length 1 1/7 1 3 0.375 0.034 0.107 0.188 0.703 0.176Range 1/3 1/9 1/3 1 0.125 0.026 0.036 0.063 0.249 0.062
2.667 4.254 9.333 16.000 1.000 1.000 1.000 1.000 4.000 1.000
Weightings To
Prioritize User
NeedsCalculation:
1/ 2.667= 0.375Sum Rows Calculate Row Average
(Called the Row Average of the
Normalized Columns (RANC) method)
Assign Scores Using a Scale
Sum Column
Developed by Thomas L. Saaty in the 1970s. The AHP provides a comprehensive and rational framework for structuring a decision problem, for representing and quantifying its elements, for relating those elements to overall goals, and for evaluating alternative solutions.
• Determine Priorities
• Develop Utility Curves
• Develop Scoring Scales
2
Importance
1 Equal
3 Moderate
5 Strong
7 Very Strong
9 Extreme
77
Capturing SME input
Participants will be asked to evaluate criteria through a series of pairwise comparisons, which calculates each factor’s relative importance.
To capture these comparisons, each participant uses a remote control to assign a number (1-9) that reflects the relative weight or importance of each decision factor.
Criteria #1 Criteria #2
1 Intensity of Importance
2 3 4 5 6 87 99 8 7 6 5 34 2
Speed Reliability Length Range sum row avgSpeed 1 3 1 3 0.375 0.705 0.107 0.188 1.375 0.344
Reliability 1/3 1 7 9 0.125 0.235 0.750 0.563 1.673 0.418Length 1 1/7 1 3 0.375 0.034 0.107 0.188 0.703 0.176Range 1/3 1/9 1/3 1 0.125 0.026 0.036 0.063 0.249 0.062
2.667 4.254 9.333 16.000 1.000 1.000 1.000 1.000 4.000 1.000
• Determine Priorities
• Develop Utility Curves
• Develop Scoring Scales
2
# Inputs = (N * N-1)/2
88
Methodology for collecting group input
Pairwise Comparisons ensure all combinations are considered independently
The Analytic Hierarchy Process is used to determine the criteria
priority. Pairwise comparisons are made for each unique combination.
During the session, the facilitator will focus on areas of disagreement by identifying divergent points of view and encouraging participants to elaborate.
Participants are encouraged to state their opinions and change their minds – and votes – if they so desire. During facilitated discussion, participants may continue to disagree on the relative importance of different decision factors but they will have a better understanding of different views within the group.
• Determine Priorities
• Develop Utility Curves
• Develop Scoring Scales
2
99
Decision Analysis at multiple levels?
Mission Areas
UIC (ASSIGNED TO RESPORG)
Capability
Area 1Capability
Area 2
Capability Area 3
PESTO Resource Elements
Capability 2 Capability 3Capability 1
O
T
S
E
P
NTA2NTA1
Joint Capability Areas
(Future)
NUMBER OF INPUTS
REQUIRED
~10
~50
~250
~3000
• Determine Priorities
• Develop Utility Curves
• Develop Scoring Scales
2
1010
Relative Scoring Example
Solution Mission A Formula % Importance
Car 1 =1/4 25%
Tank
Plane 3 =3/4 75%
Boat
TOTAL 4 100%
Criteria Scale
Supports Mission Success 1
Important to Mission Success 3
Critical to Mission Success 9
• Determine Priorities
• Develop Utility Curves
• Develop Scoring Scales
2
1111
Expert Choice – Great discussion/consensus using clickers, takes a lot of time
Group Systems – Great at capturing multiple user input simultaneously , can be performed remotely (surveys), requires extensive setup.
Excel AHP – Used to Prioritize Resource Categories within a level - Simple to setup, easy to facilitate potential for a lot of intersections ((n * n-1)/2), lose input variation and correlation checking
Excel (0,1,3,9) Scale – Simple, quick, needs to be strongly facilitated, lose input variation and consistency check
The choice of techniques is primarily based on number of intersections and time needed to perform the analysis
TIME TO COLLECT
Summary of Different Prioritization Techniques
1212
Scaling Examples for Alternative Scoring
Utility
Signature Improvement in undetectable range (meters)
1
0
0 1000
Utility
Range (meters to Refuel)
1
0
2142 6000
The range for the Measure of Merit for Signature is based on the improvement in detection range the vehicle realizes through IR signature reduction.
The range of the Measure of Merit for Range is based on the ORD Requirement on meters to recognition.
• Determine Priorities
• Develop Utility Curves
• Develop Scoring Scales
2
1313
Alternative Comparison Technique
DIV A
DIV B
DIV C
Capability/MOE Definition: The ability to transport passengers
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 ….
T
O
T
O
T
O
Number of Passengers
• Determine Priorities
• Develop Utility Curves
• Develop Scoring Scales
2
1414
Functional Analysis Conduct Operations
AND AND
Command and Control (C2)
Survive Threats and Conditions
Surveillance (Find/Fix/Track/ID/Assess)
Launch RecoverEmbark Team/Equip
Transit to
OpArea
Transit from
OpAreaInterdict
Escort Vessel
Force Protection
Logistics &Service Support
Plan/DirectOrganize
Train/Equip Arm/ Fuel
C2
Employment (Air)Employment (Sea)
Employment (Fwd Basing)
Maneuverability/Navigation
Employment (Sea)
Employment (Fwd Basing)
Speed
Seakeeping
OR Abort Mission
Patrol
Speed
Deploy
Waterborne Checkpoint, PSHD, HAD, Vessel Escort
Enabling Tasks (Off-board)
Enabling Functions (Off-board)
Key Capabilities/MOPs
LEGEND
Capacity
1
3 4 59
11
8
10
12
13Detectability
OR
Speed
Lethality
Sustain Force15
14
Endurance
Loiter
7
16
Staging
Survivability
6
Transportability
2
Systems Engineering-based Functional Decomposition provides structured process to define Required Mission Capabilities to assess Alternatives
Systems Engineering-based Functional Decomposition provides structured process to define Required Mission Capabilities to assess Alternatives
1515
Modeling and Simulation Techniques
Discrete Event Simulation
Behavior/Functional Modeling
Physical or Prototype Modeling
Wargaming/Simulations
Sensitivity Analysis
Linear Programming
Reliability ,Failure Modes and Effects Analysis
Mathematical/Monte Carlo or Probability Modeling
1616
Method to Communicate Analysis
Results of Analysis
1. System B
2. System A
3. System C
1717
Method to Communicate Analysis
TRL LEVELS
9-10
7-8
<6
LCCE vs Effectiveness for Ten Craft with the Least Number of Threshold Failures
50' Riverine Patrol Craft (CB-90)
47' Concept Riverine Craft
40' Riverine M-Hull Craft
40' Riverine Patrol Boat
40' Riverine Craft (SURC)
37' Catamaran HVA Escort
36' Unmanned RHIB
36' Navy STD RHIB (Cabin)
31' Riverine Assault Boat
31' Riverine Craft (SOCR)
$6,000,000
$9,000,000
$12,000,000
$15,000,000
$18,000,000
3.60 3.70 3.80 3.90 4.00 4.10 4.20 4.30 4.40 4.50Effectiveness
LCCE (FY07$)
1818
Another Method of Communicating Decisions
This quad chart represents the capability areas that should be pursued first.
Not Addressed
Communications Gear
782 Gear
Ancillary Equipment
Ammunition
M49 Telescope Scope SASRScope Sniper Rifle
SASRSniper Rifle
0
4.5
9
-5 -4 -3 -2 -1 0 1 2 3 4 5
Quad 1High PriorityPoor Capabilities
Quad 3Low PriorityPoor Capabilities
Quad 4Low PriorityGood Capabilities
Quad 2High PriorityGood Capabilities
1919
The Range of Decision Making Techniques
So what are the key considerations when selecting a technique…
How important is the Decision; who will review decision? How much data/information do you have or can you get? How much time and $ do you have?
2020
Questions
?