s thomas sfield

Practical Applications of Risk Informed Decision Making –

No Fear!

PM Challenge 2012Sharon L. Thomas

Scott L. Field

Presentation Themes

• Inherent riskiness of uncertain times

• Evolving risk management to stay relevant

• What exactly is Risk Informed Decision-Making (RIDM)?

• Why RIDM now, since we’re already doing Continuous Risk Management (CRM)?

• RIDM is scary!

• When to use RIDM and why

• Actual steps (really?!)

• Go forth & do good RIDM

2

not

Uncertainty = Reality – Perception?

• We live in uncertain times! Whether at NASA or in industry, it’s no longer business as usual – belts are tighter!

• Budgets are fluctuating, missions requirements are changing.

• Advocates and naysayers alike point to risk – known unknowns – and scarier still, unknown unknowns!

• However, simply acknowledging risk ≠ active risk management.

• Since uncertainty is rife with risk, NASA requires a more flexible or agile approach to risk management to help decision makers adapt.

3

Evolving Risk Management

• Risk is a triplet = a scenario with a chance or probability (L) of experiencing a negative consequence (C).

− Ref. NASA Procedural Requirements (NPR) 8000.4A

• In recent years, NASA Programs, Projects, and Directorates have employed Continuous Risk Management (CRM) to mitigate risk across key performance domains:

− Cost (to achieve or regain required performance, cost to mitigate risk, cost of inaction)

− Schedule (Critical Path)

− Technical

− Safety (and Mission Assurance)

− Core Capabilities (Critical Assets)

4

Evolving Risk Management (cont.)

• Why the need for more flexibility?

− CRM presupposes a linear mission or a mission that has clearly defined end-objectives.

– Today, missions can be less linear (especially for human spaceflight) and end-objectives can be dynamic as policies change.

– In this environment, decision-makers must be flexible and even innovative to readily adapt to changes in mission objectives, funding, and resources.

– To keep pace, we need to be innovative, systemic thinkers in the way we view risk!

5


• To remain relevant, risk management practices must evolve as NASA adapts/evolves into a 21st century agency:

– For spaceflight, adaptation means a suite of capabilities independent of discrete mission objectives; i.e., build a launch capability whether the ultimate destination is a near-Earth-object, the Moon, or Mars.

– As NASA redefines itself, the nature of risk management must shift as well:

• From: Identifying and mitigating risk(s) that prevent a performing organization from meeting a known requirement.

• To: identifying and mitigating performance shortfalls while helping decision-makers find the best solution to meet tactical and strategic objectives.

• RIDM can help to evolve Risk Management practices to meet this challenge!

– NPR 8000.4A uses RIDM as a pre-quel to CRM.

– This can help frame the risk discussion from a systemic perspective.

6

Destination

Mars

LunarNEO

Policy

Direction

Mission

Objective

sC

on

verg

en

ce

7

Risk Management adds RIDM to CRM in an integrated framework to:

• Foster proactive management of threats to performance

• Better inform decision-making using relevant (timely) risk information

• More effectively manage risks by focusing CRM on existing baseline performance requirements and on those emerging via the RIDM process

• Through a graded approach, adapt to varying levels of risk over time.


Risk Management – Avoiding Undesirable Consequences!

Interface is 50% right!

When should decision makers assess for risk outcomes?

Hey, we followed

the spec, I better get

paid

Not my problem! Too funny!

Huh?

RIDM is Scary

• RIDM terminology can seem daunting – not to worry, we will translate the key terms!

• A key component of RIDM per NPR 8000.4A is advocacy for quantitative risk analyses to add rigor to decision processes and trades for:

– Complex, multi-scenario projects– Projects with competing stakeholders– (fill-in your case).

• However desirable, quantitative analyses are not always feasible. Can be costly and may not be feasible for activities that are smaller, lower dollar, or in the operational phase.

– Graded approach allows you to match the rigor of analysis commensurate with the criticality and type of decision to be made.

• This presentation discusses RIDM in practical ways and outlines a RIDM case study to illustrate the how’s & why’s – let’s get started!

8

NOT

Consequence

To develop or revise:

RequirementsCapabilitiesOperations Processes

?

?

Generic Risk Model

9

Risk Event

Controls Controls/Response

Activity

Preventive Measures Reduce Severity

Activity?

RIDM can help to clarify objectives:

● Activities and Products? (“Shall requirements”)

● Support functions? (Insight/Oversight)

Consequence

Consequence

To meet:

Performance objectives, over time

(Based on Risk Scenarios, Requirements, Plans, Historical & Current Performance, Resources…)

Mitigate or Accept

Risk Model with RIDM (NPR 8000.4A)

10

Got Risk?

11

RIDM Process

• ID Scenario

• ID Stakeholders

• Performance Objectives

• Performance Measures

• Confirm Constraints

• Analyze

• Down Select

• Risk Inform Decision

Do you know where your alligators are today?

Key RIDM Terms

12

#Key Pt

Term "RIDM-Speak"RIDM

Handbook reference

Translation

1 √ Graded Approach

Determine depth of analysis needed to reduce uncertainty. Not all risk analysis can/should be quantitative. Use qualitative analysis, as appropriate.

Section 3.2.1.3,

p. 51

Rigor: Match level of analysis to significance of decision, project scale, likelihood, and/or level of impact.

2 √Performance Objective

Derived via an objectives hierarchy; objectives at each level of an organization.

General term

Mandated or delegated goals that organizations must meet at each level.

3 √Performance Measure

Metric to capture the extent to which a system, process, or activity meets a performance objective. Same measure can be used for all alternatives.

Generic term Figures of merit (FOM)

4 √Performance Commitment

Alternative-independent level of performance: Using accepted measures, compare across alternatives, and consider the risk tolerance of the decision-maker.

Sec 1.5, Part 3, pp.

19-22

Match performance objective to risk tolerance of decision-maker (to compare across alternatives at same level of confidence).

5 √Technical Basis for Deliberation (TBfD)

Evolving document of ground rules and assumptions used to structure a risk-informed decision. Specifies information needed to characterize uncertainty for each decision alternative.

p.17; App.D: Content

Guide, pp. 108-109

Basis of Estimate (BOE) that evolves with each iteration of risk analysis

Ref: NASA Risk Management Handbook NHBK_2011_3422

Iterative Risk Decision Process

13

Mission:● Technical Objectives● Tactical Operations● Schedule / Discrete Milestones● Logistics● Strategic Objectives

Systems Engineering Handbook

Mission Support:● Capability● Capacity / Infrastructure● Institutional Goals● Supply Chain● Strategic Objectives

RECAP

• Evolving Risk Management = RIDM + CRM

− Uncertainty and evolving Agency direction elevate the need for responsive Risk Management.

− CRM is still a successful tool in the Risk Management tool box.

− RIDM complements CRM by enhancing the risk analyses in a graded approach commensurate with decisions to be made.

− RIDM and CRM used together throughout your iterative decision process can ensure timely risk information.

• Risk-informed decisions = an adaptive process:

− Factoring timely risk information into key decisions adaptation to dynamic environments.

− CRM alone enables response to a potential risk.

− Adding RIDM iteratively can help with early insight into individual risks as well as overall performance risks over time = flexibility.

14

15

Perspective: RIDM-to-CRM

“Two Sides of the Same Coin”

• Start with a Performance Objective

• Frame the decision space (graded approach)

• Use CRM to analyze risk of each alternative

• Down-select alternatives

• Track/monitor via CRM

CRM-Centric

• Start with a Performance Shortfall (Risk)

• Use mini-RIDMs to evaluate/make handling decisions

• Assess qualitatively or quantitatively

• Develop/ implement mitigation

RIDM-Centric

Systems Engineering Handbook

16

Deciphering RIDM

1b. Derive Performance Measures

1a. Identify Performance Objectives

5. Risk-Normalize Commitments

3. Set the Framework (TBfD)*

6b. Select an Alternative & Document

2. IdentifyFeasible Alternatives

4. Analyze Alternatives & Document

6a. Deliberate & Decide

Map of KSC Facilities and Launch Pads

Illustrated Example*

17*Loosely based on an actual risk scenario

CARGO HAZARDOUSSERVICING FACILITY

• Decision: 1. Select best location for Short Stack Hypergolic Servicing.

2. Base decision on performance objectives, performance measures, stakeholder considerations, and domain risks of each feasible alternative.

• Alternatives:1. Existing facilities (e.g., Pad)

2. New facility

3. Other?



Iden

tifi

cati

on

of

Alt

ern

ativ

es



Ris

k A

na

lys

is o

f A

lter

nat

ives

6b. Select an Alternative &

Document

Ris

k-I

nfo

rmed

Alt

ern

ativ

e S

elec

tio

n


4. Analyze Alternatives &

Document


Step 1a. Refine the Decision

• Concise statement of a performance shortfall.

• Establish common understanding of performance constraints.

• Document assumptions and scope of decision.

18

Scenario for Off-line Servicing & Assembly

etc.

1b. Criteria for Success

• Identify all Stakeholders and their expectations.

• Define Performance Objectives.

• Characterize Performance Measures.

19

1b. Derive Performance

Measures


Iden

tifi

cati

on

of

Alt

ern

ativ

es



Ris

k A

na

lys

is o

f A

lter

nat

ives


Document

Ris

k-I

nfo

rmed

Alt

ern

ativ

e S

elec

tio

n



Document


Program Ground Ops

HQ KSC

EPA Public

?

“Show me the money!” is a cool phrase. For RM, it’s not just about the $$, we must consider all domains!

2. Feasible Alternatives

• Initially consider a range of solutions – even wild-eyed ideas okay at this point!

• Always include the “Do-Nothing” option.

• Use Graded Approach to determine appropriate level of rigor needed to refine your list of feasible alternatives.

20


Measures


Iden

tifi

cati

on

of

Alt

ern

ativ

es



Ris

k A

na

lys

is o

f A

lter

nat

ives


Document

Ris

k-I

nfo

rmed

Alt

ern

ativ

e S

elec

tio

n



Document


MPPF (existing facility)

VAB (existing facility)

Pad (existing process)

Vehicle Integration Building (new facility)

Stop-n-Go Model (new alternative)

3. Set the Framework

• Select “Key” Decision Parameters.– Note: Parameters must apply to ALL alternatives.

• Determine reasonable Performance Measures and criteria needed for a decision-maker to commit to an alternative (minimum threshold).

• Establish weighted priority (basis) for making a decision.

21


Measures


Iden

tifi

cati

on

of

Alt

ern

ativ

es



Ris

k A

na

lys

is o

f A

lter

nat

ives


Document

Ris

k-I

nfo

rmed

Alt

ern

ativ

e S

elec

tio

n



Document


Launch Availability

Life Cycle Cost

Contingency & Recovery

Safety

“Green Pad”

Flexibility Veh. Orientation

4. Analyze Alternatives

• Conduct analysis of each alternative’s key performance measures.

• Prune “unsuitable” alternatives expeditiously based on key parameters:– Reduces analysis cost / effort.

– Clarifies options for decision makers.

22


Measures


Iden

tifi

cati

on

of

Alt

ern

ativ

es



Ris

k A

na

lys

is o

f A

lter

nat

ives


Document

Ris

k-I

nfo

rmed

Alt

ern

ativ

e S

elec

tio

n



Document


MPPF (existing facility)

VAB (existing facility)

Pad (existing process)

Vehicle Integration Building (new facility)

Stop-n-Go Model (new alternative)

Additional RIDM Terminology for Step 5

23

#Key Pt

Term "RIDM-Speak"RIDM

Handbook reference

Translation

6 √Risk Normalized Performance Commitment

Create means to compare “like” perspectives across alternatives by first establishing a “common break-point of uncertainty” for each analysis.

Section 3.3.1,

pp.73-78

Normalize performance to assess across a range of options

• For any measure, allowable uncertainty is based on a number of factors

For ex. Worst-case threshold

• Set standards used to compare alternatives at the same level of uncertainty; allocated resources represent some point value in an uncertainty range (.01 – .99) for each alternative.

Avoid co-mingling analysis uncertainty with the uncertainty of an alternative; each option produces its own PDF and CDF.

• A common “confidence” level would be used to arrive at a “normalized” value of uncertainty for each alternative Apply the same confidence level (i.e.,

risk tolerance) to each alternative for a criterion (criteria can have different risk tolerance levels)

• Additional uncertainty considerations depend on risk tolerance of decision-makers. If risk-neutral, an option with uncertain performance may be preferred if it performs at the “mean value” of a performance measure pdf.

Risk-neutral not likely today!

Ref: NASA Risk Management Handbook NHBK_2011_3422

• Consider risk tolerance of the decision-maker and all key stakeholders for each performance measure.

• Determine degree of certainty required for each performance measure for comparison.

• Each performance measure should yield a probability distribution function (pdf) for each alternative.

Step 5. Risk Normalize Commitments

24



Iden

tifi

cati

on

of

Alt

ern

ativ

es



Ris

k A

na

lys

is o

f A

lter

nat

ives


Document

Ris

k-I

nfo

rmed

Alt

ern

ativ

e S

elec

tio

n



Document


Note:Pdf – Cdf applies for quantitative analyses; similarly for qualitative

assessments, one should apply the same level of confidence when evaluating alternatives (constructed scales).

Risk Normalized Commitments

• Each alternative yields its own PDF and corresponding cumulative distribution function (CDF).

25

NOTES:

1. A common “uncertainty” level would be used to arrive at a “normalized” value of uncertainty for each alternative.

2. Assumed 20% Risk Tolerance = 80% Certainty (use 80% value of CDF)

High- Level Objectives Performance Objectives Measures Pad VAB MPPF VIB S & G WeightMeets CARD Yes Yes Yes Yes Yes 0%Optimal: Exceeds L.A. reqmts No Yes Yes Yes No 7%Construction/ refurbishmt ($M) 2.5 5 4 20 6 8%Operations & Maintenance ($M) 4 2.5 4 1 1 10%Discovery 1 - Low Risk 3 1 1 1 2 6%Verification 2 - Mild Risk 2 1 1 1 1 8%Repair 3 - Moderate Risk 1 2 1 1 3 3%

4 - Potentially UnreliableOff-nominal (de-servicing) 5 - Unreliable 3 2 3 1 1 2%Minimize Pad Time***(Clean Pad) Quan (Hr) 106 1 1 1 1 11%De-couple critical path Quan (Hr) 96 12 24 12 12 2%Quantity Distance Impact Quan (hrs contact) 3 16 20 20 3 10%Hypergol Contamination Quan (H/L) 3.2 2.8 2.2 1.6 0.8 10%Adequate “clear” Y/N Yes Yes Yes Yes Yes 0%Minimize Wet Time Quan (Hr) 8 106 106 106 16 6%Scalability (flight rate) Flights/Yr 52 52 70 100 100 10%- Flexibility Re-Cycle Time (Hr) 48 48 12 12 0 7%

Tally 85 113 161 178 140 100%

Schedule

Launch Availability

Life Cycle Cost

Contingency Capabilities Flexibility and Recovery

Processing Effi ciency

Safety

Example of Analysis Documented on TBfD Matrix

26

Note: Conceptual only – data & values fabricated for illustrative purposes only.

Step 1 Step 1b Step 2 Step 3

Step 6a. Decide & Deliberate

• Apply analysis results to TBfD and make recommendation to Decision-Maker.

• Reconsider TBfD if no clear leader and report results to Decision-Maker.

• Reconsider requirements / constraints if no acceptable alternatives.

27


Measures


Iden

tifi

cati

on

of

Alt

ern

ativ

es



Ris

k A

na

lys

is o

f A

lter

nat

ives


Document

Ris

k-I

nfo

rmed

Alt

ern

ativ

e S

elec

tio

n



Document


Analysis Findings (for example):− Recommend New Facility− Document “Green Pad” Requirement− Work to Minimize Wet Time

28

Recap: Risk Management = RIDM + CRM


Measures


Identification of Alternatives

5. Risk-Normalize

Commitments

3. Set the Framework

(TBfD)*

Risk Analysis of Alternatives

Build Performance Requirements


Document

Risk-Informed Alternative Selection

2. IdentifyFeasible

Alternatives

Iterative

RIDM: Inform decision-making through use of quantitative and qualitative risk info to establish baseline performance requirements for mission support organizations, programs and projects.

CRM: Manage risk associated with implementing baseline performance requirements using a systematic and iterative process to identify, analyze, plan, track, control, communicate and document risks.


Document

6a. Deliberate &

Decide

Ref. RIDM Handbook Fig. 11. RIDM Process Steps (p.30)

TBfD:*Technical Basis for Deliberation; also known as (BOE) Basis of Estimate

Concluding Remarks

• Uncertainty, evolving objectives, scarce resources – require an evolution in the way we approach risk management.

• The key is coupling RIDM with CRM to evolve Risk Management practices to be more pro-active vs. simply reactive.

• Aim: Employ an integrative, systemic approach to yield relevant and timely information for missions and mission support.

• Doing so keeps NASA Risk Management practices fresh and relevant to enable risk-informed decisions.

• Go forth & do good RIDM (and CRM)!

29

Always consider mission support risks!

30

JSC\AC3 Sharon L. Thomas [email protected](281) 244-7668

JSC\NA Scott L. Field, [email protected](281) 335-2440

mailto:[email protected]

mailto:[email protected]

s thomas sfield

Economy & Finance

evolving risk management

risk active risk management

risk outcomes

risk informdecisiondo

risk scenarios

levels of risk

risk discussion

risk management practices