risk management using risk+ (v5)

INCREASING THE PROBABILITY OF PROGRAM SUCCESS USING RISK+

A workshop on the principles and practices of Risk+ and increasing the Probability of Program Success

1

Glen B. Alleman

Niwot Ridge LLC

A Warning We’re going to cover a lot of material in 3 hours

2

Risk involves uncertainty. Uncertainty involves probability.

3

Douglas Adams, Hitchhiker's Guide to the Galaxy 4

MOTIVATION?

Your motivation?

Your motivation is your pay packet on Friday.

Now get on with it.

– Noel Coward, English actor, dramatist, &

songwriter (1899 – 1973)

5

We have to know the underlying statistical behavior of the

processes driving the project

This means cost, schedule, and technical performance

measures with probabilistic models

We need to know how these three statistical drivers are

coupled

What drives what?

What are the multipliers between each random

variable? 6

Let’s start with the basics 7

Remember High School Statistics 8

The IMS is a collection of probabilistic processes all coupled together

9

What does this really mean?

In building a risk tolerant IMS,

we’re interested in the

probability of a successful

outcome…

“What is the probability of

making a desired completion

date?”

But the underlying statistics of

the tasks influence this

probability

The statistics of the tasks, their arrangement in a network of

tasks and correlation define how this probability based

estimated developed.

10

There are real problems with those pesky

Unknowns that get in the way of progress

Imprint of a bird on our west facing family room second story

window on a bright afternoon

The Bird survived 11

The “units of measure” of Risk

These classifications can be used to avoid asking the

“3 point” question for each task.

Anchoring and Adjustment† of all estimating

processes produces a bias.

Knowing this is necessary for credible estimates.

Classification Uncertainty Overrun

1 Routine, been done before Low 0% to 2%

2 Routine, but possible difficulties Medium to Low 2% to 5%

3 Development, with little technical difficulty Medium 5% to 10%

4 Development, but some technical difficulty Medium High 10% to 15%

5 Significant effort, technical challenge High 15% to 25%

6 No experience in this area Very High 25% to 50%

12

† Tversky and Khanemann Anchoring and Adjustment

We’re looking for knowledge of what is going to

happen in he future, with a known level of confidence

Harvard main library 13

With some principals behind us, let’s see how to use

Risk+ to address the problem of forecasting the future of

schedule and cost performance.

What is Monte Carlo Simulation? 15

A Quick Look At Monte Carlo

George Louis Leclerc,

Comte de Buffon, asked

what was the probability

that the needle would fall

across one of the lines,

marked here in green.

That outcome will occur

only if sinA l

16

Los Alamos Science, Special Issue 1987

17

Monte Carlo Simulation

Monte Carlo Simulation is named

after the city, in Monaco, of casinos

on the French Rivera.

Monte Carlo …

Examines all paths not just the critical

path.

Provides an accurate (true) estimate

of completion:

Overall duration distribution

Confidence interval (accuracy

range)

Sensitivity analysis of interacting tasks

Varied activity distribution types – not restricted to a single distribution

Schedule logic can include branching – both probabilistic and conditional

When resource loaded schedules are used – provides integrated cost and

schedule probabilistic model.

18

We need to answer the question …

What is the confidence we will complete “on or

before” at date and “at or below” at cost?

This is the question that should be asked and

answered on a periodic basis.

We need to have Schedule and Cost margin to

protect the deliverables and our Budget At

Completion.

What Are We Really After? 20

Here is some advice on how

to depict this margin and

where to place this margin.

No matter how we show

manage these two elements

in the IMS, if we don’t have

margin we are late and

over budget before we

start.

http://www.ndia.org/Divisions/Divisions/Procurement/Documents/PMSCommittee/CommitteeDocuments/

WhitePapers/NDIAScheduleMarginWhitePaperFinal-2010(2).pdf 21

Confidence levels for margin change as the program proceeds

As the program proceeds we want to have

Increased accuracy

Reduced schedule risk

Increasing visual confirmation that success can be reached

Current Estimate Confidence

22

Our REAL goal here is to Manage Margin using probabilistic models

Programmatic Margin is added between Development, Production and Integration & Test phases

Risk Margin is added to the IMS where risk alternatives are identified

Margin that is not used in the IMS for risk mitigation will be moved to the next sequence of risk alternatives

This enables us to buy back schedule margin for activities further downstream

This enables us to control the ripple effect of schedule shifts on Margin activities

5 Days Margin

5 Days Margin

Plan B

Plan A

Plan B

Plan AFirst Identified Risk Alternative in IMS

Second Identified Risk

Alternative in IMS

3 Days Margin Used

Downstream

Activities shifted to

left 2 daysDuration of Plan B < Plan A + Margin

2 days will be added

to this margin task

to bring schedule

back on track

23

Sensitivity Analysis

The schedule sensitivity of a task measures the closeness with which change in the task duration matches change in the project duration over the simulation.

This closeness is the correlation between changes in individual activities and their impacts on other activities.

A task with high schedule sensitivity is more likely to be a major driver of the project duration than a lower ranked task.

: Models of the Schedule

24

Task Criticality Analysis

A measure of the frequency that an activity in the

project schedule is critical (Total Float = 0) in a

simulation

If a task is critical in 500 of the 1,000 iterations of

the simulation, it has a Criticality Index of 0.5

The higher the criticality index, the more certain it is

that the task will always be critical in the project


25

Cruciality shows each task’s tolerance to risk

Cruciality = Schedule Sensitivity x Criticality

Schedule Sensitivity can be statistically misleading:

A task with high sensitivity may not be on or near the critical path.

Thus a reduction in that task’s duration may have little effect on the project duration.

Cruciality sharpens the analytical focus:

It highlights critical or near–critical activities with high.

Schedule Sensitivity

These tasks are most likely to drive project duration.


26

Guiding the Risk Factor Process means weighting each level of risk

For tasks marked “Low” a reasonable

approach is to score the maximum

10% greater than the minimum.

The “Most Likely” is then scored as a

geometric progression for the

remaining categories with a common

ratio of 1.5

Tasks marked “Very High” are bound

at 200% of minimum.

No viable project manager would like

a task grow to three times the planned

duration without intervention

The geometric progress is somewhat

arbitrary but it should be used instead

of a linear progression

Min Most

Likely

Max

Low 1.0 1.04 1.10

Low+ 1.0 1.06 1.15

Moderate 1.0 1.09 1.24

Moderate+ 1.0 1.14 1.36

High 1.0 1.20 1.55

High+ 1.0 1.30 1.85

Very High 1.0 1.46 2.30

Very High+ 1.0 1.68 3.00

: Examples of Monte Carlo

27

Progressive Risk Factors

A geometric progression (1.534) of risk can be

used.

The phrases associated with increasing risk have

been shown at the Naval Research Laboratory to

correlate with an engineers “sense” of increasing

risk.


28

Risk Factor Attributes

The “narrative” for each risk factor needs to be developed.

Each description is dependent on…

Discipline

Program stage

Complexity

Historical data

Current “risk state” of the program

This is currently missing from our efforts to quantify schedule and cost risk.


29

Accuracy

Given a specified final cost or project duration, what is the probability of achieving this cost or duration?

Frequentist approach

Over many different projects, four out of five will cost less or be completed in less time than the specified cost or duration.

Bayesian approach

We would be willing to bet at 4 to 1 odds that the project will be under the 80% point in cost or duration.

Accuracy is needed to plan reserves.

Accuracy is needed when comparing competing proposals.

: What is the Purpose of Project Risk Analysis?

30

Structured Thinking

All estimates will be in error to some degree of variance.

Trying to quantify these errors will result in bounds too wide to be useful for decision making.

Risk analysis should be used to

Think about different aspects of the project

Try to put numbers against probabilities and impacts

Discuss with colleagues the different ideas and perceptions

Thinking things through carefully results in

Which elements of the programmatic and technical risk are represented in the IMS.

The process becomes more valuable than the numbers.

: What is the Purpose of Project Risk Analysis?

31

To properly use Schedule Margin†

Work must be represented in single units – either

task or work packages.

The overall schedule margin must be related to the

variation of individual units of work.

The importance of the units of work must be shared

among all participants (ordinal ranking of work and

its risk).

The schedule must be reasonable in some units of

measure shared by all the participants.

† “Protecting Earned Value Schedules with Schedule Margin,” Newbold, Budd, and Budd,

http://www.prochain.com/pm/articles/ProtectingEVSchedules.pdf

32

Let’s Apply a Monte Carlo Simulation Tool The Monte Carlo trolley, or FERMIAC,

was invented by Enrico Fermi and

constructed by Percy King. The drums

on the trolley were set according to

the material being traversed and a

random choice between fast and slow

neutrons.

Another random digit was used to

determine the direction of motion,

and a third was selected to give the

distance to the next collision. The

trolley was then operated by moving

it across a two dimensional scale

drawing of the nuclear device or

reactor assembly being studied.

The trolley drew a path as it rolled,

stopping for changes in drum settings

whenever a material boundary was

crossed. This infant computer was

used for about two years to

determine, among other things, the

change in neutron population with

time in numerous types of nuclear

systems. 33

Most Likely Isn’t Likely to be the Most Likely

When we say “most likely” what do we think this

actually means?

If you pick the wrong meaning, your Monte

Carlo model will be seriously flawed.

A Small Diversion 35

The problem with “Most Likelies”

For each activity the “best” estimate is …

The “most likely” duration – the mode of the distribution of

durations? (Mode is the number that appears most often)

It’s 50th percentile duration – the median of the distribution?

(Median is the number in the middle of all the numbers)

It’s expected duration – the mean of the distribution? (Mean

is the average of all the numbers)

These definitions lead to values that are almost always

different from each other.

Rolling up the “best” estimate of completion is almost

never one of these.

36

Durations are Probability Estimates not Single Point Values

We know this because…

“Best” estimate is not the only possible estimate, so other estimates must be considered “worse.”

Common use of the phrase “most likely duration” assumes that other possible durations are “less likely.”

“Mean,” “median,” and “mode” are statistical terms characteristic of probability distributions.

This implies activity distributions have probability distributions

They are random variables drawn from the probability distribution function (pdf).

“Actual” project duration is an uncertain quality that can be modeled as a sum of random variables

The pdf may be known or unknown.

37

Task Most Likely ≠ Project Most Likely

PERT assumes probability distribution of the project times is the same as the tasks on the critical path.

Because other paths can become critical paths, PERT consistently underestimates the project completion time. 1 + 1 = 3

: Managing Uncertainty in the IMS

3 38

Probability Distribution Function is the Lifeblood of good planning

Probability of occurrence as a function of the number of samples.

“The number of times a task duration appears in a Monte Carlo simulation.”

: Managing Uncertainty in the IMS

39

Remember the quote about statistics

Lies, Damn Lies, and Statistics

– Benjamin Disraeli

But we know better, we know that

any estimate without a variance is

not trustworthy.

We know that the variances have

to be calibrated from past

performance to be credible

40

One should expect that the expected

can be prevented, but the unexpected

should have been expected.

— Augustine Law XLV

A “Real World” Schedule Analysis 42

This is a must own book for everyone in our business. It defines

fundamental Laws of program and business management, which

are many times ignored – like the one above

Our Starting Point

Risk+ Installed

Let’s define the

needed fields

These are used

by Risk+ to hold

information and

run the

application.

43

If there are conflicts, you can make changes in Risk+ to work around your fields.

A Simple IMS 44

By simple it means serial cascaded work efforts.

Initial Field Usage

Minimum Remaining Duration

The duration that is least you’d expect this task to complete in

Most Likely Remaining Duration

The ML (Mode) of the duration

Maximum Remaining Duration

The duration that is the most you’d expect this task to complete in

Task Reporting ID

The tasks we want to watch

45

Define a View and Table for Risk+

Start with the Gantt View and Entry Table

Set up both to match the Risk+ field usage

Use the default if

there are no field

conflicts

46

Fields used for Risk+ example 47

Let’s actually “doing something”

Initialize the Most Likely.

This sets the Most Likely

duration to the same value

that is in the “Duration” field

of your IMS.

The “planned duration” now

becomes the ML duration.

If this “planned duration” is

bogus then your model will be

as well.

Choose wisely.

48

Now the ML = DURATION step

All the DURATION values have been moved to the ML field.

But remember our discussion of the ML’s

Choose them carefully

The next we’ll set the upper and lower limits of that ML value

Using risk factors.

OK, 3 point estimates if you have to.

49

Let’s do this the simple way

Let’s pick MEDIUM confidence.

MEDIUM means

–25%

+25%

And a NORMAL (Gaussian) curve

50

Let’s have Risk+ do something for us

Enter a “1” in the RPT field (Number 1)

This marks that ROW in the schedule as a work

activity we want to see the Monte Carlo output for

51

Now we’re ready to run

The RISK ANALYSIS command

starts the process going.

Let’s make 200 iteration and

look at the DURARTION

ANALYSIS for the activities we

are watching.

52

This is nice but what actually is Risk + doing?

Risk+ is picking a random number from under the normal distribution within the range of the

Least remaining and most remaining

This is not some ordinary random number it is chosen through an algorithm called the Latin Hypercube - more on that later.

Risk+ then plugs that number into the “real” DURATION field and does that for all the DURATIONS in the schedule

Then the F9 key is pressed and the date is recorded for the finish of UID 41.

This is done 200 times and a histogram of all the dates that appeared for those 200 time is recorded.

53

And We Get

Date: 11/29/2011 4:32:17 PM

Samples: 500

Unique ID: 19

Name: End Work Package 3

Completion Std Deviation: 2.06 days

95% Confidence Interval: 0.18 days

Each bar represents 1 day

Completion Date

Fre

qu

en

cy

Cu

mu

lative

Pro

ba

bili

ty

Mon 3/12/12Fri 3/2/12 Tue 3/20/12

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

0.18

0.20 Completion Probability Table

Prob ProbDate Date

0.05 Wed 3/7/12

0.10 Thu 3/8/12

0.15 Thu 3/8/12

0.20 Fri 3/9/12

0.25 Fri 3/9/12

0.30 Fri 3/9/12

0.35 Mon 3/12/12

0.40 Mon 3/12/12

0.45 Mon 3/12/12

0.50 Mon 3/12/12

0.55 Tue 3/13/12

0.60 Tue 3/13/12

0.65 Tue 3/13/12

0.70 Wed 3/14/12

0.75 Wed 3/14/12

0.80 Wed 3/14/12

0.85 Thu 3/15/12

0.90 Thu 3/15/12

0.95 Fri 3/16/12

1.00 Tue 3/20/12

54

Learning to Speak in Risk+

Risk +shows use the probability of finish “on or

before” a date

It does NOT show the probability of success.

But even the “on or before” term is loaded with

special meaning.

It means for the 500 iterations of Risk+ using the

upper and lower bounds of the duration, drawn

from the probability density function (pdf) with the

Normal (Gaussian) shape, 60% of the finish dates

were recorded to be on or before 3/12/12.

55

Medium confidence for a large project

Date: 11/30/2011 6:05:35 PM

Samples: 200

Unique ID: 17

Name: (SA) Systems Requirements Completed



Each bar represents 2 days

Completion Date

Fre

qu

en

cy

Cu

mu

lative

Pro

ba

bili

ty

Wed 5/16/12Wed 5/2/12 Mon 6/4/12

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.03

0.05

0.08

0.10

0.13

0.15

0.17

0.20


Prob ProbDate Date

0.05 Fri 5/4/12

0.10 Wed 5/9/12

0.15 Thu 5/10/12

0.20 Fri 5/11/12

0.25 Mon 5/14/12

0.30 Mon 5/14/12

0.35 Tue 5/15/12

0.40 Tue 5/15/12

0.45 Wed 5/16/12

0.50 Wed 5/16/12

0.55 Thu 5/17/12

0.60 Thu 5/17/12

0.65 Fri 5/18/12

0.70 Mon 5/21/12

0.75 Mon 5/21/12

0.80 Tue 5/22/12

0.85 Wed 5/23/12

0.90 Thu 5/24/12

0.95 Mon 5/28/12

1.00 Mon 6/4/12

56

Low confidence for a large project

Date: 11/30/2011 10:30:05 PM

Samples: 200

Unique ID: 17

Name: (SA) Systems Requirements Completed



Each bar represents 3 days

Completion Date

Fre

qu

en

cy

Cu

mu

lative

Pro

ba

bili

ty

Thu 5/24/12Tue 4/24/12 Wed 6/27/12

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.02

0.04

0.06

0.08

0.10

0.12

0.14


Prob ProbDate Date

0.05 Thu 5/3/12

0.10 Tue 5/8/12

0.15 Wed 5/9/12

0.20 Mon 5/14/12

0.25 Tue 5/15/12

0.30 Thu 5/17/12

0.35 Fri 5/18/12

0.40 Mon 5/21/12

0.45 Wed 5/23/12

0.50 Wed 5/23/12

0.55 Fri 5/25/12

0.60 Mon 5/28/12

0.65 Wed 5/30/12

0.70 Wed 5/30/12

0.75 Fri 6/1/12

0.80 Mon 6/4/12

0.85 Wed 6/6/12

0.90 Fri 6/8/12

0.95 Thu 6/14/12

1.00 Wed 6/27/12

57

58

Let’s run some

simulations

Now that the schedule can be produced using probabilistic methods, it’s time to talk about the cost.

Cost does not have a linear relationship with schedule unfortunately.

Basic Principles of Probabilistic Cost

: Basic Principles of Probabilistic Cost

60

Basic Principles with Probabilistic Cost Estimating are coupled with scheduling

Cost estimates usually involve many CERs

Each of these CERs has uncertainty (standard error)

CER input variables have uncertainty (technical uncertainty)

Must combine CER uncertainty with technical uncertainty for many CERs in an estimate

Usually cannot be done arithmetically; must use simulation to roll up costs derived from Monte Carlo samples

Add and multiply probability distributions rather than numbers

Statistically combining many uncertain, or randomly varying, numbers

Monte Carlo simulation

Take random sample from each CER and input parameter, add and multiply as necessary, then record total system cost as a single sample

Repeat the procedure thousands of times to develop a frequency histogram of the total system cost samples

This becomes the probability distribution of total system cost


61

The Cost Probability Distributions as a function of the weighted cost drivers

$

Cost Driver (Weight)

Cost = a + bXc

Cost

Estimate

Historical data point

Cost estimating relationship

Standard percent error bounds Technical Uncertainty

Combined Cost Modeling

and Technical Uncertainty

Cost Modeling Uncertainty


62

The Risk Adjusted Cost Estimate Connected To The IMS

In the risk–adjusted cost estimate, we now combine discrete risk events and the uncertainty of the input distributions with the uncertainty of the CERs

Since the input distributions tend to be right–skewed, the expected cost tends to be larger than the baseline estimate

In addition, the risk–adjusted cost distribution tends to be wider than the baseline estimate

The difference between the expected cost of the risk–adjusted estimate and the expected cost of the baseline estimate is, by definition, the amount of RISK dollars included in the risk–adjusted estimate


63

Baseline versus Risk Adjusted Cost Estimates Usually Show a Cost Increase

Baseline vs. Risk-Adjusted Estimates

0 50 100 150 200 250 300 350

FY$M

Lik

eli

ho

od

Baseline:

Mean = $102.6M

Std Dev = $29.8M

Risk–Adjusted:

Mean = $122.6M

Std Dev = $42.8M


64

The S–Curve for Cost Modeling

Cumulative Distribution Function

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

$60 $80 $100 $120 $140 $160 $180 $200

FY00$M

Cu

mu

lati

ve

Pro

ba

bilit

y

Baseline Estimate

Mean $102.6M

50th percentile

$114.7M

Risk–adjusted

Estimate Mean

$122.6M

80th percentile

$153.5M


65

The Real Question Always Returns to… “But How Much Does It Cost? Really?”

This is impossible to answer precisely

Decision–makers and cost analysts should always think of a cost estimate as a probability distribution, NOT as a deterministic number

The best we can provide is the probability distribution – If we think we can be any more precise, we’re fooling ourselves

It is up to the decision–maker to decide where he/she wants to set the budget

The probability distribution provides a quantitative basis for making this determination

Low budget = high probability of overrun

High budget = low probability of overrun


66

Just having the pictures is necessary, but knowing

what they mean is required.

Making changes to the IMS to increase the

Probability of Program Success is the primary

outcome from Monte Carlo Simulation.

Some More Parts to using Risk+ 68

Without Integrating $, Time, and TPM you’re driving in the rearview mirror

Technical Performance (TPM)

69

Risk Management Demands a Well Defined Process

Statistics of a Triangle Distribution

Mode = 2000 hrs

Median = 3415 hrs

Mean = 3879 hrs

Minimum

1000 hrs

Maximum

6830 hrs

50% of all possible values are

under this area of the curve. This

is the definition of the median

Basic Statistics

71

TPM Trends & Responses directly impact risk and credibility of the IMS

Dr. Falk Chart – modified

25kg

23kg

28kg

26kg

PDR SRR SFR CA TRR CDR

ROM in Proposal

Design Model

Bench Scale Model Measurement

Detailed Design Model

Prototype Measurement

Flight 1st Article

Tech

nic

al P

erfo

rman

ce M

easu

re

Veh

icle

Wei

ght

72

Not A Mitigation Plan

Mitigation is too late, the risk has

turned into an issue. The money

has been spent, and the time has

passed.

73

Ordinal versus Cardinal

A variable is ordinally measurable if ranking is possible for values of the variable. For example, a gold medal reflects superior performance to a silver or bronze medal in the Olympics, or you may prefer French toast to waffles, and waffles to oat bran muffins. All variables that are cardinally measurable are also ordinally measurable, although the reverse may not be true.

A variable is cardinally measurable if a given interval between measures has a consistent meaning, i.e., if the measure corresponds to points along a straight line. For example, height, output, and income are cardinally measurable.

74

Ordinal Cardinal

Correcting Ordinal Risk Scales

Classify and calibrate risk ranking in units

meaningful to the decision makers

Risk rank 1, 2, 3, 4, is NOT sufficient

The Risk Rank must have a measurable value connected

to the actual behavior of the system being assessed

Calibration coefficients between ordinal probability

and consequences should also be used.

Ordinal analysis assumes ordering of the risks.

Cardinal analysis provides objective measures of

probability and consequential impact.

75

E

D

C

B

A

A B C D E

Level Likelihood Value

E Near Certainty E ≥ 90%

D Highly Likely 74% ≤ D ≤ 90%

C Likely 40% ≤ C ≤ 60%

B Low Likelihood 20% ≤ B ≤ 40%

A Not Likely A ≤ 20%

Level Technical Performance Schedule Cost

A Minimal or no consequence to technical performance.

Minimal or no impact Minimal or no impact

B Minor reduction in technical performance or supportability.

Able to meet key dates Budget increase or unit production cost increases. < (1% of Budget)

C

Moderate reduction in technical performance or supportability with limited impact on program objectives.

Minor schedule slip. Able to meet key milestones with no schedule float.

Budget increase or unit production cost increase < (5% of Budget)

D

Significant degradation in technical performance or major shortfall in supportability.

Program critical path affected

Budget increase or unit production cost increase < (10% of Budget)

E Severe degradation in technical performance.

Cannot meet key program milestones. Slip > X months

Exceeds budget increase or unit production cost threshold

Never multiply Likelihood by outcome. They are not “numbers,” they a probability distributions. Only convolution is possible

76

These are Cardinal measures of probability of occurrence and consequential impact

Example of Ordinal Probability Complexity Scale†

Definition of the Ordinal Scale Ranking Scale Level

Greater than 20% of the interface design has been

altered because of modifications to the ICD’s. E

Greater than 15% but less than 20% of the interface

design has been altered because of modifications of the

ICD’s.

D

Greater than 10% but less than 15% of the interface

design has been altered because of modifications of the

ICD’s.

C

At least 5% but less than 10% of the interface design has

been altered because of modifications of the ICD’s. B

At least 5% of the interface design has been altered

because of modifications of the ICD’s. A

† Effective Risk Management: Some Keys to Success, Edmund Conrow, AIAA Press, 2003

77

A “real” risk Ordinal Ranking Table

Risk

Rank Percent Variance Interpretation of Risk Ranking

A – 5% ≤ A ≤ 10% Normal business, technical & manufacturing

processes are applied

B – 5% ≤ B≤ 15%

Normal business & technical processes are

applied; new or innovative manufacturing

processes

C – 5% ≤ C ≤ 35% Flight software development & certification

processes

D – 10% ≤ D ≤ 25% Build & qualification of flight components,

subsystems & systems

E – 10% ≤ E ≤ 35% Flight software qualification

F – 5% ≤ F ≤ 175% ISS thermal vacuum acceptance testing

78

Untimely and unrealistic Latest Revised Estimates (LRE) Progress not monitored in a regular and

consistent manner Lack of vertical and horizontal traceability

cost and schedule data for corrective action Lack of internal surveillance and controls Managerial actions not demonstrated using

Earned Value

Inattention to budgetary responsibilities Work authorizations that are

not always followed Issues with Budget and data

reconciliation Lack of an integrated

management system Baseline fluctuations and

frequent replanning Current period and retroactive

changes Improper use of management

reserve EV techniques that do not

reflect actual performance Lack of predictive variance

analysis

Project Train Wrecks Occur When There is…

79

Our Final Check List

Set up the Risk+ fields, flags, views, and tables for the program standard IMS.

Build an IMS that passes the DCMA 14 Point Assessment with all GREEN.

Build the Ordinal Risk Ranking table for the various risk categories on the program.

Assign risk ranking to each activities in the IMS, with the variances defined in the Ordinal Table.

Run Risk+ to see the confidence in the deliverables.

Develop the needed schedule margin to protect the delivery to at least the 80% confidence level.

80

Advice from the school of hard knocks

Put margin in front of critical deliverables.

Build a margin burn down chart and allocate schedule margin just like you do MR for the PMB.

This real world advice is counter to the current DCMA guidance.

81

Putting This New Knowledge To Work 83

Managing margin is what Risk+ is all about

-40

-20

0

20

40

60

80

100

Critica

l P

ath

- T

ime

Re

se

rve

CP Total Float

Acceptable Rate of Float Erosion

Linear (CP Total Float )

Time Now October 31, 2005

Spacecraft Contract Delivery

December 10, 2007

Float Erosion: Critical Path Time Usage

84

How much margin do we need?

The Missing Link: Schedule Margin Management, Rick Price, PS–10, PMI–CPM EVM World 2008

85

Deterministic versus Probabilistic

Baseline

Plan

80%

Mean

Missed

Launch

Period

Launch

Period

Ready

Early

Sep 2011

Oct 2011

Nov 2001

Dec 2011

Jan 2012

Feb 2012

Mar 2012

Apr 2012

Margin

Risk

Margin

Current Plan

with risks is the

stochastic schedule

CD

R

PD

R

SR

R

FR

R

AT

LO

20%

Current Plan

with risks is the

deterministic schedule

Plan

The probability

distribution can

vary as a

function of time

86

References 88

References

“Protecting Earned Value with Schedule Margin,”


Depicting Schedule Margin in the Integrated Master Schedule,

http://www.ndia.org/Divisions/Divisions/Procurement/Documents/PMSCommittee/C

ommitteeDocuments/WhitePapers/NDIAScheduleMarginWhitePaperFinal-

2010(2).pdf

Effective Risk Management: Some Keys to Success, Second Edition, Edmund Conrow,

AIAA Press.

How to Lie with Statistics, Darrell Huff, Norton, 1954 (Available in paper back at

any good book store)

DID DI–MGMT–81650 “A management method for accommodating schedule

contingencies. It is a designated buffer and shall be identified separately and

considered part of the baseline.

89



http://www.ndia.org/Divisions/Divisions/Procurement/Documents/PMSCommittee/CommitteeDocuments/WhitePapers/NDIAScheduleMarginWhitePaperFinal-2010(2).pdf





References

Interfacing Risk and Earned Value Management, Association for Project Management,

150 West Wycombe Road, High Wycombe, Buckinghamshire, HP12 3AE, United

Kingdom.

Practice Standard for Earned Value Management, Second Edition, Project

Management Institute, 2011.

Effective Opportunity Management for Projects, David Hillson, Taylor and Francis,

2004.

Measuring Time: Improving Project Performance Using Earned Value, Mario

Vanhoucke, Springer, 2009.

Performance Based Earned Value, Paul Solomon and Ralph Young, Wiley, 2007.

Effective Risk Management: Some Keys to Success, Edmund Conrow, AIAA Press,

2003.

90

91

Niwot Ridge LLC

4347 Pebble Beach Drive

Niwot, Colorado 80503

(: 303.241.9633

-: [email protected]

risk management using risk+ (v5)

Technology