mitre february 2005 paul r. garvey chief scientist center for acquisition and systems analysis cost...

MITREMITRE

February 2005February 2005

Paul R. GarveyPaul R. Garvey

Chief ScientistChief ScientistCenter for Acquisition and Systems AnalysisCenter for Acquisition and Systems Analysis

Cost Risk Analysis Cost Risk Analysis Without Statistics!!Without Statistics!!

MITREMITREMITREMITREMITRE Paper MP 050000001©The MITRE Corporation, All Rights Reserved

Approved For Public Release

MITREMITRE2

The QuestionThe Question

A government agency asked the question A government agency asked the question

““Can a valid cost risk analysis (that is traceable and Can a valid cost risk analysis (that is traceable and defensible) be conducted with minimal (to no) defensible) be conducted with minimal (to no) reliance on Monte Carlo simulation or other reliance on Monte Carlo simulation or other statistical methods”?statistical methods”?

The question is motivated by the agency’s unsatisfactory The question is motivated by the agency’s unsatisfactory experiences in developing and implementing Monte Carlo experiences in developing and implementing Monte Carlo simulations to derive “risk-adjusted” costs of future simulations to derive “risk-adjusted” costs of future systemssystems

MITRE Paper MP 050000001©The MITRE Corporation, All Rights Reserved

MITREMITRE3

What These Charts PresentWhat These Charts Present

These charts present an approach that addresses the These charts present an approach that addresses the question posed by the agencyquestion posed by the agency

The approach reflects a “minimum acceptable” method The approach reflects a “minimum acceptable” method whereby a technically valid measure of cost risk can be whereby a technically valid measure of cost risk can be derived without relying on statistical methodsderived without relying on statistical methods

Optional AugmentationOptional Augmentation A “statistically-light” augmentation A “statistically-light” augmentation to this approach is also presented that enables the agency to this approach is also presented that enables the agency to assess probabilities that a future system’s cost will (or to assess probabilities that a future system’s cost will (or will not) be exceededwill not) be exceeded

Best PracticeBest PracticeMonte CarloMonte Carlo simulation, and the supportive statistical theory, simulation, and the supportive statistical theory, practiced routinely by the practiced routinely by the communitycommunity, remains a best technical , remains a best technical

practice for conducting a cost risk analysispractice for conducting a cost risk analysis


MITREMITRE4

The Big Picture: The Big Picture: What is Risk? What is Uncertainty?What is Risk? What is Uncertainty?

Uncertainty vs Risk*Uncertainty vs Risk* RiskRisk is the chance of loss or injury. In a situation that is the chance of loss or injury. In a situation that includes favorable and unfavorable events, includes favorable and unfavorable events, risk is the probability an unfavorable event occurs

UncertaintyUncertainty is the indefiniteness about the outcome of a is the indefiniteness about the outcome of a situation - it includes favorable and unfavorable eventssituation - it includes favorable and unfavorable events

We analyze uncertainty for the purpose of measuring We analyze uncertainty for the purpose of measuring risk!risk!

In systems engineering this analysis might focus on In systems engineering this analysis might focus on measuring the measuring the riskrisk of of {failing to achieve performance objectives}, , {overrunning the budgeted cost}, or , or {delivering the system too late to meet user needs}; ; these are examples of three unfavorable eventsthese are examples of three unfavorable events

* Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y., 10016. MITRE Paper MP 050000001©The MITRE Corporation, All Rights Reserved

MITREMITRE5

The Big Picture: The Big Picture: What is Cost Risk Analysis and Why Conduct It?What is Cost Risk Analysis and Why Conduct It?

What is Cost Uncertainty Analysis?*What is Cost Uncertainty Analysis?* Cost uncertainty analysis is a Cost uncertainty analysis is a

process of quantifying the cost impacts of process of quantifying the cost impacts of uncertainties associated with a system’s technical definition and cost estimation associated with a system’s technical definition and cost estimation methodologiesmethodologies

What is Cost Risk Analysis?*What is Cost Risk Analysis?* Cost risk analysis is a process of Cost risk analysis is a process of

quantifying the cost impacts of quantifying the cost impacts of risks associated with a system’s associated with a system’s technical definition and cost estimation methodologiestechnical definition and cost estimation methodologies

What is Cost Risk?What is Cost Risk? Cost risk is a measure of the chance that, due Cost risk is a measure of the chance that, due to unfavorable events, the planned or budgeted cost of a project to unfavorable events, the planned or budgeted cost of a project will be exceeded will be exceeded

Why Conduct the Analysis?Why Conduct the Analysis? To produce a defensible assessment To produce a defensible assessment of the level of cost to budget such that this cost has an acceptable of the level of cost to budget such that this cost has an acceptable probability of not being exceeded probability of not being exceeded


* Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y., 10016.

MITREMITRE6

A Minimum Acceptable MethodA Minimum Acceptable Method…Nonstatistical…Nonstatistical

Given the “what” and “why” of cost risk analysis, a Given the “what” and “why” of cost risk analysis, a minimum acceptable method is one that operates on a minimum acceptable method is one that operates on a set of specified scenarios that, if they occurred, would set of specified scenarios that, if they occurred, would result in costs higher than the level planned or result in costs higher than the level planned or budgetedbudgeted

These scenarios do not represent extreme worst cases; These scenarios do not represent extreme worst cases; rather, they reflect a set of conditions that a decision-rather, they reflect a set of conditions that a decision-maker would want to have budget to guard against, maker would want to have budget to guard against, should any or all of them occur should any or all of them occur

For purposes of this discussion we’ll characterize this For purposes of this discussion we’ll characterize this minimum acceptable method as the “Scenario-Driven” minimum acceptable method as the “Scenario-Driven” approach to cost risk analysisapproach to cost risk analysis


MITREMITRE7


A Scenario-Driven approach to cost risk analysis derives from A Scenario-Driven approach to cost risk analysis derives from what could be called “sensitivity analysis”, but with one what could be called “sensitivity analysis”, but with one difference...difference...

Instead of arbitrarily varying one or more variables to measure Instead of arbitrarily varying one or more variables to measure the sensitivity (or change) in cost, the Scenario-Driven the sensitivity (or change) in cost, the Scenario-Driven approach approach specifies a well-defined set of technical and/or specifies a well-defined set of technical and/or programmatic conditions that collectively affect a number programmatic conditions that collectively affect a number of cost-related variables and associated WBS cost of cost-related variables and associated WBS cost elements in a way that increase cost beyond what was elements in a way that increase cost beyond what was planned or budgetedplanned or budgeted

Multiple scenarios can be hypothesized, specified, and costed Multiple scenarios can be hypothesized, specified, and costed by the analysis team; the one considered most critical to by the analysis team; the one considered most critical to guard the system’s cost against is the scenario the risk guard the system’s cost against is the scenario the risk analysis should be based on; call this the analysis should be based on; call this the Prime ScenarioPrime Scenario


MITREMITRE8


This approach does not rely on statistics to produce a This approach does not rely on statistics to produce a valid risk-adjusted value of a system’s total costvalid risk-adjusted value of a system’s total cost

However, with the incorporation of just two statistical However, with the incorporation of just two statistical inputs the Scenario-Driven approach can also provide inputs the Scenario-Driven approach can also provide credible percentiles of a system’s total costcredible percentiles of a system’s total cost

The following charts describe this approach and The following charts describe this approach and illustrate it via an example computationillustrate it via an example computation


MITREMITRE9

““Scenario-Driven” Cost Risk AnalysisScenario-Driven” Cost Risk Analysis…Nonstatistical…Nonstatistical

Step 1Step 1 Start with the Baseline (or Point Estimate) Cost Start with the Baseline (or Point Estimate) CostDefine Define CostCostSys, PESys, PE as the total cost of a system, where as the total cost of a system, where CostCostSys, PE Sys, PE is the sum of the cost is the sum of the cost

element costs summed across the system’s work breakdown structure (WBS) element costs summed across the system’s work breakdown structure (WBS) without any adjustment for risk/uncertainty; define this cost as the Baseline or Point without any adjustment for risk/uncertainty; define this cost as the Baseline or Point Estimate costEstimate cost

Step 2Step 2 From the Prime Scenario From the Prime Scenario Define Define CostCostSys, b*Sys, b* as the total cost of a system, where as the total cost of a system, where CostCostSys, b* Sys, b* is the sum of the cost is the sum of the cost

element costs summed across the system’s work breakdown structure (WBS) element costs summed across the system’s work breakdown structure (WBS) adjusted for risks you want to guard against (from a cost overrun perspective); adjusted for risks you want to guard against (from a cost overrun perspective); define this cost as the cost derived from the Prime Scenario; define this cost as the cost derived from the Prime Scenario; CostCostSys, b*Sys, b* is the “risk- is the “risk-

adjusted” total cost of the systemadjusted” total cost of the system

Step 3Step 3 Derive a Measure of Cost Risk (CR) Derive a Measure of Cost Risk (CR)From Step 1 and Step 2 the amount of cost risk is given byFrom Step 1 and Step 2 the amount of cost risk is given by

CR = CR = CostCostSys,b*Sys,b* - - CostCostSys, PESys, PE CR is also a measure of the amount of reserve dollars needed to guard against the CR is also a measure of the amount of reserve dollars needed to guard against the Prime ScenarioPrime Scenario


MITREMITRE10


This approach (although simple in nature) is a valid cost risk This approach (although simple in nature) is a valid cost risk analysis; why?analysis; why?The process of defining these scenarios identifies the The process of defining these scenarios identifies the technical and cost estimation risks inherent in the technical and cost estimation risks inherent in the baseline estimate and enables these risks to become baseline estimate and enables these risks to become fully visible/traceable to decision-makersfully visible/traceable to decision-makers

Defining the “Prime Scenario” from the set of specified Defining the “Prime Scenario” from the set of specified scenarios will guard for at least those sets of risks (e.g., scenarios will guard for at least those sets of risks (e.g., weight growth, SLOC growth, estimation risks (e.g., weight growth, SLOC growth, estimation risks (e.g., levels of effort for SEPM)) considered critical to guard levels of effort for SEPM)) considered critical to guard the baseline cost againstthe baseline cost against

These scenarios are essentially discrete passes through These scenarios are essentially discrete passes through a Monte Carlo simulation; thus, this approach can be a Monte Carlo simulation; thus, this approach can be linked directly to the classical/best practice methodlinked directly to the classical/best practice method

A properly done Monte Carlo simulation must always be A properly done Monte Carlo simulation must always be premised on a set of reasonably well-defined scenariospremised on a set of reasonably well-defined scenarios

Defining these scenarios will build the rational, Defining these scenarios will build the rational, traceable, and analytic basis behind a “derived” traceable, and analytic basis behind a “derived” measure of cost risk (i.e., the amount of cost risk measure of cost risk (i.e., the amount of cost risk reserve needed to guard the baseline cost against the reserve needed to guard the baseline cost against the specified risks)specified risks)

All the above is essentially why we do a cost risk All the above is essentially why we do a cost risk analysis!analysis!

The cost of a scenario is just the sum of the “dots” across the The cost of a scenario is just the sum of the “dots” across the WBS cost element costs; think of this cost as a single pass WBS cost element costs; think of this cost as a single pass through the Monte Carlo process; the above figure is from the through the Monte Carlo process; the above figure is from the reference below*reference below*

* Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y., 10016. MITRE Paper MP 050000001©The MITRE Corporation, All Rights Reserved

MITREMITRE11


What Don’t You Get?What Don’t You Get?Recall that, by definition, cost risk is a measure of the Recall that, by definition, cost risk is a measure of the chancechance that, due to that, due to unfavorable events, the planned or budgeted cost of a project will be exceeded unfavorable events, the planned or budgeted cost of a project will be exceeded

Because this approach is nonstatistical, confidence measures are not produced; Because this approach is nonstatistical, confidence measures are not produced; that is, the probabilities (i.e., the chances) that that is, the probabilities (i.e., the chances) that CostCostSys, PESys, PE or or CostCostSys, b*Sys, b* will (or will will (or will

not) be exceeded are not generatednot) be exceeded are not generated

Can Confidence Measures be Incorporated?Can Confidence Measures be Incorporated?

Yes, Yes, the following charts describe how this can be done while still staying the following charts describe how this can be done while still staying within this general approachwithin this general approach

This is an This is an Optional AugmentationOptional Augmentation to the Scenario-Driven approach to the Scenario-Driven approach


MITREMITRE12

““Scenario-Driven” Cost Risk Analysis Scenario-Driven” Cost Risk Analysis

Optional Optional

AugmentationAugmentation


MITREMITRE13

““Scenario-Driven” Cost Risk AnalysisScenario-Driven” Cost Risk AnalysisOptional Augmentation: Incorporating a Statistical AngleOptional Augmentation: Incorporating a Statistical Angle

Assumption 1Assumption 1 Assess the probability (alpha) that the Assess the probability (alpha) that the “true” cost of the system will fall in the interval “true” cost of the system will fall in the interval

[[ CostCostSys, PESys, PE , , CostCostSys, b* Sys, b* ]]

Assumption 2*Assumption 2* Assume the statistical distribution for Assume the statistical distribution for CostCostSysSys is uniformly distributed with probability alpha is uniformly distributed with probability alpha that the “true” cost falls in the interval that the “true” cost falls in the interval

[[ CostCostSys, PESys, PE , , CostCostSys, b* Sys, b* ]]

Let X = CostLet X = CostSys Sys , a, a11 = Cost = CostSys, PESys, PE , and b , and b11 = Cost = CostSys, b* Sys, b*

x

)(xf X

2

1 2

1

ab1

a 1a 1b b

** Note: Within this Note: Within this assumption it is assumption it is further assumed that further assumed that the distribution of the distribution of the total probability the total probability across the interval across the interval [[aa, , bb] is as shown to ] is as shown to the rightthe right


MITREMITRE14

““Scenario-Driven” Cost Risk AnalysisScenario-Driven” Cost Risk AnalysisOptional Augmentation: Incorporating a Statistical AngleOptional Augmentation: Incorporating a Statistical Angle

Given the three values alpha, Given the three values alpha, CostCostSys, PESys, PE , , andand CostCostSys, b* Sys, b* the the minimum cost minimum cost aa and the maximum cost and the maximum cost bb can be can be derived from the formulas belowderived from the formulas below

Given Given aa and and bb we can then determine any percentile we can then determine any percentile needed from the probability formula belowneeded from the probability formula below

ProbProb ( (CostCostSysSys << xx) = () = (xx - - aa)/()/(bb - - aa)), where, where x x is dollars is dollars

x

)( xfX

2

1 2

1

ab 1

a 1a 1b b

21

21

1)( bXProb

21

22)()( 11 baba

XProbXProb

2

1212

1212

22

11

11

)()(

)()()(

ab

baba

abXVar

XMedXEXMean

X

ab

ab 11 111

ab

ab

21

111 )( abaa

21

111 )( abbb



MITREMITRE15

““Scenario-Driven” Cost Risk AnalysisScenario-Driven” Cost Risk AnalysisOptional Augmentation: Incorporating a Statistical Angle: Example Calculation IOptional Augmentation: Incorporating a Statistical Angle: Example Calculation I

Required Inputs

Dollars ($M)x

This is This is CostCostSysSys; the total; the total

cost of the systemcost of the system

In this example, In this example, xx = $34M is the derived 50th percentile cost; that is, = $34M is the derived 50th percentile cost; that is, ProbProb ( (CostCostSysSys << 34) = (34 - 26.5)/(41.5 - 26.5) = 1/2 34) = (34 - 26.5)/(41.5 - 26.5) = 1/2

a = $26.5M b = $41.5MCostSys, b*

b1 = $40M

x

)(xf X

2

1 2

1

ab1

a 1a 1b b

CostSys, PE

a1 = $28M

Alpha = 0.8

Dollars ($M)

Pro

babi

lity

that

P

roba

bilit

y th

at C

ost

Cost

Sys

Sys is

is

less

tha

n or

equ

al t

o le

ss t

han

or e

qual

to

xx

Outputs ax

abxCostProb Sys

1)(

0


MITREMITRE16

““Scenario-Driven” Cost Risk AnalysisScenario-Driven” Cost Risk AnalysisOptional Augmentation: Incorporating a Statistical Angle: Example Calculation IIOptional Augmentation: Incorporating a Statistical Angle: Example Calculation II

For this approach, an For this approach, an assessment of “alpha” is assessment of “alpha” is requiredrequired

In some cases, that may be In some cases, that may be hard to justify; but let’s hard to justify; but let’s consider some reasonable consider some reasonable possibilitiespossibilities

From past experience, we From past experience, we often see the value of often see the value of CostCostSys, PESys, PE fall between the fall between the

20th and 30th percentiles of 20th and 30th percentiles of the distribution function of a the distribution function of a project’s total cost project’s total cost CostCostSysSys


x

)(xf X

2

1 2

1

ab1

a 1a 1b b

So, if alpha = 0.60 then So, if alpha = 0.60 then CostCostSys, PESys, PE falls at the 20th falls at the 20th

percentile; if alpha = 0.75 then percentile; if alpha = 0.75 then CostCostSys, PESys, PE falls at the 12.5th falls at the 12.5th

percentile; if alpha = 0.90, percentile; if alpha = 0.90, then then CostCostSys, PESys, PE falls at the 5th falls at the 5th

percentile percentile


MITREMITRE17


Thus, we can run the Thus, we can run the analysis with alpha at 0.60, analysis with alpha at 0.60, 0.75, and 0.90 and develop 0.75, and 0.90 and develop a range of “risk-adjusted” a range of “risk-adjusted” costs associated with costs associated with each alphaeach alpha

The “true” alpha will likely The “true” alpha will likely “almost-always” fall within “almost-always” fall within this alpha rangethis alpha range

where X = Costwhere X = CostSys Sys , a, a11 = Cost = CostSys, PESys, PE , and b , and b11 = Cost = CostSys, b* Sys, b*

If If alphaalpha = 0.60 then = 0.60 then ProbProb ( (CostCostSysSys << aa11) = 0.20 ) = 0.20 and and ProbProb ( (CostCostSysSys << bb11) = 0.80) = 0.80



x

)(xf X

2

1 2

1

ab1

a 1a 1b b


MITREMITRE18


24

= 0.60 = 0.75 = 0.90

= 0.60

= 0.90 Dollars ($M)x

axab

xCostProb Sys

1

)(CumProb 0.600 0.750 0.900

0 0.857 0.929 0.9760.050 0.893 0.957 1.0000.100 0.929 0.986 1.0240.150 0.964 1.014 1.0480.200 1.000 1.043 1.0710.250 1.036 1.071 1.0950.300 1.071 1.100 1.1190.350 1.107 1.129 1.1430.400 1.143 1.157 1.1670.450 1.179 1.186 1.1900.500 1.214 1.214 1.2140.550 1.250 1.243 1.2380.600 1.286 1.271 1.2620.650 1.321 1.300 1.2860.700 1.357 1.329 1.3100.750 1.393 1.357 1.3330.800 1.429 1.386 1.3570.850 1.464 1.414 1.3810.900 1.500 1.443 1.4050.950 1.536 1.471 1.4291 1.571 1.500 1.452

The values in the three The values in the three columns from the left-most columns from the left-most column are derived from, column are derived from, and are specific to, the and are specific to, the input parameters input parameters aa11 = 28 = 28

and and bb11 = 40 given for this = 40 given for this

example.example.

Multiply the point estimate Multiply the point estimate by the values shown (in the by the values shown (in the three columns from the three columns from the left-most column) to left-most column) to determine the costs determine the costs associated with the associated with the probabilities shown in the probabilities shown in the left-most column (e.g., the left-most column (e.g., the 75th percentile cost, for 75th percentile cost, for alpha = 0.60, is (1.393)alpha = 0.60, is (1.393)($28M) = $39M($28M) = $39M

0

0.60.600

0.70.755

0.90.900

Again, suppose Again, suppose aa11 = $28M and = $28M and bb11 = $40M; then the value = $40M; then the value

of of CostCostSysSys at (say) the 75th percentile, for alpha equal to at (say) the 75th percentile, for alpha equal to

0.60, 0.75, and 0.90, respectively is…0.60, 0.75, and 0.90, respectively is…

75th Percentile: [$37.33M, $38M, $39M]75th Percentile: [$37.33M, $38M, $39M]

95th Percentile: [$40M, $41.2M, $43M] 95th Percentile: [$40M, $41.2M, $43M]

37.33 38 39

alpha = 0.90 alpha = 0.60alpha = 0.75

alpha = 0.90 alpha = 0.60alpha = 0.75 The decision-maker The decision-maker then selects a value then selects a value that best reflects that best reflects his/her judgement of his/her judgement of the cost needed to the cost needed to guard against the guard against the cost risk posed by cost risk posed by the Prime Scenariothe Prime Scenario


MITREMITRE19

““Scenario-Driven” Cost Risk AnalysisScenario-Driven” Cost Risk AnalysisObservationsObservations

This approach offers the following features This approach offers the following features Provides an analytic argument for deriving the amount of cost reserve needed to guard against Provides an analytic argument for deriving the amount of cost reserve needed to guard against a well-defined “Prime Scenario”a well-defined “Prime Scenario”

Brings the discussion of “scenarios” and their credibility to the decision-makers; this is a more Brings the discussion of “scenarios” and their credibility to the decision-makers; this is a more meaningful topic to focus on, instead of the statistical abstractions the classical analysis can meaningful topic to focus on, instead of the statistical abstractions the classical analysis can sometimes createsometimes create

Does not require the use of statistical methods to develop a valid measure of cost risk reserveDoes not require the use of statistical methods to develop a valid measure of cost risk reserve

Percentiles (confidence measures) can be designed into the approach with a minimum set of Percentiles (confidence measures) can be designed into the approach with a minimum set of statistical assumptions (only two required)statistical assumptions (only two required)

Percentiles (as well as mean, median (50th%), variance, etc) can be calculated algebraically Percentiles (as well as mean, median (50th%), variance, etc) can be calculated algebraically and thus can be executed in near-real time within a simple spreadsheet environment; Monte and thus can be executed in near-real time within a simple spreadsheet environment; Monte Carlo simulation is not needed!Carlo simulation is not needed!

The approach fully supports traceability and focuses decision-makers attention on the key The approach fully supports traceability and focuses decision-makers attention on the key issues that have the potential to drive cost higher than expectedissues that have the potential to drive cost higher than expected

The uniform distribution assumption is a conservative assumption in the sense that the The uniform distribution assumption is a conservative assumption in the sense that the distribution of probabilities is linear across the interval where it’s defineddistribution of probabilities is linear across the interval where it’s defined

Does not require analysts develop probability distribution functions for all the uncertain Does not require analysts develop probability distribution functions for all the uncertain variables in a WBS, which can be time-consuming and hard to justifyvariables in a WBS, which can be time-consuming and hard to justify

Process will automatically allocate the cost risk reserve dollars into the WBS Process will automatically allocate the cost risk reserve dollars into the WBS


MITREMITRE20

SummarySummary

The approach described herein is valid in the broad The approach described herein is valid in the broad context of the principal objective of a cost risk analysis context of the principal objective of a cost risk analysis

These scenarios represent a discrete set of “passes” These scenarios represent a discrete set of “passes” through a Monte Carlo simulationthrough a Monte Carlo simulation

Thus, you’re not capturing the full range of potential cost Thus, you’re not capturing the full range of potential cost outcomes (driven by favorable and unfavorable events) --outcomes (driven by favorable and unfavorable events) --only the outcomes as specified by the scenarios (which only the outcomes as specified by the scenarios (which are driven by unfavorable events)are driven by unfavorable events)


MITREMITRE21

SummarySummary…concluded…concluded


Again, Monte Carlo Again, Monte Carlo simulation remains a best simulation remains a best technical practicetechnical practice

However, further work is However, further work is needed by the community needed by the community on ways to simplify the on ways to simplify the presentation of its outputs presentation of its outputs (and meaning) to senior (and meaning) to senior decision-makers decision-makers

* Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y., 10016. http://www.sceaonline.net/content.asp?contentid=203

mitre february 2005 paul r. garvey chief scientist center for acquisition and systems analysis cost...

Documents

cost uncertainty analysis

valid cost risk analysis

budgeted cost

level of cost

number of cost

future systems cost

valid measure of cost

cost estimation methodologies