cost drivers learning event, 2 nd november 2005 1 correlation tutorial raymond covert, mcr, llc...
TRANSCRIPT
Cost Drivers Learning Event, 2nd November 2005
1
Correlation Tutorial
Raymond Covert, MCR, LLC
Timothy Anderson, The Aerospace Corporation ([email protected])
This tutorial was developed by the authors at:
The Aerospace Corporation
15049 Conference Center Drive,
Suite 600
Chantilly, VA 20151
Copyright © 2004 The Aerospace Corporation
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 2
Outline
1. Introduction to Correlation in Risk Analysis
2. A Statistical View of Cost Analysis
3. Types of Correlation
4. The Correlation Matrix
5. Deriving Correlation Coefficients
Cost Drivers Learning Event, 2nd November 2005
3
Part 1Introduction to Correlation
in Risk AnalysisPurpose of Section - To Answer These 6
Questions About Correlation:1. Who Should Understand It
2. What Is It
3. Why Is It Used
4. Where Is It Used
5. When Is It Used
6. How Is It Used
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 4
Who and What
Who Should Understand Correlation ? Correlation should be understood by all cost analysts
performing quantitative cost risk analysis.
What is Correlation?Ref. 1
A measure of association between two variables. It measures how strongly the variables are related, or
change, with each other. If two variables tend to move up or down together, they are
said to be positively correlated. If they tend to move in opposite directions, they are said to be negatively correlated.
The most common statistic for measuring association is the Pearson correlation coefficient, P.
1) www.statlets.com/usermanual/glossary.htm1) www.statlets.com/usermanual/glossary.htm
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 5
Correlation in Risk Analysis (2) Why is correlation used?
To quantify the effects of statistical dependence when performing algebra on random variables.
It has a large impact on the statistical properties of the results, particularly when many random variables are involved.
Example: Dice Roll. What happens when we roll 2 dice and add their result? Assume 3 cases:
Case 1: Uncorrelated. Outcome of 1 die is independent from the other.
Case 2: Negatively correlated. Outcome of 1 die relate to the outcome of the other. If one die is a “6”, the other must be “1”.
Case 3: Positively correlated. Outcome of 1 die is same as the other.
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 6
Example: Dice Roll Roll of the die gives an equal chance of getting an
outcome (1,2,3,4,5 or 6) Equal, discrete probability Uniform discrete distribution of probabilities Variance, 2 = 3.5
What happens when we sum 2 correlated dice?
Roll of Die
0
0.2
0.4
0.6
0.8
1
1 2 3 4 5 6
Roll of Die
Pro
bab
ility
Probability of x, P(x) = 1/6Probability of x, P(x) = 1/6
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 7
Example: Dice RollSum of Dice: Uncorrelated
0
0.2
0.4
0.6
0.8
1
2 3 4 5 6 7 8 9 10 11 12
Sum of Dice
Pro
bab
ility
Sum of Dice: Correlation =+1
0
0.2
0.4
0.6
0.8
1
2 3 4 5 6 7 8 9 10 11 12
Sum of Dice
Pro
bab
ility
Sum of Dice: Correlation =-1
0
0.2
0.4
0.6
0.8
1
2 3 4 5 6 7 8 9 10 11 12
Sum of Dice
Pro
bab
ility
Case 1: = 0Case 1: = 0
Case 2: = -1Case 2: = -1
Case 2: = +1Case 2: = +1
Triangular, discrete shapeModerate variance, 2=6
Mean = 7
Triangular, discrete shapeModerate variance, 2=6
Mean = 7
P(7) = 1P(<>7)=0
No variance, 2=0Mean = 7
P(7) = 1P(<>7)=0
No variance, 2=0Mean = 7
Uniform, discrete shapeP(each even)=1/6, P(odd) =0
Wide variance, 2=14Mean = 7
Uniform, discrete shapeP(each even)=1/6, P(odd) =0
Wide variance, 2=14Mean = 7
2 + 1 = 32 + 1 = 3
5 + 2 = 75 + 2 = 7
2 + 2 = 42 + 2 = 4
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 8
What We Learned From Dice Roll What we learned about the effects of correlation on sums of dice:
It affects the variance and shape It doesn’t affect the mean = 0 changes shape to a discrete triangular distribution =-1 changes shape and removes variance =+1 preserves shape, adds the most variance, and is the same as
multiplying by 2
The sum of dice example used a discretely distributed random variable, but the same rules apply for continuously distributed random variables. Uniform Triangular Normal Lognormal Weibull
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 9
Where and When Correlation Is Used
Where is correlation used? When performing algebra on random variables. Quantifying the effects of random variables in cost
estimates. Summing costs of WBS elements. Multiplying costs by random variables (i.e. Inflation). In exponentiation of one random variable with another (i.e.
learning curves).
When is correlation used? Whenever we have random variables in our estimates. When we use Monte Carlo Simulations. In analytic statistical sums.
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 10
How Correlation Is Used Directly
Through a correlation matrix,
Indirectly By neglecting correlation, we are defining = 0 By “reusing” random variables, we are defining = 1
Example: We define inflation as a random variable and use the same random variable throughout our cost estimate
By multiplying random variables by a constant, we are defining = 1 Example: We define spacecraft weight as a random variable
and use fractions of it to define weights of different subsystems.
0.11.02.0
1.00.11.0
2.01.00.1
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 11
Part 1 Summary All cost analysts performing quantitative cost risk
analysis should understand correlation Correlation measures how strongly the variables are
related, or change, with each other. Correlation affects the variance and shape, but not
the mean We use correlation frequently, but may not even
know it
Cost Drivers Learning Event, 2nd November 2005
12
Part 2A Statistical View of Cost Analysis
Purpose of Section: To Understand the following1. Costs are uncertain quantities
2. Costs can be treated as random variables
3. How correlation affects variance
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 13
A Statistical View of Cost Analysis
WBS Element Costs are Uncertain Quantities That Have “Probability” Distributions And Statistical Characteristics such as Mean, Median, Mode Costs are Random Variables
Our Goal in Cost Risk Analysis is to Combine Element Cost Distributions to Generate Probability Distribution of Total Cost Use Monte-Carlo or Other Statistical Procedure Quantify Confidence in “Best” Estimate of Total Cost, e.g., Mode Read off Mean, 50th Percentile Cost, 70th Percentile Cost, etc.,
from Cumulative Distribution to Estimate Amount of Risk Dollars Needed
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 14
Elements of Risk Model
Cost DriversCost Drivers
Risk DriversRisk Drivers
AssumptionsAssumptions
Cost EstimateModel
Cost EstimateModel
Quantified RiskQuantified Risk
Our Cost / Risk Model Quantifies:Costs
Effects of UncertaintyUncertainty in Program Assumptions
Risks to Program
Our Cost / Risk Model Quantifies:Costs
Effects of UncertaintyUncertainty in Program Assumptions
Risks to Program
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 15
Example Cost Drivers
Component, Assembly, Propellant Weights Cooling Requirements Data-Processing Requirements Power Requirements Solar Array Area Orbit Altitude Thrust Requirements Special Mission Equipment
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 16
Example Risk Drivers Beyond-State-of-the-Art
Technology Cooling Processing Survivability Power Laser Communications
Unusual Production Requirements Large Quantities (Space
Systems) Toxic Materials Yields
Tight Schedules Undeveloped Technology Software Development Supplier Viability
System Integration Multi-contractor Teams System Testing
Limited Resources Program Funding Stretch-
Out Premature Commitment to
RDT&E Phase Unforeseen Events
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 17
Cost-element Probability Distributions
Best Estimate
Low Risk
High Risk
Low Cost, High Risk vs.
High Cost, Low Risk
Best Estimate
Narrow Symmetric distribution: equal
Probability of actual cost higher or lower than best estimate
Narrow Symmetric distribution: equal
Probability of actual cost higher or lower than best estimate
Wide, Right Skewed distributionLower point
estimate, but high probability of actual
cost greater than point estimate
Wide, Right Skewed distributionLower point
estimate, but high probability of actual
cost greater than point estimate
These curves tell two very
different stories
These curves tell two very
different stories
Would you believe both could come
from the same estimate?
Would you believe both could come
from the same estimate?
mode = mean
mode = mean
meanmean
modemode
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 18
Correlation Affects the Variance are Costs of WBS Elements (Random
Variables) and n = number of WBS elements
Total Cost =
Mean of Total Cost =
Variance of Total Cost =
=
A Very Important RelationshipA Very Important Relationship
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 19
Variance Measures Dispersion Small
Large
X
Area = 1.00
X
Area = 1.00
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 20
Does Correlation Matter? If WBS-Element Costs are Uncorrelated (all ij = 0),
Variance of Total Cost =
If WBS-Element Costs are Correlated,
Variance of Total Cost =– Positive Correlations Increase Dispersion
– Negative Correlations Reduce Dispersion
If (“Worst” Case) All Correlations,
Variance of Total Cost =
“Ignoring” Correlation Issue is Tantamount to Setting all ij = 0
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 21
Yes, Correlation Matters Suppose for Simplicity
There are n Cost Elements
Each
Each
Total Cost
C C Cn1 2, , ,
Var C i 2
Corr C Ci j, 1
Var C Var C Var C Var Cik
n
i
n
j i
n
i j
1 1
1
12
n n n 2 21
n n 2 1 1
Correlation 0 1
VarC n2 n n 21 1 n22
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 22
Magnitude of Correlation Impact
Percent Underestimation of Total-Cost Sigma When Correlation Assumed to be 0 instead of is 100% times ...
Percent Overestimation of Total-Cost Sigma When Correlation Assumed to be 1 instead of is 100% times ...
Var C
Var C
n n n
n n
n
n
1 1
1 1 1 11
n n n
n n n
1 1
1 11
1
1 1
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 23
Maximum Possible Underestimation of Total-cost Sigma
Percent Underestimated * 100% When
Correlation Assumed to be 0 Instead of
1
1
1 1n
0
20
40
60
80
100
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Actual Correlation
Per
cent U
nder
estim
ated
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Actual Correlation
Per
cent U
nder
estim
ated
n = 10
n = 30
n = 100n = 1000
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 24
Maximum Possible Overestimation of Total-Cost Sigma
Percent Overestimated When Correlation Assumed to be 1 Instead of
0
50
100
150
200
250
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Actual Correlation
Pe
rce
nt
Ove
res
tim
ate
d
Limit as n
n =10
0
50
100
150
200
250
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Actual Correlation
Pe
rce
nt
Ove
res
tim
ate
d
Limit as n
n =10
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 25
Part 2 Summary
In this section we learned: Costs are uncertain quantities Costs can be treated as random variables Correlation affects variance
Especially when we are summing large numbers of WBS elements
Remember that in the total cost distribution: The means add The standard deviation is the square root of the variance
The variance =
Cost Drivers Learning Event, 2nd November 2005
26
Part 3Types of Correlation
Purpose of Section is to learn about:1. Functional and Statistical Correlation
2. Pearson and Spearman Correlation
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 27
Types of Correlation
Functional (Causal) Correlation Between cost drivers (Cost Engineering Tools) Between CERs (Cost dependent CERs – SEITPM)
Statistical Correlation Between CER errors
Residual analysis (USCM,SSCM, NAFCOM) Retro-ICE method Estimated based on Number of WBS items
Between Engineering drivers Between complexity, weight, power, etc.
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 28
Functional Correlation Between Cost Drivers
Cost drivers are functionally correlated Beginning of Life Power, EPS Weight, Solar Array Weight,
Battery Weight, RCS Weight, etc…
Use Sizing equations from cost engineering tools to examine the causal relationship
Two Good Examples: Electrical Power System Sizing Reaction Control System Sizing
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 29
Functional (Causal) CorrelationBetween Cost Drivers
POWERREQUIREMENT
LOAD CONTROL
UNIT POWER /VOLTAGEOUTPUT
PRIMARY BUSVOLTAGE
POWERLEVELS
POWERDISTRIBUTION
POWER /VOLTAGE LOSS
PRIMARY POWERSOURCE
REQUIREMENT
EOL SOLARARRAY POWERREQUIREMENT
ORBITAL DATA
EOL SOLARPOWER OUTPUT
BATTERYISOLATION
DIODEPOWER LOSS
SOLAR ARRAYPARASITIC
LOSSES
BATTERYDISCHARGEVOLTAGE /
POWER
BOL SOLARARRAY POWEROUTPUT
SOLAR ARRAYPOWER LOSS
FACTOR
SOLAR ARRAY SHUNT
DISSIPATIONREQUIREMENTS
THEORETICALBATTERY PEAK
DISCHARGEENERGY REQT.
MISSION CYCLES
DEPTH OFDISCHARGE
REQUIREDBATTERY
DISCHARGECAPACITY
NUMBER OFBATTERIES IN
PARALLEL
BATTERYCHARGER I/OVOLTAGE AND
OUTPUTPOWER
POWERSYSTEMSPECIFICWEIGHT /POWER
ACTUAL BATTERYDEPTH OF
DISCHARGE
Example: Electrical Power System SizingExample: Electrical Power System Sizing
Equations in Design loop are functional correlationsEquations in Design loop are functional correlations
Power Requirement drives EPS WeightPower Requirement drives EPS Weight
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 30
Functional CorrelationBetween Cost Drivers
Feedback Example: Reaction Control System Feedback Example: Reaction Control System
Reaction Control SelectionNatural Disturbance Torques SV
Dimensions
Solar Array Pointing ErrorSlew Rate
Antenna Pointing ErrorSlew Rate
Inertia:SV BodyAntennasSolar Array
Reaction Torque
RCS Size
RCS Power
RCS Weight
GimbalTorques
RCS Weight and size drives SV Body InertiaRCS Weight and size drives SV Body Inertia
RCS Power drives Solar Array InertiaRCS Power drives Solar Array Inertia
RCS Size drives SV DimensionsRCS Size drives SV Dimensions
Equations in Design loop are functional correlationsEquations in Design loop are functional correlations
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 31
There are two basic types of Cost Estimating Relationships (CERs).
1. Design Parameter Dependent: Subsystem Hardware (HW) CERs which use weight (or other design parameter) as a base.
2. Cost Dependent: Systems Engineering, Integration and Test, and Program Management (SEITPM) CERs which use estimated cost as a base.
CERs are Used Serially in Risk Analysis.1. Subsystem HW estimated costs are driven by subsystem
estimated weights (or other cost drivers).2. SEITPM estimated costs are driven by HW cost estimates.3. So, the variance of the SEITPM cost estimate is functionally
correlated to the HW cost estimates.
Functional Correlation Between CERs
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 32
Mathematically Total estimate is a sum of the subsystem and SEITPM
costs:*
Subsystem estimates follow the form:
SEITPM estimate follows the form:
Est
N
iiEstEst SEITPM$SS$TOTAL$
1,
ib
iiiEstiWTaSS$ )(,
N
ii
biiS
bEst
iWTaxaxSEITPM$1
)(where
*Note: The “Total cost” is actually a sum of the subsystem and SEITPM costs. It is not the sum of all costs associated with a spacecraft.
Error terms for SS estimates Error terms for SS estimates
Error term for SEITPM
Error term for SEITPM
Weight Weight
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 33
Mathematically
So, the SEITPM estimate is actually represented by:
And the total estimate is represented by:
S
bN
ii
biiEst
iWTaaSEITPM$
1
)(
S
bN
ii
bii
N
ii
biiEst
ii WTaaWTaTOTAL$
11
)()(
Functional correlation Functional correlation
SS CERs SS CERs SEITPM SEITPM
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 34
Statistical Correlation Statistical Correlation
Between CER errors. Residual analysis (USCM,SSCM, NAFCOM). Retro-ICE method. Estimated based on Number of WBS items.
Between Engineering drivers. Between complexity, weight, power, etc.
Look at 2 types of statistical correlation Pearson’s correlation Spearman’s correlation
When you have dataWhen you have data
When you have to guess
When you have to guess
When you have data but don’t
know functionalrelationships
When you have data but don’t
know functionalrelationships
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 35
Two Types of Statistical Correlation Pearson Product-Moment Linear Correlation
if and only if X and Y are linearly related, i.e., the least-squares linear relationship between X and Y allows us to predict Y precisely, given X
= proportion of variation in Y that can be explained on the basis of a least-squares linear relationship between X and Y
if and only if the least-squares linear relationship between X and Y provides no ability to predict Y, given X
Spearman Rank Correlation if and only if the largest value of X corresponds to the
largest value of Y , the second largest, ... , etc.
if and only if the largest value of X corresponds to the smallest value of Y, etc.
if and only if the rank of a particular X among all X values provides no ability to predict the rank of the corresponding Y among all Y values
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 36
Pearson “Product-Moment” Correlation Suppose X and Y are Two Random Variables
are their Expected Values (“Means”)
True Theorem:
False Theorem:
“Covariance” of X and Y
“Variance” of X
“Variance” of Y
“Correlation” of X and Y =
X YE X E Y ,
22, XEXEXXCovXVar
E X Y E X E Y X Y
E XY E X E Y
Cov X Y E XY E X E Y,
22, YEYEYYCovYVar
Corr X YCov X Y
Var X Var Y,
,
Cov X Y XY X Y, Var X Var YX Y 2 2, ,
XY
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 37
Pearson Correlation Measures Linearity A Statistical Relationship Between Two Random
Variables X and Y
Realizations of Y, Given Actual Values of X:
X
Y
0X
Y
0
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 38
Spearman Rank Correlation Coefficient Data Structure
Statistics Theorem: Spearman Rank Correlation Coefficient Equals Pearson (Linear) Correlation Coefficient Calculated
Between the Two Sets of Ranks
CASE
RANK OF Xi VALUE Xj VALUE
DIFFERENCE
SQUARED DIFFERENCE
#1 r1 c1 d1 = c1 - r1 d12
#2 r2 c2 d2 = c2 - r2 d22
#3 r3 c3 d3 = c3 - r3 d32
.
.
.
.
.
.
.
.
.
. . . . . .
.
.
.
#4 rn cn dn = cn - rn dn2
SUMS nn( )1
2 nn( )1
2 d c r 0 d 2
ss
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 39
Linear vs. Rank Correlation
0 0
-0.4
.
.s
076
0..13
.
.s
LINEARLINEAR POWERPOWER “KNEE”“KNEE”
ROOTROOT DECAY w/ OUTLIERDECAY w/ OUTLIER RANDOM w/ OUTLIERRANDOM w/ OUTLIER
More NonlinearMore Nonlinear
Linear Data gives similar and sLinear Data gives similar and s
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 40
What Does Correlation Measure?
PEARSON Correlation Measures Extent of LINEARITY of a Relationship Between Two Random Variables
SPEARMAN Correlation Measures Extent of MONOTONICITY of a Relationship Between Two Random Variables
A way to remember:
…L M N O P Q R S…
A way to remember:
…L M N O P Q R S…
Rank
Is Spearman
Rank
Is SpearmanLinear Is PearsonLinear Is Pearson
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 41
Part 3 Summary
Two types of correlation Functional and Statistical Correlation Functional correlation affects Cost Drivers and Cost
Dependent CERs Statistical correlation is an observed relationship between
data
Two types of Correlation Statistics Pearson and Spearman Correlation or Linear and Rank Correlation They are similar when the data is linear They are different when data is not linear
Cost Drivers Learning Event, 2nd November 2005
42
Part 4The Correlation Matrix
Purpose of Section is to learn :1. Correlation in Risk Rollups
2. Anatomy of a Correlation Matrix
3. How to use a Correlation Matrix
4. Which Common Cost Models Handle Correlation
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 43
Cost-Risk Procedure
Use analytic or Statistical sampling methods to arrive at a total cost distribution
WBS-ELEMENTTRIANGULAR DISTRIBUTIONS
MERGE WBS-ELEMENT COST DISTRIBUTIONSINTO TOTAL-COST DISTRIBUTION
BEST ESTIMATE COST
(MOST LIKELY)
70th PERCENTILE
COST
RISKDOLLAR
S
$
Note: Addition of risk dollars brings confidence that total appropriation (best estimate plus risk dollars) is sufficient to fund program.
L1 B1H1 $
L2 = B2 H2 $
L3 B3 H3 $
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 44
Total Cost Variance
Remember from Part 1, the Total cost variance,
T Correlation matrix (full matrix)
Vector of standard deviations (cost space)
Excel Commands SIGMA_TOT=SQRT(MMULT(MMULT(TRANSPOSE(SIGMA),RHO),SIGMA))
kj
k
jjk
n
k
n
kkTotal
1
121
22 2
Correlation is Essential in calculating variance!
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 45
Representing Correlation Matrices
Full Matrix (Have to use this when you use analytic function: )
Upper Triangular:
Lower Triangular:
1 0.2 0.14 0.37 0.20.2 1 0.06 0.15 0.12
0.14 0.06 1 -0.2 0.060.37 0.15 -0.2 1 0.15
0.2 0.12 0.06 0.15 1
1 0.2 0.14 0.37 0.21 0.06 0.15 0.12
1 -0.2 0.061 0.15
1
10.2 1
0.14 0.06 10.37 0.15 -0.2 1
0.2 0.12 0.06 0.15 1
All 3 representations mean the same thingAll 3 representations mean the same thing
TT
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 46
Representing Correlation Matrices
Single value shorthand:
This means all of the off diagonal terms are the same value
The Rules: Always positive definite Diagonal terms always 1.0 Off diagonal terms are correlation values Columns and rows are transposed, j,k = k,j
Now for some practical examples
11
11
1
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 47
Cost Risk Rollup ProcedureWBS Items Segment
(Launch, Space, Ground)
Want to form a Distribution for
Total Cost of Program
Spacecraft Bus Subsystems
Payload Subsystems
Payload SEITPM(function of payload subsystem cost)
System(Spacecraft Bus, Payload)
Subsystem 1
Subsystem 2
Subsystem N
f($)
Subsystem 1
Subsystem 2
Subsystem N
f($)
Launch
Other Elements
Spacecraft Bus SEITPM(function of spacecraft bus subsystem cost)
Inter-element Correlation needed for All Rollup (Summed)
Costs
Inter-element Correlation needed for All Rollup (Summed)
Costs
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 48
Spacecraft Bus: USCM7 Correlation Coefficients
Correlation coefficients for USCM7 Weight based, Mean Unbiased Percentage Error (MUPE) CERs Average correlation coefficient = 0.160
AD
CS
NR
AG
EN
R
CO
MM
NR
EP
SN
R
IAT
NR
PR
OG
NR
ST
RC
NR
TH
ER
NR
TT
CN
R
AD
CS
T1
AK
MT
1
CO
MM
T1
EP
ST
1
IAT
T1
LO
OS
T1
PR
OG
T1
ST
RC
T1
TH
ER
T1
TT
CT
1
ADCSNR 1.000 -0.067 -0.096 -0.035 0.035 0.012 0.413 0.605 0.121 -0.095 0.983 -0.122 0.099 0.564 0.139 0.089 -0.047 -0.057 0.092AGENR 1.000 -0.028 0.525 -0.079 0.127 0.091 -0.230 -0.125 0.416 0.001 0.085 -0.043 -0.163 -0.189 0.033 0.146 0.151 0.232COMMNR 1.000 0.888 0.884 0.966 0.762 0.281 0.850 -0.166 0.305 -0.176 0.157 0.368 0.884 -0.158 0.109 0.037 -0.004EPSNR 1.000 0.265 0.604 0.409 0.003 0.337 0.237 0.011 -0.275 0.076 0.342 0.021 -0.049 0.465 0.123 0.035IATNR 1.000 0.721 0.615 0.331 0.747 -0.037 0.391 -0.133 -0.028 0.501 0.265 -0.145 0.113 -0.014 -0.189PROGNR 1.000 0.697 0.222 0.868 -0.065 0.145 -0.191 -0.044 0.444 0.329 -0.191 -0.000 -0.125 0.019STRCNR 1.000 0.837 0.761 -0.001 0.117 -0.214 -0.113 0.418 0.173 -0.018 0.220 -0.103 0.069THERNR 1.000 0.077 -0.200 0.662 -0.171 -0.053 0.514 0.102 -0.010 -0.063 -0.165 0.092TT CNR 1.000 -0.149 0.475 -0.118 -0.071 0.519 0.294 -0.178 -0.111 -0.095 0.022ADCST1 1.000 -0.100 0.614 0.421 -0.262 -0.354 0.543 0.676 -0.029 0.655AKMT1 1.000 -0.006 0.292 0.855 0.286 0.176 -0.003 -0.027 0.052COMMT1 1.000 0.266 -0.454 -0.088 0.777 0.729 0.126 0.391EPST1 1.000 -0.150 -0.145 0.381 0.388 -0.007 0.520IATT1 1.000 0.448 -0.144 -0.224 -0.014 -0.320LOOST1 1.000 -0.336 -0.097 -0.074 -0.169PROGT1 1.000 0.421 -0.039 0.481STRCT1 1.000 -0.175 0.285THERT1 1.000 -0.140TT CT1 1.000
These correlation coefficients should not be used for all spacecraft cost models
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 49
The “Big” WBS Case Study (ISS Risk Estimate) Suppose a risk analyst diligently applies distributions to all costs at
the “level of estimating” – this is good. Assume that:
There are 300 cost elements (N=300) There are about four cost elements in each subsystem (n=4) There are (N/n = 75 subsystems) Correlation is defined between all elements within a subsystem
This means: That
WBS elements are correlated Only about 1% of the cost elements are correlated Risk is very narrow and understated
The correlation appears “just-off-the diagonal” of the correlation matrix – This is bad.
010033.0
89700
900
299*300
3*4)4/300(
)1(*
)1(*)/(
NN
nnnN
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 50
“Just-Off-Diagonal” Correlation
Some tools cannot support this function Some nominal statistical correlation does exist Even a few percent makes a big difference with a big WBS
1 0.5 0.51 0.5
1 0.5 0.5 0.51 0.5 0.5
1 0.51 0.5 0.5 0.5 0.5
1 0.5 0.5 0.51 0.5 0.5
1 0.51 0.5 0.5 0.5 0.5
1 0.5 0.5 0.51 0.5 0.5
1 0.51 0.5 0.5 0.5
1 0.5 0.51 0.5
1 0.5 0.5 0.51 0.5 0.5
1 0.51
These Inter-Subsystem WBS Elements are Effectively Uncorrelated
These Intra-Subsystem WBS Elements are Correlated
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 51
How to Use Correlation Matrices
Typically, we wouldn’t want to define all of the correlation coefficients for a big WBS (>10 elements)
We can break it up into parts, get the statistics and then sum at higher levels Reduces the size of correlation matrices Provides Risk Breakout by WBS Summary Level
Lets use an example of a “Big” WBS with 40 elements
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 52
Yuck 40 Individual WBS Elements and the correlation
matrixSEITPM
Systems EngineeringIntegration & TestProgram ManagementConfiguration ManagementData
SpaceSpace Vehicle SEITPMSpace Vehicle
Spacecarft BusBus Systems EngineeringBus I&TBus PMBus DataStructures & MechanismsThermal ControlAttitude Determination & ControlTTC / C&DHPropulsionElecrical PowerLOOSAGE
PayloadPL SEITPMOptical TelescopePanchomatic SensorMultispectral SensorSpectrometerMagnetometerGravitometerUV Sensor
GroundGround SEITPMGround TerminalMission PlanningSatellite OPS / ControlData Archive and Dissemination
LaunchLaunch VehicleLaunch Systems IntegrationLaunch Vehicle IntegrationOn-Orbit CheckoutLaunch Vehicle SE
OperationsSEITPMMaintenanceMission PlanningMission OpsData Archive and Dissemination
1 0.2 0.4 0.3 0.3 0.1 0.1 0.3 0.2 0.1 0.1 0.1 0 0.2 0.3 0.1 0 0.2 0.4 0.5 0.5 0.1 0.1 0.4 0.4 0.1 0.1 0.4 0.2 0 0.1 0.3 0.2 0.3 0.3 0.4 0.3 0.21 0.1 0.2 0.3 0 0.2 0.4 0.2 0.4 0.3 0.3 0.2 0.5 0.1 0.1 0.1 0.3 0.5 0.3 0 0.4 0.1 0.1 0.4 0.1 0.1 0.5 0 0.3 0.1 0.1 0.4 0.2 0.5 0.2 0.5 0
1 0.3 0.5 0.3 0.3 0.2 0.1 0.4 0.3 0.2 0.5 0 0.1 0.5 0.4 0.4 0.4 0.4 0.5 0.3 0.4 0.5 0.2 0 0 0.4 0.1 0.3 0.5 0.3 0 0.1 0.2 0.4 0.5 0.41 0.5 0.1 0.3 0.5 0.2 0.1 0.1 0.3 0.1 0.2 0.5 0.3 0.5 0.3 0.3 0.3 0.2 0.5 0.3 0.1 0.4 0.3 0.1 0.5 0.4 0.4 0.5 0.4 0.4 0.4 0.2 0.4 0.5 0.2
1 0.2 0.4 0.1 0 0.3 0.3 0.4 0 0.2 0.4 0.5 0.3 0.2 0.1 0.2 0.5 0.2 0.4 0.5 0.2 0.2 0.4 0.4 0.4 0.5 0.3 0.4 0.5 0 0.2 0.3 0.3 0.21 0.1 0.2 0 0.1 0.2 0.3 0 0.5 0.2 0.1 0.3 0.1 0.3 0.3 0.2 0.2 0.2 0.1 0.1 0.4 0.1 0.1 0.2 0.4 0.4 0.2 0 0.5 0.5 0 0.1 0.1
1 0.3 0.2 0.3 0.3 0.1 0.1 0.1 0.4 0 0.1 0.3 0.4 0.2 0.4 0.4 0.2 0.1 0.1 0.3 0 0.4 0.3 0.5 0.1 0.4 0.5 0.5 0.1 0.1 0.2 0.41 0.4 0.4 0.2 0.2 0.5 0.5 0 0.4 0 0.2 0.2 0.3 0.1 0.4 0.2 0 0.1 0.4 0.1 0.1 0.1 0 0.1 0.4 0.4 0 0.3 0.5 0 0.3
1 0.1 0 0.1 0 0.2 0.5 0.4 0.3 0.2 0 0.5 0 0.4 0.3 0.5 0.1 0.1 0 0.4 0.2 0.1 0.4 0.2 0.1 0.1 0.1 0.4 0.3 0.11 0.4 0 0.2 0.5 0.1 0.3 0.5 0.4 0.3 0.1 0.1 0.5 0.4 0 0.5 0.1 0.5 0.1 0.2 0.1 0.2 0.1 0.1 0.2 0.3 0.5 0.1 0.2
1 0.4 0.3 0.3 0.1 0.1 0.1 0.3 0.5 0.3 0.1 0.5 0.1 0.3 0.4 0.2 0.2 0.2 0.5 0.1 0.5 0.2 0.3 0 0.1 0.1 0.2 0.51 0.2 0.1 0 0.1 0.3 0 0.3 0.3 0.4 0.2 0.5 0.1 0 0 0.2 0.2 0.1 0.4 0.3 0 0.3 0.2 0.3 0.4 0.3 0.3
1 0.2 0.1 0.1 0.2 0.4 0.5 0.4 0 0.3 0.5 0.2 0.2 0.3 0 0.4 0.1 0.1 0.2 0.1 0.2 0.4 0.3 0.2 0.1 0.11 0.2 0.1 0.2 0.3 0.1 0.5 0.2 0.4 0.4 0.5 0.5 0.4 0.4 0.2 0.3 0.2 0.4 0.3 0.1 0 0.1 0.5 0.3 0.3
1 0.2 0.1 0.1 0.2 0.4 0.2 0.2 0.3 0.5 0.4 0.2 0.4 0.4 0 0.3 0.4 0.4 0.2 0.3 0 0.4 0.2 0.31 0.3 0.1 0 0 0.2 0.2 0.3 0.4 0 0.1 0.4 0.1 0.3 0.1 0.4 0.1 0.3 0.2 0.2 0.1 0.4 0.1
1 0.3 0.4 0.3 0.4 0.1 0.4 0 0.4 0.1 0.4 0.1 0.4 0.3 0.5 0.2 0.2 0.3 0.4 0.5 0.1 0.41 0.3 0.3 0.2 0 0.3 0.5 0.4 0.4 0.4 0.4 0.2 0.4 0.2 0.1 0.3 0.3 0.5 0.4 0.5 0.4
1 0 0.5 0.3 0 0.3 0.3 0.2 0.5 0.4 0.1 0.3 0.4 0.4 0.4 0.5 0 0.2 0.5 0.31 0.3 0.1 0.5 0.3 0.1 0.2 0.3 0.1 0.1 0.4 0.4 0.2 0.1 0.1 0.3 0.1 0.2 0.1
1 0.1 0.1 0.1 0.2 0.4 0.3 0.5 0.4 0.1 0 0.5 0.2 0.4 0.1 0.2 0.5 0.31 0.3 0.3 0.2 0.5 0.3 0.1 0.4 0.3 0.3 0.2 0.2 0.3 0 0.5 0.3 0.4
1 0.1 0.1 0.5 0.3 0.5 0.4 0.1 0.2 0.3 0.2 0.5 0 0.3 0.1 0.41 0.2 0.1 0.4 0.3 0.3 0.3 0.1 0.4 0.2 0.2 0.2 0.2 0.5 0.3
1 0.5 0.5 0.2 0.3 0.1 0.3 0.4 0.3 0.4 0.3 0.3 0.2 0.31 0.3 0.2 0.5 0.3 0.5 0.1 0.4 0.3 0.1 0.3 0.1 0.2
1 0.5 0.3 0.1 0.1 0.2 0.1 0.4 0.2 0.2 0.5 0.21 0.2 0.4 0.5 0.4 0.1 0 0.5 0.3 0.3 0.1
1 0.1 0.2 0.3 0.1 0 0.4 0.2 0.3 0.51 0.5 0.2 0.2 0.3 0.1 0.4 0.4 0.4
1 0.1 0.1 0.3 0.2 0 0 0.41 0.2 0.2 0.4 0.2 0.1 0.4
1 0.4 0.1 0.2 0.3 0.21 0.4 0.3 0.3 0.1
1 0.1 0 0.21 0 0.4
1 0.21
Imagine 300 WBS Elements!Imagine 300 WBS Elements!
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 53
Big Correlation Matrix LayoutAA
BA
CA
DA
EA
AB
BB
CB
DB
EB
AC
BC
CC
DC
EC
AD
BD
CD
DD
ED
AE
BE
CE
DE
EE
A= SETPM (5 elements)
B= Space Segment(20 elements)
C= Ground Segment(5 elements)
D= Launch Segment(5) elements)
E= Operations Segment(5 elements)
Each Block Represents a group of inter element correlationsThe full matrix requires (40*39)/2=780 different correlations
Each Block Represents a group of inter element correlationsThe full matrix requires (40*39)/2=780 different correlations
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 54
Multilevel Risk
Look at the problem one small set of pieces at a time
Now do the same for SPACE, LAUNCH, GROUND, O&M
Mean SigmaSEITPM 11.22 2.70
Systems Engineering 1.2 0.24Integration & Test 1.8 0.72Program Management 0.9 0.27Configuration Management 7.2 2.16Data 0.12 0.048
SEITPM Mean = Sum(Means)SEITPM Sigma =SQRT(MMULT(MMULT(TRANSPOSE(sigma),correl_matrix),sigma))
SEITPM Mean = Sum(Means)SEITPM Sigma =SQRT(MMULT(MMULT(TRANSPOSE(sigma),correl_matrix),sigma))
1 0.3 0.3 0.3 0.30.3 1 0.3 0.3 0.30.3 0.3 1 0.3 0.30.3 0.3 0.3 1 0.30.3 0.3 0.3 0.3 1
SEITPMSEITPM
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 55
Space Element Risk In the Space Element, first break-out the Bus calculation
Space Vehicle SEITPM is one line item,
Let’s assume we already calculated mean and sigma for the Payload, like we did for the Bus
Mean SigmaSpacecarft Bus 40 7.48
Bus Systems Engineering 3.2 1.12Bus I&T 3.3 1.32Bus PM 1 0.25Bus Data 0.5 0.2Structures & Mechanisms 1 0.3Thermal Control 1 0.3Attitude Determination & Control 8 2.4TTC / C&DH 10 3Propulsion 6 1.8Elecrical Power 3 0.9LOOS 2 0.6AGE 1 0.3
1 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.250.25 1 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.250.25 0.25 1 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.250.25 0.25 0.25 1 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.250.25 0.25 0.25 0.25 1 0.25 0.25 0.25 0.25 0.25 0.25 0.250.25 0.25 0.25 0.25 0.25 1 0.25 0.25 0.25 0.25 0.25 0.250.25 0.25 0.25 0.25 0.25 0.25 1 0.25 0.25 0.25 0.25 0.250.25 0.25 0.25 0.25 0.25 0.25 0.25 1 0.25 0.25 0.25 0.250.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 1 0.25 0.25 0.250.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 1 0.25 0.250.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 1 0.250.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 0.25 1
S/C BusS/C Bus
Mean SigmaPayload 83.0 17.74
Mean SigmaSpace Vehicle SEITPM 43.1 19.38
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 56
Space Element Risk
Now we roll-up 3 Items:
Use a small correlation matrix:
The result is:
Mean SigmaSpace Vehicle SEITPM 43.1 19.38Spacecarft Bus 40.0 7.48Payload 83.0 17.74
1 0.4 0.40.4 1 0.40.4 0.4 1
Mean SigmaSPACE 166.1 35.26
Cost
0
0.002
0.004
0.006
0.008
0.01
0.012
0.014
0 100 200 300 400
PD
F
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 57
Economics of Multi-Level Risk After summing all of the elements, we used :
144 Correlation Coefficients vs. 780 (one big matrix)
Views of Risk at all roll-up levels Easier to obtain values for correlation coefficients
We will discuss this in the next part
# Elements # RhosTotal System 5 10SEITPM 5 10Space Element 0 0
12 668 28
Ground 5 10Launch 5 10Ops 5 10
Number of Rhos 144
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 58
What We Just DidAA
BA
CA
DA
EA
AB
BB
CB
DB
EB
AC
BC
CC
DC
EC
AD
BD
CD
DD
ED
AE
BE
CE
DE
EE
A= SETPM (5 elements)
B= Space Segment(20 elements)
C= Ground Segment(5 elements)
D= Launch Segment(5) elements)
E= Operations Segment(5 elements)
Relied on AA, BB, CC, DD, and EE correlationRelied on AA, BB, CC, DD, and EE correlation
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 59
MathematicallyStep1: Calculate
; Where , and
Step 2: Need correlation coefficients of partition AA, BB and all s to
calculate
BA ,
5
4
3
2
1
B
A
2
1
A
5
4
3
B
BA ,
5445
53354334
522542243223
5115411431132112
25
24
23
22
21
2 2
Tot
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 60
Mathematically
Step 3: Calculate total variance using
This is useful when :• We know the correlation between subsystem elements • But not the correlation between subsystems from different elements to
each other (i.e., thermal control SS in spacecraft to ground Command and control CSCIs)
• But do have an idea of correlation between higher-level elements like space to ground.
ABBAABBATot 2222
522542243223511541143113 BAAB
54455335433425
24
232112
22
21
522542243223511541143113
22
AB
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 61
Mathematically
11
21
31
41
51
12
22
32
42
52
13
23
33
43
53
14
24
34
44
54
15
25
35
45
55
522542243223511541143113 BAAB
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 62
Models That Handle Correlation
PRICE*
NAFCOM*
SSCM*
USCM + OthersCannot
Full
Partial
SimulationAnalyticDiscrete
FRISK
@RISK and CB
Monte Carlo Simulators Cost Model w/ Risk*
SEER*
RI$K
SICM
With Analytic or Simulation Method & correlationStudy
Type of Risk Analysis Method
Lev
el o
f C
orr
elat
ion
Sp
ecif
icat
ion
© 2003 The Aerospace Corporation
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 63
Part 4 Summary
Showed how correlation is used in Risk Rollups Provided the anatomy of a correlation matrix
1’s on diagonals Correlation coefficients on off-diagonals Rows and columns are transposes of each other
How to use a correlation matrix Breaking down big risk jobs into smaller pieces Easier to understand Easier to correlate
Showed which common cost models handle correlation
Cost Drivers Learning Event, 2nd November 2005
64
Part 5Deriving
Correlation CoefficientsPurpose of Section is to learn how to derive correlation
coefficients:1. When data is available
2. When you have to make an educated guess
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 65
Deriving Correlation Coefficients 2 Ways to derive correlation coefficients
Data Available:(CADRE, CERs)Data Available:(CADRE, CERs)
No Data:Educated Guess
No Data:Educated Guess
ResidualAnalysisResidualAnalysis
Retro-ICE
Retro-ICE
Causal Guess
Causal Guess
N-Effect Guess
N-Effect Guess
StatisticalStatistical Non-StatisticalNon-Statistical
Effective
Effective
Knee in curve(Steve Book Method)
Knee in curve(Steve Book Method)
Cost Drivers Learning Event, 2nd November 2005
66
Determining Correlation When Data is Available
Data Available:(CADRE, CERs)Data Available:(CADRE, CERs)
No Data:Educated Guess
No Data:Educated Guess
ResidualAnalysisResidualAnalysis
Retro-ICE
Retro-ICE
Causal Guess
Causal Guess
N-Effect Guess
N-Effect Guess
StatisticalStatistical Non-StatisticalNon-Statistical
Effective
Effective
Knee in curve(Steve Book Method)
Knee in curve(Steve Book Method)
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 67
Determining Correlation When Data is Available Statistical Correlation
Residual analysis (USCM,SSCM, NAFCOM) (HARD) Need Database of cost and cost drivers+ CERs+ CER errors Only have good bus correlations with standard models
Retro-ICE method (HARD) Need actual cost data from several similar programs + similar WBS
structure+ total error+ similar models Estimated based on Number of WBS items (EASY)
Need number of WBS items + typical uncertainty Strong function of number of correlated elements Decreases with number of correlated elements
Functional (Causal) Correlation Between cost drivers (HARD)
Need Cost Engineering Tools (CDC, SWAP model) Between CERs (EASY)
Use cost dependent CERs (SEITPM, etc) linked to summary costs in model
© 2003 The Aerospace Corporation
Cost Drivers Learning Event, 2nd November 2005
68
Residual Analysis Method
Data Available:(CADRE, CERs)Data Available:(CADRE, CERs)
No Data:Educated Guess
No Data:Educated Guess
ResidualAnalysisResidualAnalysis
Retro-ICE
Retro-ICE
Causal Guess
Causal Guess
N-Effect Guess
N-Effect Guess
StatisticalStatistical Non-StatisticalNon-Statistical
Effective
Effective
Knee in curve(Steve Book Method)
Knee in curve(Steve Book Method)
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 69
Statistical Correlation From Residual Analysis
Percentage error or standard error are a measure of residual errors
Uncertainty and risk calculations Use residual errors to represent uncertainty Correlation between residuals
Cost vs. Weight
0
500
1000
1500
2000
2500
3000
0 20 40 60 80 100
Weight (lbs)
Co
st (
$K)
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 70
Deriving Correlation Coefficients Sample calculation using randomly generated
numbers Error Xi and Error Yi represent regression residuals for 2
CERs (X and Y) for 8 programs
22mimi
mimi
jk
yyxx
yyxx
PROGRAM Error, Xi Error, Yi (Xi-Xm) (Yi-Ym) (Xi-Xm)(Yi-Ym) (Xi-Xm)^2 (Yi-Ym)^21 0.5404 0.4224 0.1102 0.0167 0.0018 0.0121 0.00032 0.4943 0.3719 0.0641 -0.0339 -0.0022 0.0041 0.00113 0.4496 0.4340 0.0194 0.0282 0.0005 0.0004 0.00084 0.0088 0.2598 -0.4214 -0.1460 0.0615 0.1776 0.02135 0.5679 0.4291 0.1377 0.0234 0.0032 0.0190 0.00056 0.4486 0.5126 0.0184 0.1069 0.0020 0.0003 0.01147 0.7960 0.5357 0.3659 0.1300 0.0475 0.1339 0.01698 0.1359 0.2804 -0.2943 -0.1253 0.0369 0.0866 0.0157
SUM 0.1513 0.4340 0.0681MEAN 0.4302 0.4057RHO 0.8804 = 0.151 / SQRT(0.434 * 0.068)
Cost Drivers Learning Event, 2nd November 2005
71
Retro-ICE Method
Data Available:(CADRE, CERs)Data Available:(CADRE, CERs)
No Data:Educated Guess
No Data:Educated Guess
ResidualAnalysisResidualAnalysis
Retro-ICE
Retro-ICE
Causal Guess
Causal Guess
N-Effect Guess
N-Effect Guess
StatisticalStatistical Non-StatisticalNon-Statistical
Effective
Effective
Knee in curve(Steve Book Method)
Knee in curve(Steve Book Method)
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 72
Retro-ICE Method
Another way to look at correlation is the effect on the variance of the total vs. the variance of the components
Example: SPACE Segment Contains 3 elements: Space SEITPM, Spacecraft Bus,
Payload
What you will need: Actuals for some programs (8 or so) Estimates using your “new” cost model or method (You will
be doing a Retro-ICE)
Use the equation for Pearson correlation:
22mimi
mimi
jk
yyxx
yyxx
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 73
Retro-ICE Example
Start with a table of actuals and estimates for 8 programs and the 3 WBS elements you wish to determine correlation
8 Programs8 Programs
Actual CostsActual Costs Re-Estimated Costs(From Retro-ICEs)
Re-Estimated Costs(From Retro-ICEs)
Program SV SEITPM Bus Payload SV SEITPMBus PayloadA 33 55 30 28 50 28B 42 40 80 40 30 42C 40 60 45 45 68 47D 28 35 33 28 40 30E 24 42 22 20 35 20F 72 80 100 50 75 45G 16 20 23 8 15 18H 28 60 14 18 50 13
Actuals Estimates
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 74
Retro-ICE Example Calculate the Residuals (Actuals – Estimate)
For Each WBS Element (and each program) For the Total
Find the standard deviation, , of each set of residuals For Each WBS Element (8.05, 6.63, 21.26) For the Total (30.53)
Program SV SEITPM Bus Payload TotalA 5 5 2 12B 2 10 38 50C -5 -8 -2 -15D 0 -5 3 -2E 4 7 2 13F 22 5 55 82G 8 5 5 18H 10 10 1 21Stdev 8.048957342 6.631903 21.26029 30.53306
Estimating Error
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 75
Retro-ICE Example Construct the correlation matrix
Ones on the diagonal Use Excel “CORREL” function on the off-diagonals Remember the row/column transpose rule
Check your work:
The result should match the standard deviation of the total, 30.53
1 0.525211 0.6394730.525210993 1 0.342461
0.63947252 0.342461 1
8.05 6.63 21.26 1 0.52521 0.63947 8.05* 0.52521 1 0.34246 * 6.63 = 30.53
0.63947 0.34246 1 21.26( )^2( )^2
Cost Drivers Learning Event, 2nd November 2005
76
Effective Correlation
Data Available:(CADRE, CERs)Data Available:(CADRE, CERs)
No Data:Educated Guess
No Data:Educated Guess
ResidualAnalysisResidualAnalysis
Retro-ICE
Retro-ICE
Causal Guess
Causal Guess
N-Effect Guess
N-Effect Guess
StatisticalStatistical Non-StatisticalNon-Statistical
Effective
Effective
Knee in curve(Steve Book Method)
Knee in curve(Steve Book Method)
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 77
Effective Correlation Effective correlation is different from average
correlation, or the average of the correlation values in the upper (or lower) triangle of the correlation matrix
Effective correlation is weighted by the value of the standard deviation of the constituent elements
The effective correlation may be much different from the average correlation
Look at SSCM as an example
n
k
k
jkj
n
kTotal
eff
2
1
1
1
22
2
© 2003 The Aerospace Corporation
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 78
Effective Correlation SSCM has an average correlation of 0.04, but an
effective correlation of 0.10 Effective correlation was calculated with SSCM the
following way:1. Calculate the total cost error of SSCM
• For each data point in the database: Calculate the sum of actual cost database Calculate the sum of the estimated costs Determine the percentage error, and take the average
2 For each data point, calculate the SSCM error by multiplying the sum of actual costs by the SSCM percent error, SSCM and square to get
TOT
%24SSCM
© 2003 The Aerospace Corporation
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 79
3. Then, for each data point, calculate i by multiplying the actual cost for each WBS element by its respective percent error
4. Calculate the dot product of i with itself to get i
5. Now use the following formula to get the effective correlation, eff for that data point.
6. Finally, get the average of the effective correlations to get eff for the model
The eff for SSCM is 0.10 We should figure this out for all of our models!
Effective Correlation
n
k
k
jkj
n
kTotal
eff
2
1
1
1
22
2
© 2003 The Aerospace Corporation
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 80
Effective Correlation with the Retro ICE Method
Remember the Retro Ice Example?
Use the effective correlation equation to solve for eff
Program SV SEITPM Bus Payload TotalA 5 5 2 12B 2 10 38 50C -5 -8 -2 -15D 0 -5 3 -2E 4 7 2 13F 22 5 55 82G 8 5 5 18H 10 10 1 21Stdev 8.048957342 6.631903 21.26029 30.53306
Estimating Error
508.0)26.21*63.626.21*05.863.6*05.8(*2
)26.2163.605.8(53.30
2
2222
2
1
1
1
22
n
k
k
jkj
n
kTotal
eff
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 81
Effective Correlation with the Retro ICE Method
Retro ICE correlation Matrix Average = 0.502
Retro ICE effective correlation eff =0.508
1 0.525211 0.6394730.525210993 1 0.342461
0.63947252 0.342461 1
1 0.508 0.5080.508 1 0.5080.508 0.508 1
Cost Drivers Learning Event, 2nd November 2005
82
Determining Correlation When Data is Not Available
Data Available:(CADRE, CERs)Data Available:(CADRE, CERs)
No Data:Educated Guess
No Data:Educated Guess
ResidualAnalysisResidualAnalysis
Retro-ICE
Retro-ICE
Causal Guess
Causal Guess
N-Effect Guess
N-Effect Guess
StatisticalStatistical Non-StatisticalNon-Statistical
Effective
Effective
Knee in curve(Steve Book Method)
Knee in curve(Steve Book Method)
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 83
The Problem
It is not always possible to calculate statistical correlation between WBS elements. May be insufficient data to determine statistical correlation. May be no known functional relationship between WBS
elements.
Yet, there may be reason to believe increases or decreases in the cost of a certain WBS element are likely to cause corresponding increases or decreases in the cost of another WBS element.
In cases such as these, it is still desirable to construct a correlation matrix in order to ensure a truer picture of the total cost variance.
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 84
Potential Solutions What can you do if you cannot construct a correlation
matrix from statistical or empirical means? Assume independence
Same as a correlation matrix of zeros. Extremely easy – but, as we have shown, it is WRONG!
Use Knee-in-Curve Method (Steve Book’s Rule of Thumb) When in doubt, assume all correlation values are 0.2. Captures about 80% of the variance compared to assuming
independence. Easy to do. Use the “N-effect”
Modulate our guess at correlation by preserving total error of the estimate
Develop a subjective correlation matrix Excellent results if you can do it accurately. Can fill out an entire correlation matrix this way, but is
somewhat difficult.
Cost Drivers Learning Event, 2nd November 2005
85
Knee-in-Curve Method (Steve Book’s Rule of Thumb)
Data Available:(CADRE, CERs)Data Available:(CADRE, CERs)
No Data:Educated Guess
No Data:Educated Guess
ResidualAnalysisResidualAnalysis
Retro-ICE
Retro-ICE
Causal Guess
Causal Guess
N-Effect Guess
N-Effect Guess
StatisticalStatistical Non-StatisticalNon-Statistical
Effective
Effective
Knee in curve(Steve Book Method)
Knee in curve(Steve Book Method)
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 86
Steve Book’s Rule of Thumb
According to Dr. Steve Book…
“Ignoring” correlation issue is not a good strategy Tantamount to setting all correlations equal to zero
Problem will not go away
Actual numerical values of inter-WBS-element correlations are difficult to establish, but your estimate and range will be closer to the truth if you use “reasonable” nonzero correlations rather than zeroes.
“Ignoring” correlation issue is not a good strategy Tantamount to setting all correlations equal to zero
Problem will not go away
Actual numerical values of inter-WBS-element correlations are difficult to establish, but your estimate and range will be closer to the truth if you use “reasonable” nonzero correlations rather than zeroes.
From: 1999 Cost Risk Analysis Seminar, Manhattan Beach, CA
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 87
Steve Book’s Rule of Thumb
Dr. Book plotted the theoretical underestimation of percent total cost standard deviation when correlation is assumed to be zero rather than its true value, .
0
20
40
60
80
100
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Actual Correlation
Per
cent U
nder
estim
ated
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Actual Correlation
Per
cent U
nder
estim
ated
n = 10
n = 30
n = 100n = 1000
From: 1999 Cost Risk Analysis Seminar, Manhattan Beach, CA
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 88
Steve Book’s Rule of Thumb
For example, given a 30 x 30 correlation matrix in which each actual correlation coefficient is 0.2, if you were to instead assume each correlation coefficient’s value is zero, then you would underestimate the standard deviation of the resulting cost probability distribution by about 60%.
Dr. Book argues that since the “knee” of these curves lies at about 0.2, then it is better to populate an unknown correlation matrix with 0.2’s, or 0.3’s, rather than zeros. Doing so will reasonably capture a substantial amount of
cost variance over doing nothing.
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 89
Steve Book’s Rule of Thumb
Again, according to Dr. Book…
Repeat: “ignoring” correlation is equivalent to assuming that risks are uncorrelated, e.g., all correlations are zero
If you don’t know the “exact” correlation (and nobody does), you will be closer to the truth if you use “reasonable” nonzero correlations rather than zeroes 0.2 is at “knee” of curve on previous charts, thereby providing
most of the benefits at the least loss of accuracy
Higher values are also worth considering
Repeat: “ignoring” correlation is equivalent to assuming that risks are uncorrelated, e.g., all correlations are zero
If you don’t know the “exact” correlation (and nobody does), you will be closer to the truth if you use “reasonable” nonzero correlations rather than zeroes 0.2 is at “knee” of curve on previous charts, thereby providing
most of the benefits at the least loss of accuracy
Higher values are also worth considering
From: 1999 Cost Risk Analysis Seminar, Manhattan Beach, CA
Cost Drivers Learning Event, 2nd November 2005
90
“N-effect” Correlation
Data Available:(CADRE, CERs)Data Available:(CADRE, CERs)
No Data:Educated Guess
No Data:Educated Guess
ResidualAnalysisResidualAnalysis
Retro-ICE
Retro-ICE
Causal Guess
Causal Guess
N-Effect Guess
N-Effect Guess
StatisticalStatistical Non-StatisticalNon-Statistical
Effective
Effective
Knee in curve(Steve Book Method)
Knee in curve(Steve Book Method)
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 91
“N-effect” Correlation
As N increases, the effective correlation (eff) will decrease in reaction to the central limit theorem. This is the “N-effect”
Why? There is a fundamental limit to the predictive capability of our CERs. Just by breaking the WBS up into more pieces doesn’t improve our estimates.
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 92
“N-effect” Correlation Average Correlation* in models seem to be sensitive to number
(N) of CERs
As N increases, decreases
© 2003 The Aerospace Corporation
Maximum Possible Underestimation of Total-Cost Sigma
0
10
20
30
40
50
60
70
80
90
100
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Actual Correlation
Per
cen
t U
nd
eres
tim
ated
NAFCOM N= 55
USCM7 Bus N= 19
USCM7 FU Bus N= 11
USCM8 Bus N= 17
USCM8 Comm N= 13
SSCM N= 9
•The average correlation is different from the effective correlation, but the effect is similar
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 93
Determining Correlation from the Number of WBS Items
There appears to be a trend between the number of WBS Elements (N) in a cost model and the derived average correlation coefficient (AVG) and effective correlation EFF
EFF is a single number used to fill the correlation matrix
As N increases, EFF decreases We looked at the following models:
NAFCOM (NASA/ Air Force Cost Model) USCM7 (Unmanned Space Vehicle Cost Model, Ver. 7) USCM8 (Unmanned Space Vehicle Cost Model, Ver. 8) SSCM (Small Satellite Cost Model)
11
11
1
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 94
Determining Correlation from Number of WBS Items
If we see a trend in the chart of percent under-estimation of sigma vs. effective correlation, we have a sound basis for determining correlations when the number of WBS elements grows.
If the actual percent underestimated is k then the N-effect correlation N for a model with N CERs would be:
So, for k=50%:
1
1
1001
12
N
k
N
10 0.333 15 0.214 20 0.158 30 0.103 50 0.061
100 0.030 150 0.020 200 0.015 300 0.010 500 0.006
© 2003 The Aerospace Corporation
1
3
1
14
1
15.1
1
1
1
1001
1
2
2
NNNN
k
Cost Drivers Learning Event, 2nd November 2005
95
Causal Guess Method of Subjective Correlation(Tim Anderson’s Method)
Data Available:(CADRE, CERs)Data Available:(CADRE, CERs)
No Data:Educated Guess
No Data:Educated Guess
ResidualAnalysisResidualAnalysis
Retro-ICE
Retro-ICE
Causal Guess
Causal Guess
N-Effect Guess
N-Effect Guess
StatisticalStatistical Non-StatisticalNon-Statistical
Effective
Effective
Knee in curve(Steve Book Method)
Knee in curve(Steve Book Method)
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 96
Subjective Correlation
It has previously been shown that it is possible to derive the empirical residual correlation coefficients of a cost model such as USCM, NAFCOM or SSCM. However, this method requires exclusive use of either of
these two cost models to be effective.
One alternative method is to subjectively develop approximate correlation coefficients between WBS elements. This can be as simple as determining whether any two WBS
elements are correlated by a small amount, or by a large amount, and whether that correlation is positive or negative.
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 97
Subjective Correlation An example of a subjective correlation decision table
might look like the following:
For example, if you believe two WBS elements have a small amount of positive correlation, then you would choose a correlation value of 0.3.
Positive
correlation Negative
correlation
Uncorrelated 0 0
Small amount of correlation
0.3 -0.3
Large amount of correlation
0.75 -0.75
Subjective Correlation Coefficients
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 98
Subjective Correlation
Using this technique, it is only necessary to make the following argument between any two WBS elements:
Thus, if the answer were that a change in the cost of one WBS element might cause a minor, similar change in the other WBS element, then one would assign a correlation value of 0.3 between the two WBS elements.
“If circumstances cause the cost of one WBS element to change, is the other WBS element likely to change also? If so,
how substantially and in what direction?”
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 99
But Does It Work? The subjective scoring scheme shown previously is
based on averages. As the figure below shows, the values 0.0, ±0.3, and ±0.75 are the average values in each of their respective ranges.
The idea is this: While one might not know the true correlation between WBS elements, if instead one can subjectively bucket the correlation into one of these five ranges, then a correlation matrix composed of these averages should give approximately the same results as if the true correlations were known.
Negative Correlation Positive Correlation
-0.75 -0.3 0.0 0.3 0.75
0 0.2 0.40.3 0.5 0.70.60.1 0.8 1.00.9-0.9 -0.7-0.8 -0.6 -0.4-0.5-1.0 -0.3 -0.1-0.2
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 100
But Does It Work? To test the hypothesis, suppose we substitute the true
correlation coefficients in a typical correlation matrix with their corresponding averages from the subjective correlation table.
If the resulting cost probability distributions are similar, then we can use the technique with some confidence.
An important point, however, is that the subjectively derived values should reflect the actual values.
If this is true, then this method should give a more precise answer than Steve Book’s Rule of Thumb method.
We will see in the following example
Cost Drivers Learning Event, 2nd November 2005
101
Application of Correlation Methods When Data is Not Available
Cost Risk Analysis of A Small Earth Orbiting
Visible Imaging Sensor
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 102
Example Consider an estimate developed using the following
cost model:Work Breakdown Structure Parameter, X Input data range
RDT&E CER (FY00$K) SE
1. Payload
1.1 Visible Light Sensor Aperture diameter (m) 0.2 - 1.2 128,827 X 0.562 19,3362. Spacecraft
2.1 Structure Structure weight (kg) 54-392 157 X 0.83 38%
2.2 Thermal Thermal weight (kg) 3 - 48 394 X 0.635 45%
2.3 Electrical Power System
X1 = EPS weight (kg), X 2 = BOL
power (wt)
X1: 31 - 491, X 2:
100 - 2400 2.63 (X 1 X2) 0.71236%
2.4 Telemetry, Tracking and Command TT&C weight (kg) 12 - 65 545 X 0.761 57%
2.5 Attitude Determination and Control ADCS weight (kg) 20 - 160 464 X 0.867 48%
3. Integration, Assy. & TestSpacecraft bus + payload total RDT&E cost 2,703 - 395,529 969 + 0.215 X 46%
4. Program Mgmt.Spacecraft bus + payload total RDT&E cost 4,607 - 523,757 1.963 X 0.841 36%
5. Ground Support EquipmentSpacecraft bus + payload total RDT&E cost 24,465 - 581,637 9.262 X 0.642 34%
Work Breakdown Structure Parameter, X Input data range T1 CER (FY00$K) SE1. Payload
1.1 Visible Light Sensor Aperture diameter (m) 0.2 - 1.2 51,469 X 0.562 7,7342. Spacecraft
2.1 Structure Structure weight (kg) 54-560 13.1 X 39%
2.2 Thermal Thermal weight (kg) 3 - 87 50.6 X 0.707 61%
2.3 Electrical Power System EPS weight (kg) 31 - 573 112 X 0.763 44%
2.4 Telemetry, Tracking and Command TT&C weight (kg) 13 - 79 635 X 0.568 41%
2.5 Attitude Determination and Control ADCS weight (kg) 20 - 192 293 X 0.777 34%
3. Integration, Assy. & TestSpacecraft bus weight + payload weight 155 - 1,390 10.4 X 44%
4. Program Mgmt.Spacecraft bus + payload total recurring cost
15,929 - 1,148,084 0.341 X 39%
6. Launch & Orbital Operations SupportSpacecraft bus weight + payload weight 348 - 1,537 4.9 X 42%
RDT&E Cost Model (FY00$K) (less fee)
T1 Cost Model (FY00$K) (less fee)
Reprinted from Space Mission Analysis and Design, 3rd Edition, Wertz and Larson
Reprinted from Space Mission Analysis and Design, 3rd Edition, Wertz and Larson
Reprinted fromSpace Mission AnalysisAnd Design, 3rd Edition.Wertz and Larson
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 103
Input Variables
Suppose this spacecraft has the following set of (mean) input variables:
Cost Driver ValueAperture diameter (m) 1.04Payload Wt (kg) 212Structure Wt (kg) 85Thermal Wt. (kg) 11EPS Wt. (kg) 318BOL Power (W) 1061TT&C Wt. (kg) 52ADCS Wt. (kg) 148
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 104
Functional Relationships Integration and Test, Program Management and LOOS are
functionally correlated to PL and S/C bus cost The input variables are functionally correlated through the
following sizing relationships with their error terms () Payload Wt. = 200 * (Aperture Diameter)^1.5 + PL
Structure Wt. = 0.4 * Payload Wt. + STR
Thermal Wt. = 0.05 * Payload Wt. + TH
BOLP = 5.0 * Payload Wt. + BOLP
EPS Wt. = 0.3 * BOLP + EPS
TTC Wt. = 50 + 0.01 * Payload Wt. + TTC
ADCS Wt = 0.7 * Payload Wt. + ADCS
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 105
Input Parameter Error Terms
Aperture Diameter has a discrete 20% probability of being 1.0m, and a 80% probability of being 1.2m.
Assume the error terms () for the sizing relationships are all triangular probability density functions defined by Low = 0.9 Most Likely = 1.0 High =1.4
1.000.90 1.40
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 106
Cost Estimate The resulting cost estimate has the following
(deterministic) value:
Cost Driver ValueAperture diameter (m) 1.04Payload Wt (kg) 212Structure Wt (kg) 85Thermal Wt. (kg) 11EPS Wt. (kg) 318BOL Power (W) 1061TT&C Wt. (kg) 52ADCS Wt. (kg) 148
RDT&E (FY00$K) $M1. Payload 131.7 1.1 Visible Light Sensor 131.72. Spacecraft 77.2 2.1 Structurre 6.3 2.2 Thermal 1.8 2.3 Electrical Power System 22.7 2.4 Telemetry, Tracking and Command 11.0 2.5 Attitude Determination and Control 35.43. Integration, Assy. & Test 45.94. Program Mgt. 59.56. Launch & Orbital Operations Support 24.1TOTAL RDT&E 338.4
T1 (FY00$K) $M1. Payload 52.6 1.1 Visible Light Sensor 52.62. Spacecraft 30.7 2.1 Structurre 1.1 2.2 Thermal 0.3 2.3 Electrical Power System 9.1 2.4 Telemetry, Tracking and Command 6.0 2.5 Attitude Determination and Control 14.33. Integration, Assy. & Test 8.64. Program Mgt. 28.46. Launch & Orbital Operations Support 4.0TOTAL T1 148.8
Total RDT&E + T1 487.2
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 107
“Actual” Correlation Matrix Suppose the “actual” correlation matrix is as follows:
Ad
ePL
eST
R
eTH
eEP
S
eBO
LP
eTT
C
eAD
CS
RD
T&
E P
ayload
RD
T&
E S
tructure
RD
T&
E T
hermal
RD
T&
E E
PS
RD
T&
E T
T&
C
RD
T&
E A
DC
S
RD
T&
E IA
&T
RD
T&
E P
M
RD
T&
E G
SE
T1 P
ayload
T1 S
tructure
T1 T
hermal
T1 E
PS
T1 T
T&
C
T1 A
DC
S
T1 IA
&T
Ad 1.000 0.950 0.800 0.600 0.308 0.450 0.600 0.751 0.684 0.134 0.471 0.319 0.399 0.050 0.207 0.025 0.316 0.159 0.204 0.164 0.202 0.476 0.188 0.342ePL 1.000 0.540 0.412 0.247 0.354 0.450 0.654 0.009 0.183 0.066 0.357 0.323 0.423 0.110 0.015 0.083 0.397 0.149 0.069 0.072 0.273 0.259 0.339eSTR 1.000 0.436 0.540 0.423 0.005 0.010 0.267 0.144 0.020 0.491 0.213 0.343 0.078 0.359 0.088 0.043 0.444 0.122 0.123 0.223 0.165 0.023eTH 1.000 0.353 0.258 0.117 0.006 0.337 0.270 0.491 0.242 0.063 0.063 0.036 0.188 0.314 0.016 0.084 0.156 0.192 0.417 0.209 0.118eEPS 1.000 0.840 0.119 0.428 0.468 0.002 0.086 0.289 0.254 0.157 0.415 0.371 0.142 0.038 0.468 0.218 0.101 0.205 0.314 0.294eBOLP 1.000 0.187 0.235 0.358 0.412 0.187 0.245 0.101 0.252 0.427 0.098 0.296 0.477 0.050 0.116 0.160 0.191 0.430 0.020eTTC 1.000 0.334 0.454 0.088 0.224 0.046 0.286 0.186 0.478 0.405 0.459 0.264 0.145 0.142 0.246 0.357 0.093 0.326eADCS 1.000 0.240 0.367 0.078 0.024 0.451 0.367 0.328 0.481 0.003 0.329 0.053 0.342 0.460 0.368 0.387 0.355RDT&E Payload 1.000 0.508 0.071 -0.424 -0.029 -0.681 0.359 -0.565 -0.208 -0.498 0.586 0.181 -0.121 -0.054 -0.571 0.812RDT&E Structure 1.000 0.411 -0.778 0.162 0.381 -0.851 -0.776 -0.612 -0.583 0.493 -0.894 0.392 -0.474 -0.660 -0.071RDT&E Thermal 1.000 -0.059 0.881 0.652 0.705 0.171 -0.566 0.882 0.011 0.415 -0.978 0.816 0.613 0.275RDT&E EPS 1.000 -0.990 0.371 -0.177 0.987 -0.953 0.921 -0.021 0.497 -0.668 0.319 -0.658 -0.696RDT&E TT&C 1.000 -0.180 -0.570 0.216 0.126 -0.216 0.735 -0.701 -0.435 -0.965 -0.986 -0.931RDT&E ADCS 1.000 0.401 0.139 -0.325 0.785 0.251 -0.175 0.559 -0.460 -0.426 0.011RDT&E IA&T 1.000 -0.182 0.600 -0.318 0.055 -0.813 0.068 0.781 0.468 -0.236RDT&E PM 1.000 -0.984 0.695 0.548 0.435 -0.528 -0.735 0.759 0.153RDT&E GSE 1.000 0.400 0.224 0.965 -0.566 0.275 -0.389 0.052T1 Payload 1.000 0.138 -0.893 -0.224 -0.981 -0.264 0.610T1 Structure 1.000 0.186 -0.126 0.049 0.528 -0.989T1 Thermal 1.000 -0.002 -0.542 0.217 -0.681T1 EPS 1.000 -0.039 0.969 -0.399T1 TT&C 1.000 -0.927 0.382T1 ADCS 1.000 -0.314T1 IA&T 1.000T1 PMT1 GSE
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 108
Total Cost Distribution: Actual Correlation Values
Total cost distribution with “actual” correlation values:
$M
Lognormal: Mean = $534.7M Std Dev = $126.6M
Lognormal: Mean = $534.7M Std Dev = $126.6M
Note: This is the distribution that followsas a result of using the“actual” correlationcoefficients. The standard deviation is 24% of the mean.
Note: This is the distribution that followsas a result of using the“actual” correlationcoefficients. The standard deviation is 24% of the mean.
Frequency Chart
.000
.027
.054
.081
.109
0
135.7
271.5
407.2
543
0.0 312.5 625.0 937.5 1250.0
5,000 Trials 0 Outliers
Forecast: Total RDT&E + T1
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 109
Total Cost Distribution: No Statistical Correlation
We have included functional correlation, but no statistical correlation between the error terms,
Under these circumstances, the total cost distribution has the following appearance and statistics:
Frequency Chart
.000
.031
.062
.093
.124
0
154.5
309
463.5
618
0.0 312.5 625.0 937.5 1250.0
5,000 Trials 0 Outliers
Forecast: Total RDT&E + T1
Note: This is a narrow distribution. The standard deviation is 15%of the mean. This is very different from the percent errors of our CERs, which ranged from 34% to 61%.
Note: This is a narrow distribution. The standard deviation is 15%of the mean. This is very different from the percent errors of our CERs, which ranged from 34% to 61%.
$M
Lognormal: Mean = $525.8M Std Dev = $79.7M
Lognormal: Mean = $525.8M Std Dev = $79.7M
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 110
Total Cost Distribution: Knee-in-Curve Method
Using Steve Book’s Rule of Thumb, the correlation is set to 0.2, and the total cost distribution has the following appearance and statistics:
$M
Note: This has caused the standard deviation to shift substantially compared to the zero correlation case.
The standard deviation grew by over 76% from the uncorrelated case.
Note: This has caused the standard deviation to shift substantially compared to the zero correlation case.
The standard deviation grew by over 76% from the uncorrelated case.
Frequency Chart
.000
.019
.039
.058
.078
0
97.25
194.5
291.7
389
0.0 312.5 625.0 937.5 1250.0
5,000 Trials 1 Outlier
Forecast: Total RDT&E + T1
Lognormal: Mean = $535.3M Std Dev = $148.2M
Lognormal: Mean = $535.3M Std Dev = $148.2M
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 111
Total Cost Distribution: “N-Effect” Method Using the N-Effect method, the correlation is 0.125,
and the total cost distribution has the following appearance and statistics:
Note: This has caused the standard deviation to shift substantially compared to the zero correlation case.
The standard deviation grew by over 59% from the uncorrelated case.This is almost exactly the Standard error of the “actual correlation” case.
Note: This has caused the standard deviation to shift substantially compared to the zero correlation case.
The standard deviation grew by over 59% from the uncorrelated case.This is almost exactly the Standard error of the “actual correlation” case.
$M
Frequency Chart
.000
.022
.044
.066
.088
0
110.5
221
331.5
442
0.0 312.5 625.0 937.5 1250.0
5,000 Trials 0 Outliers
Forecast: Total RDT&E + T1
Lognormal: Mean = $532.6M Std Dev = $127.0M
Lognormal: Mean = $532.6M Std Dev = $127.0M
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 112
“Subjective” Correlation Matrix The corresponding “subjective” correlation matrix
using the causal guess method is as follows:
Ad
ePL
eST
R
eTH
eEP
S
eBO
LP
eTT
C
eAD
CS
RD
T&
E P
ayload
RD
T&
E S
tructure
RD
T&
E T
hermal
RD
T&
E E
PS
RD
T&
E T
T&
C
RD
T&
E A
DC
S
RD
T&
E IA
&T
RD
T&
E P
M
RD
T&
E G
SE
T1 P
ayload
T1 S
tructure
T1 T
hermal
T1 E
PS
T1 T
T&
C
T1 A
DC
S
T1 IA
&T
Ad 1.000 0.750 0.750 0.750 0.300 0.300 0.750 0.750 0.750 0.300 0.300 0.300 0.300 0.000 0.300 0.000 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.300ePL 1.000 0.750 0.300 0.300 0.300 0.300 0.750 0.000 0.300 0.000 0.300 0.300 0.300 0.300 0.000 0.000 0.300 0.300 0.000 0.000 0.300 0.300 0.300eSTR 1.000 0.300 0.750 0.300 0.000 0.000 0.300 0.300 0.000 0.300 0.300 0.300 0.000 0.300 0.000 0.000 0.300 0.300 0.300 0.300 0.300 0.000eTH 1.000 0.300 0.300 0.300 0.000 0.300 0.300 0.300 0.300 0.000 0.000 0.000 0.300 0.300 0.000 0.000 0.300 0.300 0.300 0.300 0.300eEPS 1.000 0.750 0.300 0.300 0.300 0.000 0.000 0.300 0.300 0.300 0.300 0.300 0.300 0.000 0.300 0.300 0.300 0.300 0.300 0.300eBOLP 1.000 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.000 0.300 0.300 0.000 0.300 0.300 0.300 0.300 0.000eTTC 1.000 0.300 0.300 0.000 0.300 0.000 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.300 0.000 0.300eADCS 1.000 0.300 0.300 0.000 0.000 0.300 0.300 0.300 0.300 0.000 0.300 0.000 0.300 0.300 0.300 0.300 0.300RDT&E Payload 1.000 0.750 0.000 -0.300 0.000 -0.750 0.300 -0.750 -0.300 -0.300 0.750 0.300 -0.300 0.000 -0.750 0.750RDT&E Structure 1.000 0.300 -0.750 0.300 0.300 -0.750 -0.750 -0.750 -0.750 0.300 -0.750 0.300 -0.300 -0.750 0.000RDT&E Thermal 1.000 0.000 0.750 0.750 0.750 0.300 -0.750 0.750 0.000 0.300 -0.750 0.750 0.750 0.300RDT&E EPS 1.000 -0.750 0.300 -0.300 0.750 -0.750 0.750 0.000 0.300 -0.750 0.300 -0.750 -0.750RDT&E TT&C 1.000 -0.300 -0.750 0.300 0.300 -0.300 0.750 -0.750 -0.300 -0.750 -0.750 -0.750RDT&E ADCS 1.000 0.300 0.300 -0.300 0.750 0.300 -0.300 0.750 -0.300 -0.300 0.000RDT&E IA&T 1.000 -0.300 0.750 -0.300 0.000 -0.750 0.000 0.750 0.300 -0.300RDT&E PM 1.000 -0.750 0.750 0.750 0.300 -0.750 -0.750 0.750 0.300RDT&E GSE 1.000 0.300 0.300 0.750 -0.750 0.300 -0.300 0.000T1 Payload 1.000 0.300 -0.750 -0.300 -0.750 -0.300 0.750T1 Structure 1.000 0.300 -0.300 0.000 0.750 -0.750T1 Thermal 1.000 0.000 -0.750 0.300 -0.750T1 EPS 1.000 0.000 0.750 -0.300T1 TT&C 1.000 -0.750 0.300T1 ADCS 1.000 -0.300T1 IA&T 1.000T1 PMT1 GSE
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 113
Total Cost Distribution:Causal Guess Subjective Values
Total cost distribution with “subjective” correlation values:
$M
Note: This is the distribution that followsas a result of using the“subjective” correlationcoefficients.
The mean is nearly identical to the “actual” case, and the standarddeviation has increasedby approximately 4%.
Note: This is the distribution that followsas a result of using the“subjective” correlationcoefficients.
The mean is nearly identical to the “actual” case, and the standarddeviation has increasedby approximately 4%.
Frequency Chart
.000
.029
.057
.086
.114
0
142.5
285
427.5
570
0.0 312.5 625.0 937.5 1250.0
5,000 Trials 0 Outliers
Forecast: Total RDT&E + T1
Lognormal: Mean = $531.2M Std Dev = $121.2M
Lognormal: Mean = $531.2M Std Dev = $121.2M
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 114
Summary of Results
0
0.001
0.002
0.003
0.004
0.005
0.006
200 300 400 500 600 700 800 900 1000
Total Cost $M
Lik
liho
od
mean sigma
Actual 534.7 126.6
Uncorrelated 525.8 79.7
Knee-in-curve 535.3 148.2
N-Effect 532.6 127
Causal guess 531.2 121.2
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 115
Summary of Results
Ignoring correlation understated the total cost sigma The N-Effect and Causal Guess methods produced
the best results The Knee-in curve method was close, but provided
the largest variance
Cost Drivers Learning Event, 2nd November 2005
16 September 2005 116
Part 5 Review We learned how to derive correlation coefficients:
When data is available When you have to make an educated guess
The Methods: Statistical (Data is available)
Residual analysis Retro-ICE Effective correlation, eff
When data is not available Knee-in-curve method (Steve Book’s Rule of Thumb) N-effect guess Subjective guess (Tim Anderson’s Method)
Example cost risk analysis using subjective methods