forecasting cost and schedule performance
TRANSCRIPT
Forecasting ACAT1 Program Performance in the Presence of Statistical UncertaintyTime series forecasting of project cost, schedule, and technical performance to increase the probability of success
1
IPMC2013
GlenB.AllemanNiwotRidge,[email protected]
2
Complexity, Risk, and Uncertainty
drive projectperformance
Toimproveprojectsuccesswemustdealwiththisuncertaintywithcredibleforecastsoffutureperformance
Why We Should Care About This?
§ The only certainty is that nothing is certain – Pliny the Elder, Roman Scholar 23-79 CE
§ When will your nice forecast come in contact with reality –Program Manager, Rocky Flats Environmental Technology Site
§ Our culture encodes a strong bias either to neglect to ignore variation
3
The Simple Fact of All Programs
§ All data variables on the program are actually random variables– Cost– Schedule– Technical Performance Measures– Measures of Performance – Measures of Effectiveness– Technical Risk– Programmatic Risk
4
Some Root Causes of Disappointment
§ Our measures of progress, using Earned Value Management, do not explicitly consider compliance with Technical Performance Measures for the products being measured.
§ The performance numbers we gather have had all their statistical variances wiped out through the accumulation process.
§ Without the underlying knowledge of these statistical variances, our biases toward the central tendency emerge – the flaw of averages (missing the variance)
5
Plansbasedonaverage assumptionsarewrongonaverage
What to do about these Root Causes
§ Collect data in a single place to allow time series analysis of these raw data as a normal analysis process– We Have the Central Repository
§ Develop time series analysis tools to forecast credible outcomes from past performance– The R system is free, and can read directly
the CR data§ Use this analysis to have a conversation
about the Probability of Program Successbetween the government and the contract
6
We Currently Apply EV Metrics As If They Were Single Point Estimates, with no Variance
7
Whatwedon’tknowisthese“Cummtodate”numbershavehiddenalltheunderlying variancesandthereisa67%probabilitywe’regoing to“stepinit”inthenext3months
CPI/SPI“Cummtodate”shows0.93/1.07,“we’redoinggoodsofar
Let’s Start with the End in Mind
§ We have 48 months of CPI data in the Central Repository (CR)
§ What will CPI look like in the next 6 months, with confidence intervals?
8
CPI
48MonthsofCPIdata
PossibleCPIvalues6
months inthefuture
Making Forecasts from CR time series§ Using R to plot the CPI time series from our
notional project– CPI from Jan of 2009 to Dec of 2012
§ We want a six month future forecast of the possible values of CPI
§ R can do this with a few commands> CPIts=ts(CPI, start=c(2009, 1), end=c(2012, 12), frequency=12)
> fitCPI=HoltWinters((CPIts, beta=FALSE, gamma=FALSE)> Plot(forecast(CPIts,6))
§ That’s it, that’s all that’s needed to make the plot of the forecast of CPI on the previous page
9Andthebestofall– thistoolisFREE!
NOW WHAT?
With that simple demonstration, let’s look to see why this is important for increasing the Probability of Program Success (PoPS) is we have the time series data in the Central Repository.First of all we have this data already. All we have to do is mine the data in the same way retailers, oil and gas exploration, bio-pharma, transportation firms, and every student in finance does
10
WeneedPrinciples,Practices,andProcessestobuildacredibleEstimateAtCompletionand
managetheprogramalongtheway
11
Let’s Start With Our Current EV Process
§ We hide the underlying variance by wiping out the variances of past performance.
§ We make linear, non-statistical projections with this Cumm to Data
§ We ignore the coupling and correlation between statistical processes
§ We ignore the impact of risk on all forecasts
12
And Look at the Missing Pieces of an Earned Value Management System
§ Earned Value Management fails to deal with the stochastic nature of project work– Critical path is considered static– EV variables don’t embody the statistical
nature of the performance and forecasting– Real-world activities are always statistical with
probabilistic behaviors – The network of activities in the IMS not
modeled as a dynamically coupled system of random processes
13
And Finish with the Positive
§ We have all the data in the CR to make a statistically sound forecast of future performance
§ Each Control Account – and maybe even Work Package – has EV data on a monthly basis for each WBS element
§ With that we can determine …– Coupling and correlation between WBS elements– The underlying stochastic processes of the time
series – Other statistically sound forecasting parameters
from this raw data
14
ThisisthepossiblyunforeseenbenefitoftheCR– notjustviewing butanalyzing innewandpowerways,usingtoolsandprocessesfromotherareas – allforfree
LET’S LOOK AT A NOTIONALLY REAL PROGRAM
The TSAS notional program being used by PARCA with IDA support is a UAV with flight avionics subsystem, baselined in an IMP/IMS with resource loaded, risk adjusted Technical Performance Measures
15
We Think our IMS Looks Like This
16
In Reality the IMS Looks Like This, Plus 1,000’s of More Activities
17
But It’s Actually Far More Complex Than That, It’s a Non-linear Relationship
18
§ The programmatic and planning dynamics act as a coupled system§ The “system response” is the transfer function between input and
output§ Understanding this transfer
function may appear beyond our interest …– But it is part of the
stochastic dynamic response to disruptions in our plans
– “What if” really means “what if” at this point in the response curve of the system
§ Most often this system is also non-linear even though our example is not
Inputs
Outputs
Thisshowslinear, it’sneverlinear
THEMES OF THIS BRIEFING
We’re not going to solve the problem of forecasting program performance in the 45 minutes. We’re not even going to touch on the problem. The best we can do is establish a collection of themes that can be addressed with processes, data, and Essential Views created from our efforts with PARCA
19
IPMC2013
GlenB.AllemanNiwotRidge,[email protected]
Top Level Themes
§ Earned Value metrics are linear representations of project performance derived from prior stochastic processes– SPI/CPI are Cumulative-To-Date measures that
wash out underlying variances.– Current period is single period extension of the
static accumulation of past performance– Information about the past dynamics of the
system is lost– Forecasting the future in this manner ignores the
very data needed to create credible confidence intervals
20
Top Level Themes (Continued)
§ Forecasts of the future using the current EV techniques are: – linear, – non-risk adjusted, – non-performance factor adjusted – Projections from the current base point
measurement.§ No statistical inference or probabilistic
projections are available in the EVM formulas between Control Accounts, within the Control Accounts history, and any consideration of future probabilistic performance of the Control Account
21
Top Level Themes (Continued)
§ Past performance can be used to forecast future performance.
§ We need not only the forecast value but a confidence interval on that value.
§ These confidence intervals come from the underlying statistics and the related probabilities.
§ Statistical forecasting, using time series analysis of past performance, is mandatory for any credible forecast of a project’s future performance.
22
Top Level Themes (Concluded)
§ “All models are wrong, some are useful”§ Time Series Analysis: Forecasting and Control,
George E. P. Box and Gwilym M. Jenkins, Holden-Day, 1976
§ Box actually said …
23
The Core Question Here
24
Canwedetermineaproject’sfutureperformanceintheabsenceofastatisticalforecastingprocess?
TheanswershouldbeNO
UncertaintyiscreatedbyIncompleteKnowledge;NotbyIgnorance
Risk
25
Unspoken Truths of Performance Management
§ Activities in the IMS have non-linear relationships for cost and schedule
§ Each activities have a probabilistic function for the duration, many times unknown, but knowable
§ Multiple inputs to any forecast produce multiple outputs
§ Correlation between the inputs is usually not defined
26
All Program Activities have Naturally Occurring Uncertainties – Aleatory
§ Naturallyoccurringuncertaintyanditsresultingrisk,impactstheprobabilityofasuccessfuloutcome.– Whatistheprobabilityofmakingadesiredcompletiondateorcosttarget?
27
§ Thestatisticalbehavioroftheseactivities,theirarrangementinanetworkofactivities,andcorrelationbetweentheirbehaviorsisthesourceofprogrammaticrisk
§ Addingmargin istheonlywaytoprotecttheoutcomefromtheimpactofthesenaturallyoccurringuncertainties
Raw Material For Forecasting Cost And Schedule Performance
28
Start With The Earned Value Numbers
§ CV/SV and CPI / SPI– That’s it, that’s all we’ve got from the PMB
§ Cumulative to date at the Control Account level§ Current period at the Control Account level
§ Cumulative values wipe out the underlying variance
§ Current period performance not adjusted for past variance
§ Both cumulative and current period not adjusted for risk
29
Add Technical Performance Numbers
§ Technical Performance Measures– Technical behaviors of the tangible deliverables
§ Measures of Performance– System performance
§ Measures of Effectiveness– Customer facing performance of the capabilities
§ Key Performance Parameters– JROC – cost, schedule, sustainment, training,
interoperability– Program Specific
30
…Producearangeofoutcomeswithconfidenceintervals
Our goal in forecasting is to …
31
ForecastsProducePatternsOfPossibleFutureOutcomes
32
33
TheBiggestQuestionis…DoEVmetricsdescribeCause,Correlation,or
Consequences?
Some Forecasting Principles
§ Reliable forecasting is a critical component of project planning, controlling, and risk management.
§ At-Completion forecast made before the project starts is the basis of credible project management.– This is done with a model of the reducible and
irreducible uncertainties in cost, schedule, and technical performance
§ Execution phase forecasting serves as a leading indicator of project success using the same uncertainty models
34
Common Features of a Credible Forecast
§ Forecasting assumes the underlying causal system that existed in the past will continue in the future
§ Forecasts are not perfect, actual values differ from predicted values. The presence of randomness precludes a perfect forecast
§ Forecast accuracy decreases as the time horizon extends
35
Some Important Terms
§ The term confidence interval is applied to interval estimates for fixed but unknown parameters
§ The term prediction interval is an interval estimate for an (unknown) future value.– A prediction interval consists of an upper and
a lower limit at a prescribed probability, which are referred to as prediction bounds
36
A core limitation of the current approach to performance forecasting
§ Standard EVM technique for forecasting the final cost at completion is not applicable to forecasting the project duration at completion†
§ This leads to inconsistent assumptions about the relationships between past performance and future performance– Using CPI creates a future forecast will be the
same as past performance
37†Short1993,Vandevoorde andVanhoucke2006,Leach2005,andLipke2003.
38
Statisticalmodelsarestoriesabouthowthedatacametobewhatitisanhowwecanmakedecisionsbasedonthathistory
Forecasting Methods
§ Probabilistic Forecasting – explicit uncertainties in project performance and errors in measurement provide prediction bounds on the predicted values
§ Integrative Forecasting Methodologies –collect all relevant information from different sources in a mathematically correct manner
§ Consistent Forecasting Methodologies –methods that can be applied to both cost and schedule performance forecasts
39
Elements of a good forecast
§ Timely§ Accurate§ Reliable§ Meaningful units§ Easy to use§ Actionable
40
Three Types of Forecasts
§ Judgmental – Subjective analysis of subjective inputs– The Management EAC is a judgmental
forecast§ Associative Models – Analyzes historical
data to reveal relationships between (easily or in advance) observable quantities and forecast quantities. Uses this relationship to make predictions.
§ Time Series – Objective analysis historical data assuming the future will be like the past
41
Forecasting Techniques
§ Moving average – taking the value of a chosen number of previous periods to use as an estimate for the next period
§ Weighted average – same as moving average except giving greater weight to more recent periods
§ Linear regression – a mathematical algorithm is using past data to create a line showing the direction. The line can then be carried forward to create a forecast
§ Exponential smoothing –combines most recent actual figure with the previous period’s forecast in order forecast an upcoming period
§ Regression analysis – equations that are used to analyze the relationship between a dependent variable and one or more independent variables.
§ Time series – uses recent history is a good predictor of the near future
§ Trend analysis – finds trends in historic data that can be used to make forecasts.
42
Capturingandprocessingdatarequireswork.Makingforecastswiththisdaya essentiallyFree
43
44
§ We’ve got the start of the data capture process§ Cost and schedule information available at the
Control Account level on a monthly basis§ In order to build the Control Account data stream,
the Work Package are used§ Use the Work Package information as well
§ By The Way – UN/CEFACT is a place in Geneva Switzerland, not the name of an XML protocol – Palais des Nations CH-1211 Geneva 10 Switzerland
45
An Example of Additional Insight Gained By Comparing Schedules Against Planned Program
Deliverables
46
FY05 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13
1
25
20
15
10
5
MilestoneCo
unt
3/05Plan(Rev.A)
PreliminaryRev.H5/07Plan(Rev.E)
006
119
264
286
970
530
600
(actualfrom5/07plan)
(1actualfrom5/07plan)
3/05Plan
12/08Prel.RevH5/07Plan
006
1115
3719
51525
14220
19250
2500M/SCount
Cum
3/05Plan
12/08Prel.RevH5/07Plan
M/SCountRate
ActualPlansoftheJamesWebSpaceTelescope(JWST)
JWST Schedules and Deliverables by Subsystems
47
Spacecraft
Sunshield
PDRCDRTestComplI&TCompl
FY05 FY06 FY07 FY08 FY09 FY10 FY11 FY12
OTE
3/05Plan(Rev.A)
PreliminaryRev.H5/07Plan(Rev.E)
3/05Plan
5/07Plan
0
0
6
1
1
9
2
6
4
2
8
6
9
7
0
5
3
0
6
0
0
(actualfrom5/07plan)
(1actualfrom5/07plan)
ISIM
0
0
6
1
1
15
3
7
19
5
15
25
14
22
0
19
25
0
25
0
0
M/SCountRate
M/SCountCum
STR-PDRSTR-CDRETU-NIRSpecETU-MIRIETU-NIRCamETU-FGSETU-I&TFM-NIRCamFM-FGSFM-I&TFM-MIRIFM-NIRSpec
OTEPDRStartPolishOTECDR1stMirrorDeliveryFinalMirrorDelivery
PDRCDRStructComplPropI&T
3/05Plan
Prel.RevH
5/07Plan
Prel.RevH
Exhibit30:DeliverablesPlannedvsActuals
Probabilities of Success Results
48
Case 2: BaselinewithUncertainty,DiscreteRisks@25probability
34.8%
50.8%
57.4%
30Sept2022
Probability ofmeetingbothcostandscheduletargets
Probability ofmeetingtargetedschedule
Probability ofmeetingtargetedcost
Exhibit31:ProbabilityofMeetingCostandScheduleTargets
Intheend,leadingindicatorsmustbeusedtotellushowtotakecorrectiveactionsbeforeweendupintheditch
49
50
OneLastThought…
Askhowcannon-statistical,non-probabilistic,erasedvariancemeasuresofpastperformancebe
usedforanycredibleforecastfutureperformance?
NiwotRidge,L.L.C.4347PebbleBeachDriveLongmont,Colorado80503
Performance-BasedProjectManagement®IntegratedMasterPlan
IntegratedMasterScheduleEarnedValueManagementDCMA/DCAAValidation
ProgrammaticandTechnicalRiskManagementProposalSupport Service
51