[email protected], (703)633-8300 x4536, 12/16/2015, 1 briefing, 35 th adodcas, scea, pmi,...
TRANSCRIPT
[email protected], (703)633-8300 x4536, 04/21/23, 1
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Schedule and Cost GrowthR. L. Coleman, J. R. Summerville, M. E. Dameron
35th ADoDCAS
PMI 2002 National Conference
November, 2002
[email protected], (703)633-8300 x4536, 04/21/23, 2
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Outline• Descriptive Statistics• Investigating the Hypothesis• Is There a Curve?• Normalizing for Dollar Size• Correction Factors and Their Use
– Correcting EACs and Risk Models
• Analysis Conclusions• Modeling Schedule Duration in Networks• How Networks Operate
– Some Toy Problems
[email protected], (703)633-8300 x4536, 04/21/23, 3
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Background
• At the MDA* Risk Working Group of 29/30 May 01, Schedule Risk was a major topic
• Action Item:– Investigate Schedule Risk
• Content variation • Cost risk* • PERT• Time and budget constraints
* The subject of this paper
This work was conducted for and funded by the IC CAIG and MDA
This work was conducted for and funded by the IC CAIG and MDA
[email protected], (703)633-8300 x4536, 04/21/23, 4
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
The Hypothesis
• Many people believe1 a graph of cost growth vs. schedule growth as illustrated below:
1.0
Schedule Growth Factor
Cost Growth Factor
1.0
1 E. g., Cost Risk Schedule – CEAC, Dr. M. Anvari, First BMDO Cost Symposium, 4 October 2001
[email protected], (703)633-8300 x4536, 04/21/23, 5
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
The Data
• We analyzed data from the RAND Cost Growth Database with both the following characteristics:– Programs with E&MD only
• Because growth is different for those with and without PDRR
– Programs with schedule data in the requisite fields
• There were 59 points. The analysis follows.
[email protected], (703)633-8300 x4536, 04/21/23, 6
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Descriptive Statistics for Schedule Growth
• We will look at these descriptive statistics in the following slides– Distribution shape– Scatter plots– Dollar weighting
[email protected], (703)633-8300 x4536, 04/21/23, 7
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Schedule Growth Factor Histogram
0
5
10
15
20
25
30
35
0.68 0.94 1.19 1.44 1.69 1.95 2.20 2.45 2.71 More
Bin (SGF)
Fre
qu
ency
Schedule Growth Distribution
CDF for Schedule Growth
PDF for Schedule Growth
These two graphs look much like CGF graphs, but the PDF is tighter here, and the CDF is
steeper.
These two graphs look much like CGF graphs, but the PDF is tighter here, and the CDF is
steeper.
The distribution is highly skewed
The distribution is highly skewed
Schedule Growth FactorCumulative Distribution Function
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4 2.6 2.8 3.0
SGF
Per
cen
tile
Note this region
[email protected], (703)633-8300 x4536, 04/21/23, 8
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Basic Statistics of Schedule ChangeAnalyzed data only
• Mean 1.29
• Standard Deviation 0.54
• CV 42%
• 75th %-ile 1.46
• 61st %-ile 1.29
• 50th %-ile 1.11
• 25th %-ile 1.00
• Shrinkers 9/59 15.3%
• Steady 12/59 20.3%
• Stretchers 38/59 64.4%
The distribution is highly skewed,as was seen in the histogram
The distribution is highly skewed,as was seen in the histogram
But, many programs have little-to-no growth
But, many programs have little-to-no growth
There is some dispersion and tendency to extremes
There is some dispersion and tendency to extremes
Observations
[email protected], (703)633-8300 x4536, 04/21/23, 9
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Basic Scatterplots – SGF & Sked vs. Dollar Size
We see the usual size effect, analogous to that in CGF graphsBigger programs have less schedule growth
We see the usual size effect, analogous to that in CGF graphsBigger programs have less schedule growth
SGF vs Size
0.00
1.00
2.00
3.00
4.00
5.00
0 2000 4000 6000 8000 10000
Size
SG
F
Sked Grow th vs Size
-20
0
20
40
60
80
0 2000 4000 6000 8000 10000
Size ($M)
Sche
dule
Gro
wth
(m
o)
[email protected], (703)633-8300 x4536, 04/21/23, 10
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
RAND 93 - Procurement
0.50
1.00
1.50
2.00
2.50
3.00
3.50
0 10000 20000 30000 40000
DE Proc baseline (FY96$)
Pro
c C
GF
The “1/x Pattern”
Contract Data - RDT&E
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
$0 $200 $400 $600
Original Estimate
Co
st
Gro
wth
Facto
r
RAND 93 - RDT&E
1
2
2
3
3
4
4
0 2000 4000 6000 8000 10000
DE Baseline (FY 96$)
DE
Co
st G
row
th F
acto
r
SW Growth
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
5.50
0 500000 1000000 1500000
Orginial Estimate
Gro
wth
Fa
cto
r
The 1/x pattern is virtually
universal.
[email protected], (703)633-8300 x4536, 04/21/23, 11
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
CGF and SGF vs. Cost SizeCGF and SGF vs Size
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
0 2000 4000 6000 8000 10000
Size ($K)
CG
F &
SG
F
CGF
SGF
1 Pt Removed for zoom
The pattern is similar, but CGF is generally more extreme:• Higher highs• Lower lows** See later plot
The pattern is similar, but CGF is generally more extreme:• Higher highs• Lower lows** See later plot
[email protected], (703)633-8300 x4536, 04/21/23, 12
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Basic Scatterplots – Dollar Size vs. Length
At Phase 2 start, there is a vague connection between length and sizeAt end, there is no connection
We would not say that longer programs are costlier
At Phase 2 start, there is a vague connection between length and sizeAt end, there is no connection
We would not say that longer programs are costlier
Program Size vs Schedule LengthInitial
y = 24.273x - 90.545R2 = 0.0659
0
2000
4000
6000
8000
10000
0 50 100 150
Length (mos)
Siz
e
Program Size vs Schedule LengthFinal
y = 2.5013x + 1532.7R2 = 0.001
0
2000
4000
6000
8000
10000
0 50 100 150 200
Length (mos)
Siz
e
[email protected], (703)633-8300 x4536, 04/21/23, 13
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Basic Scatterplots – Length vs. $ Size
At Phase 2 start, there is a vague connection between size and lengthAt end, there is no connection
We would not say that costlier programs are longer
At Phase 2 start, there is a vague connection between size and lengthAt end, there is no connection
We would not say that costlier programs are longer
Program Schedule Length vs SizeInitial
y = 0.0027x + 55.119R2 = 0.0659
0
20
40
60
80
100
120
140
0 2000 4000 6000 8000 10000
Size
Len
gth
(m
os)
Program Schedule Length vs SizeFinal
y = 0.0004x + 72.394R2 = 0.001
0
50
100
150
200
0 2000 4000 6000 8000 10000
Size
Len
gth
(m
os)
[email protected], (703)633-8300 x4536, 04/21/23, 14
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Basic Scatterplots – Cost Growth
There is no obvious connection between CGF and SGFThere is no obvious connection between CGF and SGF
Phase 2 SGF vs EMD only CGF
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
0.00 1.00 2.00 3.00 4.00
DE Only CGF
Ph
ase
II S
GF
Tw o Points omitted for Zoom-in
[email protected], (703)633-8300 x4536, 04/21/23, 15
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Basic Scatterplots - Length
There is a slight tendency for longer programs to grow less
There is a slight tendency for longer programs to grow less
SGF vs Length
y = 3.4944x-0.2609
R2 = 0.0854
0 .0 0
0 .50
1.0 0
1.50
2 .0 0
2 .50
3 .0 0
3 .50
4 .0 0
4 .50
5.00
30 50 70 90 110 13 0
Length
SG
F
[email protected], (703)633-8300 x4536, 04/21/23, 16
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Weighting by Length- and Dollar-Size
Schedule growth is less than cost growth
Weighting by Length- and Dollar-Size both reinforce size effects
Schedule growth is less than cost growth
Weighting by Length- and Dollar-Size both reinforce size effects
Raw vs Wtd GrowthAs GFs
0.000.250.500.751.001.251.501.75
RawAvg
L-w tdmean
$-WtdMn
L-WtShrinkFactor
$-WtShrinkFactor
CGF
SGF
Dollar Weighting shows a more severe effect
[email protected], (703)633-8300 x4536, 04/21/23, 17
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
CGF & SGF both Sorted
0.5
1
1.5
2
1 6
11 16
21
26
31
36
41
46
51
56
Develop CGF
Phase 2 SGF
CGF & SGF both Sorted
0
1
2
3
4
1 6 11 16 21 26 31 36 41 46 51 56
Develop CGF
Phase 2 SGF
Sorted Graphs
This graph is a zoom-in
Sorted CGF shows more growth than Sorted SGF(To the left and right of the x-intercept, Pink y-values are more extreme)
Sorted CGF shows more growth than Sorted SGF(To the left and right of the x-intercept, Pink y-values are more extreme)
[email protected], (703)633-8300 x4536, 04/21/23, 18
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Correlation and Other Joint Effects
[email protected], (703)633-8300 x4536, 04/21/23, 19
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Correlation and Other Joint Effects Between Schedule Growth and Cost Growth
• We will look for correlation– Parametric– Non-parametric– Trends in sorted data
• We will investigate the hypothesis for schedule growth vs. cost growth– We will normalize by dollar size to eliminate
any inadvertent distortion
CGF
SGF
[email protected], (703)633-8300 x4536, 04/21/23, 20
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Correlation - Parametric
There is no linear parametric correlationThere is no linear parametric correlation
Phase 2 SGF vs EMD only CGFAbridged
y = -0.0018x + 1.2284R2 = 5E-06
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
0.00 1.00 2.00 3.00 4.00
DE Only CGF
Ph
ase
II S
GF
Tw o Points omitted for regression
[email protected], (703)633-8300 x4536, 04/21/23, 21
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Correlation – Non-Parametric
• Test – Cox Stewart Test for Trend test statistic of 18 is
within the critical values of 8.41 and 18.59• The non-parametric test cannot reject no correlation• Used CGF Sort because CGF had less ties, thus less ambiguity
– Previous parametric test cannot reject no correlation
– Moving averages of CGF do not show a rise
• Conclusion: Cannot reject “no correlation”• Visual presentations follow
[email protected], (703)633-8300 x4536, 04/21/23, 22
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Patterns in SGF and CGF
There is no strong rising pattern in either CGF or SGF after sorting on the other
There is no strong rising pattern in either CGF or SGF after sorting on the other
CGF after SGF Sort
0.5
1
1.5
2
2.5
3
1 5 9 13
17
21
25
29
33
37
41
45
49
53
57
DevelopCGF
20 per.Mov. Avg.(DevelopCGF)
SGF after CGF Sort
0.5
1
1.5
2
2.5
3
1 5 9 13 17 21 25 29 33 37 41 45 49 53 57
Phase 2SGF
20 per.Mov. Avg.(Phase 2SGF)
The gentle rise here conforms with
the near-critical test statistic
[email protected], (703)633-8300 x4536, 04/21/23, 23
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Investigating the Hypothesis
CGF
SGF
[email protected], (703)633-8300 x4536, 04/21/23, 24
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
CGF by SGF Regimes
11.11.21.31.41.5
LT 1.0 EQ 1.0 LT 1.2 LT 1. 4 LT 1.6 LT 1.8
SGF Regime
CG
F A
ve
rag
e
0
5
10
15
20
Avg CGF
Count
CGF by Regime
Programs divided into SGF Regimes show a marked pattern, like the hypothesis suggested
Programs divided into SGF Regimes show a marked pattern, like the hypothesis suggested
Largest CGF
Larger CGFs, but Some small n’s
Smallest CGF
[email protected], (703)633-8300 x4536, 04/21/23, 25
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
CGF by SGF Regimes
11.11.21.31.41.5
LT 1.0 EQ 1.0 LT 1.2 LT 1. 4 LT 1.6 LT 1.8
SGF Regime
CG
F A
ve
rag
e
0
5
10
15
20
Avg CGF
Count
CGF by Regime
Programs divided into SGF regimes look somewhat like the hypothesis suggested they would
Programs divided into SGF regimes look somewhat like the hypothesis suggested they would
[email protected], (703)633-8300 x4536, 04/21/23, 26
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
There is a Patternbut
Is There a Curve?CGF
SGF
[email protected], (703)633-8300 x4536, 04/21/23, 27
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Is there a curve?
• There is no pattern on either side of the data
Stretch SGF vs CGF
0
2
4
6
8
10
12
14
16
18
20
0 1 2 3 4 5
SGF
CG
F
Shrink SGF vs CGF
0
0.5
1
1.5
2
2.5
3
0 0.2 0.4 0.6 0.8 1 1.2
SGF
CG
F
CGF
SGF
[email protected], (703)633-8300 x4536, 04/21/23, 28
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Is there a Curve?
CGF by SGF Regimes
11.11.21.31.41.5
LT 1.0 EQ 1.0 LT 1.2 LT 1. 4 LT 1.6 LT 1.8
SGF Regime
CG
F A
ve
rag
e
0
5
10
15
20
Avg CGF
Count
There is no reasonable grouping of the
stretchers that will produce a curve.
Any grouping of points has the same average.
CGF
SGF
[email protected], (703)633-8300 x4536, 04/21/23, 29
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Normalizing for Dollar SizeTo Remove Inadvertent Dollar Size Distortion
[email protected], (703)633-8300 x4536, 04/21/23, 30
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Size Normalization• We know there is a size effect in CGF
• We think there is a size effect in SGF
• We must investigate schedule effects free from size effects– First we will look at a scatter plot
– Then we will normalize1 all programs for dollar size, and compare to actuals
• If there is a pattern in any regime, we will worry
• If there is no regime pattern, we can conclude there is no dollar size distortion
• We chose to correct out dollar-size because it is stronger, and because we were worried about a length and SGF correlation causing mischief if we tried to correct it out
1 See backup for norming algorithm
[email protected], (703)633-8300 x4536, 04/21/23, 31
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
CGF by SGF Regime and Size
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
0 2000 4000 6000 8000 10000
Size (Dollars)
CG
F
Sked Grow
Sked Steady
Sked Shrink
One outlier left out for zoom in
Is there a Dollar-Size Bias?
Programs in the 3 regimes show no clear size bias, but a clear growth bias
Programs in the 3 regimes show no clear size bias, but a clear growth bias
“Steady” programs
are probably
attenuated vertically (growth
bias)
“Shrink” programs may be attenuated
horizontally (size bias)
“Growth” programs span the full range horizontally and
vertically
[email protected], (703)633-8300 x4536, 04/21/23, 32
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Normed vs Actual CGFs by Regime
Averages for size-normed programs show the same patterns, so there is no size distortion
Averages for size-normed programs show the same patterns, so there is no size distortion
Note: Corrected 20 Apr 02. Minor differences
CGFs vs SGF Regime
1.43
1.12
1.29
1.43
1.12
1.25
1.00
1.10
1.20
1.30
1.40
1.50
LT 1.0 EQ 1.0 GT 1.0
SGF Regime
CG
F Avg Act
Avg Norm'd
[email protected], (703)633-8300 x4536, 04/21/23, 33
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
CGFs vs SGF Regime
1.43
1.12
1.29
1.43
1.12
1.25
1.00
1.10
1.20
1.30
1.40
1.50
LT 1.0 EQ 1.0 GT 1.0
SGF Regime
CG
F Avg Act
Avg Norm'd
Normed vs Actual CGFs by Regime
Both sets of bars look like the hypothesis suggested they wouldBoth sets of bars look like the hypothesis suggested they would
[email protected], (703)633-8300 x4536, 04/21/23, 34
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Hypothesis – The Answer• The Hypothesis was about right
– The below is all we can say for sure– Some liberties have been taken with
the graph
1.0
Schedule Growth Factor
Cost Growth Factor
1.0
1.43
1.121.24
NB 1: Nominal has growth
CGF
SGF
NB 2: The curve is not validated, just the 3 regimes
[email protected], (703)633-8300 x4536, 04/21/23, 35
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Correction Factors and Their Use
[email protected], (703)633-8300 x4536, 04/21/23, 36
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
CGFs vs SGF Regime as Percent of Average
114%
90%
100%
80%
90%
100%
110%
120%
130%
140%
150%
LT 1.0 EQ 1.0 GT 1.0
SGF Regime
CG
F a
s %
of
Av
era
ge
S
GF
CGFs vs SGF Regime as Percent of SGF=1.0
127%
100%
111%
80%
90%
100%
110%
120%
130%
140%
150%
LT 1.0 EQ 1.0 GT 1.0
SGF Regime
CG
F a
s %
of
SG
F =
1.0
Correction Factors and Their Use
• We must correct for schedule growth, if we can predict it. The form of the correction is unclear:
These factors describe what happens if schedules change.
We might use these factors to adjust an EAC if a schedule changed.
We might use these factors to correct a risk model’s nominal growth
factors
[email protected], (703)633-8300 x4536, 04/21/23, 37
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Conclusions• Schedule growth is less extreme than cost growth
– But patterns are the same
• There is a cost-size and length effect, just as for cost growth– Dollar-larger programs lengthen less – Longer programs lengthen less
• Neither cost nor length predict the other
• There is a difference in cost growth by schedule-growth regimeRelative to
Relative to
Regime CGF No ChangeAverage
– Programs that shorten 1.42 1.25 1.14– Programs that stay the same 1.13 1.00 0.91– Programs that lengthen 1.24 1.09 1.00
• We now have tools to correct EACs and risk analysesThe hypothesis was essentially true
But there is no curve in evidence
The hypothesis was essentially trueBut there is no curve in evidence
[email protected], (703)633-8300 x4536, 04/21/23, 38
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Modeling Schedule Duration of Networks
[email protected], (703)633-8300 x4536, 04/21/23, 39
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Schedule Growth Distributions• For schedule network models, a distribution is useful to model
durations
• We will provide a distribution for program-level network schedule growth– Useable for confidence intervals and predictions for single programs
– Useable for systems of systems, to simulate component systems as single entities
• This section will provide a detailed analysis for fitting the schedule growth data to a distribution– Lognormal and Extreme Value distributions show the most promise
– Extreme Value is the most theoretically compelling• Extreme value distributions are used to model the largest of a set of random variables,
and networks complete when the last event is finished
[email protected], (703)633-8300 x4536, 04/21/23, 40
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Schedule Growth Factor CDFsPhase 2 DE only (n=59)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.50 1.00 1.50 2.00 2.50
Extreme Value
Lognormal
Empirical CDF
Best Fits vs. Empirical Data
• Extreme Value Distribution is what we expect theoretically• Extreme Value more peaked, appears to represent data better than Lognormal • But we will see the number of 1.0’s in the data base (schedules finishing “on
time”) creates problems in the fit statistics
Note disproportionate amount of 1.0’s
Schedule Growth Factor PDFs w/HistogramPhase 2 DE only (n=59)
0.0
0.5
1.0
1.5
2.0
0.00 1.00 2.00 3.00 4.00
Extreme Value
Lognormal
Histogram
Note disproportionate number of 1.0’s
[email protected], (703)633-8300 x4536, 04/21/23, 41
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Why are Values of 1 more Common?And who cares?
• There is intense pressure to complete on time, and late finishes are easily discerned
• The consequence of an early finish is to “ship” a flawed system– Flaws can be fixed after testing
• There is a temptation to drag out work if you are done early
• Perhaps the implication is that the customer should put less emphasis on finish time and more on test results?
• In any event, it is altogether likely that there would be cosmetic 1.0 SGFs, and the data would seem to reflect that
• We will find a way to deal with this in the analysis, and recommend a modeling approach
[email protected], (703)633-8300 x4536, 04/21/23, 42
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Extreme Value Distribution Fit• The CDF of the data is oddly shaped due to a large number of 1.0’s and
fails a Kolmogorov-Smirnov test for the Extreme Value Distribution
• We believe the disproportionate amount of 1.0’s is politically motivated and not a natural occurrence– This causes a “gap” between the empirical and fitted distributions
• We will next examine a hypothetical distribution with the 1.0’s redistributed along the “gap” area (using the Ext Val fit)
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
0.50 1.00 1.50 2.00 2.50
K-S stat = 0.161
95% Critical Value (n=59)
= 0.1131
1. Lilliefors methodology applied to Extreme Value distribution to generate critical value with Monte Carlo simulation
Empirical Schedule
Growth CDF vs Fitted
Extreme Value
Note “gap” caused by
1.0’s
“gap”
[email protected], (703)633-8300 x4536, 04/21/23, 43
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
The Hypothetical “Natural” CDF Original and Revised Empiricals
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.50 1.00 1.50 2.00 2.50
Extreme Value Fit
Revised Empirical
Empirical CDF
12 points respread
12 points at 1.0
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
0.50 1.00 1.50 2.00 2.50
K-S stat = 0.093
95% Critical Value (n=59)
= 0.113
1.0’s redistributed along the “gap” area (in red) better
represents what we believe to be the “natural” distribution
The revised empirical produces an Extreme Value fit with K-S stat below
the critical value. This suggests Extreme Value is a good representation of the
natural SGF distribution
Revised Empirical and Extreme Value Fit
Extreme Value: = 1.12 = 0.28
[email protected], (703)633-8300 x4536, 04/21/23, 44
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
What the test showsAnd what it doesn’t show
• The redistributed data pass a K-S test
• But, the test cannot take the redistribution of data into account– This is analogous to loss of degrees of freedom,
but the literature provides no remedy
• We fully realize that this is not a “valid statistical test”– But it strongly suggests that the underlying
distribution is the Extreme Value distribution
[email protected], (703)633-8300 x4536, 04/21/23, 45
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Hybrid Distribution Alternative• The hypothetical natural (re-distributed) distribution is reasonable for use
– But, if you wish to capture the effects of too many programs appearing to finish “on schedule” then a hybrid distribution should be examined
• To do this we must consider the probability of 1.0 vs. the rest of the outcomes as discrete cases– P(1.0) = 12/59 = 20.3%
– P(Extreme Value) = 79.7%
• The Extreme Value parameters would then be estimated from the data with the 1.0’s removed
0.50 1.00 1.50 2.00 2.50
20.3% (i.e. 12/59) probability of 1.0Hybrid Schedule
Growth PDF with Histogram
(original SGF data)
79.7% probability of Extreme Value Distribution
(fitted w/o 1.0’s)
[email protected], (703)633-8300 x4536, 04/21/23, 46
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Hybrid Distribution Alternative
Extreme Value fit to data without 1.0s:K-S stat is less than the critical value. The Extreme Value is a good representation of
this data.
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
0.50 1.00 1.50 2.00 2.50
K-S stat = 0.087
95% Critical Value (n=47)
= 0.1261
Extreme Value: = 1.16 = 0.32
1. Lilliefors methodology applied to Extreme Value distribution to generate critical value with Monte Carlo simulation
Simulated Hybrid Schedule Growth CDF
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.50 1.00 1.50 2.00 2.50
Hybrid 1000 trials Empirical CDF Plot Data
Hybrid Model
.000
.079
.157
.236
.314
0.25 0.81 1.38 1.94 2.50
Overlay Chart
Results of simulation combining this distribution with a discrete 20.3% probability of a 1.0
[email protected], (703)633-8300 x4536, 04/21/23, 47
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Distribution Conclusions
• We have shown that the Extreme Value distribution is well supported as the natural distribution
• We have shown that the pieces of the hybrid distribution fit the data– And, the hybrid reproduces the actuals well
• We recommend using the hybrid– But if “political” or “cosmetic” effects are absent,
we recommend using the hypothetical natural distribution
[email protected], (703)633-8300 x4536, 04/21/23, 48
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
How Networks OperateSome “Toy Problems”
[email protected], (703)633-8300 x4536, 04/21/23, 49
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Independent Tasks• Tasks 1 and 2 begin at the same
time and are independent
• Both tasks must be complete before the system is ready
• Duration is modeled as a uniform distribution ranging from Estimated – Note that it is symmetric!
• What is the Expected Duration?
EndStart
Task 1Duration 9
Task 2Duration 10
[email protected], (703)633-8300 x4536, 04/21/23, 50
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Task 1 Task 2 Max Dur7.02 11.91 11.917.08 11.62 11.628.22 11.27 11.27
10.00 10.91 10.919.94 8.77 9.949.03 10.94 10.949.54 8.39 9.54
10.05 10.09 10.0910.33 11.22 11.2210.59 11.64 11.64
Average 9.18 10.68 10.91Criticality 20% 80%
Independent Tasks
The average system duration is 10.91 months … longer than the estimated duration of either component task
The “shorter” Task 1 is the critical path 20% of
the time!
EndStart
Task 2Duration 10
Task 1Duration 9
Each task is uniformly distributed from –20% to +20%
of the expected duration
[email protected], (703)633-8300 x4536, 04/21/23, 51
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Comparisons with Constant Critical Path
S E
S E
ES
S E
S E
10
10
5
5
5
9
5
4
4
1
5
5
5
5
These all have Critical Path = 10
[email protected], (703)633-8300 x4536, 04/21/23, 52
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Network Comparisons
9.00 9.50 10.00 10.50 11.00 11.50
(4-5)//(5-5)crosslink
(4-5)//(5-5)
9//10
5-5
10
Ne
two
rk
Average DurationMean 80th%-ile
Comparisons with Constant CP
S E
S E
ES
S E
S E
10
10
5
5
5
9
5
4
4
1
5
5
5
5
Parallel is bad
Serial is good
Cross links are bad
Durations were modeled as uniform distributions ranging from 20% of the estimate. 5000 iterations were run.
Serial is good
These all have CP = 10 … but their probabilistic durations are all different
[email protected], (703)633-8300 x4536, 04/21/23, 53
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Network Growth Effect vs Number of Parallel Tasks
1.00
1.10
1.20
1.30
1.40
1.50
1.60
1.70
1.80
0 20 40 60 80 100
Number of Parallel Identical Tasks
Sch
edu
le G
row
th F
acto
r (N
etw
ork
)
10% CV
20% CV
29% CV
39% CV
Network Schedule Growth As a Function of Network Complexity … Parallel-Task Toy Problem
• This is another toy problem, to see what happens to a network as identical parallel tasks are added
Increasing the
number of tasks
increases the
schedule stretch
[email protected], (703)633-8300 x4536, 04/21/23, 54
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Network Growth Effect vs Coefficient of Variation (no mean shift)
0%
20%
40%
60%
80%
100%
0 0.2 0.4 0.6 0.8 1
CV
Sch
edu
le G
row
th F
acto
r (N
etw
ork
)
Mean Stretch (Triangular)Mean Stretch (Normal)80th %-ile Stretch (Triangular)80th %-ile stretch (Normal)
Network Schedule Growth As a Function of Task Variance … Changing-CV Toy Problem
• This is a real network, with changing variance, to see what happens as variance grows
Increasing the
variance of tasks
increases the
schedule stretch
[email protected], (703)633-8300 x4536, 04/21/23, 55
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
“Toy Problem” Conclusions• The duration of a network will be longer than any of the
component legs• Parallel tasks lengthen the average duration
– Independent tasks that must finish at the same time should make you worry about schedule
– The more parallel tasks, the more you stretch
• Serial tasks decrease the average duration– Serial tasks should make you feel a bit better about schedule– However, breaking a single task into smaller pieces will not improve your
schedule
• Interdependencies (cross links) increase the average duration– Tasks that depend on two or more other tasks should make you worry about
schedule
• Greater variability of the tasks will make the schedule duration grow
[email protected], (703)633-8300 x4536, 04/21/23, 56
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Backup
[email protected], (703)633-8300 x4536, 04/21/23, 57
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Size Adjustments
[email protected], (703)633-8300 x4536, 04/21/23, 58
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Prediction Equation - RAND RDT&E
RDT&E DE only
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
0 5000 10000 15000
Baseline
CG
F
Actual
Predicted
Note that data is sparse on the right (large programs)
SSE = 72.56
RDT&E Predicted CGF = 1.8 * (MSII Baseline FY96$M)-0.3 + 1.1RDT&E Predicted CGF = 1.8 * (MSII Baseline FY96$M)-0.3 + 1.1
[email protected], (703)633-8300 x4536, 04/21/23, 59
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Prediction Equation - RAND RDT&ERDT&E DE only
zoom-in
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
0 1000 2000 3000 4000 5000
Baseline
CG
FActual
Predicted
RDT&E Predicted CGF = 1.8 * (MSII Baseline FY96$M)-0.3 + 1.1RDT&E Predicted CGF = 1.8 * (MSII Baseline FY96$M)-0.3 + 1.1
[email protected], (703)633-8300 x4536, 04/21/23, 60
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Dispersion – Bounds
This graph shows the actual data, the CGF prediction line, and the Bounds. The next slide will zoom-in.
This graph shows the actual data, the CGF prediction line, and the Bounds. The next slide will zoom-in.
R&D DE only
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
0 2000 4000 6000 8000 10000 12000 14000
Baseline (FY96$M)
CG
FActual
Upper
Predicted
Lower
[email protected], (703)633-8300 x4536, 04/21/23, 61
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Dispersion – Bounds
R&D DE onlyzoom in
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
0 200 400 600 800 1000
Baseline
CG
F
Actual
Upper
Predicted
Lower
Note that the Upper and
Lower bounds are not
symmetric. Also, dispersion
is higher for smaller projects … an effect that is captured by the bounds.
[email protected], (703)633-8300 x4536, 04/21/23, 62
Briefing, 35th ADoDCAS, SCEA, PMI, 2002
TASC
Basic Statistics of Schedule ChangeAll available schedule data compared to analyzed data
Statistic Analyzed All Observations• Mean 1.29 1.25• Standard Deviation 0.54 0.51• CV 42% 41%• n 59 98
• 75th %-ile 1.46 1.365• %-ile of the mean 61% 63%• 50th %-ile 1.11 1.03• 25th %-ile 1.00 1.00
• Shrinkers 15.3% 20.4%• Steady 20.3% 22.4%• Stretchers 64.4% 57.1%
The larger data set is somewhat
less skewed
The larger data set is somewhat
less skewed
The larger data set has slightly less dispersion
The larger data set has slightly less dispersion
The two data sets are quite similar,
but, use the smaller
one as your basis
The two data sets are quite similar,
but, use the smaller
one as your basis