monte carlo simulation for software cost estimation - …csse.usc.edu/events/2002/cocomo17/pete...
TRANSCRIPT
![Page 1: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/1.jpg)
Monte Carlo Simulation forSoftware Cost Estimation
Pete MacDonaldFatma Mili, PhD.
![Page 2: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/2.jpg)
2
Definition
• Software Maintenance - The activities involved in implementing a set of relatively small changes to an existing program. Barry Boehm, Software Engineering Economics, Prentice-Hall, 1981
![Page 3: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/3.jpg)
3
Quotes• “Estimating maintenance is rather a tricky
problem.” Caper Jones, Estimating Software Costs, McGraw-Hill 1998
• “Inaccurate size estimates automatically translate into questionable budgets.” Eduardo Miranda, Improving Subjective Estimates Using Paired Comparisons, IEEE Software, Jan./Feb. 2001
![Page 4: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/4.jpg)
4
In the Beginning• Before you can have Software Maintenance,
there exists new development• For new or substantially new development
estimation is accomplished via COCOMO, etc.
• After several projects, REVIC estimates were within 10% of actuals if SLOC estimate was accurate
![Page 5: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/5.jpg)
5
Typical New Development
• Starts with a set of user requirements• Analyze these requirements to develop
software requirements• Produce a design• Implement the design by coding and testing
![Page 6: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/6.jpg)
6
Software Maintenance
• Frequently performed by “newer” employees• Understand the problem
– There is a known problem with the code– Typically requires instrumenting the code
• After the problem is identified– The code is modified and tested.– Changes made to the code are then reflected in
the analysis and design documents
![Page 7: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/7.jpg)
7
Maintenance Life-Cycle Model
Understandthe Problem
Implementthe Change
Verify theChange
Document theChange
![Page 8: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/8.jpg)
8
Discussion• Understand the problem - typically requires instrumenting
the software to discover the problem location and/or potential solutions
• Implement the change - coding a potential correction into the software
• Verify the Change - test the change to verify the problem has been corrected
• Document the Change - Update the Software Requirements and/or Design Documents to reflect change in the software
![Page 9: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/9.jpg)
9
SW estimating at GD
• Evolved via software process improvement activities
• Initially performed using REVIC– Once calibrated, estimates were within 10% if
SLOC estimate was correct• In ‘95 started software maintenance of
existing software• Attempted to apply REVIC
![Page 10: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/10.jpg)
10
Initial Approach
• Difficult to estimate % of change – Design, Code & Test
• Looked at effort required– Used an average – Eliminated high and low outliers
• Went to engineering judgement– More accurate but difficult to quantify
![Page 11: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/11.jpg)
11
Initial Approach continued
• The way REVIC was applied led to inaccurate estimates
• Reverted to Engineering Estimates• Example - Modification requires 25 SLOC
– Is percentage of the Program or File?– Are Integration changes only the amount of
tests changed or include the effort to execute the required test cases?
![Page 12: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/12.jpg)
12
Second Approach
• Effort data for changes were available• Discarded the outliers• Take mean of remaining data and use as a
“typical” Change• Tended to cause an under estimation.• Again Reverted to Engineering Estimate
![Page 13: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/13.jpg)
13
Effort DistributionHistogram
020406080100120140160180200
36 15827
839
851
863
875
887
899
811
1812
3813
5814
7815
9817
1818
38
Frequency
![Page 14: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/14.jpg)
14
Engineering Judgment
• Accomplished by senior developers• May incorporate “Delphi technique”• Difficult to quantify and/or justify• Lacks the structure of a parametric model• Historical data only in the memory of the
engineer
![Page 15: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/15.jpg)
15
Monte Carlo Simulation• Necessitates the use of multiple random
distribution functions– Single distribution would like using the mean
• Distributions are slightly overlapping subsets of historical effort data
• Run multiple simulations to produce an estimate– Typically requires a minimum of 100 runs– Take the average of the runs.
![Page 16: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/16.jpg)
16
Current Approach• Apply Monte Carlo Simulation Model• Senior Developers classify changes as
– Simple, Typical or Difficult• Each classification has its own distribution
function• Implementing a change into a baseline can
induce the requirement to include additional changes
![Page 17: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/17.jpg)
17
Model Overview
Distributionby
Classification
SimpleChanges
TypicalChanges
DifficultChanges
DefectsInduced
• Total effort = Sum of the hours estimated to all changes plus the effort to correct induced defects
![Page 18: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/18.jpg)
18
Defining the Distribution Functions
• Historical effort data is collected and maintained in ascending order
• Must have at least 40 data points to generate a distribution function
• Must validate each distribution– Mean square error < 0.005– Chi-Square and Kolmogorov- Smirnov tests > 0.05
![Page 19: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/19.jpg)
19
An Example
• Given 30 changes to be implemented as follows – 10 easy– 15 typical– 5 difficult
• Produce an estimate to implement the changes.
![Page 20: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/20.jpg)
20
Example Continued
Disc(1, 0.33,2, 0.83,3, 1.0)
75 + Logn (130, 70)
Σ = 8067hours
Tria (39,55,7, 170)
Disc(1, 0.80,2, 1.0)
505 + Expo (293)
![Page 21: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/21.jpg)
21
Results utilizing ArenaProject: SW Estimator Run execution date : 9/24/2002Analyst: Pete MacDonald Model revision date: 9/24/2002
OUTPUTS
Identifier Average Half-width Min Max ReplicationsDuration 1976.9 26.61 1651.1 2405.2 100Effort 8065.9 346.18 4526.6 14681. 100
Simulation run time: 0.08 minutes.
![Page 22: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/22.jpg)
22
Comparison of MethodsNo. of
Changes 36 30 16
Actual 14710 9429 8468 Typical Fix 9000 7500 4000
% Error 38% 20% 53% 3rd Try 11465 7115 6936 % Error 22% 14% 18% Current 14369 9848 8023 % Error 2.5% 4.5% 5%
![Page 23: Monte Carlo Simulation for Software Cost Estimation - …csse.usc.edu/events/2002/cocomo17/Pete Macdonald-Fatma Mili... · Monte Carlo Simulation for Software Cost Estimation](https://reader034.vdocuments.mx/reader034/viewer/2022051722/5aa556f87f8b9a517d8d1b97/html5/thumbnails/23.jpg)
23
Where do we go from here• Automate generation of random distribution
functions.• Evaluate induced error data.• Evaluate the application of Monte Carlo
simulation as a Systems Engineering estimation tool.
• Include Engineer effectiveness factor.