presentation on 'why cant people estimate' event, 23rd june 2016
TRANSCRIPT
Why Can’t People Estimate: Estimation Bias and Mitigation With SEER Introduction
David Simms
Galorath International
© 2016 Copyright Galorath Incorporated
Key Points for today . . .
Estimates can be better, reducing bias & strategic mis-estimation… Parametrics can
help.
© 2016 Copyright Galorath Incorporated 2
Tempering with an
“outside view” can mitigate some bias
Without care estimates are usually biased (even with experts)
Some of the most important
business decisions about a
project are made at the time of
minimum knowledge and
maximum uncertainty.
ESTIMATION & PLANNING: An Complete Estimate Defined
• An estimate is the most knowledgeable statement you can make at a particular point in time regarding:
• Effort / Cost
• Schedule
• Staffing
• Risk
• Reliability
• Estimates more precise with progress
• A WELL FORMED ESTIMATE IS A
DISTRIBUTION
4
Human Nature: Humans Are Optimists
Harvard Business Review explains this Nobel Prize Winning Phenomenon:
• Humans seem hardwired to be optimists
• Routinely exaggerate benefits and discount costs
Delusions of Success: How Optimism Undermines Executives' Decisions (Source: HBR Articles | Dan Lovallo, Daniel Kahneman | Jul 01, 2003)
5
Solution - Temper with “outside view”:
Past Measurement Results, traditional forecasting, risk
analysis and statistical parametrics can help
Don’t remove optimism, but balance optimism and
realism
Estimation Methods 1 of 2
Model
Category Description Advantages Limitations
Guessing Off the cuff estimates
Quick
Can obtain any answer
desired
No Basis or substantiation
No Process
Usually Wrong
Analogy Compare project with past
similar projects.
Estimates are based on
actual experience. Truly similar projects must exist
Expert
Judgment
Consult with one or more
experts.
Little or no historical data
is needed; good for new or
unique projects.
Experts tend to be biased;
knowledge level is sometimes
questionable; may not be
consistent.
Top Down
Estimation
A hierarchical decomposition
of the system into
progressively smaller
components is used to
estimate the size of a
software component.
Provides an estimate
linked to requirements and
allows common libraries to
size lower level
components.
Need valid requirements.
Difficult to track architecture;
engineering bias may lead to
underestimation.
Estimation Methods 2 of 2
Model Category Description Advantages Limitations
Bottoms Up
Estimation
Divide the problem into
the lowest items.
Estimate each item…
sum the parts.
Complete WBS
can be verified.
The whole is generally bigger than the
sum of the parts.
Costs occur in items that are not
considered in the WBS.
Design To Cost
Uses expert judgment to
determine how much
functionality can be
provided for given
budget.
Easy to get under
stakeholder
number.
Little or no engineering basis.
Simple CER’s
Equation with one or
more unknowns that
provides cost / schedule
estimate.
Some basis in
data.
Simple relationships may not tell the
whole story.
Historical data may not tell the whole
story.
Comprehensive
Parametric Models
Perform overall estimate
using design
parameters and
mathematical
algorithms.
Models are usually
fast and easy to
use, and useful
early in a program;
they are also
objective and
repeatable.
Models can be inaccurate if not
properly calibrated and validated;
historical data may not be relevant to
new programs; optimism in parameters
may lead to underestimation.
8
Improving Estimation Is Important
• Estimation is critical for defining success for all types of projects
• Many organizations treat estimation as a second rate activity and do not invest in the process
• Everyone estimates….but the process and the quality or success rate varies greatly
• Estimation and measurement go hand in hand – BOTH contribute to improvement
• Having a repeatable estimation process is critical to both estimating AND to successful projects
Cognitive Bias: How Fair Are We (Source BeingHuman.org)
•Human beings exhibit inherent errors in thinking
•Perception has more to do with our desires—with how we want to view ourselves—than with reality." Behavioral economist Dan Ariely
Cognitive bias: Tendency to make
systematic decisions based on PERCEPTIONS
rather than evidence
•Our brains using shortcuts (heuristics) that sometimes provide irrational conclusions Researchers theorize in
the past, biases helped survival
•from deciding how to handle our money
•to relating to other people
•to how we form memories “Bias affects everything:
© 2016 Copyright Galorath Incorporated 10
The Planning Fallacy (Kahneman & Tversky, 1979)
•Manifesting bias rather than confusion
•Judgment errors made by experts and laypeople alike
•Errors continue when estimators aware of their nature
Judgment errors are systematic & predictable,
not random
•Underestimate costs, schedule, risks
•Overestimate benefits of the same actions Optimistic due to overconfidence
ignoring uncertainty
•“inside view” focusing on components rather than outcomes of similar completed actions
•FACT: Typically past more similar assumed
•even ventures may appear entirely different
Root cause: Each new venture
viewed as unique
© 2016 Copyright Galorath Incorporated 11
Reference Class Forecasting
Attempt to force the outside view and
eliminate optimism and
misrepresentation
Choose relevant “reference class”
completed analogous projects
Compute probability distribution
Compare range of new projects to
completed projects
Provide an “outside view” focus on
outcomes of analogous projects
© 2014 Copyright Galorath Incorporated 12
Best predictor of performance is actual performance of implemented comparable projects (Nobel Prize
Economics 2002
Predicts outcome of planned action based on actual outcomes in a reference class: similar actions to that being forecast.
Dealing With the “Problem of Assumptions” • Assumptions are essential Write them down but…
Wrong assumptions can drive an estimate to uselessness
• Use an assumption verification process
© 2016 Copyright Galorath Incorporated 13
1. Identify assumptions
2. Rank order assumptions based on
estimate impact
3. Identify high ranking assumptions
that are risky
4. Clarify high ranking, high risk assumptions
& quantify what happens if those
assumptions change
5. Adjust range of SEER inputs to describe the
uncertainty in assumption
We should always listen to the experts . . . . Shouldn’t we?
© 2016 Copyright Galorath Incorporated 14
Trouble Starts By Bias or Strategic Mis- Estimation Ignoring Iron Triangle
• Typical Trouble: Mandated features needed within specific time by given resources
• At least one must vary otherwise quality suffers and system may enter impossible zone!
Quality Resources Schedule
Scope (features, functionality)
Sometimes strategic mis-estimation is used to get projects started or to win
Some customers think price to win is strategic mis-estimation (it is not)
Plans Based on Assumptions About Average Conditions Usually Go Wrong
• Executive desire to “Show me the number” forces averages
• When average is used to represent an uncertain quantity….
• it ends up distorting the results
• Because it ignores the impact of the inevitable variations
• Flaw may be fixed by tools such as Monte Carlo Analysis
• Adds a distribution & probability to “the number”
16
Adding Reality to Estimates – Example – 2 (Source SEI) Step Best Expected Worst
1 27 30 75
2 45 50 125
3 72 80 200
4 45 50 125
5 81 90 225
6 23 25 63
7 32 35 88
8 41 45 113
9 63 70 175
10 23 25 63
500
What would you forecast the schedule duration to be
now?
Monte Carlo Analysis To Quantify and Mitigate Bias
90% confidence, project under 817
days
Almost guaranteed to miss the 500
days
50% confidence,
project will be under 731 days
Example: Project Cost Alone Is not The Cost of IT Failure (Source: HBR)
• Case Study: Levi Strauss
• $5M ERP deployment contracted
• Risks seemed small
• Difficulty interfacing with customer’s systems
• Had to shut down production
• Unable to fill orders for 3 weeks
• $192.5M charge against earnings on a $5M IT project failure
“IT projects touch so many aspects of organization they pose a new singular risk”
http://hbr.org/2016/09/why-your-it-project-may-be-riskier-than-you-think/ar/1
Causes of Project Failure Source: POST Report on UK Government IT Projects
Lack of a clear link between the project and the organisation’s key
strategic priorities, including agreed measures of success.
1.
Lack of clear senior management and ministerial ownership and leadership 2.
Lack of effective engagement with Stakeholders 3.
Lack of skills and proven approach to project management and risk
management
4.
Lack of understanding of and contact with the supply industry at senior levels
within the organisation
5.
Evaluation of proposals driven by initial price rather than long-term
value for money (especially securing the delivery of business benefits)
6.
Too little attention to breaking development and implementation into
manageable steps
7.
Inadequate resources and skill to deliver the total delivery portfolio 8.
Source: POST Report on UK
Government IT Projects
Fallacy of Silent Evidence What about what we don’t know?
How confident would you feel if the Silent Evidence was visible?
What is Parametric Estimating?
• Parametric Approach:
• Employs cost estimating relationships
• Uses mathematical equations
• Allows estimation based on measurement of key parameters that influence the relationship
• Based on technical or physical characteristics of the product and production environment
• SEER’s Parameters Cover 5 Key Areas
• People
• Process
• Technology
• Platform (Environment)
• Size
22
Draw Out Range By Obtaining 3 Estimates
• Optimistic value (sopt)
• Most likely value (sm)
• Pessimistic value (spess)
• Expected value (EV)
© 2016 Copyright Galorath Incorporated 23
6
)4( pessmopt sssEV
Sophisticated Schedule Modeling: Essential For Viable Project Plans
For a given Size, Complexity and Technology Work Expands To Fill Time
(Work expands due to lack of pressure)
Minimum Time To Complete
(Effort reduces to reduce schedule)
Optimal Effort for staff
(Lower Effort for Longer Schedule)
Effort Increase
due to Longer Schedule
Calendar Time
Effort
Month
s
Reasonable Solution Range
Software Solution
Range vs. Point Estimates (Source US Army)
Range o
f Ris
k &
Uncert
ain
ty
Estimating Accuracy Trumpet
Technical and Program Maturity
RO
M -
30%
to
+7
5%
Para
me
tric
-
10%
to
+2
0%
An
alo
gy
-15%
to
+3
0%
En
gin
ee
rin
g
-5%
to
+15%
Actu
al
-3%
to
+1
0%
Target Cost
Point estimate is most likely within range estimate
with higher potential for cost increase
Range estimate provides a degree of risk and uncertainty
+75%
-30%
B A Materiel Solution Analysis
Systems Acquisition Sustainment
Technology
Development
Production &
Deployment
Operations &
Support
Engineering and
Manufacturing Development
C
Pre-Systems Acquisition
What can a parametric model tell you?
26
What is likely to happen
Feel lucky?
Firm Fixed Price?
Understand the risk before you commit!
Difference Between Bottoms Up and Parametric
• “Bottom-up” or Manual Approach (Detailed Work Measurement)
• Measures everything directly and exactly
• Little or no analysis of alternatives
• Predictive value is limited at best
• Parametrics
• Measures or estimates key parameter values and ranges
• Understand (mathematically) their effect on time, effort
• Once relationships are known, parameter values can be changed and the effects evaluated
© 2016 Copyright Galorath Incorporated 27
SEER MAJOR Components
• SEER-SEM – software/application development, maintenance, integration and testing
• SEER-H – system, hardware and electronics development, production and support
• SEER-IT – IT infrastructure, services and operations and systems engineering
• SEER-MFG – hardware manufacturing and assembly
• SEER-SYS – Systems Engineering
• SEER-Cost IQ Case based reasoning estimation
© 2016 Copyright Galorath Incorporated 29
Multiple Estimation Methods Used
• Knowledge based parametrics
• Built-in estimating relationships, ready to use
• Can be tuned and calibrated to improve accuracy
• Expert judgment – utilize SME inputs for a task
• Analogy – estimate based on past examples
• Top down – estimate total cost and allocate
• Bottom up – estimate each detailed task or process step
© 2016 Copyright Galorath Incorporated 30
SEER Enterprise Costing Solution: Parts thru Total Ownership Cost
SEER-MFG
SEER-SEM SEER-IT
Detailed SEER-H Estimate:
• Development • Production
• Operation & Support
• System Level Costing
Detailed Estimate
Parametric Estimate
Software/IT Estimate Actual Procurement
Cost Data
SEER-H
© 2016 Copyright Galorath Incorporated 31
SEER – Risk Driven Estimates The Engine For Project Evaluation
Least, likely, and most inputs provide a range of
cost and schedule outcomes
Confidence (probability) can be set and displayed for any estimated item
• SEER predicts outcomes
• SEER uses inputs to develop probability distributions
• The result is a probabilistic estimate
• SEER will predict a likely range of outcomes
• Monte Carlo provides project-level assessments of risk
© 2016 Copyright Galorath Incorporated 32
Bia
s M
itig
ation:
Ris
k
Knowledge Bases
• SEER-H uses knowledge bases to derive and preset initial parameter settings based on industry experience
• Choose knowledge bases for:
• Application (Aircraft Stabilizer, Analog Display, Electro-Mechanical Control, etc.)
• Platform (Air-Manned, Air-Unmanned, Groung Mobile, etc.)
• Acquisition Category (Build to Print, Buy & Integrate, Make, Modification, etc.)
• Standard (Commercial, Military, Space, etc.)
• Operations & Support (Remove & Replace, Repair, Scheduled Maintenance, etc.)
• Users may add their own knowledge bases if desired
Bia
s M
itig
ation:
Com
ple
ted A
ctu
als
© 2016 Copyright Galorath Incorporated 34
SEER for Software Example: Top 10 Impacts & Sensitivity Analysis
Estimates Are Validated Against Data (Yours and/or Other)
SEER-SEM
Estimate
Your Data
Regression
Trendline
Galorath
Benchmark
Trendline
Your History
Data
36
Project Monitoring & Control “When Performance is Measured Performance Improves”
• Adds Performance Measurement (Earned Value) methods to SEER-SEM
• Accepts progress & expenditure inputs
• Provides cost, schedule, and time variances
• Provides cost, schedule, and time indices
• Performance-based cost & schedule Estimate at Completion
• Displays health and status indicators
© 2016 Copyright Galorath Incorporated
36
SEER-SEM Total Cost of Ownership
Why Should We Care: Impacts ROI & development decisions can impact maintenance costs.
37
Staff Vs Maintenance Rigor
0
500
1000
1500
2000
2500
3000
3500
1 7 13 19 25 31 37 43 49 55 61 67 73 79 85
Time
sta
ff h
ou
rs p
er
mo
nth
develop
Rigor vhi+
Rigor nom
Rigor vlo
Bia
s M
itig
ation:
Forg
ott
en C
osts
Searchable Cost Catalogs
• Catalogs used for actual cost metrics for common tasks and parts
• Configurable to the customers requirements
• Refresh estimates based on the latest metrics
© 2016 Copyright Galorath Incorporated 38
Bia
s M
itig
ation:
Analo
gie
s
Ply Cost Estimator – CATIA V5
© 2016 Copyright Galorath Incorporated 39
Bia
s M
itig
ation:
via
ble
cost
trades
Cost IQ - Obtaining Cost Directly From Limited Requirements
A Case Based Reasoning system that can transform high level requirements and specifications into a cost modeling workup within a sophisticated cost estimating model.
This approach can provide even concept planners with an advanced understanding of the cost of
potential designs.
© 2016 Copyright Galorath Incorporated 40
Bia
s M
itig
ation:
Refe
rence c
lasses
• Scenarios can be generated proactively or incidentally
• Deliberately defining a standard estimation WBS pattern
• Turning an exemplar estimate into a WBS pattern
Scenarios
• Pre-configured WBS patterns may be stored as a “scenario”
• Scenarios may be configured with options for the user to select which elements to include
• The WBS structure will be loaded with the user’s tailoring
© 2016 Copyright Galorath Incorporated 41
Bia
s M
itig
ation:
Com
ple
ted A
ctu
als
SEER-DB
• Store all estimates in a secure repository
• Secure access to individuals or user groups
• Supports estimate revisions and locking
© 2016 Copyright Galorath Incorporated 43
Bia
s M
itig
ation:
Actu
al H
isto
ry
SEER – In Summary
• SEER Parametric cost estimation software provides estimates of Effort, Cost, Schedule For Information Technology projects & Program’s
• Gives a consistent framework for estimating and planning projects
• Helps you understand why projects cost what they do
© 2016 Copyright Galorath Incorporated 44
Key Points for today . . .
Estimates can be better, reducing bias & strategic mis-estimation… Parametrics can
help.
© 2016 Copyright Galorath Incorporated 45
Tempering with an
“outside view” can mitigate some bias
Without care estimates are usually biased (even with experts)
At Galorath – We Do Estimation
Galorath solutions: Tools and consulting
Galorath works with industry leaders to help adopt and improve organizational estimation best practices
• Over 30 years in business
• Hundreds of customers in government & industry; many Fortune 500
• Headquartered in El Segundo, CA; International office in Andover, U.K. with staff in the U.S. and
Europe
• Professional services organization provides consulting and training