modeling, simulation & data mining: answering tough cost ... · a model is a tool used to mimic a...
TRANSCRIPT
-
Modeling, Simulation & Data Mining: Answering Tough Cost, Date
& Staff Forecasts Questions
Troy Magennis (Focused Objective)
Larry Maccherone (Rally)
-
Pain Point
My Boss “Needs” A Date…
-
Getting Quantitative
Evidence
-
Assessing & Communicating
Risk / Uncertainty
-
Arm my teams (and yours) with the tools and techniques
to solve these problems
My Mission
-
2 Minutes About Larry
• Larry is a Pisces who enjoys skiing, reading and wine (red, or white in outdoor setting)
• We have a lot in common… over to Larry!
-
Metrics & Measurement
-
Why measure?
Feedback
Diagnostics
Forecasting
Lever
-
When to NOT take a shot
Good players?
• Monta Ellis
– 9th highest scorer (8th last season)
• Carmelo Anthony (Melo)
– 8th highest scorer (3rd last season)
-
The ODIM Framework
better Measurement
better Insight
better Decisions
better Outcomes
-
What is normal?
Cumulative -> 0.1% 2.3% 15.9% 50.0% 84.1% 97.7% 99.9%
Roughly -> 85% 98%
-
Are you normal?
-
You will be wrong by…
• 3x-10x when assuming Normal distribution
• 2.5x-5x when assuming Poisson distribution
• 7x-20x if you use Shewhart’s method
Heavy tail phenomena are not incomprehensible… but they cannot be
understood with traditional statistical tools. Using the wrong tools is incomprehensible.
~ Roger Cooke and Daan Nieboer
-
Bad application of control chart Control is an illusion, you infantile
egomaniac. Nobody knows what's gonna happen next: not on a freeway, not in an airplane, not inside our own bodies and
certainly not on a racetrack with 40 other infantile egomaniacs.
~Days of Thunder
-
Time in Process (TIP) Chart A good alternative to control chart
-
Collection
• Perceived cost is high
• Little need for explicit collection activities
• Use a 1-question NPS survey for customer and employee satisfaction
• Plenty to learn in passive data from ALM and other tools
• How you use the tools will drive your use of metrics from them
-
Summary of how to make good metric choices
• Start with outcomes and use ODIM to make metrics choices.
• Make sure your metrics are balanced so you don’t over-emphasize one at the cost of others.
• Be careful in your analysis. The TIP chart is a good alternative to control chart. Troy’s approach is excellent for forecasting. We’ve shown that there are many out there that are not so good.
• Consider collection costs. Get maximal value out of passively gathered data.
Data visualization is like photography. Impact is a function of perspective, illumination, and focus.
~Larry Maccherone
-
Flaw of Averages, Risk & Monte Carlo Sim
-
A model is a tool used to mimic a
real world process
A tool for low-cost experimentation
-
Monte Carlo Simulation
Monte Carlo Simulation?
Performing a simulation of a model multiple times using
random input conditions and recording the frequency of
each result occurrence
-
Scrum
Backlog This Iteration Deployed
2 5
8
Run Sim Total Iterations
1 3
2 2
3 5
4 3
5 4
6 2
… …
-
Kanban
Backlog Design Develop Test Deployed
2
1 – 2 days 1 – 5 days 1 – 2 days
Run Time Total
1 5
2 4
3 3
4 9
5 5
6 6
… …
-
Result versus Frequency (50 runs)
More Often
Less Often Result Values – For example, Days
15 10 20
Fre
qu
en
cy o
f R
esu
lt
1
5
10
15
20
25
-
Result versus Frequency (250 runs)
More Often
Less Often Result Values – For example, Days
15 10 20
Fre
qu
en
cy o
f R
esu
lt
1
5
10
15
20
25
-
Result versus Frequency (1000+ runs)
More Often
Less Often Result Values – For example, Days
15 10 20
Fre
qu
en
cy o
f R
esu
lt
1
5
10
15
20
25
-
Key Point
There is NO single forecast result
There will always be many possible results, some more likely
-
Time to Complete Backlog
50% Possible Outcomes
50% Possible Outcomes
When pressed for a single number, we often give the average.
Like
liho
od
-
Time to Complete Backlog
95% Outcomes 5%
Monte Carlo Simulation Yields More Information – 95% Common.
Like
liho
od
-
Key Point
“Average” is NEVER an option WARNING: Regression lines
are most often “average”
-
But, I.T. gets worse
-
1 2 3
Planned Backlog
Perf. Issues
Vendor Delay
Time to Delivery
Like
liho
od
Promised New Average
50% Possible Outcomes
-
Key Point
Risks play a BIG role in forecasts
Yes, more than backlog.
-
Velocity is NOT Linear nor is defect rate, scope-creep, story
expertise requirements, team skill, etc.
-
Date for likelihood
Likelihood (0-100%)
-
Key Point Forecasts should be presented with the
right amount of uncertainty
-
PAIN POINT Demo: Forecasting… My Boss “Needs” a Date…
-
In this demo
• Basic Scrum and Kanban Modeling
• How to build a simple model
– SimML Modeling Language
– Visual checking of models
– Forecasting Date and Cost
– The “Law of Large Numbers”
-
Demo: Finding What Matters Most Cost of Defects & Staff Analysis
-
Actively Manage
Ignore for the moment
Sensitivity Report
-
Staff Skill Impact Report
Explore what staff changes have the greatest impact
-
Key Point
Modeling helps find what matters
Fewer estimates required
-
In this demo
• Finding what matters most
– Manual experiments
– Sensitivity Testing
• Finding the next best 3 staff skill hires
• Minimizing and simplifying estimation
– Grouping backlog
– Range Estimates
– Deleting un-important model elements
-
Demo: Finding the Cost / Benefit of Outsourcing
-
Outsourcing Cost & Benefits
• Outsourcing often controversial
– Often fails when pursued for cost savings alone
– Doesn’t always reduce local employment
– An important tool to remain competitive
– I.Q. has no geographic boundaries
• Many models
– Entire project
– Augmentation of local team
-
Build Date & Cost Matrix
1 x Estimates
1.5 x Estimates
2 x Estimates
1 x Staff Best Case
1.5 x Staff Midpoint
2 x Staff Worst Case
Benefit = (Baseline Dev Cost – New Dev Cost) - Cost of Delay + Local Staff Cost Savings
-
$(150,000)
$(100,000)
$(50,000)
$-
$50,000
$100,000
$150,000
1 1.5 2
1x Multiplier
1.5x Multiplier
2x Multiplier
NOT LINEAR & NOT YOUR PROJECT
-
In this demo
• Model the impact of various outsourcing models
-
New Project Rules of Thumb…
• Cost of Delay plays a significant role
– High cost of delay project poor candidates
– Increase staffing some compensation
• Knowledge transfer and ramp-up time critical
– Complex products poor candidates
– Captive teams better choices for these projects
• NEVER as simple as direct lower costs!
-
Assessing and Communicating Risk
-
Speaking Risk To Executives
• Buy them a copy of “Flaw of Averages” • Show them you are tracking & managing risk • Do
– “We are 95% certain of hitting date x” – “With 1 week of analysis, that may drop to date y” – “We identified risk x, y & z that we will track weekly”
• Don’t – Give them a date without likelihood
• “February 29th 2013”
– Give them a date without risk factors considered • “To do the backlog of features, February 29th, 2013”
-
We spend all our time estimating here
1 2 3
**Major risk events have the predominate role in deciding where deliver actually occurs **
Plan Performance Issues
External Vendor Delay
-
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
-
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
-
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
-
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
-
Key Points
• There is no single release date forecast
• Never use Average as a quoted forecast
• Risk factors play a major role (not just backlog)
• Data has shape: beware of Non-Normal data
• Measurement → Insight → Decisions → Outcomes : Work Backwards!
• Communicate Risk early with executive peers
-
Call to action
• Read these books
• Download the software FocusedObjective.com
• Follow @AgileSimulation
• Follow @LMaccherone
http://www.focusedobjective.com/
-
Please Submit an Eval Form!
We want to learn too!
-
BEST PRACTICES
-
Model (a little)
Visually Test
Monte-Carlo Test
Sensitivity Test
The Model Creation
Cycle
-
Baseline
Make Single
Change
Compare Results
Make Informed
Decision(s)
The Experiment
Cycle
-
Best Practice 1
Start simple and add ONE input condition at a time.
Visually / Monte-carlo test
each input to verify it works
-
Best Practice 2
Find the likelihood of major events and estimate delay E.g. vendor dependencies,
performance/memory issues, third party component
failures.
-
Best Practice 3
Only obtain and add detailed estimates and opinion to a
model if Sensitivity Analysis says that input is material
-
Best Practice 4
Use a uniform random input distribution UNTIL sensitivity
analysis says that input is influencing the output
-
Best Practice 5
Educate your managers’ about risk. They will still want a “single” date for planning, but let them decide 75th or
95th confidence level (average is NEVER an option)
-
SIMULATION EXAMPLES
Return to main presentation…
-
unlikely
certain
Forecasts Return to main presentation…
-
unlikely
certain
Forecasts
50% Possible
Outcomes
50% Possible Outcomes
Return to main presentation…
-
Actively Manage
Ignore for the moment
Sensitivity Report Return to main presentation…
-
Staff Skill Impact Report
Explore what staff changes have the greatest impact
Return to main presentation…
-
Return to main presentation…
-
Focused Objective
• Risk Tools for Software Dev
• Scrum/Agile Simulation
• Kanban/Lean Simulation
• Forecasting Staff, Date & Cost
• Automated Sensitivity Analysis
• Data Reverse Engineering
• Consulting / Training
• Book
-
We Use & Recommend: EasyFit
• MathWave.com
• Invaluable for
– Analyzing data
– Fitting Distributions
– Generating Random Numbers
– Determining Percentiles
-
Free Images: MorgueFile.com
• http://www.morguefile.com/corporate/about
– Calendar: http://www.opticgroove.com.au
– Calculator: http://www.therising-sun.us
– Dice: [email protected]
http://www.morguefile.com/corporate/abouthttp://www.morguefile.com/corporate/abouthttp://www.morguefile.com/corporate/abouthttp://www.opticgroove.com.au/http://www.opticgroove.com.au/http://www.opticgroove.com.au/http://www.therising-sun.us/http://www.therising-sun.us/http://www.therising-sun.us/http://www.therising-sun.us/mailto:[email protected]