evidence based decision-making - lean product development
TRANSCRIPT
Evidence-based Decision-making for Lean Product Development
presented byKevin Burns@ MN AEG
Jan. 17, 2017
[email protected], @kevinbburns
My work history and experience
[email protected], @kevinbburns 3
[email protected], @kevinbburns 4
Peace Corp Recruitment and Public AffairsStory of how we used technology to improve Peace Corp recruitment.
Switch from USPS to email
Switch from manual data entry to wild-card search in gopher email system and screen scraping results, an early for of ETL.
Conduct direct email campaigns when spam still meant ‘meat in a can’
[email protected], @kevinbburns 5
[email protected], @kevinbburns 6
Open Discussion
Who’s measuring value, outcomes, and/or impacts today?
How are you measuring them?
If you’re not measuring them, why not?
Who’s measuring cost?
How are you measuring cost?
[email protected], @kevinbburns 7
The 1st Agile Principle
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
(should we change valuable to beneficial impact?)
How do we define value (impact) and how do we measure it?
Not all Projects (or Features) are created equal.
[email protected], @kevinbburns 8
Is value determined by delivery on time, on budget, and on scope?
Is the scope delighting the customer?Are they using everything we delivered?
[email protected], @kevinbburns 9
In a survey of 4 products, 65% of the features were rarely or never used.
How much money could have been saved if we never built them?
In the Waterfall project world, we have to ask for everything we can think of because capital will end at the end of the project. Instead we should be asking what has the most value in terms of the business outcome and/or impact and how are we going to measure it.
[email protected], @kevinbburns 10
[email protected], @kevinbburns 11
[email protected], @kevinbburns 12
Marty Cagan Quotes
• Customers don’t know what they want. It’s very hard to envision the solution you want without actually seeing it.
• At least 2/3 of our ideas are never going to work. The other 1/3 will take 3-4 iterations to get right.
• The role of the product manager is to discover a product that is valuable, usable, and feasible. Product, design, and engineering work together to arrive at optimal solution.
[email protected], @kevinbburns 13
[email protected], @kevinbburns 14
What we measure is changing
Business CustomerPO, SM, BL
Software EngineeringAD, DD, DA
UserUX, BA, QA, SME
BusinessValuable
Design Usable
TechnicallyFeasible
INNOVATIVESOLUTION
[email protected], @kevinbburns 15
Lean Startup
• Are we asking what are Minimum Viable (Valuable) Product and how do we know when we’ve delivered it?
• Use a scientific method to measure, learn and pivot or preserver.
• Use meaningful quantitative objective measure to evaluate impact.
• Can you use A/B testing?
[email protected], @kevinbburns 16
MVPInnovation
UserUX, BA, QA, SME
Business Valuable
Design Usable
Software EngineeringAD, DD, DA
Business CustomerPO, SM, BL
Use scientific method (measurable) to learn
and discovery your Minimum Viable
(Valuable) Product (MVP)
Technically Feasible
MVP innovations emerge from Conversations
[email protected], @kevinbburns 17
Value and/or Impact driven culture
• Are we measuring the Cost vs Benefit at all levels of our work items?• Portfolio• Program• Project• Feature/Capability• Story/Requirement• Tasks/Test
• Are we measuring the Impact our features have on our customers?
• The act of sizing helps us define done and value
• Use story telling and test statements create understanding of value and DoD
[email protected], @kevinbburns 18
Don Reinertsen’s Seven Big Ideas2nd Generation Lean Product Development1. Understand your economics2. Manage your queues3. Exploit variability4. Enable smaller batches5. Control WIP and start rates6. Prioritize based on economics7. Accelerate feedback
[email protected], @kevinbburns 19
1. Understand Your Economics
• In product development all difficult decisions involve multiple variables.
• Making decisions that affect multiple variables requires quantification.
• Doing quantification, isn’t as hard as you might think.
[email protected], @kevinbburns 20
A Typical Question
Should we operate our testing process at 80% utilization with a 2-week queue, or 90% utilization with a 4 week queue?
[email protected], @kevinbburns 21
Making Economic Decisions
• Waste• Cycle time• Variability• Efficiency• Revenue• Unit Cost• Value-Added
Proxy Variable Space
Tran
sfor
mat
ions
Life Cycle Profits
Economic Space
[email protected], @kevinbburns 22
Modeling Process
Model Expense Overrun
Model Cost
Overrun
Model Value
Shortfall
Model Schedule
Delay
Model Risk
Change
Create Baseline Model
Determine Total Profit Impact of Missing a MOP
Calculate Sensitivity Factors
[email protected], @kevinbburns 23
Guidelines for decision-making
• Set strategic goals/guidelines for decision-making so low-level tactical decision can be decentralized while still being aligned within strategic goals/objectives.
• While good intentioned, centralized decision-making is often slow and suboptimal because it lacks the context of all the variable at play at tackle level
[email protected], @kevinbburns 24
2. Manage Your Queues
• Many product developers assume higher utilization leads to faster development.
• They neither measure nor manage the invisible queues in their process.
• Consequently, they underestimate the true cost of overloading their processes.
• Such overloads severely hurt all aspects of development performance.
[email protected], @kevinbburns 25
Effect of Capacity Utilization
10 20 30 40 50 60 70 80 90 100
05
1015
20
Que
ue S
ize
% Capacity [email protected], @kevinbburns 26
Managing Queues
Cost of Excess Delay
Total Cost
Dollars
Excess Product Development Resources
Minimize Total Cost to Maximize Profits
[email protected], @kevinbburns 27
Why Queues Matter
• Queues Create…• Longer cycle time• Lower Quality• More variability• Increased risk• More overhead• Less motivation
Managing queues is the key to improving product development economics
[email protected], @kevinbburns 28
3. Exploit Variability
• In manufacturing it is always desirable to reduce variability
• In product development eliminating variability eliminates innovation
• We must understand the specific conditions that make variability valuable and manage our process to create these conditions
• We need development process that function in the presence of variability
[email protected], @kevinbburns 29
Asymmetric Payoff and Option Pricing
Expected Price Payoff vs Price
Expected Payoff
Expe
cted
Pay
off
Price
PricePrice
Prob
abili
ty
Payo
ff Strike Price
Strike Price
[email protected], @kevinbburns 30
Higher Variability Raises the PayoffEx
pect
ed P
ayof
f
Price
Strike Price
Payoff SD = 15
Payoff SD = 5
Option Price = 2, Strike Price = 50, Mean Price – 50, Standard Deviation = 5 and 15
[email protected], @kevinbburns 31
4. Enable Smaller Batches
• When work products are invisible, batch sizes are invisible• When batch sizes are invisible, product developers pay little attention
to them• Many companies institutionalize large batch sizes• Batch size reduction is attractive because it is fast, easy, cheap,
granular, leveraged, and reversible• It is a great starting point for LPD
Batch Size Queues Cycle Time
X 0.5 X 0.5 X [email protected], @kevinbburns
32
Drawing Review Process
200
10 Weeks
20
1 Week
Unreviewed Drawings
Large Batch Small Batch
[email protected], @kevinbburns33
Benefits of Small Batch Testing
Higher ValidityFewer Open Bugs
Faster Cycle Time
Early Feedback
Less Debug Complexity
More Efficient Debugging
More Uptime
Smaller Change
Fewer Status Reports
Less Requirement Changes
Faster Learning
Lower Cost Changes
Cheaper Debugging
Cheaper Testing
Less Non-Value-Added
Better Code
Cheaper Correction
Better Economics
[email protected], @kevinbburns 34
Setting Batch Size
Transaction Cost
Cost
Items per Batch
Economic Batch Size
1 2 3 4 5 6 7 8 9 10
05
1015
20
Total Cost
[email protected], @kevinbburns 35
5. Control WIP and Start Rates
• Many developers incorrectly assume that the sooner they start work, the sooner they will finish it
• They are constantly tempted to start too much work
• This dilutes resources and causes long transit time through their processes
• A long transit time hurts efficiency, quality and responsiveness
[email protected], @kevinbburns 36
Little’s Formula
• By constraining WIP in development processes we can control cycle time
• This approach, which is known as Lean Kanban, is currently growing rapidly in software development
https://en.wikipedia.org/wiki/Little%27s_law
MeanResponseTime = MeanNumberInSystem / MeanThroughput
[email protected], @kevinbburns 37
Control Number of Active Projects
1
2
3
4
1
2
3
4
COD Savings of Project 1 and 2 Late Start Advantages for Project 3 and 4
Time to Deliver
Time to Deliver Time to Deliver
[email protected], @kevinbburns 38
Avoid Long Planning Horizons
• The further out you plan, the less likely your forecast will be accurate
• Don’t do detailed analysis on things beyond a quarter
• Market conditions change everyday, this can change requirements
• Changing requirements cause churn (waste)
[email protected], @kevinbburns 39
Visual WIP Control Boards
Ready Queue Coding Ready to Test Testing Done
WIP constraints = 10
13
14
15
16
11
10
9
8
7
6
5
4
3
2
112
?
[email protected], @kevinbburns 40
6. Sequence Work Correctly
• The sequence in which work is processed is called the queuing discipline
• By changing the queuing discipline we can reduce the cost of a queue without decreasing the size of the queue
• Since manufacturing has homogeneous flows it always uses FIFO (First-In-First-Out)
• For the non-homogeneous flows of product development other approaches have better economics
[email protected], @kevinbburns 41
Use FIFO for Homogeneous Flow
First-In First-Out
Cost of
Delay
12
3A
B
Time
Cost
Delay CostLast-In First-Out
Cost of
Delay 12
3
AB
Time
Cost
Project Duration Cost of Delay
1 3 3
2 3 3
3 3 3
[email protected], @kevinbburns 42
Weighted Shortest Job First (WSJF) for Non-homogenous flow
High Weight First
Cost of
Delay
1
23
AB
Time
Cost
Delay Cost
Low Weight First
Cost of
Delay
A
B
Time
Cost
Project Duration Cost of Delay
Weight = COD/Duration
1 1 10 10
2 3 3 1
3 10 1 0.1
1
23
160 796 % Reduction in COD
[email protected], @kevinbburns 43
7. Create Faster Feedback
• When queues and batch sizes are large feedback is slow
• Slow feedback hurts quality, efficiency, and cycle time
• Feedback speed has enormous economic leverage in product development, but it is rarely explicitly managed
[email protected], @kevinbburns 44
The Front-Loaded Lottery
• A lottery ticket pays $3000 to winning three digit number• You can pick the number in two ways:
• Pay $3 to select all three digits at once• Pay $1 for the first digit, find out if it is correct, then choose if you wish to pay
$1 for the second digit, and then choose if you wish to pay $1 for the third digit.
[email protected], @kevinbburns 45
Value of Feedback
100%
Spend $1
Savings = $0.90Savings = $0.99
10%1%
0 $1 $2 $3
Probabilityof
Occurrence
Cumulative [email protected], @kevinbburns 46
References
• Don Reinertsen• https://www.youtube.com/watch?v=L6v6W7jkwok&spfreload=1#t=16.242347• https://www.amazon.com/gp/product/1935401009/ref=as_li_tl?ie=UTF8&camp
=1789&creative=9325&creativeASIN=1935401009&linkCode=as2&tag=reinertassoci-20&linkId=U56AHKXXQVN4VZFY
• https://en.wikipedia.org/wiki/Little%27s_law
[email protected], @kevinbburns 48