pushing the bottleneck: predicting and addressing the next, next thing
DESCRIPTION
Finding bottlenecks in our software delivery processes is often pretty easy. But once we squash one bottleneck, another team becomes the limiting factor. This presentation looks how bottlenecks work, and how to predict the next bottleneck you'll need to work on.TRANSCRIPT
![Page 1: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/1.jpg)
Pushing the bottleneck
Predicting what will break next in your SDLC
![Page 2: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/2.jpg)
Lead Consultant & Tech Evangelist
Eric is Lead Consultant at IBM UrbanCode Products where I help customers get the most out of their build, deploy and release processes.
Today he works with customers and industry leaders to figure out this DevOps thing.
Eric [email protected]@EricMinick
![Page 3: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/3.jpg)
The plan
Theory of constraints in a nutshell
Finding bottlenecks
Predicting the next bottleneck
Common bottleneck pushing patterns
Q&A
![Page 4: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/4.jpg)
Theory of constraints in a nutshell
![Page 5: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/5.jpg)
A bottleneck is a constraint
Maximum pouring speed
![Page 6: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/6.jpg)
Only by increasing flow through the constraint can overall throughput be increased*
Making the bottlewide down here does
not help
* The Goal: a process of ongoing improvement. Goldratt eta al
![Page 7: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/7.jpg)
Optimize at your constraint
Can pour faster now
![Page 8: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/8.jpg)
Your system must have a measureable goal
My goal is to empty a wine bottle quickly
![Page 9: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/9.jpg)
Simplified five step plan
1. Identify the system’s most severe constraint
2. Decide how to get the most out of the constraint
3. Subordinate everything else to the above constraint
4. Make changes to expand constraint’s capacity
5. Once constraint is relieved, return to step 1
![Page 10: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/10.jpg)
Example: Slow QA cycles
1. Identify the system’s most severe constraint
2. Decide how to get the most out of the constraint
3. Subordinate everything else to the above constraint
4. Make changes to expand constraint’s capacity
5. Once constraint is relieved, return to step 1
1. It takes too long to test our changes. Everyone else waits
2. Testers should focus on exploratory testing
3. Devs help with regression tests. Ops prioritizes QA.
4. Dev & QA work together to automate regression tests
5. Find the next bottleneck
![Page 11: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/11.jpg)
Wait, what’s our goal in software?
Generally: turn ideas into business value
Measuring “business value” hard
Emptying wine bottles only relevant if dev speed is the constraint and you believe in the Ballmer Peak (http://xkcd.com/323/)
![Page 12: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/12.jpg)
Wait, what’s our goal in software?
Generally: turn ideas into business value
Measuring “business value” hard
Features delivered minus bugs is a decent approximation
– …but rewards building useless features.
Emptying wine bottles only relevant if dev speed is the constraint and you believe in the Ballmer Peak (http://xkcd.com/323/)
![Page 13: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/13.jpg)
Three key measures
Lag time: how long from idea to value?
Throughput: how much delivered value per unit time?
Cost: what does it cost to deliver value?
![Page 14: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/14.jpg)
Finding the bottlenecks
![Page 15: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/15.jpg)
Most teams can feel the constraint
What are you waiting on?
Where’s the pain?
Constraints before you in a process feel like not enough
work.
Constraints after you in a process are annoying or
painful
![Page 16: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/16.jpg)
If it hurts, do it more often
Painful processes often grow exponentially worse with large batch sizes.
Examples
– Integration work
– Releases
– Bug Triage
– Updating databases
– Visiting the dentist
http://martinfowler.com/bliki/FrequencyReducesDifficulty.html
Time between doing it
Pain
![Page 17: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/17.jpg)
Use Lean techniques to measure
What does it take to get a change from idea to production?
–At each phase measure wait time and work time
Long wait times indicate large batch sizes or backlogs
image credits: http://commons.wikimedia.org/wiki/File:Diagram_spaghetti_kilka_produktow.PNGhttp://www.michaelnygard.com/blog/2008/02/outrunning_your_headlights.html
![Page 18: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/18.jpg)
Manual fix & verify spaghetti
![Page 19: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/19.jpg)
Bug fix & verify value stream
720
3600
240
2880
720
3600
240
15
15
120
60
15
15
60
Waiting Working
12000 300
1. Feature build
2. Build deployed
3. Bug reported
4. Dev fixes
5. Fix built
6. Fix deployed
7. Tester verifies
![Page 20: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/20.jpg)
Seeing the “next” bottleneck
![Page 21: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/21.jpg)
QA Scenario
Dev Produces a nightly build
Twice weekly, 2 hour deploy
to Test Lab
3 Days to test each drop
Constraint
![Page 22: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/22.jpg)
Dev produces a nightly build
Twice weekly, 2
hour deploy to Test Lab
1.5 Days to test each drop
If we improve test speed, our constraint moves.
New Constraint
![Page 23: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/23.jpg)
Examining a constraint
• Manual process• Limited staff• Production releases have priority
Why can we only deploy twice a week to QA?
![Page 24: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/24.jpg)
Tackling the constraint
• Manual process• Limited staff• Production releases have priority
Why can we only deploy twice a week to QA?
• Automate processes• Hire more staff• Prioritize QA Releases
Options
![Page 25: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/25.jpg)
Imagine a 1 day test cycle
Dev produces a nightly build
2 hour deploy to Test Lab
1.0 Days to test each
drop
¼ day deploy downtime becomes turns 1 day test cycle into two days.
Next Constraint
![Page 26: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/26.jpg)
Tackling the constraint
• Manual process• Limited staff• Production releases have priority
Why can we only deploy twice a week to QA?
• Automate processes• Hire more staff• Prioritize QA deploys
Options
Short term approach
Long term approach
![Page 27: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/27.jpg)
Measuring utilization helps with this prediction
“Feeling” the pain isn’t enough to predict the next constraint
There may be no pain at the next constraint today
![Page 28: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/28.jpg)
When something is free, it is used more
Example: Amazon Prime. For $79/yr, customers get free 2 day shipping on everything.
“…Customers spent as much as 150% more at Amazon after they became Prime members.
Subscribers not only ordered more often … they started buying things at Amazon that they
probably wouldn’t have in the past” *
* http://business.time.com/2013/03/18/amazon-prime-bigger-more-powerful-more-profitable-than-anyone-imagined/
![Page 29: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/29.jpg)
Consider implications of making something free
If builds, deploy, regression tests are free…
We’re going to be testing lots more. Better buy some
hardware.
![Page 30: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/30.jpg)
Common “next bottleneck” patterns
![Page 31: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/31.jpg)
After build, deploy is next
Build guy & deploy guy used to be in sync
A CI server can do hundreds of builds per day
Agile tends to make Operations a constraint
Knowing my stuff compiles at all times is great. I
want to know if it passes functional
tests too.
![Page 32: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/32.jpg)
As tests shift left, expensive tests constrain
Developers run more integration tests
Some tests use expensive resources
–pay per use web services, mainframes, production systems…
Stubbing those resources becomes important
– known as “service virtualization”
![Page 33: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/33.jpg)
Concurrent agile dev requires more test labs
More code is compiling, and set to be released “soon”
Need more environments to test changes in
– Pressure for Platform as a Service
![Page 34: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/34.jpg)
Frequent releases demand fast feedback
Frequent releases enable experimentation
Monitoring of business outcomes required
Architectural pressures to support A/B testing
Stop ignoring difference between features delivered
and value delivered
![Page 35: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/35.jpg)
If we keep chasing constraints, where does it end?
![Page 36: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/36.jpg)
You may end up with Continuous Deployment
Build
Run thousands of tests
Deploy to some servers
Monitor
Deploy to remaining servers
![Page 37: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/37.jpg)
Summary
Optimizations other than at the constraint don’t help
“Breaking” one constraint will expose the next
Patterns or analysis can be used to predict next constraint
Continual Improvement, Continual Improvement, Continual Improvement, Continual Improvement…
![Page 38: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/38.jpg)
IBM will collaborate with you to understand your current situation,
goals and constraints. The Assessment and Planning
Workshop will aim to capture sufficient information to make specific recommendations for
improvement and implementation.
Intended Audience:- Key leadership from practice areas and
stakeholder organizations Value Proposition
- Clear recommendations for capability improvements aligned to your business goals
- Initial Architecture- Adoption roadmap based on proven best
practices Activities
- Workshop planning- Assessment and Planning Workshop- Collaborative discussion on current status, future
goals and adoption requirements- Produce Deliverable
Deliverables- Current Status and Improvement
Recommendations- Architecture - Adoption Roadmap
Assessment and Planning Workshop
![Page 39: Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing](https://reader033.vdocuments.mx/reader033/viewer/2022060108/554f8fe1b4c9052a518b530f/html5/thumbnails/39.jpg)
More resources
Urbancode.com/resources
Continuous Delivery Maturity Model
Deployment Automation Basics
Applying Lean Principals to Software Delivery
Blogs.urbancode.com
Twitter.com/UrbanCode
Facebook.com/IBMUrbanCodeProducts
SlideShare.net/Urbancode