![Page 1: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/1.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndspsimple software solutions to big business problems.
Making Software. Better.
Smoothing the continuous delivery pathA tale of two teams
![Page 2: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/2.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
What’s this all about?
2
Continuous
DeliveryDone Well
![Page 3: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/3.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
What’s this (Continuous Delivery) all about?
3
“... producing valuable
software in short
cycles and ensure that the software
can be reliably released
at any time”
- https://en.wikipedia.org/wiki/Continuous_delivery
![Page 4: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/4.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Continuous doughnuts
4
Fast feedback
Risk Reduction
T: 0:00:000 T: 2:00:000
![Page 5: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/5.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
There’s no (CD) silver bullet
5
1 (php) monolith50 deploys / day1.5 billion page views / monthhttp://goo.gl/XIB29i
>600 java microservices>100 deploys / day~2 billion requests per day
http://goo.gl/ZJBiK6
![Page 6: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/6.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Prefer Continuous Improvement over Continuous Delivery
▪ .Net monolith▪ 26 -> 24 -> 34 releases
per year▪ https://goo.gl/KF9xXw
6
![Page 7: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/7.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp 7
People first, practices/process second, products last
![Page 8: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/8.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
.Net monolith team - how they roll
8
▪ Private sector
▪ Significant legacy code
▪ High daily traffic (~100 million
visits)
▪ ~8 product teams
▪ Weekly deployments:
– Pick release candidate
– Regression test
– Dog food
– Deploy
![Page 9: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/9.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Scala microservices team - how they roll
9
▪ .Gov - HMRC Digital
▪ 2 years old (greenfield)
▪ Calendar spikes (Visits peak is ~4
million)
▪ >30 product teams
▪ Multiple deployments per day
– Teams “own” their own services
– “You build it, you run it” - https://goo.gl/DyfeOf
![Page 10: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/10.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Continuous
DeliveryDone Well
Patterns for successful practice
10
![Page 11: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/11.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #1 - Healthy CI - mind & body
11
![Page 12: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/12.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #1 - Healthy CI - mind & body
12
Almostgreen!
![Page 13: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/13.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #2 - Testing as an activity, not a phase
13
Product Owner
Define
Dev
Build
QA
Test
Web Ops
Deploy
![Page 14: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/14.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #2 - Testing as an activity, not a phase
14
Product Owner
Define
Dev
Build
QA
Test
Web Ops
Deploy
Test
![Page 15: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/15.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #3 - Healthy automated tests
15
Unit Tests
API
UI
![Page 16: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/16.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #3 - (Un)healthy automated tests
16
API
UI
Unit Tests
The “Hour Glass”
![Page 17: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/17.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
UI
Best practice #3 - (Un)healthy automated tests
17
API
The “Ice Cream Cone”
Unit
Tests
![Page 18: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/18.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #3 - Healthy automated tests
18
Unit Tests
API
UI
The “Tear Drop”
![Page 19: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/19.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #4 - Low (manual) cost deployment (and rollback)
19
VS.
![Page 20: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/20.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #5 - Metrics, monitoring and alerting
20
http://seluxkanaur.deviantart.com/art/Sauron-s-Eye-6-420535318
“Big data”
Social media
User behaviour
Business metrics
Service performance
Service errors
Disk and Network
Monitor all things!
![Page 21: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/21.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Metrics, monitoring and alerting - getting it right
21
Detect
Alert Respond
Display
Analyse
Time & Pain increasing!
![Page 22: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/22.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Metrics, monitoring and alerting - getting it wrong
22
Detect
Respond
Display
Analyse
Time & Pain increasing!Alert
![Page 23: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/23.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Metrics, monitoring and alerting - where and how?
23
Detect
Alert Respond
DisplayAnalyse
![Page 24: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/24.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Metrics, monitoring and alerting - stories from the trenches
24
Detect
Alert Respond
DisplayAnalyse
![Page 25: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/25.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice summary
1. Healthy CI (mind and body)
2. Testing as an activty, not a phase
3. Tear drop shape automated tests (focus on behaviour, with loose
coupling)
4. Low (manual) cost deployment and rollback
5. Metrics, monitoring and reporting
25
Product Owner
Define
Dev
Build
QA
Test
Web Ops
Deploy
Test
Unit Tests
API
UI
Detect
Alert Respond
Display Analyse
![Page 26: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/26.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Shared Pain
26
Continuous
DeliveryDone Well
![Page 27: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/27.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Shared pain #1 - Zero down-time releases
27
https://en.wikipedia.org/wiki/VO2_max
![Page 28: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/28.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Shared pain #2 - Manual deployment steps
28
https://en.wikipedia.org/wiki/Tailor https://en.wikipedia.org/wiki/Zara_(retailer)
VS.
![Page 29: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/29.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Shared pain summary
1. Zero down time releases
2. Manual deployment steps
29
VS.
![Page 30: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/30.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Team specific accelerators
30
Continuous
DeliveryDone Well
![Page 31: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/31.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - 5 why’s
31
● Create a timeline.
● Gather those involved.
● Walk through the timeline.
● Choose an entry point event:
○ Event
○ Why?
○ Process improvement
● Repeat x5 (ish)
![Page 32: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/32.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - 5 why’s - example
32
Event Why? Improvement
Error threshold alert triggered.
Calls to service X were timing out.
Implement failover / circuit breaker.
Calls to service X were timing out.
Queries run by Service X were taking too long.
Alert on long running queries.
Queries run by Service X were taking too long.
Index was missing for a new field.
...
... ... ...
![Page 33: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/33.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - automated release notes - the problem #1
33
Much confusion and
pain
![Page 34: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/34.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - automated release notes - the problem #2
34
Agile Project Management
Tool
Source control system
Features
Stories
Defects
LoginReporting
PaymentsAPI’s
![Page 35: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/35.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - automated release notes - the solution
35
Agile Project Management
Tool
Source control system
Features
Stories
Defects
Login
1. Identify source changes
2. Identify stories/defects
3. Identify areas
4. Group by story/defect
5. For each, summarise areas and authors
6. Group by:a. What’s done
b. What’s in progress
c. Untracked work
Reporting
PaymentsAPI’s
![Page 36: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/36.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Team specific accelerators
36
Continuous
DeliveryDone Well
![Page 37: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/37.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - teams “own” their own microservices
37
▪ “You build it, you run it” - https://goo.gl/DyfeOf
▪ Prefer service over library
dependencies
![Page 38: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/38.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - backward compatible API’s (principle of least surprise)
38
https://en.wikipedia.org/wiki/VO2_max
Production
Staging
QA
Dev
Everyone shares(and cares)
![Page 39: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/39.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Team specific accelerators summary
Monolith:
1. 5 -why’s
2. Automated release note
Microservices:
3. Microservice dependencies, not library
dependencies
4. Backward compatible API’s
39
Agile Project Management
Tool
Source control system
Features
Stories
Defects
LoginReporting
Payments
API’s
![Page 40: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/40.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
References and resources
40
▪ Build Quality In (promo code - LP2015 -
https://goo.gl/ZRuQKw)
▪ Continuous Delivery
▪ Implementing Lean Software
Development
▪ Release It
![Page 41: © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing](https://reader037.vdocuments.mx/reader037/viewer/2022103101/5697bf721a28abf838c7e687/html5/thumbnails/41.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndspsimple software solutions to big business problems.
Making Software. Better.
Thank You
Twitter@EqualExperts@lyndsp
LinkedInlinkedin.com/company/equal-expertsuk.linkedin.com/in/lyndsp
UNITED KINGDOM
+44 203 603 7830
Equal Experts UK Ltd
30 Brock Street
London NW1 3FG
INDIA
+91 20 6607 7763
Equal Experts India Private Ltd
Office No. 4-C
Cerebrum IT Park No. B3
Kumar City, Kalyani Nagar
Pune, 411006
Webwww.equalexperts.comwww.lyndsayp.com
CANADA
+1 403 775 4861
Equal Experts Devices Inc
205 - 279 Midpark way S.E.
T2X 1M2
Calgary, Alberta
PORTUGAL
+
Equal Experts Portugal
Rua Tomás da Fonseca
- Torres de Lisboa
Torre G, 5º Andar
1600-209 Lisboa
Thank You