![Page 1: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/1.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndspsimple software solutions to big business problems.
Making Software. Better.
Smoothing the continuous delivery pathA tale of two teams
![Page 2: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/2.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
What’s this all about?
2
Continuous
DeliveryDone Well
![Page 3: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/3.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
What’s this (Continuous Delivery) all about?
3
“... producing valuable
software in short cycles and ensure that the software
can be reliably released
at any time”
- https://en.wikipedia.org/wiki/Continuous_delivery
![Page 4: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/4.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Continuous doughnuts
4
Fast feedback
Risk Reduction
T: 0:00:000 T: 2:00:000
![Page 5: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/5.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
There’s no (CD) silver bullet
5
1 (php) monolith50 deploys / day1.5 billion page views / monthhttp://goo.gl/XIB29i
>600 java microservices>100 deploys / day~2 billion requests per day
http://goo.gl/ZJBiK6
![Page 6: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/6.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Prefer Continuous Improvement over Continuous Delivery
▪ .Net monolith▪ 26 -> 24 -> 34 releases
per year▪ https://goo.gl/KF9xXw
6
![Page 7: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/7.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp 7
People first, practices/process second, products last
![Page 8: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/8.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
.Net monolith team - how they roll
8
▪ Private sector▪ Significant legacy code▪ High daily traffic (~100 million
visits)▪ ~8 product teams▪ Weekly deployments:
– Pick release candidate– Regression test– Dog food– Deploy
![Page 9: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/9.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Scala microservices team - how they roll
9
▪ .Gov - HMRC Digital▪ 2 years old (greenfield)▪ Calendar spikes (Visits peak is ~4
million)▪ >30 product teams▪ Multiple deployments per day
– Teams “own” their own services– “You build it, you run it” -
https://goo.gl/DyfeOf
![Page 10: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/10.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Continuous
DeliveryDone Well
Patterns for successful practice
10
![Page 11: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/11.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #1 - Healthy CI - mind & body
11
![Page 12: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/12.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #1 - Healthy CI - mind & body
12
Almostgreen!
![Page 13: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/13.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #2 - Testing as an activity, not a phase
13
Product Owner
Define
Dev
Build
QA
Test
Web Ops
Deploy
![Page 14: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/14.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #2 - Testing as an activity, not a phase
14
Product Owner
Define
Dev
Build
QA
Test
Web Ops
Deploy
Test
![Page 15: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/15.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #3 - Healthy automated tests
15
Unit Tests
API
UI
![Page 16: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/16.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #3 - (Un)healthy automated tests
16
API
UI
Unit Tests
The “Hour Glass”
![Page 17: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/17.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
UI
Best practice #3 - (Un)healthy automated tests
17
API
The “Ice Cream Cone”
Unit
Tests
![Page 18: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/18.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #3 - Healthy automated tests
18
Unit Tests
API
UI
The “Tear Drop”
![Page 19: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/19.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #4 - Low (manual) cost deployment (and rollback)
19
VS.
![Page 20: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/20.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice #5 - Metrics, monitoring and alerting
20
http://seluxkanaur.deviantart.com/art/Sauron-s-Eye-6-420535318
“Big data”Social mediaUser behaviourBusiness metricsService performanceService errorsDisk and Network
Monitor all things!
![Page 21: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/21.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Metrics, monitoring and alerting - getting it right
21
DetectAlert Respon
d
DisplayAnalyse
Time & Pain increasing!
![Page 22: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/22.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Metrics, monitoring and alerting - getting it wrong
22
DetectRespon
d
DisplayAnalyse
Time & Pain increasing!Alert
![Page 23: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/23.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Metrics, monitoring and alerting - where and how?
23
DetectAlert Respon
d
Display Analyse
![Page 24: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/24.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Metrics, monitoring and alerting - stories from the trenches
24
DetectAlert Respon
d
Display Analyse
![Page 25: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/25.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Best practice summary
1. Healthy CI (mind and body)
2. Testing as an activty, not a phase
3. Tear drop shape automated tests (focus on behaviour, with loose
coupling)
4. Low (manual) cost deployment and rollback
5. Metrics, monitoring and reporting
25
Product Owner
Define
Dev
Build
QA
Test
Web Ops
Deploy
Test
Unit Tests
API
UI
Detect
Alert Respond
Display Analyse
![Page 26: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/26.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Shared Pain
26
Continuous
DeliveryDone Well
![Page 27: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/27.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Shared pain #1 - Zero down-time releases
27
https://en.wikipedia.org/wiki/VO2_max
![Page 28: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/28.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Shared pain #2 - Manual deployment steps
28
https://en.wikipedia.org/wiki/Tailor https://en.wikipedia.org/wiki/Zara_(retailer)
VS.
![Page 29: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/29.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Shared pain summary
1. Zero down time releases
2. Manual deployment steps
29
VS.
![Page 30: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/30.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Team specific accelerators
30
Continuous
DeliveryDone Well
![Page 31: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/31.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - 5 why’s
31
● Create a timeline.● Gather those involved.● Walk through the timeline.● Choose an entry point event:
○ Event○ Why?○ Process improvement
● Repeat x5 (ish)
![Page 32: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/32.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - 5 why’s - example
32
Event Why? Improvement
Error threshold alert triggered.
Calls to service X were timing out.
Implement failover / circuit breaker.
Calls to service X were timing out.
Queries run by Service X were taking too long.
Alert on long running queries.
Queries run by Service X were taking too long.
Index was missing for a new field.
...
... ... ...
![Page 33: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/33.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - automated release notes - the problem #1
33
Much confusion and
pain
![Page 34: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/34.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - automated release notes - the problem #2
34
Agile Project Management
Tool
Source control system
Features
Stories
Defects
LoginReporting
PaymentsAPI’s
![Page 35: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/35.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - automated release notes - the solution
35
Agile Project Management
Tool
Source control system
Features
Stories
Defects
Login
1. Identify source changes2. Identify stories/defects3. Identify areas4. Group by story/defect5. For each, summarise
areas and authors6. Group by:
a. What’s doneb. What’s in progressc. Untracked work
Reporting
PaymentsAPI’s
![Page 36: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/36.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Team specific accelerators
36
Continuous
DeliveryDone Well
![Page 37: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/37.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - teams “own” their own microservices
37
▪ “You build it, you run it” - https://goo.gl/DyfeOf
▪ Prefer service over library dependencies
![Page 38: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/38.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Accelerators - backward compatible API’s (principle of least surprise)
38
https://en.wikipedia.org/wiki/VO2_max
ProductionStaging
QADev
Everyone shares(and cares)
![Page 39: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/39.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
Team specific accelerators summary
Monolith:1. 5 -why’s2. Automated release noteMicroservices:3. Microservice dependencies, not library
dependencies4. Backward compatible API’s
39
Agile Project Management
Tool
Source control system
Features
Stories
Defects
LoginReporting
PaymentsAPI’s
![Page 40: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/40.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndsp
References and resources
40
▪ Build Quality In (promo code - LP2015 -
https://goo.gl/ZRuQKw)
▪ Continuous Delivery
▪ Implementing Lean Software
Development
▪ Release It
![Page 41: Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer](https://reader036.vdocuments.mx/reader036/viewer/2022062522/5871f14a1a28ab5c348b5951/html5/thumbnails/41.jpg)
© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts
@lyndspsimple software solutions to big business problems.
Making Software. Better.
Thank You
Twitter@EqualExperts@lyndsp
LinkedInlinkedin.com/company/equal-expertsuk.linkedin.com/in/lyndsp
UNITED KINGDOM
+44 203 603 [email protected] Experts UK Ltd30 Brock StreetLondon NW1 3FG
INDIA
+91 20 6607 [email protected] Experts India Private LtdOffice No. 4-CCerebrum IT Park No. B3Kumar City, Kalyani NagarPune, 411006
Webwww.equalexperts.comwww.lyndsayp.com
CANADA
+1 403 775 [email protected] Experts Devices Inc205 - 279 Midpark way S.E.T2X 1M2Calgary, Alberta
PORTUGAL
+ [email protected] Experts PortugalRua Tomás da Fonseca - Torres de LisboaTorre G, 5º Andar1600-209 Lisboa
Thank You