© equal experts uk ltd and lyndsayp ltd 2015 @equalexperts @lyndsp simple software solutions to big...

© Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts

@lyndspsimple software solutions to big business problems.

Making Software. Better.

Smoothing the continuous delivery pathA tale of two teams


@lyndsp

What’s this all about?

2

Continuous

DeliveryDone Well


@lyndsp

What’s this (Continuous Delivery) all about?

3

“... producing valuable

software in short

cycles and ensure that the software

can be reliably released

at any time”

- https://en.wikipedia.org/wiki/Continuous_delivery


@lyndsp

Continuous doughnuts

4

Fast feedback

Risk Reduction

T: 0:00:000 T: 2:00:000


@lyndsp

There’s no (CD) silver bullet

5

1 (php) monolith50 deploys / day1.5 billion page views / monthhttp://goo.gl/XIB29i

>600 java microservices>100 deploys / day~2 billion requests per day

http://goo.gl/ZJBiK6


@lyndsp

Prefer Continuous Improvement over Continuous Delivery

▪ .Net monolith▪ 26 -> 24 -> 34 releases

per year▪ https://goo.gl/KF9xXw

6


@lyndsp 7

People first, practices/process second, products last


@lyndsp

.Net monolith team - how they roll

8

▪ Private sector

▪ Significant legacy code

▪ High daily traffic (~100 million

visits)

▪ ~8 product teams

▪ Weekly deployments:

– Pick release candidate

– Regression test

– Dog food

– Deploy


@lyndsp

Scala microservices team - how they roll

9

▪ .Gov - HMRC Digital

▪ 2 years old (greenfield)

▪ Calendar spikes (Visits peak is ~4

million)

▪ >30 product teams

▪ Multiple deployments per day

– Teams “own” their own services

– “You build it, you run it” - https://goo.gl/DyfeOf


@lyndsp

Continuous

DeliveryDone Well

Patterns for successful practice

10


@lyndsp

Best practice #1 - Healthy CI - mind & body

11


@lyndsp

Best practice #1 - Healthy CI - mind & body

12

Almostgreen!


@lyndsp

Best practice #2 - Testing as an activity, not a phase

13

Product Owner

Define

Dev

Build

QA

Test

Web Ops

Deploy


@lyndsp

Best practice #2 - Testing as an activity, not a phase

14

Product Owner

Define

Dev

Build

QA

Test

Web Ops

Deploy

Test


@lyndsp

Best practice #3 - Healthy automated tests

15

Unit Tests

API

UI


@lyndsp

Best practice #3 - (Un)healthy automated tests

16

API

UI

Unit Tests

The “Hour Glass”


@lyndsp

UI

Best practice #3 - (Un)healthy automated tests

17

API

The “Ice Cream Cone”

Unit

Tests


@lyndsp

Best practice #3 - Healthy automated tests

18

Unit Tests

API

UI

The “Tear Drop”


@lyndsp

Best practice #4 - Low (manual) cost deployment (and rollback)

19

VS.


@lyndsp

Best practice #5 - Metrics, monitoring and alerting

20

http://seluxkanaur.deviantart.com/art/Sauron-s-Eye-6-420535318

“Big data”

Social media

User behaviour

Business metrics

Service performance

Service errors

Disk and Network

Monitor all things!


@lyndsp

Metrics, monitoring and alerting - getting it right

21

Detect

Alert Respond

Display

Analyse

Time & Pain increasing!


@lyndsp

Metrics, monitoring and alerting - getting it wrong

22

Detect

Respond

Display

Analyse

Time & Pain increasing!Alert


@lyndsp

Metrics, monitoring and alerting - where and how?

23

Detect

Alert Respond

DisplayAnalyse


@lyndsp

Metrics, monitoring and alerting - stories from the trenches

24

Detect

Alert Respond

DisplayAnalyse


@lyndsp

Best practice summary

1. Healthy CI (mind and body)

2. Testing as an activty, not a phase

3. Tear drop shape automated tests (focus on behaviour, with loose

coupling)

4. Low (manual) cost deployment and rollback

5. Metrics, monitoring and reporting

25

Product Owner

Define

Dev

Build

QA

Test

Web Ops

Deploy

Test

Unit Tests

API

UI

Detect

Alert Respond

Display Analyse


@lyndsp

Shared Pain

26

Continuous

DeliveryDone Well


@lyndsp

Shared pain #1 - Zero down-time releases

27

https://en.wikipedia.org/wiki/VO2_max


@lyndsp

Shared pain #2 - Manual deployment steps

28

https://en.wikipedia.org/wiki/Tailor https://en.wikipedia.org/wiki/Zara_(retailer)

VS.

https://en.wikipedia.org/wiki/Tailor


@lyndsp

Shared pain summary

1. Zero down time releases

2. Manual deployment steps

29

VS.


@lyndsp

Team specific accelerators

30

Continuous

DeliveryDone Well


@lyndsp

Accelerators - 5 why’s

31

● Create a timeline.

● Gather those involved.

● Walk through the timeline.

● Choose an entry point event:

○ Event

○ Why?

○ Process improvement

● Repeat x5 (ish)


@lyndsp

Accelerators - 5 why’s - example

32

Event Why? Improvement

Error threshold alert triggered.

Calls to service X were timing out.

Implement failover / circuit breaker.

Calls to service X were timing out.

Queries run by Service X were taking too long.

Alert on long running queries.

Queries run by Service X were taking too long.

Index was missing for a new field.

...

... ... ...


@lyndsp

Accelerators - automated release notes - the problem #1

33

Much confusion and

pain


@lyndsp

Accelerators - automated release notes - the problem #2

34

Agile Project Management

Tool

Source control system

Features

Stories

Defects

LoginReporting

PaymentsAPI’s


@lyndsp

Accelerators - automated release notes - the solution

35


Tool


Features

Stories

Defects

Login

1. Identify source changes

2. Identify stories/defects

3. Identify areas

4. Group by story/defect

5. For each, summarise areas and authors

6. Group by:a. What’s done

b. What’s in progress

c. Untracked work

Reporting

PaymentsAPI’s


@lyndsp

Team specific accelerators

36

Continuous

DeliveryDone Well


@lyndsp

Accelerators - teams “own” their own microservices

37

▪ “You build it, you run it” - https://goo.gl/DyfeOf

▪ Prefer service over library

dependencies

https://goo.gl/DyfeOf


@lyndsp

Accelerators - backward compatible API’s (principle of least surprise)

38

https://en.wikipedia.org/wiki/VO2_max

Production

Staging

QA

Dev

Everyone shares(and cares)


@lyndsp

Team specific accelerators summary

Monolith:

1. 5 -why’s

2. Automated release note

Microservices:

3. Microservice dependencies, not library

dependencies

4. Backward compatible API’s

39


Tool


Features

Stories

Defects

LoginReporting

Payments

API’s


@lyndsp

References and resources

40

▪ Build Quality In (promo code - LP2015 -

https://goo.gl/ZRuQKw)

▪ Continuous Delivery

▪ Implementing Lean Software

Development

▪ Release It


@lyndspsimple software solutions to big business problems.

Making Software. Better.

Thank You

Twitter@EqualExperts@lyndsp

LinkedInlinkedin.com/company/equal-expertsuk.linkedin.com/in/lyndsp

UNITED KINGDOM

+44 203 603 7830

[email protected]

Equal Experts UK Ltd

30 Brock Street

London NW1 3FG

INDIA

+91 20 6607 7763

[email protected]

Equal Experts India Private Ltd

Office No. 4-C

Cerebrum IT Park No. B3

Kumar City, Kalyani Nagar

Pune, 411006

Webwww.equalexperts.comwww.lyndsayp.com

CANADA

+1 403 775 4861

[email protected]

Equal Experts Devices Inc

205 - 279 Midpark way S.E.

T2X 1M2

Calgary, Alberta

PORTUGAL

+

[email protected]

Equal Experts Portugal

Rua Tomás da Fonseca

- Torres de Lisboa

Torre G, 5º Andar

1600-209 Lisboa

Thank You

mailto:[email protected]


http://www.equalexperts.com/

http://www.equalexperts.com/

http://www.lyndsayp.com/





© equal experts uk ltd and lyndsayp ltd 2015 @equalexperts @lyndsp simple software solutions to big...

Documents

delivery equal experts

glkf9xxw6 equal experts

glzjbik6 equal experts

ones continuous delivery

continuous improvement

delivery frequency

continuous delivery

practice continous delivery