VINCENT KOK DEVELOPMENT MANAGER ATLASSIAN @VINCENTKOK

Repairing a Confluence of broken windows

Big cool statistic

2,569

Add-Ons in Marketplace

Monolith

https://flic.kr/p/nA4S49

https://flic.kr/p/nA4S49

Big cool statistic

2,569

Add-Ons in Marketplace

Cool stories

http://geekandpoke.typepad.com/geekandpoke/2011/08/hudson-status-monitor.html

http://geekandpoke.typepad.com/geekandpoke/2011/08/hudson-status-monitor.html

Big cool statistic

2,569

Add-Ons in Marketplace

Got issues?

https://flic.kr/p/fSopv4

https://flic.kr/p/fSopv4

Big cool statistic

2,569

Add-Ons in Marketplace

Journey

https://flic.kr/p/8WQutt

https://flic.kr/p/8WQutt

C O N F L U E N C E P I P E L I N E

M I C R O S E RV I C E E N V Y

4 P R O B L E M S

Agenda

C U LT U R E

Big cool statistic

2,569

Add-Ons in Marketplace

Confluence Pipeline

https://flic.kr/p/jus4Bt

https://flic.kr/p/jus4Bt

Nr of daily master builds

286

Nr of tests

31992

Nr of build agents

800

Flakes

NaN

Size of Pipeline team

3

Pipeline stages

OuterInner Release

Confluence Pipeline

Publish Deploy

ReleaseInner Outer

A1 A2

True story

Publish Deploy

ReleaseInner Outer

A1 A2

A1

A2Manual release

branch

Development Env

Big cool statistic

2,569

Add-Ons in Marketplace

Not Great

https://flic.kr/p/7D1D1N

https://flic.kr/p/7D1D1N

< 1 hour

Big cool statistic

2,569

Add-Ons in Marketplace

Culture

https://flic.kr/p/DsetEn

https://flic.kr/p/DsetEn

GreenDex

GreenDex

ResultsBeliefsExperience Actions

The Oz Principle

The way youve been shaped

Beliefs are based on your experience

You take actions based on your beliefs

The outcome of our actions

Clear goal Early results Momentum

Theory in real life

Big cool statistic

2,569

Add-Ons in Marketplace

Microservice Envy

https://flic.kr/p/b7ak8e

https://flic.kr/p/b7ak8e

Smaller Clear ownership Independent Lifecycle

Microservice traits

Big cool statistic

2,569

Add-Ons in Marketplace

Monolith

https://flic.kr/p/nA4S49

https://flic.kr/p/nA4S49

< 1 hour

Once a Day

ReleasesOwnershipComplexity Testing

Examples

Impact

Early result

Iterate

Big cool statistic

2,569

Add-Ons in Marketplace

#1: Complexity

https://flic.kr/p/cFM3cd

https://flic.kr/p/cFM3cd

Show me complexity

Outer

A1 A2

A1

A2Manual release

branch

Development Env

Manual tasks Flawed model Late feedback

Impact

Big cool statistic

2,569

Add-Ons in Marketplace

Fix the model

https://flic.kr/p/8ykpkW

https://flic.kr/p/8ykpkW

Early result

Outer

A1

A2

Manual release

branch

Development Env

A2

Fork/Join details

Inner loop Build times Build stability

Iterate

Big cool statistic

2,569

Add-Ons in Marketplace

#2: Ownership

https://flic.kr/p/bfRuoX

https://flic.kr/p/bfRuoX

Getting the model wrong

Fire fighting Broken windows No improvements

Impact

Dev Speed Feedback loop MTTR

Impact

Ownership

Development Teams

Synergy

Site Reliability Engineering

Pipeline

Early result

Visibility

Measure everything

Move the needle

Visibility

Measure everything

Move the needle

Visibility

Measure everything

Move the needle

Engagement Model Ownership Enforce by automation

Iterate

Big cool statistic

2,569

Add-Ons in Marketplace

#3: Testing

https://flic.kr/p/cBbxuQ

https://flic.kr/p/cBbxuQ

Unit

Service

UI

Unit

Service

UI

Slowness Duplication Flakes

Impact

Time Culture Broken windows

The effect of flakes

Retries as a pain killerAutomatic retries to relieve some pain. Remember this is a workaround

Early result

Big cool statistic

2,569

Add-Ons in Marketplace

Fix the root cause

https://flic.kr/p/J44Nm

https://flic.kr/p/J44Nm

Iterate

Test strategy Flake policy

DeleteIssueDetect Fix

Sustainable environment

Big cool statistic

2,569

Add-Ons in Marketplace

#4: Releases

https://flic.kr/p/91MFkh

https://flic.kr/p/91MFkh

Release cycle

Big cool statistic

2,569

Add-Ons in Marketplace

Congestion

https://flic.kr/p/bRgBZP

https://flic.kr/p/bRgBZP

Dogfooding becomes hard

Big bangEnds at master Cancellations

Impact

For once a free lunchFork and Join solved most of our problems already

Early result

Big cool statistic

2,569

Add-Ons in Marketplace

Dogfooding

https://flic.kr/p/dxpLTm

https://flic.kr/p/dxpLTm

Big cool statistic

2,569

Add-Ons in Marketplace

Feedback

https://flic.kr/p/dxaAdk

https://flic.kr/p/dxaAdk

Big cool statistic

2,569

Add-Ons in Marketplace

Soak time

https://flic.kr/p/fmfG2q

https://flic.kr/p/fmfG2q

Decouple code and features

Merge often One process

Feature flags

Big cool statistic

2,569

Add-Ons in Marketplace

One more time

https://flic.kr/p/bsBaMn

https://flic.kr/p/bsBaMn

< 1 hour

Once a Day

ReleaseOwnershipComplexity Testing

4 problems

Clear goal Early results Momentum

Experience & Beliefs

http://geekandpoke.typepad.com/.a/6a00d8341d3df553ef015392a77686970b-pi

http://geekandpoke.typepad.com/.a/6a00d8341d3df553ef015392a77686970b-pi

Thank you!

VINCENT KOK DEVELOPMENT MANAGER ATLASSIAN @VINCENTKOK