: replacing a jet engine mid-flight
TRANSCRIPT
![Page 1: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/1.jpg)
Replacing a Jet Engine Mid-flight
Or, how we launched new architecture for
planet-scale distributed system at Google
Aysylu GreenbergAugust 22, 2016
![Page 2: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/2.jpg)
Aysylu Greenberg
@aysylu22
![Page 3: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/3.jpg)
Build System StoryPhoto by imanka / CC BY
![Page 4: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/4.jpg)
Towards Distributed Build System
![Page 5: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/5.jpg)
Towards Distributed Build System
![Page 6: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/6.jpg)
Towards Distributed Build System
![Page 7: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/7.jpg)
Towards Distributed Build System
![Page 8: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/8.jpg)
Towards Distributed Build System
![Page 9: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/9.jpg)
Towards Distributed Build System
![Page 10: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/10.jpg)
Towards Distributed Build System
![Page 11: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/11.jpg)
Towards Distributed Build System
![Page 12: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/12.jpg)
Towards Distributed Build System
![Page 13: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/13.jpg)
Towards Distributed Build System
![Page 14: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/14.jpg)
Towards Distributed Build System
![Page 15: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/15.jpg)
Towards Distributed Build System
![Page 16: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/16.jpg)
Towards Distributed Build System
![Page 17: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/17.jpg)
Towards Distributed Build System
![Page 18: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/18.jpg)
Towards Distributed Build System
![Page 19: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/19.jpg)
Towards Distributed Build System
![Page 20: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/20.jpg)
Towards Distributed Build System
![Page 21: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/21.jpg)
Build System StoryPhoto by imanka / CC BY
![Page 22: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/22.jpg)
Architecture UpgradePhoto by magilla03 / CC BY SA
![Page 23: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/23.jpg)
BuildRabbit:From Experimental Project to Key Piece of Infrastructure
Photo by magilla03 / CC BY SA
![Page 24: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/24.jpg)
From Push to Pull: Client / Server
Client for
User
BuildRabbit
Scheduler
BuildRabbit
Worker
![Page 25: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/25.jpg)
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
From Push to Pull: Build Service
![Page 26: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/26.jpg)
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
From Push to Pull: Build Service
![Page 27: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/27.jpg)
Client
for
User
BuildRabbit
Scheduler
BuildRabbit
Worker
![Page 28: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/28.jpg)
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
Client
for
User
BuildRabbit
Scheduler
BuildRabbit
Worker
![Page 29: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/29.jpg)
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
Client
for
User
BuildRabbit
Scheduler
BuildRabbit
Worker
RESILIENCE
LOGIC
RESILIENCE LOGIC
![Page 30: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/30.jpg)
Architecture UpgradePhoto by magilla03 / CC BY SA
![Page 31: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/31.jpg)
Continuous
integration
system
BuildRabbit Blaze
Source
System
engineers &
teams
Release
infrastructure
Integration
testing
infrastructure
Build
artifact
storage
![Page 32: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/32.jpg)
Continuous
integration
system
BuildRabbit
Source
System
engineers &
teams
Release
infrastructure
Integration
testing
infrastructure
Build
artifact
storage
IN THE
CLOUD
![Page 33: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/33.jpg)
REPLACING A JET ENGINE MID-FLIGHT
![Page 34: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/34.jpg)
Confidential + Proprietary
![Page 35: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/35.jpg)
Photo by moonjazz / CC BY SA
Migration with Zero Downtime
![Page 36: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/36.jpg)
MIGRATE BACKENDS FIRST
![Page 37: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/37.jpg)
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
![Page 38: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/38.jpg)
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
![Page 39: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/39.jpg)
LOTS OF THROWAWAY CODE
MIGRATE BACKENDS FIRST
![Page 40: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/40.jpg)
Photo by goosmurf / CC BY
![Page 41: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/41.jpg)
FOCUS ON MIXED MODE
LOTS OF THROWAWAY CODE
MIGRATE BACKENDS FIRST
![Page 42: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/42.jpg)
FOCUS ON MIXED MODE
LOTS OF THROWAWAY CODE
MIGRATE BACKENDS FIRST
>>> Compatibility window is several months
![Page 43: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/43.jpg)
TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODE
LOTS OF THROWAWAY CODE
MIGRATE BACKENDS FIRST
![Page 44: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/44.jpg)
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
![Page 45: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/45.jpg)
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
![Page 46: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/46.jpg)
DECOUPLE LAUNCH OF SERVICES
TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODE
LOTS OF THROWAWAY CODE
MIGRATE BACKENDS FIRST
![Page 47: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/47.jpg)
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
![Page 48: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/48.jpg)
RPCeventstream
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
![Page 49: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/49.jpg)
HAVE A SOLID ROLLBACK PLAN
DECOUPLE LAUNCH OF SERVICES
TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODE
LOTS OF THROWAWAY CODEMIGRATE BACKENDS FIRST
![Page 50: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/50.jpg)
MAXIMUM VISIBILITY INTO SYSTEM STATE
HAVE A SOLID ROLLBACK PLAN
DECOUPLE LAUNCH OF SERVICES
TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODELOTS OF THROWAWAY CODE
![Page 51: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/51.jpg)
GRADUAL ROLLOUT
MAXIMUM VISIBILITY INTO SYSTEM STATE
HAVE A SOLID ROLLBACK PLAN
DECOUPLE LAUNCH OF SERVICES
TARGET LAUNCH-FRIENDLY CLIENTSFOCUS ON MIXED MODE
![Page 52: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/52.jpg)
PRACTICE BEFORE LAUNCH
GRADUAL ROLLOUT
MAXIMUM VISIBILITY INTO SYSTEM STATE
HAVE A SOLID ROLLBACK PLAN
DECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTS
![Page 53: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/53.jpg)
PRACTICE BEFORE LAUNCH
GRADUAL ROLLOUT
MAXIMUM VISIBILITY INTO SYSTEM STATE
HAVE A SOLID ROLLBACK PLAN
DECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTS
![Page 54: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/54.jpg)
PRACTICE BEFORE LAUNCH, A LOT
GRADUAL ROLLOUT
MAXIMUM VISIBILITY INTO SYSTEM STATE
HAVE A SOLID ROLLBACK PLAN
DECOUPLE LAUNCH OF SERVICESTARGET LAUNCH-FRIENDLY CLIENTS
![Page 55: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/55.jpg)
Photo by moonjazz / CC BY SA
Migration with Zero Downtime
![Page 56: : Replacing a Jet Engine Mid-Flight](https://reader030.vdocuments.mx/reader030/viewer/2022021509/587fac531a28ab107e8b4983/html5/thumbnails/56.jpg)
Replacing a Jet Engine Mid-flight
Or, how we launched new architecture for
planet-scale distributed system at Google
Aysylu Greenberg@aysylu22