open business conference: continuous delivery at netflix -- powered by open source

Continuous Delivery at Netflix: Powered by Open Source

Dianne MarshSan Francisco, May 6, 2014

http://www.linkedin.com/in/diannemarsh

Building Continuous Delivery

Netflix GoalsHigh Availability

But … Move fastTools encourage Best Practices

But … Freedom to do the right thing

Teams Deploy Their Own CodeRun What You Wrote• Rapid Innovation• Rapid Detection• Rapid Response= Freedom + Responsibility

Nebula/Gradle• Netflix Build Language• Gradle plugins• Supports internal and open source

builds

https://github.com/nebula-plugins

BUILD

BUILD Jenkins Job DSLConfiguration as CodeGroovy Script Scripts go in Version Control

http://www.slideshare.net/quidryan/configuration-as-code

BUILD BAKE

Aminator• Create AMI from Base AMI• Image contains service and

everything needed to run it• Unit of Deployment for Test and

Prod• Abstracts Cloud Details

http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html

BAKE

BUILD DEPLOYBAKE

DEPLOY

Asgard: AWS Deployment ToolDeploys Netflix to the CloudRed/Black push

http://www.infoq.com/presentations/asgard

CANARY ANALYSISTest, Int, ProdChoose where to deployRun canary analysisScale up new instancesTurn on traffic to new ASGTurn off traffic to old ASGWait … analyze … continue

Asgard Developer Portal

GLISTEN Extending Asgard’s WorkflowAutomated Red/Black Push

Run canary/analysisScale up new instancesTurn on traffic/Run Tests/Turn off trafficWait … analyze … continue

http://techblog.netflix.com/2013/09/glisten-groovy-way-to-use-amazons.html

One Button Deployment?

Regional IsolationLimit Impact of Human Error• Stagger deployments?• Canary testing per region?

Know your service!

Multi-Region ConsistencyBuild Tooling to:• Schedule Deployments• Prefer off peak• Choose next available

region automatically• Provide high visibility

per region

BUILD

DEPLOY

BAKE

Simian Army• Chaos Monkey• Latency Monkey• Janitor Monkey• Conformity Monkey (and more!)

Test resiliency at runtime

http://www.infoq.com/presentations/netflix-resiliency-failure-cloud

Send in the Conformity MonkeyHave deployments diverged? Balance regional consistency

with regional isolation Provide meaningful thresholds Build best practices into

tooling and reporting

Clean up with the Janitor Monkey• Disassociate unused EIPs• Delete unassociated

Amazon EBS volumes• Delete older Amazon

EBS snapshots• Leverage Amazon S3

Object Expiration

Key Elements for Netflix• Value Self-service • Test Everywhere• Build Awareness of Multiple Regions• Avoid peak times• Roll back quickly and easily• Be Cloud Native

Netflix OSS

More at netflix.github.com

Thanks!

Dianne Marsh (@dmarsh)dmarsh@netflix.com

Thanks!