open business conference: continuous delivery at netflix -- powered by open source
DESCRIPTION
How can we engineer velocity by weaving together tooling and culture with software development to expose and elevate highly effective practices? This talk describes various components of Netflix’s continuous delivery platform -- much of which is available in open source. I’ll show how these pieces fit together and allow us to build scaffolding so that we’re comfortable with software developers making the decision to push the button for prod deployment -- and helps them to recover if necessary. As a result, we can run fast, trusting our tooling and our culture. I’ll also describe how we test our resiliency through simulating failure, unleashing the monkeys (Simian Army) on our production environment. Because if you’re afraid of cute little monkeys, imagine how afraid you’ll be of a production environment that offers those same risks but doesn’t give you an opportunity to test your response to those dangers. Throughout this talk, I hope that you will challenge yourself to consider how your company can "shift the curve" through tooling and to achieve a high velocity environment without negatively impacting reliability.TRANSCRIPT
![Page 1: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/1.jpg)
![Page 2: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/2.jpg)
Continuous Delivery at Netflix: Powered by Open Source
Dianne MarshSan Francisco, May 6, 2014
http://www.linkedin.com/in/diannemarsh
![Page 3: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/3.jpg)
Building Continuous Delivery
![Page 4: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/4.jpg)
Netflix GoalsHigh Availability
But … Move fastTools encourage Best Practices
But … Freedom to do the right thing
![Page 5: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/5.jpg)
Teams Deploy Their Own CodeRun What You Wrote• Rapid Innovation• Rapid Detection• Rapid Response= Freedom + Responsibility
![Page 6: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/6.jpg)
Nebula/Gradle• Netflix Build Language• Gradle plugins• Supports internal and open source
builds
https://github.com/nebula-plugins
BUILD
![Page 7: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/7.jpg)
BUILD Jenkins Job DSLConfiguration as CodeGroovy Script Scripts go in Version Control
http://www.slideshare.net/quidryan/configuration-as-code
![Page 8: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/8.jpg)
BUILD BAKE
![Page 9: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/9.jpg)
Aminator• Create AMI from Base AMI• Image contains service and
everything needed to run it• Unit of Deployment for Test and
Prod• Abstracts Cloud Details
http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html
BAKE
![Page 10: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/10.jpg)
BUILD DEPLOYBAKE
![Page 11: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/11.jpg)
DEPLOY
Asgard: AWS Deployment ToolDeploys Netflix to the CloudRed/Black push
http://www.infoq.com/presentations/asgard
![Page 12: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/12.jpg)
CANARY ANALYSISTest, Int, ProdChoose where to deployRun canary analysisScale up new instancesTurn on traffic to new ASGTurn off traffic to old ASGWait … analyze … continue
![Page 13: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/13.jpg)
Asgard Developer Portal
![Page 14: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/14.jpg)
GLISTEN Extending Asgard’s WorkflowAutomated Red/Black Push
Run canary/analysisScale up new instancesTurn on traffic/Run Tests/Turn off trafficWait … analyze … continue
http://techblog.netflix.com/2013/09/glisten-groovy-way-to-use-amazons.html
![Page 15: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/15.jpg)
One Button Deployment?
![Page 16: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/16.jpg)
Regional IsolationLimit Impact of Human Error• Stagger deployments?• Canary testing per region?
Know your service!
![Page 17: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/17.jpg)
Multi-Region ConsistencyBuild Tooling to:• Schedule Deployments• Prefer off peak• Choose next available
region automatically• Provide high visibility
per region
![Page 18: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/18.jpg)
BUILD
DEPLOY
BAKE
![Page 19: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/19.jpg)
Simian Army• Chaos Monkey• Latency Monkey• Janitor Monkey• Conformity Monkey (and more!)
Test resiliency at runtime
http://www.infoq.com/presentations/netflix-resiliency-failure-cloud
![Page 20: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/20.jpg)
Send in the Conformity MonkeyHave deployments diverged? Balance regional consistency
with regional isolation Provide meaningful thresholds Build best practices into
tooling and reporting
![Page 21: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/21.jpg)
Clean up with the Janitor Monkey• Disassociate unused EIPs• Delete unassociated
Amazon EBS volumes• Delete older Amazon
EBS snapshots• Leverage Amazon S3
Object Expiration
![Page 22: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/22.jpg)
Key Elements for Netflix• Value Self-service • Test Everywhere• Build Awareness of Multiple Regions• Avoid peak times• Roll back quickly and easily• Be Cloud Native
![Page 23: Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source](https://reader034.vdocuments.mx/reader034/viewer/2022042713/53fe28e48d7f72d92d8b460b/html5/thumbnails/23.jpg)
Netflix OSS
More at netflix.github.com