continuous integration and deployment best practices on aws (arc307) | aws re:invent 2013
DESCRIPTION
With AWS, companies now have the ability to develop and run their applications with speed and flexibility like never before. Working with an infrastructure that can be 100 percent API driven enables businesses to use lean methodologies and realize these benefits. This in turn leads to greater success for those who make use of these practices. In this session, we talk about some key concepts and design patterns for continuous deployment and continuous integration, two elements of lean development of applications and infrastructures.TRANSCRIPT
![Page 1: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/1.jpg)
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Continuous Integration and Deployment Best Practices on AWS - ARC307 Leo Zhadanovsky, Senior Solutions Architect, AWS @leozh JP Schneider, DevOps / Internet Jedi, Mozilla Foundation @jdotp November 13th, 2013
![Page 2: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/2.jpg)
Who Am I?
I work for AWS I worked for the DNC 2009-2012
I was embedded in the DevOps Team @ OFA
AWS does not endorse
political candidates
![Page 3: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/3.jpg)
Who Am I?
JP, DevOps for Mozilla Foundation Previous gig DevOps at OFA 2012
Before that, Ops at Threadless @jdotp
Mozilla Foundation does
endorse animated cats
![Page 4: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/4.jpg)
CONTINUOUS INTEGRATION
![Page 5: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/5.jpg)
What is Continuous Integration?
Changes to code automatically deployed to mainline branch • After passing unit and mock tests
Makes changes to code and deployments iterative, not monolithic Bugs are detected quickly Allows rapid development Helps automate deployments
![Page 6: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/6.jpg)
DEVELOPER
![Page 7: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/7.jpg)
SOURCE CODE REPOSITORY
![Page 8: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/8.jpg)
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
![Page 9: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/9.jpg)
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
![Page 10: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/10.jpg)
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
PICK TASKS
![Page 11: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/11.jpg)
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
SUBMIT CODE
![Page 12: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/12.jpg)
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
SCHEDULE BUILD
![Page 13: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/13.jpg)
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
RECURRENT BUILDS
![Page 14: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/14.jpg)
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
CODE FETCH
![Page 15: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/15.jpg)
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
CODE QUALITY TESTS
TEST RESULTS
![Page 16: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/16.jpg)
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
BUILD OUTPUT
![Page 17: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/17.jpg)
SOURCE CODE REPOSITORY
PROJECT MANAGEMENT SERVER
CONTINUOUS INTEGRATION SERVER
DOCS
BINARIES & PACKAGES
DEV FACING NOTIFICATIONS
![Page 18: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/18.jpg)
![Page 19: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/19.jpg)
SOURCE CODE REPOSITORY
DNS
CONTINUOUS INTEGRATION SERVER
PROJECT MANAGEMENT SERVER
BUILDS
![Page 20: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/20.jpg)
PAIN POINTS • UNIT TESTS INCOMPLETE • MOCK TESTS MAINTENANCE • EXPENSIVE TEST ENVIRONMENT • TEST ENVIRONMENT ≠ PRODUCTION • DEPLOYMENT CYCLES
![Page 21: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/21.jpg)
ON-DEMAND
PAY AS YOU GO
ELASTIC
![Page 22: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/22.jpg)
= PROGRAMMABLE
PLATFORM
![Page 23: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/23.jpg)
![Page 24: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/24.jpg)
![Page 25: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/25.jpg)
![Page 26: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/26.jpg)
IF YOU CAN PROGRAM IT YOU CAN AUTOMATE IT
![Page 27: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/27.jpg)
A lot of options…
Configuration Management Systems • Puppet • Chef • Saltstack
Deployment Frameworks • AWS Elastic Beanstalk • AWS OpsWorks • Ansible • Fabric • Capistrano
Infrastructure Management • AWS CloudFormation
![Page 28: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/28.jpg)
Bake an AMI Configure dynamically
Time consuming configuration (startup time)
Static configurations (less change management)
Bootstrapping
![Page 29: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/29.jpg)
Bake an AMI Configure dynamically
Continuous deployment (latest code)
Environment specific (dev-test-prod)
Bootstrapping
![Page 30: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/30.jpg)
Obama for America
awsofa.info
![Page 31: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/31.jpg)
Web-Scale Applications
![Page 32: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/32.jpg)
500k+ IOPS DB Systems
![Page 33: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/33.jpg)
Services API
![Page 34: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/34.jpg)
![Page 35: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/35.jpg)
Typical Charts
![Page 36: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/36.jpg)
![Page 37: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/37.jpg)
How?
![Page 38: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/38.jpg)
Ingredients Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB
![Page 39: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/39.jpg)
Infrastructure, Configuration Management & Monitoring
Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB
![Page 40: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/40.jpg)
Configuration Management: Puppet
In mid-2011, we looked at options for configuration management and chose Puppet We needed to make it scale, and to get it to work with state-less, horizontally scalable infrastructure How did we do this?
![Page 41: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/41.jpg)
Bootstrapping Puppet with CloudInit
CloudInit is built into Ubuntu and Amazon Linux • Allows you to
pass bootstrap parameters in Amazon EC2 user-data field, in YAML format
![Page 42: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/42.jpg)
Bootstrapping Puppet with CloudInit
Don’t store creds in puppet manifests, store them in private Amazon S3 buckets Either pass Amazon S3 creds through CloudInit: Even better – avoid this by using AWS Identity and Access Management (IAM) roles and AWS Unified CLI’s S3 client
![Page 43: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/43.jpg)
Bootstrapping Puppet with CloudInit Built-in puppet support
Use certname with %i for instance id to name the node Puppetmaster must have auto sign turned on • Use security groups and/or NACLs for network-level security
In nodes.pp, use regex to match node names
![Page 44: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/44.jpg)
Puppet Tips
Use a base class to define your standard install
![Page 45: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/45.jpg)
Use runstages Don’t store credentials in puppet, store them in private Amazon S3 buckets • Use AWS IAM to secure the credentials bucket/folders within that bucket
Puppet Tips
![Page 46: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/46.jpg)
Puppet Tips
Use puppet only for configuration files and what makes your apps unique For undifferentiated parts of apps, use Amazon S3 backed RPM/Debian repositories • Can be either public or private repos, depending on your needs
• Amazon S3 Private RPM Repos: http://git.io/YAcsbg • Amazon S3 Private Debian Repos: http://git.io/ecCjWQ
![Page 47: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/47.jpg)
Puppet Tips
By using packages for applications deploys, you can set ensure => latest, and just bump the package in the repo to update Log everything with rsyslog/graylog/loggly/NewRelic/splunk
![Page 48: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/48.jpg)
Scaling the Puppet Masters
Use an Auto Scaling group for puppet masters • Min size => 2, use multiple Availability Zones
Either have them build themselves off of existing puppet masters in the group or off packages stored in Amazon S3 and bootstrapped through user-data Auto-sign must be on
![Page 49: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/49.jpg)
One thing that is difficult to prepare for…
![Page 50: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/50.jpg)
They had this built for the previous 3 months, a on the East Coast.
![Page 51: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/51.jpg)
They had this built for the previous 3 months, a on the East Coast.
We built this part in 9 hours to be safe.
AWS + Puppet +
Netflix Asgard + WAN Optimization Software +
DevOps =
Cross-Continent Fault-Tolerance On-Demand
![Page 52: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/52.jpg)
ARC205 Thursday @ 3:00PM Lando 4303 Deploying the ‘League of Legends’ Data
Pipeline with Chef
![Page 53: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/53.jpg)
Mozilla Foundation
![Page 54: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/54.jpg)
Webmaker.org circa 2012 • Included Apps, non-SOA: Thimble, Popcorn, Goggles • ~20 pushes of new software in 2012 • Operations and Development interacted mostly through bugzilla
tickets for deploys • Hosting in physical datacenter at Mozilla
Webmaker.org circa early 2013 • Deciding to go 12-factor, SOA in app layer • Weekly pushes of Popcorn on train model • Operations and Development interacted mostly through bugzilla
tickets for deploys
![Page 55: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/55.jpg)
April 2013
Webmaker begins rebuilding entire platform SOA, 12-factor in node.js exclusively Moving apps into AWS and DevOps / CI
![Page 56: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/56.jpg)
Since April 2013…. Openbadges, Webmaker combine for: 1339 Pushes
Pushes Per Day to Staging / Prod
Pushes Per Day (Staging and Prod)
![Page 57: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/57.jpg)
Who? • ~30 Paid Developers • Hundreds of Students • Thousands of Contributors • One DevOps / Internet Jedi • Multiple Teams
How? • Puppet, Jenkins, Fabric • Tight feedback loops:
Newrelic, Opsview • Culture Shift
• Staging Envs • Brave devs iterate,
keeping work in-context • Visible Ops • Cross-train developers in
operations
![Page 58: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/58.jpg)
What changed?
![Page 59: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/59.jpg)
1) Know How You Were Doing Before
2) Know What Changed When
3) Know How You Are Now Doing
= The confidence to try more things and try them faster, with minimum viable planning.
![Page 60: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/60.jpg)
Deployment
Pipeline
![Page 61: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/61.jpg)
ARC312 Friday @ 1:30PM Lando 4206 SmugMug’s Zero-Downtime Migration to
AWS
![Page 62: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/62.jpg)
AWS CLOUDFORMATION
STACK-BASED DEPLOYMENT SERVICE
![Page 63: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/63.jpg)
AWS CLOUDFORMATION TEMPLATE
![Page 64: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/64.jpg)
![Page 65: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/65.jpg)
{ "Description" : "Create RDS with username and password", "Resources" : { "MyDB" : { "Type" : "AWS::RDS::DBInstance", "Properties" : { "AllocatedStorage" : "500", "DBInstanceClass" : "db.m1.small", "Engine" : "MySQL", "EngineVersion" : "5.5", "MasterUsername" : "MyName", "MasterUserPassword" : "MyPassword" } } } }
![Page 66: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/66.jpg)
"AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "mysql" : [], "mysql-server" : [], "httpd" : [], "php" : [], "php-mysql" : [] } }, "sources" : { "/var/www/html" : "https://s3.amazonaws.com/my-builds/build-v4.zip" } }
![Page 67: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/67.jpg)
{ "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Type" : "String" } }, }
![Page 68: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/68.jpg)
CLOUDFORMATION TEMPLATE
PROCEDURAL DEFINITION
Create it programmatically
KNOWN CONFIGURATION Store stack configuration in source control
PARAMETER DRIVEN
Dynamic and user-driven templates
COLLABORATION Share templates with ease as just files
![Page 69: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/69.jpg)
![Page 70: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/70.jpg)
APPLICATION VERSIONS
+ INFRASTRUCTURE
VERSIONS
![Page 71: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/71.jpg)
AWS CLOUDFORMATION
TEMPLATE
![Page 72: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/72.jpg)
ARC203 Wednesday @ 4:15 Lando 4303 How Adobe Deploys
Refreshing the Entire Stack Every Time
![Page 73: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/73.jpg)
![Page 74: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/74.jpg)
Mars Rover Landing by the #’s
NASA TV = HD stream, 1080p, ~1 mb/s per viewer Expecting peak of ~1m viewers All playback devices (iOS, Android, Flash, HTML5, blah) Once in a lifetime moment in history (no crashy crashy)
FUN FUN FUN
![Page 75: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/75.jpg)
Mars Rover Landing by the #’s
NASA TV = HD stream, 1080p, ~1 mb/s per viewer Expecting peak of ~1m viewers All playback devices (iOS, Android, Flash, HTML5, blah) Once in a lifetime moment in history (no crashy crashy) NASA says we can’t use their live stream setup It’s 6 days before the landing It’s the same week as the Olympics Available technical resources from JPL: Brett and Khawaja
NO
T FUN
NO
T FUN
NO
T FUN
![Page 76: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/76.jpg)
The sticky wicket:
HD Video stream size = 4mb never changes easy to cache
manifest.f4m size = 4kb New every 4 sec caching difficult
![Page 77: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/77.jpg)
The plan
Design a solution around our limits • Max connections to origin = 6 • Max streams per cache node = 20 • Local Latency = critical • US-WEST-1 capacity reserved for S3 static images
Test the snot out of it Hang on!
![Page 78: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/78.jpg)
![Page 79: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/79.jpg)
ok, so one of them is a rocket scientist..
![Page 80: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/80.jpg)
“The 42 pack”
![Page 81: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/81.jpg)
LOAD TESTING
![Page 82: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/82.jpg)
![Page 83: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/83.jpg)
![Page 84: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/84.jpg)
Benchmarking
![Page 85: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/85.jpg)
US-East Cache Node Performance 25.3 Gbps
![Page 86: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/86.jpg)
Only ~42Mbps Impact on US-East FMS Origin Servers
![Page 87: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/87.jpg)
![Page 88: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/88.jpg)
ARC303 Friday @ 1:30PM Delfino 4003 Unmeltable Infrastructure at Scale:
Using Apache Kafka, Twitter Storm and Elastic Search on AWS
![Page 89: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/89.jpg)
CONTINUOUS DEPLOYMENT
SMALL, FREQUENT CHANGES CONSTANTLY INTEGRATING INTO
PRODUCTION
![Page 90: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/90.jpg)
KEY = ITERATION
![Page 91: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/91.jpg)
ITERATION =
MODIFY THE SYSTEM TO BETTER MEET THE EXPECTATIONS OF
YOUR USERS
![Page 92: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/92.jpg)
11.6s
Mean time between
deployments (weekday)
1,079
Max number of deployments in a
single hour
10,000
Mean number of hosts
simultaneously receiving a deployment
30,000
Max number of hosts
simultaneously receiving a deployment
DEPLOYMENTS AT AMAZON.COM
![Page 93: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/93.jpg)
SOFTWARE DEPLOY
≠ PRODUCT LAUNCH
![Page 94: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/94.jpg)
![Page 95: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/95.jpg)
1.5 BILLION PAGE VIEWS
OCTOBER 2012
$83 MILLION IN TRANSACTIONS 4.2 MILLION ITEMS SOLD
![Page 96: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/96.jpg)
30 DEPLOYS PER DAY 1 DEPLOY EVERY 20 MINUTES
![Page 97: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/97.jpg)
"Production is truly the only place you can validate your code."
![Page 98: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/98.jpg)
![Page 99: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/99.jpg)
![Page 100: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/100.jpg)
![Page 101: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/101.jpg)
![Page 102: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/102.jpg)
AWS OPSWORKS INTEGRATED APPLICATION
MANAGEMENT
DMG 304 Thursday @ 3:00PM Murano 3206 AWS OPSWORKS UNDER THE HOOD
![Page 103: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/103.jpg)
![Page 104: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/104.jpg)
14 BILLION REQUESTS/MONTH 50 000 DATABASE UPDATES / SEC
NO CACHE
![Page 105: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/105.jpg)
DATA-DRIVEN ARCHITECTURES
![Page 106: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/106.jpg)
METRICS @ETSY
![Page 107: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/107.jpg)
METRICS @OBAMA FOR AMERICA
![Page 108: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/108.jpg)
COST-ORIENTED ARCHITECTURES
![Page 109: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/109.jpg)
PHP+APACHE+VARNISH
NGINX+NODEJS
![Page 110: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/110.jpg)
![Page 111: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/111.jpg)
CONTINUOUS INTEGRATION
CONTINUOUS DEPLOYMENT
![Page 112: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/112.jpg)
CONTINUOUS DEPLOYMENT
= CONTINUOUS
EXPERIMENTATION
![Page 113: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/113.jpg)
CONTINUOUS DEPLOYMENT
= CONTINUOUS
IMPROVEMENT
![Page 114: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/114.jpg)
INNOVATE
![Page 115: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/115.jpg)
« Want to increase innovation? Lower the cost of failure »
Joi Ito
![Page 116: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/116.jpg)
SPEED AND AGILITY
Experiment Often
Fail quickly at
a low cost
More Innovation
Experiment Infrequently
Failure is expensive
Less
Innovation
“ON-PREMISES”
![Page 117: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/117.jpg)
AWS re:Invent Pub Crawl
Join the AWS Startup Team this evening at the AWS Pub Crawl When: Wednesday November 13, 5:30pm - 7:30pm Where: Canaletto at The Venetian, 2nd Floor Who Will Be There: Startups, The AWS Startup Team, Startup Launch Companies and AWS re:Invent Hackathon winners
![Page 118: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/118.jpg)
Startup Spotlight Sessions with Dr. Werner Vogels Thurs. Nov 14, Marcello Room 4406
SPOT 203 - Fireside Chats – Startup Founders, 1:30-2:30pm – Eliot Horowitz, CTO of MongoDB – Jeff Lawson, CEO of Twilio – Valentino Volonghi, Chief Architect of AdRoll
SPOT 204 - Fireside Chats – Startup Influencers, 3:00-4:00pm – Albert Wegner, Managing Partner at Union Square Ventures – David Cohen, Founder and CEO of TechStars
SPOT 101 - Startup Launches, 4:15-5:15pm – 5 companies powered by AWS launching at AWS re:Invent 2013
![Page 119: Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013](https://reader033.vdocuments.mx/reader033/viewer/2022042713/540dd9358d7f728d7e8b4b35/html5/thumbnails/119.jpg)
Please give us your feedback on this presentation
As a thank you, we will select prize winners daily for completed surveys!
ARC 307