continuous deployment practices, with production, test and development environments running on aws

Chris Munns, Solutions Architect, Chris Barclay, Senior

Product Manager, and Mike Limcaco, Solutions Architect

Continuous Deployment Practices, with Production, Test

and Development Environments Running on AWS

This talk:

• Not going to spend a lot of time talking about Continuous Integration(CI) and Continuous Deployment(CD) philosophy

• Will spend more time talking about how AWS can help you if practicing CI and CD are your goals

• Examples to get you thinking, but not the only way

• AWS + Open Source solutions

• Continuous Integration

• Continuous Deployment



Techniques and tools to

implement continuous

processes of applying quality

control in general small

pieces of effort, applied

frequently, to improve the

quality of software, and to

reduce the time taken to

deliver it.



Techniques and tools to

improve the process of

software delivery, resulting in

the ability to rapidly, reliably,

and repeatedly push out

enhancements and bug fixes

to customers at low risk and

with minimal manual

overhead.



Getting code from

developers’ brains,

through their fingers,

to production quickly

and efficiently, with

positive results.

Continuous Integration & Deployment on AWS

• Treat infrastructure as code

• Automate the testing/deploy process end to end

• Make sure environments mimic each other as closely as possible

• Use repeatable patterns between environments at a different scale

• Use different cost models where it makes sense

• Simplify and streamline the deploy process

• Let AWS services handle control flows

• Track everything (instance metrics, application metrics, logs)

In today’s infrastructure, everything is code.

From the applications developers are writing, to

your configuration management tools, to things like

CloudFormation templates or scripts that call AWS APIs.

Since Infrastructure is code, let’s treat it like code!

– Not JUST Revision control!

– Make use of bug tracking/ticketing systems

– Peer reviews of changes before they happen

– Establish infrastructure code patterns/designs

– Test infrastructure changes like code changes

Let’s talk about

the journey our

code is going to

take to production

1.Code gets written

2.Code gets tested

3.Code gets deployed

4.Code gets consumed

1.Code gets written – Someone writes code and commits to revision

control system

– Hooks in revision control, system kicks off CI work

2.Code gets tested – Unit tests, integration tests, db tests, smoke tests,

UI tests

– “Light green, trap clean” OR GOTO STEP 1

3.Code gets deployed – Ship out that code

4.Code gets consumed – Customers use it, love it, victory, profit, vacation in

Bora Bora

1.Software ( tools, services, scripts )

2.Infrastructure Environments ( dev, test, prod )

3.Process ( deploy, monitor, alert, track )

1.Software ( tools, services, scripts )

2.Infrastructure Environments ( dev, test, prod )

3.Process ( deploy, monitor, alert, track )

We need tools to help work with all of the above quickly and more efficiently

First stop on our journey: Continuous Integration-ville

• Help prove code quality and function repeatedly with predefined results

• Lots of options; self hosted, open source, closed source, and SaaS

Continuous Integration - Jenkins

“An extendable open source continuous integration server”

• Open Source • Well established and used by many • Has plugins for EC2/SQS/SNS/CloudFormation!

• Supports spot pricing! • Supports the ability to put workers into a

“standby” mode by stopping instead of terminating

• Scales well • Easily add more EC2 instances as workers

• Flexible • Easy to get started

Internal Git CI Server

Pre-commit Hook

Testing Environment Subnet

CI Workers

Test Chef Cookbooks w/ FoodCritic after each Git Commit.

Test Chef Cookbooks w/ FoodCritic after each Git Commit.

Blue dot is good!

Where is our code going?

Infrastructure Environments

A bad thing people do:

“Developers develop locally on their laptops, mostly OS X based, then deploy to

production, which is Ubuntu. Each laptop has a slightly different setup, and we

don’t maintain software versions across the whole team.”

– Dev and prod not in sync

– Dev not in sync with all of dev

– No testing tier between dev and prod


A bad thing people do:

“Developers develop locally on their laptops, mostly OS X based, then deploy to

production, which is Ubuntu. Each laptop has a slightly different setup, and we

don’t maintain software versions across the whole team.”

– Dev and prod not in sync

– Dev not in sync with all of dev

– No testing tier between dev and prod

“it worked fine on my laptop”

Region

Amazon Route 53

Amazon CloudFront

Customer Traffic

Amazon S3

Availability Zone

Availability Zone

Availability Zone

Internet Gateway

VPC Subnet VPC Subnet VPC Subnet VPC Subnet



Instance Instance

Instance

NAT

Bastion/Chef

ELB ELB

RDS DB Instance

RDS DB Instance Standby (Multi-AZ) Instance

AWS CloudFormation

Amazon CloudWatch

Amazon SNS

Potential RDS DB Instance Read

Replica

Instance Instance

Dev Environment VPC Subnet

DEV WEB ELB

Dev Stack Tier 1

Dev Stack Tier 2

Dev MySQL DB Instance DEV APP ELB

Our Development Infrastructure

Region

Amazon Route 53

Amazon CloudFront

Customer Traffic

Amazon S3

Availability Zone

Availability Zone

Availability Zone

Internet Gateway




Instance Instance

Instance

NAT

Bastion/Chef

ELB ELB

RDS DB Instance


AWS CloudFormation

Amazon CloudWatch

Amazon SNS

Potential RDS DB Instance

Read Replica

Instance Instance


DEV WEB ELB

Dev Stack Tier 1

Dev Stack Tier 2



Region

Amazon Route 53

Amazon CloudFront

Customer Traffic

Amazon S3

Availability Zone

Availability Zone

Availability Zone

Internet Gateway




Instance Instance

Instance

NAT

Bastion/Chef

ELB ELB

RDS DB Instance


AWS CloudFormation

Amazon CloudWatch

Amazon SNS


Replica

Instance Instance


DEV WEB ELB

Dev Stack Tier 1

Dev Stack Tier 2




Developers &

Operations


DEV WEB ELB Dev Stack

Tier 1

Dev Stack Tier 2

Dev MySQL DB Instance

DEV APP ELB

VPN TUNNEL

VPN facing VPC Subnet

Internet Gateway

VPN Endpoint

Dev Admin Instance

NAT Instance

Amazon S3

Amazon DynamoDB

Amazon SQS

Amazon CloudFront

Amazon Route 53

Our Development &Test Infrastructure

Developers &

Operations Internal

Git CI Server

Pre-commit Hook


CI Workers



Tier 1

Dev Stack Tier 2


DEV APP ELB

VPN TUNNEL


Internet Gateway

VPN Endpoint

Dev Admin Instance

NAT Instance

Amazon S3

Amazon DynamoDB

Amazon SQS

Amazon CloudFront

Amazon Route 53


• Be prepared to be running multiple environments

– Development

– Testing/QA

– Staging/Pre-prod

– Production

• They should be running as close to the same stack as possible

• Use configuration management and infrastructure orchestration tools

• No one off hosts

• A goal: Go from nothing to fully running instances without human intervention

This all seems like a lot of work,

and potentially costly.

But it doesn’t need to be!

Infrastructure Automation

We want to be able to rapidly stand up environments as we need to.

Sounds like we need some automation tools?

– CloudFormation

– Elastic Beanstalk

– OpsWorks

– Chef

– Puppet

AWS CloudFormation

"WebServer" : { "Type" : "AWS::EC2::Instance", "Properties" : { "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, "64" ]}, "SpotPrice" : { "Ref" : "SpotPrice" }, "InstanceType" : m1.large", "SecurityGroups" : [{ "Ref" : "InstanceSecurityGroup" }], "KeyName" : { "Ref" : "KeyName" }, "UserData": { "Fn::Base64" : { "Fn::Join" : ["", [ "#!/bin/bash -v\n", "yum update -y aws-cfn-bootstrap\n”, "curl -L http://www.opscode.com/chef/install.sh | bash\n", "cd /etc/chef\n", "/usr/bin/wget http://",{ "Ref" : "ChefServerIP" },"/chef/validation.pem\n", "/usr/bin/wget http://",{ "Ref" : "ChefServerIP" },"/chef/client.rb\n", "/bin/chown -R chef:chef /etc/chef\n",


Developers &

Operations Internal

Git CI Server

Pre-commit Hook


CI Workers



Tier 1

Dev Stack Tier 2


DEV APP ELB

VPN TUNNEL


Internet Gateway

VPN Endpoint

Dev Admin Instance

NAT Instance

Amazon S3

Amazon DynamoDB

Amazon SQS

Amazon CloudFront

Amazon Route 53

Our Development/Test Infrastructure

Developers &

Operations Internal

Git CI Server

Pre-commit Hook


CI Workers



Tier 1

Dev Stack Tier 2


DEV APP ELB

VPN TUNNEL


Internet Gateway

VPN Endpoint

Dev Admin Instance

NAT Instance

Amazon S3

Amazon DynamoDB

Amazon SQS

Amazon CloudFront

Amazon Route 53

AWS Elastic Beanstalk & OpsWorks

Elastic Beanstalk:

• Application container framework similar to a PaaS

• Deploy your application into Elastic Beanstalk and it takes care of building a self healing, auto-scaling, multi-AZ infrastructure

• Allows you to turn some of the knobs under the hood to tweak

• Considered one of the easiest places to start with hosting an application on AWS

OpsWorks:

• Build multi-layer application stacks

• Ties in with Chef for a large degree of flexibility and customization

• Makes deploying applications easier

• More flexible than Elastic Beanstalk, but requires a bit more knowledge

Our Development/Test Infrastructure

Developers &

Operations Internal

Git CI Server

Pre-commit Hook


CI Workers



Tier 1

Dev Stack Tier 2


DEV APP ELB

VPN TUNNEL


Internet Gateway

VPN Endpoint

Dev Admin Instance

NAT Instance

Amazon S3

Amazon DynamoDB

Amazon SQS

Amazon CloudFront

Amazon Route 53

Elastic Beanstalk or OpsWorks

Region

Amazon Route 53

Amazon CloudFront

Customer Traffic

Amazon S3

Availability Zone

Availability Zone

Availability Zone

Internet Gateway




Instance Instance

Instance

NAT

Bastion/Chef

ELB ELB

RDS DB Instance


AWS CloudFormation

Amazon CloudWatch

Amazon SNS


Replica

Instance Instance

Elastic Beanstalk or OpsWorks

Imagine you had an infrastructure you could turn on/off on

demand, make use of spare capacity at a lower cost,

and/or make a reservation for capacity based on your

usage needs and save money doing so.

Oh right, on AWS you do.

Using the Right Cost Model – EC2

• On Demand

• Reserved Instance ( RI ) – 40%+ savings

• Spot – 80%+ savings

Each has its place. For development infrastructure, there

are often places for each:

• On Demand – Developer instances started/stopped daily

• Reserved Instances – Code repository, CI master, DBs

• Spot – CI workers, tiers of dev infrastructure that can tolerate going

away for a bit


Developers &

Operations Internal

Git CI Server

Pre-commit Hook


CI Workers



Tier 1

Dev Stack Tier 2


DEV APP ELB

VPN TUNNEL


Internet Gateway

VPN Endpoint

Dev Admin Instance

NAT Instance

Amazon S3

Amazon DynamoDB

Amazon SQS

Amazon CloudFront

Amazon Route 53

RESERVED INSTANCES

SPOT/ON-DEMAND

RRS S3, CloudFront Price Classes,

DynamoDB RC

Now that we know where our code is going,

how is it getting there?

Ship that code!

• How are we going to deploy our code?

– File shipping:

• Just text files

• Binaries

– Package bundling:

• RPMs

• Tarballs

– As an AMI:

• Bundle one of the above into an AMI

• How fast do we need to do this?

• Across how many instances?

• How do we roll back (or forward)?

Yes, you can technically ship your code to AWS in a box. See Import/Export.

File Shipping Deploy Method

• Can be easier to work with than AMI method and package bundling

• Push out the code

– From Git/SVN/staging host

• Rolling restarts for web/application servers

• Leave existing hosts in place

• Have to worry about the cut over period

• Have to worry about feasibility of roll back/forward

• Can do deploy time schema changes (though a bad idea!)

• Have to worry about tracking what version is live for building new hosts

Deploying – Package Building

• Depending on the language/deployment method, you might need to take the

time to package your code.

– RPM

– Deb

– Something else?

• Throw this in as a step after a successful CI run.

Look at using tools like FPM to manage building packages for different

distributions.

AMI Deployment Method

• Code gets bundled into an AMI, we then deploy that AMI

– Pluses

• Very atomic

• New shouldn’t effect older versions

• Can deploy alongside current

• Easy tools to automate

– Cons

• Bit more work involved

• Have to think about where your data is persisting

• Schema updates potentially harder to package in

• Leverage configuration management tools in automation process

A quick aside - Schema updates

Schema changes tied to deployments are a huge blocker to moving fast.

– Hard to undo a change

– Can take a long time on SQL-based databases

Unlink this from code deploys:

– Flag on/off new features that touch the database in new ways

– Don’t make destructive database changes until no code touches that data

• No deletes, alters to live data! Ever!

– When altering existing data, opt instead to create a parallel column, copy data to new

column, then delete old

– Use “shadow queries” to test new functions/data sources for a percentage of users

before turning live to all

AMI Deployment Method - Building


Fully Functional AMI

OS-Only AMI

Partially Configured AMI



OS-Only AMI


Least flexible to maintain



OS-Only AMI


Most amount of post-boot work




OS-Only AMI


Most amount of post-boot work


Try and find a happy medium here

AMI Deployment Method - Deploying

Blue/Green Deploys

– We stand up a duplicate part of our

infrastructure and slowly cut traffic

over to it

• Shift via DNS

• Makes it easy to do testing of new

features

• Makes it easy to roll back

– As we shift more traffic over, let auto-

scaling grow/shrink our instances of

the new or old application

• Shut down the old when no traffic

there

Amazon Route 53

EC2 Instances

ELB

100%

DynamoDB MySQL RDS Instance

ElastiCache Cache Node


Blue/Green Deploys



over to it

• Shift via DNS


features






there

Amazon Route 53

EC2 Instances

ELB

EC2 Instances

ELB

90% 10%




Blue/Green Deploys



over to it

• Shift via DNS


features






there

Amazon Route 53

EC2 Instances

ELB

EC2 Instances

ELB

50% 50%




Blue/Green Deploys



over to it

• Shift via DNS


features






there

Amazon Route 53

EC2 Instances

ELB

EC2 Instances

ELB

0% 100%




Blue/Green Deploys



over to it

• Shift via DNS


features






there

Amazon Route 53

EC2 Instances

ELB

100%




Netflix – Asgard

– Open Source tool

– Released in 2012

– “web-based tool for managing

cloud-based applications and

infrastructure.

– Helps do Blue/Green Deploys

– Capable of much more!

But how do we do all this quickly and

easily many times a day?

We need robots

We need robots

Amazon SWF

Automating the Process with Robots

Amazon Simple Workflow (SWF)

• Orchestration tool across your infrastructure

• Use it as a middle layer to pass messages and setup tasks to be completed

• Break down individual tasks into different workers

• You define logic between workers

• Anything that can be scripted, can be made into a worker task

• Built in retries, timeouts, logging

• Low cost, reliability, and scalability built in

Amazon SWF

Deciders Workers

YOUR CODE = &

Automating the Process with Robots

Amazon Simple Workflow (SWF)

• Orchestration tool across your infrastructure

• Use it as a middle layer to pass messages and setup tasks to be completed

• Break down individual tasks into different workers

• You define logic between workers

• Anything that can be scripted, can be made into a worker task

• Built in retries, timeouts, logging

• Low cost, reliability, and scalability built in

Amazon SWF

Deciders Workers

YOUR ROBOTS = &

Automating the Process

Workers:

• Bundling code into an RPM – WORKER

• Making a new AMI with this RPM – WORKER

• Deploying a new CloudFormation stack with this RPM – WORKER

• Swapping DNS over to our new stack – WORKER

• Copy AMI across to another region for DR – WORKER

• Clean up old AMIs – WORKER

You get the picture.

Workers

YOUR CODE = ROBOTS


Developers &

Operations Internal

Git CI Server

Pre-commit Hook


CI Workers



Tier 1

Dev Stack Tier 2


DEV APP ELB

VPN TUNNEL


Internet Gateway VPN

Endpoint

Dev Admin Instance

NAT Instance

Amazon S3

Amazon DynamoDB

Amazon SQS

Amazon CloudFront

Amazon Route 53

Amazon SWF

3. Deploy RPM to Dev Environment

2. Build an RPM

1. After CI run kicks off SWF

Execution

Decider Determines next step

Our code has arrived at its destination

Our code has arrived at its destination

But what now?

Monitoring/Logging Infrastructure

• Need to know what’s going on

• Spend the time required to do this well

• Share access to these tools with whole team

• Track every single resource that you can

• Alert on services, their availability, response times

• Make use of different cost models for different parts of this stack

• Try to keep log and other monitoring data for as long a possible

– 6 months? 1 year? Multiple years?

Monitoring/Logging Infrastructure

Tools:

• Logging – Logstash

• Check out Kibana!

– Graylog2

– Syslog-ng/rsyslog/syslog

• Metrics – CloudWatch

– Ganglia

– Graphite

• Monitoring – Nagios

– Munin

– Sensu

HOST LEVEL

METRICS

AGGREGATE LEVEL

METRICS

LOG ANALYSIS BUILD METRICS

AWS Marketplace can help AWS Online Software Store • Customer can find, research, buy software

• Simple pricing, aligns with EC2 usage model

• Launch in minutes

• Marketplace billing integrated into your AWS

account

• 600+ products across 23 categories

Learn more at: http://aws.amazon.com/marketplace

Developer Tool Categories Include • Bug Tracking

• Monitoring

• Source Control

• Testing

http://aws.amazon.com/marketplace

http://aws.amazon.com/marketplace

Continuous Integration & Deployment on AWS

• Treat infrastructure as code.

• Automate the testing/deploy process end to end.

• Make sure environments mimic each other as closely as possible.

• Use repeatable patterns between environments at a different scale.

• Use different cost models where it makes sense.

• Simplify and streamline the deploy process.

• Let AWS services handle control flows.

• Track everything (instance metrics, application metrics, logs).

Thanks for listening!

continuous deployment practices, with production, test and development environments running on aws

Technology