AWS London Loft: CloudFormation Workshop

AWS London Loft: CloudFormation WorkshopTemplated AWS Resources

Tom Maddox

Solutions Architect

tmaddox@amazon.co.uk

Who am I?

Gardener (Capacity Planning)

Motorcyclist (Agility)

Mobile App Writer

Problem Solver

Technology Geek

Solutions Architect

What will we cover?

Introduction to CloudFormation

Template anatomy

Bootstrapping instances

Managing stacks

Best Practices

Demo

Managing complex cloud

architectures

Modern cloud architectures have lots of bits

Modern cloud architectures have lots of bits

S3 buckets

Auto Scaling groups

EC2 instances

CloudFront distributions

Elastic Load Balancing

RDS databases

Elasticache clusters

Security groups

SNS topics & subscriptions

EBS volumes

CloudWatch alarms

Elastic IPs

VPCs

Route tables

IAM users, roles, and policies

Route 53 hosted zones

Use the console?

Time consuming

Error prone

Not repeatableNot (easily) auditable

S3 buckets

Auto Scaling groups

EC2 instances

CloudFront distributions

Elastic Load Balancing

RDS databases

Elasticache clusters

Security groups

SNS topics & subscriptions

EBS volumes

CloudWatch alarms

Elastic IPs

VPCs

Route tables

IAM users, roles, and policies

Route 53 hosted zones

Write a script?

What happens if an API call fails?

How do I roll back?

How can I update my environment?How long should I pause for?

S3 buckets

Auto Scaling groups

EC2 instances

CloudFront distributions

Elastic Load Balancing

RDS databases

Elasticache clusters

Security groups

SNS topics & subscriptions

EBS volumes

CloudWatch alarms

Elastic IPs

VPCs

Route tables

IAM users, roles, and policies

Route 53 hosted zones

Use CloudFormation?

AWS Application Management Services

Elastic Beanstalk OpsWorks CloudFormation EC2

Convenience Control

Higher-level services Do it yourself

CloudFormation Overview

Define & manage your infrastructure as code (JSON templates)

Declarative approach with managed infrastructure dependencies

Self documenting (no more spreadsheets of SG rules)

Intelligent updating of resources and automated rollback capabilities

AWS API interactions taken care for you

Reproducibility

Versioning

Continuous integration for your complete

stack

Version Control Jenkins

Test

Live

Amazon

S3

AWS

CloudFormation

App commit

Infra commit

Pull

Deploy new

template

Deploy

new app

Continuous integration for your complete

stack

Version Control Jenkins

Test

Live

Amazon

S3

AWS

CloudFormation

App commit

Infra commit

Pull

Promote new template

Promote

new app

Template anatomy

Template structure

"Parameters"

"Mappings"

"Conditions"

"Resources"

"Outputs"

Resources

Parameters

"Parameters" : {

InstanceType : {

Description : The EC2 Instance Type to launch.,

Type : String,

AllowedValues : [t1.micro, m1.small, m1.medium]

}

},

InstanceType : { Ref : InstanceType }

Outputs

"Outputs" : {

"InstancePublicDnsName" : {

"Description" : "The public DNS name of the newly created EC2 instance",

"Value" : { Fn::GetAtt" : [ "Ec2Instance, PublicDnsName ] }

}

}

Conditions

"Environment" : {

"Description" : "Specifies if this a Dev QA or Prod Environment",

"Type" : "String",

"Default" : "Dev",

"AllowedValues" : [ "Dev", "QA", "Prod"]

},

"Conditions" : {

"ProdEnvironment" : { "Fn::Equals" : [ { "Ref" : "Environment" }, "Prod" ]}

},

"InstanceType" : { "Fn::If" : [ "ProdEnvironment", m3.2xlarge, m3.medium ] }

Mappings

"Mappings" : {

"RegionMap" : {

"us-east-1" : { "32" : "ami-6411e20d", "64" : "ami-7a11e213" },

"us-west-1" : { "32" : "ami-c9c7978c", "64" : "ami-cfc7978a" },

"eu-west-1" : { "32" : "ami-37c2f643", "64" : "ami-31c2f645" },

"ap-southeast-1" : { "32" : "ami-66f28c34", "64" : "ami-60f28c32" },

"ap-northeast-1" : { "32" : "ami-9c03a89d", "64" : "ami-a003a8a1" }

}

},

"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "32"]},

Helpful Links

http://aws.amazon.com/cloudformation/aws-cloudformation-templates/

Search: AWS CloudFormation Templates

Helpful solution-based templates (eg: Active Directory domain forest, or LAMP stack)

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/CHAP_TemplateQuickRef.html

Search: AWS CloudFormation Snippets

Handy snippets for common AWS services (eg: Autoscaling group)

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-reference.html

- Search: AWS CloudFormation Template Reference

- API docs for each CloudFormation resource type

http://aws.amazon.com/cloudformation/aws-cloudformation-templates/http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/CHAP_TemplateQuickRef.htmlhttp://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-reference.html

Demo!

Bootstrapping instances

Use AWS::CloudFormation::Init

Declarative Specify the end state, not how to get there

Debugable cfn-init.log can integrate into CloudWatch Logs

Secure Supports IAM roles

Updatable - Supported as part of the stack update process

Bring your own tools Use cfn-init to bootstrap Chef, Puppet etc.

Use AWS::CloudFormation::Init

"Metadata" : {

"AWS::CloudFormation::Init" : {

"webapp-config": {

"packages" : {},

"sources" : {},

"files" : {},

"groups" : {},

"users" : {},

"commands" : {},

"services" : {}

}

}

},

Using CloudFormation::Init to update

instances

"packages" : {},

"sources" : {},

"files" : {},

"groups" : {},

"users" : {},

"commands" : {},

"services" : {}

1. Update instance metadata in the template

2. UpdateStack

Instance

Metadata

cfn-hup

3. AWS CloudFormation daemon updates configuration

Managing your stacks

Organize by layers & environments

Frontend Services

E-Commerce Website

Backend Services

Search, Payment Gateway, Reviews, Recommendations

Shared Services

Common Monitoring tools, Queues

BaseNetwork

VPCs, Subnets, VPNs, NATs

Identity IAM Users, Groups, Roles

Apply SOA principles

"Parameters" : {

RecommendationsEndPoint : {

Description : URL of the recommendations ELB,

Type : String

}

},

"Outputs" : {

"RecommendationsEndPoint" : {

"Description" : "URL of the recommendations ELB",

"Value" : { Fn::GetAtt" : [ "RecommendationsELB, PublicDnsName ] }

}

}

E-Commerce Website Recommendations Engine

Depends On

Wire

Nested stacks for reusability

ELB_AND_AS

Resources : {

ELB,

AutoScaling

}

Website1

Resources : {

NestedStack,

RDS

}

Website2

Resources : {

NestedStack,

DynamoDB

}

Website1

Resources : {

ELB,

AutoScaling,

RDS

}

Website2

Resources : {

ELB,

AutoScaling,

DynamoDB

}

Deploying Applications

Choose a deployment style

In-place Blue-Green

Stacks

Templates

Amazon

Route 53

Use AutoScalingRollingUpdate for zero downtime

"UpdatePolicy" : {

"AutoScalingRollingUpdate" : {

"MaxBatchSize" : 2,

"MinInstancesInService" : 2,

"PauseTime" : PT20M

}

}

Use

AutoScalingRollingUpdate

for zero downtime

Developer deploys new template

containing change to Launch

Configuration

Auto Scaling group

CloudFormation Stack

CloudFormation replaces a single batch of EC2 instances, the others remain in service

ELB health check confirms new instances are healthy (otherwise rollback)

Next batch of EC2 instances are replaced

Optimizing the ASG UpdatePolicy

On test environments you can optimize for speed and replace all instances

at once

Once live, you should update the ASG in batches making sure you dont have

downtime

for example

"UpdatePolicy": {

"AutoScalingRollingUpdate": {

"PauseTime": "PT0S",

"MaxBatchSize": 6",

"MinInstancesInService": "0"

}

}

"UpdatePolicy": {

"AutoScalingRollingUpdate": {

"PauseTime": "PT15S",

"MaxBatchSize": "2",

"MinInstancesInService": "2"

}

}

For a service with an ASG with 6 instances

TEST LIVE

Best Practices

Validate your templates

ValidateTemplate API validates:

JSON syntax

Absence of circular dependencies

Template structure

Use parameter types

"Parameters" : {