aws cloudformation under the hood (dmg303) | aws re:invent 2013

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

AWS CloudFormation Under the Hood

Adam Thomas, Amazon Web Services

DJ Edwards, Amazon Web Services

November 14, 2013

So, what is CloudFormation?

This talk will not answer that question

• DMG201 - Zero to Sixty: AWS

CloudFormation – Has already happened, but will be available online

• Hands-on Labs – Working with CloudFormation

– Launching and Managing a Web Application with

CloudFormation

– Creating an Amazon Virtual Private Cloud (VPC) with

CloudFormation

This talk will answer these questions:

• What is a custom resource?

• What can they do for me?

• How do I write one for myself?

• What’s new in cfn-init?

Custom Resources

What can custom resources do?

• Add New Resources

• Interact with the CloudFormation Workflow

• Inject dynamic data into a stack

• Extend the capabilities of existing resources

What is a custom resource?

• An SNS topic…

• …hooked up to a service that can: – Respond to JSON messages from CloudFormation

– Manage the lifecycle of resources

How are custom resources defined? "myCustomResource" : {

"Type": "Custom::MyCustomResource",

"Version" : "1.0",

"Properties" : {

"ServiceToken": "arn:aws:sns:us-east-1:84969EXAMPLE:CRTest",

"CustomProperty" : "foo"

}

}

How are custom resources defined? "myCustomResource" : {

"Type": “Custom::MyCustomResource",

"Version" : "1.0",

"Properties" : {

"ServiceToken": "arn:aws:sns:us-east-1:84969EXAMPLE:CRTest",

"CustomProperty" : "foo"

}

}

How are custom resources defined? "myCustomResource" : {

"Type": “Custom::MyCustomResource",

"Version" : "1.0",

"Properties" : {

"ServiceToken": "arn:aws:sns:us-east-1:84969EXAMPLE:CRTest",

"CustomProperty" : "foo"

}

}

What can custom resources do?

• Add New Resources

• Interact with the CloudFormation Workflow

• Inject dynamic data into a stack

• Extend the capabilities of existing resources

Adding New Resources

• Something that can be Created, Updated,

and/or Deleted

• Can be a software resource – Database schema, Docker container

Meet Steve

• Steve loves RDBMS

• The schema is very

important to Steve – it

defines his application

• Running SQL scripts by

hand is Steve’s worst

nightmare

Steve’s requirements

• The Template should define the schema explicitly

• The schema should be updated by updating the stack

• If the update fails, the schema should roll back

Steve’s solution

• Steve is very familiar with

Liquibase

• Liquibase supports JSON

formatting!

• Steve writes a custom

resource with inline JSON

schema

DB Schema Template Snippet "appSchema" : {

"Type" : "Custom::DatabaseSchema",

"Properties" : {

"databaseChangeLog" : [{ "changeSet" : {

"id" : "1",

"author" : “adam",

"changes" :

[{ "createTable" : {

"tableName" : "person",

"columns" : …

DB Schema Template Snippet "appSchema" : {

"Type" : "Custom::DatabaseSchema",

"Properties" : {

"databaseChangeLog" : [{ "changeSet" : {

"id" : "1",

"author" : “adam",

"changes" :

[{ "createTable" : {

"tableName" : "person",

"columns" : …

DB Schema Demo

What can custom resources do?

• Add New Resources

• Interact with the CloudFormation Workflow

• Inject dynamic data into a stack

• Extend the capabilities of existing resources

Interacting with the CloudFormation

Workflow • Use custom resources as a hook into

create/update/delete workflows

• Built-in example: WaitCondition

• Can react to workflow, halt it, or fail it under

certain conditions

Meet Frank

• Frank analyzes data

stored on EBS

• Frank uses

CloudFormation’s

Snapshot on Delete

feature to save his

analysis results

Frank’s requirements

• Frank wants a consistent EBS snapshot when the stack is deleted

• Before CloudFormation attempts to detach his EBS volume, it should: – Cleanly shut down his

analysis service

– Unmount the volume

Why is this a challenge?

• CloudFormation can detach volumes without

any issues – if you never mount them

• What CloudFormation does not do, it cannot

undo

• Custom resources let you model your steps

within the workflow

Frank’s solution

• 3 simple bash scripts

• A “local” Custom

Resource – runs directly

on the instance

• Create and Update mount

the drive; Delete

unmounts it.

Volume Mount Template Snippet “VolumeAttach" : {

"Type" : "AWS::EC2::VolumeAttachment",

"Properties" : …

},

"VolumeMount" : {

"Type" : "Custom::VolumeMount",

"Properties" : {

"Device" : “/dev/xvdh”,

“MountPoint” : “/mnt/analysis”

}

}

Volume Mount Template Snippet “VolumeAttach" : {

"Type" : "AWS::EC2::VolumeAttachment",

"Properties" : …

},

"VolumeMount" : {

"Type" : "Custom::VolumeMount",

"Properties" : {

"Device" : “/dev/xvdh”,

“MountPoint” : “/mnt/analysis”

}

}

Volume Mount Template Snippet “VolumeAttach" : {

"Type" : "AWS::EC2::VolumeAttachment",

"Properties" : …

},

"VolumeMount" : {

"Type" : "Custom::VolumeMount",

"Properties" : {

"Device" : “/dev/xvdh”,

“MountPoint” : “/mnt/analysis”

}

}

Volume Mount Template Snippet “VolumeAttach" : {

"Type" : "AWS::EC2::VolumeAttachment",

"Properties" : …

},

"VolumeMount" : {

"Type" : "Custom::VolumeMount",

"Properties" : {

"Device" : “/dev/xvdh”,

“MountPoint” : “/mnt/analysis”

}

}

Volume Mount Demo

What can custom resources do?

• Add New Resources

• Interact with the CloudFormation Workflow

• Inject dynamic data into a stack

• Extend the capabilities of existing resources

Injecting Dynamic Data into a Stack

• Parameters are standard route into a stack – Allow free-form user input

– Constrainable, but on a per-stack level

• Mappings are traditionally used to map human-

readable input to static values – AMI IDs, instance type architectures, regional URLs

Injecting Data into a Stack

• Custom resources allow for centralized selection

logic

• Lookups in: – S3

– DynamoDB/RDS

– APIs (EC2.DescribeImages, etc)

– Third Party datastore

Meet Bill

• Bill is the head of operations

at a large tech firm

• Each of Bill’s 44 services

must run on a fully validated

and tested AMI

• Bill keeps track of these

AMIs in a sweet multi-

tabbed Excel spreadsheet

Bill’s requirements

• New AMIs should be rolled out centrally

• Bill does not want to edit the Mappings section of 44 templates for every release

• Bill wants to audit where AMIs are being used

Bill’s solution

• A manifest of named,

approved AMIs stored in a

versioned S3 file

• A simple python script that

looks up the AMI ID by

region and os,

architecture, and version

AMI Lookup Template Snippet "AMILookup": {

"Type": "Custom::AmiLookup",

"Properties": {

"os": "ubuntu",

"version": “13.04",

"arch": "64"

}

},

"WebServer": {

"Type": "AWS::EC2::Instance",

"Properties": {

"ImageId" : { “Ref" : “AMILookup” }

}

}

AMI Lookup Template Snippet "AMILookup": {

"Type": "Custom::AmiLookup",

"Properties": {

"os": "ubuntu",

"version": "13.04",

"arch": "64"

}

},

"WebServer": {

"Type": "AWS::EC2::Instance",

"Properties": {

"ImageId" : { “Ref" : “AMILookup” }

}

}

AMI Lookup Template Snippet "AMILookup": {

"Type": "Custom::AmiLookup",

"Properties": {

"os": "ubuntu",

"version": "13.04",

"arch": "64"

}

},

"WebServer": {

"Type": "AWS::EC2::Instance",

"Properties": {

"ImageId" : { “Ref" : “AMILookup” }

}

}

AMI Lookup Demo

What can custom resources do?

• Add New Resources

• Interact with the CloudFormation Workflow

• Inject dynamic data into a stack

• Extend the capabilities of existing resources

Extending Resource Capabilities

• CloudFormation is concerned only with Create,

Update, and Delete

• Some services, like AutoScaling, have lifecycles

outside of these phases

• No place in template to encapsulate long-

running, resource-based business logic

Meet Tom

• Tom manages a fleet of

virtual desktops in AWS

• Tom uses AutoScaling for

consistent fleet size

• Tom’s users use VNC to

connect to their virtual

desktops

Tom’s requirements

• Servers should be named

using his clever, easy-to-

remember Simpsons

scheme

• Names should be

recycled as machines are

replaced

Tom’s solution

• Python scripts respond to

Auto Scaling notifications

to manage Route53

records

• Names are managed in a

simple DynamoDB table

Auto Scaled DNS Snippet (1 of 2) "DNSProcessor" : {

"Type": "Custom::DNSProcessor",

"Properties": {

"HostedZoneId" : { "Ref" : "HostedZone" },

"DNSPattern" : {"Fn::Join" : [".",[“{{simpsons_name}}", { "Ref" : "AWS::Region" }, “{{hosted_zone_name}}"]] }

}

},

Auto Scaled DNS Snippet (1 of 2) "DNSProcessor" : {

"Type": "Custom::DNSProcessor",

"Properties": {

"HostedZoneId" : { "Ref" : "HostedZone" },

"DNSPattern" : {"Fn::Join" : [".",[“{{simpsons_name}}", { "Ref" : "AWS::Region" }, “{{hosted_zone_name}}"]] }

}

},

Auto Scaled DNS Snippet (2 of 2) "WebServerGroup" : {

"Type" : "AWS::AutoScaling::AutoScalingGroup",

"Properties" : {

"NotificationConfiguration" : {

"TopicARN" : { "Fn::GetAtt" : ["DNSProcessor", “Topic"] },

"NotificationTypes" : [ "autoscaling:EC2_INSTANCE_LAUNCH","autoscaling:EC2_INSTANCE_TERMINATE"]

},

"Tags" : [{ "Key" : "ProcessorId",

"Value" : { "Ref" : "DNSProcessor" },

"PropagateAtLaunch" : false }]

}

}

Auto Scaled DNS Snippet (2 of 2) "WebServerGroup" : {

"Type" : "AWS::AutoScaling::AutoScalingGroup",

"Properties" : {

"NotificationConfiguration" : {

"TopicARN" : { "Fn::GetAtt" : ["DNSProcessor", “Topic"] },

"NotificationTypes" : [ "autoscaling:EC2_INSTANCE_LAUNCH","autoscaling:EC2_INSTANCE_TERMINATE"]

},

"Tags" : [{ "Key" : "ProcessorId",

"Value" : { "Ref" : "DNSProcessor" },

"PropagateAtLaunch" : false }]

}

}

Auto Scaled DNS Demo

Building Your Own Custom Resource

• Write code to respond to Create, Update, and

Delete events

• Route Custom Resource SNS Topic to an SQS

Queue for maximum fault tolerance

Can you give me a diagram?

Other resources access Custom Resource attributes via GetAtt and Ref

Stack workflow continues

CloudFormation processes JSON message and stores result

Custom Resource creates resource and returns JSON message

CloudFormation sends CREATE notification to Custom Resource

CloudFormation Stack Workflow starts building Custom Resource

How about an architectural overview?

AWS CloudFormation

Custom Resource Topic SQS Queue

Region

Auto scaling Group

Custom Resource

Implementation

Can you add VPC?

AWS CloudFormation

Custom Resource Topic SQS Queue

Region

Corporate Data center

Existing Service VPN Custom Resource

Implementation

What makes for a good resource?

• Good resources are: idempotent – One unique request, n times == one unique response

• Immediately usable when complete

• Can be deleted cleanly from any state

• Represent one standalone piece of functionality – Embedded resources look convenient, but are hard to update

– Elastic Load Balancers embed Policies, which can depend on

each other, yet this is not modeled in the template

You keep telling me it’s simple…

• It’s really simple if you use aws-cfn-resource-bridge

• Cross-platform hook-based daemon

• Simply supply scripts for Create, Update, and Delete

• Open source (Apache 2.0)

• Install or fork from https://github.com/aws/aws-cfn-resource-bridge

Example Code

• All examples from this talk are available at

https://github.com/awslabs/aws-cfn-custom-

resource-examples

• Stealing from others is the easiest way to get

started – And the best way to use CloudFormation!

cfn-init

cfn-init

• Simple library for “getting bits on the box”

• Install packages, download files, start services

• Works on Windows, Linux, and any platform with

Python 2.6, 2.7

{{cfn-init}}

• Fn::Join can be hard to follow

• Many configuration files are largely boilerplate

• Files can process Mustache templates

• Simply add context

Wordpress config "/var/www/html/wordpress/wp-config.php" : {

"content" : { "Fn::Join" : ["", [

"<?php\n",

"define('DB_NAME', '", {"Ref" : "DBName"}, "');\n",

"define('DB_USER', '", {"Ref" : "DBUser"}, "');\n",

"define('DB_PASSWORD', '", {"Ref" : "DBPassword" }, "');\n",

"define('DB_HOST', '", {"Fn::GetAtt" : [“MyDB", "Endpoint.Address"]},"');\n",

"define('DB_CHARSET', 'utf8');\n",

"define('DB_COLLATE', '');\n",

"define('AUTH_KEY', 'f@A17vs{ mO0}:&I,6SB.QzV`E?!`/tN5:~GZX%=@ZA%!_T0-]9>g]4ll6~,6G|R');\n",

Wordpress config "/var/www/html/wordpress/wp-config.php" : {

"content" : { "Fn::Join" : ["", [

"<?php\n",

"define('DB_NAME', '", {"Ref" : "DBName"}, "');\n",

"define('DB_USER', '", {"Ref" : "DBUser"}, "');\n",

"define('DB_PASSWORD', '", {"Ref" : "DBPassword" }, "');\n",

"define('DB_HOST', '", {"Fn::GetAtt" : [“MyDB", "Endpoint.Address"]},"');\n",

"define('DB_CHARSET', 'utf8');\n",

"define('DB_COLLATE', '');\n",

"define('AUTH_KEY', 'f@A17vs{ mO0}:&I,6SB.QzV`E?!`/tN5:~GZX%=@ZA%!_T0-]9>g]4ll6~,6G|R');\n",

Wordpress config "define('SECURE_AUTH_KEY', 'gTFTI|~rYHY)|mlu:Cv7RN]GQ^3ngyUbw;L0o!12]0c-ispR<-yt3qj]xjquz^&9');\n",

"define('LOGGED_IN_KEY', 'Jd:HG9M)1p5t2<v~+R-vd{p-Q*|*RB^&PUI{vIrydAEEiV!{HS{jN:nErCmLv`p}');\n",

"define('NONCE_KEY', '4aMj4KZV;,Gu7(B|qOCve[c5?*J5x1+x93i:Ey6hh/6jXh+V_{V4+hw!qE^d*U,-');\n",

"define('AUTH_SALT', '_Y_&8m)FH)Cns)8}Yb8b88KDSn:p1#p(qBa<~VW&Y1v}P.*9/8S8@P`{mkNxV lC');\n",

"define('SECURE_AUTH_SALT', '%nG3Ag41^Lew5c86,#zbN:yPFs.GA5a)z5*:Oce1>v6uF~D`,.o1pzS)F8[bM9i[');\n",

"define('LOGGED_IN_SALT', '~K<y+Ly+_Ww1~dtq>;rSQ^+{P5/k|=!]k%RXAF-Y@XMY6GSp+wJ5{(|rCzaWjZ%/');\n",

"define('NONCE_SALT', ',Bs_*Y9:b/1Z:apVLHtz35uim|okkA,b|Jt[-&Nla=T{<l_#D?~6Tj-.2.]FonI~');\n",

Wordpress config "define('WPLANG' , '');\n",

"define('WP_DEBUG' , false);\n",

"$table_prefix = 'wp_';\n",

"if ( !defined('ABSPATH') )\n",

" define('ABSPATH', dirname(__FILE__) . '/');\n",

"require_once(ABSPATH . 'wp-settings.php');\n"

]] }

}

{{cfn-init}} Wordpress Config “files” : {

"/var/www/html/wordpress/wp-config.php" : {

“source” : “https://github.com/FAKEPATH/wp-config.mustache”,

“context” : {

“DbEndpoint” : {“Fn::GetAtt” : [“MyDB”, “Endpoint.Address”]},

“DbName” : { “Ref” : “DbName” },

“DbUser” : { “Ref” : “DbUser” },

“DbPassword” : { “Ref” : “DbPassword” }

}

}

}

{{cfn-init}} Wordpress Config “files” : {

"/var/www/html/wordpress/wp-config.php" : {

“source” : “https://github.com/FAKEPATH/wp-config.mustache”,

“context” : {

“DbEndpoint” : {“Fn::GetAtt” : [“MyDB”, “Endpoint.Address”]},

“DbName” : { “Ref” : “DbName” },

“DbUser” : { “Ref” : “DbUser” },

“DbPassword” : { “Ref” : “DbPassword” }

}

}

}

{{cfn-init}} Wordpress Config “files” : {

"/var/www/html/wordpress/wp-config.php" : {

“source” : “https://github.com/FAKEPATH/wp-config.mustache”,

“context” : {

“DbEndpoint” : {“Fn::GetAtt” : [“MyDB”, “Endpoint.Address”]},

“DbName” : { “Ref” : “DbName” },

“DbUser” : { “Ref” : “DbUser” },

“DbPassword” : { “Ref” : “DbPassword” }

}

}

}

Roleplaying

• cfn-init can use roles to download from S3

• Secured files are not just for proprietary code – Non-AWS credentials

– Private service endpoints

– Dynamic code (enabling or disabling features)

Roleplaying Template Snippet “AWS::CloudFormation::Authentication” : {

“roleCreds” : {

“type” : “S3”,

“roleName” : “MyS3Role”

}

}

…

“files” : {

“/etc/secrets.txt” : {

“source” : “https://s3.amazonaws.com/mybucket/secrets.txt”,

“authentication” : “roleCreds”

}

}

Roleplaying Template Snippet “AWS::CloudFormation::Authentication” : {

“roleCreds” : {

“type” : “S3”,

“roleName” : “MyS3Role”

}

}

…

“files” : {

“/etc/secrets.txt” : {

“source” : “https://s3.amazonaws.com/mybucket/secrets.txt”,

“authentication” : “roleCreds”

}

}

Roleplaying Template Snippet “AWS::CloudFormation::Authentication” : {

“roleCreds” : {

“type” : “S3”,

“roleName” : “MyS3Role”

}

}

…

“files” : {

“/etc/secrets.txt” : {

“source” : “https://s3.amazonaws.com/mybucket/secrets.txt”,

“authentication” : “roleCreds”

}

}

cfn-hup

• Not new, but not often used in samples

• Installed in same package as cfn-init

• Available as Linux and Windows service

• Listens for changes to the stack and runs scripts

when they occur – Usually just runs or re-runs cfn-init

Custom Resources vs. cfn-hup

• Custom Resources require an SNS topic, and

usually an SQS queue

• cfn-hup cannot interact with CloudFormation

workflow – Workflow will not wait for cfn-hup

– cfn-hup cannot fail workflow

– cfn-hup cannot inject data into stack

Summary

• Custom Resources let you extend

CloudFormation beyond the existing Resource

Library

• For more than just “things that can be created”

• cfn-init lets you use Mustache and Roles to

create simple, secure configuration

Corner us in the Developer Lounge

Adam Thomas DJ Edwards

Please give us your feedback on this

presentation

As a thank you, we will select prize

winners daily for completed surveys!

DMG303 - AWS CloudFormation Under the Hood