© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
AWS CloudFormation Under the Hood
Adam Thomas, Amazon Web Services
DJ Edwards, Amazon Web Services
November 14, 2013
So, what is CloudFormation?
This talk will not answer that question
• DMG201 - Zero to Sixty: AWS
CloudFormation – Has already happened, but will be available online
• Hands-on Labs – Working with CloudFormation
– Launching and Managing a Web Application with
CloudFormation
– Creating an Amazon Virtual Private Cloud (VPC) with
CloudFormation
This talk will answer these questions:
• What is a custom resource?
• What can they do for me?
• How do I write one for myself?
• What’s new in cfn-init?
Custom Resources
What can custom resources do?
• Add New Resources
• Interact with the CloudFormation Workflow
• Inject dynamic data into a stack
• Extend the capabilities of existing resources
What is a custom resource?
• An SNS topic…
• …hooked up to a service that can: – Respond to JSON messages from CloudFormation
– Manage the lifecycle of resources
How are custom resources defined? "myCustomResource" : {
"Type": "Custom::MyCustomResource",
"Version" : "1.0",
"Properties" : {
"ServiceToken": "arn:aws:sns:us-east-1:84969EXAMPLE:CRTest",
"CustomProperty" : "foo"
}
}
How are custom resources defined? "myCustomResource" : {
"Type": “Custom::MyCustomResource",
"Version" : "1.0",
"Properties" : {
"ServiceToken": "arn:aws:sns:us-east-1:84969EXAMPLE:CRTest",
"CustomProperty" : "foo"
}
}
How are custom resources defined? "myCustomResource" : {
"Type": “Custom::MyCustomResource",
"Version" : "1.0",
"Properties" : {
"ServiceToken": "arn:aws:sns:us-east-1:84969EXAMPLE:CRTest",
"CustomProperty" : "foo"
}
}
What can custom resources do?
• Add New Resources
• Interact with the CloudFormation Workflow
• Inject dynamic data into a stack
• Extend the capabilities of existing resources
Adding New Resources
• Something that can be Created, Updated,
and/or Deleted
• Can be a software resource – Database schema, Docker container
Meet Steve
• Steve loves RDBMS
• The schema is very
important to Steve – it
defines his application
• Running SQL scripts by
hand is Steve’s worst
nightmare
Steve’s requirements
• The Template should define the schema explicitly
• The schema should be updated by updating the stack
• If the update fails, the schema should roll back
Steve’s solution
• Steve is very familiar with
Liquibase
• Liquibase supports JSON
formatting!
• Steve writes a custom
resource with inline JSON
schema
DB Schema Template Snippet "appSchema" : {
"Type" : "Custom::DatabaseSchema",
"Properties" : {
"databaseChangeLog" : [{ "changeSet" : {
"id" : "1",
"author" : “adam",
"changes" :
[{ "createTable" : {
"tableName" : "person",
"columns" : …
DB Schema Template Snippet "appSchema" : {
"Type" : "Custom::DatabaseSchema",
"Properties" : {
"databaseChangeLog" : [{ "changeSet" : {
"id" : "1",
"author" : “adam",
"changes" :
[{ "createTable" : {
"tableName" : "person",
"columns" : …
DB Schema Demo
What can custom resources do?
• Add New Resources
• Interact with the CloudFormation Workflow
• Inject dynamic data into a stack
• Extend the capabilities of existing resources
Interacting with the CloudFormation
Workflow • Use custom resources as a hook into
create/update/delete workflows
• Built-in example: WaitCondition
• Can react to workflow, halt it, or fail it under
certain conditions
Meet Frank
• Frank analyzes data
stored on EBS
• Frank uses
CloudFormation’s
Snapshot on Delete
feature to save his
analysis results
Frank’s requirements
• Frank wants a consistent EBS snapshot when the stack is deleted
• Before CloudFormation attempts to detach his EBS volume, it should: – Cleanly shut down his
analysis service
– Unmount the volume
Why is this a challenge?
• CloudFormation can detach volumes without
any issues – if you never mount them
• What CloudFormation does not do, it cannot
undo
• Custom resources let you model your steps
within the workflow
Frank’s solution
• 3 simple bash scripts
• A “local” Custom
Resource – runs directly
on the instance
• Create and Update mount
the drive; Delete
unmounts it.
Volume Mount Template Snippet “VolumeAttach" : {
"Type" : "AWS::EC2::VolumeAttachment",
"Properties" : …
},
"VolumeMount" : {
"Type" : "Custom::VolumeMount",
"Properties" : {
"Device" : “/dev/xvdh”,
“MountPoint” : “/mnt/analysis”
}
}
Volume Mount Template Snippet “VolumeAttach" : {
"Type" : "AWS::EC2::VolumeAttachment",
"Properties" : …
},
"VolumeMount" : {
"Type" : "Custom::VolumeMount",
"Properties" : {
"Device" : “/dev/xvdh”,
“MountPoint” : “/mnt/analysis”
}
}
Volume Mount Template Snippet “VolumeAttach" : {
"Type" : "AWS::EC2::VolumeAttachment",
"Properties" : …
},
"VolumeMount" : {
"Type" : "Custom::VolumeMount",
"Properties" : {
"Device" : “/dev/xvdh”,
“MountPoint” : “/mnt/analysis”
}
}
Volume Mount Template Snippet “VolumeAttach" : {
"Type" : "AWS::EC2::VolumeAttachment",
"Properties" : …
},
"VolumeMount" : {
"Type" : "Custom::VolumeMount",
"Properties" : {
"Device" : “/dev/xvdh”,
“MountPoint” : “/mnt/analysis”
}
}
Volume Mount Demo
What can custom resources do?
• Add New Resources
• Interact with the CloudFormation Workflow
• Inject dynamic data into a stack
• Extend the capabilities of existing resources
Injecting Dynamic Data into a Stack
• Parameters are standard route into a stack – Allow free-form user input
– Constrainable, but on a per-stack level
• Mappings are traditionally used to map human-
readable input to static values – AMI IDs, instance type architectures, regional URLs
Injecting Data into a Stack
• Custom resources allow for centralized selection
logic
• Lookups in: – S3
– DynamoDB/RDS
– APIs (EC2.DescribeImages, etc)
– Third Party datastore
Meet Bill
• Bill is the head of operations
at a large tech firm
• Each of Bill’s 44 services
must run on a fully validated
and tested AMI
• Bill keeps track of these
AMIs in a sweet multi-
tabbed Excel spreadsheet
Bill’s requirements
• New AMIs should be rolled out centrally
• Bill does not want to edit the Mappings section of 44 templates for every release
• Bill wants to audit where AMIs are being used
Bill’s solution
• A manifest of named,
approved AMIs stored in a
versioned S3 file
• A simple python script that
looks up the AMI ID by
region and os,
architecture, and version
AMI Lookup Template Snippet "AMILookup": {
"Type": "Custom::AmiLookup",
"Properties": {
"os": "ubuntu",
"version": “13.04",
"arch": "64"
}
},
"WebServer": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId" : { “Ref" : “AMILookup” }
}
}
AMI Lookup Template Snippet "AMILookup": {
"Type": "Custom::AmiLookup",
"Properties": {
"os": "ubuntu",
"version": "13.04",
"arch": "64"
}
},
"WebServer": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId" : { “Ref" : “AMILookup” }
}
}
AMI Lookup Template Snippet "AMILookup": {
"Type": "Custom::AmiLookup",
"Properties": {
"os": "ubuntu",
"version": "13.04",
"arch": "64"
}
},
"WebServer": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId" : { “Ref" : “AMILookup” }
}
}
AMI Lookup Demo
What can custom resources do?
• Add New Resources
• Interact with the CloudFormation Workflow
• Inject dynamic data into a stack
• Extend the capabilities of existing resources
Extending Resource Capabilities
• CloudFormation is concerned only with Create,
Update, and Delete
• Some services, like AutoScaling, have lifecycles
outside of these phases
• No place in template to encapsulate long-
running, resource-based business logic
Meet Tom
• Tom manages a fleet of
virtual desktops in AWS
• Tom uses AutoScaling for
consistent fleet size
• Tom’s users use VNC to
connect to their virtual
desktops
Tom’s requirements
• Servers should be named
using his clever, easy-to-
remember Simpsons
scheme
• Names should be
recycled as machines are
replaced
Tom’s solution
• Python scripts respond to
Auto Scaling notifications
to manage Route53
records
• Names are managed in a
simple DynamoDB table
Auto Scaled DNS Snippet (1 of 2) "DNSProcessor" : {
"Type": "Custom::DNSProcessor",
"Properties": {
"HostedZoneId" : { "Ref" : "HostedZone" },
"DNSPattern" : {"Fn::Join" : [".",[“{{simpsons_name}}", { "Ref" : "AWS::Region" }, “{{hosted_zone_name}}"]] }
}
},
Auto Scaled DNS Snippet (1 of 2) "DNSProcessor" : {
"Type": "Custom::DNSProcessor",
"Properties": {
"HostedZoneId" : { "Ref" : "HostedZone" },
"DNSPattern" : {"Fn::Join" : [".",[“{{simpsons_name}}", { "Ref" : "AWS::Region" }, “{{hosted_zone_name}}"]] }
}
},
Auto Scaled DNS Snippet (2 of 2) "WebServerGroup" : {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties" : {
"NotificationConfiguration" : {
"TopicARN" : { "Fn::GetAtt" : ["DNSProcessor", “Topic"] },
"NotificationTypes" : [ "autoscaling:EC2_INSTANCE_LAUNCH","autoscaling:EC2_INSTANCE_TERMINATE"]
},
"Tags" : [{ "Key" : "ProcessorId",
"Value" : { "Ref" : "DNSProcessor" },
"PropagateAtLaunch" : false }]
}
}
Auto Scaled DNS Snippet (2 of 2) "WebServerGroup" : {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties" : {
"NotificationConfiguration" : {
"TopicARN" : { "Fn::GetAtt" : ["DNSProcessor", “Topic"] },
"NotificationTypes" : [ "autoscaling:EC2_INSTANCE_LAUNCH","autoscaling:EC2_INSTANCE_TERMINATE"]
},
"Tags" : [{ "Key" : "ProcessorId",
"Value" : { "Ref" : "DNSProcessor" },
"PropagateAtLaunch" : false }]
}
}
Auto Scaled DNS Demo
Building Your Own Custom Resource
• Write code to respond to Create, Update, and
Delete events
• Route Custom Resource SNS Topic to an SQS
Queue for maximum fault tolerance
Can you give me a diagram?
Other resources access Custom Resource attributes via GetAtt and Ref
Stack workflow continues
CloudFormation processes JSON message and stores result
Custom Resource creates resource and returns JSON message
CloudFormation sends CREATE notification to Custom Resource
CloudFormation Stack Workflow starts building Custom Resource
How about an architectural overview?
AWS CloudFormation
Custom Resource Topic SQS Queue
Region
Auto scaling Group
Custom Resource
Implementation
Can you add VPC?
AWS CloudFormation
Custom Resource Topic SQS Queue
Region
Corporate Data center
Existing Service VPN Custom Resource
Implementation
What makes for a good resource?
• Good resources are: idempotent – One unique request, n times == one unique response
• Immediately usable when complete
• Can be deleted cleanly from any state
• Represent one standalone piece of functionality – Embedded resources look convenient, but are hard to update
– Elastic Load Balancers embed Policies, which can depend on
each other, yet this is not modeled in the template
You keep telling me it’s simple…
• It’s really simple if you use aws-cfn-resource-bridge
• Cross-platform hook-based daemon
• Simply supply scripts for Create, Update, and Delete
• Open source (Apache 2.0)
• Install or fork from https://github.com/aws/aws-cfn-resource-bridge
Example Code
• All examples from this talk are available at
https://github.com/awslabs/aws-cfn-custom-
resource-examples
• Stealing from others is the easiest way to get
started – And the best way to use CloudFormation!
cfn-init
cfn-init
• Simple library for “getting bits on the box”
• Install packages, download files, start services
• Works on Windows, Linux, and any platform with
Python 2.6, 2.7
{{cfn-init}}
• Fn::Join can be hard to follow
• Many configuration files are largely boilerplate
• Files can process Mustache templates
• Simply add context
Wordpress config "/var/www/html/wordpress/wp-config.php" : {
"content" : { "Fn::Join" : ["", [
"<?php\n",
"define('DB_NAME', '", {"Ref" : "DBName"}, "');\n",
"define('DB_USER', '", {"Ref" : "DBUser"}, "');\n",
"define('DB_PASSWORD', '", {"Ref" : "DBPassword" }, "');\n",
"define('DB_HOST', '", {"Fn::GetAtt" : [“MyDB", "Endpoint.Address"]},"');\n",
"define('DB_CHARSET', 'utf8');\n",
"define('DB_COLLATE', '');\n",
"define('AUTH_KEY', 'f@A17vs{ mO0}:&I,6SB.QzV`E?!`/tN5:~GZX%=@ZA%!_T0-]9>g]4ll6~,6G|R');\n",
Wordpress config "/var/www/html/wordpress/wp-config.php" : {
"content" : { "Fn::Join" : ["", [
"<?php\n",
"define('DB_NAME', '", {"Ref" : "DBName"}, "');\n",
"define('DB_USER', '", {"Ref" : "DBUser"}, "');\n",
"define('DB_PASSWORD', '", {"Ref" : "DBPassword" }, "');\n",
"define('DB_HOST', '", {"Fn::GetAtt" : [“MyDB", "Endpoint.Address"]},"');\n",
"define('DB_CHARSET', 'utf8');\n",
"define('DB_COLLATE', '');\n",
"define('AUTH_KEY', 'f@A17vs{ mO0}:&I,6SB.QzV`E?!`/tN5:~GZX%=@ZA%!_T0-]9>g]4ll6~,6G|R');\n",
Wordpress config "define('SECURE_AUTH_KEY', 'gTFTI|~rYHY)|mlu:Cv7RN]GQ^3ngyUbw;L0o!12]0c-ispR<-yt3qj]xjquz^&9');\n",
"define('LOGGED_IN_KEY', 'Jd:HG9M)1p5t2<v~+R-vd{p-Q*|*RB^&PUI{vIrydAEEiV!{HS{jN:nErCmLv`p}');\n",
"define('NONCE_KEY', '4aMj4KZV;,Gu7(B|qOCve[c5?*J5x1+x93i:Ey6hh/6jXh+V_{V4+hw!qE^d*U,-');\n",
"define('AUTH_SALT', '_Y_&8m)FH)Cns)8}Yb8b88KDSn:p1#p(qBa<~VW&Y1v}P.*9/8S8@P`{mkNxV lC');\n",
"define('SECURE_AUTH_SALT', '%nG3Ag41^Lew5c86,#zbN:yPFs.GA5a)z5*:Oce1>v6uF~D`,.o1pzS)F8[bM9i[');\n",
"define('LOGGED_IN_SALT', '~K<y+Ly+_Ww1~dtq>;rSQ^+{P5/k|=!]k%RXAF-Y@XMY6GSp+wJ5{(|rCzaWjZ%/');\n",
"define('NONCE_SALT', ',Bs_*Y9:b/1Z:apVLHtz35uim|okkA,b|Jt[-&Nla=T{<l_#D?~6Tj-.2.]FonI~');\n",
Wordpress config "define('WPLANG' , '');\n",
"define('WP_DEBUG' , false);\n",
"$table_prefix = 'wp_';\n",
"if ( !defined('ABSPATH') )\n",
" define('ABSPATH', dirname(__FILE__) . '/');\n",
"require_once(ABSPATH . 'wp-settings.php');\n"
]] }
}
{{cfn-init}} Wordpress Config “files” : {
"/var/www/html/wordpress/wp-config.php" : {
“source” : “https://github.com/FAKEPATH/wp-config.mustache”,
“context” : {
“DbEndpoint” : {“Fn::GetAtt” : [“MyDB”, “Endpoint.Address”]},
“DbName” : { “Ref” : “DbName” },
“DbUser” : { “Ref” : “DbUser” },
“DbPassword” : { “Ref” : “DbPassword” }
}
}
}
{{cfn-init}} Wordpress Config “files” : {
"/var/www/html/wordpress/wp-config.php" : {
“source” : “https://github.com/FAKEPATH/wp-config.mustache”,
“context” : {
“DbEndpoint” : {“Fn::GetAtt” : [“MyDB”, “Endpoint.Address”]},
“DbName” : { “Ref” : “DbName” },
“DbUser” : { “Ref” : “DbUser” },
“DbPassword” : { “Ref” : “DbPassword” }
}
}
}
{{cfn-init}} Wordpress Config “files” : {
"/var/www/html/wordpress/wp-config.php" : {
“source” : “https://github.com/FAKEPATH/wp-config.mustache”,
“context” : {
“DbEndpoint” : {“Fn::GetAtt” : [“MyDB”, “Endpoint.Address”]},
“DbName” : { “Ref” : “DbName” },
“DbUser” : { “Ref” : “DbUser” },
“DbPassword” : { “Ref” : “DbPassword” }
}
}
}
Roleplaying
• cfn-init can use roles to download from S3
• Secured files are not just for proprietary code – Non-AWS credentials
– Private service endpoints
– Dynamic code (enabling or disabling features)
Roleplaying Template Snippet “AWS::CloudFormation::Authentication” : {
“roleCreds” : {
“type” : “S3”,
“roleName” : “MyS3Role”
}
}
…
“files” : {
“/etc/secrets.txt” : {
“source” : “https://s3.amazonaws.com/mybucket/secrets.txt”,
“authentication” : “roleCreds”
}
}
Roleplaying Template Snippet “AWS::CloudFormation::Authentication” : {
“roleCreds” : {
“type” : “S3”,
“roleName” : “MyS3Role”
}
}
…
“files” : {
“/etc/secrets.txt” : {
“source” : “https://s3.amazonaws.com/mybucket/secrets.txt”,
“authentication” : “roleCreds”
}
}
Roleplaying Template Snippet “AWS::CloudFormation::Authentication” : {
“roleCreds” : {
“type” : “S3”,
“roleName” : “MyS3Role”
}
}
…
“files” : {
“/etc/secrets.txt” : {
“source” : “https://s3.amazonaws.com/mybucket/secrets.txt”,
“authentication” : “roleCreds”
}
}
cfn-hup
• Not new, but not often used in samples
• Installed in same package as cfn-init
• Available as Linux and Windows service
• Listens for changes to the stack and runs scripts
when they occur – Usually just runs or re-runs cfn-init
Custom Resources vs. cfn-hup
• Custom Resources require an SNS topic, and
usually an SQS queue
• cfn-hup cannot interact with CloudFormation
workflow – Workflow will not wait for cfn-hup
– cfn-hup cannot fail workflow
– cfn-hup cannot inject data into stack
Summary
• Custom Resources let you extend
CloudFormation beyond the existing Resource
Library
• For more than just “things that can be created”
• cfn-init lets you use Mustache and Roles to
create simple, secure configuration
Corner us in the Developer Lounge
Adam Thomas DJ Edwards
Please give us your feedback on this
presentation
As a thank you, we will select prize
winners daily for completed surveys!
DMG303 - AWS CloudFormation Under the Hood