aws cloudformation best practices at 99designs
DESCRIPTION
How we use AWS CloudFormation at 99designsTRANSCRIPT
Text
CloudFormation at 99designs@lox / @99designs
How we use CFN
Each product team has their own formations in their product git repository.
Some teams use it to provision Docker environments, some use it to provision complete production servers
Increasingly, more teams are using it to provision all of the AWS resources that make up a web stack, and document that process in an OPS readme.
Infrastructure as code, versioned templates side-by-side with code
PrinciplesBuild a hierarchy of loosely-coupled stacks
Group resources that are strongly coupled
Template inputs for things that are dependencies (databases, AMIs, external ELBs, Route53 zones)
Template outputs should make up inputs to other stacks
Design based around things that change at the same time, e.g. split ELBs and auto-scaling app server groups
Principles
Keep it simple
Automate the building of pieces that can be assembled by humans
Invest the most automation in parts of the system that change the most frequently.
JSON is hard to read, keep templates short
Layers on Layers
Lots of tools for managing, transforming and generating CFN templates
https://github.com/cloudtools/troposphere
https://github.com/cotdsa/cumulus
At present we don’t use any of these, but might in future
Best Practice:
Use WaitHandle, WaitCondition and DependsOn
WaitHandles wait for resources to finish provisioning
Enforce ordering with DependsOn
Errors in UserData should signal the WaitHandle immediately. Failing fast means quicker iterating on templates.
"WaitHandle" : { "Type" : "AWS::CloudFormation::WaitConditionHandle"},"WaitCondition" : { "Type" : "AWS::CloudFormation::WaitCondition", "DependsOn" : "AppServers", "Properties" : { "Handle" : { "Ref" : "WaitHandle" }, "Timeout" : "300", "Count" : { "Ref" : "NumberOfAppServers" } }}
"# signal to the waithandle\n","cfn-signal -e $? -r \"Setup complete\" '", { "Ref" : "WaitHandle" }, "'\n"
Best Practice:
Use aws-cfn-bootstrap to keep userdata short
Userdata is the hardest part to read, so keep it short
A single call to cfn-init means consolidated error handling
Configure resources declaratively via Metadata
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ "#!/bin/bash -x\n", "exec &>/home/ubuntu/boot.log\n", "tail -F /var/log/cfn-init.log &\n",
"# install cfn-bootstrap\n", "apt-get -y install python-setuptools\n", "easy_install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n", "cfn-init -v ", " --region ", { "Ref" : "AWS::Region" }, " --resource LaunchConfig", " --stack ", { "Ref" : "AWS::StackName" },"\n",
"# signal to the waithandle\n", "cfn-signal -e $? -r \"Setup complete\" '", { "Ref" : "WaitHandle" }, "'\n"]]}
Best Practice:
Use AWS metadata for init and authentication
Download private setup files from S3
Install packages via yum/apt
Execute ad-hoc commands
Keeps error handling to a minimum, DRY code
"LaunchConfig" : { "Type" : "AWS::AutoScaling::LaunchConfiguration", "Metadata" : { "AWS::CloudFormation::Authentication" : { "S3AccessCreds" : { "type" : "S3", "roleName" : { "Ref" : "IAMRole" }, "buckets" : [ "my-secrets" ] }}, "AWS::CloudFormation::Init" : { "config" : { "sources" : { "/root/provision" : { "Fn::Join" : ["", ["https://s3.amazonaws.com/my-secrets/", { "Ref" : "ScriptsTarball" } ]] }}}}}}
Best Practice:
Security groups for shared access, instance roles for everything else
For inter-template security groups, use named security groups
Using instance roles provides token based credentials to the instance, compatible with aws cli tools
Avoid leaking credentials via templates
"Parameters" : { "RDSAccessSecurityGroup" : { "Description" : "Security group to connect to RDS via", "Type" : "String", "Default" : "rds-access" }, "BeanstalkdAccessSecurityGroup" : { "Description" : "Security group to connect to beanstalkd via", "Type" : "String", "Default" : "beanstalkd-access" }, "RedisAccessSecurityGroup" : { "Description" : "Security group to connect to elasticache/redis", "Type" : "String", "Default" : redis-access" },}
Best Practice:
Bootstrap instances from scripts in private s3 buckets
Rather than checking out a git repo, when you create a stack, link it to a tarball of scripts
Easy to iterate, consistent instance launches for future auto-scale events
"LaunchConfig" : { "Type" : "AWS::AutoScaling::LaunchConfiguration", "Metadata" : { "AWS::CloudFormation::Authentication" : { "S3AccessCreds" : { "type" : "S3", "roleName" : { "Ref" : "IAMRole" }, "buckets" : [ "secret-stuff" ] } }, "AWS::CloudFormation::Init" : { "config" : { "sources" : { "/root/provision" : { "Fn::Join" : ["", ["https://s3.amazonaws.com/secret-stuff/", { "Ref" : "ProvisionSlug" } ]] } }, }}}}
Best Practice:
Automate monitoring
CloudWatch is great, use it
Automate alarms, don’t leave them as an afterthough
Good clouds are ones with monitoring!
"SNSTopic" : { "Type" : "AWS::SNS::Topic", "Properties" : { "Subscription" : [{ "Endpoint" : { "Ref": "AlertEmailAddress" }, "Protocol" : "email" }] }},"CPUAlarmDB" : { "Type" : "AWS::CloudWatch::Alarm", "Properties" : { "AlarmDescription": { "Fn::Join" : ["", [{ "Ref" : "RDSInstance" }, "DB CPU Utilization"]]}, "MetricName": "CPUUtilization", ... "AlarmActions": [ { "Ref": "SNSTopic" } ], "Dimensions": [{ "Name": "DBInstanceIdentifier", "Value": { "Ref": "RDSInstance" } }], "ComparisonOperator": "GreaterThanOrEqualToThreshold" }},
Best Practice:
NoEcho property for passwords
Prevent information leak for sensitive parameters
"DockerPassword": { "NoEcho": "true", "Description" : “Docker index password", "Type": "String", "MinLength": "1", "MaxLength": "41" }
Questions we have
Reuse beyond copy/paste?
Elegant AMI selection using Fn::select?
Update stacks, or destroy/recreate?
CFN for Route53?
Future plans
CloudWatch streaming logs, see http://blogs.aws.amazon.com/application-management/post/TxPYD8JT4CB5UY/View-CloudFormation-Logs-in-the-Console
Nested templates to manage dependent templates
Elastic Beanstalk
Questions?
Resources
https://gist.github.com/lox/66a78542b2c14a3f773d
http://blogs.aws.amazon.com/application-management/blog/tag/CloudFormation
http://harish11g.blogspot.com.au/2014/08/amazon-cloudformation-templates-automation-Amazon-CFT-AWS-top-best-practices-tips.html
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-authentication.html
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html