aws cloudformation under the hood (dmg303) | aws re:invent 2013

Download AWS CloudFormation under the Hood (DMG303) | AWS re:Invent 2013

Post on 15-Jan-2015

5.552 views

Category:

Technology

5 download

Embed Size (px)

DESCRIPTION

You already know that AWS CloudFormation is a powerful tool for provisioning and managing your AWS infrastructure, but did you know that it can also provision and manage resources outside of AWS? Did you know that CloudFormation can fully bootstrap your EC2 instances, securely download data from S3, and even supports Mustache templates? In this session you will go on a deep dive, touring of some of CloudFormation's most advanced features with a member of the team that built the service. Explore custom resources, cfn-init, S3 authentication, and Mustache templates in a series of technical demos with code samples available for download afterwards.

TRANSCRIPT

  • 1. AWS CloudFormation Under the Hood Adam Thomas, Amazon Web Services DJ Edwards, Amazon Web Services November 14, 2013 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. So, what is CloudFormation? 3. This talk will not answer that question DMG201 - Zero to Sixty: AWS CloudFormation Has already happened, but will be available online Hands-on Labs Working with CloudFormation Launching and Managing a Web Application with CloudFormation Creating an Amazon Virtual Private Cloud (VPC) with CloudFormation 4. This talk will answer these questions: What is a custom resource? What can they do for me? How do I write one for myself? Whats new in cfn-init? 5. Custom Resources 6. What can custom resources do? Add New Resources Interact with the CloudFormation Workflow Inject dynamic data into a stack Extend the capabilities of existing resources 7. What is a custom resource? An SNS topic hooked up to a service that can: Respond to JSON messages from CloudFormation Manage the lifecycle of resources 8. How are custom resources defined? "myCustomResource" : { "Type": "Custom::MyCustomResource", "Version" : "1.0", "Properties" : { "ServiceToken": "arn:aws:sns:us-east-1:84969EXAMPLE:CRTest", "CustomProperty" : "foo" } } 9. How are custom resources defined? "myCustomResource" : { "Type": Custom::MyCustomResource", "Version" : "1.0", "Properties" : { "ServiceToken": "arn:aws:sns:us-east-1:84969EXAMPLE:CRTest", "CustomProperty" : "foo" } } 10. How are custom resources defined? "myCustomResource" : { "Type": Custom::MyCustomResource", "Version" : "1.0", "Properties" : { "ServiceToken": "arn:aws:sns:us-east-1:84969EXAMPLE:CRTest", "CustomProperty" : "foo" } } 11. What can custom resources do? Add New Resources Interact with the CloudFormation Workflow Inject dynamic data into a stack Extend the capabilities of existing resources 12. Adding New Resources Something that can be Created, Updated, and/or Deleted Can be a software resource Database schema, Docker container 13. Meet Steve Steve loves RDBMS The schema is very important to Steve it defines his application Running SQL scripts by hand is Steves worst nightmare 14. Steves requirements The Template should define the schema explicitly The schema should be updated by updating the stack If the update fails, the schema should roll back 15. Steves solution Steve is very familiar with Liquibase Liquibase supports JSON formatting! Steve writes a custom resource with inline JSON schema 16. DB Schema Template Snippet "appSchema" : { "Type" : "Custom::DatabaseSchema", "Properties" : { "databaseChangeLog" : [{ "changeSet" : { "id" : "1", "author" : adam", "changes" : [{ "createTable" : { "tableName" : "person", "columns" : 17. DB Schema Template Snippet "appSchema" : { "Type" : "Custom::DatabaseSchema", "Properties" : { "databaseChangeLog" : [{ "changeSet" : { "id" : "1", "author" : adam", "changes" : [{ "createTable" : { "tableName" : "person", "columns" : 18. DB Schema Demo 19. What can custom resources do? Add New Resources Interact with the CloudFormation Workflow Inject dynamic data into a stack Extend the capabilities of existing resources 20. Interacting with the CloudFormation Workflow Use custom resources as a hook into create/update/delete workflows Built-in example: WaitCondition Can react to workflow, halt it, or fail it under certain conditions 21. Meet Frank Frank analyzes data stored on EBS Frank uses CloudFormations Snapshot on Delete feature to save his analysis results 22. Franks requirements Frank wants a consistent EBS snapshot when the stack is deleted Before CloudFormation attempts to detach his EBS volume, it should: Cleanly shut down his analysis service Unmount the volume 23. Why is this a challenge? CloudFormation can detach volumes without any issues if you never mount them What CloudFormation does not do, it cannot undo Custom resources let you model your steps within the workflow 24. Franks solution 3 simple bash scripts A local Custom Resource runs directly on the instance Create and Update mount the drive; Delete unmounts it. 25. Volume Mount Template Snippet VolumeAttach" : { "Type" : "AWS::EC2::VolumeAttachment", "Properties" : },"VolumeMount" : { "Type" : "Custom::VolumeMount", "Properties" : { "Device" : /dev/xvdh, MountPoint : /mnt/analysis } } 26. Volume Mount Template Snippet VolumeAttach" : { "Type" : "AWS::EC2::VolumeAttachment", "Properties" : },"VolumeMount" : { "Type" : "Custom::VolumeMount", "Properties" : { "Device" : /dev/xvdh, MountPoint : /mnt/analysis } } 27. Volume Mount Template Snippet VolumeAttach" : { "Type" : "AWS::EC2::VolumeAttachment", "Properties" : },"VolumeMount" : { "Type" : "Custom::VolumeMount", "Properties" : { "Device" : /dev/xvdh, MountPoint : /mnt/analysis } } 28. Volume Mount Template Snippet VolumeAttach" : { "Type" : "AWS::EC2::VolumeAttachment", "Properties" : },"VolumeMount" : { "Type" : "Custom::VolumeMount", "Properties" : { "Device" : /dev/xvdh, MountPoint : /mnt/analysis } } 29. Volume Mount Demo 30. What can custom resources do? Add New Resources Interact with the CloudFormation Workflow Inject dynamic data into a stack Extend the capabilities of existing resources 31. Injecting Dynamic Data into a Stack Parameters are standard route into a stack Allow free-form user input Constrainable, but on a per-stack level Mappings are traditionally used to map humanreadable input to static values AMI IDs, instance type architectures, regional URLs 32. Injecting Data into a Stack Custom resources allow for centralized selection logic Lookups in: S3 DynamoDB/RDS APIs (EC2.DescribeImages, etc) Third Party datastore 33. Meet Bill Bill is the head of operations at a large tech firm Each of Bills 44 services must run on a fully validated and tested AMI Bill keeps track of these AMIs in a sweet multitabbed Excel spreadsheet 34. Bills requirements New AMIs should be rolled out centrally Bill does not want to edit the Mappings section of 44 templates for every release Bill wants to audit where AMIs are being used 35. Bills solution A manifest of named, approved AMIs stored in a versioned S3 file A simple python script that looks up the AMI ID by region and os, architecture, and version 36. AMI Lookup Template Snippet "AMILookup": { "Type": "Custom::AmiLookup", "Properties": { "os": "ubuntu", "version": 13.04", "arch": "64" } }, "WebServer": { "Type": "AWS::EC2::Instance", "Properties": { "ImageId" : { Ref" : AMILookup } } } 37. AMI Lookup Template Snippet "AMILookup": { "Type": "Custom::AmiLookup", "Properties": { "os": "ubuntu", "version": "13.04", "arch": "64" } }, "WebServer": { "Type": "AWS::EC2::Instance", "Properties": { "ImageId" : { Ref" : AMILookup } } } 38. AMI Lookup Template Snippet "AMILookup": { "Type": "Custom::AmiLookup", "Properties": { "os": "ubuntu", "version": "13.04", "arch": "64" } }, "WebServer": { "Type": "AWS::EC2::Instance", "Properties": { "ImageId" : { Ref" : AMILookup } } } 39. AMI Lookup Demo 40. What can custom resources do? Add New Resources Interact with the CloudFormation Workflow Inject dynamic data into a stack Extend the capabilities of existing resources 41. Extending Resource Capabilities CloudFormation is concerned only with Create, Update, and Delete Some services, like AutoScaling, have lifecycles outside of these phases No place in template to encapsulate longrunning, resource-based business logic 42. Meet Tom Tom manages a fleet of virtual desktops in AWS Tom uses AutoScaling for consistent fleet size Toms users use VNC to connect to their virtual desktops 43. Toms requirements Servers should be named using his clever, easy-toremember Simpsons scheme Names should be recycled as machines are replaced 44. Toms solution Python scripts respond to Auto Scaling notifications to manage Route53 records Names are managed in a simple DynamoDB table 45. Auto Scaled DNS Snippet (1 of 2) "DNSProcessor" : { "Type": "Custom::DNSProcessor", "Properties": { "HostedZoneId" : { "Ref" : "HostedZone" }, "DNSPattern" : {"Fn::Join" : [".",[{{simpsons_name}}", { "Ref" : "AWS::Region" }, {{hosted_zone_name}}"]] } } }, 46. Auto Scaled DNS Snippet (1 of 2) "DNSProcessor" : { "Type": "Custom::DNSProcessor", "Properties": { "HostedZoneId" : { "Ref" : "HostedZone" }, "DNSPattern" : {"Fn::Join" : [".",[{{simpsons_name}}", { "Ref" : "AWS::Region" }, {{hosted_zone_name}}"]] } } }, 47. Auto Scaled DNS Snippet (2 of 2) "WebServerGroup" : { "Type" : "AWS::AutoScaling::AutoScalingGroup", "Properties" : { "NotificationConfiguration" : { "TopicARN" : { "Fn::GetAtt" : ["DNSProcessor", Topic"] }, "NotificationTypes" : [ "autoscaling:EC2_INSTANCE_LAUNCH","autoscaling:EC2_INSTANCE_TERMINATE"] }, "Tags" : [{ "Key" : "ProcessorId", "Value" : { "Ref" : "DNSProcessor" }, "PropagateAtLaunch" : false }] } } 48. Auto Scaled DNS Snippet (2 of 2) "WebServerGroup" : { "Type" : "AWS::AutoScaling::AutoScalingGroup", "Properties" : { "NotificationConfiguration" : { "TopicARN" : { "Fn::GetAtt" : ["DNSProcessor", Topic"] }, "NotificationTypes" : [ "autoscaling:EC2_INSTANCE_LAUNCH","autoscaling:EC2_INSTANCE_TERMINATE"] }, "Tags" : [{ "Key" : "ProcessorId", "Value" : { "Ref" : "DNSProcessor" }, "PropagateAtLaunch" : false }] } } 49. Auto Scaled DNS Demo 50. Building Your Own Custom Resource Write code to respond to Create, Update, and Delete events Route Custom Resource SNS Topic to an SQS Queue for maximum fault tolerance 51. Can you give me a diagram? CloudFormation Stack Workflow starts building Custom ResourceCloudFormation sends CREATE notification to Custom Resource Custom Resource creates resource and returns JSON message CloudFormation processes JSON message and stores result Stack workflow continues O