aws power tools: advanced aws cloudformation and cli

42
AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Power Tools: Advanced AWS CloudFormation and CLI Alex Corley Solutions Architect

Upload: amazon-web-services

Post on 13-Aug-2015

399 views

Category:

Technology


7 download

TRANSCRIPT

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved.

AWS Power Tools: Advanced AWS CloudFormation and CLI

Alex CorleySolutions Architect

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

“How do I plan my stacks?”

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Organize by layers & environmentsLayers of stacks

Environments

Dev1Dev2

IntegProd

Frontend Services

• Consumer website, seller website, mobile backend

Backend Services

• Search, payments, reviews, recommendations

Shared Services

• CRM DBs, common monitoring /alarms, subnets, security groups

Base Network

• VPCs, Internet gateways, VPNs, NATs

Identity • IAM users, groups, roles

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Apply service-oriented architecture

Food catalog website Ordering website

Customer DB service

Inventory service

Recommendations service

Analytics serviceFulfillment

service

Paymentservice

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Apply service-oriented architecture

Food catalog website Customer DB service

“Outputs” : {

“CustDBEndPoint”:””

}

“Parameters” : {

“CustDBEndPoint”:””

}

Depends on

“Wire”

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Nested stacks

Website1

“Resources” : { “ELB”, “AutoScaling”,

“RDS”}

Website2

“Resources” : { “ELB”, “AutoScaling”, “DynamoDB”}

ELB_AND_AS

“Resources” : { “ELB”, “AutoScaling”}

Website1

“Resources” : { “NestedStack”, “RDS”}

Website2

“Resources” : { “NestedStack”, “DynamoDB”}

for reusability & specialization

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

“How do I ensure error-free stack creation?”

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Validate your templates

ValidateTemplate API action validates:• JSON syntax• Absence of circular dependencies• Template structure

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

*NEW* Use parameter types"Parameters" : { “aVpcId" : { "Type" : "AWS::EC2::VPC::Id" }, “bSubnetIds" : { "Type" : "List<AWS::EC2::Subnet::Id>" }, “cSecurityGroups" : { "Type" : "List<AWS::EC2::SecurityGroup::Id>" }, “dKeyPair" : { "Type" : "AWS::EC2::KeyPair::KeyName" }

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Use parameter constraints "Parameters" : { "SourceCIDRForSSH" : { "Description" : "CIDR block to allow SSH from", "Type" : "String", "MinLength" : "9", "MaxLength" : "18", "AllowedPattern" : "^([0-9]+\\.){3}[0-9]+\\/[0-9]+$"}

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

“How do I deploy and bootstrap software and data?”

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

"MyInstance": { "Type": "AWS::EC2::Instance", "Metadata" : { "AWS::CloudFormation::Init" : { "webapp-config": { "packages" : {}, "sources" : {}, "files" : {}, "groups" : {}, "users" : {}, "commands" : {}, "services" : {}

Use AWS::CloudFormation::Init Declarative Reusable Grouping & ordering Debug-able Updatable Highly secure BIOT™ (Bring in other tools)

ow.ly/DiNCm

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Use AWS::CloudFormation::Init

"YourInstance": { "Metadata": { "AWS::CloudFormation::Authentication": { "S3AccessCreds": { "type": "S3", "roleName": { "Ref" : "InstanceRole"}, "buckets" : ["your-bucket"] } }, "AWS::CloudFormation::Init": {}

Supports role-based auth

Securely download

Choose auth type. IAM Role is recommended

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Use AWS::CloudFormation::Init

"AWS::CloudFormation::Init": {"webapp-config": {

"packages" : {}, "sources" : {}, "files" : {}, "groups" : {}, "users" : {}, "commands" : {}, "services" : {}

Declarative

"sources" : {"/etc/myapp" :

"https://s3.amazonaws.com/mybucket/myapp.tar.gz"

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Use AWS::CloudFormation::InitSupports updates

"packages" : {}, "sources" : {}, "files" : {}, "groups" : {}, "users" : {},"commands" : {}, "services" : {}

1. Update instance metadata in the template

2. UpdateStack

Instance

Metadata

cfn-hup

3. AWS CloudFormation daemon updates configuration

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Use AWS::CloudFormation::Init

"install_chef" : {},

"install_wordpress" : { "commands" : { "01_get_cookbook" : {}, ..., "05_configure_node_run_list" : { "command" : "knife node run_list add -z `knife node list -z` recipe[wordpress]", "cwd" : "/var/chef/chef-repo", "env" : { "HOME" : "/var/chef" }

Flexibility to bring in other tools such as Chef

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Use AWS::CloudFormation::Init"UserData": {

"# Get the latest CloudFormation helper scripts package\n","yum update -y aws-cfn-bootstrap\n",

"# Trigger CloudFormation::Init configuration \n","/opt/aws/bin/cfn-init --stack ", {"Ref": "AWS::StackId"}, " --resource WebServerInstance ", " --region ", {"Ref": "AWS::Region"}, "\n",

"# Signal completion\n","/opt/aws/bin/cfn-signal –e $? --stack ", {"Ref": "AWS::StackId"}, " --resource WebServerInstance ", " --region ", {"Ref": "AWS::Region"}, "\n"

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Use CloudWatch Logs for debugging"install_logs": { "packages" : { ... "awslogs" ... }, "services" : { ... "awslogs" ... } "files": { "/tmp/cwlogs/cfn-logs.conf": {}

file = /var/log/cfn-init.loglog_stream_name = {instance_id}/cfn-init.log

file = /var/log/cfn-hup.loglog_stream_name = {instance_id}/cfn-hup.log

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Use CloudWatch Logs for debugging

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

CloudFormation Demo

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

CloudFormation Template: Web Application

Availability Zone - A

DMZ – 10.0.0.0/24

Web – 10.0.3.0/24

Database – 10.0.6.0/24

Availability Zone - B

DMZ – 10.0.1.0/24

Web – 10.0.4.0/24

Database – 10.0.7.0/24

Availability Zone - C

DMZ – 10.0.2.0/24

Web – 10.0.5.0/24

Database – 10.0.8.0/24

AWS CloudFormatio

n

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

CloudFormation Template – Production, Test, Dev

Production Test Development

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

CloudFormation Template – Multiple Regions

us-east-1 us-west-1 us-west-2

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

CloudFormation Template – Second Environment

Availability Zone - A

DMZ – 10.1.0.0/24

Web – 10.1.3.0/24

Database – 10.1.6.0/24

Availability Zone - B

DMZ – 10.1.1.0/24

Web – 10.1.4.0/24

Database – 10.1.7.0/24

Availability Zone - C

DMZ – 10.1.2.0/24

Web – 10.1.5.0/24

Database – 10.1.8.0/24

AWS CloudFormatio

n

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

CloudFormation Parameters

"Parameters": {"Environment": { "Default": "Development", "Type": "String",

"AllowedValues": [ "Production", "Staging", "Development" ], "Description": "Name of Environment", "ConstraintDescription": "Production or Staging or Development"

}

}

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

CloudFormation Mappings

"Mappings": {

"AWSRegionAvailabilityZones": {

"us-east-1": { "1": "us-east-1a", "2": "us-east-1b", "3": "us-east-1d" },

"us-west-1": { "1": "us-west-1a", "2": "us-west-1b", "3": "us-west-1c" },

"us-west-2": { "1": "us-west-2a", "2": "us-west-2b", "3": "us-west-2c" }

}

}

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

CloudFormation Mappings"Environment2Vpc": {

"Production": {

"VpcCidr": "10.0.0.0/16",

"PublicSubnet1Cidr": "10.0.0.0/24",

"PublicSubnet2Cidr": "10.0.1.0/24",

"PublicSubnet3Cidr": "10.0.2.0/24",

"WebSubnet1Cidr": "10.0.3.0/24",

"WebSubnet2Cidr": "10.0.4.0/24",

"WebSubnet3Cidr": "10.0.5.0/24",

"PrivateSubnet1Cidr": "10.0.6.0/24",

"PrivateSubnet2Cidr": "10.0.7.0/24",

"PrivateSubnet3Cidr": "10.0.8.0/24"

}

}

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

CloudFormation Subnets"PublicSubnet1": {

"Type": "AWS::EC2::Subnet",

"Properties": {

"AvailabilityZone": { "Fn::FindInMap": [ "AWSRegionAvailabilityZones", { "Ref": "AWS::Region" }, "1" ] },

"VpcId": { "Ref": "VPC" },

"CidrBlock": { "Fn::FindInMap": [ "Environment2Vpc", { "Ref": "Environment" }, "PublicSubnet1Cidr" ] },

"Tags": [

{ "Key": "Name", "Value": { "Fn::Join": [ "-", [ { "Ref": "Environment" }, "DMZ" ] ] } },

{ "Key": "Environment", "Value": { "Ref": "Environment" } },

{ "Key": "Application", "Value": { "Ref": "AWS::StackName" } },

{ "Key": "Network", "Value": "DMZ" }

]

}

}

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Using the CLI

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

When Should I Script?• Whenever possible:

– If it can be scripted, script it!– AWS services have API reference guides publicly available.– Scripts and AWS CloudFormation templates can be used to quickly

launch, start, and stop dev and test environments.

AWS CLI

AWS CloudFormation

template

stack

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Key Features of the AWS CLI• It’s a unified tool to manage your AWS services• It can be installed via MSI, a cross-platform bundle, or pip• It’s a Python package• Run aws configure to set access

keys and default region/output format• All commands in the AWS CLI

have a structure similar to:– $ aws <service> <operation>– ex: $ aws ec2 describe-instances

Details for commands can be found online and using the help subcommand.

AWS CLI

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

AWS CLI Configuration Values• The aws configure <subcommand>

operation gives you programmatic access to configuration variables– list provides you with all of the current configuration data.– get returns the value of a single configuration variable.– set sets the value of a single configuration variable.

• Example: aws configure get region prints the currently set region to standard output

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

EC2 Instance Profiles

• Assign an IAM role to an EC2 instance on launch

• EC2 securely delivers credentials to the instance

• EC2 rotates credentials

• AWS SDKs, AWS CLI, and Tools for PowerShell have support for instance profiles

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Handy AWS CLI Command Options:• --output is a useful way to make your returned information easier to read and

parse.• There are three output formats:

JSON Text Table

{ “Places”: [ { “City”: “Seattle”, “State”: “WA” }, { “City”: “Las Vegas”, “State”: “NV” } [}

PLACES Seattle WA PLACES Las Vegas NV

----------------------------| SomeOperationName |+--------------------------+|| Places |||+------------+-----------+||| City | State |||+------------+-----------+||| Seattle | WA |||| Las Vegas | NV |||+------------+-----------+|

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Handy AWS CLI Command Options:

• In the AWS CLI, a profile is a group of configuration values

• Since profiles store access key information, you can use them to manage access to resources; i.e., you can have separate dev and test profiles

• aws configure --profile <profile name> creates a new profile with that name

• Can be combined with the list, get, and set subcommands– To set the default region setting for the “dev2” profile to eu-west-1:

aws configure set region eu-west-1 --profile dev2– Alternatively, that command can be constructed this way:

aws configure set profile.dev2.region eu-west-1

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Handy AWS CLI Command Options:• --query uses JMESPath to filter response data down to include only

the things you want– JMESPath is a query language for JSON– jmespath.org has web-based tutorials to help learn the language

• If a command generates a response, the response body is parsed into JSON

• If a --query option was used, this data is then filtered based on the query and converted into the requested output format (JSON, text, or table)

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Using• --query '<keyname>[*].<key>' will return all values in keyname

with that key:--query 'Capitals[*].City'

Input JSON Query Result

{ "Capitals": [ { "Nation": "USA", "City": "Washington" }, { "Nation": "Japan", "City": "Tokyo" }, { "Nation": "Brazil", "City": "Brasília" } ],}

[ "Washington", "Tokyo", "Brasília"]

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Using Waiters: Polling Instance States• What's the best way to make sure the AWS CLI waits for an EC2 instance to reach

the running state before performing any other operations?

Script it, right?

#!/bin/bashinstance_id=$(aws ec2 run-instances --image-id ami-12345 \ --query 'Reservations[].Instances[].InstanceID' \ --output text)instance_state=$(aws ec2 describe-instances –instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name')while [ "$instance_state" != "running" ]do sleep 1 instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name')done

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Using Waiters: Polling Instance States• What's the best way to make sure the AWS CLI waits for an EC2 instance to

reach the running state before performing any other operations?

Script it, right?

#!/bin/bashinstance_id=$(aws ec2 run-instances --image-id ami-12345 \ --query 'Reservations[].Instances[].InstanceID' \ --output text)instance_state=$(aws ec2 describe-instances –instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name')while [ "$instance_state" != "running" ]do sleep 1 instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name')done

Problems with this?

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Using Waiters: Polling Instance States• Let's rewrite this script using a waiter:

#!/bin/bashinstance_id=$(aws ec2 run-instances --image-id ami-12345 \ --query 'Reservations[].Instances[].InstanceID' \ --output text)aws ec2 wait instance-running --instance-ids $instance_id

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

CLI Demo

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015

Thank You.This presentation will be loaded to SlideShare the week following the Symposium.

http://www.slideshare.net/AmazonWebServices

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015