(and AWS Lambda)
& aoepeople!
E-Commerce: Magento
CMS: TYPO3
Portals: ZF, Symfony,…
Mobile Searchperience: ElasticSearch
250+ people world-wide
(in 8 locations)
Global Enterprise Projects
Infrastructure: AWS
CloudFormation Lambda “Immutable”
/i(m)ˈmyo͞odəb(ə)l/
adjective
unchanging over time or unable to be changed.
“disposable” “ephemeral”
Pet Cattle
not disposable disposable
disposable
not disposable
disposable
Static Resources Build VPC Build
Build Bucket
IAM Setup
VPC
Static Resources
Build
Manual Setup
separate
CloudFormation
Stacks use resources
from underlying stacks as input
parameters Lambda functions,
Monitoring,…
Build Bucket
IAM Setup
Environment A (e.g. prod) Environment B (e.g. stage)
VPC VPC
Static Resources Static Resources
Build Build Build Build
Build X Build X+1 Build X Build X+1
Manual Setup
different scopes (build, environment,
account, global…)
Build Bucket
IAM Setup
Environment A (e.g. prod) Environment B (e.g. stage) Environment C (e.g. qa) Environment D (e.g. dev)
IAM Setup
VPC VPC VPC VPC
Static Resources Static Resources Static Resources Static Resources
Build Build Build Build Build Build Build Build
Build X Build X+1 Build X Build X+1 Build X Build X+1 Build X Build X+1
Manual Setup Manual Setup
Build Bucket Access
Private subnets
Public subnets
ElastiCache (Redis)
with replication groups
for cache and sessions
RDS (multi-az) with
DB subnet group
Bastion
server
s3: media
storage*
Route 53: DNS
configuration CloudFront
distribution
SSL
Certificates
Security group for Varnish servers
Security group for Magento servers
Security group for Load Balancer
Static Resources
Build
Auto Scaling group
Auto Scaling group
Elastic Load
Balancer
Auto-
Scaling
Group
Launch
Configurati
on
Scaling
Policy
Build
Auto Scaling group
Auto Scaling group
Elastic Load
Balancer
Auto-
Scaling
Group
Launch
Configurati
on
Scaling
Policy
Auto Scaling group Auto Scaling group
+1
Baking AMIs
“Chef vs. Puppet?”
http://fbrnc.net/blog/2015/11/
how-to-provision-an-ec2-instance
“…Ansible!” “BASH!”
Keep it simple…!
✔ ✔ (✔)
the “last mile”
most underestimated CFN feature!
var r = require('cfn-response'); exports.handler = function (event, context) { […] var res = {}; if (event.RequestType == 'Create') { res.Password = randomPassword(20); } r.send(event, context, r.SUCCESS, res); };
"IndexerDb": { "Type": "AWS::RDS::DBInstance", "Properties": { [...] "MasterUserPassword": {"Fn::GetAtt": ["GenerateDbPassword", "Password"]}, [...] } },
"GenerateDbPassword": { "Type": "Custom::PasswordGenerator", "Properties": { "ServiceToken": {"Ref": "PasswordGeneratorArn"} } },
• launch some ASGs (set DesiredCapacity) • create database passwords • tag all resources (incl. ElastiCache!) • restore database and media files • (one-time) install scripts (db migrations,…) • detect Varnish backends • wait for healthy backends in the ELBs • run infrastructure tests • cache warming • update Route53 records sets • delete old stacks • …
DesiredCapacity =
Number of Instances in current ASG x 1.2
better safe than sorry…
DesiredCapacity =
Number of Instances in current ASG x 1.2
DesiredCapacity =
Number of Instances in current ASG x 1.2
ScalingPolicy takes care…
"CountInstances": { "Type": "Custom::InstanceCounter", "Properties": { "ServiceToken": {"Ref": "InstanceCounter"}, "AutoScalingGroupTags": [ {"Key": "Environment", "Value": "prod"}, {"Key": "Type", "Value": "Frontend"} ], "Min": 1, "Max": 10, "Factor": "1.5" } },
"FrontendAsg": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { [...] "DesiredCapacity": {"Fn::GetAtt": ["CountInstances", "Count"]}, "Tags": [ {"Key": "Environment", "Value": "prod", "PropagateAtLaunch": true}, {"Key": "Type", "Value": “Frontend", "PropagateAtLaunch": true} ] } },
Update Route53 record set
Update Route53 record set
Delete old stacks
"UpdateR53": { "DependsOn": [ "MagentoWaitCondition", "MagentoWorkerWaitCondition", "Elb"], "Type": "Custom::Route53Update", "Properties": { "ServiceToken": {"Ref": "R53Updater"}, "Name": {"Fn::Join": ["", [ "www-", {"Ref": "EnvironmentName"} ]]}, "HostedZoneId": {"Ref": "HostedZoneId"}, "AliasTargetDNSName": {"Fn::GetAtt": ["Elb", "DNSName"]}, "AliasTargetHostedZoneId": {"Fn::GetAtt": ["Elb", "CanonicalHostedZoneNameID"]}, "Comment": "Updated via CloudFormation/Lambda" } },
"DeleteStacks": { "Condition": "DeleteOldStacks", "DependsOn": ["UpdateR53"], "Type": "Custom::StackDeleter", "Properties": { "ServiceToken": {"Ref": "StackDeleter"}, "TagFilter": { "Environment": {"Ref": "EnvironmentName"}, "Type": "Deployment"}, "ExceptStackName": {"Ref": "AWS::StackName"} } }
+1
Baking AMIs
Tools+CI
Have fun filling out 47 form fields! :)
aoepeople/stackformation
command-line tool (Symfony console, uses
AWS SDK for PHP)
integrates nicely into your CI (Jenknis,…)
pre-process
Userdata
Lambda JS “Template”
superset of CFN template (JSON)
Parameters lookup from other stacks, env vars,…
Template
Parameter
Values
create/ update
Stack policies, Tags,…
Stack magento-stage-build-5
Stack magento-stage-build-6
Stack magento-prod-build-6
Stack magento-prod-build-5
CloudFormation Template
merge & pre-process
“CloudFormation+X” Template(s)
+ Dynamic Parameters
+ Stack Policies
+ Behavior
+ Tags …
Blueprint magento-{env:ENVIRONMENT}-build-{env:BUILD}
Blueprint magento-{env:ENVIRONMENT}-setup
Stack magento-stage-setup
Stack magento-prod-setup
blueprints: - stackname: 'magento-{env:BUILD}' template: 'magento.template' stackPolicy: 'policy.json' OnFailure: 'DO_NOTHING' parameters: Build: '{env:BUILD}' KeyPair: '{var:KeyPair}' VPC: '{resource:setupstack:VPC}' Subnet: '{resource:setupstack:Subnet}' InstanceSg: '{resource:setupstack:InstanceSg}' InstanceProfile: '{output:setupstack:InstanceProfile}' BootAmi: 'ami-06116566' tags: Environment: 'prod' Build: '{env:BUILD}'
enforce “immutability” by denying updates!
aoepeople/cfn-vpc
aoepeople/cfn-lambdahelper
aoepeople/cfn-amibaker
via composer
so we can integrate this into our CI pipeline…
•
aoepeople/awsinspector
command-line tool (Symfony console, uses
AWS SDK for PHP)
Domain models for PHP
$repository = new \AwsInspector\Model\Elb\Repository(); $dns = $repository->findElbsByTags([ 'Environment' => 'deploy', 'Build' => 554, 'Type' => 'Frontend’ ])->getFirst()->getDNSName();
> bin/awsinspector.php ec2:ssh -t Environment:prod –c Type –c Build
filter by tag
Please select an instance [0] i-1033ed9b (Type: Frontend; Environment: prod; Build: 477) [1] i-4ff36ec8 (Type: Backend; Environment: prod ; Build: 477) [2] i-5ab4322b (Type: Worker; Environment: prod; Build: 477) [3] i-705ad42f (Type: Worker; Environment: prod; Build: 476) >
• will take jump hosts into account (ProxyCommand)
• auto-detects your local
(encrypted) private keys
• multiplexed ssh connections
• run commands directly
logins, orders,…
deployments, scaling activity,…
JMeter response time, error rate,…
CPU, load, network I/O,…
correlate metrics from
various sources
Grafana ElasticSearch (Service)
time-series database
leftovers from “disposed” resources
…
“Ready for testing”
(WaitCondition)
setup terminate
instances
• Install JMeter
• Download
testcase from
S3
• Signal “Ready”
and wait
• Create Security
allowing access
to all IPs and
attach to UAT’s
ELB
Auto-Scaling Group
of Load Generator Instances
Spin up test
environment (incl. restoring db+media
snapshot from prod)
Run Stress Test
Delete test
environment* Delete self*
1.
2.
3. 4.
Time
timeout 3600 jmeter –n –t testcase.jmx
*via Lambda custom resource
Pushing samples in near real-time
via custom JMeter backend
listener
“Testing Completed”
(WaitCondition)
JMeter test
CloudFormation Stack
CloudFormation
Stack
Grafana ElasticSearch (Service)
CloudWatch
Lambda
find relevant resources for the current deployment (via tags),
collect metrics, and push them to ElasticSearch
CloudFormation Stack
• Install JMeter
• Download
testcase from S3
• Signal “Ready”
and wait
• Create SG
allowing access
to all IPs and
attach to UAT’s
ELB
Auto-Scaling Group of Load Generator Instances
Run Stress Test 2.
Time
timeout 3600 jmeter –n –t testcase.jmx
Delete test
environment* Delete self*
3. 4.
*via Lambda custom resource
Lambda
Grafana ElasticSearch (Service)
CloudWatch find relevant resources for the
current deployment (via tags),
collect metrics and push them to
ElasticSearch
…
Spin up test
environment (incl. restoring db+media
snapshot from prod)
1.
CloudFormation
Stack
setup terminate
instances JMeter test
Pushing samples in near real-time
via custom JMeter backend listener
“Ready for testing”
(WaitCondition)
“Testing Completed”
(WaitCondition)
https://blogs.aws.amazon.com/application-
management/post/Tx38Z5CAM5WWRXW/Faster-Auto-Scaling-in-AWS-
CloudFormation-Stacks-with-Lambda-backed-Custom-Resou
Provisioning
“Ready for baking”
(WaitCondition)
EC2
AMI Baker
(Lambda) Node.js SDK
AWS CLI
“Baking Completed”
(WaitCondition)
shutdown
(-> terminate)
ec2.createImage
aws ec2 wait image-available
AMI Baker
(Lambda) Node.js SDK
ec2.deleteImage
Custom
CloudFormation
Resource
Time
•
Follow me on twitter!
My blog