choose your own aws adventure

╔══════════════════════════════════════════╗ ║ CHOOSE YOUR OWN AWS ADVENTURE ║ ║ ║ ║ Noah Zoschke ║ ║ noah@convox.com ║ ║ @nzoschke ║ ║ ║ ║ 1/28/2016 ║ ╚══════════════════════════════════════════╝

CONVOX

CONVOX DEVOPS TOOLS AND SERVICES

Eliminate complex and boring parts of architecting a system on AWS

Automate the hard parts of operating software on AWS

Integrate the best pieces from AWS and Docker ecosystem

Open source everything to share code and expertise

Support and maintain for app and infrastructure lifecycle

CONVOX OPEN SOURCE PAAS ⟷ IAAS

Racks ⟷ ASG, CF, Dynamo, EC2, ECS, IAM, VPC

Apps ⟷ CF, ECS, ELB

Scale ⟷ ASG, CF, ECS

Environments ⟷ KMS, S3

Builds ⟷ S3, ECR

Logs ⟷ CloudWatch Logs, Kinesis, Lambda

Metrics ⟷ CloudWatch Metrics, Elasticsearch

Notifications ⟷ SNS

CHOOSE YOUR OWN ADVENTURE

BUILD AND PACKAGING DECISION TREE

Build AMIs

What OS? ubuntu / redhat / systems / dockerd

Build packages

What format? RPMs / debs / tgzs

Push code

What tool? Ansible / Chef / CodeDeploy

Build Docker images

What registry? DockerHub / Quay / v1 / v2 / ECR

BUILD AND PACKAGING ADVENTURE: DOCKER IMAGES / PRIVATE REGISTRY

┌─────────────────────────────────────────┐ │ convox API │ │ POST /apps/httpd/build │ ┌──────────────────┐ │ │ │ │ │ ┌─────────────────────────────┐ │ ┌──────────────────┐ │ │ │ │ pull httpd-web:BLASTBUILD │◀───┼───────┤ │ ┌────────────────┐ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │◀───────────┤ ECS Instance │ │ ├─────┼─────▶│ pull rabbitmq │ │ │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ └────────────────┘ │ │ │ ┌─────────────────────────────┐ │ │ │ │ │ │ │tag httpd-rabbit:BQUWNCMIYZG │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ ┌────────────────┐ │ DockerHub │ │ ┌─────────────────────────────┐ │ │ │ │ │ │ or │ │ │ push $REGISTRY_HOST ├────┼──────▶│ │◀───────────┤ ECS Instance │ │Private Registries│ │ └─────────────────────────────┘ │ │ Convox Registry │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ └────────────────┘ │ ├─────┼─────▶│ pull debian │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ ┌────────────────┐ │ │ │ │ build Dockerfile │ │ │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │◀───────────┤ ECS Instance │ │ │ │ ┌─────────────────────────────┐ │ │ │ │ │ │ │ │ │ tag httpd-web:BQUWNCMIYZG │ │ │ │ └────────────────┘ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ └──────────────────┘ │ │ push $REGISTRY_HOST ├────┼──────▶└───────┬──────────┘ │ └─────────────────────────────┘ │ │ ▲ │ ┌─────────────────────────────┐ │ │ │ │ │tag httpd-worker:BQUWNCMIYZG │ │ ▼ │ │ └─────────────────────────────┘ │ ┌──────────┴───────┐ │ ┌─────────────────────────────┐ │ │ │ │ │ ... │ │ │ S3 │ │ └─────────────────────────────┘ │ │ │ │ │ └──────────────────┘ │ │ └─────────────────────────────────────────┘

BUILD AND PACKAGING R&D

Apr: AMIs are way too slow for CD workflow

May: DockerHub requirement doesn’t feel right

May: Running a Docker v1 registry with S3 store works!

Jun: v1 registry is flaky and impossible to debug and fix

Jun: v2 registry introduces lots of new stuff and incompatibilities

Oct: AWS announces ECR at re:Invent. Wait…

Dec: ECR GA

ECR MIGRATION ADVENTURE: CUSTOM RESOURCES

"RegistryRepository": { "Type": "Custom::ECRRepository", "Version": "1.0", "Properties": { "ServiceToken": { "Fn::GetAtt": [ "CustomTopic", "Arn" ] }, "Name": { "Ref": "AWS::StackName" } } },

"CustomTopic": { "Type": "AWS::Lambda::Function", "Properties": { "Code": { "S3Bucket": { "Fn::Join": [ "-", [ "convox", { "Ref": "AWS::Region" } ] ] }, "S3Key": { "Fn::Join": [ "", [ "release/", { "Ref": "Version" }, "/formation.zip" ] ] } }, "Handler": "lambda.external", "MemorySize": "128", "Role": { "Fn::GetAtt": [ "CustomTopicRole", "Arn" ] }, "Runtime": "nodejs", "Timeout": "30" } },

ECR MIGRATION ADVENTURE: PROGRAMMABLE INFRASTRUCTURE

$ convox update

Next deploy creates RegistryRepository Resource

Next deploy sees RegistryRepository Output pushes to it

Keep v1 registry / data around for rollbacks

ECR MIGRATION ADVENTURE: GLOBAL INFRASTRUCTURE

+ "RegionHasRegistry": { + "Fn::Equals": [ + { + "Ref": "AWS::Region" + }, + "us-east-1" + ] + },

ECR MIGRATION ADVENTURE: SUPPORT

watching for new regions

debugging ECR 500s on pushes

adding retry logic

adding error rate metrics

watching metrics around v1 usage

APP LOGS DECISION TREE

Use Amazon

What service? Kinesis / CloudWatch

Use 3rd Party

What vendor? Papertrail / Loggly / Splunk

Build your own pipeline

What system? syslog / logspout / logstash

App interface

stdout/stderr, log files, log drivers

APP LOGS ADVENTURE: AGENT, DOCKER APIS, KINESIS,

LAMBDA

┌──────────────────────────────────────────────────────────┐ ┌──────────────────┐ │ EC2 Instance in ECS Cluster │ │ app1 Kinesis │ │ │ │ ┌────────┐ │ ┌───────────────────────────────────────────┐ │ ┌──────────────┐ ┌──────────────────────────────────┐ │ ┌─┼───▶│shard 1 │ │──┐ │ Lambda w/ EventSourceMapping │ │ │ │ │ │ │ │ │ └────────┘ │ │ │ ┌──────────────────────────────────────┐ │ │ │ │ │ │ │ │ └──────────────────┘ │ │ │function(event, context) { │ │ │ │ app1 │ │ app2 │ │ │ │ │ │ event.records.forEach(function(r) { │ │ │ │ web.1 │ │ worker.1 │ │ │ │ │ │ winston.info(r.kinesis.data) │ │ │ │ │ │ │ │ │ └─┼▶│ }) │──┼────────▶┌───────────────┐ │ │ │ │ │ │ │ │ │ context.done() │ │ │ │ │ └──────────────┘ └──────────────────────────────────┘ │ │ ┌──────────────────┐ │ │} │ │ │ │ │ │ │ │ │ │ app2 Kinesis │ │ │ │ │ │ │ │ │ ┌─────────────────────┘ │ │ │ ┌────────┐ │ │ └──────────────────────────────────────┘ │ │ Syslog Server │ │ ▼ ▼ │ │ │ ┌─▶│shard 1 │ │ │ ┌────────────────────────────────┐ │ │ │ │ ┌────────────┐ ┌────────────┐─────────────┼───┘ │ │ └────────┘ │ │ │function(event, context) { ... }│──┼────────▶│ │ │ │ dockerd │◀─────────────│convox/agent│─────────────┼─────┼─┘ ┌────────┐ │ │ └────────────────────────────────┘ │ │ │ │ └────────────┘ └────────────┘─────────────┼─────┼───▶│shard 2 │ │ │ ┌────────────────────────────────┐ │ │ │ │ ▲ ┌────────────────────────────────────┐ │ │ └────────┘ │────┼─▶│function(event, context) { ... }│───────┼────────▶└───────────────┘ │ │ │GET docker /events (create) │ │ │ . │ │ └────────────────────────────────┘ │ │ ▼ │ GET ENV "Kinesis", "Process"│ │ │ . │ │ │ │ ┌────────────┐ │ GET Docker /logs?follow=1 │ │ │ . │ └───────────────────────────────────────────┘ │ │ ecs-agent │ │ PUT Kinesis /records │ │ │ ┌────────┐ │ │ └────────────┘ └────────────────────────────────────┘ │ │ │shard N │ │ │ │ │ └────────┘ │ └──────────────────────────────────────────────────────────┘ └──────────────────┘

APP LOGS R&D

Apr: Kinesis is rad

Oct: Kinesis + Lambda is really rad. Forward logs to Papertrail.

Nov: What about Kinesis to other systems?

Dec: What about CloudWatch Logs?

Jan: docker awslogs driver is great code

APP LOGS ADVENTURE: DOCKER/DOCKER INTEGRATION

APP LOGS ADVENTURE: INFRA AS JAVASCRIPT IN JAVASCRIPT

CONVOX PHILOSOPHY

Integrate, don’t invent

All-in on Docker Container and Image workflow

All-in on pure AWS solutions (where available)

Stub in missing pieces where not available

Build and exercise automated infrastructure migration paths

Remove custom pieces when no longer needed

noah@convox.com @nzoschke

_ _ _ _ | |_| |__ __ _ _ __ | | _____| | | __| '_ \ / _` | '_ \| |/ / __| | | |_| | | | (_| | | | | <\__ \_| \__|_| |_|\__,_|_| |_|_|\_\___(_)

I’m hiring...