amazon ecs deep dive

© 2016, Amazon Web Serv ices, Inc. or its Af f iliates. All rights reserv ed.

Konstantin Wilms, Solutions Architect

June 21st, 2016

Amazon EC2 Container

Service Deep Dive

Agenda

Infrastructure Setup

Infrastructure Management

PaaS on ECS

Amazon ECS Infrastructure

Setup

Amazon ECS Cluster Setup

Cluster Setup with AWS CloudFormation

CloudFormation supports ECS cluster, service and task

definition resources

Use AWS::IAM::Role to create ECS service role and

container instances role

Launch container instances using

AWS:AutoScaling::LaunchConfiguation and

AWS:AutoScaling::AutoScalingGroup


"Resources" : {

"ECSCluster": {

"Type": "AWS::ECS::Cluster"

},

"ECSAutoScalingGroup" : {

"Type" : "AWS::AutoScaling::AutoScalingGroup",

"Properties" : {

"VPCZoneIdentifier" : { "Ref" : "SubnetID" },

"LaunchConfigurationName" : { "Ref" : "ContainerInstances" },

"MinSize" : "1",

"MaxSize" : { "Ref" : "MaxSize" },

"DesiredCapacity" : { "Ref" : "DesiredCapacity" }

},

[…]

},


"ContainerInstances": {

"Type": "AWS::AutoScaling::LaunchConfiguration",

"Metadata" : {

"AWS::CloudFormation::Init" : {

"config" : {

"commands" : {

"01_add_instance_to_cluster" : {

"command" : { "Fn::Join": [ "", [ "#!/bin/bash\n", "echo ECS_CLUSTER=", { "Ref": "ECSCluster" }, " >> /etc/ecs/ecs.config" ] ] }

}

},

[…]

}

}

}

Cluster Setup with AWS OpsWorks

One ECS Cluster layer per stack

One cluster can only be associated with one stack

Cluster Setup with AWS OpsWorks

Update OpsWorks IAM role to allow ecs:* actions

Add instances to layer (24/7, time-based, load-based)

Manage security updates, user permission and access

Amazon ECR Setup

Amazon ECR Setup

You have read and write access to the repositories you

create in your default registry, i.e.

<aws_account_id>.dkr.ecr.us-east-1.amazonaws.com

Repository names can support namespaces, e.g. team-

a/web-app.

Repositories can be controlled with both IAM user access

policies and repository policies.

Amazon ECR Setup

# Authenticate Docker to your Amazon ECR registry

> aws ecr get-login

docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us-east-1.amazonaws.com

> docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us-east-1.amazonaws.com

# Create a repository called ecr-demo

> aws ecr create-repository --repository-name ecr-demo

# Push an image to your repository

> docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecr-demo:v1

Amazon ECR Docker Credential Helper

Available today - http://bit.ly/25yXdC3

Place the docker-credential-ecr-login binary on your PATH

Set the contents of ~/.docker/config.json file to be:

{ "credsStore": "ecr-login" }

Push and pull images from ECR without docker login

http://bit.ly/25yXdC3

Amazon ECS Infrastructure

Management

Amazon EC2 Simple Systems

Manager (SSM)

Amazon EC2 Simple Systems Manager (SSM)

Use Amazon EC2 SSM to execute commands on container

instances, e.g. yum update

• Add AmazonEC2RoleForSSM

to instances IAM role to

process Run Commands

• Install SSM Agent

• Create SSM document –

similar to CloudInit userdata

• Lock down AWS-* documents

Monitoring & Logging

Monitoring with Amazon CloudWatch

Metric data sent to CloudWatch in 1-minute periods and

recorded for a period of two weeks

Available metrics: CPUReservation, MemoryReservation,

CPUUtilization, MemoryUtilization

Available dimensions: ClusterName, ServiceName


Use the Amazon CloudWatch Monitoring Scripts to monitor

additional metrics, e.g. disk space:

# Edit crontab

> crontab -e

# Add command to report disk space utilization to CloudWatch every five minutes

*/5 * * * * <path_to>/mon-put-instance-data.pl --disk-space-util --disk-space-used --disk-space-avail --disk-path=/ --from-cron

Configuring Logging in Task Definition

logConfiguration task definition parameter

Requires version 1.18 or greater of the Docker Remote API

Maps to docker run --log-driver option

Log drivers: json-file, syslog, journald, gelf, fluentd,

awslogs

"logConfiguration": {

"logDriver": "awslogs",

"options": {

"awslogs-group": "awslogs-wordpress",

"awslogs-region": "us-west-2"

}

}

CloudWatch Logging with awslogs

Amazon CloudWatch Logs




Amazon S3

Amazon Kinesis

AWS Lambda

Amazon Elasticsearch Service

Amazon ECS Store

Stream

Process

Search

Logging Amazon ECS API with AWS CloudTrail

{

"eventVersion": "1.03",

"userIdentity": {…},

"eventTime": "2015-10-12T13:57:33Z",

"eventSource": "ecs.amazonaws.com",

"eventName": "CreateCluster",

"awsRegion": "eu-west-1",

"sourceIPAddress": "54.240.197.227",

"userAgent": "console.amazonaws.com",

"requestParameters": {

"clusterName": "ecs-cli"

},

Logging Amazon ECS API with AWS CloudTrail

"responseElements": {

"cluster": {

"clusterArn": "arn:aws:ecs:eu-west-1:560846014933:cluster/ecs-cli",

"pendingTasksCount": 0,

"registeredContainerInstancesCount": 0,

"status": "ACTIVE",

"runningTasksCount": 0,

"clusterName": "ecs-cli",

"activeServicesCount": 0

}

},

[…]

Monitoring Amazon ECS with Datadog

Monitoring Amazon ECS with Sysdig Cloud

Scaling Amazon ECS

Setup ECS Cluster with AutoScaling

Create LaunchConfiguration

• Pick instance type

depending on resource

requirements, e.g. memory

or CPU

• Use latest Amazon Linux

ECS-optimized AMI, other

distros available

Create AutoScaling group and

set to cluster initial size

Auto Scaling your Amazon ECS Cluster

Create CloudWatch alarm

on a metric, e.g.

MemoryReservation

Configure scaling policies to

increase and decrease the

size of your cluster

Auto Scaling your Amazon ECS services

Service Discovery &

Configuration Management

Service Discovery with ECS Services & Route 53

Route 53 private hosted zone

Set search path on hosts with DHCP option sets

Define ECS services with ELB

Create CNAMEs for each ELB

Service Discovery with ECS Services & Route 53

Task

Task TaskTask

ECS

Service

Application

router, e.g. nginx

Internal ELB with

CNAME, e.g. api.example.com

Route 53 private

zone, e.g. example.com

Service Discovery with Weaveworks

DNS interface for cross-host

container communication

Gossip protocol to share

grouped updates

Overlay network between hosts

Service Discovery and Configuration

Management with ConsulThree main components:

• Consul agent - Runs on each node, responsible for checking the health of the services and of the node itself.

• One or more Consul servers - Store and replicate data, leader elected using the Raft consensus algorithm

• Registrator agent - Automatically register/deregisters services based on published ports and metadata from the container environment variables defined in the ECS task definition


Management with Consul

EC

S C

lus

ter

consul-server

ECS Instance

consul-agent

registrator

ECS Instance

Back end 1

Back end 2

consul-agent

registrator

ECS Instance

Front end

EC

S C

lus

ter


Management with etcd

etcd

registrator

ECS Instance

Container 1

Container 2

confd etcd

registrator

ECS Instance

Container 1

Container 2

confd etcd

registrator

ECS Instance

Container 1

Container 2

confd

Security

ECS IAM Policies and Roles

The ECS agent calls the ECS APIs on your behalf, so

container instances require an IAM policy and role that

allows these calls.

The ECS service scheduler calls the EC2 and ELB APIs on

your behalf to register and deregister container instances

with your load balancers.

Use AmazonEC2ContainerServiceforEC2Role and

AmazonEC2ContainerServiceRole managed policies

(respectively)

ECR IAM Policies and Roles

ECR uses resource-based permissions to control access.

By default, only the repository owner has access to a

repository.

You can apply a policy document that allows others to

access your repository.

Use managed policies for IAM users or roles that allow

differing levels of control:

AmazonEC2ContainerRegistryFullAccess,

AmazonEC2ContainerRegistryPowerUser or

AmazonEC2ContainerRegistryReadOnly

Image Vulnerability Scanning with Twistlock

Secrets Management

Task

ECS Cluster

Container instance

PaaS on ECS

AWS Elastic Beanstalk

Uses Amazon ECS to coordinate deployments to

multicontainer Docker environments

Takes care of tasks including cluster creation, task

definition and execution

AWS Elastic Beanstalk

Elastic Beanstalk uses a Dockerrun.aws.json file that

describes how to deploy containers.

The Dockerrun.aws.json file includes three sections:

• AWSEBDockerrunVersion: Set to "2" for multicontainer

Docker environments.

• containerDefinitions: An array of container definitions.

• volumes: Creates mount points in the container instance

that a container can use.

Convox

Convox

# Initialize your app and create default manifest

> convox init

# Locally build and run your app as declared in the manifest

> convox start

# Create app

> convox apps create my_app

# Deploy app, output ELB DNS name

> convox deploy

[...]

web: http://my_app-1234567890.us-east-1.elb.amazonaws.com

Remind Empire

Control layer on top of Amazon ECS that provides a

Heroku like workflow

Any tagged Docker image can be deployed to Empire as

an app

• When you deploy a Docker image to Empire, it will

extract a Procfile from the WORKDIR

• Each process type in the Procfile maps directly to an

ECS Service

Remind Empire

Routing layer backed by internal ELBs

• An application that specifies a web process will get an

internal ELB attached to its ECS Service

• When a new internal ELB is created, an associated

CNAME record is created in Route53 under the internal

TLD, enabling service discovery via DNS

Thank You!

amazon ecs deep dive

Technology