Justgiving's AWS journey

Manchester AWS Meetup@owainperry

What?

We had to migrate to AWS to enable scale

More dynamic resource

Micro services

Have a set of general principles

Our Devops

Use Chef to provision infrastructure

Overview of resource providers

Json define infrastructure

Building AMI's

Booting up

Garbage collection

Application information

Fails.

The future

Raise Money for Charities

Turnover ~400 Million a year

Process UK gift aid

Text Giving

Crowd funding

~1000 nodes

Stephen Sutton

Stephen Sutton

Huge stained loadsViral traffic Random traffic patternsMedia coverageHosting model at the time not suitable Random tidal wave loads No make up selfie Dog's home Ice bucket challenge

Move away from the monolith Micro services It's a small service! Application architecture dictates infrastructure architectureMove to micro services to implement CD Usually a component makes up part of a business domainHas one purpose Is simple to reason about Reality - various levels of granularityOwn infrasturcutre to scale independantly

Infrastructure principles

Infrastructure principles

Wanted development to define what they need Very Available Have a common repeatable patternHeavy use of automation Get everything in git Keep environments isolated (Dev/stg/prod)Be pragmatic.

Traintrack automation

Development - self service infrastructureFast to spin up infrastructure

Less Ops bottleneck

Reduce the need for AWS keys

Consistency benefits: Service architecture

Names and name conventions

Tags

Security groups

Scripting

Chef defined infrastructure

Created some base HWRP to abstract out common functionsAll changes are pushed into git CI picks up changes and pushed to chef Kicks off chef run on our infra builder nodesChef reconverges every 30 mins. One per account. (dev , stg, prod)

Heavy Weight Resource Providers

Instance

Autoscale groups

Launch configurations

Subnets

Load balancers

Security groups

IAM

DNS

aws_subnet 'subtest2' do accesskey 'accesskey' #not required secretkey 'secretkey' #not required region 'eu-west-1' vpc 'vpc-dev' azone 'eu-west-1a' cidr '10.0.18.0/24' routetable 'RTB-DEV-PRIVATE' tags [ { "a" => "1"},{"c" => "2" }] action :createend

Simple example LWRP to create a subnet Idempodency checks

LWRP built out the base HWRP 2 Base types ELB ASG LCF ASG LCF These are managed usng databagsBut we can use the resources in any way we want Build custom cookbooks etc.

{ "id": "lcftest", "type": "asg-lcf","region": "eu-west-1","availability_zones": ["eu-west-1c"],"vpc": "DEV-VPC","routetable": "RTB-DEV-PRIVATE","ami": "win2012r2_conf_std","instance_type": "t2.small","security_groups": ["GENERAL"],"launch_configuration_name": "LCF-DEV-SERVICE-LTEST","tags": [{"component": "AWS-DEV-SERVICE-LTEST"},{"owain": "PERRY"}],"aws_autoscale_group_name": "ASG-DEV-SERVICE-LTEST","health_check_type": "EC2", "health_check_grace_period": 1200,"asg_min_size": 1,"asg_max_size": 1 ,"asg_desired_capacity": 1,"asg_default_cooldown": 200,"root_volume_size": 200,"subnets": [ {"name":"SUB-DEV-SERVICE-ZC","azone": "eu-west-1c","cidr": "10.0.106.0/24","routetable": "RTB-DEV-PRIVATE"} ]}

Auto scale group

Auto scale group

Elastic Load BalancerElastic Load BalancerVarnishNode 1Node 2Node 1Node 2

Micro service hosting architecture v1 Varnish is the load balancerBlue green deployments Chef switches

Packer

Packer

Build using CI base ami's with chef and updates

Version stamped and stored in database

Runs core cookbook Sensu

LDAP

System logs into ELK

Plugs into automation to push latest

Pipeline to deploy into Dev / Stg / prod

Supports multi-region

Packer pipeline

Insert GOCD pipeline pic

Bootup

Prebaked AMIs (faster especially on windows)

Instance userdata script contains the bootup script (inc chef runlist)

Kicks off initial chef run (registers with chef)

Auto scale groups used to instantiate new instances and manage machine life cycles

Garbage collection

SQS to rabbitMQ pubsubChef node removal Sensu node removal Graphite node management

Internal tools

Headup display for development Reduces need for AWS console accessStatus of Active Sides Node informationChef information

What we didn't get right...

Zombies.

AWS reboot

Cost control

Abstraction of infrastructure json

Not putting DNS into abstraction

Chef search to configure load balancer

Not automating our base AMI's from the outset

RDS per service

Security groups

Chocolatey auto update = true

Future

Multi region Simplify the json V2 data definitions DNS based blue green Sensu checks automation Aws parsing and chef parsing for sensu automationRolling instance recycling daily / weekly? Bake images in productionService discovery rather than chef search Infrastructure penetration testing Infrastructure security checks

{ "id": "dev-redis", "type": "asg-lcf-v1", "vpc": "DEV", "region": ["eu-west-1","eu-central-1"] "ami": "amazon_conf_std", "name" "LOGSTAS-REDIS" "health_check_type": "EC2", "health_check_grace_period": 1200, "cookbook":"gg-redis" "monitoring":"/status/health" "dns":["redis.justgiving.service"] "tags": ["":""]}

So..

We had to migrate to AWS to enable scale

More dynamic resource

Micro services

Have a set of general principles

Our Devops

Use Chef to provision infrastructure

Overview of resource providers

Json define infrastructure

Building AMI's

Booting up

Garbage collection

Application information

Fails.

The future

Demo?

Questions?



@owainperry

Additional slides...

Automated change

Change log automated into Jira for releasesStaging and production

Change log automated for cookbook releases

Platform to include more over time.

Automated change log

CD of micro services

Each micro service has its own Git repository

GO Pipeline

Team City build

Chef cookbook

Tests

AWS Infrastructure

Operations run book

Health check endpoint - /status/health

Blue / Green Deployment

Pipelines for chef cookbooks