Safe deployments with Blue-Green and Spinnaker

Platform Bucharest 23.02.2017

HiMihnea Dobrescu-Balaur

@mihneadbhttps://www.mihneadb.net

Deployment

• getting product out to clients

• agility matters - first to market

• speed vs risk

Rolling deployment

• classic/easiest way of doing things

• on every server: git pull; restart

• you are probably doing some variation of this

http://searchitoperations.techtarget.com/definition/rolling-deployment

Rolling deployment (cont.)

• Advantages:

• simple

• no extra hardware

• can do partial deploys

Rolling deployment (cont.)

• Disadvantages:

• might drop requests

• slow rollback

• finding the "last good state"

• testing in staging only

Blue-Green deployment

• two identical production environments

• only one serves requests

• deploy to the new one; test; route traffic

http://fbrnc.net/blog/2016/05/green-blue-deployments-with-aws-lambda-and-cloudformation

Blue-Green deployment (cont.)

• Advantages:

• testing in production

• no downtime

• quick rollback

• predictable release time

Blue-Green deployment (cont.)

• Disadvantages

• cost - 2x servers

• databases require extra work

• development time to implement

Fixing the database problem

Optimizations• connection draining

• canary

• sticky sessions

• automated tests on green

• reproducible deploys using AMIs/Docker images

Real world

• using it at Hootsuite since summer 2015

• 1K deploys in prod with no issues*

• has not affected our velocity

• deploy to prod is a mundane task

* trivial rollback in case we had any

Real world (cont.)• Tech stack:

• Docker

• Asgard

• Jenkins

• ELB

• Hubot

Our deployment pipeline

http://code.hootsuite.com/blue-green-deployments/

Our deployment pipeline (cont.)

• cluster check

• conflict check

• merge tests

• live tests

• git merge

• manual canary release

• manual blue-green switch

Spinnaker

• successor of Asgard (thanks, Netflix)

• custom build and deploy pipelines

• multiple cloud providers supported

Why Spinnaker?• easy trigger integration with Jenkins

• handles all the "cloud" stuff

• server groups

• load balancers

• security groups

• useful templates built-in

https://cloud.google.com/solutions/spinnaker-on-compute-engine

Spinnaker stack

• Jenkins

• Packer

• .deb repo

• Spinnaker core

Deploying .debs

• not everybody used to this

• lifesaver: fpm

• running your stuff - upstart

Demo!

http://www.spinnaker.io/docs/from-source-to-prod