continuous deployment strategies

Continuous Deployment Strategies

Ketan Padegaonkar

TECHNIQUES FOR IMPLEMENTATION

@snap_cisnap-ci.com

snap-ci.com © 2015 2

CONTENTSPage

Author Bio 3

Chapter 1: Continuous Deployment Strategies 4-5

1.1 Snowflake Servers 6

1.2 Phoenix Servers 7

1.3 How Does this Apply to Continuous Deployment? 8

1.4 Blue/Green Deployments 9

1.5 Canary Releases 10

Chapter 2: Blue / Green Deployments 11-12

2.1 Changing the load balancer/router 13

2.2 Changing public DNS records 14

2.3 Load Balancer + Internal DNS records 15

Chapter 3: Implementation Techniques for Canary Releases 16-17

3.1 Long running migration phase 18


AUTHOR BIO

Ketan P Senior Developer

Quintessential shaver of yaks, supporter and developer for open source, long time ThoughtWorker and builder of continuous delivery tools @goforcd and @snap_ci.

@ketanpkr github.com/ketan


Continuous Delivery Strategies


CHAPTER 1

Not very long ago, during the days of Perl/CGI and PHP (and even today);

deployments involved ftp-ing your files to the webserver directory and running a

bunch of migrations by shelling into the webserver.

Modern webapps and application servers have evolved quite a bit since then, but

many developers continue to use similar strategies to perform deployments. FTP has

been replaced by git pull followed by bundle install or npm install and then restarting

the appserver that you use.

Along with similar deployment strategies, many developers also continue to use the

same servers for months and years on end. But even with the right patches, updates,

etc, after a while it’s easy to end up with a “snowflake server”: a server with a delicate,

unique configuration that does not lend itself to change. A solution to this is the

“phoenix server.” Here’s a little more about both these concepts…


SNOWFLAKE SERVERS

While this approach may work great for small webapps, it quickly falls apart when

you’re managing anything but a couple of servers.

It is generally difficult to set up a new server if your existing server has problems. If

you’re looking to scale by adding more servers, it’s difficult to keep each server’s

software, configuration and services in sync. You may not be in a position to replicate a

production (or production-like) environment for testing.

Over time, one loses track of the magic configuration files, packages and services that

are installed on the production machines.

1.1


PHOENIX SERVERS

Some these issues can be addressed by using a bit of configuration management tools

like Chef, Puppet, or Ansible, among a few others.

These tools help avoid configuration drift, the sort of one-off changes that go

unnoticed and undocumented.

An important point to note is that these tools only apply the configuration that they are

asked to. If you applied any additional configuration outside of what these tools are

aware of, or if you forgot to un-apply or remove some configuration, you’d be out of

luck. This is what Ranjib Dey calls the Accumulator Anti-Pattern.

A much better alternative is to tear down servers periodically and apply configuration

changes every once in a while. This helps avoid and catch any configuration drift

outside of configuration management tools. Eventually one can move to deploying new

servers on every configuration change. This strategy is known as an Immutable Server.

1.2


HOW DOES THIS APPLY TO CD?

Continuous Deployment requires that at a very minimum, you have: * a solid

foundation of tests that gives you confidence in your software * a set of automation

tools and scripts that give you confidence that your deployment will succeed, or

rollback in case of issues

A few of the more popular continuous deployment strategies include blue/green

deployments and canary releases, which I’ve briefly outlined below…

1.3


BLUE / GREEN DEPLOYMENTS

Blue/green is a technique for deployments where the existing running deployment is

left in place. A new version of the application is installed in parallel with the existing

version. When the new version is ready, cut over to the new version by changing the

load balancer configuration.

This makes rollback really simple and gives time to make sure that the new version

works as expected before putting it live.

1.4

This is one of the simplest CD strategies to implement and get started with.

v1.0

WEB

SERVER

ROUTER

APP

SERVER

DB

SERVER

v1.1

v1.0

v1.1

v1.0

v1.1


CANARY RELEASES

This is named after the “canary in a coal mine” metaphor. The metaphor originates

from the times when miners used to carry caged canaries into the mines with them: if

there were any dangerous gases (methane or carbon monoxide) in the mine, the

canary would die before the gas levels reached levels hazardous to humans.

Canary releasing is similar to blue/green, although only a small amount of the servers

are upgraded. Then, using a cookie or similar, a fraction of users are directed to the

new version.

1.5

This allows for the load and functionality of the site to be tested with a small group of

users. If the application behaves as expected, migrate more and more servers to the

new version until all the users are on the new version.

This technique can also be used to do some interesting multi-variant testing and

performance testing.

v1.0

v1.1


Blue/Green Deployments


CHAPTER 2

In the previous article on Continuous Deployment Strategies, we explored at a high

level some of the popular CD strategies.

This article describes some of the implementation techniques for performing blue/

green deployments.

To recap – Blue/green is a technique for deployments where the existing running

deployment is left in place. A new version of the application is installed in parallel with

the existing version. When the new version is ready, cut over to the new version by

changing the load balancer configuration.

The basic idea behind this technique is to be able to route users to either the green set

of servers, or the blue set of servers.

This can be achieved in multiple ways:

Changing the load balancer/router

Changing public DNS records

Load Balancer + Internal DNS records


CHANGING THE LOAD BALANCER / ROUTER2.1

Using this technique you would change the load balancer or request router to point to

either the blue server or the green server.

v1.0

WEB

SERVER

ROUTER

APP

SERVER

DB

SERVER

v1.1

v1.0

v1.1

v1.0

v1.1


CHANGING PUBLIC DNS RECORDS2.2

Using this technique involves load-balancing at the DNS level by changing either the

CNAME or the A records. When users request a DNS entry for your website, your DNS

server could return the IP address for either the blue server or the green server. This

particular approach would require that the time to live (TTL) is set to a very small value.

This particular approach is not recommended for changing public DNS records because

some DNS servers run some browsers don’t respect the low TTL values on the DNS entry,

they will cache the DNS records for longer than the TTL. This may leave a subset of your

users not being able to access your website.

DNS

SERVER

ex

am

ple

.co

m

1.2

.3

.4

OR

5.6

.7

.8

green.example.com => 1.2.3.4

blue.example.com => 5.6.7.8


LOAD BALANCER + INTERNAL DNS RECORDS2.3

This approach is a combination of the two techniques we’ve seen so far. In this technique,

instead of changing public DNS records, we’ll change internal (private) DNS records. Since

the internal network and the DNS server can be managed and configured to respect TTLs,

this approach works a lot better than changing public DNS records.

DNS

SERVER

inte

rn

al.

ex

am

ple

.co

m

19

2.1

68

.1

.1

OR

19

2.1

68

.2

.1

green.internal.example.com => 192.168.1.1

blue.internal.example.com => 192.168.2.1


Implementation Techniques for Canary Releases


CHAPTER 3In the previous article on Continuous Deployment Strategies, we explored at a high

level some of the popular CD strategies.This article describes some of the

implementation techniques for performing Canary Releases.

Just like Blue/Green Deployments, one would start by deploying the application to a

small subset of your servers. Once the subset of servers is deployed to, you may then

route requests for a few users to the new set of servers.

This strategy lets you do a lot of interesting things:

Test the performance of the application.

Perform A/B tests based on demographics and user profiles, for example “users

between the ages of 20 – 25 living in Iowa.”

Load Balancer + Internal DNS records.

As your confidence in the deployment improves, you can deploy your application to

more and more servers, and route more users to the new servers.

v1.0

v1.1


LONG RUNNING MIGRATION PHASE

Because one or more versions of your application may be running in production for

some period of time, your application and its components (webservices, microservices,

database) needs to be backward-compatible so that it works with at least one or two

previous versions of your application. This Parallel-Change Pattern is a simple and

effective way to implement backward-incompatible changes between application

interfaces.

3.1

continuous deployment strategies

Technology