Presenter: Stein Welberg

Continuous DeliveryHow it’s done at a SAAS start-up

About Me

Studied at Twente University

Waterpolo Player

Identity & Access management expert

CTO @ Onegini

Why am I here?

Sell Onegini

Explain about Continuous Delivery

Inspire you to think about CD

Steal engineers ;-)

What is Onegini

SaaS Platform: onegini.me

SSO solution to give external users access to your organisation

Focus on ease of use

Focus on security

Continuous Integration

"Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day.

Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly."- Martin Fowler

Continuous Delivery

Key concepts

Release early, release often

Small releases

Development close to business

Test the sh*t out of your application

Not just the code, also infra setup, db schema, etc!

Have software that can always go live!

Treat anything as software

Automate, automate, automate… automate!!!

Full deployment pipeline

Summary: Continuous Delivery is Continuous Integration on steriods

Continuous Delivery @ Onegini

A new release into production every 4 weeks

Devs could release anytime, business can’t keep up with us ;-)

Fully automated Deployments (infrastructure as code)

For all Business critical applications

Automated (integration / acceptance) testing

Critical issues are hardly ever found by our testers

Not all corner cases are tested automatically

Our road to Continuous Delivery

The challenges

Ruby (on Rails) and Java combined. Could not easily drop a war into e.g.

Google App engine or Azure.

Plans to use a graph / nosql database in the near future

Run (and be able to customize) and roll own virtual machine

Completely automated build environment testable and repeatable

No (manual) installation or installation manual ;-)

No SSH to the servers

No manual database changes

Time needed to update the production cluster must be less than one hour

Choices we made

Cloud Provider: Amazon

Advantages

IAAS: so you can host anything you want

We didn’t know exactly how many machines we needed or what kind of software stack

was necessary to complete the job

A lot of tooling exists

Great API’s

Relatively cheap when you are small

Free tier

Disadvantages

Small machines are really slow

Speed to cost ratio is quite bad

Choices we made

Rolling entire machines

Advantages

Machines stay fresh

Every machine is exactly the same (does not matter which environment)

Up- and downscaling is very easy and can be automated

Creating a machine can be automated and tested

Disadvantages

Need to manually run OS updates if you don’t roll machines often enough

No state inside the machine

Database needs to be backwards compatible

Property files need to be present on the machines (can be extracted)

Advantages

MySQL or other DB type (Oracle, MS SQL) is available

Easy to setup

Disadvantages

Only marginal customisability. Heavy performance tuning is not possible.

Write performance is not great

Backup and restore results in a different hostname

Choices we made

AWS RDS Database

Choices we made

Installation automation: Chef Solo

Advantages

Cookbooks in ruby

Standalone

No need to manually login to a server

Disadvantages

Updating cookbooks does break our build sometimes

Dependant of the community for updates of cookbooks

We have forked some cookbooks because they were broken or out of date

Troubleshooting can be time consuming

Choices we made

Base image

Advantages

Saves time: not necessary to install / run initial OS updates on the clones

Install software that is necessary on the entire stack

Disadvantages

If the base AMI contains an error all images do!

Choices we made

Asgard

Open-source tool by Netflix written in Groovy

Web-based AWS cloud management and deployment

Advantages

It provides the “rolling entire machines” approach

Uses the Amazon API’s

Provides it’s own API’s for automation

Amazon does not provide a UI to create Auto scaling groups

Disadvantages

Can only be used on Amazon AWS

Asgard

Cloud deployment model

Asgard

Cloud deployment model

Asgard

Fast rollback

Asgard

Fast rollback

Asgard

Fast rollback

Asgard

Fast rollback

Asgard

Fast rollback

Asgard

Fast rollback

Build Architecture Overview

Demo

Asgard

Chef Cookbook

Deploying a new version to test

Lessons learned

Needed to clean up old Amazon AMI’s, volumes, launch configurations,

etc

A lot of custom ruby scripts

AMI builder

Base AMI builder

Cleanup

Custom way to prevent building the entire pipeline in jenkins

Automating to starting a new machine and connecting to it via SSH is

quite ugly..

Amazon reuses IP addresses, this sometimes results in a fingerprint

conflict which causes a pipeline failure

The future

Amazon VPC setup

Network segmentation

In and egress filtering

Migration to another data center

Host data in a data center that is not owned by an American organisation