the road to continuous deployment: a case study (dpc16)

THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY

MICHIEL ROOK


ABOUT ME

▸ Java, PHP & Scala contractor

▸ PHP since ’99

▸ Maintainer of Phing

▸ Dutch Web Alliance

▸ http://www.linkedin.com/in/michieltcs

▸ @michieltcs

http://www.linkedin.com/in/michieltcs


THIS TALK

▸ Background

▸ The approach

▸ Process / standards

▸ Build pipelines

▸ Results & lessons learned


THE SYSTEM - SAN DIEGO

▸ ... or the Big Ball Of Mud

▸ Large legacy monolith

▸ Generates significant income

▸ Slow

▸ Complex, lots of moving parts

▸ Technical debt

SAN DIEGO FRONTEND

MYSQL DB

SAN DIEGO BACKEND

LOAD BALANCERS / VARNISH

ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK

SAN DIEGO FRONTEND

SAN DIEGO FRONTEND

SAN DIEGO FRONTEND

SAN DIEGO BACKEND

SAN DIEGO BACKEND

SAN DIEGO BACKEND

MEMCACHE FTP EXT. SERVICES

SOLR


THE SYSTEM - SAN DIEGO

▸ 2.5% code coverage

▸ Fragile tests

▸ Low velocity

▸ Frequent outages / bugs / issues

▸ Frustrated team

▸ Low confidence modifying existing code

REFACTOR? REBUILD?

A NEW STACK


THE STACK


APPROACH

▸ Strangler pattern

▸ Proxy to switch between old/new

▸ Migrate individual pages

ORIGINAL MONOLITH

PROXY

SERVICEORIGINAL MONOLITH

ORIGINAL MONOLITH

SERVICE SERVICE

SERVICE

PROXY

DB

DBDB

DB

DB DB


APPROACH

▸ Services per domain object (job, jobseeker, ...)

▸ Services behind load balancers

▸ Access legacy db’s

▸ Continuous deployment

▸ Containers

▸ Frontends are services

SAN DIEGO

ELASTIC SEARCHLEGACY

DB

JOB SERVICE

RMQ

ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK

MONGO DB

ITBANEN

JOBSEEKER SERVICE

NVBINTERMEDIAI


PROXY

# Serve job detail page from new codeRewriteRule ^/vacature/.* /app.php [L,PT]# Proxy php files to php5-fpm ProxyPassMatch ^/(app.php(/.*)?)$ fcgi://127.0.0.1:9000/opt/webapp/web/$1# Proxy everything else to SanDiegoProxyPass / ${SAN_DIEGO__URL} nocanonProxyPassReverse / ${SAN_DIEGO__URL}ProxyPreserveHost On


FEATURE TOGGLES, A/B TESTS

PROCESS


PROCESS

▸ Scrum, 1 week sprints

▸ TDD / BDD

▸ Definition of Done

▸ Team mindset / experience

▸ Focus on value

▸ Replace old features with new (legacy becomes obsolete)

CONTINUOUS EVERYTHING

DEV TEST ACCEPTANCE PRODUCTION

DEV TEST ACCEPTANCE PRODUCTION

CONTINUOUS DELIVERY

CONTINUOUS DEPLOYMENT

ONLY COMMIT TO MASTER

PAIR PROGRAMMING

BOY SCOUT RULE

QUALITY GATES

100% CODE COVERAGE

DASHBOARDS

BUILD PIPELINES

AUTOMATE REPEATABLE THINGS


AUTOMATION

▸ Builds

▸ Testing

▸ Deployments

▸ Orchestration

▸ Config management

EVERY COMMIT GOES TO PRODUCTION


DEFENSE IN DEPTH

UNIT TESTS

SCENARIOS

INTEGRATION

SMOKE / PERF. TESTS

MANUAL TESTING?


UNIT TESTSpublic function testGetById(){ $expectedJob = $this->getJob(); $this->jobRepository->getById($expectedJob->getId()) ->shouldBeCalled() ->willReturn($expectedJob); $this->assertEquals( $expectedJob, $this->jobService->getById($expectedJob->getId() ); }


SCENARIOS

Scenario: As an API user I need to be able to retrieve a job Given there is a valid job When I retrieve a valid job Then I should get a valid job resource


BUILDING

<project name="JobService" default="build"> <target name="build"> <delete file="jobservice.tar" quiet="true"/> <tar destfile="jobservice.tar"> <fileset dir="src"> <include name="*.php"/> </fileset> </tar> </target> </project>


DOCKER

FROM php:7.0-apache

ADD vhost.conf /etc/apache2/sites-available/000-default.conf

ADD jobservice.tar /opt/webapp

RUN chown -R www-data:www-data /opt/webapp


DEPLOYING

PULL IMAGE

REMOVE FROM LOAD BALANCER

STOP CONTAINER

START NEW CONTAINER

WAIT FOR PORT

SMOKE TESTS / HEALTH CHECKS

ADD TO LOAD BALANCER


DEPLOYING

- name: remove instance from load balancer haproxy: state=disabled host={{ inventory_hostname }} backend=jobservice delegate_to: "{{ item }}" with_items: groups.haproxy - name: stop & remove old container docker: name: jobservice state: absent image: jobservice - name: start container docker: name: jobservice state: present image: jobservice:{{ BUILD_NUMBER }} ports: - "8080:8080"


DEPLOYING

- name: perform health check uri: url: http://localhost:8080/_health status_code: 200 timeout: 30 return_content: true changed_when: False register: result until: result.content is defined and result.content.find("jobservice") != -1 retries: 10 delay: 3 - name: add instance to load balancer haproxy: state=enabled host={{ inventory_hostname }} backend=jobservice delegate_to: "{{ item }}" with_items: groups.haproxy

http://localhost:8080/_health


BUILD PIPELINE

node { stage 'Run tests' sh "phpunit" sh "behat" stage 'Build docker image' sh "phing build" sh "docker build -t jobservice:${env.BUILD_NUMBER} ." sh "docker push jobservice:${env.BUILD_NUMBER}" stage 'Deploy acceptance' sh "ansible-playbook -e BUILD_NUMBER=${env.BUILD_NUMBER} -i acc deploy.yml" stage 'Deploy production' sh "ansible-playbook -e BUILD_NUMBER=${env.BUILD_NUMBER} -i prod deploy.yml" }


BUILD PIPELINE

FEEDBACK!

RESULTS


RESULTS

▸ Total build time per service < 10 minutes

▸ Significantly improved page load times

▸ Improved audience stats (time on page, pages per session, session duration, traffic, seo ranking, etc)

▸ Increased confidence and velocity

▸ Experimented with new tech/stacks (angular, jvm, eventsourcing)

▸ More fun


LESSONS LEARNED

▸ Team acceptance

▸ Change is hard

▸ Overhead of weekly sprint; requires discipline

▸ Docker orchestration

▸ Issues with traffic between Amazon <-> on-premise datacenter

▸ Javascript testing


LESSONS LEARNED

▸ Experience with new tech

▸ Stability of build pipelines

▸ Management/leadership buy-in

▸ Not enough focus on replacing legacy application


QUESTIONS & DISCUSSION

Please leave feedback at https://joind.in/talk/4dab5

Thank you!

https://joind.in/talk/4dab5

the road to continuous deployment: a case study (dpc16)

Software