microservice and continuos delivery

Continuous Delivery in a microservices environment A success story

Luca Mazzaferro [email protected]

Outline:

•  About me

•  About HolidayCheck

•  CD and micro-services at HolidayCheck

▪  I was born in Rome where I spent most of my life until 2013 when I moved to Munich.

▪  Former physicist, member of ATLAS experiment @CERN during the roaring master degree and Ph.D. years. Mainly focusing in data analysis and grid computing.

▪  In June 2015, I sold my soul to the business, leaving the public research for happily joining HolidayCheck as devops engineer

About Me

About HolidayCheck

1

Holidaycheck means…

Holidays and travel bookings

1

Trustable reviews



2

1

Trustable reviews


Finding the best price

2

3


1

Trustable reviews


… and much more.

2

3

4

Finding the best price


Hotels: 659.000

Dests: 195 Countries 52880 Towns

Reviews: 6.900.000

Photos: 7.470.000

Visit/Month: 13.66 Million

Tour operators: 100

About HolidayCheck Some numbers

HolidayCheck is the largest review, booking and price comparing portal for travel and holidays in German language.

CD and micro-services at HolidayCheck A bit of History

Pre HC microservices era: Not so much time ago…

The Deployment Day

Pre HC micro-services era: Not so much time ago…

ABC

1 Day 1 Week 2 Weeks

C B A

ABC

Development

Collect &

Merge Test & Deploy


Results: ●  Low deployment rate ●  Frequent full roll backs ●  Many inter teams

dependencies ●  Hard team planning ●  High stress


The Elephant Release


Dev

Test

Merge

Deploy

Fback


Results:

●  Low deployments rate

●  Teams queued

●  Inter teams dependencies

●  Still monolithic application


Continuous Delivery is the ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands

of users, safely and quickly in a sustainable way.*

A bit better but still far away from here:

* continuousdelivery.com

CD and micro-services at HolidayCheck Our solution

The migration to micro-services The ingredients

●  Migration plan ●  New teams structure and responsibilities ●  New technology stack

1st Ingredient: Migration plan

The 1st ingredient

1.  Slow migration: from mobile to desktop 2.  One section after another: less traffic to higher 3.  Getting rid of old legacy software (when possible)

Migration Plan:

2nd Ingredient: New teams structure &

responsibilities

•  Preferred small cross functional teams (5-6 people) •  Devops Culture adopted via in-team devops engineer •  Teams responsible for their own products (microservices) •  Inter-teams dependencies removed and autonomy enforced

2nd ingredient

PO/TO SM Devs Devops Eng.

Ops Team

Teams structure & reposibilities

Organiza(onswhichdesignsystems...areconstrainedtoproducedesignswhicharecopiesofthecommunica(onstructuresoftheseorganiza(ons[M.Conway]

*

3rd Ingredient: New technology stack

-  The Dust project -  The Alerts and

Monitoring -  CD at HC

The new technology stack The Dust project

Creation of Dust Team (Ops + Dev): •  Setup the new infrastructure

focused to run micro-services •  Develop specific tools for micro-

services deployments •  Support teams for services

“containerization”

Dust Environment

The new technology stack The Dust project

Dust Environment

Main advantages:

•  Easily scalable

•  Ready for Continuous

Deployments

•  Services easy to monitor

•  Isolated environment

•  Enforce team autonomy


-  The Dust project -  The Alerts and


The new technology stack

•  Metric based Alerts

•  Logs •  System metrics

Default Monitoring

•  Teams defined dashboards

Custom Monitoring

New approach to monitoring and alerting

•  Required for each service •  Based on the user experience

(mainly error rate and response time)

•  Customizable by the teams •  Automatically deployed

Alerts You can’t survive without

Alerts A simple overview

A

B C

DUST

NGINX metrics

Customizable Metrics

Alerts defined on user experience: -  Error rate (5XX, 4XX) -  Response Time

On Call Duty

Teams

Traffic

•  Automatically deployed •  Based on logs and metrics •  Useful for

debugging criticalities discovery performances issue discovery

•  Accessible by devs and ops.

Default Monitoring Makes your life easier

•  One centralize system for all the logs

•  Search and visualize •  No configuration needed •  Easy to integrate in GCE

Default Monitoring Logs

Servers

Microservices

GCE

•  Grafana shows all the metrics collected by Prometheus

•  Custom metrics •  Custom dashboards

Default Monitoring Custom Metrics

Servers

Microservices

GCE

Default Monitoring Custom Metrics

NGINX Grafana dashboard

•  External service mainly used by micro-services

•  Error analysis •  Services maps •  Metrics monitoring

Default Monitoring NewRelic

Servers

Microservices

GCE

Default Monitoring NewRelic Services Maps

Services Maps

•  Team defined metrics and dashboards

•  Freedom to chose tools and technologies

•  Mainly for service quality and performances monitoring

•  Data & Business analysis

Custom Monitoring Can drive business decisions


-  The Dust Project -  The Alerts and


1 Develop, commit and push

Continuous Delivery @HolidayCheck Our Vision

Staging

Live

Team Server

1

2

3

4

Build Basic tests (e.g. Unit, Reg.) Deploy on staging env

2

3 Sophisticated Tests (integr., accuracy)

4 Deploy in production Smoke tests

* Continuous Monitoring

MSs/Containers

100

Teams

12 Deployment Rate

Daily/Weekly

Main languages: Scala/Go/ReactJS

Main Backend Techs ElasticSearch/

MongoDB/Kafka

HC Infrastructure in Numbers

Currently we are at the end of a migration phase in Google Cloud Engine.

Conclusions The adoption of microservices

enforced team autonomy, enabled the Continuous Delivery and

increased the deployment rate.

The entire infrastructure is more stable, reliable, maintainable

and scalable

A smooth migration in Google has been helped only by the

previous micro-services adoption.

www.holidaycheck.de

HolidayCheck Solutions Neumarkter Str. 61 | D- 81673 München Luca Mazzaferro DevOps Engineer T: Tel +41 (0) 71 686 9000 [email protected]

Contact.

THANK YOU FOR YOUR ATTENTION!

Questions?

microservice and continuos delivery

Software