london devops - orc

OrcZero downtime deployments made easy

Tomas Doran (t0m)@bobtfish<bobtfish@bobtfish.net>https://github.com/bobtfish/

Problem space:

•Multiple heterogeneous applications

•Multiple servers per app

•Multiple environments

•Custom deployment solutions / semi-manual deployment

Observed problems:

•Support nightmare

•Failed / inconsistent deployments

•New applications extremely costly to deploy

Solution:

•Application infrastructure contract

•Defined interface for infrastructure to interact with your application:

•How to launch

•How to query health

•Any application which complies gets continuous deployment - free

Reliable

•Contract conformity checked by a common test suite

•CI builds to check deployability

Predictable

•Deployment is model driven

•Store desired state in ‘cmdb’

•Audit live servers, adjust reality to conform to desired state

•State machine - easy to test!

Safe•Each possible step in a deployment is

a state transition

•Sample live state after each transition - if model doesn’t match reality, stop!

•Mcollective agent - flexible per-user + fact based permissions for free

Architecture

Participation?

•Need to be able to cleanly remove instances from load balancer

•Make load balancer health check do 2 things:

•Check app = healthy

•Check participation = true

•Tatin - tiny REST service to drop state into

Contract URIs

•App:

•http://localhost:xxxx/info/health

•http://localhost:xxxx/info/version

•Participation:

•http://localhost:yyyy/app_name/environment_name/instance_name

deployapp methods

•get status (health + version)

•enable participation

•disable participation

•upgrade version

State machine

Multiple instances

•Discover all instances

•Work out next transition for each instance

•Each transition has a weight

•Sort by weight - take highest weighted

•Repeat until no transitions left

Simple example

AppApp

DesireDesiredd

versioversionn

ActualActualversioversio

nn

DesireDesiredd

partpart

ActualActualpartpart ActionAction WeighWeigh

tt

host1 5 4 Y YDisable part

1

host2 5 4 Y YDisabl

epart

1

Simple example

AppApp

DesireDesiredd

versioversionn

ActualActualversioversio

nn

DesireDesiredd

partpart

ActualActualpartpart ActionAction WeighWeigh

tt

host1 5 4 Y NUpgra

de2

host2 5 4 Y YDisabl

epart

1

Simple example

AppApp

DesireDesiredd

versioversionn

ActualActualversioversio

nn

DesireDesiredd

partpart

ActualActualpartpart ActionAction WeighWeigh

tt

host1 5 5 Y NEnable

part3

host2 5 4 Y YDisabl

epart

1

Simple example

AppApp

DesireDesiredd

versioversionn

ActualActualversioversio

nn

DesireDesiredd

partpart

ActualActualpartpart ActionAction WeighWeigh

tt

host1 5 5 Y Y None 0

host2 5 4 Y YDisabl

epart

1

Simple example

AppApp

DesireDesiredd

versioversionn

ActualActualversioversio

nn

DesireDesiredd

partpart

ActualActualpartpart ActionAction WeighWeigh

tt

host1 5 5 Y Y None 0

host2 5 4 Y NDisabl

epart

1

Simple example

AppApp

DesireDesiredd

versioversionn

ActualActualversioversio

nn

DesireDesiredd

partpart

ActualActualpartpart ActionAction WeighWeigh

tt

host1 5 5 Y Y None 0

host2 5 4 Y NUpgra

de2

Simple example

AppApp

DesireDesiredd

versioversionn

ActualActualversioversio

nn

DesireDesiredd

partpart

ActualActualpartpart ActionAction WeighWeigh

tt

host1 5 5 Y Y None 0

host2 5 5 Y Y None 0

A/B groups

•Can run A/B groups of the same app with different versions

•Can upgrade non-participating group, test, then swap participation.

•Same model just works

Status

•Orc, deployapp + contract test harness open source

•Docs still very lacking.

•You need to plug in your own repository module (and your own launcher for non jar apps)

Todo

•Better deployment (public gem+debs)

•Deployapp configuration rewrite

•http artifact fetcher

•More app launchers - rack/wsgi/psgi apps!

•Docs - specifically load balancer configs + installation.

Code

•The ideas here are way more important than the code

•Development sponsored by TIM Group

•Yes, of course we are hiring :)

•https://github.com/youdevise/orc

Thanks!

https://github.com/youdevise/orc

Questions?