building a paas with docker, consul and python

Download Building a PaaS with Docker, Consul and Python

Post on 15-Apr-2017




4 download

Embed Size (px)


  • Building a PaaS with Docker, Consul and Python

    by Konstantin Nazarov

  • I work in Tarantool And I've built a PaaS this way

  • I'll share how to build one It is simple

    The experience is highly portable

    You can start small and grow iteratively

    Fits your requirements

    The tech stack is widely known

  • 1. docker host

    2. docker host

    3. orchestrator

    4. consul

    5. web UI

  • Why?

  • Why build your own PaaS?

    If you develop your own product and sell it

    Small initial investment to solve lifecycle problems

    A way to fit your requirements exactly

    Keep your operations lean

    To enable fast experiments

  • When not to build it

    When you run off-she-shelf software

    If you have large monolithic services

  • What a PaaS should do

    Run the code you give to it

    Abstract away the OS

  • Let's build it

  • We will build progressively

    On each step there will be a working system

    Let's build it

  • A few building blocks

  • 1. Python Because it is simple

    There are bindings for almost all existing stuff

  • 2. Docker

    Has remote HTTP API to run stuff

    Has convenient packaging format

  • 2. Docker: the good


    Well documented

  • 2. Docker: the bad


    Weak networking

  • 2. Docker: Alternatives

    fabric (yes, as an RPC)



  • 3. Consul

    Fault-tolerant key-value storage

    Service registry with active checks

    Easily deployed

  • 3. Consul: alternatives



  • MK1: Smart command-line

    client to Docker/Consul

  • Why?

    quick to implement

    very high value compared to effort

  • What it should do






  • Example usage

    $ mypaas run git://gitserver/project.git v1.2


    $ mypaas upgrade fddf3f v1.3

    $ mypaas rm fddf3f


  • How

    Docker API is exposed on physical servers

    Physical servers are registered in consul

    CLI connects to a known consul host

    Docker API is used to build app container "in place"

  • How to choose physical nodes

    By maximum memory requirements

    By conventional CPU units

    By number of services running

  • Upgrading versions

    Stop the running container

    Start new container inheriting volumes from the old

    On success, remove old container

    On failure, restart old container

  • Result

    A working PaaS

  • Improving MK1: health checks

    Consul API can be used to register health checks

    Consul can run commands in docker containers

    The CLI can poll consul for service statuses

  • Wiring things together via network

  • Docker networking hell

    There are overlay networks (UDP encapsulation)

    And macvlan (adding new MAC addresses to eth)

    And openvswitch with DPDK

    And god knows what else (BGP routing anyone?)

  • Let's use plain bridges


  • Let's use plain bridges

    Docker IPAM doesn't know about other nodes

    IP conflicts are possible

    So we have to use our own IPAM

    Write allocated IPs to consul KV

    Set IPs explicitly

  • IPAM

  • MK2: running as a service

  • Why? Limiting access to production servers

    Active monitoring of business logic

    Concurrent access

  • How?




  • Why these tools?

    Everything fits in one app

    Orchestration code is easier to write in async mode

    Quick to implement

    Your CLI becomes an HTTP API client

  • Separate state Orchestrator itself should be stateless

    Orchestrator should show the system overview

    You probably need basic auth at this step

  • MK3: admin UI

  • Why?

    Easier to manage

    Easier to debug

  • Example

  • How?

    flask templates


  • What to have here

    Overview of physical servers and their state

    Overview of your services and their state


  • MK4: delayed tasks and active checks

  • Why?

    Distributed cron

    Data extraction


  • Tools


    Docker exec API

  • How?

    On server start, spawn a worker fiber

    In the fiber, poll consul and run your code

    Or start worker fibers on demand

    Send notification emails upon completion

  • MK5: metrics and time series

  • Why?

    Historical data matters for problem solving

    See how well your new code is behaving over time

  • How?

    install prometheus and hook it up to consul

    use prometheus API to query aggregates

  • Exporting metrics to prometheus

    Either add support to your service

    Prometheus protocol is very simple!

    Or collect via the orchestrator

  • Recap

  • Recap docker-python - running stuff

    python-consul - storing information about running stuff

    flask - serving admin UI

    flask-restful - providing HTTP API

    bootstrap - making your web page less ugly

    gevent - running delayed tasks and async code

    prometheus - storing time series

  • Thanks!Konstantin Nazarov

    Building orchestrators is not that hard.



View more >