docker, ansible and practical experiences....ansible and docker • two relevant modules for ansible...

PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING

Docker, Ansible and practical experiences.

Kristian Lyngstøl, Redpill LinproOslo, June, 2016


Background

• Senior Systems Consultant for Redpill Linpro

• Volunteer at The Gathering

• Worked with:

• Docker for 3 years

• Configuration management for 7 years

• GNU/Linux for 17 years

• C/Perl/PHP/Python/JavaScript/Basic/AWK/+++ for 23 years.

• A well established knack for finding weird and obscure bugs.

• Trust issues.


Docker in one sentence

A client/server implementation for building and maintaining file archives

and for running them in isolated environments.


What?

• Define an image in a Dockerfile using simple commands (Start FROM this image, ADD this file, RUN this command, EXPOSE this port, etc)

• Upon changing a Dockerfile, only a small part of the image has to be rebuilt.

• Run as many copies of said image as you want. Each running instance is a container.

• Any changes made in a container is gone if you restart it. Sort of.


Dockerfile

FROM debian:jessie

RUN aptget update

RUN aptget y install varnish

RUN rm /etc/varnish/default.vcl

ADD extras/misc/varnish.vcl /etc/varnish/default.vcl

CMD varnishd a :80 f /etc/varnish/default.vcl F

EXPOSE 80


Docker pros

• Great for development and testing

• Enforces a well defined environment (sort of)

• “Just works”

• Lighter than a virtual machine

• Theoretically better hand-off to operations


Docker cons

• Overly pragmatic. Immature

• Still IPv4-centric in 2016.

• Hides complexity and dependencies. Black boxes everything.

• Infantile approach to logging.

• Obvious development work flow. Non-obvious production work flow.

• No obvious non-disruptive way to combine with existing applications.

• Requires considerable commitment to utilize fully


The big one: Systemd++

• Horrible “religious war” between Docker developers and systemd developers.

• Just running 5-20 containers shouldn't require any real orchestration at all, but does.

• Makes integration with existing systems a pain.

• Forces an “all or nothing” situation


Ansible in one sentence

Minimalistic way to perform well-defined tasks across multiple

machines.


What?

• No centralized controller. No daemon. Just command line.

• Write a YAML fine defining tasks that are potentially linked. (“Install this package repository”, “Enable this service”, “Build this docker container”)

• Write an inventory. Can be a single hostname in a file.

• Run ansible-playbook and it will connect to relevant hosts as needed.


Ansible playbook:

hosts: all

become: true

tasks:

apt: name=nano state=absent

git: repo=https://github.com/techserver/gondul.git

dest=/opt/gondul


Ansible pros

• Extremely easy to get started.

• Can be combined with existing configuration management or orchestration.

• Doesn't require dramatic changes.


Ansible cons

• Simple. For better or worse: Just tool.

• Replacing existing configuration management can be tricky

• Performance issues if you have many nodes (Python and parallelism isn't a match made in heaven)

• Several modules are somewhat immature


The Docker challenge

• How do you actually run these things in production?

• Lack of systemd-integration makes things less smooth than it could have been. Specially for smaller projects.

• Many emerging tools, all requiring big commitments

• Mesos, kubernetes, openshift?

• Basically, things haven't really settled yet.


Ansible and Docker

• Two relevant modules for Ansible 2.0.x

• “docker_image” to build an image

• “docker” to run a container (off of an image)

• Ansible 2.1.0 (released recently) expands and improves this. Notably introducing support for “docker compose”.

• Works well for smaller installations

• Everything can be kept in a single git repository

• Can be used for larger installations, but requires you to do a lot of thinking that others have done for you in competing solutions.


Short Intermission: The Gathering

• Annual non-profit digital festival attracting 6000-8000 people for 5 days each easter for gaming, concerts, socializing, esports, cosplay, shows and more.

• 400 volunteers, about 35 of which are part of the technical crew.

• 10,500 MAC addresses, 300-400 switches, access points and routers.

• Great test-bed for technology.

• http://www.gathering.org


Pictures courtesy of Joachim Tingvold


A few days earlier...


Various fiber cables Meru Wireless Access Points

Servers we need in production within 24-96 hours




About 200 Juniper EX2200 48-port switches


A second Juniper QFX10002-72q being installed($200,000-$250,000 each, 72 40Gbit ports or 24 100GBit ports)



~400 Volunteers



My job: Providing tools needed to ensure a good technical delivery.


Monitoring: Gondul

• Custom built to match the work flow and demand.

• Simple and robust design.

• Postgres database, 3-4 stand-alone data collectors, REST API, Varnish cache, JavaScript/HTML frontend.

• Used for other events too – open source.

• Served around 600 HTTP requests per second during TG16. About 300-400 rows inserted per second.

• Delay from data collection to visibility typically sub-second.

(Hi, Dream Hack).


Differences between Gondul and other projects

• Only needs to run it for 9 days.

• Simulating traffic from 7000 concurrent devices ranging from cell phones to laptops and x-boxes mixed with DDoS attacks is impossible.

• Can't force people to do boring things. If it isn't motivating at some level, it wont get done.

• Hard to introduce new developers to the project until they've seen how it's used.


Development and deployment

• Plan for changes: Development ahead of the event has to anticipate rapid changes to code and physical infrastructure.

• Self-documentation important.

• Easy (re-)deployment is key.

• Ridiculously overpowered hardware provided by gracious sponsors has to be utilized.

• Docker was an attractive alternative.


Attempt 1: Docker + systemd + homebrew

• Made transition between development in containers to deployment on hardware easier

• Largely self documenting

• Systemd integration was absolutely horrible and varied wildly between developers' individual systems.

• Nearly made me deaf when a docker container decided to reset my “host” volume to 100%.

• Development during the event was efficient, but awkward.


Attempt 2: Docker + Ansible

• Cleaner Dockerfiles (no systemd integration)

• Closer coupling to containers.

• No homebrew == leverage existing Ansible modules to get bonus features

• Finally getting some much needed test cases

• Development can still take place during the event and ahead of time

• Production database non-containerized. Development database containerized.


Demo

Example: Building

hosts: all

become: false

vars:

images:

name: "nmsdbtest"

links: []

name: "nmsfronttest"

links: [ "nmsdbtest:db" ]

name: "nmsvarnishtest"

links: [ "nmsfronttest:nmsfront" ]

name: "nmscollectortest"

links: [ "nmsdbtest:db" ]

tasks:

name: make all

docker_image:

state: build

name: "{{ item.name }}"

dockerfile: /opt/nms/build/test/{{ item.name }}.Dockerfile

path: "/opt/nms"

with_items: "{{ images }}"

Example: Basic tests

name: workaround to get nmsvarnishfrontip

shell: "docker inspect nmsvarnishtest | grep IPAddress | sed 's/[^09.]//g'"

register: ip

name: Display IP

debug:

msg: "Varnish test is available at http://{{ ip.stdout }}/"

name: test index

uri: url="http://{{ ip.stdout }}/"

name: test public api without data

uri:.

url: "http://{{ ip.stdout }}{{ item }}"

with_items: "{{ simple_urls }}"


Immaturity

• Neither Docker nor Ansible's docker modules can give me the IP reliably.

• “IPAdress” in docker still means legacy-IP, because why be part of the solution when you can be part of the problem.

• The docker “fact” of ansible has issues, even in recent 2.1.0.

• Specifically: Only shows facts for one container.

• Not accessible in a pretty way (e.g.: a list, not a dictionary)

• I assume these issues will be resolved with time


Deploying?

Just run it on a different host. Simple run-time configuration for picking

database server.


Potential and challenges

• Persistent data in a container is still painful, but possible.

• Being able to use the same solution to deploy non-docker infrastructure is a nice benefit.

• Bootstrapping the host.

• Experimenting with using Ansible to control and at least partially configure network equipment.

• If we used persistent infrastructure: Use a registry to do handover from development to production.


Questions?

docker, ansible and practical experiences....ansible and docker • two relevant modules for ansible...

Documents