flocker documentation - read the docs · 2019-04-02 · flocker documentation, release 0.2.1...

Flocker DocumentationRelease 0.2.1

ClusterHQ

October 09, 2014

Contents

1 Introduction to Flocker 31.1 Motivation for Building Flocker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.3 Initial Implementation Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.4 User Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Getting Started 72.1 Installing Flocker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Tutorial: Deploying and Migrating MongoDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.3 Flocker Application Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.4 Flocker Feature Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3 Advanced Documentation 353.1 What’s New . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.2 Using Flocker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363.3 Configuring Flocker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373.4 Volume Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.5 Data-Oriented Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403.6 Setting up External Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413.7 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413.8 Cleaning Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4 Getting Involved 434.1 Contributing to Flocker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.2 Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

5 Areas of Potential Future Development 535.1 Flocker Volume Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

6 FAQ 576.1 ZFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576.2 Future Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

7 Authors 59

i

Flocker Documentation, Release 0.2.1

Flocker is a data volume manager and multi-host Docker cluster management tool. With it you can control your datausing the same tools you use for your stateless applications. This means that you can run your databases, queues andkey-value stores in Docker and move them around as easily as the rest of your app.

With Flocker’s command line tools and a simple configuration language, you can deploy your Docker-based applica-tions onto one or more hosts. Once deployed, your applications will have access to the volumes you have configuredfor them. Those volumes will follow your containers when you use Flocker to move them between different hosts inyour Flocker cluster.

Contents:

Contents 1


2 Contents

CHAPTER 1

Introduction to Flocker

1.1 Motivation for Building Flocker

Flocker lets you move your Docker containers and their data together between hosts. This means that you can runyour databases, queues and key-value stores in Docker and move them around as easily as the rest of your app. Evenstateless apps depend on many stateful services and currently running these services in Docker containers in productionis nearly impossible. Flocker aims to solve this problem by providing an orchestration framework that allows you toport both your stateful and stateless containers between environments.

Docker allows for multiple isolated, reproducible application environments on a single node: “containers”. Applica-tion state can be stored on a local disk in “volumes” attached to containers. And containers can talk to each other andthe external world via specified ports.

But what happens if you have more than one node?

• Where do containers run?

• How do you talk to the container you care about?

• How do containers across multiple nodes talk to each other?

• How does application state work if you move containers around?

The diagram below provides a high level representation of how Flocker addresses these questions.

3


1.2 Architecture

Below is a high-level overview of Flocker’s architecture. For more information, you can follow along with a tutorialthat walks you through deploying and migrating MongoDB or read more in our advanced documentation.

1.2.1 Flocker - Orchestration

• Flocker can run multiple containers on multiple nodes.

• Flocker offers a configuration language to specify what to run and where to run it.

1.2.2 Flocker - Routing

• Container configuration includes externally visible TCP port numbers.

• Connect to any node on a Flocker cluster and traffic is routed to the node hosting the appropriate container(based on port).

• Your external domain (www.example.com) configured to point at all nodes in the Flocker cluster(192.0.2.0, 192.0.2.1)

1.2.3 Flocker - Application State

• Flocker manages ZFS filesystems as Docker volumes. It attaches them to your containers.

• Flocker provides tools for copying those volumes between nodes.

• If an application container is moved from one node to another, Flocker automatically moves the volume with it.

1.2.4 Application Configuration

• Application configuration describes what you want to run in a container.

– it identifies a Docker image

– an optional volume mount point

– externally “routed” ports

• This configuration is expected to be shared between development, staging, production, etc environments.

• Flocker 0.1 does not support automatic re-deployment of application configuration changes.

1.2.5 Deployment Configuration

• Deployment configuration describes how you want your containers deployed.

– which nodes run which containers.

• This configuration can vary between development, staging, production, etc environments.

– Developer might want to deploy all of the containers on their laptop.

– Production might put database on one node, web server on another node, etc.

• Reacting to changes to this configuration is the primary focus of Flocker 0.1.

4 Chapter 1. Introduction to Flocker


1.3 Initial Implementation Strategy

• This is the 0.1 approach.

• Future approaches will be very different; feedback is welcome.

• All functionality is provided as short-lived, manually invoked processes.

• flocker-deploy connects to each node over SSH and runs flocker-reportstate to gather the clusterstate.

• flocker-deploy then connects to each node over SSH and runs flocker-changestate to make thenecessary deployment changes.

• Nodes might connect to each other over SSH to copy volume data to the necessary place.

1.3.1 flocker-changestate

• This is installed on nodes participating in the Flocker cluster.

• Accepts the desired global configuration and current global state.

• Also looks at local state - running containers, configured network proxies, etc.

• Makes changes to local state so that it complies with the desired global configuration.

– Start or stop containers.

– Push volume data to other nodes.

– Add or remove routing configuration.

1.3.2 Managing Containers

• geard is used to start, stop, and enumerate containers.

• geard works by creating systemd units.

• systemd units are a good way to provide admin tools for:

– Logging and state inspection.

– Starting/stopping (including at boot).

– Inter-unit dependency management.

1.3.3 Managing Volumes

• Volumes are ZFS filesystems.

• Volumes are attached to a Docker “data” container.

• geard automatically associates the “data” container’s volumes with the actual container.

– Association is done based on container names by geard.

• Data model * Volumes are owned by a specific node.

– Node A can push a copy to node B but node A still owns the volume. Node B may not modify its copy.

– Volumes can be “handed off” to another node, i.e. ownership is changed. Node A can hand off the volumeto node B. Then node B is now the owner and can modify the volume and node A no longer can.

1.3. Initial Implementation Strategy 5

https://github.com/openshift/geard


• Volumes are pushed and handed off so as to follow the containers they are associated with.

– This happens automatically when flocker-deploy runs with a new deployment configuration.

1.3.4 Managing Routes

• Containers claim TCP port numbers with the application configuration that defines them.

• Connections to that TCP port on the node that is running the container are proxied (NAT’d) into the containerfor whatever software is listening for them there.

• Connections to that TCP port on any other node in the Flocker cluster are proxied (NAT’d) to the node that isrunning the container.

• Proxying is done using iptables.

1.4 User Experience

• Flocker provides a command-line interface for manually deploying or re-deploying containers across nodes.

• The tool operates on two distinct pieces of configuration:

– Application

– Deployment

• Your sysadmin runs a command like flocker-deploy deployment-config.ymlapplication-config.yml on their laptop.

6 Chapter 1. Introduction to Flocker

CHAPTER 2

Getting Started

Flocker is a lightweight volume and container manager. It lets you:

• Define your application as a set of connected Docker containers

• Deploy them to one or multiple hosts

• Easily migrate them along with their data between hosts

The goal of Flocker is to simplify the operational tasks that come along with running databases, key-value stores,queues and other data-backed services in containers. This Getting Started guide will walk you step-by-step throughinstalling Flocker and provide some tutorials that demonstrate the essential features of Flocker.

2.1 Installing Flocker

As a user of Flocker you will need to install the flocker-cli package which provides command line tools tocontrol the cluster. This should be installed on a machine with SSH credentials to control the cluster nodes (e.g., ifyou use our Vagrant setup then the machine which is running Vagrant).

There is also a flocker-node package which is installed on each node in the cluster. It contains theflocker-changestate, flocker-reportstate, and flocker-volume utilities. These utilities arecalled by flocker-deploy (via SSH) to install and migrate Docker containers and their data volumes.

Note: For now the flocker-node package is pre-installed by the Vagrant configuration in the tutorial.

Note: If you’re interested in developing Flocker (as opposed to simply using it) see Contributing to Flocker.

2.1.1 Installing flocker-cli

Linux

Before you install flocker-cli you will need a compiler, Python 2.7, and the virtualenv Python utility in-stalled. On Fedora 20 you can install these by running:

alice@mercury:~$ sudo yum install @buildsys-build python python-devel python-virtualenv

On Ubuntu or Debian you can run:

alice@mercury:~$ sudo apt-get install gcc python2.7 python-virtualenv python2.7-dev

7


Then run the following script to install flocker-cli:

linux-install.sh

#!/bin/sh

# Create a virtualenv, an isolated Python environment, in a new directory called# "flocker-tutorial":virtualenv --python=/usr/bin/python2.7 flocker-tutorial

# Upgrade the pip Python package manager to its latest version inside the# virtualenv. Some older versions of pip have issues installing Python wheel# packages.flocker-tutorial/bin/pip install --upgrade pip

# Install flocker-cli and dependencies inside the virtualenv:echo "Installing Flocker and dependencies, this may take a few minutes with no output to the terminal..."flocker-tutorial/bin/pip install --quiet https://storage.googleapis.com/archive.clusterhq.com/downloads/flocker/Flocker-0.2.1-py2-none-any.whlecho "Done!"

Save the script to a file and then run it:

alice@mercury:~$ sh linux-install.sh...alice@mercury:~$

The flocker-deploy command line program will now be available in flocker-tutorial/bin/:

alice@mercury:~$ cd flocker-tutorialalice@mercury:~/flocker-tutorial$ bin/flocker-deploy --version0.2.1alice@mercury:~/flocker-tutorial$

If you want to omit the prefix path you can add the appropriate directory to your $PATH. You’ll need to do this everytime you start a new shell.

alice@mercury:~/flocker-tutorial$ export PATH="${PATH:+${PATH}:}${PWD}/bin"alice@mercury:~/flocker-tutorial$ flocker-deploy --version0.2.1alice@mercury:~/flocker-tutorial$

OS X

Install the Homebrew package manager.

Make sure Homebrew has no issues:

alice@mercury:~$ brew doctor...alice@mercury:~$

Fix anything which brew doctor recommends that you fix by following the instructions it outputs.

Add the ClusterHQ/flocker tap to Homebrew and install flocker:

alice@mercury:~$ brew tap ClusterHQ/tap...alice@mercury:~$ brew install flocker-0.2.1...alice@mercury:~$ brew test flocker-0.2.1

8 Chapter 2. Getting Started

http://brew.sh


...alice@mercury:~$

You can see the Homebrew recipe in the homebrew-tap repository.

The flocker-deploy command line program will now be available:

alice@mercury:~$ flocker-deploy --version0.2.1alice@mercury:~$

2.2 Tutorial: Deploying and Migrating MongoDB

The goal of this tutorial is to teach you to use Flocker’s container, network, and volume orchestration functionality.By the time you reach the end of the tutorial you will know how to use Flocker to create an application. You will alsoknow how to expose that application to the network and how to move it from one host to another. Finally you willknow how to configure a persistent data volume for that application.

This tutorial is based around the setup of a MongoDB service. Flocker is a generic container manager. MongoDB isused only as an example here. Any application you can deploy into Docker you can manage with Flocker.

If you have any feedback or problems, you can Talk to Us.

2.2.1 Before You Begin

Requirements

To replicate the steps demonstrated in this tutorial, you will need:

• Linux, FreeBSD, or OS X

• Vagrant (1.6.2 or newer)

• VirtualBox

• At least 10GB disk space available for the two virtual machines

• The OpenSSH client (the ssh, ssh-agent, and ssh-add command-line programs)

• bash

• The mongo MongoDB interactive shell (see below for installation instructions)

You will also need flocker-cli installed (providing the flocker-deploy command). See Installing flocker-cli.

Setup

Installing MongoDB

The MongoDB client can be installed through the various package managers for Linux, FreeBSD and OS X. If you donot already have the client on your machine, you can install it by running the appropriate command for your system.

Ubuntualice@mercury:~$ sudo apt-get install mongodb-clients...alice@mercury:~$

2.2. Tutorial: Deploying and Migrating MongoDB 9

https://github.com/ClusterHQ/homebrew-tap

https://docs.vagrantup.com/

https://www.virtualbox.org/


Red Hat / Fedoraalice@mercury:~$ sudo yum install mongodb...alice@mercury:~$

OS X Install Homebrew

alice@mercury:~$ brew update...alice@mercury:~$ brew install mongodb...alice@mercury:~$

Other Systems See the official MongoDB installation guide for your system.

Creating Vagrant VMs Needed for Flocker

Note: If you already have a tutorial environment from a previous release see Upgrading the Vagrant Environment.

Before you can deploy anything with Flocker you’ll need a node onto which to deploy it. To make this easier, thistutorial uses Vagrant to create two VirtualBox VMs.

These VMs serve as hosts on which Flocker can run Docker. Flocker does not require Vagrant or VirtualBox. You canrun it on other virtualization technology (e.g., VMware), on clouds (e.g., EC2), or directly on physical hardware.

For your convenience, this tutorial includes Vagrantfile which will boot the necessary VMs. Flocker and itsdependencies will be installed on these VMs the first time you start them. One important thing to note is that theseVMs are statically assigned the IPs 172.16.255.250 (node1) and 172.16.255.251 (node2). These two IPaddresses will be used throughout the tutorial and configuration files. If these addresses conflict with your localnetwork configuration you can edit the Vagrantfile to use different values. Note that you will need to make thesame substitution in commands used throughout the tutorial.

Note: The two virtual machines are each assigned a 10GB virtual disk. The underlying disk files grow to about 5GB.So you will need at least 10GB of free disk space on your workstation.

1. Create a tutorial directory:

alice@mercury:~/$ mkdir flocker-tutorialalice@mercury:~/$ cd flocker-tutorialalice@mercury:~/flocker-tutorial$

2. Download the Vagrant configuration file by right clicking on the link below. Save it in the flocker-tutorialdirectory and preserve its filename.

Vagrantfile

# -*- mode: ruby -*-# vi: set ft=ruby :

# This requires Vagrant 1.6.2 or newer (earlier versions can’t reliably# configure the Fedora 20 network stack).Vagrant.require_version ">= 1.6.2"


http://brew.sh/

http://docs.mongodb.org/manual/installation/

https://docs.vagrantup.com/


# Vagrantfile API/syntax version. Don’t touch unless you know what you’re doing!...

alice@mercury:~/flocker-tutorial$ lsVagrantfilealice@mercury:~/flocker-tutorial$

3. Use vagrant up to start and provision the VMs:

alice@mercury:~/flocker-tutorial$ vagrant upBringing machine ’node1’ up with ’virtualbox’ provider...==> node1: Importing base box ’clusterhq/flocker-dev’...... lots of output ...==> node2: ln -s ’/usr/lib/systemd/system/docker.service’ ’/etc/systemd/system/multi-user.target.wants/docker.service’==> node2: ln -s ’/usr/lib/systemd/system/geard.service’ ’/etc/systemd/system/multi-user.target.wants/geard.service’alice@mercury:~/flocker-tutorial$

This step may take several minutes or more as it downloads the Vagrant image, boots up two nodes and down-loads the Docker image necessary to run the tutorial. Your network connectivity and CPU speed will affect howlong this takes. Fortunately this extra work is only necessary the first time you bring up a node (until you destroyit).

4. After vagrant up completes you may want to verify that the two VMs are really running and accepting SSHconnections:

alice@mercury:~/flocker-tutorial$ vagrant statusCurrent machine states:

node1 running (virtualbox)node2 running (virtualbox)...alice@mercury:~/flocker-tutorial$ vagrant ssh -c hostname node1node1Connection to 127.0.0.1 closed.alice@mercury:~/flocker-tutorial$ vagrant ssh -c hostname node2node2Connection to 127.0.0.1 closed.alice@mercury:~/flocker-tutorial$

5. If all goes well, the next step is to configure your SSH agent. This will allow Flocker to authenticate itself to theVM:

If you’re not sure whether you already have an SSH agent running, ssh-add can tell you. If you don’t, you’llsee an error:

alice@mercury:~/flocker-tutorial$ ssh-addCould not open a connection to your authentication agent.alice@mercury:~/flocker-tutorial$

If you do, you’ll see no output:

alice@mercury:~/flocker-tutorial$ ssh-addalice@mercury:~/flocker-tutorial$

If you don’t have an SSH agent running, start one:

alice@mercury:~/flocker-tutorial$ eval $(ssh-agent)Agent pid 27233alice@mercury:~/flocker-tutorial$



6. Finally, add the Vagrant key to your agent:

alice@mercury:~/flocker-tutorial$ ssh-add ~/.vagrant.d/insecure_private_keyalice@mercury:~/flocker-tutorial$

You now have two VMs running and easy SSH access to them. This completes the Vagrant-related setup.

Upgrading the Vagrant Environment

The Vagrantfile used in this tutorial installs an RPM package called flocker-node on both the nodes. If youalready have a tutorial environment from a previous release, you’ll need to ensure that both tutorial nodes are runningthe latest version of flocker-node before continuing with the following tutorials.

First check the current Flocker version on the nodes. You can do this by logging into each node and running theflocker-reportstate command with a --version argument.

alice@mercury:~/flocker-tutorial$ ssh [email protected] flocker-reportstate --version

Only proceed if you find that you are running an older version of Flocker than 0.2.1.

If you find that you are running an older version, you now need to rebuild the tutorial environment.

This will ensure that you have the latest Flocker version and that you are using a pristine tutorial environment.

Warning: This will completely remove the existing nodes and their data.

If you have the original Vagrantfile, change to its parent directory and run vagrant destroy.

alice@mercury:~/flocker-tutorial$ vagrant destroynode2: Are you sure you want to destroy the ’node2’ VM? [y/N] y

==> node2: Forcing shutdown of VM...==> node2: Destroying VM and associated drives...==> node2: Running cleanup tasks for ’shell’ provisioner...

node1: Are you sure you want to destroy the ’node1’ VM? [y/N] y==> node1: Forcing shutdown of VM...==> node1: Destroying VM and associated drives...==> node1: Running cleanup tasks for ’shell’ provisioner...alice@mercury:~/flocker-tutorial$

Next delete the cached SSH host keys for the virtual machines as they will change when new VMs are created. Failingto do so will cause SSH to think there is a security problem when you connect to the recreated VMs.

alice@mercury:~/flocker-tutorial$ ssh-keygen -f "$HOME/.ssh/known_hosts" -R 172.16.255.250alice@mercury:~/flocker-tutorial$ ssh-keygen -f "$HOME/.ssh/known_hosts" -R 172.16.255.251

Delete the original Vagrantfile and then download the latest Vagrantfile and run vagrant up.

alice@mercury:~/flocker-tutorial$ vagrant upBringing machine ’node1’ up with ’virtualbox’ provider...Bringing machine ’node2’ up with ’virtualbox’ provider...alice@mercury:~/flocker-tutorial$

Alternatively, if you do not have the original Vagrantfile or if the vagrant destroycommand fails, you can remove the existing nodes directly from VirtualBox. The two vir-tual machines will have names like flocker-tutorial_node1_1410450919851_28614 andflocker-tutorial_node2_1410451102837_79031.


https://www.virtualbox.org/manual/ch01.html#idp55629568


2.2.2 Moving Applications

Note: If you haven’t already, make sure to install the flocker-cli package before continuing with this tutorial.

Starting an Application

Let’s look at an extremely simple Flocker configuration for one node running a container containing a MongoDBserver.

minimal-application.yml

"version": 1"applications":

"mongodb-example":"image": "clusterhq/mongodb"

minimal-deployment.yml

"version": 1"nodes":

"172.16.255.250": ["mongodb-example"]"172.16.255.251": []

Notice that we mention the node that has no applications deployed on it to ensure that flocker-deploy knows thatit exists. If we hadn’t done that certain actions that might need to be taken on that node will not happen, e.g. stoppingcurrently running applications.

Next take a look at what containers Docker is running on the VM you just created. The node IPs are those which werespecified earlier in the Vagrantfile:

alice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESalice@mercury:~/flocker-tutorial$

From this you can see that there are no running containers. To fix this, use flocker-deploy with the simpleconfiguration files given above and then check again:

alice@mercury:~/flocker-tutorial$ flocker-deploy minimal-deployment.yml minimal-application.ymlalice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES4d117c7e653e clusterhq/mongodb:latest mongod 2 seconds ago Up 1 seconds 27017/tcp, 28017/tcp mongodb-examplealice@mercury:~/flocker-tutorial$

flocker-deploy has made the necessary changes to make your node match the state described in the configurationfiles you supplied.

Moving an Application

Let’s see how flocker-deploy can move this application to a different VM. Recall that the Vagrant configurationsupplied in the setup portion of the tutorial started two VMs. Copy the deployment configuration file and edit it so thatit indicates the application should run on the second VM instead of the first. The only change necessary to indicatethis is to change the original IP address, 172.16.255.250, to the address of the other node, 172.16.255.251.The new file should be named minimal-deployment-moved.yml.

minimal-deployment-moved.yml




"172.16.255.250": []"172.16.255.251": ["mongodb-example"]

Note that nothing in the application configuration file needs to change. Moving the application only involves updatingthe deployment configuration.

Use flocker-deploy again to enact the change:

alice@mercury:~/flocker-tutorial$ flocker-deploy minimal-deployment-moved.yml minimal-application.ymlalice@mercury:~/flocker-tutorial$

docker ps shows that no containers are running on 172.16.255.250:


and that MongoDB has been successfully moved to 172.16.255.251:

alice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES4d117c7e653e clusterhq/mongodb:latest mongod 3 seconds ago Up 2 seconds 27017/tcp, 28017/tcp mongodb-examplealice@mercury:~/flocker-tutorial$

At this point you have successfully deployed a MongoDB server in a container on your VM. You’ve also seen howFlocker can move an existing container between hosts. There’s no way to interact with it apart from looking at thedocker ps output yet. In the next section of the tutorial you’ll see how to expose container services on the host’snetwork interface.

2.2.3 Exposing Ports

Each application running in a Docker container has its own isolated networking stack. To communicate with anapplication running inside the container we need to forward traffic from a network port in the node where the containeris located to the appropriate port within the container. Flocker takes this one step further: an application is reachableon all nodes in the cluster, no matter where it is currently located.

Let’s start a MongoDB container that exposes the database to the external world.

port-application.yml


"mongodb-port-example":"image": "clusterhq/mongodb""ports":- "internal": 27017

"external": 27017

port-deployment.yml


"172.16.255.250": ["mongodb-port-example"]"172.16.255.251": []

We will once again run these configuration files with flocker-deploy:



alice@mercury:~/flocker-tutorial$ flocker-deploy port-deployment.yml port-application.ymlalice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES4d117c7e653e clusterhq/mongodb:latest mongod 2 seconds ago Up 1 seconds 27017/tcp, 28017/tcp mongodb-port-examplealice@mercury:~/flocker-tutorial$

This time we can communicate with the MongoDB application by connecting to the node where it is running. Usingthe mongo command line tool we will insert an item into a database and check that it can be found. You should try tofollow along and do these database inserts as well.

Note: To keep your download for the tutorial as speedy as possible, we’ve bundled the latest development release ofMongoDB in to a micro-sized Docker image. You should not use this image for production.

If you get a connection refused error try again after a few seconds; the application might take some time to fully startup.

alice@mercury:~/flocker-tutorial$ $ mongo 172.16.255.250MongoDB shell version: 2.4.9connecting to: 172.16.255.250/test> use example;switched to db example> db.records.insert({"flocker": "tested"})> db.records.find({}){ "_id" : ObjectId("53c958e8e571d2046d9b9df9"), "flocker" : "tested" }

We can also connect to the other node where it isn’t running and the traffic will get routed to the correct node:

alice@mercury:~/flocker-tutorial$ mongo 172.16.255.251MongoDB shell version: 2.4.9connecting to: 172.16.255.251/test> use example;switched to db example> db.records.find({}){ "_id" : ObjectId("53c958e8e571d2046d9b9df9"), "flocker" : "tested" }

Since the application is transparently accessible from both nodes you can configure a DNS record that points at bothIPs and access the application regardless of its location. See Setting up External Routing for more details.

At this point you have successfully deployed a MongoDB server and communicated with it. You’ve also seen howexternal users don’t need to worry about applications’ location within the cluster. In the next section of the tutorialyou’ll learn how to ensure that the application’s data moves along with it, the final step to running stateful applicationson a cluster.

2.2.4 Data Volumes

The Problem

By default moving an application from one node to another does not move its data along with it. Before proceedinglet’s see in more detail what the problem is by continuing the Exposing Ports example.

Recall that we inserted some data into the database. Next we’ll use a new configuration file that moves the applicationto a different node.

port-deployment-moved.yml




"172.16.255.250": []"172.16.255.251": ["mongodb-port-example"]

alice@mercury:~/flocker-tutorial$ flocker-deploy port-deployment-moved.yml port-application.ymlalice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES4d117c7e653e clusterhq/mongodb:latest mongod 2 seconds ago Up 1 seconds 27017/tcp, 28017/tcp mongodb-port-examplealice@mercury:~/flocker-tutorial$

If we query the database the records we’ve previously inserted have disappeared! The application has moved but thedata has been left behind.

alice@mercury:~/flocker-tutorial$ mongo 172.16.255.251MongoDB shell version: 2.4.9connecting to: 172.16.255.251/test> use example;switched to db example> db.records.find({})>

The Solution

Unlike many other Docker frameworks Flocker has a solution for this problem, a ZFS-based volume manager. Anapplication with a Flocker volume configured will move the data along with the application, transparently and with noadditional intervention on your part.

We’ll create a new configuration for the cluster, this time adding a volume to the MongoDB container.

volume-application.yml


"mongodb-volume-example":"image": "clusterhq/mongodb""ports":- "internal": 27017

"external": 27017"volume":

# The location within the container where the data volume will be# mounted:"mountpoint": "/data/db"

volume-deployment.yml


"172.16.255.250": ["mongodb-volume-example"]"172.16.255.251": []

Then we’ll run these configuration files with flocker-deploy:

alice@mercury:~/flocker-tutorial$ flocker-deploy volume-deployment.yml volume-application.ymlalice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES4d117c7e653e clusterhq/mongodb:latest mongod 2 seconds ago Up 1 seconds 27017/tcp, 28017/tcp mongodb-volume-examplealice@mercury:~/flocker-tutorial$

Once again we’ll insert some data into the database:



alice@mercury:~/flocker-tutorial$ $ mongo 172.16.255.250MongoDB shell version: 2.4.9connecting to: 172.16.255.250/test> use example;switched to db example> db.records.insert({"the data": "it moves"})> db.records.find({}){ "_id" : ObjectId("53d80b08a3ad4df94a2a72d6"), "the data" : "it moves" }

Next we’ll move the application to the other node.

volume-deployment-moved.yml


"172.16.255.250": []"172.16.255.251": ["mongodb-volume-example"]

alice@mercury:~/flocker-tutorial$ flocker-deploy volume-deployment-moved.yml volume-application.ymlalice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES4d117c7e653e clusterhq/mongodb:latest mongod 2 seconds ago Up 1 seconds 27017/tcp, 28017/tcp mongodb-volume-examplealice@mercury:~/flocker-tutorial$

This time however the data has moved with the application:

alice@mercury:~/flocker-tutorial$ mongo 172.16.255.251MongoDB shell version: 2.4.9connecting to: 172.16.255.251/test> use example;switched to db example> db.records.find({}){ "_id" : ObjectId("53d80b08a3ad4df94a2a72d6"), "the data" : "it moves" }

At this point you have successfully deployed a MongoDB server and communicated with it. You’ve also seen howFlocker allows you to move an application’s data to different locations in a cluster as the application is moved. Younow know how to run stateful applications in a Docker cluster using Flocker.

The virtual machines you are running will be useful for testing Flocker and running other examples in the documenta-tion. If you would like to shut them down temporarily you can run vagrant halt in the tutorial directory. You canthen restart them by running vagrant up. If you would like to completely remove the virtual machines you can runvagrant destroy.

2.3 Flocker Application Examples

You can find below examples of how to deploy some common applications with Flocker. Each example includesinstructions and Flocker configuration files to download that can be used immediately with the virtual machines createdin the MongoDB tutorial.

2.3.1 Using Environment Variables

MySQL Example

Flocker supports passing environment variables to a container via its Application Configuration. This example willuse a configured environment variable to set the root user password for a MySQL service running inside a container.

2.3. Flocker Application Examples 17


Create the Virtual Machines

You can reuse the Virtual Machines defined in the Vagrant configuration for the MongoDB tutorial. If you have sinceshutdown or destroyed those VMs, boot them up again:

alice@mercury:~/flocker-tutorial$ vagrant upBringing machine ’node1’ up with ’virtualbox’ provider...==> node1: Importing base box ’clusterhq/flocker-dev’...

Download the Docker Image

The Docker image used by this example is quite large, so you should pre-fetch it to your nodes.

alice@mercury:~/flocker-tutorial$ ssh -t [email protected] docker pull mysql:5.6.17...alice@mercury:~/flocker-tutorial$ ssh -t [email protected] docker pull mysql:5.6.17...alice@mercury:~/flocker-tutorial$

Note: The mysql:5.6.17 Docker image is used in this example for compatibility with ZFS. Newer versions ofthe MySQL Docker image enable asynchronous I/O, which is not yet supported by ZFS on Linux.

Launch MySQL

Download and save the following configuration files to the flocker-tutorial directory:

mysql-application.yml


"mysql-volume-example":"image": "mysql:5.6.17""environment":

"MYSQL_ROOT_PASSWORD": "clusterhq""ports":- "internal": 3306


"mountpoint": "/var/lib/mysql"

mysql-deployment.yml


"172.16.255.250": ["mysql-volume-example"]"172.16.255.251": []

Now run flocker-deploy to deploy the MySQL application to the target Virtual Machine.

alice@mercury:~/flocker-tutorial$ flocker-deploy mysql-deployment.yml mysql-application.ymlalice@mercury:~/flocker-tutorial$



Connect to MySQL & Insert Sample Data

You can now use the mysql client on the host machine to connect to the MySQL server running inside the container.Connect using the client to the IP address of the Virtual Machine. In this case the example has exposed the defaultMySQL port 3306 so it is not required to specify a connection port on the command line:

alice@mercury:~/flocker-tutorial$ mysql -h172.16.255.250 -uroot -pclusterhq

Welcome to the MySQL monitor. Commands end with ; or \g....mysql> CREATE DATABASE example;Query OK, 1 row affected (0.00 sec)

mysql> USE example;Database changedmysql> CREATE TABLE ‘testtable‘ (‘id‘ INT NOT NULL AUTO_INCREMENT,‘name‘ VARCHAR(45) NULL,PRIMARY KEY (‘id‘)) ENGINE = MyISAM;Query OK, 0 rows affected (0.05 sec)

mysql> INSERT INTO ‘testtable‘ VALUES(’’,’flocker test’);Query OK, 1 row affected, 1 warning (0.01 sec)

mysql> quitBye

alice@mercury:~/flocker-tutorial$

Create a New Deployment Configuration and Move the Application

Download and save the following configuration file to your flocker-tutorial directory:

mysql-deployment-moved.yml


"172.16.255.250": []"172.16.255.251": ["mysql-volume-example"]

Then run flocker-deploy to move the MySQL application along with its data to the new destination host:

alice@mercury:~/flocker-tutorial$ flocker-deploy mysql-deployment-moved.yml mysql-application.ymlalice@mercury:~/flocker-tutorial$

Verify Data Has Moved

Confirm the application has moved to the target Virtual Machine:

alice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES51b5b09a46bb mysql:5.6.17 /bin/sh -c /init 7 seconds ago Up 6 seconds 0.0.0.0:3306->3306/tcp mysql-volume-examplealice@mercury:~/flocker-tutorial$

And is no longer running on the original host:




You can now connect to MySQL on its host and confirm the sample data has also moved:


Welcome to the MySQL monitor. Commands end with ; or \g....mysql> SHOW DATABASES;+--------------------+| Database |+--------------------+| information_schema || example || mysql || performance_schema |+--------------------+4 rows in set (0.02 sec)

mysql> USE example;Reading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -A

Database changedmysql> SELECT * FROM ‘testtable‘;+----+--------------+| id | name |+----+--------------+| 1 | flocker test |+----+--------------+1 row in set (0.01 sec)

mysql>

This concludes the MySQL example.

2.3.2 Running PostgreSQL






alice@mercury:~/flocker-tutorial$ ssh -t [email protected] docker pull postgres...alice@mercury:~/flocker-tutorial$ ssh -t [email protected] docker pull postgres...alice@mercury:~/flocker-tutorial$



Launch PostgreSQL

Download and save the following configuration files to your flocker-tutorial directory:

postgres-application.yml


"postgres-volume-example":"image": "postgres""ports":- "internal": 5432


# The location within the container where the data volume will be# mounted; see https://github.com/docker-library/postgres/blob/docker/Dockerfile.template"mountpoint": "/var/lib/postgresql/data"

postgres-deployment.yml


"172.16.255.250": ["postgres-volume-example"]"172.16.255.251": []

Now run flocker-deploy to deploy the PostgreSQL application to the target Virtual Machine.

alice@mercury:~/flocker-tutorial$ flocker-deploy postgres-deployment.yml postgres-application.ymlalice@mercury:~/flocker-tutorial$

Confirm the container is running in its destination host:

alice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESf6ee0fbd0446 postgres:latest /bin/sh -c /init 7 seconds ago Up 6 seconds 0.0.0.0:5432->5432/tcp postgres-volume-examplealice@mercury:~/flocker-tutorial$

Connect to PostgreSQL

You can now use the psql client on the host machine to connect to the PostgreSQL server running inside the container.Connect using the client to the IP address of the Virtual Machine, using the port number exposed in the applicationconfiguration:

alice@mercury:~/flocker-tutorial$ psql postgres --host 172.16.255.250 --port 5432 --username postgrespsql (9.3.5)Type "help" for help.

postgres=#

This verifies the PostgreSQL service is successfully running inside its container.

Insert a Row into the Database

postgres=# CREATE DATABASE flockertest;CREATE DATABASEpostgres=# \connect flockertest;psql (9.3.5)



You are now connected to database "flockertest" as user "postgres".flockertest=# CREATE TABLE testtable (testcolumn int);CREATE TABLEflockertest=# INSERT INTO testtable (testcolumn) VALUES (3);INSERT 0 1flockertest=# SELECT * FROM testtable;testcolumn

------------3

(1 row)

flockertest=# \quit

Move the Application


postgres-deployment-moved.yml


"172.16.255.250": []"172.16.255.251": ["postgres-volume-example"]

Then run flocker-deploy to move the PostgreSQL application along with its data to the new destination host:

alice@mercury:~/flocker-tutorial$ flocker-deploy postgres-deployment-moved.yml postgres-application.ymlalice@mercury:~/flocker-tutorial$



alice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES51b5b09a46bb clusterhq/postgres:latest /bin/sh -c /init 7 seconds ago Up 6 seconds 0.0.0.0:5432->5432/tcp postgres-volume-examplealice@mercury:~/flocker-tutorial$



You can now connect to PostgreSQL on its host and confirm the sample data has also moved:

alice@mercury:~/flocker-tutorial$ psql postgres --host 172.16.255.251 --port 5432 --username postgrespsql (9.3.5)Type "help" for help.

postgres=# \connect flockertest;psql (9.3.5)You are now connected to database "flockertest" as user "postgres".flockertest=# select * from testtable;testcolumn

------------



3(1 row)

This concludes the PostgreSQL example.

2.3.3 Linking Containers

Elasticsearch, Logstash & Kibana

Flocker provides functionality similar to Docker Container Linking. In this example you will learn how to deployElasticSearch, Logstash, and Kibana with Flocker, demonstrating how applications running in separateDocker containers can be linked together such that they can connect to one another, even when they are deployed onseparate nodes.

The three applications are connected as follows:

• Logstash receives logged messages and relays them to ElasticSearch.

• ElasticSearch stores the logged messages in a database.

• Kibana connects to ElasticSearch to retrieve the logged messages and present them in a web interface.




Download the Docker Images

The Docker images used by this example are quite large, so you should pre-fetch them to your nodes.

alice@mercury:~/flocker-tutorial$ ssh -t [email protected] docker pull clusterhq/elasticsearch...alice@mercury:~/flocker-tutorial$ ssh -t [email protected] docker pull clusterhq/logstash...alice@mercury:~/flocker-tutorial$ ssh -t [email protected] docker pull clusterhq/kibana...alice@mercury:~/flocker-tutorial$


Deploy on Node1



http://docs.docker.com/userguide/dockerlinks/


elk-application.yml


"elasticsearch":"image": "clusterhq/elasticsearch""ports":- "internal": 9200


"mountpoint": "/var/lib/elasticsearch/""logstash":"image": "clusterhq/logstash""ports":- "internal": 5000

"external": 5000"links":- "local_port": 9200

"remote_port": 9200"alias": "es"

"kibana":"image": "clusterhq/kibana""ports":- "internal": 8080

"external": 80

elk-deployment.yml


"172.16.255.250": ["elasticsearch", "logstash", "kibana"]"172.16.255.251": []

Run flocker-deploy to start the three applications:

alice@mercury:~/flocker-tutorial$ flocker-deploy elk-deployment.yml elk-application.ymlalice@mercury:~/flocker-tutorial$

Connect to Kibana

Browse to port 80 on Node1 (http://172.16.255.250:80) with your web browser. You should see the Kibana webinterface but there won’t be any messages yet.


http://172.16.255.250:80


Generate Sample Log Messages

Use telnet to connect to the Logstash service running in the Virtual Machine and send some sample JSON data.

alice@mercury:~/flocker-tutorial$ telnet 172.16.255.250 5000{"firstname": "Joe", "lastname": "Bloggs"}{"firstname": "Fred", "lastname": "Bloggs"}^]

telnet> quitConnection closed.alice@mercury:~/flocker-tutorial$

Now refresh the Kibana web interface and you should see those messages.



Move ElasticSearch to Node2



"172.16.255.250": ["logstash", "kibana"]"172.16.255.251": ["elasticsearch"]

Then run flocker-deploy to move the Elasticsearch application along with its data to the new destinationhost:


Now verify that the ElasticSearch application has moved to the other VM:

alice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES894d1656b74d clusterhq/elasticsearch:latest /bin/sh -c ’source / 2 minutes ago Up 2 minutes 9300/tcp, 0.0.0.0:9200->9200/tcp elasticsearchalice@mercury:~/flocker-tutorial$


alice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESabc5c08557d4 clusterhq/kibana:latest /usr/bin/twistd -n w 45 minutes ago Up 45 minutes 0.0.0.0:80->8080/tcp kibana



44a4ee72d9ab clusterhq/logstash:latest /bin/sh -c /usr/loca 45 minutes ago Up 45 minutes 0.0.0.0:5000->5000/tcp logstashalice@mercury:~/flocker-tutorial$

Now if you refresh the Kibana web interface, you should see the log messages that were logged earlier.

This concludes the Elasticsearch-Logstash-Kibana example. Read more about linking containers in ourConfiguring Flocker documentation.

2.4 Flocker Feature Examples

You can find below examples of how to deploy some common applications with Flocker. Each example includesinstructions and Flocker configuration files to download that can be used immediately with the virtual machines createdin the MongoDB tutorial.

2.4.1 Using Environment Variables

MySQL Example

Flocker supports passing environment variables to a container via its Application Configuration. This example willuse a configured environment variable to set the root user password for a MySQL service running inside a container.






alice@mercury:~/flocker-tutorial$ ssh -t [email protected] docker pull mysql:5.6.17...alice@mercury:~/flocker-tutorial$ ssh -t [email protected] docker pull mysql:5.6.17...alice@mercury:~/flocker-tutorial$

Note: The mysql:5.6.17 Docker image is used in this example for compatibility with ZFS. Newer versions ofthe MySQL Docker image enable asynchronous I/O, which is not yet supported by ZFS on Linux.

Launch MySQL


mysql-application.yml

2.4. Flocker Feature Examples 27



"mysql-volume-example":"image": "mysql:5.6.17""environment":

"MYSQL_ROOT_PASSWORD": "clusterhq""ports":- "internal": 3306


"mountpoint": "/var/lib/mysql"

mysql-deployment.yml


"172.16.255.250": ["mysql-volume-example"]"172.16.255.251": []

Now run flocker-deploy to deploy the MySQL application to the target Virtual Machine.

alice@mercury:~/flocker-tutorial$ flocker-deploy mysql-deployment.yml mysql-application.ymlalice@mercury:~/flocker-tutorial$

Connect to MySQL & Insert Sample Data

You can now use the mysql client on the host machine to connect to the MySQL server running inside the container.Connect using the client to the IP address of the Virtual Machine. In this case the example has exposed the defaultMySQL port 3306 so it is not required to specify a connection port on the command line:


Welcome to the MySQL monitor. Commands end with ; or \g....mysql> CREATE DATABASE example;Query OK, 1 row affected (0.00 sec)

mysql> USE example;Database changedmysql> CREATE TABLE ‘testtable‘ (‘id‘ INT NOT NULL AUTO_INCREMENT,‘name‘ VARCHAR(45) NULL,PRIMARY KEY (‘id‘)) ENGINE = MyISAM;Query OK, 0 rows affected (0.05 sec)

mysql> INSERT INTO ‘testtable‘ VALUES(’’,’flocker test’);Query OK, 1 row affected, 1 warning (0.01 sec)

mysql> quitBye

alice@mercury:~/flocker-tutorial$

Create a New Deployment Configuration and Move the Application


mysql-deployment-moved.yml




"172.16.255.250": []"172.16.255.251": ["mysql-volume-example"]

Then run flocker-deploy to move the MySQL application along with its data to the new destination host:

alice@mercury:~/flocker-tutorial$ flocker-deploy mysql-deployment-moved.yml mysql-application.ymlalice@mercury:~/flocker-tutorial$



alice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES51b5b09a46bb mysql:5.6.17 /bin/sh -c /init 7 seconds ago Up 6 seconds 0.0.0.0:3306->3306/tcp mysql-volume-examplealice@mercury:~/flocker-tutorial$



You can now connect to MySQL on its host and confirm the sample data has also moved:


Welcome to the MySQL monitor. Commands end with ; or \g....mysql> SHOW DATABASES;+--------------------+| Database |+--------------------+| information_schema || example || mysql || performance_schema |+--------------------+4 rows in set (0.02 sec)

mysql> USE example;Reading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -A

Database changedmysql> SELECT * FROM ‘testtable‘;+----+--------------+| id | name |+----+--------------+| 1 | flocker test |+----+--------------+1 row in set (0.01 sec)

mysql>

This concludes the MySQL example.



2.4.2 Linking Containers

Elasticsearch, Logstash & Kibana

Flocker provides functionality similar to Docker Container Linking. In this example you will learn how to deployElasticSearch, Logstash, and Kibana with Flocker, demonstrating how applications running in separateDocker containers can be linked together such that they can connect to one another, even when they are deployed onseparate nodes.

The three applications are connected as follows:

• Logstash receives logged messages and relays them to ElasticSearch.

• ElasticSearch stores the logged messages in a database.

• Kibana connects to ElasticSearch to retrieve the logged messages and present them in a web interface.




Download the Docker Images

The Docker images used by this example are quite large, so you should pre-fetch them to your nodes.



Deploy on Node1


elk-application.yml


"elasticsearch":"image": "clusterhq/elasticsearch"


http://docs.docker.com/userguide/dockerlinks/


"ports":- "internal": 9200


"mountpoint": "/var/lib/elasticsearch/""logstash":"image": "clusterhq/logstash""ports":- "internal": 5000

"external": 5000"links":- "local_port": 9200

"remote_port": 9200"alias": "es"

"kibana":"image": "clusterhq/kibana""ports":- "internal": 8080

"external": 80

elk-deployment.yml


"172.16.255.250": ["elasticsearch", "logstash", "kibana"]"172.16.255.251": []

Run flocker-deploy to start the three applications:


Connect to Kibana

Browse to port 80 on Node1 (http://172.16.255.250:80) with your web browser. You should see the Kibana webinterface but there won’t be any messages yet.


http://172.16.255.250:80


Generate Sample Log Messages

Use telnet to connect to the Logstash service running in the Virtual Machine and send some sample JSON data.

alice@mercury:~/flocker-tutorial$ telnet 172.16.255.250 5000{"firstname": "Joe", "lastname": "Bloggs"}{"firstname": "Fred", "lastname": "Bloggs"}^]

telnet> quitConnection closed.alice@mercury:~/flocker-tutorial$

Now refresh the Kibana web interface and you should see those messages.

Move ElasticSearch to Node2



"172.16.255.250": ["logstash", "kibana"]"172.16.255.251": ["elasticsearch"]

Then run flocker-deploy to move the Elasticsearch application along with its data to the new destinationhost:




Now verify that the ElasticSearch application has moved to the other VM:

alice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES894d1656b74d clusterhq/elasticsearch:latest /bin/sh -c ’source / 2 minutes ago Up 2 minutes 9300/tcp, 0.0.0.0:9200->9200/tcp elasticsearchalice@mercury:~/flocker-tutorial$


alice@mercury:~/flocker-tutorial$ ssh [email protected] docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESabc5c08557d4 clusterhq/kibana:latest /usr/bin/twistd -n w 45 minutes ago Up 45 minutes 0.0.0.0:80->8080/tcp kibana44a4ee72d9ab clusterhq/logstash:latest /bin/sh -c /usr/loca 45 minutes ago Up 45 minutes 0.0.0.0:5000->5000/tcp logstashalice@mercury:~/flocker-tutorial$

Now if you refresh the Kibana web interface, you should see the log messages that were logged earlier.

This concludes the Elasticsearch-Logstash-Kibana example. Read more about linking containers in ourConfiguring Flocker documentation.


CHAPTER 3

Advanced Documentation

3.1 What’s New

Note: If you already have a tutorial environment from a previous release see Upgrading the Vagrant Environment.

3.1.1 v0.2

• Moving volumes between nodes is now done with a two-phase push that should dramatically decrease applica-tion downtime when moving large amounts of data.

• Added support for environment variables in the application configuration.

• Added basic support for links between containers in the application configuration.

3.1.2 v0.1

Everything is new since this is our first release.

3.1.3 Known Limitations

• This release is not ready for production and should not be used on publicly accessible servers or to store datayou care about. Backwards compatibility is not a goal yet.

• Changes to the application configuration file will often not be noticed by flocker-deploy, and there is noway to delete applications or volumes. Choose new names for your applications if you are making changes tothe application configuration.

You can learn more about where we might be going with future releases by:

• Stopping by the #clusterhq channel on irc.freenode.net.

• Visiting our GitHub repository at https://github.com/ClusterHQ/flocker.

• Reading Areas of Potential Future Development.

35

https://github.com/ClusterHQ/flocker


3.2 Using Flocker

Flocker manages which containers are running and on what hosts. It also manages network configuration for thesecontainers (between them and between containers and the world). And Flocker also creates and replicates volumes.All of this functionality is available via a simple invocation of the flocker-deploy program. This program isincluded in the flocker-cli package. If you haven’t installed that package yet, you may want to do so now.

3.2.1 Command Line Arguments

flocker-deploy takes just two arguments. The first of these is the path to a deployment configuration file. Thesecond is the path to an application configuration file.

$ flocker-deploy clusterhq_deployment.yml clusterhq_app.yml

The contents of these two configuration files determine what actions Flocker actually takes. The configuration filescompletely control this; there are no other command line arguments or options. See Configuring Flocker for detailsabout these two files.

You can run flocker-deploy anywhere you have it installed. The containers you are managing do not need to berunning on the same host as flocker-deploy.

3.2.2 Authentication

Setup

flocker-deploy lets you manage containers on one or more hosts. Before flocker-deploy can do this itneeds to be able to authenticate itself to these hosts. Flocker uses SSH to communicate with the hosts you specifyin the deployment configuration file. It requires that you configure SSH access to the root user in advance. Therecommended configuration is to generate an SSH key (if you don’t already have one):

$ ssh-keygen

Then add it to your SSH key agent:

$ ssh-add <path to key file>

Finally add it to the authorized_keys file of each host you want to manage:

$ ssh-copy-id -i <path to key file> root@<hostname>

This will allow flocker-deploy to connect to these hosts (as long as the key is still available in your key agent).

If you have a different preferred SSH authentication configuration which allows non-interactive SSH authenticationyou may use this instead.

Other Keys

flocker-deploy will generate an additional SSH key. This key is deployed to each host you manage with Flockerand allows the hosts to authenticate to each other.

36 Chapter 3. Advanced Documentation

https://en.wikipedia.org/wiki/Ssh-keygen

https://en.wikipedia.org/wiki/Ssh-agent


3.3 Configuring Flocker

Flocker operates on two configuration files: application and deployment. Together these configurations define a de-ployment.

The configuration is represented using yaml syntax.

3.3.1 Application Configuration

The application configuration consists of a version and short, human-meaningful application names and the parametersnecessary to run those applications.

The required parameters are version and applications. For now the version must be 1.

The parameters required to define an application are:

• image

This is the name of the Docker image which will be used to start the container which will run the application.Optionally, this may include a tag using the <name>:<tag> syntax.

For example, an application which is meant to use version 1.0 of ClusterHQ’s flocker-dev Docker image isconfigured like this:

"image": "clusterhq/flocker-dev:v1.0"

The following parameters are optional when defining an application:

• ports

This is an optional list of port mappings to expose to the outside world. Connections to the external port on thehost machine are forwarded to the internal port in the container.

"ports":- "internal": 80"external": 8080

• links

This is an optional list of links to make to other containers, providing a mechanism by which your containers cancommunicate even when they are located on different hosts. Linking containers in Flocker works by populatinga number of environment variables in the application specifying a link.

The environment variables generated are named for the specified alias and local port, while their values will pointto the configured remote port. For example, given an application configuration containing a links section asfollows:

"links":- "local_port": 80"remote_port": 8080"alias": "apache"

The above configuration will produce environment variables in that application using the same format as gener-ated by Docker:

APACHE_PORT_80_TCP=tcp://example.com:8080APACHE_PORT_80_TCP_PROTO=tcpAPACHE_PORT_80_TCP_ADDR=example.comAPACHE_PORT_80_TCP_PORT=8080

3.3. Configuring Flocker 37


Warning: As you may have noticed in the example above, unlike Docker links, the destination port will notbe the port used to create the environment variable names. Flocker implements linking via the ports exposedto the network, whereas Docker creates an internal tunnel between linked containers, an approach that is notcompatible with the deployment of links across multiple nodes.

Note: Only TCP links are supported by Flocker, therefore the TCP portion of the environment variable namesand the tcp value of the _PROTO and _TCP variables are not configurable.

• volume

This specifies that the application container requires a volume. It also allows you to specify where in thecontainer the volume will be mounted via the mountpoint key. The value for this key must be a string givingan absolute path.

"volume":"mountpoint": "/var/www/data"

• environment

This is an optional mapping of key/value pairs for environment variables that will be applied to the applicationcontainer. Keys and values for environment variables must be strings and only ASCII characters are supportedat this time.

"environment":"foo": "bar""baz": "qux"

Here’s an example of a simple but complete configuration defining one application:


"site-clusterhq.com":"image": "clusterhq/clusterhq-website""environment":

"WP_ADMIN_USERNAME": "administrator""WP_ADMIN_PASSWORD": "password"

"ports":- "internal": 80


"mountpoint": "/var/mysql/data"

3.3.2 Deployment Configuration

The deployment configuration specifies which applications are run on what nodes. It consists of a version and amapping from node names to application names.

The required parameters are version and applications. For now the version must be 1.

Here’s an example of a simple but complete configuration defining a deployment of one application on one host:


"node017.example.com":"site-clusterhq.com"



3.4 Volume Manager

Flocker comes with a volume manager, a tool to manage volumes that can be attached to Docker containers. Ofparticular note is the ability to push volumes to different machines.

3.4.1 Configuration

Each host in a Flocker cluster has a universally unique identifier (UUID) for its volume manager. By default the UUIDis stored in /etc/flocker/volume.json.

The volume manager stores volumes inside a ZFS pool called flocker.

3.4.2 Volume Ownership

Each volume is owned by a specific volume manager and only that volume manager can write to it. To begin with avolume is owned by the volume manager that created it. A volume manager can push volumes it owns to anothermachine, copying the volume’s data to a remote volume manager. The copied volume on that remote volume managerwill continue to be owned by the local volume manager, and therefore the remote volume manager will not be ableto write to it. A volume manager can also handoff a volume to a remote volume manager, i.e. transfer ownership.The remote volume manager becomes the owner of the volume and subsequently it is able to write to the volume. Thevolume manager that did the handoff ceases to own the volume and subsequently is not allowed to write to the volume.

Volumes are mounted read-write by the manager which owns them. They are mounted read-only by any other managerwhich has a copy.

3.4.3 Implementation Details

Each volume is a ZFS dataset. Volumes are created with three parameters:

• The UUID of the volume manager that owns the volume. The creating volume manager’s UUID (see above) isused to supply a value for this parameter.

• The logical name; this must be the same as the name of the container it will be mounted in. For example, for acontainer named "myapp-mongodb" a volume called "myapp-mongodb" will be created.

• A mount path, indicating where within a container the volume will be mounted. For example, for a MongoDBserver this would be "/var/lib/mongodb" since that is where MongoDB stores its data.

The ZFS dataset name is a combination of the UUID and the logical name, e.g. 1234.myapp-mongodb.

Docker Integration

Volumes are exposed to Docker by creating a container with a "-data" suffix that mounts the volume inthe appropriate location. For example, if you create a volume called "myapp-mongodb" with mount point"/var/lib/mongodb" then a container called "myapp-mongodb-data" will be created that has the volumemounted at that path.

You can then use this volume manually using --volumes-from:

$ docker run --volumes-from myapp-mongodb-data --name myapp-mongodb openshift/centos-mongodb

The myapp-mongodb container will now have a volume mounted at /var/lib/mongodb pointing at the ZFSdataset managed by Flocker.

3.4. Volume Manager 39


Even easier, geard and therefore the Flocker orchestration system will automatically mount volumes from"myapp-mongodb-data" if you create a unit called "myapp-mongodb".

Push and Handoff

Push and handoffs are currently done over SSH between nodes, with ad hoc calls to the flocker-volumecommand-line tool. In future releases this will be switched to a real protocol and later on to communication betweenlong-running daemons rather than short-lived scripts. (See #154.)

When a volume is pushed a zfs send is used to serialize its data for transmission to the remote machine, whichdoes a zfs receive to decode the data and create or update the corresponding ZFS dataset.

If the sending node determines that it has a snapshot of the volume in common with the receiving node (as determinedusing flocker-volume snapshot) then it will construct an incremental data stream based on that snapshot.This can drastically reduce the amount of data that needs to be transferred between the two nodes.

Handoff involves renaming the ZFS dataset to change the owner UUID encoded in the dataset name. For example,imagine two volume managers with UUIDs 1234 and 5678 and a dataset called mydata.

Action Volume Manager 1234 Volume Manager 5678

1. Create mydata on 12341234.mydata (owner)

2. Push mydata to 56781234.mydata (owner) 1234.mydata

3. Handoff mydata to 56785678.mydata 5678.mydata (owner)

3.5 Data-Oriented Clustering

3.5.1 Minimal Downtime Volume Migration

Flocker’s cluster management logic uses the volume manager (see Volume Manager) to efficiently move containers’data between nodes.

Consider a MongoDB application with a 20GB volume being moved from node A to node B. The naive implementationwould be:

1. Shut down MongoDB on node A.

2. Push all 20GB of data to node B with no database running.

3. Hand off ownership of the volume to node B, a quick operation.

4. Start MongoDB on node B.

This method would cause significant downtime.

Instead Flocker uses a superior two-phase push:

1. Push the full 20GB of data in the volume from node A to a node B. Meanwhile MongoDB continues to run onnode A.

2. Shut down MongoDB on node A.

3. Push only the changes that were made to the volume since the last push happened. This will likely be orders ofmagnitude less than 20GB, depending on what database activity happened in the interim.


https://github.com/ClusterHQ/flocker/issues/154


4. Hand off ownership of the volume to node B, a quick operation.

5. Start MongoDB on node B.

MongoDB is only unavailable during the time it takes to push the incremental changes from node A to node B.

3.6 Setting up External Routing

Flocker allows you to expose public ports on your applications. For example, you can export port 8443 on an HTTPSserver running inside a container as an externally visible port 443 on the host machine. Because Flocker runs on acluster of nodes your web application might run on different nodes at different times. You could update the DNSrecord every time a container moves. However, updating DNS records can take anywhere from a minute to a fewhours to take effect for all clients so this will impact your application’s availability. This is where Flocker’s routingfunctionality comes in handy.

When an external route is configured (e.g. on port 443) Flocker routes that port on all nodes to the node where yourapplication is running. You can therefore move a node and then change your DNS configuration appropriately withoutincurring any downtime.

3.6.1 No-Change DNS Configuration

What’s more it is also possible to configure your DNS records in such a way that no DNS changes are necessary whenapplications move to different nodes. Specifically, the DNS record for your application should be configured to pointat all IPs in the cluster.

For example, consider the following setup:

www.example.com has a DNS record pointing at two different nodes’ IP. Every time you connect towww.example.com your browser will choose one of the two IPs at random.

• If you connect to port 80 on the node2 — which is not hosting the container — the traffic will be routed on tonode1.

• If you connect to port 80 on node1 you will reach the web server that is listening on port 8080 within acontainer.

Note that if nodes are in different data centers and you pay for bandwidth this configuration will require you to pay forforwarded traffic between nodes.

3.7 Debugging

3.7.1 Logging

The Flocker processes running on the nodes will write their logs to /var/log/flocker/. The log files are named<processname>-<pid>.log, e.g. flocker-volume-1234.log.

Logs from the Docker containers are written to systemd’s journal with a unit name constructed with a ctr- prefix. Forexample if you’ve started an application called mymongodb you can view its logs by running the following commandon the node where the application was started:

$ journalctl -u ctr-mymongodb

3.6. Setting up External Routing 41

http://www.freedesktop.org/software/systemd/man/journalctl.html


3.8 Cleaning Up

Flocker does not currently implement a tool to purge containers and state from deployment nodes that have hadapplications and volumes installed via flocker-deploy. Adding a cleanup tool is on the Flocker developmentpath for a later release.

Until this feature is available, you may wish to manually purge deployment nodes of all containers and state createdby Flocker. This will enable you to test, play around with Flocker or repeat the deployment process (for example, ifyou have followed through the tutorial and would like to clean up the virtual machines to start again without having todestroy and rebuild them).

Note: This process will destroy all applications and their associated data deployed by Flocker on the target node. Inaddition, the verbatim commands documented below will destroy all Docker containers and Geard units on the targetnode, regardless of whether or not they were deployed via Flocker. Proceed at your own risk and only if you fullyunderstand the effects of executing these commands.

You can run the necessary cleanup commands via SSH. The tutorial’s virtual machines are created with IP addresses172.16.255.250 and 172.16.255.251. Be sure to replace the example IP address in the commands belowwith the actual IP address of the node you wish to purge.

3.8.1 Stopping Containers

Docker containers must be stopped before they can be removed.

alice@mercury:~/flocker-mysql$ ssh [email protected] ’docker ps -q | xargs --no-run-if-empty docker stop’

3.8.2 Removing Containers

alice@mercury:~/flocker-mysql$ ssh [email protected] ’docker ps -aq | xargs --no-run-if-empty docker rm’

These commands list the ID numbers of all the Docker containers on each host, including stopped containers and thenpipes each ID to the docker rm command to purge.

3.8.3 Removing Gear Units

alice@mercury:~/flocker-mysql$ ssh [email protected] gear purge

3.8.4 Removing ZFS Volumes

To remove ZFS volumes created by Flocker, you can list the volumes on each host and then use the unique IDs inconjunction with the zfs destroy command.

alice@mercury:~/flocker-mysql$ ssh [email protected] ’zfs list -H -o name’flockerflocker/e16d5b2b-471d-4bbe-be23-d58bbc8f1b94.mysql-volume-examplealice@mercury:~/flocker-mysql$ ssh [email protected] ’zfs destroy -r flocker/e16d5b2b-471d-4bbe-be23-d58bbc8f1b94.mysql-volume-example’

Alternatively if you wish to destroy all data sets created by Flocker, you can run the following command:

alice@mercury:~/flocker-mysql$ zfs destroy -r flocker




CHAPTER 4

Getting Involved

4.1 Contributing to Flocker

4.1.1 Introduction

ClusterHQ develops software using a variation of the Ultimate Quality Development System.

• Each unit of work is defined in an issue in the issue tracker and developed on a branch.

• Code is written using test-driven development.

• The issue is closed by merging the branch (via a GitHub pull request).

• Before a branch is merged it must pass code review.

• The code reviewer ensures that the pull request:

– Follows the coding standard (Python’s PEP 8).

– Includes appropriate documentation.

– Has full test coverage (unit tests and functional tests).

– The tests pass in the continuous integration system (Buildbot).

– Resolves the issue.

• The code reviewer can approve the pull request for merging as is, with some changes, or request changes and anadditional review.

4.1.2 Talk to Us

Have questions or need help? Besides filing a GitHub issue with feature requests or bug reports you can also join uson the #clusterhq channel on the irc.freenode.net IRC network or on the flocker-users Google Group.

4.1.3 Development Environment

• To run the complete test suite you will need ZFS, geard and Docker installed. geard requires an operatingsystem with systemd. The recommended way to get an environment with these installed is to use the includedVagrantfilewhich will create a pre-configured Fedora 20 virtual machine. Vagrant 1.6.2 or later is required.Once you have Vagrant installed (see the Vagrant documentation) you can run the following to get going:

43

https://twistedmatrix.com/trac/wiki/UltimateQualityDevelopmentSystem

http://legacy.python.org/dev/peps/pep-0008/

http://build.clusterhq.com/

https://github.com/ClusterHQ/flocker/issues

https://groups.google.com/forum/?hl=en#!forum/flocker-users

http://zfsonlinux.org

https://openshift.github.io/geard/

https://www.docker.com/

http://docs.vagrantup.com/


$ vagrant up$ vagrant ssh

• You will need Python 2.7 and a recent version of PyPy installed on your development machine.

• If you don’t already have tox on your development machine, you can install it and other development depen-dencies (ideally in a virtualenv) by doing:

$ python setup.py install .[doc,dev]

4.1.4 Running Tests

You can run all unit tests by doing:

$ tox

Functional tests require ZFS, geard and Docker to be installed and in the case of the latter two running as well. Inaddition, tox needs to be run as root:

$ sudo tox

Since these tests involve global state on your machine (filesystems, iptables, Docker containers, etc.) we recom-mend running them in the development Vagrant image.

4.1.5 Documentation

Documentation is generated using Sphinx and stored in the docs/ directory. You can build it individually by running:

$ tox -e sphinx

You can view the result by opening docs/_build/html/index.html in your browser.

4.1.6 Requirements for Contributions

1. All code must have unit test coverage and to the extent possible functional test coverage.

Use the coverage.py tool with the --branch option to generate line and branch coverage reports. Thisreport can tell you if you missed anything. It does not necessarily catch everything though. Treat it as a helperbut not the definitive indicator of success. You can also see coverage output in the Buildbot details link of yourpull request. Practice test-driven development to ensure all code has test coverage.

2. All code must have documentation.

Modules, functions, classes, and methods must be documented (even if they are private). Function parametersand object attributes must be documented (even if they are private).

3. All user-facing tools must have documentation.

Document tool usage as part of big-picture documentation. Identify useful goals the user may want to ac-complish and document tools within the context of accomplishing those goals. Documentation should be asaccessible and inclusive as possible. Avoid language and markup which assumes the ability to precisely use amouse and keyboard, or that the reader has perfect vision. Create alternative but equal documentation for thevisually impaired, for example, by using alternative text on all images. If in doubt, particularly about markupchanges, use http://achecker.ca/ and fix any “Known Problems” and “Likely Problems”.

4. Add your name (in alphabetical order) to the AUTHORS.rst file.

44 Chapter 4. Getting Involved

http://sphinx-doc.org/

http://achecker.ca/


4.1.7 Project Development Process

The core development team uses GitHub issues to track planned work. Issues are organized by release milestones, andthen by subcategories:

Backlog Issues we don’t expect to do in the release. These issues don’t have any particular category label. All issuesstart in the backlog when they are filed. The requirements for an issue must be completely specified before itcan move out of the backlog.

Design Issues that we expect to work on soon. This is indicated by a design label. A general plan for accomplishingthe requirements must be specified on the issue before it can move to the Ready state. The issue is assigned tothe developer working on the plan. When there is a proposed plan the review label is added to the issue (sothat it has both design and review).

Ready Issues that are ready to be worked on. This is indicated by a ready label. Issues can only be Ready after theyhave been in Design so they include an implementation plan. When someone starts work on an issue it is movedto the In Progress category (the ready keyword is removed and the in progress label is added).

In Progress Such issues are assigned to the developer who is currently working on them. This is indicated by an inprogress label. When the code is ready for review a new pull request is opened. The pull request is added tothe Review category.

Ready for Review An issue or pull request that includes work that is ready to be reviewed. This is indicated by areview label. Issues can either be in design review (design and review) or final review (just review). Areviewer can move a design review issue to Ready (to indicate the design is acceptable) or back to Design (toindicate it needs more work). A reviewer can move a final review issue to Approved (to indicate the work isacceptable) or back to In Progress (to indicate more work is needed).

Passed Review A pull request that has some minor problems that need addressing, and can be merged once those aredealt with and all tests pass. This is indicated by an accepted label.

Done Closed issues and pull requests.

Blocked Issues that can’t be worked on because they are waiting on some other work to be completed. This isindicated by a blocked label.

You can see the current status of all issues and pull requests by visiting https://waffle.io/clusterhq/flocker. In generalissues will move from Backlog to Design to Ready to In Progress. An in-progress issue will have a branch with theissue number in its name. When the branch is ready for review a pull request will be created in the Review category.When the branch is merged the corresponding pull requests and issues will be closed.

Steps to Contribute Code

GitHub collaborators can participate in the development workflow by changing the labels on an issue. GitHub lets non-collaborators create new issues and pull requests but it does not let them change labels. If you are not a collaboratoryou may seek out assistance from a collaborator to set issue labels to reflect the issue’s stage.

1. Pick the next issue in the Ready category. Drag it to the In Progress column in Waffle (or change the label fromready to in progress in GitHub).

2. Create a branch from master with a name including a few descriptive words and ending with the issue number,e.g. add-thingie-123.

3. Resolve the issue by making changes in the branch.

4. Submit the issue/branch for review. Create a pull request on GitHub for the branch. The pull request shouldinclude a Fixes #123 line referring to the issue that it resolves (to automatically close the issue when thebranch is merged). Make sure Buildbot indicates all tests pass.

4.1. Contributing to Flocker 45

https://waffle.io/clusterhq/flocker


5. Address any points raised by the reviewer. If a re-submission for review has been requested, change the labelfrom in progress to review in GitHub (or drag it to the Ready for Review column in Waffle) and go backto step 4.

6. Once it is approved, merge the branch into master by clicking the Merge button.

7. As a small thank you for contributing to Flocker, we’d like to send you some ClusterHQ swag. Once your pullrequest has been merged, just send an email to [email protected] with your t-shirt size, mailing addressand a phone number to be used only for filling out the shipping form. We’ll get something in the mail to you.

Steps to Contribute Reviews

1. Pick a pull request in GitHub/Waffle that is ready for review (review label/Review category).

2. Use the continuous integration information in the PR to verify the test suite is passing.

3. Verify the code satisfies the Requirements for Contribution (see above).

4. Verify the change satisfies the requirements specified on the issue.

5. Think hard about whether the code is good or bad.

6. Leave comments on the GitHub PR page about any of these areas where you find problems.

7. Leave a comment on the GitHub PR page explicitly approving or rejecting the change. If you accept the PRand no final changes are required then use the GitHub merge button to merge the branch. If you accept the PRbut changes are needed move it to the Review Passed column in Waffle or change its label from review toapproved. If you do not accept the PR move it to the In Progress column in Waffle or change its label fromreview to in progress.

4.2 Infrastructure

Contents:

4.2.1 Vagrant

There is a Vagrantfile in the base of the repository, that is pre-installed with all of the dependencies required torun flocker.

See the Vagrant documentation for more details.

Boxes

There are several vagrant boxes.

Development Box (vagrant/dev) The box is initialized with the yum repositories for ZFS and for dependenciesnot available in Fedora and installs all the dependencies. This is the box the Vagrantfile in the root of therepository is based on.

Tutorial Box (vagrant/tutorial) This box is initialized with the yum repositories for ZFS and Flocker, and hasFlocker pre-installed. This is the box the tutorial is based on.


mailto:[email protected]

http://docs.vagrantup.com/v2/


Building

To build one of the above boxes, run the build script in the corresponding directory. This will generate aflocker-<box>-<version>.box file.

Upload this file to Google Cloud Storage, using gsutil:

gsutil cp -a public_read flocker-dev-$(python ../../setup.py --version).box gs://clusterhq-vagrant/

(If you’re uploading the tutorial box the image will be flocker-tutorial-... instead offlocker-dev-....)

Then add a version on Vagrant Cloud (flocker-dev) or Vagrant Cloud (flocker-tutorial) as applicable. The version onVagrant Cloud should be the version with - replaced with ..

Testing

It is possible to test this image locally before uploading. First add the box locally:

vagrant box add --name clusterhq/flocker-dev flocker-dev-$(python ../../setup.py --version).box

This adds the box with version 0. Then change config.vm.box_version to = 0 in the appropriateVagrantfile, and then destroy and re-upload that vagrant image.

It is also possible to build a vagrant image based on RPMs from a branch. If you pass a branch name to build, thenit will use the RPMs from the latest build of that branch on Buildbot. In this case, the box name will not include aversion number.

4.2.2 Building RPMs

To build flocker RPMs, run the following commands:

python setup.py sdistpython setup.py generate_speccp dist/Flocker-$(python setup.py --version).tar.gz ~/rpmbuild/SOURCESsudo yum-builddep flocker.specrpmbuild -ba flocker.spec

The above commands require the rpmdevtools and yum-utils packages installed.

Flocker depends on a number of packages which aren’t available in fedora, or newer versions than are availablethere. These packages are available from our Copr repository. To enable yum to find them, put the repo file in/etc/yum.repos.d/.

4.2.3 Release Process

Outcomes

By the end of the release process we will have:

• a tag in version control

• a Python wheel in the ClusterHQ package index

• Fedora 20 RPMs for software on the node and client

• documentation on docs.clusterhq.com

4.2. Infrastructure 47

https://console.developers.google.com/project/apps~hybridcluster-docker/storage/clusterhq-vagrant/

https://developers.google.com/storage/docs/gsutil?csw=1

https://vagrantcloud.com/clusterhq/flocker-dev

https://vagrantcloud.com/clusterhq/flocker-tutorial

https://copr.fedoraproject.org/coprs/tomprince/hybridlogic/

https://copr.fedoraproject.org/coprs/tomprince/hybridlogic/repo/fedora-20-x86_64/tomprince-hybridlogic-fedora-20-x86_64.repo

http://archive.clusterhq.com

https://docs.clusterhq.com


If this is a major or minor release (i.e. not a weekly development release), we will also have:

• download links on https://clusterhq.com

Prerequisites

Software

• Fedora 20 (rpmbuild, createrepo, yumdownloader) - it may be possible to install these on Ubuntuthough.

You are advised to perform the release from a Flocker development machine , which will have all the requisitesoftware pre-installed.

• a web browser

• an up-to-date clone of the Flocker repository

• an up-to-date clone of the homebrew-tap repository

Access

• A Read the Docs account (registration), with maintainer access to the Flocker project.

• Access to Google Cloud Storage using gsutil.

Preliminary Step: Pre-populating RPM Repository

Warning: This only needs to be done if the dependency packages for Flocker (i.e. geard and Python libraries)change; it should not be done every release. If you do run this you need to do it before running the release processabove as it removes the flocker-cli etc. packages from the repository!

These steps must be performed from a machine with the ClusterHQ Copr repository installed. You caneither use the Flocker development environment or install the Copr repository locally by running curlhttps://copr.fedoraproject.org/coprs/tomprince/hybridlogic/repo/fedora-20-x86_64/tomprince-hybridlogic-fedora-20-x86_64.repo>/etc/yum.repos.d/hybridlogic.repo

mkdir repoyumdownloader --destdir=repo geard python-characteristic python-eliot python-idna python-netifaces python-service-identity python-treq python-twistedcreaterepo repogsutil cp -a public-read -R repo gs://archive.clusterhq.com/fedora/20/x86_64

mkdir srpmyumdownloader --destdir=srpm --source geard python-characteristic python-eliot python-idna python-netifaces python-service-identity python-treq python-twistedcreaterepo srpmgsutil cp -a public-read -R srpm gs://archive.clusterhq.com/fedora/20/SRPMS

Preparing for a release

1. Choose a version number:

• Release numbers should be of the form x.y.z e.g.:


https://clusterhq.com

https://github.com/ClusterHQ/flocker.git

https://github.com/ClusterHQ/homebrew-tap.git

https://readthedocs.org/accounts/signup/

https://readthedocs.org/dashboard/flocker/users/

https://console.developers.google.com/project/apps~hybridcluster-docker/storage/archive.clusterhq.com/

https://developers.google.com/storage/docs/gsutil


export VERSION=0.1.2

2. File a ticket:

(a) Set the title to “Release flocker $VERSION”

(b) Assign it to yourself

3. Create a clean, local working copy of Flocker with no modifications:

git clone [email protected]:ClusterHQ/flocker.git "flocker-${VERSION}"

4. Create a branch for the release and push it to GitHub:

git checkout -b release/flocker-${VERSION} origin/mastergit push origin --set-upstream release/flocker-${VERSION}

5. Update the version numbers in:

• the yum install line in docs/gettingstarted/linux-install.sh and

• the box_version in docs/gettingstarted/tutorial/Vagrantfile

• docs/gettingstarted/installation.rst (including the sample command output)

• Commit the changes:

git commit -am "Bumped version numbers"

6. Ensure the release notes in NEWS are up-to-date:

XXX: Process to be decided. See https://github.com/ClusterHQ/flocker/issues/523

git commit -am "Updated NEWS"

7. Ensure copyright dates in LICENSE are up-to-date:

XXX: Process to be decided. See https://github.com/ClusterHQ/flocker/issues/525

git commit -am "Updated copyright"

8. Push the changes:

git push

9. Ensure all the tests pass on BuildBot:

Go to the BuildBot web status and force a build on the just-created branch.

10. Do the acceptance tests:

XXX: See https://github.com/ClusterHQ/flocker/issues/315

Release

1. Change your working directory to be the Flocker release branch working directory.

2. Create (if necessary) and activate the Flocker release virtual environment:

mkvirtualenv flocker-release-${VERSION}pip install --editable .[release]

Note: The example above uses virtualenvwrapper but you can use virtualenv directly if you prefer.


https://github.com/ClusterHQ/flocker/blob/master/docs/gettingstarted/linux-install.sh

https://github.com/ClusterHQ/flocker/blob/master/docs/gettingstarted/tutorial/Vagrantfile

https://github.com/ClusterHQ/flocker/blob/master/docs/gettingstarted/installation.rst



http://build.clusterhq.com/boxes-flocker


https://pypi.python.org/pypi/virtualenvwrapper

https://pypi.python.org/pypi/virtualenv


3. Tag the version being released:

git tag --annotate "${VERSION}" "release/flocker-${VERSION}" -m "Tag version ${VERSION}"git push origin "${VERSION}"

4. Go to the BuildBot web status and force a build on the tag.

Note: We force a build on the tag as well as the branch because the RPMs built before pushing the tag won’thave the right version. Also, the RPM upload script currently expects the RPMs to be built from the tag, ratherthan the branch.

You force a build on a tag by putting the tag name into the branch box (without any prefix).

5. Set up gsutil authentication by following the instructions from the following command:

$ gsutil config

6. Build python packages and upload them to archive.clusterhq.com

python setup.py sdist bdist_wheelgsutil cp -a public-read \

"dist/Flocker-${VERSION}.tar.gz" \"dist/Flocker-${VERSION}-py2-none-any.whl" \gs://archive.clusterhq.com/downloads/flocker/

7. Build RPM packages and upload them to archive.clusterhq.com

admin/upload-rpms "${VERSION}"

8. Build and upload the tutorial vagrant box.

9. Build tagged docs at Read the Docs:

(a) Go to the Read the Docs dashboard.

(b) Enable the version being released.

(c) Wait for the documentation to build. The documentation will be visible athttp://docs.clusterhq.com/en/${VERSION} when it has been built.

(d) Set the default version to that version.

Warning: Skip this step for weekly releases and pre-releases. The features and documentation inweekly releases and pre-releases may not be complete and may not have been tested. We want newusers’ first experience with Flocker to be as smooth as possible so we direct them to the tutorial forthe last stable release. Other users choose to try the weekly releases, by clicking on the latest weeklyversion in the ReadTheDocs version panel.

(e) Force Read the Docs to reload the repository, in case the GitHub webhook fails, by running:

curl -X POST http://readthedocs.org/build/flocker

10. Update the Homebrew recipe

The aim of this step is to provide a version specific homebrew recipe for each release.

• Checkout the homebrew-tap repository.

git clone [email protected]:ClusterHQ/homebrew-tap.git

• Create a release branch


http://build.clusterhq.com/boxes-flocker

https://readthedocs.org/dashboard/flocker/versions/

http://docs.clusterhq.com/en



git checkout -b release/flocker-${VERSION%pre*} origin/mastergit push origin --set-upstream release/flocker-${VERSION%pre*}

• Create a flocker-{VERSION}.rb file

Copy the last recipe file and rename it for this release.

• Update recipe file

– Update the version number

The version number is included in the class name with all dots and dashes removed. e.g. classFlocker012 < Formula for Flocker-0.1.2

– Update the sha1 checksum.

sha1sum "dist/Flocker-${VERSION}.tar.gz"ed03a154c2fdcd19eca471c0e22925cf0d3925fb dist/Flocker-0.1.2.tar.gz

– Commit the changes and push

git commit -am "Bumped version number and checksum in homebrew recipe"git push

• Test the new recipe on OS X with Homebrew installed

Try installing the new recipe directly from a GitHub link

brew install https://raw.githubusercontent.com/ClusterHQ/homebrew-tap/release/flocker-${VERSION}/flocker.rb

• Make a pull request

Make a homebrew-tap pull request for the release branch against master, with a Refs #123 line in thedescription referring to the release issue that it resolves.

11. Make a pull request on GitHub for the release branch against master, with a Fixes #123 line in the de-scription referring to the release issue that it resolves.

Update Download Links

Warning: Skip this entire step for weekly releases.

XXX Update download links on https://clusterhq.com:

XXX Arrange to have download links on a page on https://clusterhq.com. See:

• https://github.com/ClusterHQ/flocker/issues/359 and

• https://www.pivotaltracker.com/n/projects/946740/stories/75538272

4.2.4 Release Schedule and Version Numbers

Goals

The goals of the release schedule are to:

• Make new features and bug fixes available to users as quickly as possible.


http://brew.sh





https://www.pivotaltracker.com/n/projects/946740/stories/75538272


• Practice releasing so that we are less likely to make mistakes.

• Improve the automation of releases through experience.

Schedule

We will make a new release of Flocker each week. This will proceed according to the Release Process. The releaseswill happen on Tuesday of each week. If nobody is available in the ClusterHQ organization to create a release, theweek will be skipped.

After each release is distributed, the engineer who performed the release will create issues for any improvementswhich could be made. The release engineer should then spend 4-8 hours working on making improvements to releaseprocess. If there is an issue that will likely take over 8 hours then they should consult the team manager before startingthem.

Version Numbers

Released version numbers take the form of X.Y.Z. The current value of X is 0 until the project is ready for production.

Y is the “marketing version”. ClusterHQ’s marketing department is made aware of the content of a release ahead oftime. If the marketing department decides that this release is sufficiently important to publicize then Y is incrementedand Z is set to 0.

Z is incremented for each standard weekly release.

Patch Releases

ClusterHQ will not be producing patch releases until the project is ready for production.


CHAPTER 5

Areas of Potential Future Development

Flocker is an ongoing project whose direction will be guided in large part by the community. The list below includessome potential areas for future development. It is not all encompassing or indicative of what definitely will be built.Feedback welcome and encouraged.

• Support for atomic updates.

• Scale-out for stateless containers.

• API to support managing Flocker volumes programmatically.

• Statically configured continuous replication and manual failover.

• No-downtime migrations between containers.

• Automatically configured continuous replication and failover.

• Multi-data center support.

• Automatically balance load across cluster.

• Roll-back a container to a snapshot.

Detailed plans have been made in some areas:

5.1 Flocker Volume Manager

The Flocker Volume Manager (FVM) provides snapshotting and replication of Flocker volumes. It has the ability topush volumes to remote nodes, track changes to those volumes, and roll them back to earlier states.

Although initially built on top of ZFS, FVM should eventually be capable of being backed on a number of filesystems.As such a generic data model is required.

5.1.1 Data Model

Motivation:

• ZFS has some peculiarities in its model when it comes to clones, e.g. promoting a clone moves snapshots fromoriginal dataset to the clone.

• Having clones be top-level constructs on the same level as originating dataset is a problem, since they are closelytied to each other both in terms of usage and in administrative “cleaning up old data” way.

• We don’t want to be too tied to the ZFS model (or terminology!) in case we want to switch to Btrfs or someother system. Especially given conflicting terminology - Btrfs “snapshots” are the same as ZFS “clones”.

53


• When it comes to replication, it is probably useful to differentiate between “data which is a copy of what theremote host has” and “local version”, in particular when divergence is a potential issue (e.g. can be caused byerroneous failover). In git you have “origin/branchname” vs. the local “branchname”, for example.

We are therefore going to be using the following model for CLI examples below:

• A “volume” is a tree of “branches”.

• A “tag” is a named read-only pointer to the contents of a branch at a given point in time; it is attached to thevolume, and is not mounted on the filesystem.

• Given volume called “mydata”, “mydata/trunk” is (by convention) is the main branch from which otherbranches originate, “mydata/branchname” is some other branch, and “mytag@mydata” is a tag.

• Branches’ full name includes the Flocker instance they came from (by default let’s say us-ing its hostname), e.g. “somehost/myvolume/trunk”. “dataset/branch” is short-hand for the current host, e.g. “thecurrenthost.example.com/dataset/branch”. Ina replication scenario we could have “remote.example.com/datavolume/trunk” and“thecurrenthost.example.com/datavolume/trunk” (aka “datavolume/trunk”) as abranch off of that.

• Local branches are mounted on the filesystem, and then exposed to Docker, e.g. “myvolume/trunk” isexported via a Docker container called “flocker:myvolume/trunk” (“flocker:” prefix is not a Dockerfeature, just a proposed convention for naming our containers).

• Remote branches are not mounted, but a local branch can be created off of them and then that is auto-mounted.

Implementation Notes - ZFS

The names of volumes, branches and tags do not map directly onto the ZFS naming system.

Each Flocker instance has a UUID, with a matching (unique across a Flocker cluster) human readable name, typicallythe hostname. We can imagine having two Flocker instances on same machine (with different pools) for testing,so don’t want to require hostname. This is the first part of the <Flocker instance UUID>/<volume UUID>/<branchname> triplet of branch names - in human-exposed CLI we probably want to use human names though, not UUIDs.Branches are known to be local if branch’s specified Flocker instance matches the UUID of the Flocker process that ismanaging it.

Volumes have UUIDs, and a matching (cluster unique?) human readable name. Tags are indicated by having asnapshot with a user attributes indicating it is a tag, the tag name and the volume name. However, not all ZFSsnapshots will be exposed as tags. E.g. the fact that a snapshot is necessary for cloning (and therefore branch creation)is an implementation detail; sometimes you want to branch off a tag, but if you want to branch off of latest version thefact that a snapshot is created needn’t be exposed.

A remote branch exists if there is a non-tag ZFS snapshot naming it, i.e. the snapshot has a user attribute indicatingwhich branch it’s on (e.g. “thathost/somevolume/abranch”).

In either case the ZFS-level snapshot name is the Flocker instance UUID + the timestamp when it was generated.

A local branch exists due to local existence ZFS dataset, one of:

1. A root dataset (“trunk”), if this is the primary host (whatever that means).

2. A clone of a remote branch snapshot.

3. A clone of a local branch snapshot.

The branch name is stored as a user attribute on the ZFS dataset. Dataset names can be the branch human readablenames, since only one Flocker instance will ever be setting them.

54 Chapter 5. Areas of Potential Future Development


In cases where we can’t use attributes the data will be in a local database of some sort. E.g. ZFS properties areinherited automatically (not the behavior we want), which might lead to some corrupt state in crashes if the low-levelAPIs don’t allow bypassing this. . .

Implementation Notes - Btrfs

Btrfs does not have a concept of clones - it just has snapshots, and they are mounted and writable. As such theproposed model should also work with Btrfs. Btrfs appears to lack promotion, but that can be emulated via renames.It’s not clear if Btrfs has the “can’t delete parent if it has children” restriction, though it may just keep around extradisk storage in that case.

5.1. Flocker Volume Manager 55


56 Chapter 5. Areas of Potential Future Development

CHAPTER 6

FAQ

• ZFS– Flocker uses ZFS. What about the ZFS licensing issues?– But if ZFS isn’t part of mainline Linux proper, it won’t benefit from rigorous testing. How do you

know it’s stable?• Future Functionality

– How does Flocker integrate with Kubernetes / Mesos / Deis / CoreOS / my favorite orchestrationframework?

– If I clone a 2GB database five times, won’t I need a really large server with 10 GB of disk?– If I clone a database five times, how does maintaining five different versions of the database work?

Flocker is under active deployment and we receive a lot of questions about how this or that will be done in a futurerelease. You can find these questions in the Future Functionality section below. You can also view ideas for futureversions of Flocker.

If you want to get involved in a discussion about a future release or have a question about Flocker today, get in touchon our Freenode IRC channel #clusterhq or the flocker Google group.

6.1 ZFS

6.1.1 Flocker uses ZFS. What about the ZFS licensing issues?

There is a good write up of the ZFS and Linux license issues on the ZFS on Linux website. In short, while ZFS won’tbe able to make it into mainline Linux proper due to licensing issues, “there is nothing in either license that preventsdistributing it in the form of a binary module or in the form of source code.”

6.1.2 But if ZFS isn’t part of mainline Linux proper, it won’t benefit from rigoroustesting. How do you know it’s stable?

ZFS on Linux is already in use in companies and institutions all over the world to the tune of hundreds of petabytes ofdata. We are also rigorously testing ZFS on Linux to make sure it is stable. ZFS is production quality code.

57

https://groups.google.com/forum/#!forum/flocker-users

http://zfsonlinux.org/faq.html#WhatAboutTheLicensingIssue


6.2 Future Functionality

6.2.1 How does Flocker integrate with Kubernetes / Mesos / Deis / CoreOS / myfavorite orchestration framework?

Over time, we hope that Flocker becomes the de facto way for managing storage volumes with your favorite orches-tration framework. We are interested in expanding libswarm to include support for filesystems and are talking with thevarious open source projects about the best way to collaborate on storage and networking for volumes. If you’d likework with us on integration, get in touch on our Freenode IRC #clusterhq or the flocker Google group. You canalso submit an issue or a pull request if you have a specific integration that you’d like to propose.

6.2.2 If I clone a 2GB database five times, won’t I need a really large server with 10GB of disk?

Thankfully no. This is where ZFS makes things really cool. Each clone is essentially free until the clone is modified.This is because ZFS is a copy-on-write filesystem, so a clone is just a set of block pointers. It’s only when a block ismodified that the data is copied, so a 2GB database that is cloned five times still just uses 2GB of disk space until acopy is modified. That means, when the database is modified, only the changes are written to disk, so your are onlystoring the net new data. This also makes it really fast to create database clones.

6.2.3 If I clone a database five times, how does maintaining five different versionsof the database work?

The idea will be that cloning the app and the database together in some sense allows the containers to maintain whatwe call independent “links” between 10 instances of the app server (deployed at different staging URLs) and therespective 10 different instances of the cloned database. This works because e.g. port 3306 inside one app server getsrouted via an ephemeral port on the host(s) to 3306 inside the corresponding specific instance of the database.

The upshot if which is that you shouldn’t need to change the apps at all, except to configure each clone with a differentURL.

58 Chapter 6. FAQ

https://groups.google.com/forum/#!forum/flocker-users

CHAPTER 7

Authors

Flocker is maintained by ClusterHQ and is licensed under the Apache Software License 2.0.

The following people and organizations contributed to its development; please add your name in alphabetical orderwith your first pull request:

• ClusterHQ (formerly Hybrid Logic Ltd.)

• Scotch Media (Base for documentation theme)

59

flocker documentation - read the docs · 2019-04-02 · flocker documentation, release 0.2.1...

Documents