container orchestrator smackdown @continouslifecycle


Run Docker & Kubernetes on Exoscale

https://github.com/exoscale/multi-master-kubernetes


How important is orchestration and what is it for?● Might not need it for small apps

● No orchestration == manual orchestration

● Manually place containers, network, scale, check, update

● Microservices & Cloud Native Applications


Design principles for Cloud Native Applications:

● Design for Performance responsiveness, concurrency, efficiency

● Design for Automation automate dev & ops tasks

● Design for Resiliency fault-tolerance, self-healing

● Design for Elasticity automatic scaling

● Design for Delivery minimise cycle-time, automate deployment

● Design for Diagnosability cluster-wide logs, traces & metrics


Let’s buy some socks...


Microservice reference application

● Intended to help people getting started with Microservices

● Great for comparing frameworks, test driving new tools...

● Inspired by the "Pet Store" for Java Frameworks

... and “TodoMVC” for JavaScript

Implementations for 10+ Cloud/Container environments:https://github.com/microservices-demo/microservices-demo/tree/master/deploy


Architecture


Comparing Orchestrators


Comparing orchestrators● All work and are improving rapidly

● Understand the differences

● Understand your requirements

● Please don't roll your own!


The players● Kubernetes

● Mesos (different workloads)

● Docker Swarm Mode

● Plus others

○ Nomad, PaaSs...


Side note - the Borg/Omega papers● Influential papers from Google● Lessons learnt from 10 years with containers● Google contributed cgroups to the Linux kernel, cgroups and linux

namespaces are the heart of containers


Docker Swarm Mode


Docker Swarm Mode

● New in Docker 1.12

● Docker Inc's official solution

● Part of core distribution

● Major improvement over TOS (“The Original Swarm”)


Core components

● Manager nodes

○ coordinate via Raft

○ no need for separate etcd/zookeeper

● Worker nodes


Usability

● Swarm extends concepts from single-node Docker to multi-node setups

● If you are up to date on standard Docker concepts, you’ll pick it up quickly

● Setting up a new Swarm is easy as pie


Easy to install

$ docker swarm initSwarm initialized: current node (10vh26gyxppo6j2vyb8rcvjwj) is now a manager.

To add a worker to this swarm, run the following command:

docker swarm join \ --token SWMTKN-1-5td5x39z8jw...ccrjmkt1o8du3 \ 172.17.9.102:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.


Secure communication by default

● TLS set up using self-signed certs

● Certificates automatically rotated


Feature Set

● Services

● Networks

● Constraints and labels


Services

● Fixed number of containers are launched together and are kept running

● Two types of services: replicated or global

○ Replicated: Maintain a specified number of containers across the cluster

○ Global: Run one instance of a container on each swarm node


Networks

● Allows creating named overlay networks...

● … which are isolated, flat, encrypted virtual networks

across your Swarm nodes to launch your containers into


● Control which node a container can be scheduled on

● E.g.:

○ Only nodes labeled staging

○ Only nodes which have the image

○ Only the node running a given container (affinity rules)

Constraints and Filters


Other features

● Spread scheduling

○ chooses "least loaded" node

○ More options later

○ support for reserving &

limiting cpu/memory


Application definition

● Apps are defined in DAB can be deployed on a Swarm cluster

● Possible to scale individual containers defined in the DAB file (manual)

Testing Swarm Mode with Socks Shop:https://raw.githubusercontent.com/microservices-demo/microservices-demo/master/deploy/swarmkit/start-swarmkit-services.sh


Swarm Mode advantages

● Easy to install

● Secure by default

● “Bundled with Docker”


Swarm Mode disadvantages

● Very new

● Some Docker features unsupported (--privileged, --read-only, …)

● DAB still WIP


Kubernetes

often just “K8S”


Kubernetes

● Based on Google's experience running containers

● Many advanced features baked in:

○ Load-balancing

○ Secrets management

○ RBAC (Role Based Access Control)

○ …

● More opinionated


Core concepts

● Pods

● Labels

● Services

● Deployments

● ReplicaSets


Pods

● Groups of containers deployed and scheduled together

● Atomic unit of deployment

● Containers in a pod share IP address

● Single container pods are most common case

● Pods are ephemeral


Labels

● Key/Value pairs attached to objects (primarily pods)

○ e.g. version: dev, tier: frontend

● Label selectors then used to group objects

● Used for load-balancing etc.


Services

● Stable endpoints addressed by name

● Forward traffic to pods

● Pods are selected by labels

● Round-robin load-balancing

● Separates endpoint from implementation


Deployments & ReplicaSets

● ReplicaSets monitor status of Pods

○ start/stop pods as needed

● Deployments start/create ReplicaSets

● Rollout/Rollback & Updates


Usability

● Setting up a production grade Kubernetes-cluster from scratch requires

setting up etcd, networking plugins, DNS servers and certificate authorities.

○ Will change pretty soon with future versions of kubeadm

● Beyond initial setup, Kubernetes still has a steeper learning curve


Snap to install$kubeadm init

<master/tokens> generated token: "f0c861.753c505740ecde4c"<master/pki> created keys and certificates in "/etc/kubernetes/pki"<util/kubeconfig> created "/etc/kubernetes/kubelet.conf"<util/kubeconfig> created "/etc/kubernetes/admin.conf"<master/apiclient> created API client configuration<master/apiclient> created API client, waiting for the control plane to become ready<master/apiclient> all control plane components are healthy after 61.346626 seconds<master/apiclient> waiting for at least one node to register and become ready<master/apiclient> first node is ready after 4.506807 seconds<master/discovery> created essential addon: kube-discovery<master/addons> created essential addon: kube-proxy<master/addons> created essential addon: kube-dns

Kubernetes master initialised successfully!

You can connect any number of nodes by running:

$kubeadm join --token <token> <master-ip>


Application Definition● A combination of Pods, Replication Controllers, Replica Sets, Services and

Deployments● Each application tier is defined as a pod and can be scaled when managed

by a Deployment or ReplicationController/ReplicaSet. The scaling can be manual or automated

● Auto-scaling using a simple number-of-pods target is defined declaratively with the API exposed by ReplicationControllers or ReplicaSets

Testing Kubernetes with Sock Shop:https://github.com/microservices-demo/microservices-demo/blob/master/deploy/kubernetes/complete-demo.yaml


Kubernetes Advantages

● Easy to install (with kubeadm)

○ Currently limited to a single master installation

○ Currently limited to a single etcd installation

● Advanced features baked-in

● Lots of momentum behind the community


Kubernetes disadvantages

● Harder to get started

● Extra concepts to learn


Conclusion

● Different options with different strengths

● In some ways surprisingly similar (k8s Deployment | Swarm service)

● Hard to predict a winner

● All are much better than rolling-your-own


Trainings: Docker, Kubernetes, Microservices, Docker Security, Mesos… [email protected]

container orchestrator smackdown @continouslifecycle

Technology