Docker Multi Host NetworkingRachit Arora , IBM

BigInsights on Cloud

Agenda Docker networking – overview Native Multi Host networking (Experimental) Software defined networks SDNs Multi Host networking options

OpenVSwtich Weave Flannel , SocketPlane Etc..

Goals Make "network" a first class object Provide a Pluggable networking stack Span networks across multiple hosts Support multiple platforms

libnetwork

Updated Networking Stack in Docker

Create Networks using the Docker CLI

Multi-host Networking

Services UI

SDNs value proposal

SDNs proposes network solutions embracing cloud paradigms Massively multi-tenant

Thousands tenants, massively scalable Easy & fast (de)provisioning

Infra as code, API centric Infrastructure agnostic

L3, does not stick with lower levels (physical designs, vlans & co) Decouple infrastructure & tenants lifecycles Cross technology, vendor agnostic

Full-mesh, network agnostic and encapsulated approach Relies on L3 networks and GRE, VXLAN… tunnels for inter-hosts communication (avoid using L2) Network agnostic, til hosts can route trafic SDN Routers must route traffic between an inner virtual net and the ext. world

SDN Properties

Host #3

Host #1

Host #2

SDN network #1

SDN network #1

SDN network #1

SDN network #1

SDN network #1

RouterPhysical net.Flow encapsulation

L3 network

OpenVSwtich

Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license.

It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag).

In addition, it is designed to support distribution across multiple physical servers similar to VMware's vNetwork distributed vswitch

Host #1 Host #3Host #2

Getting Started

Container ‘app’

Container ‘db’

Brid

ge d

ocke

r0

Container ‘app’

Container ‘db’

Brid

ge d

ocke

r0

Container ‘app’

Container ‘db’

Container ‘app’

Brid

ge d

ocke

r0

Container ‘app’

TCP ports of services running in containers are

mapped

Get rid of actual Docker bridges implementation ! (Turn ICC False) If not, all containers will talk together across a same host

SetUp - OVS Setup Dev Tools

yum -y groupinstall "Development Tools"

yum -y install wget openssl-devel

yum -y install wget openssl-devel gcc make python-devel openssl-devel kernel-devel graphviz kernel-debug-devel autoconf automake rpm-build redhat-rpm-config libtool

Set up Bridge Utils yum -y install bridge-utils

Prepare OVS rpm cd ~

mkdir -p rpmbuild/SOURCES

wget http://openvswitch.org/releases/openvswitch-2.3.1.tar.gz

tar xvfz openvswitch-2.3.1.tar.gz

cd openvswitch-2.3.1/

cp ../openvswitch-2.3.1.tar.gz ~/rpmbuild/SOURCES/

sed 's/openvswitch-kmod, //g' openvswitch-2.3.1/rhel/openvswitch.spec > openvswitch-2.3.1/rhel/openvswitch_no_kmod.spec

rpmbuild -bb --nocheck ~/openvswitch-2.3.1/rhel/openvswitch_no_kmod.spec

You can find the files in ~/rpmbuild/RPMS/x86_64/.

yum -y localinstall openvswitch-2.3.1-1.x86_64.rpm

Use OpenVSwitch to create bridges on each host, for each tenant’s subnet. For instance, on host #1:

» ovs-vsctl add-br tech-br» ovs-vsctl add-port tech-br tep0 -- set interface tep0 type=internal» ifconfig tep0 192.168.1.1 netmask 255.255.255.0» ovs-vsctl add-br sdn-br0» ovs-vsctl set bridge sdn-br0 stp_enable=true

Create SDN compliant bridges

Host #1

sdn-

br0

Host #3

sdn-

br0

sdn-

br1

Host #2sd

n-br

0

sdn-

br1

Simplified view. Detailed insight exposed in later slides

Once per host: common plumbing

For each bridge: create and protect against ethernet loops using

Spanning Tree Protocol (beware, in complex/large deployments, it may

consumes a lot of CPU!)

Link SDN bridges

Host #1

sdn-

br0

Host #3

sdn-

br0

sdn-

br1

Host #2sd

n-br

0

sdn-

br1

Use OpenVSwitch to link corresponding bridges accross hosts In this example, we decided to use the full-mesh approach. On host #1:

ovs-vsctl add-port sdn-br0 gre0 --set interface gre0 type=gre options:remote_ip:1.2.3.2 ovs-vsctl add-port sdn-br0 gre1 --set interface gre1 type=gre options:remote_ip:1.2.3.3

gre0

gre1

1.2.3.1 1.2.3.2 1.2.3.3

Simplified view. Detailed insight exposed in later slides

U

Start containers and map them to the bridges

Host #1

sdn-

br0

Host #3

sdn-

br0

sdn-

br1

Host #2sd

n-br

0

sdn-

br1

Container ‘app’

Container ‘db’

Container ‘app’

Container ‘db’

Container ‘app’

Container ‘app’

Container ‘db’

Container ‘app’

Container ‘db’

Instanciate containers without pre-built interfaces to avoid plugging containers to native docker0 bridge

Use “docker run … ip = ?? -net=none” switch in “docker run” calls

Weave

Weave Weave can traverse firewalls and operate in partially connected

networks. Traffic can be encrypted, allowing hosts to be connected across

an untrusted network. With weave you can easily construct applications consisting of

multiple containers, running anywhere. Weave works alongside Docker's existing (single host)

networking capabilities, so these can continue to be used by containers.

Weave Installation

Ensure you are running Linux (kernel 3.8 or later) and have Docker (version 1.3.1 or later) installed

Weave Setup

Performance Comparison – self test Used qperf to test performance

Physical Host

tcp_bw:

bw = 117 MB/sec OVS with gre tunnel (Default)

tcp_bw:

bw = 81.9 KB/sec OVS with gre tunnel (ethtool -K ovsBridge tx off sg off tso off) ( technique for increasing

outbound throughput of high-bandwidth network connections by reducing CPU overhead , not recommended for production)

tcp_bw:

bw = 104 MB/sec Weave(Latest version) based on VXLAN (Older versions based on pcap had lower

performance)

tcp_bw:

bw = 92.3 MB/sec

Bonus: Alternate design - OpenStack Neutron paradigms Alternate design, based on OpenStack Neutron paradigms

All VMs/containers of a same tenant network are segregated inside a dedicated, local VLAN of a shared unique bridge

Full-mesh of GRE tunnels between all hosts On each host, local mapping between a local tenant network

VLAN and its GRE identifier shared across all hosts