overlay/underlay - betting on container networking

35
Betting on Container Networking Lee Calcote July 27th, 2016 Overlay Underlay http://calcotestudios.com/over-under

Upload: lee-calcote

Post on 16-Apr-2017

257 views

Category:

Software


3 download

TRANSCRIPT

Betting on ContainerNetworking

Lee CalcoteJuly 27th, 2016

OverlayUnderlay 

http://calcotestudios.com/over-under

http://calcotestudios.com/oubcn

Lee Calcoteclouds, containers, infrastructure, applications

 and their management

linkedin.com/in/leecalcote

@lcalcote

blog.gingergeek.com

[email protected]

ContainerNetworking

 

...it's complicated.

Preset Expectations           

 

Experience &Management

   

  

Reliability &Performance

same demands and measurements

developer-friendly and application-driven

simple to use and deploy for developers and operators

better or at least on par with their existing virtualized datacenter networking

ContainerNetworking

Specifications

Very interesting

but no need to actually know these

@lcalcote

(CNM)

Container Network Model

 ...is a specification proposed by Docker,

adopted by projects such as

 Plugins built by projects such as ,

, and

libnetwork

Weave

Project Calico Kuryr

(CNI)

Container Network Interface

 ...is a specification proposed byCoreOS and adopted by projects suchas , , ,

, and   Plugins created by projects such as

, , and

rkt Kurma Kubernetes CloudFoundry Apache Mesos

Weave Project Calico ContivNetworking

@lcalcote

Container Networking Specifications

Container Network ModelSpecification

@lcalcote

Remote DriversLocal Drivers

Container Network ModelTopology

@lcalcote

Network Sandbox

Endpoint

Backend Network

DockerContainer

Network Sandbox

Endpoint

DockerContainer

Network Sandbox

Endpoint

DockerContainer

Endpoint

Frontend Network

@lcalcote

Container Network Interface(CNI)

@lcalcote

Container Network InterfaceFlow

 

1. Container runtime needs to:

1. allocate a network namespace to the container and assign a container ID

2. pass along a number of parameters (CNI config) to network driver.

2. Network driver attaches container to a network and thenreports the assigned IP address back to the container runtime(via JSON schema)

@lcalcote

CNI Network(JSON)

{ "name": "mynet", "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "ipam": { "type": "host-local", "subnet": "10.22.0.0/16", "routes": [ { "dst": "0.0.0.0/0" } ] }

CNI and CNMSimilar in that each...

...are driver-based, and therefore

democratize the selection of which type of container networking 

...allow multiple network drivers to be active and usedconcurrently

each provide a one-to-one mapping of network to that network’s driver

...allow containers to join one or more networks.

...allow the container runtime to launch the network in itsown namespace

 segregate the application/business logic of connecting the container to

the network to the network driver.

 @lcalcote

CNI and CNMDifferent in that...

CNI supports any container runtime

CNM only support Docker runtime

CNI is simpler, has adoption beyond its creator CNM acts as a broker for conflict resolution

CNI is still considering its approach to arbitration

@lcalcote

Types of Container Networking

NoneLinks and AmbassadorsContainer-mappedBridgeHostOverlay 

Underlay

MACvlanIPvlanDirect Routing

Point-to-PointFan Networking 

@lcalcote

None

@lcalcote

container receives a network stack, butlacks an external network interface.

 it does, however, receive a loopback

interface.

Linksfacilitate single host connectivity

"discovery" via /etc/hosts or env vars

@lcalcote

Ambassadorsfacilitate multi-host connectivity

uses a tcp port forwarder (socat)

Web Host

MySQL

AmbassadorPHP

DB Host

PHP

AmbassadorMySQL

link link

Container-Mappedone container reuses (maps to) the networking

namespace of another container.

@lcalcote

may only be invoked when running a dockercontainer (cannot be defined in Dockerfile):

 

 --net=container=some_container_name_or_id

BridgeAh, yes, docker0

default networking for Dockeruses a host-internal networkleverages iptables for network address translation(NAT) and port-mapping

@lcalcote

Hostcontainer created shares its network namespace with the host

default Mesos networking modebetter performance easy to understand and troubleshootsuffers port conflictssecure?

@lcalcote

Overlayuse networking tunnels to delivery communication

across hosts 

Most useful in hybrid cloud scenariosor when shadow IT is needed

Many tunneling technologies existVXLAN being the most commonly used

Requires distributed key-value store

@lcalcote

K/V Store for OverlayNetworking

Docker - requires K/V store (built-in as experimental as of1.12)

WeaveMesh - does not require K/V store

WeaveNet - limited to single network; requires K/V store

Flannel -  requires K/V store

Plumgrid - requires K/V store; built-in and not pluggable

Midokura - requires K/V store; built-in and not pluggable

Calico - requires K/V store

@lcalcote

Underlaysexpose host interfaces (i.e. the physical network interface at

eth0) directly to containers running on the host

MACvlanIPvlanDirect Routing

@lcalcote

not necessarily public cloud friendly

Point-to-Point

Default rkt networking modeUses NAT (IPMASQ) by defaultCreates a virtual ethernet pair

placing one on the host and the other into the container pod

leverages iptables to provide port-forwardingfor inbound traffic to the podinternal communication between othercontainers in the pod over the loopbackinterface

@lcalcote

Internet

MACvlanallows creation of multiple virtual network interfaces behindthe host’s single physical interface

Each virtual interface has unique MAC and IP addressesassigned

with restriction: the IP address needs to be in the same broadcast

domain as the physical interface

eliminates the need for the Linux bridge, NAT and port-mapping

allowing you to connect directly to physical interface

 @lcalcote

IPvlanallows creation of multiple virtual network interfaces behind the host’s single

physical interface

Each virtual interface has unique IP addresses assigned

Same MAC address used for all containers

L2-mode containers must be on same network as host (similar to MACvlan)

L3-mode containers must be on different network than host

Network advertisement and redistribution into the network still needs to be done.

@lcalcote

MACvlan and IPvlanWhile multiple modes of networking are supported on a given host, MACvlan

and IPvlan can’t be used on the same physical interface concurrently.

ARP and broadcast traffic, the L2 modes of these underlay drivers operate

just as a server connected to a switch does by flooding and learning using

802.1d packets

IPvlan L3-mode - No multicast or broadcast traffic is allowed in.

In short, if you’re used to running trunks down to hosts, L2 mode is for you.

If scale is a primary concern, L3 has the potential for massive scale.

@lcalcote

Direct RoutingBenefits of pushing past L2 to L3

resonates with network engineers

leverage existing network infrastructure

use routing protocols for connectivity; easier to interoperate with existingdata center across VMs and bare metal servers

Better scaling

More granular control over filtering and isolating network traffic

Easier traffic engineering for quality of service

Easier to diagnose network issues

@lcalcote

Fan Networkinga way of gaining access to many more IP addresses, expanding from one assigned IP

address to 250 more IP addresses “address expansion” - multiplies the number of available IP addresses on the

host, providing an extra 253 usable addresses for each host IP

Fan addresses are assigned as subnets on a virtual bridge on the host,

IP addresses are mathematically mapped between networks

uses IP-in-IP tunneling; high performance

particularly useful when running containers in a public cloud

where a single IP address is assigned to a host and spinning up additional networks is prohibitive or

running another load-balancer instance is costly

@lcalcote

Network Capabilities andServices

 IPAM, multicast, broadcast, IPv6, load-balancing, service discovery, policy, quality

of service, advanced filtering and performance are all additional considerations to

account for when selecting networking that fits your needs.

 

@lcalcote

IPv6 and IPAMIPv6

lack of support for IPv6 in the top public clouds

reinforces the need for other networking types (overlays and fan networking)

some tier 2 public cloud providers offer support for IPv6

IPAM

most container runtime engines default to host-local for assigning addresses

to containers as they are connected to networks.

Host-local IPAM involves defining a fixed block of IP addresses to be selected.

DCHP is universally supported across the container networking projects.

CNM and CNI both have IPAM built-in and plugin frameworks for integration

with IPAM systems@lcalcote