Kubernetes networking

Introduction to overlay networks, communication models and implementation

[April 26 2016][ Murat Mukhtarov ]

Zendesk

Contents

2

● Overlay networking introduction○ Overlay concept○ Overlay network example: VXLAN

● Linux namespaces● Kubernetes networking○ Kubernetes and Docker networking comparison○ Service abstraction○ Inter POD communication○ Flannel example

● Q&A and links

Overlay networks: it is not a new paradigm

3

Stacking nature of networking allows encapsulation of different protocol stack at each network layer:- Datalink could be encapsulated in datalink, IP,

transport (e.g. UDP, IPSEC)- IP could be encapsulated in IP, MPLS, Datalink- MPLS allows simple stacking for complex

service-oriented topologies

Overlay networking concept

4

Overlay networks - encapsulation of the full tcp/ip stack including layer 2 inside transport network (UDP datagrams)

Overlay example: VXLAN

5

VXLAN overlay networking technology that allows to send Ethernet traffic encapsulated into UDP datagrams

over IP/GRE networks. Detailed description of VXLAN networking could be found in RFC7348

24 bit VNI field is VXLAN address field that could be

compared with 802.1q tag for Ethernet frames or MPLS

label.

Bare in mind MTU value when using VXLAN

Linux network namespaces

6

Network namespaces is a part of containerization technology that used by Linux kernelNetwork namespaces allows:

○ To create linux container network isolation instances (namespaces)

○ With own routing table, virtual interfaces, L2 isolation

● The tool that is used to operate with network ns: iproute2

● Network namespaces are stored in ○ /var/run/netns

● There two types of network namespaces:○ Root namespace [ ip link ]○ Non-root namespace [ ip netns .. ip link ]

Comparison network models: Kubernetes and Docker

7

Docker host example Kubernetes node example

Kubernetes: service abstraction

8

Service-oriented model:- POD represents group of containers in the

same namespace- Service represents load-balancing group of

PODs- Service could be mapped to overlay

network number, e.g. VXLAN number

Kubernetes networking model provides flexibility in terms of:- IP addressing: routable address per pod- Organizing networks: Flannel, OVS and etc.- Customising plugins: e.g. CNI- Docker0 serves as L3 (IP) gateway for PODs

behind it

Inter POD communication with overlay: general principle

9

Communication can be implemented as L2 and L3, it totally depends on chosen network model.

This particular example assumes that DHCP server resides somewhere in the network where overlay subnets are terminated.

Further flexibility could be achieved with CNI.

Inter POD communication: other examples

10

Flannel OVS

https://github.com/coreos/flannel#flannel http://kubernetes.io/docs/admin/ovs-networking/

Flannel example explanation

11

● Flannel creates overlay network on your choice:○ UDP or VXLAN encapsulation

● Flannel creates interface ○ flannel.VNI

● where VNI is number that you specified in json payload.

● Flannel interface is being assigned with ip address 10.1.X.0/16 address, where X - is random subnet number.

● Docker0 interface is assigned with address 10.1.x.1/24 all hosts behind docker0 are assigned with 10.1.x.2 - 254 addresses with 24 bit mask.

● Nodes are talking each other on switched 192.168.1.0/24 network

Apparently scaling can be questionable if we have more than 150-200 nodes in the network

Q&Ammukhtarov@zendesk.com

Links:Kubernetes network design

https://github.com/kubernetes/kubernetes/blob/release-1.2/docs/design/networking.md

Kubernetes with OVShttp://kubernetes.io/docs/admin/ovs-networking/

Kubernetes with Flannelhttps://github.com/coreos/flannel#flannel

Calico BGP projecthttps://github.com/projectcalico/calico-containers

BaGPipe BGP CNI plugin:https://github.com/murat1985/bagpipe-cni