container)networking)in)hybrid)cloud)@)scale › sites › events › files › ...gcp setup routes...

Container Networking in Hybrid Cloud @ Scale:

Sharad Murthy PayPal Inc • July 14, 2016@sharad_murthy

©2016 PayPal Inc. Confidential and proprietary.

1

Table of contents

Introduction to PayPal

Deployment Architecture

Container Networking Requirements

Solution

Q & A

©2016 PayPal Inc. Confidential and proprietary. 2

Key Statistics


Developer Statistics

x 100K Cores

x 10 PB Storage

OpenStack Cloud

> 10K Physical Servers

x 10K VMs

x 1000engineers

x 10M Lines

x 1000 Releases/year

> 1000 Services

x 1000 Builds/day

x 1000Deploy/day

9 Availability Zones

3 Regions

Deployment Architecture


data center

Availability Zone

ComplianceZone

Availability Zone

ComplianceZone

ComplianceZone

ComplianceZone

data center

Availability Zone

ComplianceZone

Availability Zone

ComplianceZone

ComplianceZone

ComplianceZone

subnetsubnet

subnet subnet

region

Hybrid Cloud


virtual private cloud

corporate data center

Availability ZoneVPC subnet

10 – 90 millisecs

1-‐10 millisecs

< 100 microsecs

region


virtual private cloud


1-‐10 millisecs

< 100 microsecs

region



1-‐10 millisecs

< 100 microsecs


Cluster Deployment


CentralController

IPAMAvailability Zone

MesosMaster

SLB/Service Registry

SystemService

LocalController

Host

2

4

5

3

10.10.1.1 –10.10.1.255

SystemServices Proxy

DockerEngine

6

1 LocalController

169.254.1.0-‐169.254.1.x

Container Networking RequirementsRequirements:− Container must be first class citizen of Network − IP Per Container or Per POD− Container interface must be assigned a routable IP and link local secondary IP − Container to Container communication must have low latency• desirable < 100 micro seconds with in AZ

− Support up to 100 containers per host • 50,000 to 100,000 containers in an Availability Zone

− Containers from different compliance zones must communicate through firewall. − IP Mobility with in Availability Zone− Distributed SLB− Distributed Firewall− ANYCAST with resilient flows− Same Solution on all clouds

Options:• Non-‐overlay: Ipvlan, Macvlan• Overlay: Vxlan, MPLS, OVS


IPVLAN

• Bridge device • Inherits MAC address of host interface• Operates in 2 modes−L2 mode• Arp is enabled

−L3 mode• Arp disabled• Relies on routing

• First available in Linux Kernel 3.19• Stable since Linux Kernel 4.2


Container Label

• Unique Container Identifier• Specification

network=<networkname>:servicetype=<type of service>:servicename=<service name>:mobile=<true/false>:id=<uuid>

where

• Network is a logical name for network • service type is type of service e.g. web, mid, zk, consul, syssvc etc• service name is application name• mobile – flag set to true or false indicating if container requires mobile IP• Id must be unique with an AZ

• Each field in Label passed to Docker as a Docker label• Label used to query IPAM for container IP


Container Networking with IPVLAN L2 Mode


TORTOR

Half Rack Half Rack

VM

Hypervisor

172.21.1.1/16 172.22.1.1/16172.21.1.21/16

172.21.1.10/16

172.21.1.11/16

172.22.1.10/16

172.22.1.10/16

LocalController

IPAM

Mobility restricted to subnet

Container IP address in

same subnet as host

CentralController

LocalController

IPVLAN IPVLAN

LocalController

IPVLAN

VM

IPVLAN in L2 Mode on AWS


ENI-‐1 ENI-‐2 ENI-‐3 ENI-‐4 ENI-‐5 ENI-‐6 ENI-‐7 ENI-‐8

IPVLAN IPVLAN IPVLAN IPVLAN IPVLAN IPVLAN IPVLAN IPVLAN

AWS VM

Max of 30 IPs per ENI for a total of 240 IP addresses per

VM

GroupOf 29

Containers

IPAMCentralController

LocalController

Container Networking with IPVLAN in L3 Mode


Key Benefit:o IP mobility in

Availability zone

Challenges:o Route table

sizeo No ECMP in

host

Compromise: o Limit number

of mobile IP addresses

o Use Bond Device in host

BGP

TORTOR

BGP

BGP AS = 1 BGP AS = 1

BGP

Half Rack

KernelIPVLAN

Bond

vlan

10.100.1.0

10.100.1.1

10.100.1.2 10.100.1.3

10.100.1.5 10.100.1.6

10.100.1.7Local

Controller BGPLocal

Controller

BGP BGP AS = 2LACP support in switch required IPAM

CentralController

10.5.6.1

10.100.1.4

Half Rack

IPVLANKernelBond

10.24.6.16

IPVLAN L3 Mode in GCP


Routes provisioned to container via

hostFrom IPAM

GCP CLI commandgcloud compute routes \create instance-‐2-‐route \-‐-‐destination-‐range 10.100.1.12/24-‐-‐next-‐hop-‐instance instance-‐2 \-‐-‐next-‐hop-‐instance-‐zone us-‐central1-‐f

GCP Cloud Installroutes

IPAM

CentralController

IPVLAN Kernel

LocalController

Instance-‐2

IPVLAN

LocalController

LocalController

LocalController

IPVLAN

IPVLANKernel

Kernel

Kernel

10.100.1.12

Control Plane


CentralController

AWS

AssignIP addressTo ENI

GCP

SetupRoutesFor IP addressRange, zone

• Create Binding• Release Binding• Extend Lease

admin• Define Network• Create Bindings• Remove Bindings:• List Bindings

IPAM

LocalController

LocaIIPAM

DockerEngine

Mesos

IPAMStore

Availability Zone

Distributed SLB with Anycast and BGP


BGP/BFD

BGP/BFD

TORTOR

BGP/BFDBGP AS = 1

BGP/BFD

BGP/BFD

Half RackHalf Rack

172.21.1.1

172.21.1.160/16

172.21.1.219/16

172.22.1.1

172.22.1.160/1610.0.0.1/32

10.0.0.1/32

Route Table:Destination Mask Next-‐hop Dist

10.0.0.1 32 172.22.1.160 110.0.0.1 32 172.22.1.210 1

172.22.1.210/16

RoundRobinLB


10.0.0.1 32 172.21.1.160 110.0.0.1 32 172.21.1.219 1

RoundRobinLB


10.0.0.1 32 172.21.1.1 210.0.0.1 32 172.22.1.1 2

RoundRobinLB

All switches must support flow state routing using ECMP with consistent hashing of 5 tuples to pin flows to next hop for high

reliability

BGP AS = 2

BGP/BFD

10.0.0.1/32

10.0.0.1/32

BGP/BFD

Micro Segmentation For Containers


TORTOR

VMVM

Hypervisor

Green-‐network

Yellow-‐network

Orange-‐network

IPAMControlPlane

MM• MPLS• VXLAN• ?

container)networking)in)hybrid)cloud)@)scale › sites › events › files › ...gcp setup routes...

Documents