cloud native networking with fd.io/vppbos.itdks.com/23e6f3d8cfbf4316a1ec5177ac087b0d.pdf•...

Cloud Native Networking with FD.io/VPP

Frank Brockners

Distinguished Engineer, Chief Technology and Architecture Office, Cisco

March 17, 2018

1

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

The way Applications are developed and deployed has changed…..

2


Microservices & Containers have changed many things…Applications are being developed and deployed very differently today

• Microservices allow you to split an application into many modular pieces, the network is how you stitch the pieces back together.

• The interconnection of the pieces results in a more complex application network which consumes lots of resources

• The performance of the cloud native network is crucial to the behavior of the overall application.

3

Pod

Pod

PodPod

Pod

Pod

Pod

It’s crucial we get ”Container Networking” right! Lets not get “Openstacked”


What Container Network Stacks Provide Today:

• Lifecycle management for application Containers

• Overlay connectivity for application Containers:

• NAT communication with external world

• Policy controlled overlay, may extend policy control to DC fabric in some cases

• Network policy addresses security/connectivity

• Designed for Data Center applications / use cases

Good start, but not sufficient for NFV use cases (Firewalls, Customs VNFs)

Container

Orchestration, Scheduling

Container

Networking Control

Container

Data-Plane

4


What Container Networks Stacks Lack for NFV Use Cases: • NVF-specific policy APIs (e.g. QoS, placement

considering network resources, etc.)

• Networking:

• NAT is a bottleneck, not suitable for NFV use cases

• VNFs often require more than 1 interface / IP address

• No support for high-speed wiring of NFVs:• To the outside world; To application containers;

Between NFV containers; Creation of Service Function Chains (mixed physical & virtual)

• Management/Control:

• Containerized NVFs not really in the data plane (except for the vswitch)

• No support for cloud-native, high-performance NVFs

• Forwarding:

• Kernel or OVS used for forwarding

Container

Orchestration,

Scheduling

Flexible & High-Speed

Container

Networking

High-Performance/Scale

Container Data-Plane

Flexible

Container

“wiring” and

control

5


Cloud-Native Network Function Needs: Summary

• Container-based

• Container stack lifecycle (same as application containers)

• 12-factor app design for management/control

• High-performance forwarding

• High-performance networking

• Seamless virtual/physical world interworking

• Common policy control with application containers (as much as possible)

• Must solve 3 key problems:

• Lifecycle management

• High-Performance Networking (wiring/config, operations)

• Easy installation, operation – policy, security

Container

Orchestration,

Scheduling

Container

Networking

Control

High-Performance/Scale

Container Data-Plane

Flexible

Container

“wiring” and

Control

LIGATO

6


Container Networking moving from Kernel to Userspace

• Userspace enables rapid upgradability, highly available (doesn't bring down node), no system call overhead, no dependency on Linux kernel networking community for features, higher performance and scale

• FD.io (dataplane), DPDK (network), SPDK (Storage) are examples

• Cloud Native apps are all connected by the network – lots of network end points to be managed, userspace offers lower overhead and higher performance

• Meltdown/Spectre bugs add a new tax for kernel networking

7


Kubernetes Networking – Key Communication patterns

4 distinct communication patterns:

8

Highly-coupled container-to-

container communications

Pod concept and

localhost communications

Pod-to-Pod communications Network Plugins

Pod-to-Service

communications

Kubernetes services

concept

External-to-Service

communications

Kubernetes services

concept

Kubernetes, Rocket…

Container Network Interface

Plugins


Kubernetes Network Plugins

• Kubernetes assumes seamless connectivity between pods, wherever it decides to place them. A networking plugin is needed to abstract the network

• Kubernetes fundamental requirements for any networking implementation

• all containers can communicate with all other containers without NAT

• all nodes can communicate with all containers (and vice-versa) without NAT

• the IP that a container sees itself as is the same IP that others see it as

• Network Plugins provide various levels of sophistication – from simple to highly feature rich and performant

9

Examples

• Cilium

• Contiv

• Contrail

• Flannel

• Google Compute Engine (GCE)

• Kube-router

• L2 networks and linux bridging

• Multus

• NSX-T

• Nuage Networks VCS

• OpenVSwitch

• OVN (Open Virtual Networking)

• Project Calico

• Romana

• Weave Net from Weaveworks

• CNI-Genie from Huawei


Container Networking Control: Contiv-VPPhttps://github.com/contiv/vpp

• Contiv is a networking plugin for Kubernetes that:

• Allocates IP addresses to Pods (IPAM)

• Programs the underlying infrastructure it uses (Linux TCP/IP stack, OVS, VPP, …) to connect the Pod’s to other Pods in the cluster and/or to the external world.

• Implements K8s network policies that define which pods can talk to each other.

• Implements K8s services; a service exposes one or more (physical) service instances implemented as K8s pods to the other pods in the cluster and/or to external clients as a virtual instance (e.g. as a virtual “service” IP address).

• Contiv is a user-space based, high-performance, high-density networking plugin for Kubernetes. Contiv-VPP leverages FD.io/VPP as the industry’s highest performance data plane

10

https://github.com/contiv/vpp


Ligato – Accelerate Cloud-Native DevelopmentPlatform and code samples for development of cloud native VNFs

LIGATO provides a mechanism for delivering and managing agents for cloud-native Network Functions to enable them to become part of the application service topology - all in user-space.

Components:

• Cloud-Native Infrastructure – a Golang platform for building cloud-native microservices

• VPP-agent - Golang implementation of a control/management plane for FD.io/VPP based cloud-native Virtual Network Functions (VNFs)

• SFC-Controller - Service Function Chain (SFC) Controller for stitching virtual and physical networks

• BGP-Agent - BGP Agent is a BGP information provider

https://ligato.github.io/

11



Network Functions can now be delivered as containers

• Take advantage of the low overhead of containers to deliver higher performance.

• Using cloud native technologies to build the Network Functions so they run in the same network and user-space as the applications.

• Network functions become part of the service topology

• Network Functions are truly just another service and can be developed/deployed using the same tools as the applications with the same velocity.

• Leverage cloud-native development & deployment velocity – #devployfaster

• Eliminates the overlay tax for services

12


Putting it all togetherEnabling Production-Grade Native Cloud Network Services at Scale

13

Contiv-VPP NetmasterCalico

SFC

Controller

Kubernetes API Proxies

Service Policy Service Topology Lifecycle

KubeletCNI

CRIContiv-VPP NetpluginCalico

Production-Grade Container Orchestration

Network Function and Network Topology Orchestration

Containerized Network Data Plane

Container

Network FunctionCNF

Agent Agent Agent

FD.io VPP

Container Switch

Agent

Container Networking

Networking Plugin

CNF

LIGATO

Contiv

Production-Grade

Container Orchestration

Cloud-native NF Orchestration

Cloud-native NF Agent platform

Containerized Fast

Data Input/ Output

Performance-Centric



GoAGENT

Containerized

Network Functions

GoAGENT

Containerized

Network Functions

SFC ControllerSFC Controller

Configuration

Operational State

Message Bus

SFC ControllerSFC ControllerApplications Tools (e.g.agentctl)

clientv1

ContainersLifecycle

Orchestration

clientv1

Data Store

GoAGENT

Configuration

Notifications

Control and Management Plane

Inter-Process Communication

Containerized

Switch


<<-- microservices -->>

40GE 100GE

CNF CNFcSwitch

Contiv-VPP Netmaster

Cloud-native Network Micro-Services Putting It All Together Now – The System Design

Functional Layered Diagram System Components

Contiv-VPP NetmasterCalico

SFC

Controller



KubeletCNI

CRIContiv-VPP NetpluginCalico




Container

Network FunctionCNF

Agent Agent Agent

FD.io VPP

Container Switch

Agent


Networking Plugin

CNF

SFC Controller

clientv1

14


Host

VPP Vswitch

CNF

VPP

10.1.0.127

…

CNF1

VPP

…

CNF2

VPP

…

…Server

Vswitch VPP

CNF

VPP

…

CNF

VPP

…

CNF3

VPP

…

…

NF1 NF2 NF3

Overlay Tunnel

Logical Representation

Physical Representation

Ingress Network

Ingress Classifier

Egress Network

Egress Classifier

TopologyPlacement

(K8s)Rendering

IngressRouter

EgressRouter

Overlay Tunnel Overlay Tunnel

Ingress Classifier Egress Classifier

Network Micro-Service Use Case:Service Function Chaining with Cloud-Native NFs

15


Ligato

Controller

Data Plane Network 1

Data Plane Network 2

Cloud (Overlay) Network

…

KubernetesEtcd

Contiv

Netmaster

(opt)

Physical

Device

Physical

Device

Physical

Device

Physical

Device…

Cloud tools

& services

CNF CNF CNF CNF CNF

Example: Distributed, multi-cloud CUPS architecture enabling Cloud Native Network Functions


Host

VPPvswitch

VNF

CN AppCN App

CN App

Host

VPPvswitch

VNF

CN-VNFCN-VNF

CN-VNF

Host

VPPvswitch

VNF

CN AppCN App

CN AppHost

VPPvswitch

VNF

CN AppCN App

CN App

Host

VPPvswitch

VNF

CN-VNFCN-VNF

CN-VNFHost

VPPvswitch

VNF

CN-VNFCN-VNF

CN-VNF Cloud

Network

Kubernetes

Cloud tools & services

Cloud-Native

Control Plane

Cloud-Native

Data Plane

VPP VPP

LIGATO Controller

CPE

CPECPE

CPE

CPE

CPE

CPE

…

…

Use Case: Cloud-Native CUPS ArchitectureEdge compute application and network service orchestration

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18

A glimpse at performance: Example topology

x86 Server

Host NW Stack

SFCController

etcd

Kubernetes

+K8s NW

Plugin

VPP VPPVPP

VPP cSwitch

CNF CNFCNF

vint1

vint1

vint2

vint2

I2xconn

NIC interface1 NIC interface2 Mgmt interface

veth

veth

veth

kafkaveth

I2bd

Topology: Containerized Switch with one or more Containerized Network Functions


memif: Shared Memory Packet Interface for VPP

• Packet based shared memory interface for user-mode application

• Container friendly (no privileged containers needed)

• Polling and interrupt mode operation:

• Interrupts simulated with linux eventfd infrastructure

• Support for interrupt masking in polling mode

• 3rd-party library - allows easy creation of applications which communicate over memif

• vpp-to-vpp, vpp-to-3rd-party and 3rd-party-to-3rd-party operation

• Support for multiple queues (incl. asymmetric configurations)

• Jumbo frames support (chained buffers)

• Secure

19


Example Benchmark: memif 10x faster than veth

0

1

2

3

4

5

6

7

8

9

10

0

2

4

6

8

10

12

14

16

1xCNF

cSwitch1core

2xCNF

cSwitch2cores

4xCNF

cSwitch4cores

Topology[1]:Memifvs.vEthBenchmarks64B Ethernetframesize

MemifMpps

vEthMpps*

MemifGbps

vEthGbps

PacketThroughput

[Mpps]

BandwidthThroughput

[Gbps]Memiffaster:10 times

64B Ethernet frame size – No Frame Loss

64B Ethernet frame size – No Frame Loss

Memif – cSwitch 1core

vEth – cSwitch 1core


One more thing…


Application Driven Service Mesh to enable Policy enforcement & Security

• Agents deployed as part of the app code to enable policy and security enforcement at the app level

• Provides a way to express application security consistently to support mutli-cloud application deployments

• Enterprise can get back consistent Networking & control when leveraging cloud-native and multi-cloud technologies Security Controls

Consistent Network & Security Control across cloud environments


• Istio provides a dedicated infrastructure layer for handling service-to-service communication

• Responsible for the reliable delivery of requests through the complex topology of services that comprise a modern, cloud native application

• Deployed as a sidecar container with the application

• Platform agnostic, runs where ever the containers run

• Envoy agent in the sidecar provides, routing, load balancing, traffic steering, stats collection, flow data – leveraging VPP for high performance

Istio Service Mesh with Envoy Agent

• Intelligent routing & Load Balancing

• Resilience Across Languages and Platforms

• Fleet-Wide Policy Enforcement

• In-Depth Telemetry and Reporting

Istio & Envoy


Network

Endpoint

Public Cloud & Serverless - Networking & Security controls are limited/non existent

On Prem

Service A

RULES

Serverless

Service D

Physical Network & SDN

Container

Networking

Container

Service C

Operator

programmed

RULES

Rules programmed by Network OperatorRules programmed

by DevOps

No controls

available

Network

Endpoint

Co-Lo

Service B

RULES

Enforcement

?

Multi-cloud environments are extremely challenging to secure


Service Mesh

Network

Endpoint

Public Cloud & Serverless - Networking & Security controls are limited/non existent

On Prem

Service A

RULES

Serverless

Service D

Physical Network & SDN

Container

Networking

Container

Service C

Operator

programmed

RULES

Rules programmed by Network OperatorRules programmed

by DevOps

No controls

available

Network

Endpoint

Co-Lo

Service B

RULES

Enforcement

?

Application Driven Service Mesh

RULES

Agents deployed part of the app –Enterprise get back consistent Networking & Security Controls


Istio CA

Kubernetes Pod

Service AEnvoy(Sidecar

Container)

Kubernetes Pod

Service BEnvoy(Sidecar

Container)

mTLS + Secure Naming

Issue & mount as

K8s secrets

Orchestrate Key & Certificate:

- Generation

- Deployment

- Rotation

- Revocation

spiffe://myorg.com/*

SAN:

“spiffe://myorg.com/ns/prod/sa/foo”

- Namespace: prod

- Service account: foo

SAN:

“spiffe://myorg.com/ns/prod/sa/bar:

- Namespace: prod

- Service account: bar

FD.io adds performance to the

Service Mesh K8s Master

Security at Scale with Istio


Complete Cloud Native Network Stack


OpenStack

ODL/ML2

Data Plane

Neutron

…

GBP

Neutron NB

VPP Renderer Topology Mgr.

VPP

Honeycomb

DPDK

Policy,

Lifecycle

ONAP/ OPNFV FastDataStacks

LIGATOContiv

Production-Grade

Container Orchestration

Cloud-native Network Function

Orchestration

Containerized Fast

Data Input/ Output

Performance-Centric


Enabling Production-Grade Native Cloud Network Services at Scale

Istio

Application Service

Mesh

Contiv NetmasterCalico

LIGATOController



KubeletCNI

CRIContiv NetmasterCalico




Container

Network FunctionCNF

Agent Agent Agent

FD.io VPP

Container Switch

Agent


Networking Plugin

CNF

Istio Service Mesh


• CNFs: Need to have cloud-native network functions in order to meet the

needs of cloud-native applications. Kernel networking is not appropriate for

cloud-native environments.

• Userspace Networking: Finish getting the kernel out of the way with

userspace host stacks

• Userspace/Cloud Native Storage: Storage needs to start down the

Userspace/Cloud Native road networking is walking: SPDK/Ceph/VPP

integration

• Unified IO: Networking/Storage: Start thinking of blocks/packets as

interchangeable units of IO with well integrated processing paths.

• Security Policy Orchestration: Integrate Application Driven security into the

development toolchain for DevOps.

What’s left to do ?


Summary

LIGATO

Contiv

Production-Grade Container Workload

Scheduling and Orchestration

Cloud-native NF Orchestration

Cloud-native NF Agent platform

Data-Plane: Containerized

Fast Data Input/ Output

Performance-Centric


IstioSecure overlay network with granular

policy control using a “sidecar” (L7 proxy)

Towards an orchestrated multi-cloud

from edge to centralized

andon-Prem to Co-lo to

public cloud


Thank you

31


References

• FD.io/VPP

• https://fd.io/

• memif:• https://docs.fd.io/vpp/17.10/libmemif_doc.html

• https://docs.google.com/presentation/d/1KrpOUUD7oHML6Plge_4E3-oq5YBun5T8pfoDiCBVk74

• Contiv-VPP

• https://github.com/contiv/vpp

• Ligato

• https://ligato.github.io/

32

https://fd.io/

https://docs.google.com/presentation/d/1KrpOUUD7oHML6Plge_4E3-oq5YBun5T8pfoDiCBVk74

https://docs.google.com/presentation/d/1KrpOUUD7oHML6Plge_4E3-oq5YBun5T8pfoDiCBVk74

https://github.com/contiv/vpp


cloud native networking with fd.io/vppbos.itdks.com/23e6f3d8cfbf4316a1ec5177ac087b0d.pdf•...

Documents