11

OpenStack Powered by Tungsten FabricSukhdev KapurDistinguished Engineer, Juniper Networks

Krzysztof KajkowskiDirector of Engineering, CodiLime

Open Infrastructure Summit, Shanghai, November 2019

2

Tungsten Fabric Architecture Overview

Physical IP Fabric(no changes)

TF CONTROLLER

Host O/SvRouter

Network / Storage Orchestration

(Config, Control, Analytics, CSN)

(Windows, Linux ….) on BMS

TOR

Compute Orchestration

Virtual Network Blue Virtual Network Red

FW

Logical View

BGP

BGP XMPPNETCONF

Host O/SvRouter

… ……

DC Computes CPE Devices Public Cloud VM

Dis

trib

ute

d P

olic

y E

nfo

rcem

ent

Cen

tral

ized

Po

licy

Defi

nit

ion

ORCHESTRATOR / APPS

Internet / WAN or Legacy Env.

Gateway

…

2

3

vRouter Architecture Overview

vRouter Agent● Exchanging control state such as routes with the Control nodes using

XMPP.● Receiving low-level configuration state such as routing instances and

forwarding policy from the Control nodes using XMPP● Reporting analytics state such as logs, statistics, and events to the

analytics nodes.● Installing forwarding state into the forwarding plane● Discovering the existence and attributes of VMs in cooperation with

the Nova agent.● Applying forwarding policy for the first packet of each new flow and

installing a flow entry in the flow table of the forwarding plane.● Proxying DHCP, ARP, DNS

vRouter Kernel/DPDK● Encapsulating packets sent from the overlay network and

de-capsulating packets received for the overlay network.● Packets received from the overlay network are assigned to a routing

instance based on the MPLS label or Virtual Network Identifier (VNI).● Doing a lookup of the destination address of the in the Forwarding

Information Base (FIB) and forwarding the packet to the correct destination.  The routes may be layer-3 IP prefixes or layer-2 MAC addresses.

● Doing RPF check before sending Virtual machine traffic to destination. This is configurable.

Host Compute

User space

Kernel space

vRouter Kernel

Virtual Machine

(Tenant A)

Virtual Machine

(Tenant B)

XMPP

Control Node

pkt0 tap-abc tap-xyz

vRouter Agent

NETLINK

vhost0

Routing Instance

Routing Instance

Config VRFs Policy Table

ethX OR bondX

3

4

vRouter Deployment ModelsKERNEL vROUTER DPDK vROUTER

SRIOV/ vROUTER COEXISTENCE SMARTNIC vROUTER

4

● vRouter runs as a user space process and uses DPDK for fast path Packet I/O.

● Full set of SDN Capabilities Supported

● Requires the VMs to have DPDK enabled for performance benefits

● vRouter fwding plane runs within the NIC

● Workloads are SRIOV-connected to the NIC

● Some workloads can directly SR-IOV into the NIC, while others go through the vRouter

● Sometimes a VNF can have multiple interfaces some of which are SRIOV-ed to the NIC

● Interfaces that are SRIOV-ed into NIC don’t get the benefits / features of vRouter

● This the normal operation where fwding plane of vRouter runs in the kernel and are connected to VMs using TAP interface (or veth pair for containers)

● vRouter itself is enhanced using other performance related features:○ TSO / LRO○ Multi-Q Virtio

VM1vRouterAgent

VM 2...VM1vRouterAgent

VNF 2...

VM1 VM...vRouterAgent

... VM1 VM...vRouterAgent

...

5

Distributed Networking for VMs, PODs, & BMS

5

KubernetesCNI

Neutron/CNI/DM/FabricSDN Controller

Edge/MC-GW

OpenStackNeutron

Edge/POP Site

Edge/POP Site

Edge/POP Site

Basic Networking:L2/L3 or L2/L3 NetworkIPAM/DHCP, DNS, Multi-Tenancy

Advanced Networking:VLAN-ID, VRRP, VIP, LB, Routes Advertisement,GW Function, Service Chaining, Traffic Steering, Flow awareness,QoS, SR-IOV/DPDK, BGP-VPN,Inter Site Federation, Health Checks, FW, IPSec/TLS Support

BMS & FabricManager

On-Prem:● Core Site● Core Distributed Site● Edge Site

6

Policy Framework

Old B e h a v i o r New B e h a v i o r

Can we use one policy to be applied in all the different deployments?

…

Web App db

App1, Deployment = Prod

Network Policy = P3

1. Reduced Complexity2. Simplified Management3. Improved Scalability

Web App db

App1, Deployment = Staging

Network Policy = P2

Web App db

App1, Deployment = Dev

Network Policy = P1

…

Web App db

App1, Deployment = Dev

Web App db

App1, Deployment = Staging

Web App db

App1, Deployment = Prod

Policy = P

6

7

Policy Framework

Web App db

App1, Deployment = Dev-AWS

…

Web App db

App1, Deployment = Dev

Web App db

App1, Deployment = Staging

Web App db

App1, Deployment = Prod

Reuse of policies across multiple clouds and with multiple orchestrators

Web App db

App1, Deployment = Dev-K8s

Web App db

App1, Deployment = Dev-Mesos

Reuse policy

Web App db

App1, Deployment = Staging-BMS

B a r e M e t a l S e r v e r s

Reuse policy

Reuse policy

Reuse policy

Policy = P

Define/Review/Approve Once → Use Everywhere

7

8

Policy Framework – Use Case Example

site = US site = EMEA

Web App

App = Finance, Deployment = Dev

Web App

App = Finance, Deployment = Prod

Web App

App = Finance, Deployment = Dev

Web App

App = Finance, Deployment = Staging

match deploymentallow https-traffic tier=web > tier=app1

allow mysql-traffic tier=app > tier=db match site2

Dev

Pro

duct

ion

Sta

ging

Legacy Data (tier =

db)

&& siteE

n f

o r c

e m

e n

tD

e f

n

Legacy Data (tier = db)

8

9

Tungsten Fabric Deployment Models with Openstack

● Two Deployment models

○ Monolithic Plugin

○ ML2 based - this is used in the demo

Neutron

ML2 Plugin

MechanismDriver

VLAN GRE VxLAN Flat

Op

envS

wit

ch

TypeDriver

Op

enD

aylig

ht

Ari

sta

Cis

co N

exu

s

Net

wor

kin

g

Op

en C

ontr

ail

9

10

● Running Tungsten Fabric SDN along with other ML2 drivers

● This facilitates:○ Running OVS, SR-IOV and vRouter based works simultaneously

○ Running OVS and SR-IOV workloads and have Tungsten Fabric manage the fabric

○ Live migration of OVS based computes to vRouter based computes

https://opendev.org/x/networking-opencontrail

Tungsten Fabric and ML2 demo

10

11

Demo Setup Overview

b1s19 - node1 b1s19 - node3 b1s19 - node4eth0 eth0 eth0

VM VMVM VMVM VMSRIOV OVS TF

vMX (on b1s19 - node2)

ge-0/0

QFX

xe-0/1

vMX

xe-0/2 xe-0/3xe-0/0

11

12

Live Migration Scenario

12

b1s19 - node1 b1s19 - node3 b1s19 - node4

VM-SRIOV

eth0

VM-OVS

eth0 eth0

50.50.50.200 50.50.50.231

LAN50.50.50.0/24

VM-MIGRATE

50.50.50.183

VM-ROUTER

50.50.50.76

VM-MIGRATE

50.50.50.183

13

The Demo

14

Questions & Answers

15

Try Tungsten Fabric

Tungsten Fabric 15 minute deployment with k8s on AWS

THANK YOU.

16