net1522bu kubernetes networking with nsx-t …...yves fauser / yasen simeonov net1522bu #vmworld...

37
NET1522BU #VMworld #NET1522BU Kubernetes Networking with NSX-T Deep Dive VMworld 2017 Content: Not for publication or distribution

Upload: others

Post on 02-Apr-2020

9 views

Category:

Documents


0 download

TRANSCRIPT

NET1522BU

#VMworld #NET1522BU

Kubernetes Networking with NSX-T Deep Dive

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

CONFIDENTIAL 2

VMworld 2017 Content: Not fo

r publication or distri

bution

Yves Fauser / Yasen Simeonov

NET1522BU

#VMworld #NET1522BU

Kubernetes Networking with NSX-T Deep Dive

VMworld 2017 Content: Not fo

r publication or distri

bution

Agenda

1 NSX-T Overview

2 Kubernetes Overview

3 NSX-T & Kubernetes Integration

4 Demo

4#NET1522BU CONFIDENTIAL

VMworld 2017 Content: Not fo

r publication or distri

bution

NSX-T Overview

VMworld 2017 Content: Not fo

r publication or distri

bution

NSX Vision: Driving NSX Everywhere

#NET1522BU CONFIDENTIAL 6

Managing security and connectivity for many heterogeneous end points

New app frameworks

Branch offices/Edge computing/IOT

End usersOn-premises data center

Automation

IT at the speed of business

Security

Inherently secure infrastructure

Application Continuity

Data center anywhere

Cloud

vCloud AirNetwork

VMworld 2017 Content: Not fo

r publication or distri

bution

Transport Nodes

NSX Manager

NSX Controllers

NSX-T Architecture

#NET1522BU CONFIDENTIAL 7

NSX Architecture and Components

Cloud Consumption• Self Service Portal

• OpenStack, K8s, Custom

Data Plane

• High Performance Data Plane

• Scale-out Distributed Forwarding Model

Management Plane (MP) Node – VM form factor

• Concurrent configuration portal

• REST API entry-point

• UI

Central Control Plane (CCP) Nodes- VM form factor

• Talks to Dataplane over a Control-Plane

Protocol

• Separation of Control and Data Plane

ESXi (+ kernel modules)

Control Plane

Management Plane

NSX Edge(L3 + Adv

Services)

Physical Infrastructure

Hypervisors

L2 Bridge(L2 Overlay-

VLAN)

KVM(+ kernel modules)

VMworld 2017 Content: Not fo

r publication or distri

bution

CCP Node CCP Node CCP Node

MP Node

NSX-T Architecture

#NET1522BU CONFIDENTIAL 8

Operations Workflow

User makes a configuration

Transport

Node

MPA LCP

Transport

Node

MPA LCP

Transport

Node

MPA LCP

XConfiguration is “persisted”

Configuration is pushed to CCP

Configuration is realized

VMworld 2017 Content: Not fo

r publication or distri

bution

Data Plane

#NET1522BU CONFIDENTIAL 9

Improved performance and resiliency

Admin

Tenants/CMP

Designed for multi-tenancy and scale

New distributed edge architecture with increased performance with

DPDK

p1 p2

HV TN1vSwitch

1

TEP

Overlay Transport Zone

TEP: Overlay Tunnel End Point

(with its own IP address)

GENEVE Tunnel

p1 p2

HV TN1vSwitch

2

TEP

Next gen overlay maintaining performance with increased flexibility

Edge

Node

Edge Cluster

Edge

Node

Edge

Node

Edge

Node

VMworld 2017 Content: Not fo

r publication or distri

bution

#NET1522BU CONFIDENTIAL 10

NSX-T Breakout Session

VMware NSX-T - Getting Started

SPL182601U (US)

SPL182601E (Europe)

NSX-T Hands On Lab

Introduction to NSX-T Architecture NET1510BU (US)

NET1510BE (Europe)

NSX-T VMworld Session & Lab

VMworld 2017 Content: Not fo

r publication or distri

bution

Kubernetes Overview

VMworld 2017 Content: Not fo

r publication or distri

bution

#NET1522BU CONFIDENTIAL 12

Kubernetes is an open-source platform for automating deployment, scaling, and operations of

application containers across clusters of hosts, providing container-centric infrastructure.

What Is Kubernetes?

VMworld 2017 Content: Not fo

r publication or distri

bution

• K8s Cluster Consists of Master(s) and Nodes

• K8s Master Components

– API Server

– Scheduler

– Controller Manager

– Dashbord

• K8s Node Components

– Kubelet

– Kube-Proxy

– Containers Runtime (Docker or Rocket)

#NET1522BU CONFIDENTIAL 13

K8s masterK8s master

K8s

Master

Controller

Manager

K8s API

Server

Key-Value

Store

dashboard

Scheduler

K8s nodeK8s node

K8s nodeK8s node

K8s Nodes

kubelet c runtime

Kube-proxy

> _ Kubectl

CLI

K8s Master(s)

Kubernetes Components

VMworld 2017 Content: Not fo

r publication or distri

bution

#NET1522BU CONFIDENTIAL 14

Pod

pause container(‘owns’ the IP stack)

10.24.0.0/16

10.24.0.2

nginxtcp/80

mgmttcp/22

loggingudp/514

IPC

External IP Traffic

A Pod is a group of one or more containers that shares an IP address and a Data Volume

Kubernetes Pod

VMworld 2017 Content: Not fo

r publication or distri

bution

#NET1522BU CONFIDENTIAL 15

Namespace: fooBase URI: /api/v1/namespaces/foo

‚redis-master‘ Pod:/api/v1/namespaces/foo/pods/redis-master

‚redis‘ service:/api/v1/namespaces/foo/services/redis-master

Namespace: barBase URI: /api/v1/namespaces/bar

‚redis-master‘ Pod:/api/v1/namespaces/bar/pods/redis-master

‚redis‘ service:/api/v1/namespaces/bar/services/redis-master

• Namespaces are a way to divide cluster resources between multiple uses

• They can be considered as Tenants

• They are a way to provide Resources Quotas, RBAC, Networking Multitenancy, and Names Overlapping

Kubernetes Namespace

VMworld 2017 Content: Not fo

r publication or distri

bution

East-West Load Balancing North-South Load Balancing

#NET1522BU CONFIDENTIAL 16

Redis Slave

Pods

redis-slave svc

10.24.0.5/16

172.30.0.24

Web Front-End

Pods

East-West Load Balancing is provided through K8s Service using ClusterIP & IPTables

Web Front-End

(e.g. Apache) Pods

Web Front-End

IngressNginx || HAProxy || etc.

LB Pods

http://*.bikeshop.com

Can be achieved through K8s Ingress or External third Party Load Balancer using NodePort

K8s Load Balancing

VMworld 2017 Content: Not fo

r publication or distri

bution

Nodeint eth0

10.240.0.4

int cbr0

10.24.2.1/24

10.24.2.2 10.24.2.3 10.24.2.4

Kubernetes Networking Topologies

#NET1522BU CONFIDENTIAL 17

Flat routed topology

Nodeint eth0

10.240.0.3

int cbr0

10.24.1.1/24

10.24.1.2 10.24.1.3 10.24.1.4

net.ipv4.ip_forward=1

net.ipv4.ip_forward=1

• Every Node is an IP Router and responsible for its Pod Subnet

• Subnets are associated with Nodes, not Tenants

• Physical Network Configuration is required

ip route 10.24.1.0/24 10.240.0.3

ip route 10.24.2.0/24 10.240.0.4

VMworld 2017 Content: Not fo

r publication or distri

bution

Kubernetes Networking Topologies

#NET1522BU CONFIDENTIAL 18

Node-to-Node overlay topology

Nodeint eth0

10.240.0.4

int cbr0

10.24.2.1/24

10.24.2.2 10.24.2.3 10.24.2.4

Nodeint eth0

10.240.0.3

int cbr0

10.24.1.1/24

10.24.1.2 10.24.1.3 10.24.1.4

net.ipv4.ip_forward=1

net.ipv4.ip_forward=1

Overlay

Key-Value

Store

Overlays are typically used to avoid Physical Network Configuration

VMworld 2017 Content: Not fo

r publication or distri

bution

NSX-T and Kubernetes Integration

VMworld 2017 Content: Not fo

r publication or distri

bution

#NET1522BU CONFIDENTIAL 20

admin@k8s-master:~$ kubectl create namespace foonamespace ”foo" created

admin@k8s-master:~$ kubectl create namespace barnamespace ”bar" created

admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foodeployment "nginx-foo" created

admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bardeployment "nginx-bar" created

Namespace: foo Namespace: bar

NSX / K8s topology

10.24.0.0/24 10.24.1.0/24 10.24.2.0/24

NAT boundary

NAT boundary

K8s nodesK8s Masters

NSX-T K8s Integration – Namespaces & Pods

VMworld 2017 Content: Not fo

r publication or distri

bution

#NET1522BU CONFIDENTIAL 21

admin@k8s-master:~$ vim no-nat-namespace.yaml

apiVersion: v1kind: Namespacemetadata:

name: no-nat-namespaceannotations:

ncp/no_snat: "true“

admin@k8s-master:~$ kubectl create –f no-nat-namespace.yamlnamespace ”no-nat-namespace" created

admin@k8s-master:~$ kubectl run nginx-no-nat --image=nginx –n no-nat-namespacedeployment "nginx-k8s" created

Namespace: no-nat-namespace

NSX / K8s topology

114.4.10.0/26

Direct Routing

114.4.10.64/26

K8s nodesK8s Masters

NSX-T K8s Integration – Routed Namespaces

VMworld 2017 Content: Not fo

r publication or distri

bution

NSX-T K8s Integration – Pods Micro-Segmentations

#NET1522BU CONFIDENTIAL 22

Option1: Predefined Label Based Rules

admin@k8s-master:~$ kubectl label pods nginx-foo-3492604561-nltrf secgroup=web -n fooPod "nginx-nsx-3492604561-nltrf" labeled

admin@k8s-master:~$ kubectl label pods nginx-bar-2789337611-z09x2 secgroup=db -n barpod "nginx-k8s-2789337611-z09x2" labeled

admin@k8s-master:~$ kubectl get pods --all-namespaces -LsecgroupNAMESPACE NAME READY STATUS RESTARTS AGE SECGROUPk8s nginx-foo-2789337611-z09x2 1/1 Running 0 58m webnsx nginx-bar-3492604561-nltrf 1/1 Running 0 1h db

Namespace: foo Namespace: bar

NSX / K8s topology

10.24.0.0/24 10.24.1.0/24 114.4.10.0/26

NAT boundary

NAT boundary

Web

• Security Groups are defined in NSX with ingress and egress policy

• Each Security Group could be micro-segmented to protect Pods from each other

DB

VMworld 2017 Content: Not fo

r publication or distri

bution

NSX-T K8s Integration – Pods Micro-Segmentations

#NET1522BU CONFIDENTIAL 23

Option 2: K8s Network Policy

admin@k8s-master:~$ vim nsx-demo-policy.yamlapiVersion: extensions/v1beta1kind: NetworkPolicymetadata:name: nsx-demo-policy

spec:podSelector:matchLabels:app: web

ingress:- from:

- namespaceSelector:matchLabels:ncp/project: db

ports:- port: 80protocol: TCP

admin@k8s-master:~$ kubectl create -f nsx-demo-policy.yaml

• State: released on K8s 1.7 (Beta on 1.6)

• Capability: Using Network Policy, users can define firewall rules to allow traffic into and out of a Namespace, and between Pods. The network policy is a Namespace property. The default is drop.

Namespace: foo Namespace: bar

NSX / K8s topology

10.24.0.0/24 10.24.1.0/24 114.4.10.0/26

NAT boundary

Routed

DB

Label: app=db

Web

Label: app=web

VMworld 2017 Content: Not fo

r publication or distri

bution

NSX-T K8s Integration – Pods Micro-Segmentations

#NET1522BU CONFIDENTIAL 24

Option 2: K8s Network Policy

$ kubectl create -f nsx-demo-policy.yaml

Dynamic Creation of Security Groups

Dynamic Creation of Security Policy based on k8s Network Policy

Once the Network Policy is applied, NSX will dynamically create source & destination Security Groups and apply the right policy

VMworld 2017 Content: Not fo

r publication or distri

bution

• Micro-Segmentation in K8s: The data model to describe segmentation policies between Namespaces, and within namespaces is called ’Network Policies’ and is released on Kubernetes 1.7 (Beta on 1.6)

Firewalling in Kubernetes

• NSX could utilize K8s Network Policies to define Dynamic Security Groups & Policies

• Capabilities are limited to K8s Network Policy capabilities

K8s Network Policy

• Security Groups & Policies could be predefined on NSX. Labels are used to specify Pods Membership

• Mapping of IP based groups, egress rules, VM based matching could be available to be used in the policy definition

Pre-Defined Label based rules

• The NSX / K8s integration intends to support both the pre-defined label based rules and K8s network policy

Firewalling in NSX / K8s

#NET1522BU CONFIDENTIAL 25

NSX-T K8s Integration – Pods Micro-Segmentations

VMworld 2017 Content: Not fo

r publication or distri

bution

#NET1522BU CONFIDENTIAL 26

Node

VM

NSX CNI

Plugin

OVS

Pods

NSX KubeProxy

K8s masterK8s master

K8s

Master

Controller

Manager

K8s API

Server

dashboard

Scheduler

• K8s Services are delivered through NSX Kube-Proxy

• Delivered as a container image, so that it can be run as a Kubernetes Daemon-Set on the Nodes

• NSX Kube-Proxy would replace the native distributed east-west load balancer in Kubernetes called Kube-Proxy

• OpenVSwitch (OVS) load-balancingis used

East-West Load Balancing

VMworld 2017 Content: Not fo

r publication or distri

bution

Once an Ingress Controller is added, NSX will define SNAT & DNAT rules

#NET1522BU CONFIDENTIAL 27

Web Front-End

Ingress

Nginx

Ingress LB Pod

http://*.demo.corp.local

10.4.0.0/24 10.4.1.0/24

10.4.0.67

North-South Load Balancing

VMworld 2017 Content: Not fo

r publication or distri

bution

• NCP is a software component provided by VMware in form of a container image, e.g. to be run as a K8s Pod

• NCP is build in a modular way, so that individual adapters can be added for different CaaS and PaaS systems

K8s / NSX Components

#NET1522BU CONFIDENTIAL 28

NSX Container Plugin (NCP)

VMworld 2017 Content: Not fo

r publication or distri

bution

Namespace creation workflow

NCP

Infra

K8s

Adapter

NSX Container Plugin

NSX

Manager

API Client

NSX

Manager

NS: foo

NSX/ K8s topology

NS: bar

K8s master

etcd

API-

Server

Scheduler

1)2)

3)

4)

1. NCP creates a ‚watch‘ on K8s API for any Namespace events

2. A user creates a new K8s Namespace

3. The K8s API Server notifies NCP of the change (addition) of Namespaces

4. NCP creates the network topology for the Namespace:

a) Requests a new subnet from the pre-configured IP block in NSX

b) Creates a logical switchc) Creates a T1 router and attaches it to

the pre-configured global T0 routerd) Creates a router port on the T1 router,

attaches it to the LS, and assigns an IP from the new subnet

K8s / NSX Workflows

#NET1522BU CONFIDENTIAL 29

Namespace / Topology creation

VMworld 2017 Content: Not fo

r publication or distri

bution

Hypervisor

(ESXi &

KVM)

Node

VM

DFW

eth2

Node

VM

DFW

eth0

Minion Mgmt.

IP Stack

eth0

Minion Mgmt.

IP Stack

mgmtnetwork

OVS

mgmtnetwork

Vla

n10

vla

n11

cifcif

eth2

vla

n10

vla

n11

OVS

cifcif

NSX CNI

Plugin

NSX CNI

Plugin

Pods

Pods

#NET1522BU CONFIDENTIAL 30

• Management Interface is Separated

from the interface used for Pods traffic

• CIF is used per K8s Pod

• CIFs are differentiated through locally

significant VLAN tags

• NSX CNI Plugin is responsible for

tagging the traffic with the right VLAN

• NCP will map the VLAN tags to a

specific CIF

NSX-T Container Interface (CIF)

VMworld 2017 Content: Not fo

r publication or distri

bution

#NET1522BU CONFIDENTIAL 31NSX-T Traceflow

NSX-T Operational Tools• Traceflow• Port Mirroring• Port Connection Tool• Spoofguard• Syslog• Port Counters• IPFIX

NSX-T Operational Tools for K8s

VMworld 2017 Content: Not fo

r publication or distri

bution

Demo

VMworld 2017 Content: Not fo

r publication or distri

bution

#NET1522BU CONFIDENTIAL 33

Enterprise-class

Networking

Advanced Security Enhanced

Operations

Full Network

Visibility

Enterprise

Support

Unified VM-to-

Pod

Networking

Pods Micro-

Segmentation

N S X - T Va l u e s f o r K 8 s

F e a t u r e s

NSX-T Values for K8s

VMworld 2017 Content: Not fo

r publication or distri

bution

Hands On Lab Self-Paced Lab

VMware NSX-T with KubernetesSPL182602U(US)

SPL182602E(Europe)

Kubernetes and VMware NSX Blog

https://blogs.vmware.com/networkvirtualization/2017/03/

kubecon-2017.html/#NET1522BU CONFIDENTIAL 34

VMworld 2017 Content: Not fo

r publication or distri

bution

Join VMUG for exclusive access to NSX

vmug.com/VMUG-Join/VMUG-Advantage

Connect with your peers

communities.vmware.com

Find NSX Resources

vmware.com/products/nsx

Network Virtualization Blog

blogs.vmware.com/networkvirtualization

Where to Get Started

Dozens of Unique NSX Sessions

Spotlights, breakouts, quick talks & group discussions

Visit the VMware Booth

Product overview, use-case demos

Visit Technical Partner Booths

Integration demos – Infrastructure, security, operations,

visibility, and more

Meet the Experts

Join our Experts in an intimate roundtable discussion

Free Hands-on Labs

Test drive NSX yourself with expert-led or self-paces

hands-on labs

labs.hol.vmware.com

Training and Certification

Several paths to professional certifications. Learn

more at the Education & Certification Lounge.

vmware.com/go/nsxtraining

Engage and Learn Experience

Try Take

#NET1522BU CONFIDENTIAL 35

VMworld 2017 Content: Not fo

r publication or distri

bution

VMworld 2017 Content: Not fo

r publication or distri

bution

VMworld 2017 Content: Not fo

r publication or distri

bution