net1522be kubernetes networking with nsx-t … al idrees yves fauser net1522be #vmworld #net1522be...
TRANSCRIPT
Ali Al IdreesYves Fauser
NET1522BE
#VMworld #NET1522BE
Kubernetes Networking with NSX-T Deep Dive
VMworld 2017 Content: Not fo
r publication or distri
bution
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not been determined.
Disclaimer
2#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Agenda
1 NSX-T Overview
2 Kubernetes Overview
3 NSX-T & Kubernetes Integration
4 Demo
3#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
5
NSX Vision: Driving NSX everywhereManaging security and connectivity for many heterogeneous end points
New app frameworks
Branch offices/Edge computing/IOT
End users
On-premises data center
Automation
IT at the speed of business
Security
Inherently secure infrastructure
Application Continuity
Data center anywhere
Cloud
vCloud AirNetwork
VMworld 2017 Content: Not fo
r publication or distri
bution
Transport Nodes
NSX Manager
NSX Controllers
NSX-T ArchitectureNSX Architecture and Components
Cloud Consumption• Self Service Portal
• OpenStack, K8s, Custom
Data Plane
• High Performance Data Plane
• Scale-out Distributed Forwarding Model
Management Plane (MP) Node – VM form factor
• Concurrent configuration portal
• REST API entry-point
• UI
Central Control Plane (CCP) Nodes- VM form factor
• Talks to Dataplane over a Control-Plane
Protocol
• Separation of Control and Data Plane
ESXi(+ kernel modules)
Control Plane
Management Plane
NSX Edge(L3 + Adv
Services)
Physical Infrastructure
Hypervisors
L2 Bridge(L2 Overlay-
VLAN)
KVM(+ kernel modules)
6#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
CCP Node CCP Node CCP Node
MP Node
NSX-T ArchitectureOperations Workflow
7
User makes a configuration
Transport
Node
MPA LCP
Transport
Node
MPA LCP
Transport
Node
MPA LCP
XConfiguration is “persisted”
Configuration is pushed to CCP
Configuration is realized
#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Data Plane
Improved performance and resiliency
Admin
Tenants/CMP
Designed for multi-tenancy and scale
New distributed edge architecture with increased performance with
DPDK
p1 p2
HV TN1vSwitc
h1
TEP
Overlay Transport Zone
TEP: Overlay Tunnel End Point
(with its own IP address)
GENEVE Tunnel
p1 p2
HV TN1vSwitc
h2
TEP
Next gen overlay maintaining performance with increased flexibility
Edge
Node
Edge Cluster
Edge
Node
Edge
Node
Edge
Node
8#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX-T VMworld Session & Lab
9
NSX-T Breakout Session
VMware NSX-T - Getting Started
SPL182601U (US)
SPL182601E (Europe)
NSX-T Hands On Lab
Introduction to NSX-T Architecture NET1510BU (US)
NET1510BE (Europe)
#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
What is Kubernetes?
Kubernetes is an open-source platform for automating deployment, scaling, and operations of
application containers across clusters of hosts, providing container-centric infrastructure.
11#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Kubernetes Components
• K8s Cluster Consists of Master(s) and Nodes
• K8s Master Components
– API Server
– Scheduler
– Controller Manager
– Dashbord
• K8s Node Components
– Kubelet
– Kube-Proxy
– Containers Runtime (Docker or Rocket)
12
K8s masterK8s master
K8s
Master
Controller
Manager
K8s API
Server
Key-Value
Store
dashboard
Scheduler
K8s nodeK8s node
K8s nodeK8s node
K8s Nodes
kubelet c runtime
Kube-proxy
> _ Kubectl
CLI
K8s Master(s)
#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Kubernetes Pod
Pod
pause container(‘owns’ the IP stack)
10.24.0.0/16
10.24.0.2
nginxtcp/80
mgmttcp/22
loggingudp/514
IPC
External IP Traffic
• A Pod is a group of one or more containers that shares an IP address and a Data Volume
13#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Kubernetes Namespace
Namespace: fooBase URI: /api/v1/namespaces/foo
‚redis-master‘ Pod:/api/v1/namespaces/foo/pods/redis-master
‚redis‘ service:/api/v1/namespaces/foo/services/redis-master
Namespace: barBase URI: /api/v1/namespaces/bar
‚redis-master‘ Pod:/api/v1/namespaces/bar/pods/redis-master
‚redis‘ service:/api/v1/namespaces/bar/services/redis-master
• Namespaces are a way to divide cluster resources between multiple uses
• They can be considered as Tenants
• They are a way to provide Resources Quotas, RBAC, Networking Multitenancy, and Names Overlapping
14#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
K8s Load Balancing
East-West Load Balancing North-South Load Balancing
15
Redis Slave
Pods
redis-slave svc
10.24.0.5/16
172.30.0.24
Web Front-End
Pods
East-West Load Balancing is provided through K8s Service using ClusterIP & IPTables
Web Front-End
(e.g. Apache) Pods
Web Front-End
IngressNginx || HAProxy || etc.
LB Pods
http://*.bikeshop.com
Can be achieved through K8s Ingress or External third Party Load Balancer using NodePort
#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Nodeint eth0
10.240.0.4
int cbr0
10.24.2.1/24
10.24.2.2 10.24.2.3 10.24.2.4
Kubernetes Networking TopologiesFlat routed topology
ip route 10.24.1.0/24 10.240.0.3
ip route 10.24.2.0/24 10.240.0.4
Nodeint eth0
10.240.0.3
int cbr0
10.24.1.1/24
10.24.1.2 10.24.1.3 10.24.1.4
net.ipv4.ip_forward=1
net.ipv4.ip_forward=1
• Every Node is an IP Router and responsible for its Pod Subnet
• Subnets are associated with Nodes, not Tenants
• Physical Network Configuration is required
16#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Kubernetes Networking TopologiesNode-to-Node overlay topology
Nodeint eth0
10.240.0.4
int cbr0
10.24.2.1/24
10.24.2.2 10.24.2.3 10.24.2.4
Nodeint eth0
10.240.0.3
int cbr0
10.24.1.1/24
10.24.1.2 10.24.1.3 10.24.1.4
net.ipv4.ip_forward=1
net.ipv4.ip_forward=1
Overlay
Key-Value
Store
• Overlays are typically used to avoid Physical Network Configuration
17#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX-T K8s Integration – Namespaces & Pods
admin@k8s-master:~$ kubectl create namespace foonamespace ”foo" created
admin@k8s-master:~$ kubectl create namespace barnamespace ”bar" created
admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foodeployment "nginx-foo" created
admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bardeployment "nginx-bar" created
Namespace: foo Namespace: bar
NSX / K8s topology
10.24.0.0/24 10.24.1.0/24 10.24.2.0/24
NAT boundary
NAT boundary
K8s nodesK8s Masters
19#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX-T K8s Integration – Routed Namespaces
admin@k8s-master:~$ vim no-nat-namespace.yaml
apiVersion: v1kind: Namespacemetadata:
name: no-nat-namespaceannotations:
ncp/no_snat: "true“
admin@k8s-master:~$ kubectl create –f no-nat-namespace.yamlnamespace ”no-nat-namespace" created
admin@k8s-master:~$ kubectl run nginx-no-nat --image=nginx –n no-nat-namespacedeployment "nginx-k8s" created
Namespace: no-nat-namespace
NSX / K8s topology
114.4.10.0/26
Direct Routing
114.4.10.64/26
K8s nodesK8s Masters
20#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX-T K8s Integration – Pods Micro-SegmentationsOption1: Predefined Label Based Rules
admin@k8s-master:~$ kubectl label pods nginx-foo-3492604561-nltrf secgroup=web -n fooPod "nginx-nsx-3492604561-nltrf" labeled
admin@k8s-master:~$ kubectl label pods nginx-bar-2789337611-z09x2 secgroup=db -n barpod "nginx-k8s-2789337611-z09x2" labeled
admin@k8s-master:~$ kubectl get pods --all-namespaces -LsecgroupNAMESPACE NAME READY STATUS RESTARTS AGE SECGROUPk8s nginx-foo-2789337611-z09x2 1/1 Running 0 58m webnsx nginx-bar-3492604561-nltrf 1/1 Running 0 1h db
Namespace: foo Namespace: bar
NSX / K8s topology
10.24.0.0/24 10.24.1.0/24 114.4.10.0/26
NAT boundary
NAT boundary
Web
• Security Groups are defined in NSX with ingress and egress policy
• Each Security Group could be micro-segmented to protect Pods from each other
DB
21
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX-T K8s Integration – Pods Micro-SegmentationsOption 2: K8s Network Policy
admin@k8s-master:~$ vim nsx-demo-policy.yamlapiVersion: extensions/v1beta1kind: NetworkPolicymetadata:name: nsx-demo-policy
spec:podSelector:matchLabels:app: web
ingress:- from:
- namespaceSelector:matchLabels:ncp/project: db
ports:- port: 80protocol: TCP
admin@k8s-master:~$ kubectl create -f nsx-demo-policy.yaml
• State: released on K8s 1.7 (Beta on 1.6)
• Capability: Using Network Policy, users can define firewall rules to allow traffic into and out of a Namespace, and between Pods. The network policy is a Namespace property. The default is drop
Namespace: foo Namespace: bar
NSX / K8s topology
10.24.0.0/24 10.24.1.0/24 114.4.10.0/26
NAT boundary
Routed
DB
Label: app=db
Web
Label: app=web
22#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX-T K8s Integration – Pods Micro-SegmentationsOption 2: K8s Network Policy
$ kubectl create -f nsx-demo-policy.yaml
23
Dynamic Creation of Security Groups
Dynamic Creation of Security Policy based on k8s Network Policy
Once the Network Policy is applied, NSX will dynamically create source & destination Security Groups and apply the right policy
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX-T K8s Integration – Pods Micro-Segmentations
• Micro-Segmentation in K8s: The data model to describe segmentation policies between Namespaces, and within namespaces is called ’Network Policies’ and is released on Kubernetes 1.7 (Beta on 1.6)
Firewalling in Kubernetes
• NSX could utilize K8s Network Policies to define Dynamic Security Groups & Policies.
• Capabilities are limited to K8s Network Policy capabilities.
K8s Network Policy
• Security Groups & Policies could be predefined on NSX. Labels are used to specify Pods Membership
• Mapping of IP based groups, egress rules, VM based matching could be available to be used in the policy definition
Pre-Defined Label based rules
• The NSX / K8s integration intends to support both the pre-defined label based rules and K8s network policy.
Firewalling in NSX / K8s
24#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
East-West Load Balancing
Node
VM
NSX CNI
Plugin
OVS
Pods
NSX KubeProxy
K8s masterK8s master
K8s
Master
Controller
Manager
K8s API
Server
dashboard
Scheduler
• K8s Services are delivered through NSX Kube-Proxy.
• Delivered as a container image, so that it can be run as a Kubernetes Daemon-Set on the Nodes.
• NSX Kube-Proxy would replace the native distributed east-west load balancer in Kubernetes called Kube-Proxy.
• OpenVSwitch (OVS) load-balancing is used.
25#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
North-South Load Balancing
• Once an Ingress Controller is added, NSX will define SNAT & DNAT rules
26
Web Front-End
Ingress
Nginx
Ingress LB Pod
http://*.demo.corp.local
10.4.0.0/24 10.4.1.0/24
10.4.0.67
VMworld 2017 Content: Not fo
r publication or distri
bution
K8s / NSX ComponentsNSX Container Plugin (NCP)
• NCP is a software component provided by VMware in form of a container image, e.g. to be run as a K8s Pod.
• NCP is build in a modular way, so that individual adapters can be added for different CaaS and PaaS systems
27#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Namespace creation workflowK8s / NSX WorkflowsNamespace / Topology creation
NCP
Infra
K8s
Adapter
NSX Container Plugin
NSX
Manager
API Client
NSX
Manager
NS: foo
NSX/ K8s topology
NS: bar
K8s master
etcd
API-
Server
Scheduler
1)2)
3)
4)
1. NCP creates a ‚watch‘ on K8s API for any Namespace events
2. A user creates a new K8s Namespace
3. The K8s API Server notifies NCP of the change (addition) of Namespaces
4. NCP creates the network topology for the Namespace :
a) Requests a new subnet from the pre-configured IP block in NSX
b) Creates a logical switchc) Creates a T1 router and attaches it to
the pre-configured global T0 routerd) Creates a router port on the T1 router,
attaches it to the LS, and assigns an IP from the new subnet
28#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX-T Container Interface (CIF)
Hypervisor
(ESXi &
KVM)
Node
VM
DFW
eth2
Node
VM
DFW
eth0
Minion Mgmt.
IP Stack
eth0
Minion Mgmt.
IP Stack
mgmtnetwork
OVS
mgmtnetwork
Vla
n10
vla
n11
cifcif
eth2
vla
n10
vla
n11
OVS
cifcif
NSX CNI
Plugin
NSX CNI
Plugin
Pods
Pods
29
• Management Interface is Separated
from the interface used for Pods traffic
• CIF is used per K8s Pod
• CIFs are differentiated through locally
significant VLAN tags
• NSX CNI Plugin is responsible for
tagging the traffic with the right VLAN
• NCP will map the VLAN tags to a
specific CIF.
#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX-T Operational Tools for K8s
30
NSX-T Traceflow
NSX-T Operational Tools• Traceflow• Port Mirroring• Port Connection Tool• Spoofguard• Syslog• Port Counters• IPFIX
#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX-T Values for K8s
Enterprise-class
Networking
Advanced Security Enhanced
Operations
Full Network
Visibility
Enterprise
Support
Unified VM-to-
Pod
Networking
Pods Micro-
Segmentation
N S X - T Va l u e s f o r K 8 s
F e a t u r e s
32#NET1522BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
33
Hands On Lab Self-Paced Lab
VMware NSX-T with KubernetesSPL182602U(US)
SPL182602E(Europe)
Kubernetes and VMware NSX Blog
https://blogs.vmware.com/networkvirtualization/2017/03/
kubecon-2017.html/
VMworld 2017 Content: Not fo
r publication or distri
bution
Join VMUG for exclusive access to NSX
vmug.com/VMUG-Join/VMUG-Advantage
Connect with your peers
communities.vmware.com
Find NSX Resources
vmware.com/products/nsx
Network Virtualization Blog
blogs.vmware.com/networkvirtualization
Where to get started
Dozens of Unique NSX Sessions
Spotlights, breakouts, quick talks & group discussions
Visit the VMware Booth
Product overview, use-case demos
Visit Technical Partner Booths
Integration demos – Infrastructure, security, operations,
visibility, and more
Meet the Experts
Join our Experts in an intimate roundtable discussion
Free Hands-on Labs
Test drive NSX yourself with expert-led or self-paces
hands-on labs
labs.hol.vmware.com
Training and Certification
Several paths to professional certifications. Learn
more at the Education & Certification Lounge.
vmware.com/go/nsxtraining
Engage and Learn Experience
Try TakeVMworld 2017 Content: N
ot for publicatio
n or distribution