kubernetes container networking with nsx-t data …...kubernetes container networking with nsx-t...
Post on 22-May-2020
44 Views
Preview:
TRANSCRIPT
#vmworld
KubernetesContainer Networking with
NSX-T Data CenterDeep Dive
Yasen Simeonov, Sr. Technical Product Manager, NSBUDennis Breithaupt, Sr. Systems Engineer (NSX)
,
NET1677BE
#NET1677BE
VMworld 2018 Content: Not for publication or distribution
Disclaimer
2©2018 VMware, Inc.
This presentation may contain product features orfunctionality that are currently under development.
This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.
VMworld 2018 Content: Not for publication or distribution
Agenda
3©2018 VMware, Inc.
NSX-T IntroQuick level set on NSX-T
Kubernetes OverviewTechnical overview of Kubernetes, nomenclature & networking details
NSX-T & KubernetesDetails of the NSX-T integration with Kubernetes
DemoSeeing is believing
VMworld 2018 Content: Not for publication or distribution
4©2018 VMware, Inc.
NSX-T Data Center IntroQuick level set on NSX-T Data Center
VMworld 2018 Content: Not for publication or distribution
5©2018 VMware, Inc.
BRANCH
BRANCH
BRANCH
BRANCH
BRANCH
BRANCH
BRANCH
BRANCH
TELCO/NFV
TELCO/NFV
EDGE/IOT
TELCO/NFV
BRANCH
BRANCH
EDGE/IOT
EDGE/IOT
The Virtual Cloud NetworkConnect and Protect your BusinessVMworld 2018 Content: Not for publication or distribution
6©2018 VMware, Inc.
Identity
Apps and Data
Policy ScalabilityAnalytics and Insights
Secure Connectivity Availability
Users
Private Data Centers
VMs, Containers, Microservices
Branch Offices
Public Clouds
Telco Networks
Things
Virtual Cloud NetworkingConnect & Protectany workload across any environment
Built-in
Automated
Programmable
Application Centric
VMworld 2018 Content: Not for publication or distribution
7©2018 VMware, Inc.
NETWORKING AND SECURITY MANAGEMENT AND AUTOMATION
vRealize AutomationEnd-to-end workload automation
Network InsightNetwork discovery and insights
Cloud-Based Management Workflow Automation Blueprints / Templates Insights / Discovery Visibility
NETWORK AND SECURITY VIRTUALIZATION
AppDefenseModern application
security
NSX SD-WANby VeloCloud
WAN connectivity services
NSX Hybrid ConnectData center and cloud
workload migration
NSX Data CenterNetworking and
security for data center workloads
NSX CloudNetworking and
security for Public Cloud workloads
Security Integration Extensibility Automation Elasticity
VMware NSX PortfolioThe Foundation of the Virtual Cloud Network
VMworld 2018 Content: Not for publication or distribution
8©2018 VMware, Inc.
NSX Data Center Architecture For Private Cloud, Public Cloud & Containers
CONTROLPLANE
DATAPLANE
MANAGEMENT PLANE
Private or Public cloud infrastructure
NSX Central Controller
NSX Manager
(VPN Gateway, DirectConnect, ExpressRoute)
Public Cloud
Linux VM Windows VMNSX Cloud Gateway
VMware Cloud on AWS
Private Cloud
NSX Edge VM or Bare Metal
ESXi KVM
N-VDS N-VDS
Multi-Hypervisor
Cloud ServiceManager
Bare Metal
NSX NSX
VMworld 2018 Content: Not for publication or distribution
9©2018 VMware, Inc.
Data PlaneImproved performance and resiliency
Admin
Tenants/CMP
Designed for multi-tenancy and scale
New distributed edge architecture with increased
performance with DPDK
p1 p2
HV TN1vSwitch1
TEP
Overlay Transport Zone
TEP: Overlay Tunnel End Point
(with its own IP address)
GENEVE Tunnel
p1 p2
HV TN1 vSwitch2
TEP
Next gen overlay maintaining
performance with increased flexibility
EdgeNode
Edge Cluster
EdgeNode
EdgeNode
EdgeNode
Admin
Tenants/CMP
Designed for multi-tenancy and scale
SessionsNET1127BU VMware NSX-T™ Data Center Routing Deep Dive
VMworld 2018 Content: Not for publication or distribution
10©2018 VMware, Inc.
Kubernetes OverviewTechnical overview of Kubernetes, nomenclature & networking details
VMworld 2018 Content: Not for publication or distribution
11©2018 VMware, Inc.
Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure.
What is Kubernetes?
VMworld 2018 Content: Not for publication or distribution
12©2018 VMware, Inc.
Kubernetes Components
K8s Cluster Consists of Master(s) and Nodes
K8s Master Components• API Server• Scheduler• Controller Manager• Dashboard
K8s Node Components• Kubelet• Kube-Proxy• Containers Runtime
K8s masterK8s master
K8s Master
Controller Manager
K8s APIServer
Key-Value Store
dashboard
Scheduler
K8s nodeK8s nodeK8s nodeK8s node
K8s Nodes
kubelet c runtime
Kube-proxy
> _ Kubectl
CLI
K8s Master(s)
VMworld 2018 Content: Not for publication or distribution
13©2018 VMware, Inc.
Kubernetes Pod
A Pod is a group of one or more containers that shares an IP address and a Data Volume Pod
pause container(‘owns’ the IP stack)
10.24.0.0/16
10.24.0.2
nginxtcp/80
mgmttcp/22
loggingudp/514
IPC
External IP Traffic
VMworld 2018 Content: Not for publication or distribution
14©2018 VMware, Inc.
Kubernetes Namespace
Namespaces are a way to divide cluster resources amongst users and groups
They can be thought of as Tenants
They are a way to provide Resources Quotas, RBAC, Networking Multitenancy, and Name uniqueness
Namespace: fooBase URI: /api/v1/namespaces/foo
‚redis-master‘ Pod:/api/v1/namespaces/foo/pods/redis-master
‚redis‘ service:/api/v1/namespaces/foo/services/redis-master
Namespace: barBase URI: /api/v1/namespaces/bar
‚redis-master‘ Pod:/api/v1/namespaces/bar/pods/redis-master
‚redis‘ service:/api/v1/namespaces/bar/services/redis-master
VMworld 2018 Content: Not for publication or distribution
15©2018 VMware, Inc.
Kubernetes Service
A Kubernetes Service defines a logical set of Pods, selected with matching labels
Serves multiple functions:• Service Discovery / DNS• East/West load balancing in
the Cluster (Type: ClusterIP)
• External load balancing for L4 TCP/UDP (Type: LoadBalancer)
• External access to the service through the nodes IPs (Type: NodePort)
Redis Slave Pods
redis-slave svc
10.24.0.5
ClusterIP172.30.0.24
Web Front-EndPods
10.24.2.7
▶ kubectl describe svc redis-slaveName: redis-slaveNamespace: defaultLabels: name=redis-slaveSelector: name=redis-slaveType: LoadBalancerIP: 172.30.0.24LoadBalancer Ingress: 134.247.200.20Port: <unnamed> 6379/TCPEndpoints: 10.24.0.5:6379,
10.24.2.7:6379
DNS:
redis-slave.<ns>.cluster.local 172.30.0.24
ExternalIP134.247.200.20
DNS:
redis-slave.external.com 134.247.200.20
VMworld 2018 Content: Not for publication or distribution
16©2018 VMware, Inc.
Kubernetes Ingress
A Kubernetes Ingress Object is a L7 LoadBalancing rule that binds a hostname and url to a Service
The LoadBalancer Datapath can be implemented as an external Load Balancer or as a K8s Pod
Web Front-EndPods (shop svc)
http://www.bikeshop.com/shop
Web Front-EndPods (special-offers svc)
http://www.bikeshop.com/special-offers
LoadBalancer Datapath
(External or K8s Pods)
▶ kubectl describe ingress bikeshop-ingress-shopName: bikeshop-shopNamespace: bikeshopAddress: 100.64.240.9,134.247.200.1Default backend: default-http-backend:80 (<none>)
Rules:Host Path Backends---- ---- --------www.bikeshop.com /shop
web-svc-1:80 (<none>)
External IP: 134.247.200.1
DNS: *.bikeshop.com 134.247.200.1
VMworld 2018 Content: Not for publication or distribution
17©2018 VMware, Inc.
Kubernetes Networking Topologies
Every Node is an IP Router and responsible for its Pod Subnet
Subnets are associated with Nodes, not Tenants
Physical Network Configuration is required
Non-multitenant routed topology
Nodeint eth0
10.240.0.4
int cbr0
10.24.2.1/24
10.24.2.2 10.24.2.3 10.24.2.4
ip route 10.24.1.0/24 10.240.0.3ip route 10.24.2.0/24 10.240.0.4
Nodeint eth0
10.240.0.3
int cbr0
10.24.1.1/24
10.24.1.2 10.24.1.3 10.24.1.4
net.ipv4.ip_forward=1
net.ipv4.ip_forward=1
VMworld 2018 Content: Not for publication or distribution
18©2018 VMware, Inc.
Kubernetes Networking Topologies
Overlays are typically used to avoid Physical Network Configuration
Subnets are still associated with Nodes, not Tenants
External outbound connectivity needs SNAT using the Nodes IP
External inbound connectivity needs Node Port or Ingress in Host Network Mode
Node-to-Node overlay topology
Nodeint eth0
10.240.0.4
int cbr0
10.24.2.1/24
10.24.2.2 10.24.2.3 10.24.2.4
Nodeint eth0
10.240.0.3
int cbr0
10.24.1.1/24
10.24.1.2 10.24.1.3 10.24.1.4
net.ipv4.ip_forward=1
net.ipv4.ip_forward=1
Overlay
Key-Value Store
VMworld 2018 Content: Not for publication or distribution
19©2018 VMware, Inc.
NSX-T Data Center & KubernetesDetails of the NSX-T Data Center integration with Kubernetes
VMworld 2018 Content: Not for publication or distribution
20©2018 VMware, Inc.
Key design goals of the NSX-T Data Center Kubernetes Integration
Don't stand in the way of the developer !
Provide solutions to map the Kubernetes
constructs to enterprise networking
constructs
Secure Containers, VMs and any other
endpoints with overarching Firewall
Policies
Provide visibility & troubleshooting tools to ease the
container adoption in the enterprise
VMworld 2018 Content: Not for publication or distribution
NSX-T K8s Integration – Namespaces & Pods
admin@k8s-master:~$ kubectl create namespace foonamespace ”foo" created
admin@k8s-master:~$ kubectl create namespace barnamespace ”bar" created
admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foodeployment "nginx-foo" created
admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bardeployment "nginx-bar" created
Namespace: foo Namespace: bar
NSX / K8s topology
10.24.0.0/24 10.24.1.0/24
10.24.2.0/24
NAT boundary
NAT boundary
K8s nodesK8s Masters
Dynamic per Namespace Topology
VMworld 2018 Content: Not for publication or distribution
NSX-T K8s Integration – Routed Namespaces
admin@k8s-master:~$ vim no-nat-namespace.yaml
apiVersion: v1kind: Namespacemetadata:
name: no-nat-namespaceannotations:
ncp/no_snat: "true“
admin@k8s-master:~$ kubectl create –f no-nat-namespace.yamlnamespace ”no-nat-namespace" created
admin@k8s-master:~$ kubectl run nginx-no-nat --image=nginx –n no-nat-namespacedeployment "nginx-k8s" created
Namespace: no-nat-namespace
NSX / K8s topology
114.4.10.0/26
Direct Routing
114.4.10.64/26
K8s nodesK8s MastersVMworld 2018 Content: Not for publication or distribution
23©2018 VMware, Inc.
K8s / NSX-T Data Center Components
NCP is a software component provided by VMware in form of a container image, e.g. to be run as a K8s Pod.
NCP is build in a modular way, so that individual adapters can be added for different CaaS and PaaS systems at some point
NSX Container Plugin (NCP)
NCM Infra
K8s / OSAdapter
CloudFoundryAdapter
NSX Container Plugin
More…
NSX Manager
API Client
NSX Manager
NS: foo NS: barNSX/ K8s topology
K8s master
etcd
API-Server
Scheduler
VMworld 2018 Content: Not for publication or distribution
24©2018 VMware, Inc.
Tenancy / Topology MappingThe open source way
With most networking technologies in K8s the source IP of the traffic can't be mapped to the tenancy. This is the biggest hurdle today to get K8s integrated in enterprise IT environments
Node VM
IPTables(NAT)
vnic
mgmt IP
Pods
10.255.0.10/2410.255.0.9/24
172.16.1.11/24
Node VM
IPTables(NAT)
vnic
mgmt IP
Pods
10.255.1.3/2410.255.1.5/24
172.16.1.12/24
Physical or virtual Router
172.16.1.1/24
Tenant: fooTenant: barTenant: foo
Database (VM based or Physical)
Physical DC FirewallSNAT to Node IP
Did the traffic come from 'foo' or 'bar'?
SNAT to Node IP
VMworld 2018 Content: Not for publication or distribution
25©2018 VMware, Inc.
Tenancy / Topology MappingPersistent IPs for K8s Namespaces
With NSX-T each Tenant (Kubernetes Namespace) either gets its own SNAT IP (NAT Mode), or is directly identifiable by its source subnet (No NAT Mode)
Node VM
OpenvSwitch
10.12.5.5/2410.12.1.8/24
172.16.1.11/24
mgmt IP
vnic
Namesp. FooT1 router
PAS VMsT1 router
VLAN Trunk
NSX-T Logical Switch
Namesp. BarT1 router
172.16.1.1/24 10.12.1.1/24 10.12.5.1/24
Pods
Database (VM based or Physical)
Physical DC Firewall
A new SNAT IP is allocated on the T0 router for each Tenant for NAT Mode
In NAT Mode, the external DC Firewall and the DB can distinguish tenant 'foo' and tenant 'bar' using the source SNAT IP that is allocated to a specific Tenant.
Tenant: fooTenant: bar
In No-NAT Mode, the external DC Firewall and the DB can distinguish tenant 'foo' and tenant 'bar' using the source IP Subnet that is allocated to a specific Tenant.
VMworld 2018 Content: Not for publication or distribution
26©2018 VMware, Inc.
Infrastructure Teams can pre-create Firewall rules in existing DC physical Firewalls to allow traffic from specific workloads in K8s
The K8s user / DevOps can deploy applications that are easily identifiable in the physical network
With this feature a set of Kubernetes Workloads (Pods) can be assigned to use a specific IP or group of SNAT IPs to source their traffic from
Before this feature only a SNAT IP to a Kubernetes Namespace was assigned
Feature
Benefits
Persistent SNAT IP per K8s ServiceSpecifying the source IP Kubernetes Workloads using the K8s service
Tier0 LR
Corporate network
DB
allow – from: 134.247.100.10 (App) to: 134.247.200.9 (DB)
Tier1 LR
Kubernetes Namespace: Foo
Web-FrontendPods
App Logic Pods
K8s Svc for AppK8s Svc for Web
Namespace LS(s)
SNAT App Svc Pods to: 134.247.100.10For all other Pods
use namespace SNAT IP
VMworld 2018 Content: Not for publication or distribution
Central Visibility
With NSX-T you have deep visibility and troubleshooting tools.
VMworld 2018 Content: Not for publication or distribution
Kubernetes Metadata / NSX Logical Port Mapping
▶ kubectl get pod nsx-demo-rc-c7x65 -o yaml
apiVersion: v1kind: Podmetadata:creationTimestamp: 2018-07-25T12:05:56ZgenerateName: nsx-demo-rc-labels:
app: nsx-demoname: nsx-demo-rc-c7x65namespace: nsx-ujo
Metadata within Kubernetes like Namespace, Pod names, Labels all get copied to the NSX Logical Port as Port Tags
VMworld 2018 Content: Not for publication or distribution
Pre-Created Security Groups / Firewall rules (admin rules)NSX can be configured to collect ports and switches in dynamic security groups based on Tags (Kubernetes Metadata) and apply Firewall rules on them
Match on Port Tags
Matching Pods are part of the Group
Groups are used in Firewall sections as src and dst
VMworld 2018 Content: Not for publication or distribution
Support of Kubernetes Network Policy
Besides supporting admin pre-defined rules, NCP is also translating Kubernetes NetworkPolicy Objects to NSX security groups and Firewall rules
Admin pre-defined rules can be used concurrently in NSX, admin rules are put in sections before or after K8s network policy rules
apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:name: default-deny
spec:podSelector: {}policyTypes:- Ingress
apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:
name: nsx-demo-policyspec:
podSelector:matchLabels:
app: nsx-demopolicyTypes:- Ingressingress:- from:
- ipBlock:cidr: 100.64.160.11/32
ports:- port: 80
protocol: TCP
VMworld 2018 Content: Not for publication or distribution
Built-in Load Balancing
NCM Infra
K8s / OSAdapter
CloudFoundry Adapter
Libnetwork Adapter
NSX Container Plugin
More…
NSX Manager
API Client
NSX Manager
K8s master
etcd
API-Server
Scheduler
Virtual Server10.114.209.209HTTP and/or
HTTPS traffic
Server Pool 1
Server Pool 2Rule 2/bar/
Rule 1/foo/
LB Service
NCM Infra
K8s / OSAdapter
CloudFoundry Adapter
Libnetwork Adapter
NSX Container Plugin
More…
NSX Manager
API Client
NSX Manager
K8s master
etcd
API-Server
Scheduler
Virtual Server10.114.209.212TCP and/or
UDP traffic
Server Pool
LB Service
Built-in support for Ingress (L7 HTTP/HTTPS) and Svc Type LB (L4 TCP/UDP) in the NSX-K8s integration. Most other K8s networking choice don't support Svc Type LB (L4), and you need an additional technology like NGINX from Ingress (L7).
VMworld 2018 Content: Not for publication or distribution
K8s NSX Plugin – Child Interfaces
• CIFs (Child VIFs) are subinterfaces of parent VIFs (vnic/vm interfaces)
• The OVS on the Worker VM tags traffic coming from a POD with a local significant VLAN id
• NSX vSwitch on the Hypervisor have a logical (sub-interface) and is able to apply DFW rules to specific CIFs
• The OVS on the Worker is ‘standalone’, it is not controlled by the NSX control plane and is only programed by the NSX CNI Plugin
Hypervisor(ESXi &
KVM)
K8s WorkerVM
Pod
DFW
eth3
Pod
T1 Router
K8s WorkerVM
Pod
DFW
Pod
eth0
Minion Mgmt. IP Stack
eth0
Minion Mgmt. IP Stack
mgmtnetwork
OVS
mgmtnetwork
Vla
n1
vlan
2
cifcif
eth3
vlan
1
vlan
2
OVS
cifcif
NSX CNI
Plugin
NSX CNI
Plugin
VMworld 2018 Content: Not for publication or distribution
©2018 VMware, Inc. 38
NSX-T & Kubernetes -Demo
VMworld 2018 Content: Not for publication or distribution
Relational database used to store FAA registration relational data, e.g., Planes => Planetypes
No-SQL in-memory database used to store
high-churn Aircraft position data
Receives lives aircraft position data and feeds it
into redis no-sql cache
Automatic dependent surveillance broadcast
(ADS–B) - live feed(ADS–B)
Provides a REST API to frontend, takes data from MySQL and redis and correlates information
Python Flask and Bootstrap based Web-Frontent
http (web)
http (REST)TCP stream
RESP
RESP
MySQL
Application Overview
Planespotter: retrieve FAA aircraft registration data and airborne status
API App ServiceADS-B data feeder
Frontend
Thanks to Yves Fauser
VMworld 2018 Content: Not for publication or distribution
NSX-T Data Center Values for Containers
Enterprise-class Networking
Advanced Security
Enhanced Operations
Full Network Visibility
Enterprise Support
Unified VM-to-Container Networking
Micro-Segmentation
N S X - T V a l u e s f o r C o n t a i n e r s
F e a t u r e sVMworld 2018 Content: Not for publication or distribution
41©2018 VMware, Inc.
Join the NSX VMUG Communityvmug.com/nsxConnect with your Peerscommunities.vmware.com
Embrace the NSX Mindsetnsxmindset.comFind NSX Resourcesvmware.com/go/networkingRead the Network Virtualization Blogblogs.vmware.com/networkvirtualization
Where to get started
Attend the Networking and Security SessionsShowcases, breakouts, quick talks & group discussions
Visit the VMware BoothProduct overviews, use-case demos
Visit Technical Partner BoothsIntegration demos – Infrastructure, security, operations, visibility, and more
Meet the ExpertsJoin our experts in an intimate roundtable discussion
Free Hands-on Labslabs.hol.vmware.com
Virtual Cloud Network Guided Demovcndemo.com
VMware Education - Training and Certificationvmware.com/go/nsxtraining
Free NSX Training on Courseravmware.com/go/coursera
Engage and Learn Experience
Try Take
VMworld 2018 Content: Not for publication or distribution
DON’T FORGET TO FILL OUT YOUR SURVEY.
#vmworld #NET1677BE
VMworld 2018 Content: Not for publication or distribution
THANK YOU!
#vmworld #NET1677BE
VMworld 2018 Content: Not for publication or distribution
top related