vmworld 2013: designing network virtualization for data-centers: greenfield design and migration...

Designing Network Virtualization for Data-Centers:

Greenfield Design and Migration Scenarios

Ben Basler, VMware

Roberto Mari, VMware

TEX5350

#TEX5350

2 Confidential

This presentation may contain product features that are currently

under development.

This overview of new technology represents no commitment from

VMware to deliver these features in any generally available

product.

Features are subject to change, and must not be included in

contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new technologies or features

discussed or presented have not been determined.

Disclaimer

3

Session Objectives

Discuss Network virtualization Design principles and introduce the

VMware NSX network virtualization product

Discuss the different Data-Center design options and the benefits

that Network virtualization brings

Analyze more in detail design aspects of the VMware NSX Network

Virtualization for the vSphere/ESXi deployments

Discuss migration scenarios with a mix of physical and virtual

workloads

4

Reference Sessions

NET5716 – Advanced NSX Architecture

NET5266 – Bringing Network Virtualization to VMware

Environments with NSX

NET5270 – Virtualized Network Services Model with NSX

NET5521 - vSphere Distributed Switch - Design and Best Practices

SEC5582 – Multi-site Deployments with VMware NSX

NET5796 – Network Virtualization Concepts for Network

Administrators

NET5184 - Designing Your Next Generation Datacenter for Network

Virtualization

See NSX in action in lab HOL-SDC-1319 and HOL-SDC-1303

5

Agenda

Network Virtualization design Advantages

VMware NSX Solution for Datacenter Network Virtualization

Datacenter virtualization Design principles

NSX and vSphere Design Considerations

Data Center Migration scenarios

Key Takeaways

6

Agenda


• Network Virtualization Design Advantages

• Why Network Virtualization ?





Key Takeaways

7

Network Virtualization Design Advantages

Network virtualization leverages existing datacenter networks

Overlay Technology can be deployed over common datacenter

network design

Flexible and Rapid deployment options via adoption of virtual

Networking and Logical Networks.

SOFTWARE-DEFINED

DATACENTER

All infrastructure is virtualized and delivered as a

service, and the control of this datacenter is

entirely automated by software.

8

Why Network Virtualization ?

Physical Infrastructure

Compute Virtualization Abstraction Layer

Physical Network: A Barrier to

Software Defined Data Center

SOFTWARE-DEFINED DATACENTER SERVICES

VDC

• Provisioning is slow

• Placement is limited

• Mobility is limited

• Hardware dependent

• Operationally intensive

9

Why Network Virtualization ?

Physical Infrastructure

Compute Virtualization Abstraction Layer

Network Virtualization Abstraction Layer


VDC

Solution: Virtualize the Network

• Programmatic provisioning

• Place any workload anywhere

• Move any workload anywhere

• Decoupled from hardware

• Operationally efficient

10

Agenda



• VMware solutions and VMware NSX

• VMware NSX Functional System Overview




Key Takeaways

11

VMware NSX – the Network Virtualization Platform

Multi-Hypervisor

VMware NSX

Any Physical

Infrastructure

Compute, Storage & Network Hardware Independent


VDC


VDC


VDC

Multi-Cloud Management Platform

12

VMware NSX Functional System Overview

vSphere KVM XenServer Hyper-V*

vSwitch vSwitch vSwitch vSwitch

Hosts

Data Plane

Operations

UI

Logs/Stats

Control Plane Run-time state

Management Plane

API

API, config, etc.

HA, scale-out

NSX

Gateways

NSX Manager

NSX Controller

Consumption

Tenant UI

CMP

13

Agenda




• Virtual Networking and Physical Fabrics

• Terminology, Design Considerations and Features

• Virtualization and Distributed Service placement



Key Takeaways

14

Virtualization improves efficiency of compute clusters

Less than 40% Asset Utilization

to 90% Asset Utilization

Transformation

15

L2/L3 Traditional Fabrics - Physical Fabric Resource Utilization

Before NV With NV

MAC addresses

ARP entries

VLAN usage

STP load

# of VMs # of Tenants

VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM

Anim

ate

d S

lide

L3

L2

16

L2/L3 Traditional Fabrics - Physical Fabric Resource Utilization

Before NV With NV

MAC addresses

ARP entries

VLAN usage

STP load

# of VMs # of Tenants




L3

L2

Anim

ate

d S

lide

17

Traditional Physical Networking Configuration Tasks

L3

L2

Initial configuration

• Multi-chassis LAG

• Routing configuration

• SVIs/RVIs

• VRRP/HSRP

• STP

Instances/mappings

Priorities

Safeguards

• LACP

• STP

Instances/mappings

Priorities

Safeguards

Recurring configuration

• SVIs/RVIs

• VRRP/HSRP

• Advertise new subnets

• Access lists (ACLs)

• VLANs

• Adjust VLANs on trunks

• VLANs STP/MST

mapping

• VLANs STP/MST mapping

• Add VLANs on uplinks

• Add VLANs to server ports

Anim

ate

d S

lide

Configuration consistency !

18

Fabric Technology Trends

From 2- or 3-tier to spine/leaf

Density & bandwidth jump

ECMP for layer 3 (and layer 2, i.e.

TRILL)

Reduce network oversubscription

Wire & configure once

Uniform configurations

WAN/Internet

WAN/Internet

L3

L2

L3

L2

19

L3 Leaf-Spine Fabrics Simplified Configuration Tasks

Anim

ate

d S

lide

Initial configuration

• Multi-chassis LAG


• SVIs/RVIs

• VRRP/HSRP

• STP

Instances/mappings

Priorities

Safeguards

• LACP

• STP

Instances/mappings

Priorities

Safeguards

Recurring configuration

• SVIs/RVIs

• VRRP/HSRP

• Advertise new subnets

• Access lists (ACLs)

• VLANs

• Adjust VLANs on trunks

• VLANs STP/MST

mapping

• VLANs STP/MST mapping

• Add VLANs on uplinks

• Add VLANs to server ports

L3

L2

Simplified 2-Tier L3 Fabric

with Network virtualization

L3

L2


• SVIs/RVIs

20

Simplified Physical Infrastructure

Compute Racks

• Repeatable Rack design

• No VLANs for VMs

Infra/Storage Racks Edge Racks

WAN

Internet

• On/off ramp to physical

• Work with existing devices

• Multi-CMP

• Multi-VC

• Scalable

L2

L3

Higher Speed, Lower Cost, Non-Disruptive

21

Spine Nodes

…

…

Spine

Leaf

L3 point-to-point

10.99.1.0/31

… …

.2

.1

Spine L3 Only

(Route Table entries, ARP entries,

no mac consumption)

To Leaf

1

To Leaf N

…

L3 Downlinks

Spine connects to

leaf switches

• Interfaces configured

as routed point-to-point

L3 links.

• Links between spine

switches not required.

• In case of a spine to

leaf link failure, routing

protocol reroutes traffic

on the alternate paths.

• Aggregates the leaf

nodes and provide

connectivity between

racks.

22

Leaf Nodes

Servers facing ports have minimal

configuration

May use Link Aggregation Control

Protocol (LACP)

• Applies to same speed Interfaces

• Active/Active Bandwidth

• Fast failover times

801.Q trunks with a small set of

VLANs

• VTEP (VXLAN), management, storage,

vMotion Traffic

L2

L3

L3 Uplinks

VLAN

Boundary 802.1Q

Hypervisor 1

802.1Q Hypervisor n

. . .

23

Leaf Nodes

Storage

VXLAN

L3 Uplinks

VLAN

Boundary Mgmt: 10.66.1.x/26

vMotion 10.77.1.2/26

VXLAN: 10.88.1.x/26

Storage: 10.99.1.x/26

L2

L3

Management

802.1Q Hypervisor 1

802.1Q Hypervisor n

. . .

L3 ToR designs have

dynamic routing protocol

between leaf and spine.

• BGP, OSPF or ISIS can be used

• Rack advertises small set of

prefixes (one per VLAN/subnet).

• Equal cost paths to the other

racks prefixes.

• Switch provides default gateway

service for each VLAN subnet

vMotion

24

QoS in virtualized Data Center Designs

Leaf

Trust QoS marking

802.1Q Hypervisor

Trust or set QoS marking

No Marking/Reclassification

Spine

Virtualized environments can

carry different types of traffic

Hypervisor is a trusted

boundary, sets the respective

QoS values.

The physical switching

infrastructure to “trust” these

values. No reclassification is

necessary at the server facing

port of a leaf.

With congestion QoS values

are looked at for taking a

decision which traffic should

be queued up (and potentially

dropped) or prioritized.

25

Scalable Bandwidth Growth

WAN

Internet

Compute Racks

(1x10 Gbps) or (2x10 Gbps)

Infra Racks

1x10 Gbps

Edge Racks

(1x10 Gbps)

L2

L3

Spine

Leaf

Individual L3 Links or

L3 Port-channels (LACP)

Storage Racks

(2x10 Gbps)

1x10 Gbps

2x10 Gbps

Scale incrementally grows with: # of racks, # of paths from a leaf, #

of spines, # of ports on each Hypervisors Leaf or Spine node

Different racks with different bandwidth requirements.

Link speed and number of uplinks to satisfy bandwidth demands

26

VMware NSX Components

Compute Racks Infrastructure Racks Edge Racks

WAN

Internet

L2

L3

Controllers/Infra Mgmt.

• CMP, vCenter, SYSLOG,

network and storage mgmt.

• Massive scale

Hypervisor Service Modules

• Distributed network services (Switching,

Routing)

• Load Balancer, Switch, Firewall, Router

Gateway/Edge Software

• Integration with existing

physical infra.

• V to V / V to P

27

Network Virtualization – Networking functions in Virtual Space

Routing/NAT

Security/Firewalling

QoS

Port Mirroring

Counters

VM VM VM

Physical Fabric

NSX Controllers

NSX vSwitch

Anim

ate

d S

lide

VM-Aware Networking services

Networking services distributed at

the edge

Scale-out with the number of

Hypervisors

No choke points

28

Centralized vs. Distributed Services

VM

VM

VM

VM VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

29

Distributed Services with Network Virtualization

Perimeter NSX Edge

Dynamic Routing

(OSPF, BGP)

External

Networks

Web Logical

Switch App Logical Switch DB Logical Switch Web Logical

Switch App Logical Switch DB Logical Switch

Web Logical

Switch App Logical Switch DB Logical Switch

Logical Router Instance 1

Logical Router Instance 2

Logical Router Instance n

2 Tiers of Routing

• Distributed for E-W

• Perimeter for N-S

Dynamic Routing to

advertise Logical Networks Dynamic Routing

(OSPF, BGP)

30

Agenda





• vSphere Networking considerations

• Network Addressing and Management consideration

• vSphere scalability considerations


Key Takeaways

31

VMkernel Networking

Multi instance TCP/IP Stack

• Introduced with vSphere 5.5 and leveraged by:

• VXLAN

• NSX vSwitch transport network

• Separate routing table, ARP

table and default gateway per

stack instance

• Provides increased isolation and

reservation of networking

resources such as sockets,

buffers and heap memory

• Enables VXLAN VTEPs to use a

gateway independent from the

default TCP/IP stack

• Management, vMotion, FT, NFS,

iSCSI leverage the default

TCP/IP stack in 5.5

32

VMkernel Networking

Static Routing

• VMkernel VLANs do not extend beyond the rack in an L3 fabric design or

beyond the cluster with an L2 fabric, static routes are required for

Management, Storage and vMotion Traffic

• Host Profiles reduce the overhead of managing static routes and ensure

persistence

• Follow the RPQ (Request for Product Qualification) process for official

support of routed vMotion. Routing of IP Storage traffic also has some

caveats.

• A number of customers went

through RPQ and use routed

vMotion with full support today

• Future enhancements will

simplify ESXi host routing and

enable greater support for L3

network topologies

33

VMkernel Networking

Teaming Recommendations

• LACP (802.3ad) good option for optimal use of

available bandwidth and quick convergence.

• Load Based Teaming recommended to simplify

configuration and reduce dependencies on the

physical network, while using multiple uplinks

• NSX supports multiple VTEPs per host with VXLAN

• 2x 10Gbe NIC adapters per server is most common

• Proprietary network partitioning technologies increase

complexity and dependencies on 3rd party drivers

Overlay Networks are used for VMs

• Have only one VMkernel port per IP subnet

• Have a dedicated VLAN for each VMkernel interface

Considerations for vSphere Auto Deploy

• DHCP relay and IP helper to provision boot images

ESXi Host

NSX vSwitch

Physical Switch

See NET5521 –

VDS Design and Best Practices

34

vSphere Host (ESXi)

VMkernel Networking

L3 ToR Switch

Routed uplinks (ECMP)

Span

of V

LA

Ns

VLAN Trunk (802.1Q)

VLAN 66

Mgmt

10.66.1.25/26

GW: 10.66.1.1

VLAN 77

vMotion

10.77.1.25/26

GW: 10.77.1.1

VLAN 88

VXLAN

10.88.1.25/26

GW: 10.88.1.1

VLAN 99

Storage

10.99.1.25/26

GW: 10.99.1.1

SVI 66: 10.66.1.1/26

SVI 77: 10.77.1.1/26

SVI 88: 10.88.1.1/26

SVI 99: 10.99.1.1/26

Anim

ate

d S

lide

35

WAN

Internet

Rack 1

Infrastructure Racks Edge Racks

ToR

Rack 2 Rack N

ToR ToR ToR ToR

Compute Rack - IP Address Allocations and VLANs

Function Global VLAN ID IP Address

Management 66 10.66.R_id.x/26

vMotion 77 10.77.R_id.x/26

VXLAN 88 10.88.R_id.x/26

Storage 99 10.99.R_id.x/26

Note: Values for VLANs, IP addresses and masks are provided as an example. R_id is the rack or cluster number

vSphere Network Addressing

36

Management and Edge Rack Requirements

WAN

Internet

Leaf

L2

L3

L2

L3

L2

L3 L2

Edge 2 Edge 1

VMkernel VLANs

VLANs for Edge

VMs to Physical

Network

VMkernel VLANs

Edge N Edge N

Management Racks

• L2 between racks needed for

Management workloads such as

vCenter Server, NSX

Controllers, NSX Manager and

IP Storage

Leaf L2

L3

L2

L3

Mgmt N

Mgmt N Mgmt 1

VMkernel VLANs

VLANs for

Management

VMs

VMkernel VLANs

Edge Racks

• L2 between racks needed for

External 802.1q VLANs

Mgmt 2

L2

Edge 3 Edge 4

37

vSphere Scalability

Cluster Sizing

• vSphere 5.x HA: 32 hosts

Storage

• VAAI ATS removes SCSI reservation constraints on datastore sizing

Virtual Machines

• 10,000 powered on VMs per vCenter Server

Concurrent Operations

• Eight concurrent operations per vCenter Server

Networking

• NSX allows scaling of the network

independent to vCenter Server

• NSX for vSphere supports similar

scale. Logical networks are bound

to a vCenter Server at present

See NET5184 - Designing Your

Next Generation Datacenter for

Network Virtualization

38

Agenda






• Integrate Physical workloads with x86 NSX Gateways

• Integrate Physical workloads with 3rd party NSX Gateways

Key Takeaways

39

Migration scenarios: Connecting Physical and Virtual Worlds

Physical network (port, or VLAN)

NSX L2 Gateway

Logical network (VNI)

40

The VMware NSX Gateway

VM VM VM VM VM VM VM

Virtual Network Virtual Network

Hypervisor

NSX Gateway

VLAN VLAN

Physical

Virtual

e.g. Hosted Servers

e.g. Cloud Servers

1

1 2

2 3 4

3 4

Bare Metal, Physical Switch

or Virtual Appliance

41

NSX Gateway – What is the NSX Gateway for?

Connect physical workloads to logical Networks via a L2 service

Physical workloads are reachable on a specific VLAN or PPort

Logical Switch Logical View

Service Node

Physical View

NSX L2 Gateway

Logical network (VNI)

DNS-DHCP VLAN 10

VLAN 10 Database

VLAN 10

VLAN 10

Database

42

Connecting virtual and Physical: NSX x86 Gateways

Physical servers part of the compute infrastructure

NSX x86 Gateways w. 10Gbps performance

L2 Bridging and L3 Routing Services

P-to-V, P-to-P connectivity (STT or VXLAN)

Migrating/Virtualizing Legacy Services (same design)

Compute Racks Infra/Storage Racks Edge Racks

WAN

Internet

L2

L3

Hypervisors

w. virtual Workloads

Physical Servers

w. Hosted Services

NSX x86 Gateways

43

How do 3rd Party NSX Gateways work?

1. Registration of hardware appliance into NSX Controller (one time)

2. Creation of a L2 Gateway Service including the hardware

appliance physical port.

3. NSX API calls to connect a physical port/VLAN to a Logical

Switch

3rd Party Gateway

VXLAN

L2

L3 Virtual Network

L2

VMware NSX

1

2

3

44

Connecting virtual and Physical: NSX 3rd Party HW Gateways

3rd Party Gateways Controlled by VMware NSX

High Throughput L2 Bridging Service

P-to-V, P-to-P connectivity (VXLAN)

Same Design and Operational model

• Troubleshooting tools and Traffic statistics

Hypervisors

w. virtual Workloads

Physical Servers

w. Hosted Services

NSX 3rd Party

HW Gateways

Compute Racks Infra/Storage Racks Edge Racks

WAN

Internet

L2

L3

45

Agenda






Key Takeaways

46

VMware NSX: Key Takeaways…

… works great on top of existing networks and new fabrics

… network virtualization for vSphere and 3rd party Hypervisors

… unleashes the power of x86-based networking

… integrates physical compute with same provisioning/operations

See our NSX Demos at the main VMware Booth

and check out the NSX Hands-on Labs!

47

Reference Sessions

NET5716 – Advanced NSX Architecture

NET5266 – Bringing Network Virtualization to VMware

Environments with NSX

NET5270 – Virtualized Network Services Model with NSX

NET5521 - vSphere Distributed Switch - Design and Best Practices

SEC5582 – Multi-site Deployments with VMware NSX

NET5796 – Network Virtualization Concepts for Network

Administrators

NET5184 - Designing Your Next Generation Datacenter for Network

Virtualization

See NSX in action in lab HOL-SDC-1319 and HOL-SDC-1303

48

TAP Membership Renewal – Great Benefits

• TAP Access membership includes:

New TAP Access NFR Bundle

• Access to NDA Roadmap sessions at VMworld, PEX and Onsite/Online

• VMware Solution Exchange (VSX) and Partner Locator listings

• VMware Ready logo (ISVs)

• Partner University and other resources in Partner Central

• TAP Elite includes all of the above plus:

• 5X the number of licenses in the NFR Bundle

• Unlimited product technical support

• 5 instances of SDK Support

• Services Software Solutions Bundle

• Annual Fees

• TAP Access - $750

• TAP Elite - $7,500

• Send email to [email protected]

mailto:[email protected]


49

TAP Resources

TAP

• TAP support: 1-866-524-4966

• Email: [email protected]

• Partner Central: http://www.vmware.com/partners/partners.html

TAP Team

• Kristen Edwards – Sr. Alliance Program Manager

• Sheela Toor – Marketing Communication Manager

• Michael Thompson – Alliance Web Application Manager

• Audra Bowcutt –

• Ted Dunn –

• Dalene Bishop – Partner Enablement Manager, TAP

VMware Solution Exchange

• Marketplace support –

[email protected]

• Partner Marketplace @ VMware

booth pod TAP1


http://www.vmware.com/partners/partners.html






THANK YOU

Designing Network Virtualization for Data-Centers:

Greenfield Design and Migration Scenarios

Ben Basler, VMware

Roberto Mari, VMware

TEX5350

#TEX5350

vmworld 2013: designing network virtualization for data-centers: greenfield design and migration...

Technology

features virtualization

hypervisor vmware nsx

greenfield design

design aspects

network administrators

generation datacenter

vmware tex5350

vmware environments