architecture and design of vmware nsx-t for …...n virtualization network design n nsx-t design...

Architecture and Designof VMware NSX-T forWorkload Domains21 AUG 2018VMware Validated Design 4.3VMware NSX-T 2.2

Architecture and Design of VMware NSX-T for Workload Domains

VMware, Inc. 2

You can find the most up-to-date technical documentation on the VMware website at:

https://docs.vmware.com/

If you have comments about this documentation, submit your feedback to

[email protected]

Copyright © 2018 VMware, Inc. All rights reserved. Copyright and trademark information.

VMware, Inc.3401 Hillview Ave.Palo Alto, CA 94304www.vmware.com

https://docs.vmware.com/

mailto:[email protected]

http://pubs.vmware.com/copyright-trademark.html

Contents

About Architecture and Design of VMware NSX-T for Workload Domains 4

1 Applying the Guidance for NSX-T for Workload Domains 5

2 Architecture Overview 7

Physical Network Architecture 7

Virtual Infrastructure Architecture 9

3 Detailed Design 16

Physical Infrastructure Design 16

Virtual Infrastructure Design 21

VMware, Inc. 3

About Architecture and Design ofVMware NSX-T for Workload Domains

Architecture and Design of VMware NSX-T for Workload Domains provides detailed information about therequirements for software, tools, and external services to implement VMware NSX-T® in a shared edgeand compute cluster in an SDDC that is compliant with VMware Validated Design for Software-DefinedData Center.

PrerequisitesDeploy the management cluster according to VMware Validated Design for Software-Defined Data Centerat least in a single region. See the VMware Validated Design documentation page.

Intended AudienceThis design is intended for architects and administrators who want to deploy NSX-T in a virtualinfrastructure workload domain for tenant workloads.

Required VMware SoftwareIn addition to the VMware Validated Design for Software-Defined Data Center 4.3 deployment, you mustdownload NSX-T 2.2. You then deploy and configure NSX-T in the shared edge and compute clusteraccording to this guide. See VMware Validated Design Release Notes for more information aboutsupported product versions

VMware, Inc. 4

https://docs.vmware.com/en/VMware-Validated-Design/index.html

Applying the Guidance for NSX-T for Workload Domains 1The content in Architecture and Design of VMware NSX-T for Workload Domains replaces certain parts ofArchitecture and Design in VMware Validated Design for Software-Defined Data Center, also referred toas the Standard SDDC.

Before You Design the Compute Workload Domain withNSX-TBefore you follow this documentation, you must deploy the components for the SDDC managementcluster according to VMware Validated Design for Software-Defined Data Center at least in a singleregion. See Architecture and Design, Planning and Preparation and Deployment for Region A in the VMware Validated Design documentation.

n ESXi

n Platform Services Controller pair and Management vCenter Server

n NSX for vSphere

n vRealize Lifecycle Manager

n vSphere Update Manager

n vRealize Operations Manager

n vRealize Log Insight

n vRealize Automation with embedded vRealize Orchestrator

n vRealize Business

Designing a Compute Workload Domain with NSX-TNext, follow the guidance to design a compute workload domain with NSX-T deployed in this way:

n In general, use the guidelines about the compute workload domain and shared edge and computecluster in the following sections of Architecture and Design in VMware Validated Design for Software-Defined Data Center:

n Architecture Overview > Physical Infrastructure Architecture

n Architecture Overview > Virtual Infrastructure Architecture

VMware, Inc. 5

https://docs.vmware.com/en/VMware-Validated-Design/4.3/com.vmware.vvd.sddc-design.doc/GUID-44471514-D818-424F-AB32-57108003806C.html

https://docs.vmware.com/en/VMware-Validated-Design/4.3/com.vmware.vvd.sddc-plan.doc/GUID-75126B85-2186-46F9-9EA0-00FC55451A1F.html

https://docs.vmware.com/en/VMware-Validated-Design/4.3/com.vmware.vvd.sddc-deploya.doc/GUID-657DB777-D919-4C23-BA5E-B98D8A91CA8B.html

http://docs.vmware.com/en/VMware-Validated-Design/index.html

https://docs.vmware.com/en/VMware-Validated-Design/4.3/com.vmware.vvd.sddc-design.doc/GUID-44471514-D818-424F-AB32-57108003806C.html

n Detailed Design > Physical Infrastructure Design

n Detailed Design > Virtual Infrastructure Design

n For the sections that are available in both Architecture and Design of VMware NSX-T for WorkloadDomains and Architecture and Design, follow the design guidelines in Architecture and Design ofVMware NSX-T for Workload Domains.

First-Level Chapter Places to Use the Guidance for NSX-T

Architecture Overview n Physical Infrastructure Architecturen Workload Domain Architecturen Cluster Typesn Physical Network Architecture

n Network Transportn Infrastructure Network Architecturen Physical Network Interfaces

n Virtual Infrastructure Architecture

Detailed Design n Physical Infrastructure Designn Physical Design Fundamentalsn Physical Networking Designn Physical Storage Design

n Virtual Infrastructure Designn vCenter Server Design

n vCenter Server Deploymentn vCenter Server Networkingn vCenter Server Redundancyn vCenter Server Appliance Sizingn vSphere Cluster Designn vCenter Server Customizationn Use of TLS Certificates in vCenter Server

n Virtualization Network Designn NSX-T Design


VMware, Inc. 6

Architecture Overview 2VMware Validated Design for NSX-T enables IT organizations that have deployed VMware ValidatedDesign for Software-Defined Data Center 4.3 to create a shared edge and compute cluster that usesNSX-T capabilities.

This chapter includes the following topics:

n Physical Network Architecture

n Virtual Infrastructure Architecture

Physical Network ArchitectureVMware Validated Designs can use most physical network architectures.

Network TransportYou can implement the physical layer switch fabric of an SDDC by offering Layer 2 or Layer 3 transportservices. For a scalable and vendor-neutral data center network, use a Layer 3 transport.

VMware Validated Design supports both Layer 2 and Layer 3 transports. To decide whether to use Layer2 or Layer 3, consider the following factors:

n NSX-T service routers establish Layer 3 routing adjacency with the first upstream Layer 3 device toprovide equal cost routing for workloads.

n The investment you have today in your current physical network infrastructure.

n The benefits and drawbacks for both layer 2 and layer 3 designs.

Benefits and Drawbacks of Layer 2 TransportA design using Layer 2 transport has these considerations:

n In a design that uses Layer 2 transport, top of rack switches and upstream Layer 3 devices, such ascore switches or routers, form a switched fabric.

n The upstream Layer 3 devices end each VLAN and provide default gateway functionality.

n Uplinks from the top of rack switch to the upstream Layer 3 devices are 802.1Q trunks carrying allrequired VLANs.

Using a Layer 2 transport has the following benefits and drawbacks:

VMware, Inc. 7

Table 2‑1. Benefits and Drawbacks for Layer 2 Transport

Characteristic Description

Benefits n More design freedom.n You can span VLANs across racks.

Drawbacks n The size of such a deployment is limited because the fabricelements have to share a limited number of VLANs.

n You might have to rely on a specialized data centerswitching fabric product from a single vendor.

n Traffic between VLANs must traverse to upstream Layer 3device to be routed.

Figure 2‑1. Example Layer 2 Transport

ToR ToR

ESXiHost

Upstream L3 DeviceL3

10 GigE 10 GigE

L2

Upstream L3 Device

Benefits and Drawbacks of Layer 3 TransportA design using Layer 3 transport requires these considerations:

n Layer 2 connectivity is limited within the data center rack up to the top of rack switches.

n The top of rack switch ends each VLAN and provides default gateway functionality. The top of rackswitch has a switch virtual interface (SVI) for each VLAN.

n Uplinks from the top of rack switch to the upstream layer are routed point-to-point links. You cannotuse VLAN trunking on the uplinks.

n A dynamic routing protocol, such as BGP, connects the top of rack switches and upstream switches.Each top of rack switch in the rack advertises a small set of prefixes, typically one per VLAN orsubnet. In turn, the top of rack switch calculates equal cost paths to the prefixes it receives from othertop of rack switches.


VMware, Inc. 8

Table 2‑2. Benefits and Drawbacks of Layer 3 Transport

Characteristic Description

Benefits n You can select from many Layer 3 capable switch productsfor the physical switching fabric.

n You can mix switches from different vendors because ofgeneral interoperability between their implementation ofBGP.

n This approach is typically more cost effective because ituses only the basic functionality of the physical switches.

Drawbacks n VLANs are restricted to a single rack. The restriction canaffect vSphere Fault Tolerance, and storage networks.

Figure 2‑2. Example Layer 3 Transport

ToR

L3

ESXiHost

UpstreamSwitch

UpstreamSwitch

WAN/MPLSInternet

WAN/MPLSInternet

ToR

L2

10 GigE10 GigE

Virtual Infrastructure ArchitectureThe virtual infrastructure is the foundation of an operational SDDC. It contains the software-definedinfrastructure, software-defined networking and software-defined storage.

In the virtual infrastructure layer, access to the underlying physical infrastructure is controlled andallocated to the management and compute workloads. The virtual infrastructure layer consists of thehypervisors on the physical hosts and the control of these hypervisors. The management components ofthe SDDC consist of elements in the virtual management layer itself.

Figure 2‑3. Virtual Infrastructure Layer in the SDDC


VMware, Inc. 9

ServiceManagement

Portfolio Management

OperationsManagement

CloudManagement

Layer

Service Catalog

Self-Service Portal

Orchestration

BusinessContinuity

Fault Tolerance and Disaster

Recovery

Backup & Restore

Hypervisor

Pools of Resources

Virtualization Control

VirtualInfrastructure

Layer

Compute

Storage

Network

PhysicalLayer

Security

Replication Compliance

Risk

Governance

Virtual Infrastructure OverviewThe SDDC virtual infrastructure consists of workload domains. The SDDC virtual infrastructure includes amanagement workload domain that contains the management cluster and a virtual infrastructure workloaddomain that contains the shared edge and compute cluster.

Management ClusterThe management cluster runs the virtual machines that manage the SDDC. These virtual machines hostvCenter Server, vSphere Update Manager, NSX Manager, and other management components. Allmanagement, monitoring, and infrastructure services are provisioned to a vSphere cluster which provideshigh availability for these critical services. Permissions on the management cluster limit access only toadministrators. This limitation protects the virtual machines that are running the management, monitoring,and infrastructure services from unauthorized access. The management cluster leverages software-defined networking capabilities in NSX for vSphere.

The management cluster architecture and design is covered in the VMware Validated Design forSoftware-Defined Data Center. The NSX-T validated design does not include the design of themanagement cluster.

Shared Edge and Compute ClusterThe shared edge and compute cluster runs the NSX-T controllers, edge virtual machines and all tenantworkloads. The edge virtual machines are responsible for North-South routing between computeworkloads and the external network. This is often referred to as the on-off ramp of the SDDC.

The hosts in this cluster provide services such as high availability to the NSX-T controllers and edgevirtual machines.

Figure 2‑4. SDDC Logical Design


VMware, Inc. 10

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OS

APP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

Virtual InfrastructureManagement

NSX-VControllers

(Mgmt)

OtherManagementApplications

NSX-VEdges(Mgmt)

NSX-VManager(Mgmt)

NSX-TManager

(Compute)

NSX-TEdges

NSX-TControllers

ESXi ESXi ESXiESXi ESXi ESXi

SDDCPayload

Virtual Infrastructure Edge Compute

NSX-T Transport Zone

N-VDS (Compute) vDS (Mgmt)

NSX-V Transport Zone (Mgmt)

Shared Edge and Compute Cluster

Management Cluster

Managed by: Compute vCenter Server

Managed by:Management vCenter Server

Network: External(Internet/Corporate)

Network: Internal SDDC

vCenterServer(Mgmt)

vCenterServer

(Compute)

Transport NodesESXi

ESXi

Network Virtualization ComponentsThe NSX-T platform consists of several components that are relevant to the network virtualization design.

NSX-T PlatformNSX-T creates a network virtualization layer, which is an abstraction between the physical and virtualnetworks. You create all virtual networks on top of this layer.

Several components are required to create this network virtualization layer:

n NSX-T Manager

n NSX-T Controllers

n NSX-T Edge Nodes

n NSX-T Distributed Routers (DR)


VMware, Inc. 11

n NSX-T Service Routers (SR)

n NSX-T Logical Switches

These components are distributed in different planes to create communication boundaries and provideisolation of workload data from system control messages.

Data plane Performs stateless forwarding or transformation of packets based on tablespopulated by the control plane, reports topology information to the controlplane, and maintains packet level statistics.

The following traffic runs in the data plane:

n Workload data

n N-VDS virtual switch, distributed routing, and the distributed firewall inNSX-T

The data is carried over designated transport networks in the physicalnetwork.

Control plane Contains messages for network virtualization control. You place the controlplane communication on secure physical networks (VLANs) that areisolated from the transport networks for the data plane.

The control plane computes the runtime state based on configuration fromthe management plane. Control plane propagates topology informationreported by the data plane elements, and pushes stateless configuration toforwarding engines.

Control plane in NSX-T has two parts:

n Central Control Plane (CCP). The CCP is implemented as a cluster ofvirtual machines called CCP nodes. The cluster form factor providesboth redundancy and scalability of resources.

The CCP is logically separated from all data plane traffic, that is, afailure in the control plane does not affect existing data planeoperations.

n Local Control Plane (LCP). The LCP runs on transport nodes. It isadjacent to the data plane it controls and is connected to the CCP. TheLCP is responsible for programming the forwarding entries of the dataplane.

Management plane Provides a single API entry point to the system, persists user configuration,handles user queries, and performs operational tasks on all management,control, and data plane nodes in the system.


VMware, Inc. 12

For NSX-T, all querying, modifying, and persisting user configuration is inthe management plane. Propagation of that configuration down to thecorrect subset of data plane elements is in the control plane. As a result,some data belongs to multiple planes. Each plane uses this data accordingto stage of existence. The management plane also queries recent statusand statistics from the control plane, and under certain conditions directlyfrom the data plane.

The management plane is the only source of truth for the logical systembecause it is the only entry point for user configuration. You make changesusing either a RESTful API or the NSX-T user inteface.

For example, responding to a vSphere vMotion operation of a virtualmachine is responsibility of the control plane, but connecting the virtualmachine to the logical network is responsibility of the management plane.

Network Virtualization ServicesNetwork virtualization services include logical switches, routers, firewalls, and other components of NSX-T.

Logical Switch Reproduces switching functionality, broadcast, unknown unicast, andmulticast (BUM) traffic in a virtual environment that is decoupled from theunderlying hardware.

Logical switches are similar to VLANs because they provide networkconnections to which you can attach virtual machines. The virtual machinescan then communicate with each other over tunnels between ESXi hosts.Each logical switch has a virtual network identifier (VNI), like a VLAN ID.Unlike VLANs, VNIs scale beyond the limits of VLAN IDs.

Logical Router Provides North-South connectivity so that workloads canaccess externalnetworks, and East-West connectivity between logical networks.

A logical router is a configured partition of a traditional network hardwarerouter. It replicates the functionality of the hardware, creating multiplerouting domains in a single router. Logical routers perform a subset of thetasks that are handled by the physical router, and each can contain multiplerouting instances and routing tables. Using logical routers can be aneffective way to maximize router use, because a set of logical routers withina single physical router can perform the operations previously performed byseveral pieces of equipment.

n Distributed router (DR)


VMware, Inc. 13

A DR spans ESXi hosts whose virtual machines are connected to thislogical router, and edge nodes the logical router is bound to.Functionally, the DR is responsible for one-hop distributed routingbetween logical switches and logical routers connected to this logicalrouter.

n One or more (optional) service routers (SR).

An SR is responsible for delivering services that are not currentlyimplemented in a distributed fashion, such as stateful NAT.

A logical router always has a DR. A logical router has SRs when it is aTier-0 router, or when it is a Tier-1 router and has routing servicesconfigured such as NAT or DHCP.

NSX-T Edge Node Provides routing services and connectivity to networks that are external tothe NSX-T domain through a Tier-0 router over BGP or static routing.

You must deploy an NSX-T Edge for stateful services at either the Tier-0 orTier-1 logical routers.

NSX-T Edge Cluster Represents a collection of NSX-T Edge nodes that host multiple servicerouters in highly available configurations. At a minimum, deploy a singleTier-0 SR to provide external connectivity.

An NSX-T Edge cluster does not have a one-to-one relationship with avSphere cluster. A vSphere cluster can run multiple NSX-T Edge clusters.

Transport Node Participates in NSX-T overlay or NSX-T VLAN networking. Any node canserve as a transport node if it contains an NSX-T Virtual Distributed Switch(N-VDS) such as ESXi hosts and NSX-T Edges.

An ESXi host can serve as a transport node if it contains at least one N-VDS.

Transport Zone Controls which transport nodesa logical switch can reach. A transport zonecan span one or more vSphere clusters. Transport zones dictate whichESXi hosts and which virtual machines can participate in the use of aparticular network.

A transport zone defines a collection of ESXi hosts that can communicatewith each other across a physical network infrastructure. Thiscommunication happens over one or more interfaces defined as TunnelEndpoints (TEPs).


VMware, Inc. 14

When you create an ESXi host transport node and then add the node to atransport zone, NSX-T installs an N-VDS on the host. For each transportzone that the host belongs to, a separate N-VDS is installed. The N-VDS isused for attaching virtual machines to NSX-T logical switches and forcreating NSX-T logical router uplinks and downlinks.

NSX-T Controller As a component of the control plane, controls virtual networks and overlaytransport tunnels.

For stability and reliability of data transport, NSX-T deploys the NSX-TControllers as a cluster of three highly available virtual appliances. Theyare responsible for the programmatic deployment of virtual networks acrossthe entire NSX-T architecture.

Logical Firewall Responsible for traffic handling in and out the network according to firewallrules.

A logical firewall offers multiple sets of configurable Layer 3 and Layer 2rules. Layer 2 firewall rules are processed before Layer 3 rules. You canconfigure an exclusion list to exclude logical switches, logical ports, orgroups from firewall enforcement.

The default rule, located at the bottom of the rule table, is a catchall rule.The logical firewall enforces the default rule on packets that do not matchother rules. After the host preparation operation, the default rule is set tothe allow action. Change this default rule to a block action and enforceaccess control through a positive control model, that is, only traffic definedin a firewall rule can flow on the network.

Logical Load Balancer Provides high-availability service for applications and distributes thenetwork traffic load among multiple servers.

The load balancer accepts TCP, UDP, HTTP, or HTTPS requests on thevirtual IP address and determines which pool server to use.

Logical load balancer is supported only in an SR on the Tier-1 logicalrouter.


VMware, Inc. 15

Detailed Design 3The NSX-T detailed design considers both physical and virtual infrastructure design. It includes numbereddesign decisions and the justification and implications of each decision.

This chapter includes the following topics:

n Physical Infrastructure Design

n Virtual Infrastructure Design

Physical Infrastructure DesignThe physical infrastructure design includes design decision details for the physical network.

Figure 3‑1. Physical Infrastructure Design

ServiceManagement



CloudManagement

Layer

Service Catalog

Self-Service Portal

Orchestration

BusinessContinuity


Recovery

Backup & Restore

Hypervisor

Pools of Resources



Layer

Compute

Storage

Network

PhysicalLayer

Security


Risk

Governance

Physical Networking DesignDesign of the physical SDDC network includes defining the network topology for connecting the physicalswitches and the ESXi hosts, determining switch port settings for VLANs and link aggregation, anddesigning routing. You can use the VMware Validated Design guidance for design and deploymentwithmost enterprise-grade physical network architectures.

VMware, Inc. 16

Switch Types and Network ConnectivityFollow best practices for physical switches, switch connectivity, VLANs and subnets, and access portsettings.

Top of Rack Physical Switches

When configuring top of rack (ToR) switches, consider the following best practices:

n Configure redundant physical switches to enhance availability.

n Configure switch ports that connect to ESXi hosts manually as trunk ports. Virtual switches arepassive devices and do not support trunking protocols, such as Dynamic Trunking Protocol (DTP).

n Modify the Spanning Tree Protocol (STP) on any port that is connected to an ESXi NIC to reduce thetime to transition ports over to the forwarding state, for example using the Trunk PortFast featurefound in a Cisco physical switch.

n Provide DHCP or DHCP Helper capabilities on all VLANs used by TEP VMkernel ports. This setupsimplifies the configuration by using DHCP to assign IP address based on the IP subnet in use.

n Configure jumbo frames on all switch ports, inter-switch link (ISL), and switched virtual interfaces(SVIs).

Top of Rack Connectivity and Network Settings

Each ESXi host is connected redundantly to the ToR switches SDDC network fabric by two 10 GbE ports.Configure the ToR switches to provide all necessary VLANs via an 802.1Q trunk. These redundantconnections use features in vSphere Distributed Switch and NSX-T to guarantee that no physicalinterface is overrun and available redundant paths are used.

Figure 3‑2. Host to ToR Connectivity

ToR ToR

ESXiHost

10 GigE 10 GigE

VLANs and Subnets

Each ESXi host uses VLANs and corresponding subnets.

Follow these guidelines:

n Use only /24 subnets to reduce confusion and mistakes when handling IPv4 subnetting.


VMware, Inc. 17

n Use the IP address .253 as the (floating) interface with .251 and .252 for Virtual Router RedundancyProtocol (VRPP) or Hot Standby Routing Protocol (HSRP).

n Use the RFC1918 IPv4 address space for these subnets and allocate one octet by region andanother octet by function.

Access Port Network Settings

Configure additional network settings on the access ports that connect the ToR switches to thecorresponding servers.

Spanning Tree Protocol(STP)

Although this design does not use the Spanning Tree Protocol, switchesusually include STP configured by default. Designate the access ports astrunk PortFast.

Trunking Configure the VLANs as members of a 802.1Q trunk with the managementVLAN acting as the native VLAN.

MTU Set MTU for all VLANS and SVIs (Management, vMotion, VXLAN, andStorage) to jumbo frames for consistency purposes.

DHCP Helper Configure a DHCP helper (sometimes called a DHCP relay) on all TEPVLANs.

Physical Network Design DecisionsThe physical network design decisions determine the physical layout and use of VLANs. They alsoinclude decisions on jumbo frames and on other network-related requirements such as DNS and NTP.

Physical Network Design Decisions

Routing protocols NSX-T supports only the BGP routing protocol.

DHCP Helper Set the DHCP helper (relay) to point to a DHCP server by IPv4 address.


VMware, Inc. 18

Table 3‑1. Physical Network Design Decisions

Decision ID Design Decision Design Justification Design Implication

NSXT-PHY-NET-001

Implement the following physicalnetwork architecture:n One 10 GbE or faster port on

each ToR switch for ESXi hostuplinks

n No EtherChannel(LAG/LACP/vPC) configurationfor ESXi host uplinks

n Layer 3 device that supportsBGP

n Guarantees availability during aswitch failure.

n Uses BGP asthe only dynamicrouting protocol that is supproted byNSX-T.

n Might limit the hardwarechoice.

n Requires dynamicrouting protocolconfiguration in thephysical network.

NSXT-PHY-NET-002

Use a physical network that isconfigured for BGP routingadjacency.

n Supports flexibility in networkdesign for routing multi-site andmulti-tenancy workloads.

n Uses BGP as the only dynamicrouting protocol that is supproted byNSX-T.

Requires BGP configurationin the physical network.

NSXT-PHY-NET-003

Use two ToR switches for each rack. Supports the use of two 10-GbE links toeach server and provides redundancyand reduces the overall designcomplexity.

Requires two ToR switchesper rack which can increasecosts.

NSXT-PHY-NET-004

Use VLANs to segment physicalnetwork functions.

n Supports physical networkconnectivity without requiring manyNICs.

n Isolates the different networkfunctions of the SDDC so that youcan have differentiated services andprioritized traffic as needed.

Requires uniformconfiguration andpresentation on all the trunksmade available to the ESXihosts.

Additional Design Decisions

Additional design decisions deal with static IP addresses, DNS records, and the required NTP timesource.


VMware, Inc. 19

Table 3‑2. IP Assignment, DNS, and NTP Design Decisions


NSXT-PHY-NET-005

Assign static IP addresses to allmanagement components in theSDDC infrastructure except forNSX-T TEPs which are assignedby DHCP.

n Provides a consistent access point formanagement interfaces.

n Uses DHCP to ease the IP management ofthese interfaces.NSX-T TEPs are notaccessed by users and are only used fortunnels between ESXi hosts and edgevirtual machines.

Requires accurateIP addressmanagement.

NSXT-PHY-NET-006

Create DNS records for allmanagement nodes to enableforward, reverse, short, and FQDNresolution.

Ensures consistent resolution of managementnodes using both IP address (reverse lookup)and name resolution.

None.

NSXT-PHY-NET-007

Use an NTP time source for allmanagement nodes.

Maintains accurate and synchronized timebetween management nodes.

None.

Jumbo Frames Design Decisions

IP storage throughput can benefit from the configuration of jumbo frames. Increasing the per-framepayload from 1500 bytes to the jumbo frame setting improves the efficiency of data transfer. You mustconfigure jumbo frames end-to-end, which is feasible in a LAN environment. Select an MTU that matchesthe MTU of the physical switch ports.

According to the purpose of the workload, determine whether to configure jumbo frames on a virtualmachine. If the workload consistently transfers large amounts of network data, configure jumbo frames, ifpossible. In that case, confirm that both the virtual machine operating system and the virtual machineNICs support jumbo frames.

Using jumbo frames also improves the performance of vSphere vMotion.

Note The Geneve overlay requires an MTU value of 1600 bytes or greater.

Table 3‑3. Jumbo Frames Design Decisions


NSXT-PHY-NET-008

Configure the MTU size to at least 9000bytes (jumbo frames) on the physical switchports, vSphere Distributed Switches,vSphere Distributed Switch port groups, andN-VDS switches that support the followingtraffic types.n Geneve (overlay)n vSANn vMotionn NFSn vSphere Replication

Improves trafficthroughput.

To support Geneve,increase the MTU settingto a minimum of 1600bytes.

When adjusting the MTU packetsize, you must also configure theentire network path (VMkernelports, virtual switches, physicalswitches, and routers) to supportthe same MTU packet size.


VMware, Inc. 20

Virtual Infrastructure DesignThe virtual infrastructure design includes the NSX-T components that make up the virtual infrastructurelayer.

Figure 3‑3. Virtual Infrastructure Layer in the SDDC

ServiceManagement



CloudManagement

Layer

Service Catalog

Self-Service Portal

Orchestration

BusinessContinuity


Recovery

Backup & Restore

Hypervisor

Pools of Resources



Layer

Compute

Storage

Network

PhysicalLayer

Security


Risk

Governance

vSphere Cluster DesignThe cluster design must consider the workload that the cluster handles. Different cluster types in thisdesign have different characteristics.

vSphere Cluster Design Decision BackgroundWhen you design the cluster layout in vSphere, consider the following guidelines:

n Use fewer, larger ESXi hosts, or more, smaller ESXi hosts.

n A scale-up cluster has fewer, larger ESXi hosts.

n A scale-out cluster has more, smaller ESXi hosts.

n Compare the capital costs of purchasing fewer, larger ESXi hosts with the costs of purchasing more,smaller ESXi hosts. Costs vary between vendors and models.

n Evaluate the operational costs of managing a few ESXi hosts with the costs of managing more ESXihosts.

n Consider the purpose of the cluster.

n Consider the total number of ESXi hosts and cluster limits.


VMware, Inc. 21

vSphere High Availability DesignVMware vSphere High Availability (vSphere HA) protects your virtual machines in case of ESXi hostfailure by restarting virtual machines on other hosts in the cluster when an ESXi host fails.

vSphere HA Design Basics

During configuration of the cluster, the ESXi hosts elect a master ESXi host. The master ESXi hostcommunicates with the vCenter Server system and monitors the virtual machines and secondary ESXihosts in the cluster.

The master ESXi host detects different types of failure:

n ESXi host failure, for example an unexpected power failure

n ESXi host network isolation or connectivity failure

n Loss of storage connectivity

n Problems with virtual machine OS availability

Table 3‑4. vSphere HA Design Decisions


NSXT-VI-VC-001 Use vSphere HA toprotect all clustersagainst failures.

vSphere HA supports a robust levelof protection for both ESXi host andvirtual machine availability.

You must provide sufficient resources onthe remaining hosts so that virtualmachines can be migrated to thosehosts in the event of a host outage.

NSXT-VI-VC-002 Set vSphere HA HostIsolation Response toPower Off.

vSAN requires that the HA IsolationResponse be set to Power Off and torestart VMs on available ESXi hosts.

VMs are powered off in case of a falsepositive and an ESXi host is declaredisolated incorrectly.

vSphere HA Admission Control Policy Configuration

The vSphere HA Admission Control Policy allows an administrator to configure how the clusterdetermines available resources. In a smaller vSphere HA cluster, a larger proportion of the clusterresources are reserved to accommodate ESXi host failures, based on the selected policy.


VMware, Inc. 22

The following policies are available:

Host failures the clustertolerates

vSphere HA ensures that a specified number of ESXi hosts can fail andsufficient resources remain in the cluster to fail over all the virtual machinesfrom those ESXi hosts.

Percentage of clusterresources reserved

vSphere HA reserves a specified percentage of aggregate CPU andmemory resources for failover.

Specify Failover Hosts When an ESXi host fails, vSphere HA attempts to restart its virtualmachines on any of the specified failover ESXi hosts. If restart is notpossible, for example, the failover ESXi hosts have insufficient resources orhave failed as well, then vSphere HA attempts to restart the virtualmachines on other ESXi hosts in the cluster.

Shared Edge and Compute Cluster DesignTenant workloads run on the ESXi hosts in the shared edge and compute cluster. Because ofthe sharednature of the cluster, NSX-T Controllers and Edge virtual machines also run in this cluster. The designdecisions determine the number of ESXi hosts and vSphere HA settings and several other characteristicsof the shared edge and compute cluster.

Table 3‑5. Shared Edge and Compute Cluster Design Decisions


NSXT-VI-VC-003

Create a shared edge andcompute cluster thatcontains tenant workloadsand NSX-T Controllersand Edge virtualmachines.

Limits the footprint of the design bysaving the use of a dedicated vSphereedge cluster.

In a shared cluster there must be sperateVLANs and subnets between the theESXi host overlay VMkernel ports andEdge virtual machine overlay ports.

NSXT-VI-VC-004

Configure AdmissionControl for one ESXi hostfailure and percentage-based failover capacity.

vSphere HA protects the tenantworkloads and NSX-T Controllerinstances and Edge virtual machinesin the event of an ESXi host failure.vSphere HA powers on virtualmachines from the failed ESXi hostson any remaining ESXi hosts.

Only a single ESXi host failure istolerated before potential resourcecontention.

NSXT-VI-VC-005

Create a shared edge andcompute cluster with aminimum of four ESXihosts.

Allocating four ESXi hosts provides fullredundancy within the cluster.

Four ESXi hosts is the smallest startingpoint for the shared edge and computecluster for redundancy and performancethus increasing cost.


VMware, Inc. 23

Table 3‑5. Shared Edge and Compute Cluster Design Decisions (Continued)


NSXT-VI-VC-006

Create a resource pool forthe three NSX-TControllers and two largesized edge virtualmachines with a CPUshare level of High, amemory share of normal,and 80 GB memoryreservation.

The NSX-T components control allnetwork traffic in and out of the SDDC,and update route information for inter-SDDC communication. In a contentionsituation,these virtual machines mustreceive all the resources required.

During contention NSX-T componentsreceive more resources than all otherworkloads.As a result, monitoring andcapacity management must be aproactive activity.

NSXT-VI-VC-007

Create a resource pool forall user virtual machineswith a CPU share value ofNormal and a memoryshare value of Normal.

Creating virtual machines outside of aresource pool would have a negativeimpact on all other virtual machinesduring contention. In a shared edgeand compute cluster, the SDDC edgevirtual machines must be guaranteedresources above all other workloadsas to not impact network connectivity.Setting the share values to normalgives the SDDC edges more shares ofresources during contention ensuringnetwork traffic is not impacted.

During contention,tenantvirtual machinesmight be starved for resources andexperience poor performance. Proactivelyperform monitoring and capacitymanagement, add capacity or dedicatean edge cluster before contention occurs.

NSXT-VI-VC-008

Create a host profile forthe shared edge andcompute cluster.

Using host profiles simplifiesconfiguration of ESXi hosts andensures settings are uniform acrossthe cluster.

After you add the ESXi hosts asTransport Nodes, the host profile are nolonger usable and must be removed fromthe cluster.

The host profile is only useful for initialcluster deployment and configuration.

Compute Cluster DesignAs the SDDC expands, you can add compute-only clusters. Tenant workloads run on the ESXi hosts inthe compute cluster instances. Multiple compute clusters are managed by the Compute vCenter Serverinstance. The design determines vSphere HA settings for the compute cluster.


NSXT-VI-VC-009

Configure vSphere HA to usepercentage-based failovercapacity to ensure n+1availability.

Using explicit host failoverlimits the total availableresources in a cluster.

The resources of one ESXi host in thecluster is reserved which can causeprovisioning to fail if resources areexhausted.

Virtualization Network DesignDesign the virtualization network according to the business goals of your organization. Prevent alsounauthorized access, and provide timely access to business data.

This network virtualization design uses vSphere and NSX-T to implement virtual networking.


VMware, Inc. 24

Virtual Network Design GuidelinesThis VMware Validated Design follows high-level network design guidelines and networking bestpractices.

Design Goals

You can apply the following high-level design goals to your environment:

n Meet diverse needs. The network must meet the diverse needs of many different entities in anorganization. These entities include applications, services, storage, administrators, and users.

n Reduce costs. Server consolidation alone reduces network costs by reducing the number of requirednetwork ports and NICs, but you should determine a more efficient network design. For example,configuring two 10 GbE NICs with VLANs might be more cost effective than configuring a dozen 1-GbE NICs on separate physical networks.

n Boost performance. You can achieve performance improvements and decrease the time required toperform maintenance by providing sufficient bandwidth, which reduces contention and latency.

n Improve availability. You usually improve availability by providing network redundancy.

n Support security. Youcan support an acceptable level of security through controlled access whererequired and isolationwhere necessary.

n Improve infrastructure functionality. You can configure the network to support vSphere features suchas vSphere vMotion, vSphere High Availability, and vSphere Fault Tolerance.

Best Practices

Follow the networking best practices throughout your environment.

n Separate network services from one another for greater security and better performance.

n Use Network I/O Control and traffic shaping to guarantee bandwidth to critical virtual machines.During network contention, these critical virtual machines receive a higher percentage of thebandwidth.

n Separate network services on an NSX-T Virtual Distributed Switch (N-VDS) by attaching them tological switches with different VLAN IDs.

n Keep vSphere vMotion traffic on a separate network. When migration with vMotion occurs, thecontents of the memory of the guest operating system is transmitted over the network. You can placevSphere vMotion on a separate network by using a dedicated vSphere vMotion VLAN.

n When using pass-through devices with Linux kernel version 2.6.20 or an earlier guest OS, avoid MSIand MSI-X modes. These modes have significant performance impact.

n For best performance, use VMXNET3 virtual machine NICs.

n Ensure that physical network adapters connected to the same virtual switch are also connected to thesame physical network.


VMware, Inc. 25

Network Segmentation and VLANs

You separate different types of traffic for access security and to reduce contention and latency.

High latency on a network can impact performance. Some components are more sensitive to high latencythan others. For example, reducing latency is important on the IP storage and the vSphere FaultTolerance logging network, because latency on these networks can negatively affect the performance ofmultiple virtual machines.

According to the application or service, high latency on specific virtual machine networks can alsonegatively affect performance. Use information gathered from the current state analysis and frominterviews with key stakeholder and SMEs to determine which workloads and networks are especiallysensitive to high latency.

Virtual Networks

Determine the number of networks or VLANs that are required according tothe type of traffic.

n vSphere operational traffic.

n Management

n Geneve (overlay)

n vMotion

n vSAN

n NFS Storage

n vSphere Replication

n Traffic that supports the services and applications of the organization.

Virtual SwitchesVirtual switches simplify the configuration process by providing single pane of glass view for performingvirtual network management tasks.

Virtual Switch Design Background

vSphere Distributed Switch andNSX-T Virtual Distributed Switch (N-VDS) offer several enhancementsover standard virtual switches.

Centralizedmanagement

n A distributed switch is created and centrally managed on a vCenterServer system. The switch configuration is consistent across ESXihosts.


VMware, Inc. 26

n An N-VDS is created and centrally managed in NSX-T Manager. Theswitch configuration is consistent across ESXi and edge transportnodes.

Centralized management saves time, reduces mistakes, and lowersoperational costs.

Additional features Distributed switches offer features that are not available on standard virtualswitches. Some of these features can be useful to the applications andservices running in the organization’s infrastructure. For example, NetFlowand port mirroring provide monitoring and troubleshooting capabilities to thevirtual infrastructure.

Consider the following caveats for distributed switches:

n vSphere Distributed Switches are manageable only when vCenter Server is available. vCenter Servertherefore becomes a Tier-1 application.

n N-VDS instances are manageable only when the NSX-T Manager is available. NSX-T Managertherefore becomes a Tier-1 application.

Virtual Switch Design Decisions

The virtual switch design decisions determine the use and placement of specific switch types.


NSXT-VI-NET-001

Use N-VDS for the NSX-T based sharededge and compute cluster, and more NSX-T based compute clusters.

You need N-VDS for overlaytraffic.

N-VDS is not compatible withvSphere host profiles.

Shared Edge and Compute Cluster Switches

The shared edge and compute cluster uses a single N-VDS with a certain configuration for handled traffictypes, NIC teaming, and MTU size.

Table 3‑6. Virtual Switches for the Shared Edge and Compute Cluster

N-VDS Switch Name FunctionNumber ofPhysical NIC Ports Teaming Policy MTU

sfo01-w01-nvds01 n ESXi Managementn vSphere vMotionn vSANn NFSn Geneve Overlay (TEP)n Uplink trunking (2) for the NSX-T

Edge instances

2 n Load balance source forsystem traffic

n Faliover order for overlaytraffic

9000

sfo01-w01-uplink01 n Uplink to enable ECMP 1 Failover order 9000

sfo01-w01-uplink02 n Uplink to enable ECMP 1 Failover order 9000


VMware, Inc. 27

Figure 3‑4. Virtual Switch Design for ESXi Hosts in the Shared Edge and Compute Cluster

nic0 nic1

VLAN NFS

VLAN vMotion

VLAN Overlay (TEP)

VLAN vSAN

Sample ESXi Shared Edge and Compute Host

sfo01-w01-nvds01

VLAN ESXi Management

VLAN Trunking Uplink01

VLAN Trunking Uplink02

VLAN Trunking Edge Overlay (TEP)

Table 3‑7. Virtual Switches in the Shared Edge and Compute Cluster by Physical NIC

N-VDS Switch vmnic Function

sfo01-w01-nvds01 0 Uplink

sfo01-w01-nvds01 1 Uplink

Table 3‑8. Logical Switches in the Shared Edge and Compute Cluster

N-VDS Switch Logical Switch Name

sfo01-w01-nvds01 sfo01-w01-nvds01-management

sfo01-w01-nvds01 sfo01-w01-nvds01-vmotion

sfo01-w01-nvds01 sfo01-w01-nvds01-vsan

sfo01-w01-nvds01 sfo01-w01-nvds01-overlay

sfo01-w01-nvds01 sfo01-w01-nvds01-uplink01

sfo01-w01-nvds01 sfo01-w01-nvds01-uplink02

sfo01-w01-nvds01 sfo01-w01-nvds01-nfs


VMware, Inc. 28

Table 3‑8. Logical Switches in the Shared Edge and Compute Cluster(Continued)

N-VDS Switch Logical Switch Name

sfo01-w01-uplink01 sfo01-w01-uplink01

sfo01-w01-uplink02 sfo01-w01-uplink02

Table 3‑9. VMkernel Adapters for the Shared Edge and Compute Cluster

N-VDS Switch Logical Switch Name Enabled Services

sfo01-w01-nvds01 sfo01-w01-nvds01-management Management Traffic

sfo01-w01-nvds01 sfo01-w01-nvds01-vmotion vMotion Traffic

sfo01-w01-nvds01 sfo01-w01-nvds01-vsan vSAN

sfo01-w01-nvds01 sfo01-w01-nvds01-nfs --

Note When the NSX-T Edge virtual machine is on an N-VDS, it must use a different VLAN ID andsubnet from the ESXi hosts overlay (TEP) VLAN ID and subnet.

ESXi host TEP VMkernel ports are automatically created when you configure an ESXi host as a transportnode.

NIC TeamingYou can use NIC teaming to increase the network bandwidth available in a network path, and to providethe redundancy that supports higher availability.

Benefits and Overview

NIC teaming helps avoid a single point of failure and provides options for load balancing of traffic. Toreduce further the risk of a single point of failure, build NIC teams by using ports from multiple NIC andmotherboard interfaces.

Create a single virtual switch with teamed NICs across separate physical switches.

NIC Teaming Design Background

For a predictable level of performance, use multiple network adapters in one of the followingconfigurations.

n An active-passive configuration that uses explicit failover when connected to two separate switches.

n An active-active configuration in which two or more physical NICs in the server are assigned theactive role.

This validated design uses a non-LAG active-active configuration using the route based on physical NICload algorithm for vSphere Distributed Switch and load balance source algorithm for N-VDS. By using thisconfiguration, network cards remain active instead of remaining idle until a failure occurs.


VMware, Inc. 29

Table 3‑10. NIC Teaming and Policy

Design Quality Active-Active Active-Passive Comments

Availability ↑ ↑ Using teaming regardless of the option increases the availability of theenvironment.

Manageability o o Neither design option impacts manageability.

Performance ↑ o An active-active configuration can send traffic across either NIC, therebyincreasing the available bandwidth. This configuration provides a benefit ifthe NICs are being shared among traffic types and Network I/O Control isused.

Recoverability o o Neither design option impacts recoverability.

Security o o Neither design option impacts security.

Legend: ↑ = positive impact on quality; ↓ = negative impact on quality; o = no impact on quality.

Table 3‑11. NIC Teaming Design Decisions


NSXT-VI-NET-002 In the shared edge and computecluster, use the Load balancesource teaming policy on N-VDS.

NSX-T Virtual Distributed Switch(N-VDS)supports Load balance source and Failoverteaming policies. When you use the Loadbalance source policy, both physical NICs can beactive and carry traffic.

None.

Geneve OverlayGeneve provides the overlay capability in NSX-T to create isolated, multi-tenant broadcast domainsacross data center fabrics, and enables customers to create elastic, logical networks that span physicalnetwork boundaries.

The first step in creating these logical networks is to abstract and pool the networking resources. By usingthe Geneve overlay, NSX-T abstracts the network into a pool of capacity and separates the consumptionof these services from the underlying physical infrastructure. This model is similar to the model vSphereuses to abstract compute capacity from the server hardware to create virtual pools of resources that canbe consumed as a service. You can then organize the pool of network capacity in logical networks thatare directly attached to specific applications.

Geneve is a tunneling mechanism which provides extensibility while still using the offload capabilities ofNICs for performance improvement.

Geneve works by creating Layer 2 logical networks that are encapsulated in UDP packets. A Segment IDin every frame identifies the Geneve logical networks without the need for VLAN tags. As a result, manyisolated Layer 2 networks can coexist on a common Layer 3 infrastructure using the same VLAN ID.


VMware, Inc. 30

In the vSphere architecture, the encapsulation is performed between the virtual NIC of the guest VM andthe logical port on the virtual switch, making the Geneve overlay transparent to both the guest virtualmachines and the underlying Layer 3 network. The Tier-0 router performs gateway services betweenoverlay and non-overlay hosts, for example, a physical server or the Internet router. The NSX-T Edgevirtual machine translates overlay segment IDs to VLAN IDs, so that non-overlay hosts can communicatewith virtual machines on an overlay network.

The edge cluster hosts all NSX-T Edge virtual machine instances that connect to the corporate networkfor secure and centralized network administration.

Table 3‑12. Geneve Overlay Design Decisions


NSXT-VI-NET-003

Use NSX-T to introduceoverlay networks forworkloads.

Simplifies the networkconfiguration by using centralizedvirtual network management.

n Requires additional compute andstorage resources to deploy NSX-T components.

n Might require more training inNSX-T.

NSXT-VI-NET-004

Use overlay networks withNSX-T Edge virtual machinesand distributed routing toprovide virtualized networkcapabilities to workloads.

Creates isolated, multi-tenantbroadcast domains across datacenter fabrics to deploy elastic,logical networks that spanphysical network boundaries.

Requires configuring transportnetworks with an MTU size of at least1600 bytes.

vMotion TCP/IP StackUse the vMotion TCP/IP stack to isolate traffic for vSphere vMotion and to assign a dedicated defaultgateway for vSphere vMotion traffic.

By using a separate TCP/IP stack, you can manage vSphere vMotion and cold migration traffic accordingto the topology of the network, and as required for your organization.

n Route the traffic for the migration of virtual machines by using a default gateway that is different fromthe gateway assigned to the default stack on the ESXi host.

n Assign a separate set of buffers and sockets.

n Avoid routing table conflicts that might otherwise appear when many features are using a commonTCP/IP stack.

n Isolate traffic to improve security.

Table 3‑13. vMotion TCP/IP Stack Design Decision


NSXT-VI-NET-005

Use the vMotionTCP/IP stack forvSphere vMotiontraffic.

By using the vMotion TCP/IP stack,vSphere vMotion traffic can beassigned a default gateway on its ownsubnet and can go over Layer 3networks.

The vMotion TCP/IP stack is notavailable in the VMkernel adaptercreation wizard of vSphere DistributedSwitch. You must create the VMkerneladapter directly on the ESXi host.


VMware, Inc. 31

NSX DesignThis design implements software-defined networking by using VMware NSX-T. By using NSX-T,virtualization delivers for networking what it has already delivered for compute and storage.

In much the same way that server virtualization programmatically creates, snapshots, deletes, andrestores software-based virtual machines (VMs), NSX network virtualization programmatically creates,snapshots, deletes, and restores software-based virtual networks. The result is a transformative approachto networking that not only enables data center managers to achieve orders of magnitude better agilityand economics, but also supports a vastly simplified operational model for the underlying physicalnetwork. NSX-T is a nondisruptive solution. You can deploy it on any IP network, including existingtraditional networking models and next-generation fabric architectures, regardless of the vendor.

When administrators provision workloads, network management is a time-consuming task. You spendmost time configuring individual components in the physical infrastructure and verifying that networkchanges do not affect other devices that are using the same physical network infrastructure.

The need to pre-provision and configure networks is a major constraint to cloud deployments wherespeed, agility, and flexibility are critical requirements. Pre-provisioned physical networks enables rapidcreation of virtual networks and faster deployment times of workloads using the virtual network. As longas the physical network that you need is already available on the ESXi host to host the workload, pre-provisioning physical networks works well. However, if the network is not available on a given ESXi host,you must find an ESXi host with the available network and spare capacity to run your workload in yourenvironment.

To get around this bottleneck, decouple virtual networks from their physical counterparts. This, in turn,requires that you can programmatically recreate all physical networking attributes that are required byworkloads in the virtualized environment. Because network virtualization supports the creation of virtualnetworks without modification of the physical network infrastructure, faster network provisioning ispossible.

NSX-T DesignNSX-T components are not dedicated to a specific vCenter Server instance or vSphere construct. Youcan share them across different vSphere environments.

Although an NSX-T deployment is not associated with a vCenter Server instance, it supports only single-region deployments. This design discusses a shared edge and compute cluster and compute clusters in asingle region.


VMware, Inc. 32

Table 3‑14. NSX-T Design Decisions

Decision ID Design Decision Design Justification Design Implications

NSXT-VI-SDN-001

Deploy a single NSX-T Manager toconfigure and manage the NSX-T basedshared edge and compute cluster and alladditional NSX-T based compute clustersin a single region.

Provides support for software-definednetworking (SDN) capabilities such asload balancing, firewalls and logicalswitching.

You must install andconfigure NSX-TManager in a highlyavailable managementcluster.

NSXT-VI-SDN-002

In the management cluster, add the NSX-T Manager to the distributed firewallexclusion list of NSX for vSphere.

Ensures that the management planeis still available if a misconfigurationof the distributed firewall of NSX forvSphere occurs.

None.

Figure 3‑5. NSX-T Architecture

Management Cluster


ManagementvCenter Server

NSX-V Manager

NSX-T Manager

NSX-V Controller 1

NSX-V Controller 2

NSX-V Controller 3

NSX-T Controller 1

NSX-T Controller 2

NSX-T Controller 3

ComputevCenter Server

NSX-T ComponentsThe following sections describe the components in the solution and how they are relevant to the networkvirtualization design.


VMware, Inc. 33

NSX-T Manager

NSX-T Manager provides the graphical user interface (GUI) and the RESTful API for creating,configuring, and monitoring NSX-T components, such as logical switches.

NSX-T Manager implements the management plane for the NSX-T infrastructure. NSX-T Managerprovides an aggregated system view and is the centralized network management component of NSX-T. Itprovides a method for monitoring and troubleshooting workloads attached to virtual networks. It providesconfiguration and orchestration of the following services:

n Logical networking components, such as logical switching and routing

n Networking and edge services

n Security services and distributed firewall

NSX-T Manager also provides a RESTful API endpoint to automate consumption. Because of thisarchitecture, you can automate all configuration and monitoring operations using any cloud managementplatform, security vendor platform, or automation framework.

The NSX-T Management Plane Agent (MPA) is an NSX-T Manager component that is available on eachESXi host. The MPA is in charge of persisting the desired state of the system and for communicating non-flow-controlling (NFC) messages such as configuration, statistics, status, and real time data betweentransport nodes and the management plane.

Table 3‑15. NSX-T Manager Design Decisions


NSXT-VI-SDN-003

Deploy NSX-T Manager as a large sizevirtual appliance.

The large-size appliance supports morethan 64 ESXi hosts. The small-sizeappliance is for proof of concept and themedium size only supports up to 64 ESXihosts.

The large size requiresmore resources in themanagement cluster.

NSXT-VI-SDN-004

n Grant administrators access to boththe NSX-T Manager UI and itsRESTful API endpoint.

n Restrict end-user access to theRESTful API endpoint configuredfor end-user provisioning, such asvRealize Automation or PivotalContainer Service (PKS).

Ensures that tenants or non-provider staffcannot modify infrastructure components.

End-users typically interact only indirectlywith NSX-T from their provisioning portal.Administrators interact with NSX-T usingits UI and API.

End users have accessonly to end-pointcomponents.

NSX-T Controller

An NSX-T Controller controls virtual networks and overlay transport tunnels.

For stability and reliability of data transport, the NSX-T Controller is deployed as a cluster of three highlyavailable virtual appliances that are responsible for the programmatic deployment of virtual networksacross the entire NSX-T architecture.

The CCP is logically separated from all data plane traffic, that is, a failure in the control plane does notaffect existing data plane operations. The controller provides configuration to other NSX-T Controllercomponents such as the logical switches, logical routers, and edge virtual machine configuration.


VMware, Inc. 34

Table 3‑16. NSX-T Controller Design Decision


NSXT-VI-SDN-005 Deploy the NSX-T Controllercluster with three members for highavailability and scale.

The high availability of NSX-T Controllerreduces the downtime period if a failure ofone physical ESXi host occurs.

None.

NSX-T Virtual Distributed Switch

An NSX-T Virtual Distributed Switch (N-VDS) runs on ESXi hosts and provides physical traffic forwarding.It transparently provides the underlying forwarding service that each logical switch relies on. To achievenetwork virtualization, a network controller must configure the ESXi host virtual switch with network flowtables that form the logical broadcast domains the tenant administrators define when they create andconfigure logical switches.

NSX-T implements each logical broadcast domain by tunneling VM-to-VM traffic and VM-to-logical routertraffic using the Geneve tunnel encapsulation mechanism. The network controller has a global view of thedata center and ensures that the ESXi host virtual switch flow tables are updated as VMs are created,moved, or removed.

Table 3‑17. NSX-T N-VDS Design Decision


NSXT-VI-SDN-006 Deploy an N-VDS instance to eachESXi host in the shared edge andcompute cluster.

ESXi hosts in the shared edge andcompute cluster provide tunnel endpointsfor Geneve overlay encapsulation.

None.

Logical Switching

NSX-T logical switches create logically abstracted segments to which you can connect tenant workloads.A single logical switch is mapped to a unique Geneve segment that is distributed across the ESXi hosts ina transport zone. The logical switch supports line-rate switching in the ESXi host without the constraintsof VLAN sprawl or spanning tree issues.

Table 3‑18. NSX-T Logical Switching Design Decision


NSXT-VI-SDN-007 Deploy all workloads onNSX-T logical switches.

To take advantage of features such asdistributed routing, tenant workloadsmust be connected to NSX-T logicalswitches.

You must perform all networkmonitoring must be performed in theNSX-T Manager UI or vRealizeNetwork Insight.

Logical Routers

NSX-T logical routers provide North-South connectivity so that workloads can access external networks,and East-West connectivity between different logical networks.


VMware, Inc. 35

A logical router is a configured partition of a traditional network hardware router. It replicates thefunctionality of the hardware, creating multiple routing domains in a single router. Logical routers performa subset of the tasks that are handled by the physical router, and each can contain multiple routinginstances and routing tables. Using logical routers can be an effective way to maximize router use,because a set of logical routers within a single physical router can perform the operations previouslyperformed by several pieces of equipment.

n Distributed router (DR)

A DR spans ESXi hosts whose virtual machines are connected to this logical router, and edge nodesthe logical router is bound to. Functionally, the DR is responsible for one-hop distributed routingbetween logical switches and logical routers connected to this logical router.

n One or more (optional) service routers (SR).

An SR is responsible for delivering services that are not currently implemented in a distributedfashion, such as stateful NAT.

A logical router always has a DR. A logical router has SRs when it is a Tier-0 router, or when it is a Tier-1router and has routing services configured such as NAT or DHCP.

Tunnel Endpoint

Tunnel endpoints enable ESXi hosts to participate in an NSX-T overlay. The NSX-T overlay deploys aLayer 2 network on top of an existing Layer 3 network fabric by encapsulating frames inside packets andtransferring the packets over an underlying transport network. The underlying transport network can beanother Layer 2 networks or it can cross Layer 3 boundaries. The Tunnel Endpoint (TEP) is theconnection point at which the encapsulation and decapsulation take place.

NSX-T Edges

NSX-T Edges provide routing services and connectivity to networks that are external to the NSX-Tdeployment. You use an NSX-T Edge for establishing external connectivity from the NSX-T domain byusing a Tier-0 router using BGP or static routing. Additionally, you deploy an NSX-T Edge to supportnetwork address translation (NAT) services at either the Tier-0 or Tier-1 logical routers.

The NSX-T Edge connects isolated, stub networks to shared uplink networks by providing commongateway services such as NAT, and dynamic routing.

Logical Firewall

NSX-T uses handles traffic in and out the network according to firewall rules.

A logical firewall offers multiple sets of configurable Layer 3 and Layer 2 rules. Layer 2 firewall rules areprocessed before Layer 3 rules. You can configure an exclusion list to exclude logical switches, logicalports, or groups from firewall enforcement.

The default rule, that is located at the bottom of the rule table, is a catchall rule. The logical firewallenforces the default rule on packets that do not match other rules. After the host preparation operation,the default rule is set to the allow action. Change this default rule to a block action and enforce accesscontrol through a positive control model, that is, only traffic defined in a firewall rule can flow on thenetwork.


VMware, Inc. 36

Logical Load Balancer

The NSX-T logical load balancer offers high-availability service for applications and distributes thenetwork traffic load among multiple servers.

The load balancer accepts TCP, UDP, HTTP, or HTTPS requests on the virtual IP address anddetermines which pool server to use.

Logical load balancer is supported only on the Tier-1 logical router.

NSX-T Network Requirements and SizingNSX-T requirements impact both physical and virtual networks.

Physical Network Requirements

Physical requirements determine the MTU size for networks that carry overlay traffic, dynamic routingsupport, time synchronization through an NTP server, and forward and reverse DNS resolution.

Requirement Comments

Provide an MTU size of 1600 or greater on anynetwork that carries Geneve overlay traffic must.

Geneve packets cannot be fragmented. The MTU size must be largeenough to support extra encapsulation overhead.

This design uses an MTU size of 9000 for Geneve traffic. See Table 3‑3.

Enable dynamic routing support on the upstreamLayer 3 devices.

You use BGP on the upstream Layer 3 devices to establish routingadjacency with the Tier-0 SRs.

Provide an NTP server. The NSX-T Manager requires NTP settings that synchronize it with the restof the environment.

Establish forward and reverse DNS resolution forall management VMs.

The NSX-T Controllers do not require DNS entries.

NSX-T Component Specifications

When you size the resources for NSX-T components, consider the compute and storage requirements foreach component, and the number of nodes per component type.

Size of NSX Edge services gateways might vary according to tenant requirements. Consider all options insuch a case.

Virtual Machine vCPU Memory (GB) Storage (GB) Quantity per NSX-T Deployment

NSX-T Manager 8 (Large) 32 (Large) 140 (Large) 1

NSX-T Controller 4 16 120 3

NSX-T Edge virtualmachine

2 (Small) 4 (Small) 120 (Small) Numbers vary per use case. At least two edgedevices are required to enable ECMP routing.

4 (Medium) 8 (Medium) 120 (Medium)

8 (Large) 16 (Large) 120 (Large)


VMware, Inc. 37

Table 3‑19. NSX Edge VM Sizing Design Decision


NSXT-VI-SDN-008 Use large-size NSX Edge virtualmachines.

The large-size appliance provides all theperformance characteristics if a failure occurs.

None.

Network Virtualization Conceptual Design for NSX-TThis conceptual design for NSX-T provides the network virtualization design of the logical componentsthat handle the data to and from tenant workloads in the environment.

The network virtualization conceptual design includes a perimeter firewall, a provider logical router, andthe NSX-T logical router. It also considers the external network, internal workload networks, and themanagement network.

Figure 3‑6. NSX-T Conceptual Overview

VM VMVM VM VMVM

Internet MPLS

MgmtNetwork

External Network

Perimeter Firewall

Upstream Layer 3Devices

NSX-Т DistributedRouter (DR)

Internal Tenant Networks (Logical Switches)

vNIC-Level Distributed Firewall

NSX-T Service Router(SR)


VMware, Inc. 38

The conceptual design has the following components.

External Networks Connectivity to and from external networks is through the perimeter firewall.

Perimeter Firewall The firewall exists at the perimeter of the data center to filter Internet traffic.

Upstream Layer 3Devices

The upstream Layer 3 devices are behind the perimeter firewall and handleNorth-South traffic that is entering and leaving the NSX-T environment. Inmost cases, this layer consists of a pair of top of rack switches orredundant upstream Layer 3 devices such as core routers.

NSX-T Logical Router(SR)

The SR component of the NSX-T Tier-0 Logical Router is responsible forestablishing eBGP peering with the Upstream Layer 3 devices and enablingNorth-South routing.

NSX-T Logical Router(DR)

The DR component of the NSX-T Logical Router is responsible for East-West routing.

Management Network The management network is a VLAN-backed network that supports allmanagement components such as NSX-T Manager and NSX-T Controllers.

Internal WorkloadNetworks

Internal workload networks are NSX-T logical switches and provideconnectivity for the tenant workloads. Workloads are directly connected tothese networks. Internal workload networks are then connected to a DR.

Cluster Design for NSX-TThe NSX-T design uses management, and shared edge and compute clusters. You can add morecompute clusters for scale-out, or different workload types or SLAs.

The logical NSX-T design considers the vSphere clusters and defines the place where each NSXcomponent runs.


VMware, Inc. 39

Figure 3‑7. NSX-T Cluster Design

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OS

APP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

Virtual InfrastructureManagement

NSX-VControllers

(Mgmt)

OtherManagementApplications

NSX-VEdges(Mgmt)

NSX-VManager(Mgmt)

NSX-TManager

(Compute)

NSX-TEdges

NSX-TControllers

ESXi ESXi ESXiESXi ESXi ESXi

SDDCPayload

Virtual Infrastructure Edge Compute

NSX-T Transport Zone

N-VDS (Compute) vDS (Mgmt)

NSX-V Transport Zone (Mgmt)


Management Cluster

Managed by: Compute vCenter Server

Managed by:Management vCenter Server

Network: External(Internet/Corporate)

Network: Internal SDDC

vCenterServer(Mgmt)

vCenterServer

(Compute)

Transport NodesESXi

ESXi

Management Cluster

The management cluster contains all components for managing the SDDC. This cluster is a corecomponent of the VMware Validated Design for Software-Defined Data Center. For information about themanagement cluster design, see the Architecture and Design documentation in VMware Validated Designfor Software-Defined Data Center.

NSX-T Edge Node Cluster

The NSX-T Edge cluster is a logical grouping of NSX-T Edge virtual machines. These NSX-T Edge virtualmachines run in the vSphere edge cluster and provide North-South routing for the workloads in thecompute clusters.


VMware, Inc. 40


In the shared edge and compute cluster, ESXi hosts are prepared for NSX-T. As a result, they can beconfigured as transport nodes and can participate in the overlay network. All tenant workloads, NSX-TControllers, and NSX-T Edge virtual machines run in this cluster.

Table 3‑20. Cluster Design Decisions


NSXT-VI-SDN-009

For the compute stack, donot dedicate an edge cluster.

Simplifies configuration andminimizes the number of ESXihosts required for initialdeployment.

The NSX-T Controllers and Edge virtualmachines of the compute stack aredeployed in the shared edge and computecluster.

Because of the shared nature of the cluster,you must scale out the cluster as computeworkloads are added to avoid an impact onnetwork performance.

NSXT-VI-SDN-010

Deploy at least two large-size NSX-T Edge virtualmachines in the shared edgeand compute cluster.

Creates NSX-T Edge cluster,and meets availability and scalerequirements.

You must add the edge virtual machines astransport nodes before you add them to theNSX-T Edge cluster.

NSXT-VI-SDN-011

Apply vSphere DistributedResource Scheduler(vSphere DRS) anti-affinityrules to NSX-T Controllers.

Prevents controllers fromrunning on the same ESXi hostand thereby risking their highavailability capability.

Requires at least four physical hosts toguarantee the three NSX-T Controllerscontinue to run if an ESXi host failureoccurs.

Additional configuration is required to set upanti-affinity rules.

NSXT-VI-SDN-012

Apply vSphere DRS anti-affinity rules to the virtualmachines of the NSX-T Edgecluster.

Prevents the NSX-T Edgevirtual machines from runningon the same ESXi host andcompromising their highavailability.


High Availability of NSX-T Components

The NSX-T Manager runs on the management cluster. vSphere HA protects the NSX-T Manager byrestarting the NSX-T Manager virtual machine on a different ESXi host if a primary ESXi host failureoccurs.

The NSX-T Controller nodes run on the shared edge and compute cluster. vSphere DRS anti-affinity rulesprevent NSX-T Controller nodes from running on the same ESXi host.

The data plane remains active during outages in the management and control planes although theprovisioning and modification of virtual networks is impaired until those planes become available again.

The NSX-T Edge virtual machines are deployed on the shared edge and compute cluster. vSphere DRSanti-affinity rules prevent NSX-T Edge virtual machines that belong to the same NSX-T Edge cluster fromrunning on the same ESXi host.

NSX-T SRs for North-South routing are configured in equal-cost multi-path (ECMP) mode that supportsroute failover in seconds.


VMware, Inc. 41

Design Decisions on the Cluster Design

The NSX-T design for vSphere cluster configuration for controller and edge high availability translates inthe following design decisions:

Table 3‑21. Cluster Design Decisions


NSXT-VI-SDN-009

For the compute stack, donot dedicate an edge cluster.

Simplifies configuration andminimizes the number of ESXihosts required for initialdeployment.

The NSX-T Controllers and Edge virtualmachines of the compute stack aredeployed in the shared edge and computecluster.

Because of the shared nature of the cluster,you must scale out the cluster as computeworkloads are added to avoid an impact onnetwork performance.

NSXT-VI-SDN-010

Deploy at least two large-size NSX-T Edge virtualmachines in the shared edgeand compute cluster.

Creates NSX-T Edge cluster,and meets availability and scalerequirements.

You must add the edge virtual machines astransport nodes before you add them to theNSX-T Edge cluster.

NSXT-VI-SDN-011

Apply vSphere DistributedResource Scheduler(vSphere DRS) anti-affinityrules to NSX-T Controllers.

Prevents controllers fromrunning on the same ESXi hostand thereby risking their highavailability capability.

Requires at least four physical hosts toguarantee the three NSX-T Controllerscontinue to run if an ESXi host failureoccurs.


NSXT-VI-SDN-012

Apply vSphere DRS anti-affinity rules to the virtualmachines of the NSX-T Edgecluster.

Prevents the NSX-T Edgevirtual machines from runningon the same ESXi host andcompromising their highavailability.


Replication Mode of Logical SwitchesThe control plane decouples NSX-T from the physical network, and handles the broadcast, unknownunicast, and multicast (BUM) traffic in the logical switches.

The following options are available for BUM replication on logical switches.


VMware, Inc. 42

Table 3‑22. BUM Replication Mode of NSX-T Logical Switches

BUM Replication Mode Description

Hierarchical Two-Tier In this mode, the ESXi host transport nodes are groupedaccording to their TEP IP subnet. One ESXi host in eachsubnet is responsible for replication to a ESXi host in anothersubnet. The receiving ESXi host replicates the traffic to theESXi hosts in its local subnet.

The source ESXi host transport node knows about the groupsbased on information it has received from the NSX-T controlcluster. The system can select an arbitrary ESXi host transportnode as the mediator for the source subnet if the remotemediator ESXi host node is available.

Head-End In this mode, the ESXi host transport node at the origin of theframe to be flooded on a logical switch sends a copy to everyother ESXi host transport node that is connected to this logicalswitch.

Table 3‑23. Design Decisions on Logical Switch Replication Mode


NSXT-VI-SDN-013 Use hierarchical two-tierreplication on all logicalswitches.

Hierarchical two-tier replication is more efficientby reducing the number of ESXi hosts thesource ESXi host must replicate traffic to.

None.

Transport Zone DesignTransport zones determine which hosts can participate in the use of a particular network. A transportzone identifies the type of traffic, VLAN or overlay, and the N-VDS name. You can configure one or moretransport zones. A transport zone does not represent a security boundary.

Table 3‑24. Transport Zones Design Decisions


NSXT-VI-SDN-014

Create a single transportzone for all overlay traffic.

Ensures all logical switches areavailable to all ESXi hosts and edgevirtual machines configured asTransport Nodes.

None.

NSXT-VI-SDN-015

Create a VLAN transportzone for ESXi hostVMkernel ports.

Enables the migration of ESXi hostVMkernel ports to N-VDS.

The N-VDS name must match theN-VDS name in the overlaytransport zone.

NSXT-VI-SDN-016

Create two transport zonesfor edge virtual machineuplinks.

Enables the edge virtual machines touse equal-cost multi-path routing(ECMP).

You must specify a VLAN range,that is use VLAN trunking, on thelogical switches used as theuplinks.


VMware, Inc. 43

Network I/O Control Design for NSX-TWhen a Network I/O Control profile is attached to an N-VDS, during contention the switch allocatesavailable bandwidth according to the configured shares, limit, and reservation for each vSphere traffictype.

How Network I/O Control Works

Network I/O Control enforces the share value specified for the different traffic types only when there isnetwork contention. When contention occurs, Network I/O Control applies the share values set to eachtraffic type. As a result, less important traffic, as defined by the share percentage, is throttled, grantingaccess to more network resources to more important traffic types.

Network I/O Control also supports the reservation of bandwidth for system traffic according to the overallpercentage of available bandwidth.

Network I/O Control Heuristics

The following heuristics can help with design decisions.

Shares vs. Limits When you use bandwidth allocation, consider using shares instead of limits.Limits impose hard limits on the amount of bandwidth used by a traffic floweven when network bandwidth is available.

Limits on NetworkResource Pools

Consider imposing limits on a resource pool. For example, set a limit onvSphere vMotion traffic to avoid oversubscription at the physical networklevel when multiple vSphere vMotion data transfers are initiated on differentESXi hosts at the same time. By limiting the available bandwidth forvSphere vMotion at the ESXi host level, you can prevent performancedegradation for other traffic.

Network I/O Control Design Decisions

Based on the heuristics, this design has the following decisions.

Table 3‑25. Network I/O Control Design Decisions


NSXT-VI-SDN-017

Create and attach a NetworkI/O Control Policy on all N-DVSswitches.

Increases resiliency and performance of thenetwork.

If configured incorrectly,Network I/O Controlmight impact networkperformance for criticaltraffic types.

NSXT-VI-SDN-018

Set the share value forvSphere vMotion traffic to Low(25).

During times of network contention, vSpherevMotion traffic is not as important as virtualmachine or storage traffic.

During times of networkcontention, vMotiontakes longer than usualto complete.


VMware, Inc. 44

Table 3‑25. Network I/O Control Design Decisions (Continued)


NSXT-VI-SDN-019

Set the share value forvSphere Replication traffic toLow (25).

During times of network contention, vSphereReplication traffic is not as important as virtualmachine or storage traffic.

During times of networkcontention, vSphereReplication takes longerand might violate thedefined SLA.

NSXT-VI-SDN-020

Set the share value for vSANtraffic to High (100).

During times of network contention, vSAN trafficneeds a guaranteed bandwidth to support virtualmachine performance.

None.

NSXT-VI-SDN-021

Set the share value formanagement traffic to Normal(50).

By keeping the default setting of Normal,management traffic is prioritized higher thanvSphere vMotion and vSphere Replication butlower than vSAN traffic. Management traffic isimportant because it ensures that the hosts canstill be managed during times of networkcontention.

None.

NSXT-VI-SDN-022

Set the share value for NFStraffic to Low (25).

Because NFS is used for secondary storage,such as backups and vRealize Log Insightarchives, it is not as important as vSAN traffic.By prioritizing it lower, vSAN is not impacted.

During times of networkcontention, backups areslower than usual.

NSXT-VI-SDN-023

Set the share value for backuptraffic to Low (25).

During times of network contention, the primaryfunctions of the SDDC must continue to haveaccess to network resources with priority overbackup traffic.

During times of networkcontention, backups areslower than usual.

NSXT-VI-SDN-024

Set the share value for virtualmachines to High (100).

Virtual machines are the most important asset inthe SDDC. Leaving the default setting of Highensures that they always have access to thenetwork resources they need.

None.

NSXT-VI-SDN-025

Set the share value forvSphere Fault Tolerance toLow (25).

This design does not use vSphere FaultTolerance. Fault tolerance traffic can be set thelowest priority.

None.

NSXT-VI-SDN-026

Set the share value for iSCSItraffic to Low (25).

This design does not use iSCSI. iSCSI trafficcan be set the lowest priority.

None.

Transport Node and Uplink Policy DesignA transport node can participate in an NSX-T overlay or NSX-T VLAN network.

Several types of transport nodes are available in NSX-T.

ESXi Host TransportNodes

ESXi host transport nodes are ESXi hosts prepared and configured forNSX-T. N-VDS provides network services to the virtual machines runningon these ESXi hosts.

Edge Nodes NSX-T Edge nodes are service appliances that run network services thatcannot be distributed to the hypervisors. They are grouped in one orseveral NSX-T Edge clusters. Each cluster represents a pool of capacity.


VMware, Inc. 45

Uplink profiles define policies for the links from ESXi hosts to NSX-T logical switches or from NSX-T Edgevirtual machines to top of rack switches. By using uplink profiles, you can apply consistent configurationof capabilities for network adapters across multiple ESXi hosts or edge virtual machines. Uplink profilesare containers for the properties or capabilities for the network adapters.

Uplink profiles can use either load balance source or failover order teaming. If using load balance source,multiple uplinks can be active. If using failover order, only a single uplink can be active.

Table 3‑26. Design Decisions on Transport Nodes and Uplink Policy


NSXT-VI-SDN-027

Create an uplink profile with theload balance source teamingpolicy with two active uplinks forESXi hosts.

For increased resiliency andperformance, supports the concurrentuse of both physical NICs on the ESXihosts that are configured as transportnodes.

You can use this policy onlywith ESXi hosts. Edge virtualmachines must use the failoverorder teaming policy.

NSXT-VI-SDN-028

Create an uplink profile with thefailover order teaming policywith one active uplink and nostandby uplinks for edge virtualmachine overlay traffic.

Provides a profile that according to therequirements for Edge virtual machines.Edge virtual machines support uplinkprofiles only with a failover order teamingpolicy.

VLAN ID is required in the uplink profile.Hence, you must create an uplink profilefor each VLAN used by the edge virtualmachines.

n You create and managemore uplink profiles.

n The VLAN ID used must bedifferent than the VLAN IDfor ESXi host overlaytraffic.

NSXT-VI-SDN-029

Create two uplink profiles withthe failover order teaming policywith one active uplink and nostandby uplinks for edge virtualmachine uplink traffic.

Enables ECMP because the еdge virtualmachine can uplink to the physicalnetwork over two different VLANs.

You create and manage moreuplink profiles.

NSXT-VI-SDN-030

Add as transport nodes all ESXihosts that are prepared forNSX-T.

Enables the participation of ESXi hostsand the virtual machines on them inNSX-T overlay and VLAN networks.

Hosts run N-VDS. You mustmigrate the existing VMkernelports from the existing virtualswitches to N-VDS. Apply astaged approach where only asingle physical NIC is assignedto the N-VDS instance until allVMkernel ports are migrated toN-VDS.

NSXT-VI-SDN-031

Add as transport nodes all edgevirtual machines.

Enables the participation of edge virtualmachines in the overlay network and thedelivery of services, such as routing, bythese machines.

None.

NSXT-VI-SDN-032

Create an NSX-T Edge clusterwith the default BidirectionalForwarding Detection (BFD)settings containing the edgetransport nodes.

Satisfies the availability requirements bydefault.

Edge clusters are required to createservices such as NAT, routing to physicalnetworks, and load balancing.

None.


VMware, Inc. 46

Routing Design by Using NSX-TThe routing design considers different levels of routing in the environment, such as number and type ofNSX-T routers, dynamic routing protocol, and so on. At each level, you apply a set of principles fordesigning a scalable routing solution.

Routing can be defined in the following directions: North-South and East-West.

n North-South traffic is traffic leaving or entering the NSX-T domain, for example, a virtual machine onan overlay network communicating with an end-user device on the corporate network.

n East-West traffic is traffic that remains in the NSX-T domain, for example, two virtual machines on thesame or different logical switches communicating with each other.

Table 3‑27. Design Decisions on Routing Using NSX-T


NSXT-VI-SDN-033

Create two VLANs to enableECMP between the Tier-0router and the Layer 3 device(ToR or upstream device).

The ToR switches orupstream Layer 3 deviceshave an SVI on one of the twoVLANS and each edge virtualmachine has an interface oneach VLAN.

Supports multiple equal-cost routes onthe Tier-0 Router and provides moreresiliency and better bandwidth use inthe network.

Extra VLANs are required.

NSXT-VI-SDN-034

Deploy an Active-ActiveTier-0 router.

Supports ECMP North-South routingon all edge virtual machines in theNSX-T Edge cluster.

Active-Active Tier-0 routers cannotprovide services such as NAT. If youdeploy a specific solution thatrequires stateful services on theTier-0 router, such as VMwarePivotal Container Service, you mustprovide an additional Tier-0 router inActive-Standby mode.

NSXT-VI-SDN-035

Use BGP as the dynamicrouting protocol.

Enables the dynamic routing by usingNSX-T. NSX-T supports only BGP .

In environments where BGP cannotbe used, you must configure andmanage static routes.

NSXT-VI-SDN-036

Configure BGP Keep AliveTimer to 4 and Hold DownTimer to 12 between the ToRswitches and the Tier-0router.

Provides a balance between failuredetection between the ToR switchesand the Tier-0 router andoverburdening the ToRs with keepalive traffic.

By using longer timers to detect if arouter is not responding, the dataabout such a router remains in therouting table longer. As a result, theactive router continues to sendtraffic to a router that is down.


VMware, Inc. 47

Table 3‑27. Design Decisions on Routing Using NSX-T (Continued)


NSXT-VI-SDN-037

Do not enable GracefulRestart between BGPneighbors.

Avoids loss of traffic. Graceful Restartmaintains the forwarding table which inturn will forward packets to a downneighbor even after the BGP timershave expired causing loss of traffic.

None.

NSXT-VI-SDN-038

Deploy a Tier-1 router to theNSX-T Edge cluster andconnect it to the Tier-0 router.

Creates a two-tier routing architecturethat supports load balancers and NAT.

Because the Tier-1 is alwaysActive/Standby, creation of servicessuch as load balancers or NAT ispossible.

A Tier-1 router can only beconnected to a single Tier-0 router.

In scenarios where multiple Tier-0routers are required, you mustcreate multiple Tier-1 routers.

Virtual Network Design Example Using NSX-TDesign a setup of virtual networks where you determine the connection of virtual machines to logicalswitches and the routing between the Tier-1 router and Tier-0 router, and then between the Tier-0 routerand the physical network.

Figure 3‑8. Virtual Network Example

Physical Layer 3 Devices

Tier-1 to Tier-0 Connection

Legend:

Logical Switch 1 - 192.168.100.0/24

VLANs

Logical Switch

Tier-0RouterActive/Active

ECMP

NSX-TEdge Cluster

Edge VM 1 Edge VM 2

Tier-1Router

Linux Windows Linux

Logical Switch 2 - 192.168.200.0/24

Linux Windows Linux


VMware, Inc. 48

Monitoring NSX-TMonitor the operation of NSX-T for identifying failures in the network setup. Use the vRealize Log InsightContent Pack for VMware NSX-T to see the logs generated by the components of NSX-T in the userinterface of vRealize Log Insight.

vRealize Log Insight saves log queries and alerts, and you can use dashboards for efficient monitoring.

Table 3‑28. Design Decisions on Monitoring NSX-T


NSXT-VI-SDN-039 Install the content pack for NSX-T. Provides granular monitoring of theNSX-T infrastructure.

Requires manually installingthe content pack.

NSXT-VI-SDN-040 Configure each NSX-Tcomponent to send loginformation over syslog to thevRealize Log Insight cluster VIP.

Ensures that all NSX-T componentslog files are available for monitoringand troubleshooting in vRealize LogInsight.

Requires manuallyconfiguring syslog on eachNSX-T component.

Use of SSL Certificates in NSX-TBy default, NSX-T Manager uses a self-signed Secure Sockets Layer (SSL) certificate. This certificate isnot trusted by end-user devices or Web browsers.

As a best practice, replace self-signed certificates with certificates that are signed by a third-party orenterprise Certificate Authority (CA).

Table 3‑29. Design Decisions on the SSL Certificate of NSX-T Manager

Design ID Design Decision Design Justification Design Implication

NSXT-VI-SDN-041 Replace the NSX-T Managercertificate with a certificate that issigned by a third-party Public KeyInfrastructure.

Ensures that the communicationbetween NSX-T administrators andthe NSX-T Manager is encrypted byusing a trusted certificate.

Replacing and managingcertificates is anoperational overhead.


VMware, Inc. 49

architecture and design of vmware nsx-t for …...n virtualization network design n nsx-t design...

Documents