hypervisors networking: best practices for interconnecting...

Hypervisors networking: best practices for interconnecting with Cisco switches BRKVIR-2019

Ramses Smeyers – Customer Support Engineer

Agenda

© 2013 Cisco and/or its affiliates. All rights reserved. BRKVIR-2019 Cisco Public

What is this session about ?

Networking virtualization concepts

Hypervisor overview

– VMware vSphere ESXi 5.1

– Microsoft HyperV 3.0

– XenServer 6.1

4

Networking virtualization concepts


Hypervisor networking virtualization

VM-to-VM and VM to physical host traffic handled via software switch that lives in the hypervisor

VM-to-VM:

memory transfer

VM-to-physical:

physical adapter

6


Hypervisor networking virtualization – vSphere ESXi example

VMs don’t directly control networking hardware – x86 hw designed to be handled by only one device driver!

When a VM communicates with the outside world, it: – … passes the packet to its local device driver …

– … which in turns hands it to the hypervisor…

– … which in turns passes it to the physical NIC

ESX gives VMs several device driver options: – Strict emulation of Intel’s e1000 / e1000e

– VMware vmxnet/vmxnet3: paravirtualized!

VMs have MAC addresses that appear on the wire

7


Gratuitous ARP

8


Gratuitous ARP

9


Spanning-tree

Enable or disable ?

– No full switch traffic from 1 incoming uplink port will not be duplicated to a different uplink port spanning-tree port type edge [ trunk ]

spanning-tree portfast

With spanning-tree (vSwitch with 2 uplink ports)

– link-failure/fail back 30s block on port network disconnections

10


Switch independent / dependent

Independent no specific switch protocol / configuration is required

Dependent specific switch configuration is required (port-channel)

11

Hypervisor overview VMware vSphere ESXi

12


vNetwork standard switch

Nic: Virtual

machine NIC

vmNic: Physical

server NIC

Portgroup != VLAN

vmNic {0,1} vSwitch

13


vNetwork distributed switch

• Multiple distributed

switches can exist on 1

server

• Enterprise plus license is

required

14


VMware vSphere ESXi

Route based on originating virtual port

Route based on IP hash

Route based on IP hash + LACP (vDS)

Route based on source MAC hash

Route based on physical NIC load (vDS)

Use explicit failover order

15


Uplink connectivity possibility's

16

vSS vDS

vDS (extra):

• Route based on physical NIC load

• Route based on IP hash + LACP


Load balancing

17

Per portgroup:






Per DVUplink:



Load balancing: vSS

18

Override vSwitch0 properties


Load balancing: vDS

19

Not configurable on the

DVUplinks


Active / Standby / Unused

20

vmnic0

Ethernet1/17 is up

vminc1

Ethernet1/17 is up

vmnic2

GigabitEthernet1/0/5 is up, line

protocol is up (connected)



Default configuration

Even distribution of traffic if the number of vEth adapters > physical adapters

1 vEth can only be on 1 physical interface

Can go to 1 or multiple switches

21

VSS +

VDS


Route based on originating virtual port – Switch configuration

interface GigabitEthernet1/0/5

switchport access vlan 49

switchport mode access






22


switchport trunk encapsulation dot1q

switchport trunk allowed vlan 49

switchport mode trunk



switchport trunk encapsulation dot1q

switchport trunk allowed vlan 49

switchport mode trunk


NXOS: spanning-tree port type

edge trunk


Route based on originating virtual port - ESXi

23



Uplink based on a hash of source and destination IP address

Evenness of traffic distribution depends on the number of TCP/IP sessions and their characteristics

Physical switch sees the client MAC address on multiple ports.

Use of 802.3ad is required mode on

Needs to connect to 1 port-channel

24

VSS +

VDS

“Route based on IP

hash” does not work

with Cisco UCS-B

Fabric Interconnects

cannot do VPC


Route based on IP hash – no port-channel

25


Route based on IP hash – Switch configuration





channel-group 1 mode on






interface Port-channel1




26

bdsol-3750-05(config-if)#channel-group 1 mode ?

active Enable LACP unconditionally

auto Enable PAgP only if a PAgP device is detected

desirable Enable PAgP unconditionally

on Enable Etherchannel only

passive Enable LACP only if a LACP device is detected

bdsol-3750-05(config-if)#channel-group 1 mode


Route based on IP hash – ESXi

27



Uplink based on a hash of source and destination IP address

Evenness of traffic distribution depends on the number of TCP/IP sessions and their characteristics

Physical switch sees the client MAC address on multiple ports.

Use of 802.3ad is required active or passive


28

VDS


Route based on IP hash + LACP (vDS) - ESXi

29

Active – The port is in an active negotiating state, in which the

port initiates negotiations with remote ports by sending LACP

packets.

Passive – The port is in a passive negotiating state, in which

the port responds to LACP packets it receives but does not

initiate LACP negotiation.


Route based on IP hash + LACP (vDS) - ESXi

30

Extra LACP_MgmtPort


Route based on IP hash + LACP (vDS) – Switch configuration









channel-group 1 mode active






31

channel-group 1 mode passive


ESXi

Switch

Active Passive

Active Yes Yes

Passive Yes No



Uplink chosen based on a hash of the source MAC address (VM)

1 vEth 1 physical adapter

Even distribution of traffic if random MAC addresses

Be careful with manual MAC address assignments


32

VSS +

VDS


Route based on source MAC hash - ESXi

33


Route based on source MAC hash – Switch configuration









34



Initial placement based on “Route based on originating virtual port”

If RX or TX > 75% rebalance

VMkernel check runs every 30 seconds

35

VDS



Initial placement based on “Route based on originating virtual port”

If RX or TX > 75% rebalance

VMkernel check runs every 30 seconds

MAC moves on switch layer


36

VDS



37











38

No port-channel as

asymmetric return traffic

will be dropped



Highest order uplink from the list of active adapters


39

VSS +

VDS


Use explicit failover order - ESXi

40

All MAC’s on 1 port


Use explicit failover order – Switch configuration









41


VMware conclusion

vSS

– Switch independent Route based on originating virtual port

– Switch dependent Route based on IP hash

vDS

– Switch independent Route based on physical NIC load

– Switch dependent Route based on IP hash + LACP

42

Hypervisor overview Microsoft Hyper-V 3.0 Windows 2012

43


Uplink connectivity possibility's

Single physical NIC or Teaming

44


Enable NIC Teaming

Pre Windows 2012 Teaming provided by vendor device drivers

Windows 2012 Teaming provided native by Windows

45


Microsoft Network Adapter Multiplexor Driver

46


Uplink connectivity details

All Address hash modes Hyper-V Port mode

Switch Independent Scenario 1 Scenario 2

Switch Dependent

(Static and LACP)

Scenario 3 and 4 Scenario 5 and 6

47

Address Hash comes in 3 flavors:

• 4-tuple hashing (Source and destination TCP ports, usually with IP

addresses, this is the default hashing mode)

• 2-tuple: If the ports aren’t available (encrypted traffic such as IPsec)

then it’ll go to 2-tuple where it uses Source and destination IP

address, with or without MAC addresses

• MAC address hash: If not IP traffic, then MAC addresses are

hashed


Static / LACP / Independent

Static teaming 802.3ad draft v1

LACP 802.1ax LACP, also referred to as 802.3ad

Independent No switch configuration required

48


Standby adapter

49

bdsol-3750-05#show int GigabitEthernet1/0/3

GigabitEthernet1/0/3 is up, line protocol is up (connected)

bdsol-3750-05#show int GigabitEthernet1/0/4

GigabitEthernet1/0/4 is up, line protocol is up (connected)

bdsol-3750-05#


Single physical NIC

Normal server connection

Possibility of trunk

Multiple MAC’s on 1 switch port

50


Switch Independent - All Address hash modes

Outbound traffic is spread across all active members


Observation: traffic sticks to 1 interface

51


Switch Independent - All Address hash modes

52

Observation: All traffic sticks to 1

interface


Switch Independent - All Address hash modes – Switch configuration









53

For Your Reference


Switch Independent - Hyper-V Port mode

Outbound traffic is tagged with the port on the Hyper-V switch

All traffic with that port tag is sent on the same team member

1 MAC address will be fixed to 1 physical switch port


54


Switch Independent - Hyper-V Port mode

55


Switch Independent - Hyper-V Port mode – Switch configuration









56

For Your Reference


Switch Dependent - All Address hash modes

Outbound traffic is spread across all active members

Inbound traffic will be distributed by the switch’s load distribution algorithm


57


Switch Dependent - All Address hash modes

58


Switch Dependent - All Address hash modes – Switch configuration














Static Teaming

59









Switch Dependent - All Address hash modes – Switch configuration














LACP

60









Switch Dependent - Hyper-V Port mode

Outbound traffic is tagged with the port on the Hyper-V switch

All traffic with that port tag is sent on the same team member

1 MAC address will be fixed to 1 physical switch port



61


Switch Dependent - Hyper-V Port mode

62


Switch Dependent - Hyper-V Port mode – Switch configuration














Static Teaming

63









Switch Dependent - Hyper-V Port mode – Switch configuration














LACP

64









Hyper-V conclusion

Switch independent

– Switch Independent - Hyper-V Port mode

Switch dependent

– Switch Dependent - Hyper-V Port mode

65

Hypervisor overview XenServer 6.1

66


Uplink connectivity possibilities

67

Single physical NIC or Bonding

LACP as of XenServer 6.1


Single physical NIC

Normal server connection

Possibility of trunk

Multiple MAC’s on 1 switch port

68


Active-active

balance-slb Server Load Balancing

1 VM is fixed to 1 physical adapter

Every 10 second traffic is evaluated and balanced over all physical adapters

MAC move


69


Active-active

70


Active-active









71

For Your Reference


Active-passive

1 NIC is active, other NIC’s are passive waiting for adapter failure

Can goto 1 or multiple switches

72


Active-passive

73


Active-passive









74

For Your Reference


LACP with load balancing based on IP and port of source and destination

Outgoing traffic will be hashed over multiple uplinks based on src/dest IP and port



75



76
















77









LACP with load balancing based on source MAC address

1 VIF will use 1 outgoing physical interface (1 MAC per link)

1-1 mapping between VIF (virtual interface) and physical interface/switch port


78



79
















80









XenServer conclusion

Switch independent

– Active-Active

Switch dependent

– LACP with load balancing based on source MAC address

81

Nexus 1000V

82


Nexus 1000V

Virtual Ethernet Module(VEM)

Replaces Vmware’s virtual switch

Enables advanced switching capability on the hypervisor

Provides each VM with dedicated “switch ports”

Virtual Supervisor Module(VSM)

CLI interface into the Nexus 1000V

Leverages NX-OS

Controls multiple VEMs as a single network device

83


Load balance possibilities

84


Nexus 1000V

Multi hypervisor switch (VMware, Hyper-V* and KVM*)

Layer 2 switching: VLANs, private VLANs, VXLAN, loop prevention, multicast, virtual PortChannels, LACP, ACLs

Network management: SPAN, ERSPAN, Netflow 9, vTracker, vCenter Server Plug-in

Enhanced QOS features

Cisco vPath

Security: DHCP Snooping, IP Source Guard, Dynamic ARP inspection, Cisco TrustSec SGA support

Cisco Virtual Security Gateway

Other virtual services (Cisco ASA 1000V, Cisco vWAAS, etc..)

85


Nexus 1000V Sessions

86

LTRVIR-2005 Deploying the Nexus 1000V on ESXi and Hyper-V

BRKVIR-2017 The Nexus 1000V on Microsoft Hyper-V: Expanding the Virtual Edge

BRKVIR-3013 Deploying and Troubleshooting the Nexus 1000v virtual switch

BRKVIR-2023 Cisco Nexus 1000V InterCloud based Hybrid Cloud Architectures and Approaches


Key take-aways

Understand the hypervisor’s load-balancing mechanisms

Align the configuration on hypervisor and networking layer

All adapters are always up

MAC moves are possible depending on the load balancing algorithm

Use the correct port-channel configuration (on/active/passive)

87


Maximize your Cisco Live experience with your

free Cisco Live 365 account. Download session

PDFs, view sessions on-demand and participate in

live activities throughout the year. Click the Enter

Cisco Live 365 button in your Cisco Live portal to

log in.

Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Cisco Daily Challenge points for each session evaluation you complete.

Complete your session evaluation online now through either the mobile app or internet kiosk stations.

88

hypervisors networking: best practices for interconnecting...

Documents