vmware 10ge qos design deep dive with cisco ucs &...
TRANSCRIPT
VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus
10GE Networking
BRAD HEDLUND .com
Brad Hedlund, CCIE #5530 Data Center Architect Cisco Systems [email protected]
v 2.5
- About the Author -
Brad Hedlund Technical Solutions Architect, Data Center Cisco Systems, Inc. CCIE #5530
Blog: http://bradhedlund.com Twitter: http://twitter.com/bradhedlund E-mail: [email protected] Comments welcome.
BRAD HEDLUND .com
http://bradhedlund.com/about/
Cisco UCS Networking Overview
BRAD HEDLUND .com
6100 Fabric A
6100 Fabric B
B200 B250
CNA
FEX
A CNA CNA
FEX
B
FEX
A
FEX
B
SAN A SAN B ETH 1 ETH 2
MGMT MGMT
Chassis 1 Chassis 2
Emulex/Qlogic CNA
10GE
vNIC vNIC
Fabric A
Fabric B
vmnic0
10GE
vHBA vHBA
vhba0 vmnic1 vhba1
CNA
vSwitch / Nexus 1000V
FEX FEX
BRAD HEDLUND .com
Emulex/Qlogic CNA – Logical View
CNA
vSwitch / Nexus 1000V
FEX FEX
BRAD HEDLUND .com
vmnic0 vhba0 vmnic1 vhba1
vfc 1
vEth1
vEth2
vfc 2
Fabric Interconnect A Fabric Interconnect B
Service Profile / Fabric Selection
BRAD HEDLUND .com
Cisco VIC “Palo”
vNIC
Fabric A
Fabric B
vhba0
Cisco VIC
vHBA vNIC vNIC vNIC vNIC vNIC vNIC vNIC vHBA 58
vhba1
10GE 10GE
vSwitch / vDS / Nexus 1000V
vmnic0
vmnic1
vmnic2
vmnic3
vmnic4
vmnic5
vmnic6
vmnic7
FEX FEX
BRAD HEDLUND .com
Cisco VIC QoS: Example
vNIC 2
Fabric A
Fabric B
Cisco VIC
vHBA 1
vNIC 1
vNIC 3
vNIC 4
vNIC 5
vNIC 6
vNIC 8
vNIC 7
vHBA 2
58
FEX FEX
BRAD HEDLUND .com
0 2 4 5 6
7 7 10GE
0 1 2 4 5 6
10GE
40% 40% 10%
Strict Priority
10% 40% 40% 10% 10%
Strict Priority
1 3
RR RR
VIC Mgmt
VIC Mgmt
Bandwidth Scheduler Bandwidth Scheduler
COS 1 COS 3 COS 4 COS 5 COS 1 COS 3 COS 4 COS 5
Limit TX 1GE Limit TX 1GE
TX TX
No Drop
(8) COS Queues
No Drop
3
Cisco VIC – Logical View
vhba0
Cisco VIC
vhba1
vSwitch / Nexus 1000V
vmnic0
vmnic1
vmnic2
vmnic3
vmnic4
vmnic5
vmnic6
vmnic7
vfc 1
vEth1
vEth2
vEth3
vEth4
vfc 2
vEth6
vEth7
vEth8
vEth5
FEX A FEX B
Fabric Interconnect A Fabric Interconnect B
BRAD HEDLUND .com
I/O Consolidation with 10GE
BRAD HEDLUND .com
vmnic0
vmnic1
vmnic2
vmnic3
10GE Network Switch
VM Data
VMotion
MGMT
IP Storage
CONCERN: Bandwidth Availability
SOLUTIONS: 1. MAX bw per NIC
Rate Limits 2. MIN bw per NIC
QoS 3. Do Nothing
Let it rip
I/O Consolidation with 10GE
BRAD HEDLUND .com
vmnic1
10GE Network Switch
VM Data
VMotion
MGMT
IP Storage
CONCERN: Bandwidth Availability
SOLUTIONS: 1. MAX bw per Class
Rate Limits 2. MIN bw per Class
QoS 3. Do Nothing
Let it rip
MAX Bandwidth – Rate Limits
BRAD HEDLUND .com
vmnic0
vmnic1
vmnic2
vmnic3
10GE
Network Switch
VM Data
VMotion
MGMT
IP Storage
2.5G
2.5G
2.5G
2.5G
Sum of Limits must not exceed physical link speed Inefficient use of unused BW
This slide animated @ http://bradhedlund.com/2010/08/16/cisco-ucs-qos-vs-hp-virtual-connect-rate-limiting/
MIN Bandwidth – QoS
BRAD HEDLUND .com
vmnic0
vmnic1
vmnic2
vmnic3
10GE
Network Switch
VM Data
VMotion
MGMT
IP Storage
Minimum BW guarantee with no Maximum Efficient: unused BW available to other NICs
This slide animated @ http://bradhedlund.com/2010/08/16/cisco-ucs-qos-vs-hp-virtual-connect-rate-limiting/
MIN Bandwidth – QoS
BRAD HEDLUND .com
10GE
Network Switch
VM Data
VMotion
MGMT
IP Storage
vmnic1
Nexus 1000V Classifies Traffic
UCS 6100 Nexus 5000 Enforces Policy
Cisco VIC Enforces Policy
Traffic Considerations
Required
• Guest VM Data
• Management – VMKernel
– Service Console
• vMotion
• Central Storage Access
Optional
• Fault Tolerance (FT)
• NFS
• iSCSI
• FCoE
• Realtime – Video/Voice, Market Data
BRAD HEDLUND .com
Cisco UCS + VIC Design (1): Required Traffic
vhba0
vhba1
vSwitch or vDS
vmnic0
vmnic2
vmnic4
vmnic1
vmnic3
vmnic5
vfc 1
vEth1
vEth2
vEth3
vfc 2
vEth6
vEth7
vEth8
Fabric Interconnect A Fabric Interconnect B
VM DATA vMotion MGMT FC
Primary vmnic1 vmnic2 vmnic5 MP
Backup vmnic0 vmnic3 vmnic4 MP
Min BW 10% 1GE Max 40% 10% 40%
Class Bronze COS 1 Best Effort COS 0 Silver COS 2 COS 3 BRAD HEDLUND .com
UCS Manager QoS System Class Design #1
BRAD HEDLUND .com
• Assign each Class of Service a minimum bandwidth weighting • System wide policy
Apply QoS Policy to vNIC
BRAD HEDLUND .com
• Create QoS Policy • Assign Policy to vNIC
Cisco UCS + VIC Design (2): Req + 1 Opt
vhba0
vhba1
vSwitch or vDS
vmnic0
vmnic2
vmnic4
vmnic6
vmnic1
vmnic3
vmnic5
vmnic7
vfc 1
vEth1
vEth2
vEth3
vEth4
vfc 2
vEth6
vEth7
vEth8
vEth5
Fabric Interconnect A Fabric Interconnect B
FC NFS VM DATA vMotion MGMT
Primary vmnic1 vmnic2 vmnic5 vmnic7 MP
Backup vmnic0 vmnic3 vmnic4 vmnic6 MP
Min BW 10% 1GE Max 30% 10% 10% 40%
Class Bronze COS 1 Best Effort COS 0 Silver COS 2 Gold COS 4 COS 3 BRAD HEDLUND .com
UCS Manager QoS System Class Design #2
BRAD HEDLUND .com
• Assign each Class of Service a minimum bandwidth weighting • System wide policy
Cisco UCS + VIC Design (3): Req + 2 Opt + Realtime
vhba0
vhba1
vSwitch or vDS
vmnic2
vmnic4
vmnic6
vmnic8
vmnic1
vmnic3
vmnic5
vmnic7
vfc 1
vEth2
vEth3
vEth4
vEth5
vfc 2
vEth7
vEth8
vEth9
vEth6
Fabric Interconnect A Fabric Interconnect B
VM DATA vMotion FT MGMT NFS
vmnic9
vmnic0
vEth10
vEth1
Primary vmnic0 vmnic1 vmnic2 vmnic5 vmnic6 vmnic9 MP
Backup vmnic1 vmnic0 vmnic3 vmnic4 vmnic7 vmnic8 MP
Min BW 1GE Max 10% 10% 20% 10% 10% 40%
Class Bronze COS 1 Bronze COS 1 Plat COS 5 Gold COS 4 Best Effort COS 0 Silver COS 2 COS 3
FC
BRAD HEDLUND .com
Realtime
UCS Manager QoS System Class Design #3
BRAD HEDLUND .com
• “Platinum” Class for Realtime traffic • Lossless Ethernet and Multicast optimized • “Silver” and “Bronze” for vMotion & NFS with Jumbo Frames
Design #4: (2) 10GE NICs with Nexus 1000V Cisco UCS or Nexus 5000
vhba0
vhba1
vmnic1
vmnic0
vfc 1
vEth4
vfc 2
vEth7
Fabric Interconnect A / Nexus 5000 #1 Fabric Interconnect B / Nexus 5000 #2
NEXUS 1000V QoS Classification
Primary vmnic0 vmnic1 vmnic0 vmnic1 vmnic0 vmnic1 MP
Backup vmnic1 vmnic0 vmnic1 vmnic0 vmnic1 vmnic0 MP
Min BW 1GE Max 10% 10% 20% 10% 10% 40%
Class Bronze COS 1 Bronze COS 1 Plat COS 5 Gold COS 4 Best Effort COS 0 Silver COS 2 COS 3
VM DATA vMotion FT MGMT NFS FC Realtime
QoS Enforcement (if Cisco VIC)
QoS Enforcement
Cisco VIC or Other adapter Cisco UCS server or non-Cisco
BRAD HEDLUND .com
QoS Enforcement (if N1K v1.4)
vNIC QoS Policy for Design #4
BRAD HEDLUND .com
• Preserve and use COS markings from Nexus 1000V • Same QoS System Class settings as Design #3
Design #5: (2) 10GE NICs, no Cisco VIC, no N1K Nexus 5000
vhba0
vhba1
vmnic1
vmnic0
vfc 1
Eth 1/1
vfc 2
Eth 1/1
Nexus 5000 #1 Nexus 5000 #2
vSwitch or vDS
QoS Classification
QoS Enforcement
Primary vmnic0 vmnic1 vmnic0 vmnic1 vmnic0 vmnic1 MP
Backup vmnic1 vmnic0 vmnic1 vmnic0 vmnic1 vmnic0 MP
Min BW 1GE Max 10% 10% 20% 10% 10% 40%
Class Bronze COS 1 Bronze COS 1 Plat COS 5 Gold COS 4 Best Effort COS 0 Silver COS 2 COS 3
VM DATA vMotion FT MGMT NFS FC Realtime
Cisco VIC or Other adapter Cisco C-Series or non-Cisco
BRAD HEDLUND .com
VMware NetIOC ? FCoE?
Narrative posted at:
BRAD HEDLUND .com
http://bradhedlund.com/?p=1650