Network Tuning for Specific WorkloadsGabriel Silva Don Stanwyck
DCIM-B344
MICROSOFT CONFIDENTIAL – INTERNAL ONLY
Terminology and Basic Architecture
NIC Features for Networking
Agenda
Let’s first cover some terminology• Processor. One physical processor, which can consist of one or more nodes. A physical processor is the same as a package, a socket, or a CPU.
• Non-uniform memory architecture (NUMA) node. A set of logical processors and cache that are close to one another.
• Core. One processing unit, which can consist of one or more logical processors.
• Logical processor (LP). One logical computing engine from the perspective of the host operating system, application, or driver. In effect, an LP is a thread.
• Virtual Processor (VP). One logical computing engine from the perspective of a virtual operating system. VPs are specific to a virtual machine and are not 1:1 to LPs on the physical system
• Affinity. A preference indicated by a thread, process, or interrupt for operation on a particular processor, node, or group.
Processor or Socket
NUMA Node
Core
LP LP
Processor or SocketNUMA Node
Core
LP LP
Processor or SocketNUMA Node
Core
LP LP
Processor or SocketNUMA Node
Core
LP LP
Processor or SocketNUMA Node
Core
LP LP
Parent Partition
Basic Architecture
Hyper-V Extensible Switch
ExtensionsRouting Filtering
ACLs
Network Adapter
Virtual machine
VM BUS
LP0 LP1 LP2 LP3 VP0 VP1 VP2 VP3
VM NIC
NIC Performance Features for Networking
Problem• An Enterprise Web Server has a large volume of incoming packets
spanning a large number of incoming connections. A small set of logical processors run at 100% utilization. An IT Admin wants to scale the workload across many logical processors and get more parallelization in servicing requests
• Examples: Web Server and File Server
Receive Side Scaling (RSS)•Why do I need RSS?•Network traffic is no longer limited to one core•RSS distributes receive and send networking traffic to multiple processors for the host
•When should I enable RSS?•Enabled by default on Windows Server machines without a vSwitch created. •Useful when processing large amounts of networking traffic
Physical Machine
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
RSS Queue1
RSS Queue2
RSS Queue3
RSS Queue4
RSS Logic
Network Packet
RSS Step by Step
1. NIC performs the spreading of the network traffic by TCP/UDP flows
2. Flows are then placed into RSS Queues on the NIC
3. Networking traffic inside the RSS Queue is then indicated to the affinitized processor in the host
4. RSS CPUs are configured by the admininistrator
RSS Configuration• PowerShell Configuration• Get-NetAdapterRss• Set-NetAdapterRss• -Profile, -BaseProcessorNumber, -MaxProcessorNumber, -MaxProcessors
• Enable-NetAdapterRss• Disable-NetAdapterRss
• Profiles• Closest (Like WS08)• Closest Static • NUMA Static (Default)• NUMA Dynamic• Conservative
DynamicStatic
NUMA aware
NonNUMA
Closest Static
NUMA Dynamic
NUMA Static(default)
Closest(like WS08 R2)
Demo
RSS
Problem• An IT Pro is unable to run more VMs on a deployment server
because they realize that the incoming packet processing is saturating a limited set of logical processors on the physical host
• Examples: Private Cloud Deployments
Virtual Machine Queue (VMQ)•Why do I need VMQ?•RSS spreads traffic for one physical host and by TCP/UDP flows•Multiple guests on a system breaks that model•VMQ spreads traffic per vNIC•Each VMQ can use at most one logical CPU in the host
•when When should I enable VMQ?•Enabled by default on Windows Server machines with a vSwitch created. •Useful hosting many VMs on the same host
Parent Partition
Virtual machine
Network stack
VM NIC
Hyper-V Extensible Switch
Virtual machine
Network stack
VM NIC
VM BUS
VM Queue
VM Queue
DefaultQueue
Routing Filtering
Port1 Port2
VP0 VP1 VP2 VP3 VP4 VP5
MAC Address Filter
VMQ
VMQ Step-by-Step1. NIC performs the spreading of the network traffic by MAC address2. Flows are then placed into the appropriate VMQ on the NIC3. Networking traffic inside the VMQ is then indicated to the affinitized processor in the
host• VMQ indication saves CPU by bypassing the routing and filtering in the vSwitch
4. Once vSwitch processing is completed, vSwitch passes the traffic to the correct VM5. VMQ CPUs are configured by the admininistrator
VMQ Configuration• PowerShell Configuration• Get-NetAdapterVmq• Set-NetAdapterVmq• -BaseProcessorNumber, -MaxProcessorNumber, -MaxProcessors
• Enable-NetAdapterVmq• Disable-NetAdapterVmq• Get-NetAdapterVmqQueue
Demo
VMQ
Problem• An IT Pro is hosting an Enterprise Web Server in a VM for high
availability and to leverage existing hardware. Like the server on a physical machine, this server has a large volume of incoming packets spanning a large number of incoming connections. The Pro notices that the workload used to scale natively but in a VM the processing is bound to 1 VP and a single LP in the root. The Pro wants to scale the workload across many logical and virtual processors to get more parallelization in servicing requests
• Examples: File Servers, Web Server, Gateways
Virtual Receive Side Scaling (vRSS)•Why do I need vRSS?•Remember this from VMQ: Each VMQ can use at most one logical CPU in the host•One guest with high networking traffic is capped at a single CPU for network processing• Example: File Server, SQL Server•vRSS enables RSS inside of a VM and spreading inside the host for vSwitch processing
•When should I enable vRSS?•Disabled by default on Windows Server 2012 R2 machines •Useful when hosting one or two large VMs with high networking traffic
vRSSParent Partition Virtual machine
Hyper-V Extensible Switch
Virtual machine
VM BUS
VM Queue
VM Queue
DefaultQueue
Port1 Port2
Extensions
ACLs
VP0 VP1 VP2 VP3 VP0 VP1 VP2 VP3
VM NIC VM NICIndirection Table Indirection Table
LP0 LP1 LP2 LP3 LP4 LP5
Indirection Table
Toeplitz HashRouting Filtering
MAC Address Filter
vRSS Step-by-Step1. NIC performs the spreading of the network traffic by MAC address2. Flows are then placed into the appropriate VMQ on the NIC3. Networking traffic inside the VMQ is then indicated to the affinitized processor in the
host• VMQ indication saves CPU by bypassing the routing and filtering in the vSwitch
4. NIC applies RSS logic for the flow based on the VMQ it is indicated on to spread traffic further for vSwitch processing, if needed
5. Once vSwitch processing is completed, vSwitch passes the traffic to the correct VM6. The VM applies another level of RSS logic to now spread networking traffic inside the
VM7. VMQ CPUs are configured by the administrator in the host and RSS CPUs on the
guest side are configured by the VMs administrator
vRSS Configuration• PowerShell Configuration (Host Side)• VMQ PowerShell Cmdlets
• PowerShell Configuration (VM)• RSS PowerShell Cmdlets
Demo
vRSS
IT Pro’s Problem• An IT Pro has a VM that hosts applications that will receive data
from the network, make some decisions based on the data, and then send a response in her private cloud. The workload is very latency sensitive and the IT Pro needs to minimize the round-trip time to the microseconds from the time they receive the data and the time they send a response back out
• Examples: Financial Applications
Single Root I/O Virtulization (SR-IOV)•Why do I need SR-IOV?•SR-IOV distributes networking traffic by VM and delivers directly to the VM from the NIC•*Some adapters also have RSS for SR-IOV from the NIC
•When should I enable SR-IOV?•Disabled by default on Windows Server machines •Useful when hosting VMs with low latency requirements in a trusted environment
SR-IOV
Parent Partition Virtual machine
Network stack
VM NICVirtual
function (VF)
Hyper-V Extensible Switch
ExtensionsRouting Filtering
ACLs
SR-IOV network adapter VF
Virtual machine
Network stack
VM NIC
VM BUS
SR-IOV Step-by-Step1. NIC performs the spreading of the network traffic by MAC address 2. Flows are then placed into the appropriate VF on the NIC3. Networking traffic inside the VF is then indicated to the correct VM• SR-IOV saves CPU and decreases latency by bypassing the host and vSwitch altogether
SR-IOV Configuration• Hyper-V Extensible Switch Configuration• Get-NetAdapterSriov
• PowerShell Configuration• Get-NetAdapterSriov• Set-NetAdapterSriov• -NumVFs
• Enable-NetAdapterSriov• Disable-NetAdapterSriov• Set-NetAdapterAdvancedProperty
• RSS provides high scalability of network traffic on a single host
• VMQ distributes traffic equally amongst multiple guests on a single host with a vSwitch (1 core per vmNIC/vNIC)
• vRSS provides high scalability of network traffic on a single host to a few large VMs
• SRIOV is a low latency path for network traffic that bypasses the host and vSwitch in secure environments
NIC Offloads Key Takeaways
RSS VMQHost (No vSwitch bound to this NIC)
vSwitch (Bound to the NIC)
VM (vRSS)
Inside the VM Inside the host for vSwitch processing
NIC Offloads Key Takeaways, cont…
Parent Partition Virtual machine
Hyper-V Extensible Switch
Virtual machine
VM BUS
VMQVMQ
Port1 Port2
Extensions
ACLs
VP0 VP1 VP2 VP3 VP0 VP1 VP2 VP3
VM NIC VM NICIndirection Table
LP0 LP1 LP2 LP3 LP4 LP5
RSS
NIC Offloads altogether
NIC Teaming in WS 2012
• All Static Distribution• Different hashes (address elements, switch ports)
• Hashes were static until failure or reboot
• No explicit attempt to balance loads, just to distribute them
WS 2012 R2: New Dynamic mode
• Continuously monitors distribution of traffic
• Actively adjusts traffic based on observed load
• Will move TCP streams from team member to team member during breaks in the stream
NIC TEAM
How it works in WS 2012…
tNIC
NIC NICNIC
BACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDTCP/IP Native Stack
Each arrow represents a flowlet. In WS2012each flowlet always follows the same path theprevious flowlet from that flow did because flowlets aren’t detected and rebalancing isn’tperformed.
NIC TEAM
How it works in WS 2012 R2…
tNIC
NIC NICNIC
BACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACDBACD
Each arrow represents a flowlet. In WS2012 R2each flowlet is independently routed to the leastused NIC in the team. With MAC address rewritethe adjacent switches are unaware that flows aremoving around.
ABCDTCP/IP Native Stack
VMQ Deep Dive blog serieshttp://blogs.technet.com/b/networking/archive/2013/09/10/vmq-deep-dive-1-of-3.aspxhttp://blogs.technet.com/b/networking/archive/2013/09/24/vmq-deep-dive-2-of-3.aspxhttp://blogs.technet.com/b/networking/archive/2013/10/22/vmq-deep-dive-3-of-3.aspx
vRSS and Networking interviewhttp://channel9.msdn.com/Shows/Edge/Edge-Show-89-vRSS-and-Networking-With-Gabriel-Silva
Additional Resources
Come Visit Us in the Microsoft Solutions Experience!
Look for Datacenter and Infrastructure ManagementTechExpo Level 1 Hall CD
For More InformationWindows Server 2012 R2http://technet.microsoft.com/en-US/evalcenter/dn205286
Windows Server
Microsoft Azure
Microsoft Azurehttp://azure.microsoft.com/en-us/
System Center
System Center 2012 R2http://technet.microsoft.com/en-US/evalcenter/dn205295
Azure PackAzure Packhttp://www.microsoft.com/en-us/server-cloud/products/windows-azure-pack
Resources
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
msdn
Resources for Developers
http://microsoft.com/msdn
TechNet
Resources for IT Professionals
http://microsoft.com/technet
Sessions on Demand
http://channel9.msdn.com/Events/TechEd
Complete an evaluation and enter to win!
Evaluate this session
Scan this QR code to evaluate this session.
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.