vmware network setting requirements

1

DOCUMENT TYPE

VMWARE NETWORK SETTING REQUIREMENTS

TECHNICAL DOCUMENT

VMWARE NETWORK SETTING REQUIREMENTS VMWARE L2 NETWORK SECURITY SETTINGS

When reading through the installation guides for the Virtual Mobility Master (VMM) and Virtual Mobility Controllers (VMC), some customers may see the requirement for ‘promiscuous mode’, ‘forged transmits’, and ‘allow MAC changing’ as requirements to be enabled. Questions may arise as to why, as some documentation from VMWare and from years of ‘best practices’ are to not allow these settings due to security risks. This document will go through what these settings are, why they are required, workarounds to minimizing their impact, and other factors related to the settings.

This document is focusing on VMware and will use VMWare’s terminology and graphical images to convey some of the core concepts. However, these settings are universal to all hypervisor platforms unless said hypervisor solution natively supports promiscuous mode and forged transmits by default.

‘VSWITCHES’ AND ‘SWITCHES’, ‘PORT GROUPS’, ETC…TERMINOLOGY SOUP Before starting, it’s important to clear up some use of terminology and concepts related to this topic. First, the term ‘vSwitch’ is a VMWare specific term that refers to the virtual switch that is part of the hypervisor’s network fabric. VMWare’s vSwitch has specific behaviors and processes that are not 100% analogous to a hardware-based network switch. A good primer on VMWare’s vSwitch can be found at the link here:

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/virtual_networking_concepts.pdf

When the Virtual Mobility Master or Virtual Mobility Controller is referenced as a ‘switch’, it is specific to the switching nature of the network backplane of the controller. While the ArubaOS carries a multitude of different capabilities, it’s ultimate communication pathway to the physical network outside of the hypervisor is via its internal switch. As such, there are some factors that one needs to be aware of when talking about the VMM and VMC compared to other server-based virtualized appliances that run on VMWare ESX hypervisors.

• When servers (like Windows Server, or even ClearPass and AirWave), the mac address used by the virtual appliance is provided by the VM’s settings generated by the hypervisor and it’s a 1:1 mapping. When ‘Network Adapter 1’ maps to Eth1 on the virtual machine, those MAC addresses will match and all traffic into and out of that MAC is a 1:1 relationship.

• When the VMM and VMC are virtualized, the mapping is more complex. The VMM and VMC has two MAC addresses assigned to it (from a ‘show inventory’), but also has additional ports that share that MAC address:

o ‘Network Adapter 1’ maps to the dedicated MGMT interface

o ‘Network Adapter 2’ maps to the VMM or VMC’s ‘gig0/0/0’ interface

o The single MAC for ‘Network Adapter 2’ is what the VMM and VMC uses to send and receive switched packets to the rest of the network, across gig0/0/1, gig0/0/2, and gig0/0/3 (the last being only for VMC)

• In the case of VMMs and VMCs, which are basically virtualized switches, there are additional elements related to the VMC in terms of traffic it needs to be able to send and receive (VRRP, VLANs, STP, network broadcasts, multicast, etc) which can create additional issues depending on vSwitch policy and settings, as well as the type of traffic they will carry (broadcast traffic, multicast traffic, etc)

•

2

DOCUMENT TYPE


Within VMWare’s vSwitch architecture, there is also the concept of ‘Port Group’. These can be thought of as logical enclaves within the vSwitch itself where differentiated settings can be enabled or disabled as required. These are NOT VLAN constructs because we can have the same VLANs assigned to the vSwitch and the Port Groups within the vSwitch. Think of a Port Group as a way to logically group virtual machines within the same localized network domain, both for organizational purposes, but also for times when different vSwitch settings need to be applied. It is also a critical component to some internal VMware processes (like vMotion) but are not applicable to this topic. Specific to this topic, Port Groups are important because they allow for differentiated L2 security settings to be applied separately from the vSwitch L2 security settings applied globally to the vSwitch.

L2 SECURITY ON VSWITCHES AND PORT GROUPS When a vSwitch or Port Group is created (by default, every hypervisor has at least one single vSwitch and a single Port Group), there are L2 security settings to be aware of that apply to deploying a VMM or VMC. The following describes the function of these L2 security mechanisms. It’s important again to note that Port Groups are a construct within a vSwitch, and the Port Group can have different L2 security settings separate from the vSwitch, which VMWare created and supports for these explicit purposes. In terms of support for Aruba’s VMMs and VMCs, all of the following should be enabled.

Promiscuous Mode

VMWare’s definition of ‘Promiscuous Mode’ is as follows:

Promiscuous mode is a security policy which can be defined at the virtual switch or portgroup level in vSphere ESX/ESXi. A virtual machine, Service Console or VMkernel network interface in a portgroup which allows use of promiscuous mode can see all network traffic traversing the virtual switch.

By default, a guest operating system's virtual network adapter only receives frames that are meant for it. Placing the guest's network adapter in promiscuous mode causes it to receive all frames passed on the virtual switch that are allowed under the VLAN policy for the associated portgroup. This can be useful for intrusion detection monitoring or if a sniffer needs to analyze all traffic on the network segment.

In a more user-friendly description, putting the vSwitch or Port Group into promiscuous mode allows the VMM or VMC to hear and receive all frames destined to the Port Group or vSwitch, including frames that would otherwise be blocked by L2 security policy applied to the vSwitch, or other types of frames that would traditionally be dropped by the vSwitch by default.

Forged Transmits

When forged transmits are set to allow, the vSwitch or Port Group will allow the VM to send out frames using a different MAC address than the one assigned to the VM by the hypervisor. Important in this respect is to features like

• VLANs (dot1q trunking)

• Use of multiple interfaces (If more than gig0/0/1 is used in the data path)

• VRRP (Generated MAC of (00:00:5E:00:01:VRID)

• Multicast (MAC depends on multicast traffic type)

Allow MAC Changes

Enabling this setting allows the VM to change their unicast and allows the device to see other unicast frames. Generally, if promiscuous mode and forged transmits are enabled this should be set to ‘enable’ as well to allow the VMM or VMC to send or respond to packets sent to it.

3

DOCUMENT TYPE


WHY ARE THESE L2 SECURITY SETTINGS REQUIRED? When looking at ‘what’ the VMM and VMC are, let’s use the above information to look at the L2 packet flow from the LAN to the hypervisor, through the vSwitch and Port Group, to the VMM. In Figure 1-1, the L2 elements within the data flow are represented.

Figure 1-1

In the above figure, the VMM data plane(bottom) is used for Mobility Controller (MC) termination, admin user session traffic, VRRP IPs, VLANs, etc), and will use the MAC-AC:20 (or the VRRP MAC) as the SRC and DST for all frames (gig0/0/0 and gig0/0/1), and is what will be used for all ARP. Because the vSwitch tracks packets, any VRRP packets destined for the VRRP MAC (00:00:5E:00:01:VRID), or if any packets are destined for any resource on Gig0/0/1, because the vSwitch hasn’t learned MAC-AD:30, it will be dropped at the vSwitch. If VRRP is sent out, or a packet destined for the VRRP MAC, that will be dropped as well. The same applies for any multicast traffic. However, with those L2 security settings enabled, those packets are flooded down all active interfaces and the VMM will then receive the packets and can process the frames without issue.

In the simplest case (Figure 1-2), there are two VMMs, which are L2 connected. The Mobility Controller (MC) clusters communicate to the MMs via the VRRP IP established between them. As such, each MM needs to be able to transmit and receive from its own MAC Address as well as the VRRP MAC address at L2. If promiscuous mode was disabled, the VRRP frames would be dropped because the vSwitch would drop the frames. If forged transmits and allow MAC changing were disabled, each MM would not be able to transmit the VRRP frames because the vSwitch or Port Group would not allow the VRRP MAC through it. Same for if VLANs were used, or the VMMs needed to receive traffic that would otherwise be dropped by the vSwitch or Port Group.

Figure 1-2

4

DOCUMENT TYPE


Figure 1-3

When Virtual Mobility Controllers (VMC) are used (Figure 1-3), not only is VRRP involved, but there is also the use of VLANs and other network broadcasts like multicast that the VMCs require to operate, terminate APs, and provide connectivity and services to the users. As such, promiscuous mode, forged transmits, and allow MAC changing must be enabled. If that is not possible, hardware controllers must be used.

IS PROMISCUOUS MODE A SECURITY RISK? In short, the answer to this depends a great deal on what virtual machines are running on the vSwitch and port group that the VMM or VMC is assigned to. Assuming the vSwitch and Port Groups contain secure servers and appliances that have limited access to the appropriate admins on the same team, then generally no (Security Architects may disagree, but that is generally what they do, looking at worst case). When the access to the hypervisors and machines running on the hypervisors have restricted access to the same teams or individuals only, the risk is minimal to non-existent.

In another case, there could be machines hosted on the same vSwitch or Port Group whose admins are from different teams or business units. Worst case, there could be situations where there is a public facing web server running on the same vSwitch as other internal appliances. In these cases, the security ‘domain’ is mixed, and when promiscuous mode is enabled, a bad actor could log in to a Linux machine on said vSwitch or Port Group and sniff the packets of another machine managed by a different team or group, giving them visibility into the traffic they would otherwise not have access to. Finally, if one VM is exposed to a 3rd party for monitoring or remote admin monitoring and assistance, that 3rd party could potentially see packets from other machines on the same vSwitch or Port Group, which would certainly have significant security implications to the devices on that hypervisor.

The solution to address these L2 security issues is to first assess the security domain of the machines running on the same vSwitch or Port Group, and then construct the vSwitch and Port Groups to provide the necessary isolation. Options within VMWare are multifaceted.

The simplest design is, when the security domain is flat (see Figure 1-4) and all machines are part of the same security group and managed by the same admins, are not publicly exposed, etc, a single vSwitch and Port Group, and all VMs are placed within the same Port Group. All devices can share the same L2 Security Settings (promiscuous mode and forged transmits are enabled, etc).

In some cases, internal security policies defined outside of the VMWare admins may apply as well, regardless of reasoning. In those cases, additional steps may be required.

Figure 1-4

5

DOCUMENT TYPE


Figure 1-5

If the VMs belong to different security domains, then the use of separate Port Groups can be used to provide a more isolated security domain where the vSwitch and all Port Groups have different L2 security policies applied. In Figure 1-5, the diagram shows a two hypervisor vSwitch and Port Group architecture with multiple uplinks to the network. Each ESX host has three physical adapters, one vSwitch (called ‘uplink port group’ in the image), and four Port Groups on the vSwitch. Within the ‘Test environment’ and ‘Production’ Port Groups are two VMs each. This scenario could allow the two Port Groups to both have promiscuous mode and forged transmits enabled between them, allowing the two VMs to have full visibility between them, but not between the ‘Test Environment’ and ‘Production’ environment, providing proper isolation between the two port groups.

To provide the strongest security between vSwitches and port groups is to use dedicated physical adapter(s) and separate vSwitches. To do this, the hypervisor needs multiple physical interfaces (at least one dedicated NIC per vSwitch). In this scenario (Figure 1-6), The ‘External Network’ vSwitch maps to its own physical NIC(s) that uplinks to ‘External Network 1’, the ‘Internal Network’ vSwitch maps to its own physical NIC(s) connected to ‘Internal Network 2’, and the ‘DMZ’ vSwitch maps to its own physical NIC(s) connected to two separate physical networks. In this scenario, it provides the maximum protection between each vSwitch and port group so that a higher level of isolation exists between the vSwitches. In this scenario, every vSwitch could certainly connect to the same network as well, the protection here exists to completely isolate vSwitch and completely remove any risk of one VM being able to hear another on a different vSwitch (assuming port groups are not enough security). While not all servers will have a large number of wired NICs available, NICs are easily added to hypervisors and are relatively inexpensive.

Figure 1-6

OTHER VMWARE FEATURES AND CONFGIGURATIONS VMWare also has some additional features that may come in to play depending on the VMWare deployment configuration. As part of the larger scale hypervisor architecture (vSphere), VMWare also supports a feature called ‘vSphere Distributed Switching’ (VDS). VDS supports providing a single, managed virtual switch that extends across multiple hypervisors within a vSphere deployment. A VDS virtual switch is referred to as a ‘vNetwork Distributed Switch’ (or dvSwitch), and within a dvSwitch there can be one or multiple Distributed Port Groups (or dvPortGroup). This allows for a more unified, global virtual switching fabric to be abstracted from a single hypervisor and spreads it across multiple hypervisors within the vSphere deployment.

6

DOCUMENT TYPE


Looking at Figure 1-7, the dvSwitch can be seen spreading across multiple ESX hosts. The same concepts of vSwitches and port groups apply here and work more or less the same way. The biggest change is that this generally requires diligence in the uplink configs to ensure that all VLANs are carried properly on all the hypervisors. Additionally, at this scale, LACP and other network configs are in place that need to be validated and checked. LACP is also relevant to VMMs and VMCs deployed on a dvSwitch, to ensure that packet flow is properly handled within the dvSwitch to the VMM and VMCs.

Multiple Physical NICs tied to the same vSwitch

To provide redundancy for a vSwitch, VMWare supports multiple NICs on a single vSwitch, providing redundancy to one or multiple switches. VMWare supports multiple modes, some of which are VMware specific and proprietary, and others are based on standards-based network protocols. And within those different modes of operation are additional settings that can be applied. This document will not cover all the options (those can be read up on within VMWare’s own documentation), but will only cover what Aruba recommend the settings be to fully support the VMM and VMC. The purpose here is that, because the VMM and VMC is a virtualized switch, it is sensitive to loop detection, and an improperly configured NIC Teaming configuration on the vSwitch or dvSwitch could result in a loop detection and cause port blocking or an actual network loop to occur.

Within a vSwitch (single hypervisor), Aruba recommends that the NIC Teaming configuration use the modes identified in Table 1-1. There are other options, assuming the hypervisor is licensed with the ‘Enterprise Plus’ license, but without the additional license, using the aforementioned settings will work.

In addition, another setting that needs to be enabled on the hypervisor is to configure ‘ReversePathFwdCheckPromisc’. This setting is used to drop duplicate packets in the absence of LACP or PortChannel on the upstream switch. The ‘Aruba Virtual Appliance Installation Guide’ for AOS 8 covers these settings and how to implement them.

When looking at NIC Teaming within a dvSwitch, which requires the ‘VMWare Enterprise Plus’ license, LACP is recommended between the NIC Teaming adapters and the upstream switch. LACP or 802.11ad allows for the concurrent use of multiple NICs for both additional throughput and redundancy. The ‘Aruba Virtual Appliance Installation Guide’ for AOS 8 covers these settings and how to implement them.

Table 1-1

VMWare Virtual Switch Type NIC Teaming Options Supported

vSwitch* “Route based on Originating PortID” or “Use Explicit Failover”

dvSwitch* LACP**

* - Requires ‘ReversePathFwdCheckPromisc’ to be enabled. When changed, all L2 settings will need to be checked and re-enabled if disabled when the setting is implemented.

** - Requires VMWare Enterprise Plus license

7

DOCUMENT TYPE


CAN THE VIRTUAL MOBILITY MASTER RUN WITHOUT PROMISCOUS MODE AND FORGED TRANSMITS? First, it is imperative to state that the only officially supported configuration of a VMM or VMC is per the ‘Virtual Mobility Master and Virtual Mobility Controller Installation Guide’ found on support.arubanetworks.com. Every customer’s environment is different, and certainly within each customer’s virtual infrastructure there are vast differences from site to site, and customer to customer. If the configuration deviates from this, and an issue or problem arises where a technical support (TAC) case is required, the likely first change TAC will recommend is to enable promiscuous mode, forged transmits, and allow MAC changing, especially if TAC determines that is the root cause of the issue. Secondly, when deploying a VMC, those settings will absolutely be required to function properly.

In cases where a customer is unwilling or unable to follow the recommendations above, or is unwilling or unable to implement the protection strategies provided by VMware with Port Groups or separate, dedicated vSwitches, Aruba provides alternatives in the form of a hardware-based Mobility Master (HMM) and hardware-based Mobility Controllers (MC) that runs outside of a virtual environment. As a result, in cases where these features just cannot be enabled, the HMM alternative provides 100% complete functionality with zero impairment in terms of features, performance, or capabilities.

If, however, a customer decides to go that route anyway, and deploys a VMM with those settings disabled, the following restrictions and limitations will apply.

• The VMM can only have one virtual interface enabled, with a single IP address, and no VLAN support

• No Layer2 VMM redundancy will be available

• No support for broadcast or multicast traffic

• No guarantee that future code versions will not require it for other services or features that require it later

• No support for multiple physical NICs with NIC Teaming, there can only be a single NIC per vSwitch

The above restrictions will severely impact and limit some of the robustness of the Mobility Master architecture, but if the customer is willing to accept the risks outlined above, and the possible limitations in feature support, a customer is more than welcome to undertake that effort.

CONCLUSION: IT’S ALWAYS MORE COMPLEX THAN ORIGINALLY THOUGHT VMWare has led to an explosion in simplicity and expansibility within the data center, and rapid evolutions in how data centers and infrastructure are now deployed. However, while the fundamentals of virtualization are fairly simple, it’s a new ‘thing’ in virtualization to virtualize appliances that switch and bridge traffic. Some of these new appliances and the capabilities being introduced will sometimes involve a re-evaluation as to how these large, virtualization systems are configured and deployed.

For any future inquiries, please reach out to your local Aruba sales representative or systems engineer.

SOURCES https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/virtual_networking_concepts.pdf

https://kb.vmware.com/s/article/1002934

https://www.arubanetworks.com/assets/atmosphere/2018-Airheads/Fundamentals%20Guide%20-%20ArubaOS%208%20Fundamentals.pdf

Google image search of “site:vmware.com”

vmware network setting requirements

Documents