iaas product line architecture fabric architecture guide
Post on 03-Jun-2018
234 Views
Preview:
TRANSCRIPT
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 1/112
Infrastructure-as-a-ServiceProduct Line Architecture
Fabric Architecture Guide
Published May 2013
Authors
Adam Fazio, David Ziembicki, Joel Yoker - Microsoft ServicesContributors
Carlos Mayol Berral, Ricardo Machado, Artem Pronichkin - Microsoft Services
Reviewer
Yuri Diogenes
For the latest information, please see the Microsoft Server and Cloud Platform site.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 2/112
2 Fabric Architecture Guide
Copyright information
This document is provided "as-is". Information and views expressed in this document, including URL and otherInternet website references, may change without notice.
Some examples depicted herein are provided for illustration only and are fictitious. No real association or connection
is intended or should be inferred.
This document does not provide you with any legal rights to any intellectual property in any Microsoft product. Youmay copy and use this document for your internal, reference purposes.
Microsoft, Active Directory, BitLocker, Hyper-V, Windows PowerShell, Windows Vista, Windows, and Windows Serverare either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.All other trademarks are property of their respective owners.
© 2013 Microsoft Corporation. All rights reserved.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 3/112
Fabric Architecture Guide 3
Contents1 Introduction ..............................................................................................................................................................................
1.1 Scope ...............................................................................................................................................................................
1.2 Microsoft Private Cloud Fast Track ........................................................................................................................... 5
1.3 Microsoft Services ...........................................................................................................................................................
2 IaaS Product Line Architecture Overview .......................................................................................................................... 7
2.1 IaaS Reference Architectures ....................................................................................................................................... 7
2.2 Product Line Architecture Design Patterns ............................................................................................................ 8
2.3 Windows Licensing .........................................................................................................................................................
3 Continuous Availability over SMB Storage Pattern Overview ............................................................................... 10
4 Non-Converged Infrastructure Pattern Overview ...................................................................................................... 14
5 Converged Infrastructure Pattern Overview ................................................................................................................. 17
6 Storage Architecture .............................................................................................................................................................. 20 6.1.1 Disk Architecture ...................................................................................................................................................... 2
6.1.2 Storage Controller Architectures ....................................................................................................................... 22
6.1.3 Storage Networking ................................................................................................................................................ 25
6.1.4 Windows File Services ............................................................................................................................................ 32
6.1.5 SAN and Storage Array Features ....................................................................................................................... 36
6.1.6 Storage Management and Automation .......................................................................................................... 37
7 Network Architecture............................................................................................................................................................. 4
7.1.1 Network Architecture Patterns ........................................................................................................................... 40
7.1.2 Network Performance and Low Latency ......................................................................................................... 42
7.1.3 Network High Availability and Resiliency ....................................................................................................... 44
8 Compute Architecture ........................................................................................................................................................... 48
8.1.1 Server Architecture .................................................................................................................................................. 48
8.1.2 Failover Clustering ................................................................................................................................................... 49
9 Hyper-V Virtualization Architecture ................................................................................................................................. 54
9.1.1 Windows Server 2012 Hyper-V Features ........................................................................................................ 54
9.1.2 Windows Server 2012 Hyper-V Failover Clustering ................................................................................... 66
9.1.3 Hyper-V Guest Virtual Machine Design .......................................................................................................... 70
10 Fabric and Fabric Management ......................................................................................................................................... 73
10.1.1 Fabric ......................................................................................................................................................................
10.1.2 Fabric Management ........................................................................................................................................... 74
11 Non-Converged Architecture Pattern ............................................................................................................................. 74
11.1.1 Compute .................................................................................................................................................................
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 4/112
4 Fabric Architecture Guide
11.1.2 Network ..................................................................................................................................................................
11.1.3 Storage ....................................................................................................................................................................
12 Converged Architecture Pattern ........................................................................................................................................ 83
12.1.1 Compute .................................................................................................................................................................
12.1.2 Network .................................................................................................................................................................. 12.1.3 Storage ....................................................................................................................................................................
13 Continuous Availability over SMB Storage Architecture Pattern ......................................................................... 89
13.1.1 Compute .................................................................................................................................................................
13.1.2 Network ..................................................................................................................................................................
13.1.3 Storage ....................................................................................................................................................................
14 Multi-Tenant Designs ............................................................................................................................................................ 99
14.1.1 Requirements Gathering .................................................................................................................................. 99
14.1.2 Infrastructure Requirements ......................................................................................................................... 100
14.1.3 Multi-Tenant Storage Considerations ...................................................................................................... 101
14.1.4 Multi-Tenant Network Considerations ..................................................................................................... 104
14.1.5 Multi-Tenant Compute Considerations ................................................................................................... 110
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 5/112
Fabric Architecture Guide 5
1 Introduction
The goal of the Infrastructure-as-a-Service (IaaS) reference architecture is to help organizations developand implement private cloud infrastructures quickly while reducing complexity and risk. The IaaS productline architecture (PLA) provides a reference architecture that combines Microsoft software, consolidatedguidance, and validated configurations with partner technology such as compute, network, and storagearchitectures, in addition to value-added software components.
The private cloud model provides much of the efficiency and agility of cloud computing, with theincreased control and customization that are achieved through dedicated private resources. Byimplementing private cloud configurations that align to the IaaS PLA, Microsoft and its hardware partnerscan help provide organizations the control and the flexibility that are required to reap the potentialbenefits of the private cloud.
The IaaS PLA utilizes the core capabilities of the Windows Server operating system, Hyper-V, and SystemCenter to deliver a private cloud infrastructure as a service offering. These are also key software
components that are used for every reference implementation.
1.1 ScopeThe scope of this document is to provide customers with the necessary guidance to develop solutions fora Microsoft private cloud infrastructure in accordance with the reference architecture patterns that areidentified for use with the Windows Server 2012 operating system. This document provides specificguidance for developing fabric architectures (compute, network, storage, and virtualization layers) of anoverall private cloud solution.
Accompanying guidance is provided in the following guide for the development of an accompanyingfabric management architecture that uses System Center 2012 with Service Pack 1 (SP1):
Infrastructure-as-a-Service Fabric Management Architecture Guide
1.2 Microsoft Private Cloud Fast TrackThe Microsoft Private Cloud Fast Track is a joint effort between Microsoft and its hardware partners todeliver preconfigured virtualization and private cloud solutions. The Private Cloud Fast Track focuses onthe new technologies and services in Windows Server 2012 in addition to investments in SystemCenter 2012 with Service Pack 1.
The validated designs in the Private Cloud Fast Track are delivering a ―best-of-breed solution‖ from ourhardware partners that drive Microsoft technologies, investments, and best practices. The Private CloudFast Track has expanded the footprint, and it enables a broader choice with different architectures. Marketavailability of the Private Cloud Fast Track validated designs from our hardware partners have beenlaunched with Microsoft solutions. Please visit the Private Cloud Fast Track website for the most up-to-date information and validated solutions.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 6/112
6 Fabric Architecture Guide
1.3 Microsoft ServicesMicrosoft Services is comprised of a global team of architects, engineers, consultants, and supportprofessionals who are dedicated to helping customers maximize the value of their investment in Microsoftsoftware. Microsoft Services touches customers in over 82 countries, helping them plan, deploy, support,and optimize Microsoft technologies. Microsoft Services works closely with Microsoft Partners by sharingtheir technological expertise, solutions, and product knowledge. For more information about the solutionsthat Microsoft Services offers or to learn about how to engage with Microsoft Services and Partners,please visit the Microsoft Services website.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 7/112
Fabric Architecture Guide 7
2 IaaS Product Line Architecture OverviewThe IaaS PLA is focused on deploying virtualization fabric and fabric management technologies inWindows Server 2012 and System Center to support private cloud scenarios. This PLA includes referencearchitectures, best practices, and processes for streamlining deployment of these platforms to supportprivate cloud scenarios.
This component of the IaaS PLA focuses on delivering core foundational virtualization fabric infrastructureguidance that aligns to the defined architectural patterns within this and other Windows Server 2012private cloud programs. The resulting Hyper-V infrastructure in Windows Server 2012 can be leveraged tohost advanced workloads, and subsequent releases will contain fabric management scenarios usingSystem Center components. Scenarios relevant to this release include:
Resilient infrastructure – Maximize the availability of IT infrastructure through cost-effectiveredundant systems that prevent downtime, whether planned or unplanned.
Centralized IT – Create pooled resources with a highly virtualized infrastructure that supportsmaintaining individual tenant rights and service levels.
Consolidation and migration – Remove legacy systems and move workloads to a scalable high-performance infrastructure.
Preparation for the cloud – Create the foundational infrastructure to begin transition to a privatecloud solution.
2.1 IaaS Reference ArchitecturesMicrosoft Private Cloud programs have three main solutions as shown in figure 1. This document focuseson the open solutions model to service the Enterprise and hosting provider audiences.
Figure 1 Branches of the Microsoft Private Cloud
Each audience should use a reference architecture that defines the requirements that are necessary todesign, build, and deliver virtualization and private cloud solutions for small and medium enterprise andhosting service provider implementations. Figure 2 shows an example of these reference architectures.
SMB solutions
From 2 to 4 hosts
Up to 75 server virtual machines
Open solutions
From 6 to 64 hostsUp to 8,000 servervirtual machines
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 8/112
8 Fabric Architecture Guide
Storage infrastructure
Volumes
Volume1
Cluster Shared Volumes (CSV2)
Volume- n
Network infrastructure
Server infrastructure
Storage infrastructure
Volumes
Volume1
Cluster Shared Volumes (CSV2)
Volume- n
Network infrastructure
Server infrastructure
Small configuration
From 8 to 16computecluster nodesDedicated 2-node fabric-managementcluster
From 6 to 8compute clusternodesDedicated orintegrated fabricmanagement
Medium configuration
Figure 2 Examples of reference architectures
Each reference architecture combines concise guidance with validated configurations for the compute,network, storage, and virtualization layers. Each architecture presents multiple design patterns to enablethe architecture, and each design pattern describes the minimum requirements for each solution.
2.2 Product Line Architecture Design PatternsAs previously described, Windows Server 2012 utilizes innovative hardware capabilities and enables whatwere once considered advanced scenarios and capabilities from commodity hardware. These capabilitieshave been summarized into initial design patterns for the IaaS PLA. Identified patterns include the
following infrastructures:
Non-converged infrastructure Converged infrastructure Continuous availability over SMB storage infrastructure
Each design pattern guide outlines the high-level architecture, provides an overview of the scenario,identifies technical requirements, outlines all dependencies, and provides guidelines as to how thearchitectural guidance applies to each deployment pattern. Each pattern also includes an array of fabricconstructs in the categories of compute, network, storage, and virtualization, which comprise the pattern.Each pattern is outlined in this guide with an overview of the pattern and a summary of how each patternleverages each component area.
2.3 Windows LicensingIaaS PLA architectures use the Windows Server 2012 Standard or Windows Server 2012 Datacenter. Formore information about the Windows Server 2012 operating systems, please see Windows Server 2012 onthe Microsoft website.The packaging and licensing for Windows Server 2012 have been updated to simplify purchasing andreduce management requirements, as shown in the following table. The Windows Server 2012 editions aredifferentiated only by virtualization rights —two virtual instances for the Standard edition, and an
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 9/112
Fabric Architecture Guide 9
unlimited number of virtual instances for the Datacenter edition. Running instances can exist in a physicaloperating system environment (POSE) or in a virtual operating system environment (VOSE).
Edition Running instances in POSE Running instances in VOSE
Datacenter 1 UnlimitedStandard 1* 2
Table 1 Licensing of Windows Server 2012 editions
* When a customer is running all allowed virtual instances, the physical instance may only be used to manage andservice the virtual instances.
For more information about Windows Server 2012 licensing, see Windows Server 2012: How to Buy . Forinformation about licensing in virtual environments, see Microsoft Volume Licensing Brief: LicensingMicrosoft Server Products in Virtual Environments on the Microsoft Download Center .
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 10/112
10 Fabric Architecture Guide
3 Continuous Availability over SMB Storage Pattern OverviewThe Continuous Availability over Server Message Block (SMB) Storage pattern supports Hyper-V clustereddeployments in Windows Server 2012. Continuous availability and transparent failover are delivered overa Scale-Out File Server cluster infrastructure, and SMB shared storage by using a converged hardwareconfiguration and native capabilities in the Windows Server 2012 operating system. This pattern has threevariations:
Variation A: SMB Direct using Shared Serial Attached SCSI (SAS)/Storage Spaces Variation B: SMB Direct using Storage Area Network (SAN) Variation C: SMB 3.0-Enabled Storage
Note SMB Direct is based on SMB 3.0, and it supports the use of network adapters that have RemoteDirect Memory Access (RDMA) capability.
Variation A uses SMB Direct using Shared SAS and Storage Spaces to provide storage capabilities overdirect attached storage (DAS) technologies. This pattern combines a Scale-Out File Server cluster
infrastructure with SMB Direct to provide back-end storage that has similar characteristics to traditionalSAN infrastructures and supports Hyper-V and SQL Server workloads.
Figure 3 outlines a conceptual view of Variation A.
SAS disks
SAScontroller
SAScontroller
Scale-Out File-Server cluster
Storage Spaces
Cluster Shared Volumes(CSV2) + CSV cache
RDMANIC
RDMANIC
Hyper-V host cluster(s)
RDMANIC
RDMANIC
SMB3 DirectSMB3 Transparent Failover
SMB3 Multichannel
EthNIC
EthNIC
NICteaming
Hyper-Vextensible
switch
VMs
VMs
VHDs
LAN
Figure 3 Conceptual view of variation A
Variation B describes the use of SMB Direct with SAN-based storage, which provides the advanced
storage capabilities that are found in storage area network (SAN) infrastructures. SAN-based storagesolutions typically provide additional features beyond what can be provided natively through theWindows Server 2012 operating system by using shared direct attached ―Just a Bunch of Drives‖ (JBOD)storage technologies. Although this variation is generally more expensive, its primary trade-offs weighcapability and manageability over cost.
Variation B is similar to Variation A. It utilizes a Scale-Out File Server cluster infrastructure with SMB Direct;however, the back-end infrastructure is a SAN-based storage array. In this variation, innovative storage
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 11/112
Fabric Architecture Guide 11
capabilities that are typically associated with SAN infrastructures can be utilized in conjunction with RDMAand SMB connectivity for Hyper-V workloads.
Figure 4 outlines a conceptual view of Variation B.
Fibre Channel/iSCSI
Scale-Out File-Server cluster
Volumes
Cluster Shared Volumes(CSV2) + CSV cache
RDMANIC
RDMANIC
Hyper-V host cluster(s)
RDMANIC
RDMANIC
SMB3 DirectSMB3 Transparent Failover
SMB3 Multichannel
EthNIC
EthNIC
NICteaming
Hyper-Vextensible
switch
VMs
VMs
VHDs
LAN
Fibre Channel/iSCSI
SAN storage
Figure 4 Conceptual view of variation B
In Variation C, instead of using Scale-Out File Server clusters and SMB Direct, SMB 3.0-enabled storagedevices are used to provide basic storage capabilities, and Hyper-V workloads utilize the SMB sharedresources directly. This configuration might not provide advanced storage capabilities, but it provides anaffordable storage option for Hyper-V workloads.
Figure 5 outlines a conceptual view of Variation C.
Figure 5 Conceptual view of variation C
Although the following list of requirements is not comprehensive, the components that are listed inTable 2 are expected for the Continuous Availability over SMB Storage pattern.
SMB3 Enabled Device
Volumes
RDMANIC
RDMANIC
Hyper-V Host Cluster(s)
RDMANIC
RDMANIC
SMB3 DirectSMB3 Transparent Failover
SMB3 Multi-Channel
EthNIC
EthNIC
NICTeaming
Hyper-VExtensible
Switch
VMs
VMs
VHDsLAN
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 12/112
12 Fabric Architecture Guide
Expected Components Optional Components Dedicated fabric management hosts Support for single root I/O virtualization
(SR-IOV) network card 10 gigabit Ethernet (GbE) or higher
network connectivity SMB 3.0-enabled storage array (required for
SMB 3.0-enabled storage variation only)
Redundant paths for all storagenetworking components
Addition of support for SR-IOV network card
SMI-S–compliant management interfacesfor storage components
Addition of a certified Hyper-V extensible virtualswitch extension
Dedicated hosts for Scale-Out File Servercluster
RDMA network connectivity (RoCE orInfiniBand)
Shared SAS JBOD storage array (requiredfor SMB Direct that uses the SharedSAS/Storage Spaces variation only)
Table 2 Database instances and requirements
Table 3 outlines Windows Server 2012 features and technologies that are utilized in this pattern.
Windows Server 2012 Feature Key Scenarios
Increased VP:LP ratio Removes previous limits of 8:1 processor ratios for serverworkloads and 12:1 processor ratios for client workloads.
Increased virtual memory and DynamicMemory
Supports up to 1 TB of memory inside virtual machines.
Virtual machine guest clusteringenhancements (Fibre Channel)
Supports virtual machine guest clusters by using iSCSIconnections or by using the Hyper-V Fibre Channel adapter toconnect to shared storage.
Hyper-V extensible switch A virtual Ethernet switch that allows for third party filtering,capturing, and forwarding of extensions that are to be addedto support additional virtual-switch functionality on the Hyper-V platform.
Encrypted cluster volumes Enables support for BitLocker-encrypted Cluster SharedVolumes version 2 (CSVv2).
Cluster-aware updating Provides the ability to apply updates to running failoverclusters through coordinated patching of individual failovercluster nodes.
Offloaded data transfer (ODX) Supports storage-level transfers that use ODX technology(SAN feature).
Support for SR-IOV network Allows virtual machines the virtual functions (VF) from SR-IOV-
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 13/112
Fabric Architecture Guide 13
enabled physical network interface for increased IO.
Support for 4K physical disk Support for native 4K disk drives on hosts.
Diskless network boot with iSCSI Target Provides the network boot capability on commodity hardwareby using an iSCSI boot –capable network adapter or a softwareboot loader (such as iPXE or netBoot/i).
Quality of service (QoS) minimum bandwidth(fair share)
Assigns a certain amount of bandwidth to a given type oftraffic and helps make sure that each type of network trafficreceives up to its assigned bandwidth.
Virtual machine storage enhancements (VHDX) Supports VHDX disks that are up to 64 TB in size.
NIC Teaming, load-balancing failover (LBFO)support
Supports switch-independent and switch-dependent loaddistribution by using physical and virtual network connections.
IPsec offload Supports network adapters that are equipped with hardwarethat reduces the load on the central processing unit (CPU) byperforming this computationally intensive work.
Data center bridging (DCB) Provides hardware support for converged fabrics and allowsbandwidth allocation and priority flow control.
Storage spaces Enables cost-effective, optimally used, highly available,scalable, and flexible storage solutions in virtualized or physical
deployments.Hyper-V over SMB Direct Supports use of SMB 3.0 file shares as storage locations for
running virtual machines by using low-latency RDMA networkconnectivity.
Data deduplication Involves finding and removing duplication within data withoutcompromising its fidelity or integrity.
Note Deduplication is not recommended for files that areopen and constantly changing for extended periods of time orthat have high I/O requirements.
SMB multichannel Allows file servers to use multiple network connectionssimultaneously, which provides increased throughput andnetwork fault tolerance.
Table 3 Windows Server 2012 features and key scenarios
Key drivers that would encourage customers to select this design pattern include lower cost of ownershipand flexibility with shared SAS JBOD storage solutions. Decision points for this design pattern over othersfocus primarily on the storage aspects of the solution in combination with the innovative networkingcapabilities of SMB Multichannel and RDMA.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 14/112
14 Fabric Architecture Guide
4 Non-Converged Infrastructure Pattern OverviewThe non-converged design pattern uses a standard Hyper-V clustered deployment with non-convergedstorage (traditional SAN), networking, and compute-hardware infrastructure. The storage network andnetwork paths are isolated by using dedicated I/O adapters. Failover and scalability are achieved on thestorage network through Multipath I/O (MPIO). The TCP/IP network uses NIC Teaming.In this pattern, Fibre Channel or Internet SCSI (iSCSI) is expected to be the primary connectivity to ashared storage network. High-speed 10-gigabit Ethernet (GbE) adapters are common for advancedconfigurations of TCP/IP traffic.
Figure 6 outlines an overview of the non-converged design pattern.
SAN-based storage
Volumes
F C
F C
Hyper-V cluster
Volume1
Cluster Shared Volumes (CSV2)
Volume- n
Teamed networkinterfaces (LBFO)
i S C S I
i S C S I
Figure 6 Non-converged design pattern
The non-converged pattern is expected to have two variations:
Variation A: Fibre Channel Variation B: iSCSI
Figure 7 outlines a conceptual view of this pattern.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 15/112
Fabric Architecture Guide 15
Fibre Channel/iSCSI
SAN storage Hyper-V host cluster(s)
EthNIC
EthNIC
NICteaming(LBFO)
Hyper-Vextensible
switch
VMs
VMs
VHDs
LAN
Fibre Channel/iSCSI
Fibre Channel/iSCSI
Fibre Channel/iSCSI
Cluster Shared Volumes(CSV2) + CSV cache
Volumes
Figure 7 Non-converged design pattern variations
Although the following list of requirements is not comprehensive, this design pattern expects thecomponents that are listed in Table 4.
Expected Components Optional Components Dedicated fabric management hosts Storage-array support for ODX 10 GbE of network connectivity Additional Fibre Channel host bus adapters
(HBAs), as required, to support complex virtualFibre Channel configurations within virtualmachines
Redundant paths for all storagenetworking components
Support for single root I/O virtualization (SR-IOV) network cards
SMI-S–compliant management interfaces
for storage components
RDMA network connectivity
A certified Hyper-V extensible virtual switchextension
Network-adapter support for Data centerbridging (DCB)
Table 4 List of components for this design pattern
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 16/112
16 Fabric Architecture Guide
Table 5 outlines the Windows Server 2012 features and technologies that are utilized in this pattern.
Windows Server 2012 Feature Key ScenariosIncreased VP:LP ratio Removes previous limits of 8:1 processor ratios for server
workloads and 12:1 processor ratios for client workloads. Increased virtual memory and Dynamic Memory Supports up to 1 TB of memory inside virtual machines.
Virtual machine guest clustering enhancements(Fibre Channel)
Supports virtual machine guest clusters by using iSCSIconnections or by using the Hyper-V Fibre Channeladapter to connect to shared storage.
Hyper-V extensible switch A virtual Ethernet switch that allows for third partyfiltering, capturing, and forwarding of extensions that areto be added to support additional virtual-switchfunctionality on the Hyper-V platform.
Encrypted cluster volumes Enables support for BitLocker-encrypted Cluster SharedVolumes version 2 (CSVv2).
Cluster-aware updating Provides the ability to apply updates to running failoverclusters through coordinated patching of individualfailover-cluster nodes.
Offloaded data transfer (ODX) Support for storage-level transfers that use ODXtechnology (SAN feature).
Support for SR-IOV network Allows virtual machines the virtual functions (VF) from SR-IOV-enabled physical network interface for increased IO.
Support for 4K physical disk Supports native 4K disk drives on hosts.
Diskless network boot with iSCSI Software Target Provides the network-boot capability on commodityhardware by using an iSCSI boot –capable network adapteror a software boot loader (such as iPXE or netBoot/i).
Quality of service (QoS) minimum bandwidth (fairshare)
Assigns a certain amount of bandwidth to a given type oftraffic and helps make sure that each type of network
traffic receives up to its assigned bandwidth.Virtual machine storage enhancements (VHDX) Supports VHDX disks that are up to 64 TB in size.
NIC Teaming, load-balancing failover (LBFO)support
Supports switch-independent and switch-dependent loaddistribution by using physical and virtual networkconnections.
IPsec offload Supports network adapters that are equipped withhardware that reduces the load on the central processingunit (CPU) by performing this computationally intensivework.
Data center bridging Provides hardware support for converged fabrics, whichallows bandwidth allocation and priority flow control.
Table 5 Windows Server 2012 features and key scenarios
Key drivers that would encourage customers to select this design pattern include current capital andintellectual investments in SAN and transformation scenarios that include using an existing infrastructurefor upgrading to a newer platform. Decision points for this design pattern include storage investments,familiarity, and flexibility of hardware.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 17/112
Fabric Architecture Guide 17
5 Converged Infrastructure Pattern OverviewIn this context, a ―converged infrastructure‖ refers to sharing a network topology between traditionalnetwork and storage traffic. This typically implies Ethernet network devices and network controllers thathave particular features to provide segregation, quality of service (performance), and scalability. The resultis a network fabric that features less physical complexity, greater agility, and lower costs than those thatare associated with traditional Fiber Channel-based storage networks.
This topology supports many storage designs, including traditional SANs, SMB 3.0-enabled SANs, andWindows-based Scale-Out File Server clusters. The main points in a converged infrastructure are that allstorage connectivity is network-based, and it uses a single media (such as copper). SFP+ adapters aremore commonly used.
Server choices include converged blade systems and rack-mount servers —the key differentiators beinghow the servers connect to storage and the advanced networking features. High-density blade systemsare common and feature advanced hardware options that present physical or virtual network adapters tothe Hyper-V host that is supporting a variety of protocols.
Figure 8 depicts a converged configuration in which the following points should be noted: Host storage adapters can be physical or virtual, and they must support iSCSI, Fibre Channel over
Ethernet (FCoE), and optionally SMB Direct. Many storage devices are supported, including traditional SANs and SMB Direct –capable storage.
Figure 8 Converged infrastructure design pattern
SAN Storage
Volumes
Cluster Shared Volumes(CSV2) + CSV Cache
Fiber Channel /iSCSI
Hyper-V Host Cluster(s)
CNA
CNA
Hyper-VExtensible
Switch
VMs
VMs
VHDsLAN
Fiber Channel /iSCSI
Fiber ChanneliSCSI
SMB Direct
NICTeaming
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 18/112
18 Fabric Architecture Guide
Although the following list of requirements is not comprehensive, the components that are listed in Table6 are expected for this design pattern.
Expected Components Optional Components Dedicated fabric management hosts Storage-array support for ODX 10 GbE of network connectivity Ability to present Fibre Channel HBAs, as
required, to support complex virtual FibreChannel configurations within virtual machines
Redundant paths for all storagenetworking components
Support for SR-IOV network card
SMI-S–compliant management interfacesfor storage components
RDMA network connectivity
Quality of service (QoS) A certified Hyper-V extensible virtual switchextension
Data center bridging (DBC) –capablehardware
NIC Teaming (hardware-based orsoftware-based 1)
Table 6 List of components for this design pattern
Table 7 outlines Windows Server 2012 features and technologies that are utilized in this pattern.
1 Windows Server 2012 software-based NIC Teaming is not compatible with the following features: SR-IOV, RDMA and TCP chimney offload.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 19/112
Fabric Architecture Guide 19
Windows Server 2012 Feature Key ScenariosIncreased VP:LP ratio Removes previous limits of 8:1 processor ratios for server
workloads and 12:1 processor ratios for client workloads. Increased virtual memory and Dynamic Memory Supports up to 1 TB of memory inside virtual machines.
Virtual machine guest clustering enhancements(Fibre Channel)
Supports virtual machine guest clusters by using iSCSIconnections or by using the Hyper-V Fibre Channeladapter to connect to shared storage.
Hyper-V extensible switch Allows filtering, capturing, and forwarding of third-partyextensions to support additional virtual-switchfunctionality on the Hyper-V platform.
Encrypted cluster volumes Enables support for BitLocker-encrypted Cluster SharedVolumes version 2 (CSVv2).
Cluster-aware updating Provides the ability to apply updates to running failoverclusters through coordinated patching of individualfailover-cluster nodes.
Offloaded data transfer (ODX) Supports storage-level transfers that use ODX technology(SAN feature).
Support for SR-IOV network Allows virtual machines the virtual functions (VF) from SR-IOV-enabled physical network interface for increased IO.
Support for 4K physical disk Supports native 4K disk drives on hosts.
Diskless network boot with iSCSI Software Target Provides the network-boot capability on commodityhardware by using an iSCSI boot –capable network adapteror a software boot loader (such as iPXE or netBoot/i).
Quality of service (QoS) minimum bandwidth (fairshare)
Assigns a certain amount of bandwidth to a given type oftraffic and helps make sure that each type of networktraffic receives up to its assigned bandwidth.
Virtual machine storage enhancements (VHDX) Supports VHDX disks that are up to 64 TB in size.
Network Interface Card (NIC) teaming, load-balancing failover (LBFO) support
Supports switch-independent and switch-dependent loaddistribution by using physical and virtual networkconnections.
IPsec offload Supports network adapters that are equipped withhardware that reduces the load on the central processingunit (CPU) by performing this computationally intensivework.
Data center bridging Provides hardware support for converged fabrics, whichallows bandwidth allocation and priority flow control.
Low-latency workloads technologies Contains applications that require very fast inter-processcommunication (IPC) and inter-computercommunications, a high degree of predictability regardinglatency and transaction response times, and the ability to
handle very high message rates.
Table 7 Windows Server 2012 features and key scenarios
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 20/112
20 Fabric Architecture Guide
6 Storage Architecture6.1.1 Disk ArchitectureThe type of hard drives in the host server or in the storage array that are used by the host of file servers
will have the most significant impact on the overall performance of the storage architecture. The critical
performance factors for hard disks are:
The interface architecture (for example, SAS or SATA) The rotational speed of the drive (for example, 10K or 15K RPM) or a solid-state drive (SSD) that
does not have moving parts The read/write speed The average latency in milliseconds (ms)
Additional factors, such as the cache on the drive, and support for advanced features, such as Native
Command Queuing (NCQ) or TRIM, can improve performance and duration.
As with the storage connectivity, high input/output operations per second (IOPS) and low latency aremore critical than maximum sustained throughput when it comes to sizing and guest performance on the
Hyper-V server. During the selection of drives, this translates into selecting those that have the highest
rotational speed and lowest latency possible, and choosing when to use SSD or flash-based disks for
extreme performance.
6.1.1.1 Serial ATA (SATA)Serial ATA (SATA) drives are a low-cost and relatively high-performance option for storage. SATA
drives are available primarily in the 3-Gbps and 6-Gbps standards (SATA II and SATA III), with a
rotational speed of 7,200 RPM and average latency of around four milliseconds. Typically, SATA drives
are not designed to enterprise-level standards of reliability, although new technologies in Windows
Server 2012 such as a Resilient File System (ReFS) can help make SATA drives a viable option for
single server scenarios. However, SAS disks are required for all cluster and high availability scenarios
using Storage Spaces.
6.1.1.2 SASSAS drives are typically more expensive than SATA drives, but they can provide higher performance in
throughput and, more importantly, low latency. SAS drives typically have a rotational speed of 10k or
15k RPM with an average latency of 2 ms to 3 ms and 6 Gbps interfaces. There are also SAS SSDs.
Unlike SATA, there are SAS disks with dual interface ports that are required for using clustered storagespaces. (Details are provided in subsequent sections.) The SCSI Trade Association has a range of
information about SAS. In addition, several white papers and solutions can be found on the LSI
website .
The primary scenario for SAS drives that are used in the Continuous Availability pattern is in
conjunction with a JBOD storage enclosure, which enables the Storage Spaces feature. Aside from the
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 21/112
Fabric Architecture Guide 21
enclosure requirements that will be outlined later, the following requirements exist for SAS drives
when used in this configuration:
Drives must provide port association . Windows depends on drive enclosures to provideSES-3 capabilities such as drive-slot identification and visual drive indications (commonly
implemented as drive LEDs). Windows matches a drive in an enclosure with SES-3identification capabilities through the port address of the drive. Computer hosts can beseparate from drive enclosures or integrated into drive enclosures.
Multiport drives must provide symmetric access . Drives must provide the sameperformance for data-access commands and the same behavior for persistent reservationcommands that arrive on different ports as they provide when those commands arrive on thesame port.
Drives must provide persistent reservations . Windows can use physical disks to form astorage pool. From the storage pool, Windows can define virtual disks, called storage spaces.A failover cluster can make the pool of physical disks, the storage spaces that they define, and
the data that they contain highly available. In addition to the standard HCT qualification,physical disks should pass through the Microsoft Cluster Configuration Validation Wizard(ClusPrep tool).
In addition to the drives, the following enclosure requirements exist:
Drive enclosures must provide drive-identification services. Drive enclosures mustprovide numerical (for example, drive bay number) and visual (for example, failure LED ordrive-of-interest LED) drive-identification services. Enclosures must provide this servicethrough SCSI Enclosure Service (SES-3) commands. Windows depends on proper behavior forthe following enclosure services. Windows correlates enclosure services to drives through
protocol-specific information and their vital product data page 83h inquiry association type 1. Drive enclosures must provide direct access to the drives that they house. Enclosures
must not abstract the drives that they house (for example, form into a logical RAID disk). Ifthey are present, integrated switches must provide discovery of and access to all of the drivesin the enclosure, without requiring additional physical host connections. If possible, multiplehost connections must provide discovery of and access to the same set of drives.
Hardware vendors should pay specific attention to these storage drive and enclosure requirements for
SAS configurations when used in conjunction with the Storage Spaces feature in Windows
Server 2012.
6.1.1.3 Fibre ChannelFibre Channel disks are traditionally used in SAN arrays and provide high speed, relatively low latency,
and enterprise-level reliability. Fibre Channel drives are usually more expensive than SATA and SAS
drives. Fibre Channel disk drives typically has performance characteristics that are similar to those of
SAS drives, but they use a different interface. The choice of Fibre Channel or SAS drives is usually
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 22/112
22 Fabric Architecture Guide
determined by the choice of storage array or disk tray. In many cases, SSDs (solid-state drives) can
also be used in SAN arrays that use Fibre Channel interfaces.
6.1.1.4 Solid-State StorageSolid-state storage has several advantages over traditional spinning media disks, but it comes at a
premium cost. The most prevalent type of solid-state storage used in a disk form-factor is a solid-state drive (SSD), which will be discussed here. Some advantages include significantly lower latency,
no spin-up time, faster transfer rates, lower power and cooling requirements, and no fragmentation
concerns.
Recent years have shown greater adoption of SSDs in enterprise storage markets. These more
expensive devices are usually reserved for workloads that have high-performance requirements.
Mixing SSDs with spinning disks in storage arrays is common to minimize cost. These storage arrays
often have software algorithms that automatically place the frequently accessed storage blocks on the
SSDs and the less frequently accessed blocks on the lower-cost disks (referred to as auto-tiering),
although manual segregation of disk pools is also acceptable. NAND Flash Memory is most
commonly used in SSDs for enterprise storage.
6.1.1.5 Hybrid DrivesHybrid drives combine traditional spinning disks with nonvolatile memory or small SSDs that act as a
large buffer. This method provides the potential benefits of solid-state storage with the cost
effectiveness of traditional disks. Currently, these disks are not commonly found in enterprise storage
arrays.
6.1.1.6 Advanced Format (4K) Disk CompatibilityWindows Server 2012 introduces support for large sector disks that support 4096-byte sectors
(referred to as 4K), rather than the traditional 512-byte sectors, which ship with most hard drives
today. This change offers higher capacity drives, better error correction, and more efficient signal-to-
noise ratios.
However, this change introduces compatibility challenges. To support compatibility, two types of 4K
drives exist – 512-byte emulation (512e) and 4K native. 512e drives present a 512-byte logical sector
to use as the unit of addressing, and they present a 4K physical sector to use as the unit of atomic
write (the unit defined by the completion of read and write operations in a single operation).
6.1.2 Storage Controller ArchitecturesFor servers that will be directly connected to storage devices or arrays (which could be Hyper-V host
servers or file servers that will present storage), the choice of storage controller architecture is critical
to performance, scale, and overall cost.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 23/112
Fabric Architecture Guide 23
6.1.2.1 SATA IIISATA III controllers can operate at speeds of up to six Gbps, and enterprise-oriented controllers can
include varying amounts of cache on the controller to improve performance. PCIe/SAS host bus
adapter (HBA).
SAS controllers can also operate up to six Gbps, and they are more common in server form factorsthan SATA. With Windows Server 2012, it is important to understand the difference between host bus
adapters (HBAs) and RAID controllers.
SAS HBAs provide direct access to the disks, trays, or arrays that are attached to the controller. There
is no controller-based RAID. Disk high availability is provided by the array or by the tray itself. In the
case of Storage Spaces, high availability is provided by higher-level software layers. SAS HBAs are
common in one, two, and four port models.
To support the Storage Spaces feature in Windows Server 2012, the HBA must report the physical bus
that is used to connected devices. For example, drives connected through the SAS bus is a validconfiguration, whereas drives as connected through the RAID bus is an invalid configuration.
All commands must be passed directly to the underlying physical devices. The physical devices must
not be abstracted (that is, formed into a logical RAID device), and the bus adapter must not respond
to commands on behalf of the physical devices.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 24/112
24 Fabric Architecture Guide
Figure 9 Example SAS JBOD Storage architecture
6.1.2.2 PCIe RAID/Clustered RAIDPeripheral Component Interconnect Express (PCIe) RAID controllers are the traditional cards that arefound in servers. They provide access to storage systems, and they can include on-board RAID
technology. RAID controllers are not typically used in cluster scenarios, because clustering requires
shared storage. If Storage Spaces is used, hardware RAID should not be enabled, because Storage
Spaces handles data availability and redundancy.
Cluster RAID controllers are a relatively new type of storage interface card that can be used with
shared storage and cluster scenarios because the RAID controllers across configured servers are able
to present shared storage. In this case, the clustered RAID controller solution must pass the Cluster in
a Box Validation Kit . This step is required to help make sure that the solution provides the appropriate
storage capabilities that are necessary for failover cluster environments.
6.1.2.3 Fibre Channel HBAFibre Channel HBAs are one of the more common connectivity methods for storage, particularly in
clustering and shared storage scenarios. Some HBAs include two or four ports, each of which ranges
from one Gbps to eight Gbps. Like Windows Server 2008 R2, Windows Server 2012 supports a large
SAS Disks
SAS HBA SAS HBA
Scale-Out File Server Cluster Node
Storage Spaces
Cluster Shared Volumes (CSV v2) + CSV Cache
VHDs
SASPort
SASPort
SASPort
SASPort
SAS HBA SAS HBA
SASPort
SASPort
SASPort
SASPort
VHDs
Scale-Out File Server Cluster Node
SAS Expander SAS Expander
SAS JBOD Array with Dual Expander/Dual Port Drives
Storage Pool(s)
10Gb-ERDMA Port
10Gb-ERDMA Port
10Gb-ERDMA Port
10Gb-ERDMA Port
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 25/112
Fabric Architecture Guide 25
number of logical unit numbers (LUNs) per HBA 2. The capacity is expected to exceed the needs of
customers for addressable LUNs in a SAN.
Hyper-V in Windows Server 2012 introduces the ability to support virtual Fibre Channel adapters
within a Hyper-V guest. Although this virtualization feature is outlined in later sections of this
document, it is important to understand that the HBA ports that are to be used with virtual FibreChannel should be set up in a Fibre Channel topology that supports N_Port ID Virtualization (NPIV),
and they should be connected to an NPIV-enabled SAN. To utilize this feature, the Fibre Channel
adapters must also support devices that present logical units.
6.1.3 Storage NetworkingA variety of storage-networking protocols and scenarios exist to support traditional SAN-based
scenarios, NAS scenarios, and the newer SMB/Continuous Availability scenarios.
6.1.3.1 Fibre Channel
Historically, Fibre Channel has been the storage protocol of choice for enterprise data centers for avariety of reasons, including performance and low latency. These considerations have offset the
typically higher costs of Fibre Channel. The continually advancing performance of Ethernet from one
Gbps to 10 Gbps and beyond has led to great interest in storage protocols that use Ethernet
transports, such as iSCSI and Fibre Channel over Ethernet (FCoE).
Given the long history of Fibre Channel in the data center, many organizations have a significant
investment in a Fibre Channel –based SAN infrastructure. Windows Server 2012 continues to provide
full support for Fibre Channel hardware that is logo-certified. There is also support for virtual Fibre
Channel in guest virtual machines through a Hyper- V feature in Windows Server 2012.
6.1.3.2 iSCSIIn Windows Server 2012, the iSCSI Software Target is available as a built-in option under the file and
storage service role instead of a separate downloadable add-on, so it is easier to deploy. The iSCSI
Software Target capabilities were enhanced to support diskless network-boot capabilities and similar
continuous availability configurations as those used by Continuous Availability SMB. This
demonstrates how the storage protocols in Windows Server 2012 are designed to complement each
other across all layers of the storage stack.
In Windows Server 2012, the iSCSI Software Target feature provides network-boot capability for
commodity hardware. The iSCSI Target Server enables network boot of up to 256 computers fromoperating system images that are stored in a centralized location. This capability does not require
special hardware, but it is recommended to be used in conjunction with 10 GbE adapters that support
iSCSI boot capabilities.
2 Using the following formula, Windows Server 2012 supports a total of 261,120 LUNs per HBA – (255 LUN ID per target) × (128 targets per bus) × (8 buses per adapter).
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 26/112
26 Fabric Architecture Guide
For Hyper-V, iSCSI-capable storage provides an advantage because it is the protocol that is utilized by
Hyper-V guest virtual machines for guest clustering.
6.1.3.3 Fibre Channel over Ethernet (FCoE)A key advantage of the use of an Ethernet transport by the protocols is the ability to use a converged
network architecture. Converged networks have an Ethernet infrastructure that serves as the transportfor LAN and storage traffic. This can reduce costs by eliminating dedicated Fibre Channel switches and
reducing cables.
Fibre Channel over Ethernet (FCoE) allows the potential benefits of using an Ethernet transport, while
retaining the advantages of the Fibre Channel protocol and the ability to use Fibre Channel storage
arrays.
Several enhancements to standard Ethernet are required for FCoE. The enriched Ethernet is commonly
referred to as enhanced Ethernet or Data Center Ethernet. These enhancements require Ethernet
switches that are capable of supporting enhanced Ethernet.6.1.3.4 InfiniBand
InfiniBand is an industry-standard specification that defines an input/output architecture that is used
to interconnect servers, communications infrastructure equipment, storage, and embedded systems.
InfiniBand is a true fabric architecture that utilizes switched, point-to-point channels with data
transfers of up to 120 gigabits per second (Gbps), in chassis backplane applications and through
external copper and optical fiber connections.
InfiniBand is a low-latency, high-bandwidth interconnect that requires low processing overhead. It is
ideal to carry multiple traffic types (clustering, communications, storage, management) over a single
connection.
6.1.3.5 Switched SASAlthough switched SAS is not traditionally viewed as a storage networking technology, it is possible to
design switched SAS storage infrastructures. In fact, this can be a low cost and powerful approach
when combined with Windows Server 2012 features, such as Storage Spaces and SMB 3.0.
SAS switches enable multiple host servers to be connected to multiple storage trays (SAS JBODs) and
multiple paths between each as shown in Figure 10. Multiple path SAS implementations use a single
domain method of providing fault tolerance. Current mainstream SAS hardware supports six Gbps.
SAS switches support ―domains‖ that enable functionality similar to zoning in Fibre Channel.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 27/112
Fabric Architecture Guide 27
Figure 10 SAS switch connected to multiple SAS JBOD arrays
6.1.3.6 Network File System (NFS)File-based storage is a practical alternative to more SAN storage because it is straightforward to
provision, and it has gained viability because it is simple to provision and manage. An example of this
trend is the popularity of deploying and running VMware ESX/ESXi virtual machines from file-based
storage accessed over the Network File System (NFS) protocol. To help you utilize this, Windows
Server 2012 includes an updated Server for NFS that supports NFS 4.1 and can utilize many other
performance, reliability, and availability enhancements that are available throughout the storage stack
in Windows.
Some of the key features that are available with NFS for Windows Server 2012 include:
Storage for VMware virtual machines over NFS. In Windows Server 2012, you can confidently
deploy the Windows NFS server as a highly available storage back end for VMware virtualmachines. Critical components of the NFS stack have been designed to provide transparentfailover semantics to NFS clients.
NFS 4.1 protocol. The NFS 4.1 protocol is a significant evolution, and Microsoft delivers astandards-compliant server-side implementation in Windows Server 2012. Some of the features ofNFS 4.1 include a flexible single-server namespace for easier share management, full Kerberos v5support (including authentication, integrity, and privacy) for enhanced security, VSS snapshotintegration for backup, and Unmapped UNIX User Access for easier user account integration.
SAS Disks
SAS HBA SAS HBA
Scale-Out File Server Cluster Node
Storage Spaces
Cluster Shared Volumes (CSV v2) + CSV Cache
VHDs
SASPort
SASPort
SASPort
SASPort
SAS HBA SAS HBA
SASPort
SASPort
SASPort
SASPort
VHDs
Scale-Out File Server Cluster Node
SAS Switch SAS Switch
SAS JBOD Array with DualExpander/Dual Port Drives
Storage Pool(s)
SASExpander
SASExpander
SAS Disks
SAS JBOD Array with DualExpander/Dual Port Drives
SASExpander
SASExpander
10Gb-ERDMA Port
10Gb-ERDMA Port
10Gb-ERDMA Port
10Gb-ERDMA Port
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 28/112
28 Fabric Architecture Guide
Windows Server 2012 supports simultaneous SMB 3.0 and NFS access to the same share, identitymapping by using stores based on RFC-2307 for easier and more secure identity integration, andhigh availability cluster deployments.
Windows PowerShell. In response to customer feedback, over 40 Windows PowerShell cmdletsprovide task-based remote management of every aspect of the NFS server —from the configuring
of NFS server settings to the provisioning of shares and share permissions. Simplified identity mapping. Windows Server 2012 includes a flat file –based identity-mapping
store. Windows PowerShell cmdlets replace cumbersome manual steps to provision ActiveDirectory Lightweight Directory Services (AD LDS) as an identity-mapping store and to managemapped identities.
6.1.3.7 SMB 3.0File-based storage is a practical alternative to more SAN storage because it is straightforward to
provision, and it has gained viability because it is simple to provision and manage. An example of this
trend is the popularity of deploying and running VMware ESX/ESXi virtual machines from file-based
storage accessed over the Network File System (NFS) protocol.
Figure 11 Example of a SMB 3.0-enabled NAS
SMB Direct (SMB over RDMA)6.1.3.7.1
The SMB protocol in Windows Server 2012 includes support for RDMA network adapters, whichallows storage-performance capabilities that rival Fibre Channel. RDMA network adapters enable this
performance capability by operating at full speed with very low latency because of their ability to
bypass the kernel and perform write and read operations directly to and from memory. This capability
is possible because effective transport protocols are implemented on the adapter hardware and allow
for zero-copy networking by using kernel bypass.
By using this capability, applications (including SMB) can perform data transfers directly from
memory, through the adapter, to the network, and then to the memory of the application that is
requesting data from the file share. This means two kernel calls —one from the server and one from
the client —are largely removed from the data transfer process, resulting in greatly improved datatransfer performance. This capability is especially useful for read and write intensive workloads, such
as in Hyper-V or Microsoft SQL Server, and it results in remote file server performance that is
comparable to local storage.
SMB Direct requires:
SAS/SSD Disks
10Gb-E RDMAPort
10Gb-E RDMAPort
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 29/112
Fabric Architecture Guide 29
At least two computers running Windows Server 2012. No additional features have to be
installed, and the technology is available by default.
Network adapters that are RDMA-capable. SMB Direct supports common RDMA-capable
adapter types, including Internet Wide Area RDMA Protocol (iWARP), InfiniBand, and RDMA
over Converged Ethernet (RoCE).
SMB Direct works in conjunction with SMB Multichannel to transparently provide a combination of
exceptional performance and failover resiliency when multiple RDMA links between clients and SMB
file servers are detected. In addition, because RDMA bypasses the kernel stack, RDMA does not work
with NIC Teaming; however, it does work with SMB Multichannel, because SMB Multichannel is
enabled at the application layer.
SMB Multichannel6.1.3.7.2
SMB 3.0 protocol in Windows Server 2012 includes SMB Multichannel, which provides scalable and
resilient connections to SMB shares that dynamically create multiple connections for single sessionsor multiple sessions on single connections, depending on connection capabilities and current
demand. This capability to create flexible session to connection associations gives SMB a number of
key features:
Connection resiliency : With the ability to dynamically associate multiple connections with a
single session, SMB gains resiliency against connection failures that are usually caused by
network interfaces or components. SMB Multichannel also allows clients to actively manage
paths of similar network capability in a failover configuration that automatically switches
sessions to the available paths if one path becomes unresponsive.
Network usage : SMB can utilize receive-side scaling (RSS) –capable network interfaces alongwith the multiple connection capability of SMB Multichannel to fully use high-bandwidth
connections, such as those that are available on 10 GbE networks, during read and write
operations with workloads that are evenly distributed across multiple CPUs.
Load balancing : Clients can adapt to changing network conditions to rebalance loads
dynamically to a connection or across a set of connections that are more responsive when
congestion or other performance issues occur.
Transport flexibility : Because SMB Multichannel also supports single session to multiple
connection capabilities, SMB clients are flexible enough to adjust dynamically when newnetwork interfaces become active. This is how SMB Multichannel is automatically enabled
whenever multiple UNC paths are detected and can grow dynamically to use multiple paths
as more are added, without administrator intervention.
SMB Multichannel has the following requirements, which are organized by how SMB Multichannel
prioritizes connections when multiple connection types are available:
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 30/112
30 Fabric Architecture Guide
RDMA-capable network connections : SMB Multichannel can be used with a single
InfiniBand connection on the client and server sides or with a dual InfiniBand connection on
each server, connected to different subnets. Although SMB Multichannel offers scaling
performance enhancements in single adapter scenarios through RDMA and RSS, if available, it
cannot supply failover and load balancing capabilities without multiple paths. RDMA-capable
adapters include iWARP, InfiniBand, and RoCE.
RSS-capable network connections : SMB Multichannel can utilize RSS-capable connections
in 1-1 connection scenarios or multi-connection scenarios. As mentioned, multichannel load
balancing and failover capabilities are not available unless multiple paths exist, but it does
utilize RSS to provide scaling performance usage by spreading overhead between multiple
processors by using RSS-capable hardware.
Load balancing and failover (LBFO) or aggregate interfaces : When RDMA or RSS
connections are not available, SMB prioritizes connections that use a collection of two or
more physical interfaces. This requires more than one network interface on the client andserver, where both are configured as a network adapter team. In this scenario, load balancing
and failover are the responsibility of the teaming protocol, not SMB Multichannel, when only
one NIC Teaming connection is present and no other connection path is available.
Standard interfaces and Hyper-V virtual networks : These connection types can use SMB
Multichannel capabilities but only when multiple paths exist. For all practical intent, one GB
Ethernet connection is the lowest priority connection type that is capable of using SMB
Multichannel.
Wireless network interfaces : Wireless interfaces are not capable of multichannel operations.
When connections are not similar between client and server, SMB Multichannel will utilize available
connections when multiple connection paths exist. For example, if the SMB file server has a 10 GbE
connection, but the client has only four 1 GbE connections, and each connection forms a path to the
file server, then SMB Multichannel can create connections on each 1 GbE interface. This provides
better performance and resiliency, even though the network capabilities of the server exceed the
network capabilities of the client.
SMB Transparent Failover6.1.3.7.3
SMB Transparent Failover helps administrators configure file shares in Windows failover cluster
configurations so that they are continuously available. The use of continuously available file shares
enables administrators to perform hardware or software maintenance on any cluster node without
interrupting the server applications that are storing their data files on these file shares.
In case of a hardware or software failure, the server application nodes transparently reconnect to
another cluster node without interruption of the server application I/O operations. By using an SMB
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 31/112
Fabric Architecture Guide 31
scale-out file share, SMB Transparent Failover allows the administrator to redirect a server application
node to a different file-server cluster node to facilitate better load balancing.
SMB Transparent Failover has the following requirements:
A failover cluster that is running Windows Server 2012 with at least two nodes. Theconfiguration of servers, storage, and networking must pass all of the tests performed in theCluster Configuration Validation Wizard.
File Server role installed on all cluster nodes. Clustered file server configured with one or more file shares created with the continuously
available property. Client computers running the Windows 8 client or Windows Server 2012.
To realize the potential benefits of the SMB Transparent Failover feature, the SMB client computer
and the SMB server must support SMB 3.0, which was introduced in Windows 8 and Windows
Server 2012. Computers running down-level SMB versions, such as SMB 1.0, SMB 2.0, or SMB 2.1 can
connect and access data on a file share that has the continuously available property set, but they willnot be able to realize the potential benefits of the SMB Transparent Failover feature.
SMB Encryption6.1.3.7.4
SMB Encryption protects incoming data from unintentional snooping threats on untrusted networks,
with no additional setup requirements. SMB 3.0 in Windows Server 2012 secures data transfers by
encrypting incoming data, to protect against tampering and eavesdropping attacks. The biggest
potential benefit of using SMB Encryption instead of general solutions (such as Internet Protocol
security [IPsec]) is that there are no deployment requirements or costs beyond changing the SMB
server settings. The encryption algorithm that is used is AES-CCM, which also provides data-integrity
validation (signing).
SMB 3.0 uses a newer algorithm (AES-CMAC) for signing, instead of the HMAC-SHA256 algorithm
that SMB 2.0 uses. AES-CCM and AES-CMAC can be dramatically accelerated on most modern CPUs
that have AES instruction support.
By using Windows Server 2012, an administrator can enable SMB Encryption for the entire server, or
just specific shares. Because there are no other deployment requirements for SMB Encryption, it is an
extremely cost effective way to protect data from snooping and tampering attacks. Administrators can
turn it on simply by using the File Server Manager or Windows PowerShell.
Volume Shadow Copy Support (VSS)6.1.3.7.5
Volume Shadow Copy Service (VSS) is a framework that enables volume backups to run while
applications on a system continue to write to the volumes. A new feature called ―VSS for SMB File
Shares‖ was introduced in Windows Server 2012 to support applications that store their data files on
remote SMB file shares. This feature enables VSS-aware backup applications to perform application
consistent shadow copies of VSS-aware server applications that store data on SMB 3.0 file shares.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 32/112
32 Fabric Architecture Guide
Prior to this feature, VSS supported only the performance of shadow copies of data that was stored
on local volumes.
6.1.3.8 SMB Scale-OutOne the main advantages of file storage over block storage is the ease of configuration, paired with
the ability to configure folders that can be shared by multiple clients. SMB takes this one step fartherby introducing the SMB Scale-Out feature, which provides the ability to share the same folders from
multiple nodes of the same cluster. This is made possible by the use of the Cluster Shared Volumes
(CSV) feature, which in Windows Server 2012 supports file sharing.
For example, if you have a four-node file-server cluster that uses SMB Scale-Out, an SMB client will be
able to access the share from any of the four nodes. This active-active configuration lets you balance
the load across cluster nodes by allowing an administrator to move clients without any service
interruption. This means that the maximum file-serving capacity for a given share is no longer limited
by the capacity of a single cluster node.
SMB Scale-Out also helps keep configurations simple, because a share is configured only once to be
consistently available from all nodes of the cluster. Additionally, SMB Scale-Out simplifies
administration by not requiring cluster virtual IP addresses or by creating multiple cluster file-server
resources to utilize all cluster nodes.
SMB Scale-Out requires:
A failover cluster that is running Windows Server 2012 with at least two nodes. The clustermust pass the tests in the Cluster Configuration Validation Wizard.
File shares that are created on a Cluster Shared Volume with the high-availability property.
This is the default setting. Computers running Windows 8 or Windows Server 2012. Computers must include the
updated SMB client that supports high availability.
6.1.4 Windows File Services6.1.4.1 Storage Spaces
Storage Spaces introduces a new class of sophisticated storage virtualization enhancements to the
storage stack that incorporates two new concepts:
Storage pools : Virtualized units of administration that are aggregates of physical disk units.Pools enable storage aggregation, elastic capacity expansion, and delegated administration.
Storage spaces : Virtual disks with associated attributes that include a desired level ofresiliency, thin or fixed provisioning, automatic or controlled allocation on diverse storagemedia, and precise administrative control.
The Storage Spaces feature in Windows Server 2012 can utilize failover clustering for high availability,
and it can be integrated with CSV for scalable deployments.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 33/112
Fabric Architecture Guide 33
Figure 12 CSV v2 can be integrated with Storage Spaces
The features that Storage Spaces includes are:
Storage pooling : Storage pools are the fundamental building blocks for Storage Spaces. ITadministrators can flexibly create storage pools, based on the needs of the deployment. For
example, given a set of physical disks, an administrator can create one pool by using all of thephysical disks that are available or multiple pools by dividing the physical disks as required. Inaddition, to promote the value from storage hardware, the administrator can map a storagepool to combinations of hard disks in addition to solid-state drives (SSDs). Pools can beexpanded dynamically simply by adding more drives, thereby seamlessly scaling to cope withincreasing data growth as needed.
Multitenancy : Administration of storage pools can be controlled through access control lists(ACLs) and delegated on a per-pool basis, thereby supporting hosting scenarios that requiretenant isolation. Storage Spaces follows the familiar Windows security model; therefore, it canbe integrated fully with Active Directory Domain Services (AD DS).
Resilient storage : Storage Spaces supports two optional resiliency modes: mirroring andparity. Features such as per-pool hot spare support, background scrubbing, and intelligenterror correction enable optimal service availability despite storage component failures.
Continuous availability through integration with failover clustering : Storage Spaces isfully integrated with failover clustering to deliver continuously available service deployments.One or more pools can be clustered across multiple nodes in a single cluster. Storage Spacescan then be instantiated on individual nodes and will seamlessly migrate or fail over to adifferent node when necessary, either in response to failure conditions or because of loadbalancing. Integration with CSV 2.0 enables scalable access to data on storage clusters.
Optimal storage use : Server consolidation frequently results in multiple datasets sharing the
same storage hardware. Storage Spaces supports thin provisioning to enable businesses toeasily share storage capacity among multiple unrelated datasets, thereby promoting capacityuse. Trim support enables capacity reclamation when possible.
Operational simplicity : Fully scriptable remote management is permitted through theWindows Storage Management API, Windows Management Instrumentation (WMI), andWindows PowerShell. Storage Spaces can be managed easily through the File Services GUI inServer Manager or by using task automation with many new Windows PowerShell cmdlets.
Storage Spaces
Cluster Shared Volumes (CSV v2)
Storage Pool(s)
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 34/112
34 Fabric Architecture Guide
For single-node environments, Windows Server 2012 requires the following:
Serial or SAS-connected disks (in an optional JBOD enclosure)
For multi-server and multi-site environments, Windows Server 2012 requires the following:
Any requirements that are specified for Windows failover clustering and Windows CSVversion 2
Three or more SAS-connected disks (JBODs) to encourage compliance with WindowsCertification requirements
6.1.4.2 Resilient File System (ReFS)Windows Server 2012 introduces a new local file system called Resilient File System (ReFS). ReFS
promotes data availability and online operation, despite errors that would historically cause data loss
or downtime. Data integrity helps protect business-critical data from errors and helps make sure that
the data is available when needed. ReFS architecture provides scalability and performance in an era of
constantly growing dataset sizes and dynamic workloads.
ReFS was designed with three key goals in mind:
Maintain the highest possible levels of system availability and reliability, under theassumption that the underlying storage might be unreliable.
Provide a full end-to-end resilient architecture when it is used in conjunction with StorageSpaces, so that these two features magnify the capabilities and reliability of one another whenthey are used together.
Maintain compatibility with widely adopted and successful NTFS features, while replacingfeatures that provide limited value.
6.1.4.3 NTFS ImprovementsIn Windows Server 2012, NTFS has been enhanced to maintain data integrity when using cost-
effective industry-standard SATA drives. NTFS also provides online corruption scanning and repair
capabilities that reduce the need to take volumes offline. When they are combined, these capabilities
let you deploy very large NTFS volumes with confidence.
Two key enhancements have been made to NTFS in Windows Server 2012. The first one targets the
need to maintain data integrity in inexpensive commodity storage. This has been accomplished by
enhancing NTFS to rely only on the flush command instead of ―forced unit access‖ for all operations
that require write ordering. This improves resiliency against metadata inconsistencies that are causedby unexpected power loss. This means that you can more safely use cost-effective industry-standard
SATA drives.
NTFS availability is the focus of the second key enhancement, and this is achieved through a
combination of features, which include:
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 35/112
Fabric Architecture Guide 35
Online corruption scanning : Windows Server 2012 performs online corruption scanningoperations as a background operation on NTFS volumes. This scanning operation identifiesand corrects areas of data corruption if they occur, and it includes logic that distinguishesbetween transient conditions and actual data corruption, which reduces the need for CHKDSKoperations.
Improved self-healing : To further improve resiliency and availability, Windows Server 2012significantly increases online self-healing to resolve many issues on NTFS volumes without theneed to take the volume offline to run CHKDSK.
Reduced repair times : In the rare case of data corruption that cannot be fixed with onlineself-healing, administrators are notified that data corruption has occurred, and they canchoose when to take the volume offline for a CHKDSK operation. Furthermore, because of theonline corruption-scanning capability, CHKDSK scans and repairs only tagged areas of datacorruption. Because it does not have to scan the whole volume, the time that is necessary toperform an offline repair is greatly reduced. In most cases, repairs that would have takenhours on volumes that contain a large number of files now take seconds.
6.1.4.4 Scale-Out File Server Cluster ArchitectureIn Windows Server 2012, the following clustered file-server types are available:
Scale-Out File Server cluster for application data : This clustered file server lets you storeserver application data (such as virtual machine files in Hyper-V) on file shares, and obtain asimilar level of reliability, availability, manageability, and high performance that you wouldexpect from a storage area network. All file shares are online on all nodes simultaneously. Fileshares that are associated with this type of clustered file server are called scale-out file shares.This is sometimes referred to as active-active.
File Server for general use : This is the continuation of the clustered file server that has beensupported in Windows Server since the introduction of failover clustering. This type ofclustered file server, and thus all of the shares that are associated with the clustered fileserver, is online on one node at a time. This is sometimes referred to as active-passive ordual-active. File shares that are associated with this type of clustered file server are calledclustered file shares.
In Windows Server 2012, Scale-Out File Server cluster is designed to provide scale-out file shares that
are continuously available for file-based server application storage. Scale-out file shares provide the
ability to share the same folder from multiple nodes of the same cluster. For instance, if you have a
four-node file-server cluster that is using Server Message Block (SMB) Scale-Out, which is introduced
in Windows Server 2012, a computer that is running Windows Server 2012 can access file shares from
any of the four nodes. This is achieved by utilizing new Windows Server 2012 failover-clustering
features and new capabilities in SMB 3.0.
File-server administrators can provide scale-out file shares and continuously available file services to
server applications and respond to increased demands quickly by bringing more servers online. All of
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 36/112
36 Fabric Architecture Guide
this can be done in a production environment, and it is completely transparent to the server
application.
Key potential benefits that are provided by Scale-Out File Server cluster in Windows Server 2012include:
Active-active file shares. All cluster nodes can accept and serve SMB client requests. Bymaking the file-share content accessible through all cluster nodes simultaneously, SMB 3.0clusters and clients cooperate to provide transparent failover to alternative cluster nodesduring planned maintenance and unplanned failures without service interruption.
Increased bandwidth. The maximum share bandwidth is the total bandwidth of all file-servercluster nodes. Unlike in previous versions of Windows Server, the total bandwidth is no longerconstrained to the bandwidth of a single cluster node, but instead to the capability of thebacking storage system. You can increase the total bandwidth by adding nodes.
CHKDSK with zero downtime. CHKDSK in Windows Server 2012 is significantly enhanced todramatically shorten the time a file system is offline for repair. Clustered shared volumes(CSVs) in Windows Server 2012 take this one step further and eliminate the offline phase. ACSV File System (CSVFS) can perform CHKDSK without affecting applications that have openhandles on the file system.
Clustered Shared Volume cache. CSVs in Windows Server 2012 introduce support for a readcache, which can significantly improve performance in certain scenarios, such as a VirtualDesktop Infrastructure.
Simplified management. With Scale-Out File Server clusters, you create the Scale-Out FileServer cluster and then add the necessary CSVs and file shares. It is no longer necessary tocreate multiple clustered file servers, each with separate cluster disks, and then developplacement policies to confirm activity on each cluster node.
6.1.5 SAN and Storage Array Features6.1.5.1 Data Deduplication
Fibre Channel and iSCSI SANs often provide data deduplication functionality. By using the data
deduplication feature in Windows Server 2012, organizations can significantly improve the efficiency
of storage capacity usage. In Windows Server 2012, data deduplication provides the following
features:
Capacity optimization : Data deduplication lets you store more data in less physical space.You can achieve significantly better storage efficiency than was previously possible with
Single Instance Storage (SIS) or New Technology File System (NTFS) compression. Datadeduplication uses variable size chunking and compression, which together deliveroptimization ratios of up to 2:1 for general file servers and up to 20:1 for VHD libraries.
Scalability and performance : Data deduplication is highly scalable, resource-efficient, andnon-intrusive. It can run on dozens of large volumes of primary data simultaneously, withoutaffecting other workloads on the server.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 37/112
Fabric Architecture Guide 37
Reliability and data integrity : When you apply data deduplication, you must maintain dataintegrity. To help with data integrity, Windows Server 2012 utilizes checksum, consistency,and identity validation. In addition, to recover data in the event of corruption, WindowsServer 2012 maintains redundancy for all metadata and the most frequently referenced data.
Bandwidth efficiency alongside BranchCache : Through integration with BranchCache, the
same optimization techniques that are applied to improving data storage efficiency on thedisk are applied to transferring data over the WAN to a branch office. This integration resultsin faster file download times and reduced bandwidth consumption.
6.1.5.2 Thin Provisioning and TrimLike data deduplication, thin-provisioning technology improves the efficiency of how we use and
provision storage. Instead of removing redundant data on the volume, thin provisioning gains
efficiencies by making it possible to allocate just enough storage at the moment of storage allocation,
and then increase capacity as your business needs grow over time. Windows Server 2012 provides full
support for thinly provisioned storage arrays, which lets you get the most out of your storage
infrastructure. These sophisticated storage solutions offer just-in- time allocations, known as ―thin
provisioning,‖ and the ability to reclaim storage that is no longer needed, known as ―trim.‖
6.1.5.3 Volume CloningVolume cloning is another common practice in virtualization environments. Volume cloning can be
used for host and virtual machine volumes to improve host and virtual machine provisioning times
dramatically.
6.1.5.4 Volume SnapshotSAN volume snapshots are a common method of providing a point-in-time, instantaneous backup of
a SAN volume or LUN. These snapshots are typically block-level, and they only utilize storage capacity
as blocks change on the originating volume. Some SANs provide tight integration with Hyper-V and
integrate the Hyper-V VSS Writer on hosts and volume snapshots on the SAN. This integration
provides a comprehensive and high-performing backup and recovery solution.
6.1.5.5 Storage TieringStorage tiering is the practice of physically partitioning data into multiple distinct classes such as price
or performance. Data can be dynamically moved among classes in a tiered storage implementation,
based on access, activity, or other considerations.
Storage tiering is normally achieved through a combination of varying types of disks that are used for
different data types (for example, production, non-production, or backups). The following is an
example of storage tiering for a high I/O application like Microsoft Exchange Server.
6.1.6 Storage Management and AutomationWindows Server 2012 introduces a new, unified interface that uses WMI for comprehensive
management of physical and virtual storage, including third-party intelligent storage subsystems. The
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 38/112
38 Fabric Architecture Guide
unified interface uses WMI to provide a rich experience to IT professionals and developers by using
Windows PowerShell scripting, to help make a diverse set of solutions available.
Management applications can use a single Windows API to manage different storage types by using
SMP or standards-based protocols such as SMI-S.
The unified interface for storage management provides a core set of defined WMI and Windows
PowerShell interfaces, in addition to features for more advanced management. Figure 13 shows the
unified storage management architecture.
Figure 13 Unified storage-management architecture
The unified interface is a powerful and consistent mechanism for managing storage, which can reduce
complexity and operational costs. The storage interface provides capabilities for advanced
management of storage in addition to the core set of defined WMI and Windows PowerShell
interfaces.
6.1.6.1 SMS-SWindows Server 2012 enables storage management that is comprehensive and fully scriptable, and
administrators can manage it remotely. A WMI-based interface provides a single mechanism through
which to manage all storage, including non-Microsoft intelligent storage subsystems and virtualized
local storage (known as Storage Spaces). Additionally, management applications can use a singleWindows API to manage different storage types by using standards-based protocols such as Storage
Management Initiative Specification (SMI-S).
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 39/112
Fabric Architecture Guide 39
Figure 14 Windows Server management, storage management APIs and provider layers
For WMI-based services, the interface includes a rich set of built-in storage management features that
ISVs might find particularly helpful.
6.1.6.2 ODXWhenever possible, the speed of your virtualization platform should rival that of physical hardware.
Offloaded data transfer (ODX) support is a feature of the storage stack of Hyper-V in Windows
Server 2012. When used with offload-capable SAN storage hardware, ODX lets a storage device
perform a file copy operation without the main processor of the Hyper-V host actually reading the
content from one storage place and writing it to another.
Offloaded data transfer (ODX) enables rapid provisioning and migration of virtual machines, and it
provides significantly faster transfers of large files, such as database or video files. By offloading the
file transfer to the storage array, ODX minimizes latencies, promotes the use of array throughput, and
reduces host resource usage such as central processing unit (CPU) and network consumption. File
transfers are automatically and transparently offloaded when you move or copy files, regardless of
whether you perform drag-and-drop operations in Windows Explorer or use command-line file copy
commands. No administrator setup or intervention is necessary.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 40/112
40 Fabric Architecture Guide
7 Network ArchitectureA variety of designs and new approaches to data center networks have emerged in recent years. The
objective in most cases is to improve resiliency and performance while optimizing for highly
virtualized environments.
7.1.1 Network Architecture Patterns7.1.1.1 Hierarchical
Many network architectures include a hierarchical design with three or more tiers such as Core,
Distribution/Aggregation, and Access. Designs are driven by the port bandwidth and quantity that are
required at the edge, in addition to the ability of the distribution/aggregation and core tiers to
provide higher speed uplinks to aggregate traffic. Additional considerations include Ethernet
broadcast boundaries and limitations, and spanning tree and other loop-avoidance technologies.
Core
The core tier is the high-speed backbone for the network architecture. The core typically comprises
two modular-switch chassis to provide a variety of service and interface module options. The data-
enter core tier might interface with other network modules.
Aggregation
The aggregation (or distribution) tier consolidates connectivity from multiple access tier switch
uplinks. This tier is commonly implemented in end-of-row switches, a centralized wiring closet, or
main distribution frame (MDF) room. The aggregation tier provides high-speed switching and more
advanced features, like Layer 3 routing and other policy-based networking capabilities. The
aggregation tier must have redundant, high-speed uplinks to the core tier for high availability.
Access
The access tier provides device connectivity to the data center network. This tier is commonly
implemented by using Layer 2 Ethernet switches —typically through blade chassis switch modules or
top-of-rack (ToR) switches. The access tier must provide redundant connectivity for devices, required
port features, and adequate capacity for access (device) ports and uplink ports.
The access tier can also provide features that are related to NIC Teaming, like link aggregation control
protocol (LACP). Certain teaming solutions might require LACP switch features.
Figure 15 illustrates two three-tier network models, one providing 10 GbE to devices and the otherproviding 1 GbE to devices.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 41/112
Fabric Architecture Guide 41
Figure 15 Comparative of 10 Gb and 1 Gb Ethernet Edge topology
7.1.1.2 Flat NetworkA flat network topology is adequate for very small networks. In a flat network design, there is no
hierarchy. Each internetworking device has essentially the same job, and the network is not divided
into layers or modules. A flat network topology is easy to design and implement, and it is easy to
maintain, as long as the network stays small. When the network grows, however, a flat network isundesirable. The lack of hierarchy makes troubleshooting difficult —instead of being able to
concentrate troubleshooting efforts in just one area of the network, you might have to inspect the
entire network.
7.1.1.3 Network Virtualization (Software-Defined Networking)Hyper-V network virtualization provides the concept of a virtual network that is independent of the
underlying physical network. With this concept of virtual networks, which are composed of one or
more virtual subnets, the exact physical location of an IP subnet is decoupled from the virtual network
topology. As a result, customers can easily move their subnets to the cloud while preserving their
existing IP addresses and topology in the cloud, so that existing services continue to work unaware of
the physical location of the subnets.
Hyper-V network virtualization in Windows Server 2012 provides policy-based, software-controlled
network virtualization that reduces the management overhead that enterprises face when they
expand dedicated infrastructure-as-a-service (IaaS) clouds. In addition, it provides cloud hosting
Team
2 x 10 GbEthernet
Links
4 x 10 GbEthernet
Links
10 GbEthernet
Links
Mgmt VLAN/vNICiSCSI VLAN/vNICCSV VLAN/vNICLM VLAN/vNICVM VLAN/NIC
Top of Rack orBlade Modules
2 x 10 GbEthernet
Links
4 x 10 GbEthernet
Links
1 GbEthernet
Links
Top of Rack orBlade Modules
Mgmt iSCSI CSV LM VM VMiSCSI
Team
10 Gb Ethernet to the Edge 1 Gb Ethernet to the Edge
CoreSwitches
AggregationSwitches
AccessSwitches
VLAN(s)
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 42/112
42 Fabric Architecture Guide
providers with better flexibility and scalability for managing virtual machines to achieve higher
resource utilization.
7.1.2 Network Performance and Low Latency7.1.2.1 Data Center Bridging
Separate isolated connections for network, live migration, and management traffic make managingnetwork switches and other networking infrastructure a challenge. As data centers evolve, IT
organizations look to some of the latest innovations in networking to help solve these issues. The
introduction of 10 GbE networks, for example, helps support converged networks that can handle
network, storage, live migration, and management traffic through a single connection, reducing the
requirements and costs of IT management.
Data center bridging (DCB) refers to enhancements to Ethernet LANs that are used in data center
environments. These enhancements consolidate the various forms of network into a single
technology, known as a converged network adapter (CNA). In the virtualized environment, Hyper-V in
Windows Server 2012 can utilize DCB-capable hardware to converge multiple types of network traffic
on a single network adapter, with a maximum level of service to each.
DCB is a hardware mechanism that classifies and dispatches network traffic that depends on DCB
support on the network adapter, supporting far fewer traffic flows. It converges different types of
traffic, including network, storage, management, and live migration traffic. However, it also can
classify network traffic that does not originate from the networking stack.
7.1.2.2 Virtual Machine Queue (VMQ)The virtual machine queue (VMQ) feature allows the network adapter of the host to pass DMA
packets directly into individual virtual machine memory stacks. Each virtual machine device buffer isassigned a VMQ, which avoids needless packet copies and route lookups in the virtual switch.
Essentially, VMQ allows the single network adapter of the host to appear as multiple network
adapters to the virtual machines, to allow each virtual machine its own dedicated network adapter.
The result is less data in the buffers of the host and an overall performance improvement in I/O
operations.
The VMQ is a hardware virtualization technology that is used for the efficient transfer of network
traffic to a virtualized host operating system. A VMQ-capable network adapter classifies incoming
frames to be routed to a receive queue, based on filters that associate the queue with the virtual
network adapter of a virtual machine. These hardware queues can have affinities to different CPUs, toallow for receive scaling on a per –virtual machine network adapter basis.
Windows Server 2012 dynamically distributes the processing of incoming network traffic to host
processors, based on processor use and network load. In times of heavy network load, Dynamic VMQ
(D-VMQ) automatically uses more processors. In times of light network load, D-VMQ relinquishes
those same processors.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 43/112
Fabric Architecture Guide 43
D-VMQ requires hardware network adapters and drivers that support Network Device Interface
Specification (NDIS) 6.30.
7.1.2.3 IPsec Task OffloadIPsec protects network communication by authenticating and encrypting some or all of the content of
network packets. IPsec Task Offload in Windows Server 2012 utilizes the hardware capabilities ofserver network adapters to offload IPsec processing. This reduces the CPU overhead of IPsec
encryption and decryption significantly.
In Windows Server 2012, IPsec Task Offload is extended to virtual machines. Customers who use
virtual machines and want to help protect their network traffic by using IPsec can utilize the IPsec
hardware offload capability that is available in server network adapters. Doing so frees up CPU cycles
to perform more application-level work and leaves the per-packet encryption and decryption to
hardware.
7.1.2.4 Quality of Service (QoS)For most deployments, one or two 10 GbE network adapters should provide enough bandwidth for all
the workloads on a Hyper-V server. However, 10 GbE network adapters and switches are considerably
more expensive than the 1 GbE counterparts. To optimize the 10 GbE hardware, a Hyper-V server
requires new capabilities to manage bandwidth.
Windows Server 2012 expands the power of the quality of service (QoS) by introducing the ability to
assign a minimum bandwidth to a virtual machine or service. This feature is important for service
providers and companies that honor SLA clauses that promise a minimum network bandwidth to
customers. It is equally important to enterprises that require predictable network performance when
they run virtualized server workloads on shared hardware.
In addition to the ability to enforce maximum bandwidth, QoS in Windows Server 2012 introduces a
new bandwidth management feature: minimum bandwidth. Unlike maximum bandwidth, which is a
bandwidth cap, minimum bandwidth is a bandwidth floor, and it assigns a certain amount of
bandwidth to a given type of traffic.
7.1.2.5 Remote Direct Memory AccessSMB Direct (SMB over remote direct memory access [RDMA]) is a new storage protocol in Windows
Server 2012. It enables direct memory-to-memory data transfers between server and storage, with
minimal CPU usage, while using standard RDMA-capable network adapters. SMB Direct is supportedon three types of RDMA technology: iWARP, InfiniBand, and RoCE.
7.1.2.6 Receive Segment CoalescingReceive segment coalescing (RSC) improves the scalability of the servers by reducing the overhead for
processing a large amount of network I/O traffic. It accomplishes this by coalescing multiple inbound
packets into a large buffer.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 44/112
44 Fabric Architecture Guide
7.1.2.7 Receive-Side ScalingReceive-side scaling (RSS) spreads monitoring interrupts over multiple processors, so a single
processor is not required to handle all I/O interrupts, which was common with earlier versions of
Windows Server. Active load balancing between the processors tracks the load on the CPUs and then
transfers the interrupts as necessary.You can select which processors will be used for handling RSS requests, including processors that are
beyond 64 KB, which allows you to utilize very high-end computers that have a large number of
logical processors.
RSS works with NIC Teaming to remove a limitation in earlier versions of Windows Server, where a
choice had to be made between the use of hardware drivers or RSS. RSS will also work for User
Datagram Protocol (UDP) traffic, and it can manage and debug applications that use WMI and
Windows PowerShell.
7.1.2.8 SR-IOVThe SR-IOV standard was introduced by the PCI-SIG, the special interest group that owns and
manages PCI specifications as open industry standards. SR-IOV works in conjunction with system
support for virtualization technologies that provides remapping of interrupts and DMA, and it lets SR-
IOV–capable devices be assigned directly to a virtual machine.
Hyper-V in Windows Server 2012 enables support for SR-IOV –capable network devices and allows
directly assignment of an SR-IOV virtual function of a physical network adapter to a virtual machine.
This increases network throughput and reduces network latency, while reducing the host CPU
overhead that is required for processing network traffic.
7.1.2.9 TCP Chimney OffloadThe TCP chimney architecture offloads the data transfer portion of TCP protocol processing for one or
more TCP connections to a network adapter. This architecture provides a direct connection, called a
chimney, between applications and an offload-capable network adapter.
The chimney offload architecture reduces host network processing for network-intensive applications,
so networked applications scale more efficiently and end-to-end latency is reduced. In addition, fewer
servers are needed to host an application, and servers are able to use the full Ethernet bandwidth.
Note Virtual machine chimney, also called TCP offload, has been removed. The TCP chimney will not
be available to guest operating systems.
7.1.3 Network High Availability and ResiliencyTo increase reliability and performance in virtualized environments, Windows Server 2012 includes
built-in support for network adapter hardware that is NIC Teaming –capable. NIC Teaming is also
known as ―network -adapter teaming technology‖ and ―load -balancing failover‖ (LBFO).
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 45/112
Fabric Architecture Guide 45
7.1.3.1 NIC TeamingNIC Teaming, also known as load balancing and failover (LBFO), allows multiple network adapters to
be placed into a team for the purposes of bandwidth aggregation and traffic failover (to maintain
connectivity in the event of a network component failure.
This feature has long been available from network adapter vendors; however, NIC Teaming is nowincluded as an in-box feature with Windows Server 2012.
NIC Teaming is compatible with all networking capabilities in Windows Server 2012 with five
exceptions: SR-IOV, RDMA, native host QoS, TCP chimney, and 802.1X authentication. From a
scalability perspective, on Windows Server 2012, a maximum of 32 network adapters can be added to
a single team, and an unlimited number of teams can be created on a single host.
NIC Teaming Types7.1.3.1.1
When establishing NIC Teaming, it is required to set the teaming mode and distribution mode for the
team. Two basic sets of algorithms are used for teaming modes in NIC Teaming. These are exposed inthe UI as three options —a switch-independent mode, and two switch-dependent modes: Static
Teaming and LACP.
Switch-independent modes : These algorithms make it possible for team members to connect to
different switches because the switch does not know that the interface is part of a team at the host.
These modes do not require the switch to participate in the teaming.
Switch-dependent modes : These algorithms require the switch to participate in the teaming. Here,
all interfaces of the team are connected to the same switch.
There are two common choices for switch-dependent modes of NIC Teaming: Generic or static teaming (IEEE 802.3ad draft v1) : This mode requires configuration on the
switch and on the host to identify which links form the team. Because this is a staticallyconfigured solution, there is no additional protocol to assist the switch and host to identifyincorrectly plugged cables or other errors that could cause the team to fail. Typically, thismode is supported by server-class switches.
Dynamic teaming (IEEE 802.1ax, Link Aggregation Control Protocol [LACP]) : This modeis also commonly referred to as IEEE 802.3ad, because it was developed in the IEEE 802.3adcommittee before it was published as IEEE 802.1ax. It works by using the LACP to dynamicallyidentify links that are connected between the host and a specific switch. Typical server-classswitches support IEEE 802.1ax, but most require administration to enable LACP on the port.There are security challenges to allow an almost completely dynamic IEEE 802.1ax to operateon a switch. As a result, switches require the switch administrator to configure the switchports that are allowed to be members of such a team.
Either of these switch-dependent modes results in inbound and outbound traffic that approach the
practical limits of the aggregated bandwidth.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 46/112
46 Fabric Architecture Guide
Traffic-Distribution Algorithms7.1.3.1.2
Aside from teaming modes, two algorithms are used for traffic distribution within NIC Teaming in
Windows Server 2012. These are exposed in the UI as two options: Hyper-V Switch Port and Address
Hash.
Hyper-V Port mode : Used when virtual machines have independent MAC addresses that can be thebasis for dividing traffic. There is an advantage in using this scheme in virtualization, because the
adjacent switch always sees certain source MAC addresses on only one connected interface. This
causes the switch to "balance" the egress load (the traffic from the switch to the host) on multiple
links, based on the destination MAC address on the virtual machine.
This mode is particularly useful when Virtual Machine Queues (VMQs) is used, because a queue can
be placed on the specific network adapter where the traffic is expected to arrive. However, this mode
might not be granular enough to get a well-balanced distribution, and it will always limit a single
virtual machine to the bandwidth that is available on a single interface.
Windows Server uses the Hyper-V Switch Port as the identifier instead of the source MAC address,
because a virtual machine in some instances might be using more than one MAC address on a switch
port.
Address Hash : Creates a hash value that is based on components of the packet and then assigns
packets that have that hash value to one of the available interfaces. This keeps all packets from the
same TCP stream on the same interface. Components that can be used as inputs to the hashing
function include:
Source and destination MAC addresses
Source and destination IP addresses, with or without considering the MAC addresses (2-tuplehash)
Source and destination TCP ports, usually used with the IP addresses (4-tuple hash)
7.1.3.2 Guest Virtual Machine NIC TeamingNIC Teaming in Windows Server 2012 lets a virtual machine have virtual network adapters that are
connected to more than one virtual switch and still have connectivity, even if the network adapter that
is under that virtual switch is disconnected. This is particularly important when you are working with
features such as SR-IOV traffic, which does not go through the Hyper-V extensible switch and thus
cannot be protected by a network adapter team that is under a virtual switch.
By using the virtual machine teaming option, you can set up two virtual switches, each of which is
connected to its own SR-IOV –capable network adapter. NIC Teaming then works in one of the
following ways:
Each virtual machine can install a virtual function from one or both SR-IOV network adaptersand, if a network adapter disconnection occurs, it will fail over from the primary virtualfunction to the backup virtual function.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 47/112
Fabric Architecture Guide 47
Each virtual machine can have a virtual function from one network adapter and a non-virtualfunction interface to the other switch. If the network adapter that is associated with the virtualfunction becomes disconnected, the traffic can fail over to the other switch without losingconnectivity.
Note Because failover between network adapters in a virtual machine might result in traffic beingsent with the MAC address of the other interface, each virtual switch port that is associated with a
virtual machine by using NIC Teaming must be set to permit MAC spoofing.
7.1.3.3 NIC Teaming Feature CompatibilityTable 8 lists feature compatibility for NIC Teaming.
Feature CommentsData center bridging (DCB) Works independently of NIC Teaming. Supported if team
members support it.IPsec task offload (IPsecTO) Supported if all team members support it.
Large send offload (LSO) Supported if all team members support it.
Receive segment coalescing (RSC) Supported in hosts if any of the team members support it. Notsupported through Hyper-V switches.
Receive-side scaling (RSS) NIC Teaming supports RSS in the host. The TCP/IP stack inWindows Server 2012 programs the RSS information directlyto the team members.
Receive-side checksum offloads (IPv4,IPv6, TCP)
Supported if any of the team members support it.
RDMA Because RDMA data bypasses the Windows Server 2012
protocol stack, team members will not also support RDMA.SR-IOV Because SR-IOV data bypasses the host operating system
stack, network adapters that expose the SR-IOV feature will nolonger expose the feature if they are a member of a team.Teams can be created in virtual machines to team SR-IOVvirtual functions.
TCP chimney offload Not supported with Windows Server 2012 software-based NICTeaming.
Transmit-side checksum offloads (IPv4,IPv6, TCP)
Supported if all team members support it.
Virtual machine queues (VMQ) Supported when teaming is installed under the Hyper-Vswitch.
QoS in host/native OssUse of minimum bandwidth policies will degrade throughputthrough a team.
Virtual machine QoS Virtual machine-QoS is affected by the load-distributionalgorithm that is used by NIC Teaming. For better results, usethe Hyper-V Port load-distribution mode.
802.1X authentication Not compatible with many switches. Should not be used withNIC teaming.
Table 8 Windows Server 2012 NIC Teaming feature compatibility
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 48/112
48 Fabric Architecture Guide
8 Compute Architecture8.1.1 Server Architecture
The host server architecture is a critical component of the virtualized infrastructure, and a key variable
in the consolidation ratio and cost analysis. The ability of the host server to handle the workload of a
large number of consolidation candidates increases the consolidation ratio and helps provide the
desired cost benefit.
The system architecture of the host server refers to the general category of the server hardware.
Examples include rack mounted servers, blade servers, and large symmetric multiprocessor servers
(SMP). The primary tenet to consider when selecting system architectures is that each Hyper-V host
will contain multiple guests with multiple workloads. Processor, RAM, storage, and network capacity
are critical, as are high I/O capacity and low latency. The host server must be able to provide the
required capacity in each of these categories.
Note The Windows Server Catalog is useful for assisting customers in selecting appropriatehardware. It contains all servers, storage, and other hardware devices that are certified for Windows
Server 2008 R2 and Hyper-V. The logo program and support policy for failover-cluster solutions
changed in Windows Server 2008 R2 and Windows Server 2012, and cluster solutions are not listed in
the Windows Server Catalog. All individual components that make up a cluster configuration must
earn the appropriate "Certified for" or "Supported on" Windows Server 2008 R2 or Windows
Server 2012 designations, and they are listed in their device-specific category in the Windows Server
Catalog .
8.1.1.1 Server and Blade Network ConnectivityUse multiple network adapters or multiport network adapters on each host server. For convergeddesigns, network technologies that provide teaming or virtual network adapters can be utilized,
provided that two or more physical adapters can be teamed for redundancy and multiple virtual
network adapters or VLANs can be presented to the hosts for traffic segmentation and bandwidth
control.
8.1.1.2 Microsoft Multipath I/OMultipath I/O (MPIO) architecture supports iSCSI, Fibre Channel, and serial attached storage (SAS)
SAN connectivity by establishing multiple sessions or connections to the storage array.
Multipath solutions use redundant physical path components —adapters, cables, and switches —tocreate logical paths between the server and the storage device. If one or more of these components
should fail (causing the path to fail), multipath logic uses an alternate path for I/O, so that applications
can still access their data. Each network adapter (in the iSCSI case) or HBA should be connected by
using redundant switch infrastructures, to provide continued access to storage in the event of a failure
in a storage fabric component.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 49/112
Fabric Architecture Guide 49
Failover times vary by storage vendor, and they can be configured by using timers in the Microsoft
iSCSI Initiator driver or by modifying the parameter settings of the Fibre Channel host bus adapter
driver.
In all cases, storage multipath solutions should be used. Generally, storage vendors will build a
device-specific module on top of the Multipath I/O (MPIO) software in Windows Server 2012. Eachdevice-specific module and HBA will have its own unique multipath options and recommended
number of connections.
8.1.1.3 Consistent Device NamingWindows Server 2012 supports Consistent Device Naming (CDN), which provides the ability for
hardware manufacturers to identify descriptive names of onboard network adapters within the BIOS.
Windows Server 2012 assigns these descriptive names to each interface, providing users with the
ability to match chassis printed interface names with the network interfaces that are created within
Windows. The specification for this change is outlined in the Slot Naming PCI-SIG Engineering Change
Request .
8.1.2 Failover Clustering8.1.2.1 Cluster-Aware Updating
Cluster-aware updating (CAU) reduces server downtime and user disruption by allowing IT
administrators to update clustered servers with little or no loss in availability when updates are
performed on cluster nodes. CAU transparently takes one node of the cluster offline, installs the
updates, performs a restart (if necessary), brings the node back online, and moves on to the next
node. This feature is integrated into the existing Windows Update management infrastructure, and it
can be further extended and automated with Windows PowerShell for integrating into larger IT
automation initiatives.
CAU facilitates the cluster updating operation while running from a computer running Windows
Server 2012 or Windows 8. The computer running the CAU process is called an orchestrator. CAU
supports either of the two following modes of operation: remote-updating mode or self-updating
mode. In remote-updating mode, a computer that is remote from the cluster being updated acts as
an orchestrator. In self-updating mode, one of the cluster nodes being updated acts as an
orchestrator, and it is capable of self-updating the cluster on a user-defined schedule.
The end-to-end cluster update process by way of the CAU is cluster-aware, and it is completely
automated. It integrates seamlessly with an existing Windows Update Agent (WUA) and MicrosoftWindows Server Update Services (WSUS) infrastructure. CAU also includes an extensible architecture
that supports new plug-in development to orchestrate any node-updating tools, such as custom
software installers, BIOS updating tools, and network adapter/HBA firmware updating tools. After they
have been integrated with CAU, these tools, can work across all cluster nodes in a cluster-aware
manner.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 50/112
50 Fabric Architecture Guide
8.1.2.2 Cluster Shared Volumes2.0The Cluster Shared Volumes (CSV) feature was introduced in Windows Server 2008 R2 as a more
efficient way for administrators to deploy storage for cluster-enabled virtual machines on Hyper-V
clusters. Before CSVs, administrators had to provision a LUN on shared storage for each virtual
machine, so that each machine had exclusive access to its virtual hard disks, and mutual writeconditions could be avoided. By using CSVs, all cluster hosts have simultaneous access to a single
shared volume where storage for multiple virtual machines can be hosted, —thus, there is no need to
provision a new LUN whenever you created a new virtual guest.
Windows Server 2012 introduces a number of new capabilities with CSV 2.0, including:
Flexible application and file storage : Cluster Shared Volumes extends its potential benefitsbeyond Hyper-V to support other application workloads and flexible file storage solutions.CSV 2.0 provides capabilities to clusters through shared namespaces to share configurationsacross all cluster nodes, including the ability to build continuously available cluster-wide file
systems. Application storage can be served from the same share as data, eliminating the needto deploy two clusters (an application and a separate storage cluster) to support true highlyavailability application scenarios.
Integration with other features of Windows Server 2012 R2 : Allows for inexpensivescalability, reliability, and management simplicity through tight integration with StorageSpaces. You gain high performance and resiliency capabilities with SMB Direct and SMBMultichannel, and create more efficient storage with thin provisioning.
Single namespace : Provides a single consistent file namespace where files have the samename and path when viewed from any node in the cluster. CSV volumes are exposed asdirectories and subdirectories under the ClusterStorage root directory.
Improved backup and restore : Supports several backup and restore capabilities, includingsupport for the full feature set of VSS and support for hardware and software backup of CSVvolumes. CSVs also offer a distributed backup infrastructure for software snapshots. TheSoftware Snapshot Provider coordinates creating a CSV 2.0 snapshot, point-in-time semanticsat a cluster level, and the ability to perform remote snapshots.
Single CSV per Cluster
In the ―single CSV per cluster‖ design pattern, the SAN is configured to present a single large LUN to
all the nodes in the host cluster. The LUN is configured as a CSV in failover clustering. All files that
belong to the virtual machines that are hosted on the cluster are stored on the CSV. Optionally, data
deduplication functionality that is provided by the SAN can be utilized (if it is supported by the SAN
vendor).
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 51/112
Fabric Architecture Guide 51
Figure 16 Virtual machine on a single large CSV
Multiple CSVs per Cluster
In the ―multiple CSVs per cluster‖ design pattern, the SAN is configured to present two or more large
LUNs to all the nodes in the host cluster. The LUNs are configured as a CSV in failover clustering. All
virtual machine –related files that belong to the virtual machines that are hosted on the cluster are
stored on the CSVs.
In addition, data deduplication functionality that the SAN provides can be utilized (if supported by the
SAN vendor).
Figure 17 Virtual machines on multiple CSVs, with minimal segregation
For the single and multiple CSV patterns, each CSV has the same I/O characteristics, so that each
individual virtual machine has all of its associated virtual hard disks (VHDs) stored on one of the CSVs.
Host Boot Volumes
(if boot from SAN)Host Cluster WitnessDisk Volumes
DataDeDupe
Large CSVMultiple VMs/VHDs no data/IOoptimization
Storage
SAN
Host Boot Volumes
(if boot from SAN)Host Cluster Witness
Disk Volumes
DataDeDupe
Multiple CSVsMultiple VMs/VHDs no data/IOoptimization
Storage
SAN
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 52/112
52 Fabric Architecture Guide
Figure 18 The virtual disks of each virtual machine reside on the same CSV
Multiple I/O Optimized CSVs per Cluster
In the ―multiple I/O optimized CSVs per cluster‖ design pattern, the SAN is configured to present
multiple LUNs to all the nodes in the host cluster; however, the LUNs are optimized for particular I/O
patterns like fast sequential read performance, or fast random write performance. The LUNs are
configured as CSV in failover clustering. All VHDs that belong to the virtual machines that are hosted
on the cluster are stored on the CSVs, but they are targeted to the appropriate CSV for the given I/O
needs.
Figure 19 Virtual machines with a high degree of virtual disk segregation
OS VHD
Data VHD
Logs VHD
Large CSVMultiple VMs/VHDs nodata/IO optimization
VirtualMachine
Storage
Host Boot Volumes
(if boot from SAN)
CSV Volume 3VM Logging / sequential W I/O
CSV Volume 1VM Operating Systems
CSV Volume 4VM Staging, P2V, V2V
CSV Volume 5VM Configuration files,Volatile Memory, Pagefiles,
Host Cluster CSV Volumes
(per Cluster)
CSV Volume 2VM Database / random R/W I/O
DataDeDupe
NoDataDeDupe
Host Cluster Witness
Disk Volumes
SAN
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 53/112
Fabric Architecture Guide 53
In the ―multiple I/O optimized CSVs per cluster‖ design pattern, each individual virtual machine has all ofits associated VHDs stored on the appropriate CSV, per required I/O requirements.
Figure 20 Virtual machines with a high degree of virtual disk segregation
Note A single virtual machine can have multiple VHDs and each VHD can be stored on a different
CSV (provided that all CSVs are available to the host cluster on which the virtual machine is created).
8.1.2.3 BitLocker-Encrypted Cluster VolumesHyper-V, failover clustering, and BitLocker work together to create an ideal, highly secure platform for
private cloud infrastructure. Windows Server 2012 cluster disks encrypted with BitLocker Drive
Encryption enable better physical security for deployments outside secure data centers (providing thatthere is a critical safeguard for private cloud infrastructure) and help protect against data leaks.
8.1.2.4 Hyper-V Application MonitoringWith Windows Server 2012, Hyper-V and failover clustering work together to bring higher availability
to workloads that do not support clustering. They do so by providing a lightweight, simple solution to
monitor applications that are running on virtual machines and by integrating with the host. By
monitoring services and event logs inside the virtual machine, Hyper-V and failover clustering can
detect if the key services that a virtual machine provides are healthy. If necessary, they provide
automatic corrective action such as restarting the virtual machine or restarting a service within the
virtual machine.
8.1.2.5 Virtual Machine Failover PrioritizationVirtual machine priorities can be configured to control the order in which specific virtual machines fail
over or start. This helps make sure that high-priority virtual machines get the resources that they need
and that lower-priority virtual machines are given resources as they become available.
OS VHD
Data VHD
Logs VHD
VirtualMachine
CSV Volume 3VM Logging /sequential W I/O
CSV Volume 1VM OperatingSystems
CSV Volume 2VM Database /random R/W I/O
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 54/112
54 Fabric Architecture Guide
9 Hyper-V Virtualization Architecture9.1.1 Windows Server 2012 Hyper-V Features9.1.1.1 Hyper-V Host and Guest Scale-Up
Windows Server 2012 Hyper-V supports running on a host system that has up to 320 logical
processors on hardware and 4 terabytes (TB) of physical memory. This helps encourage compatibility
with the largest scale-up server systems.
Hyper-V in Windows Server 2012 lets you configure a virtual machine with up to 64 virtual processors
and up to 1 TB of memory, to support very large workload scenarios.
Hyper-V in Windows Server 2012 supports running up to 8,000 virtual machines on a 64-node failover
cluster. This is a significant improvement on the previous version, which supported a maximum of 16
cluster nodes and 1,000 virtual machines per cluster.
9.1.1.2 Hyper-V over SMB 3.0Prior to Windows Server 2012, remote storage options for Hyper-V were limited to expensive Fibre
Channel SAN solutions that were difficult to provision for Hyper-V guests or other more inexpensive
options that did not offer many features. By enabling Hyper-V to use SMB file shares for virtual
storage, administrators have a new option that is simple to provision with support for CSV 2.0 and
inexpensive to deploy, but also offers performance capabilities and features that rival those available
with Fibre Channel SANs.
Hyper-V over SMB requires:
One or more computers running Windows Server 2012, with the Hyper-V and File and
Storage Services roles installed. A common Active Directory infrastructure. (The servers that are running AD DS do not have to
run Windows Server 2012.) Failover clustering on the Hyper-V side, on the File and Storage Services side, or both.
Failover clustering is not required.
Hyper-V over SMB supports a variety of flexible configurations that offer several levels of capabilities
and availability, which include single-node, dual-node, and multi-node file server modes.
9.1.1.3 Virtual Machine MobilityHyper-V live migration makes it possible to move running virtual machines from one physical host to
another with no effect on the availability of virtual machines to users. Hyper-V in Windows
Server 2012 introduces faster and simultaneous live migration inside or outside a clustered
environment.
In addition to providing live migration in the most basic of deployments, this functionality facilitates
more advanced scenarios, such as performing a live migration to a virtual machine between multiple,
separate clusters to balance loads across an entire data center.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 55/112
Fabric Architecture Guide 55
If you use live migration in a clustered environment, you will see that live migrations can now use
higher network bandwidths (up to 10 gigabits) to complete migrations faster. You can also perform
multiple simultaneous live migrations to move many virtual machines in a cluster quickly.
9.1.1.4 Hyper-V Live MigrationWindows Server 2012 Hyper-V live migration lets you perform live migrations outside a failovercluster. The two scenarios for this are:
SMB-based live migration . In this instance, the hard disk of each virtual machine is stored
on a central SMB file share. You then perform a live migration of the virtual machines from
one server to another while their storage remains on the central SMB share.
“Shared -nothing” live migration . In this case, the live migration of a virtual machine from
one non-clustered Hyper-V host to another begins when the hard drive storage of the virtual
machine is mirrored to the destination server over the network. Then you perform the live
migration of the virtual machine to the destination server while it continues to run andprovide network services.
9.1.1.5 Storage MigrationWindows Server 2012 introduces support for live storage migration, which lets you move virtual hard
disks that are attached to a virtual machine that is running. When you have the flexibility to manage
storage without affecting the availability of your virtual machine workloads, you can perform
maintenance on storage subsystems, upgrade storage-appliance firmware and software, and balance
loads while the virtual machine is in use.
Windows Server 2012 provides the flexibility to move virtual hard disks on shared and non-shared
storage subsystems if a network shared folder on Windows Server 2012 SMB is visible to both Hyper-
V hosts.
Combined with live migration, storage migration also lets you move a virtual machine between hosts
on different cluster servers that are not using the same storage.
9.1.1.6 Hyper-V Extensible SwitchThe Hyper-V extensible switch in Windows Server 2012 is a Layer 2 virtual network switch that
provides programmatically managed and extensible capabilities to connect virtual machines to the
physical network. The Hyper-V extensible switch is an open platform that lets multiple vendors
provide extensions that are written to standard Windows API frameworks. The reliability of extensionsis strengthened through the Windows standard framework and the required third-party code for
functions is reduced. It is backed by the Windows Hardware Quality Labs (WHQL) certification
program. You can manage the Hyper-V extensible switch and its extensions by using Windows
PowerShell, or programmatically by using WMI or the Hyper-V Manager UI.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 56/112
56 Fabric Architecture Guide
The Hyper-V extensible switch architecture in Windows Server 2012 is an open framework that lets
third parties add new functionality such as monitoring, forwarding, and filtering into the virtual switch.
Extensions are implemented by using Network Device Interface Specification (NDIS) filter drivers and
Windows Filtering Platform (WFP) callout drivers. These public Windows platforms for extending
Windows networking functionality are used as follows:
NDIS filter drivers : Used to monitor or modify network packets in Windows. NDIS filterswere introduced with the NDIS 6.0 specification.
WFP callout drivers : Introduced in Windows Vista and Windows Server 2008, and letindependent software vendors (ISVs) create drivers to filter and modify TCP/IP packets,monitor or authorize connections, filter IPsec-protected traffic, and filter remote procedurecalls (RPCs). Filtering and modifying TCP/IP packets provides unprecedented access to theTCP/IP packet processing path. In this path, you can examine or modify outgoing andincoming packets before additional processing occurs. By accessing the TCP/IP processingpath at different layers, you can more easily create firewalls, antivirus software, diagnostic
software, and other types of applications and services.
The Hyper-V extensible switch is a module that runs in the root partition of Windows Server 2012. The
switch module can create multiple virtual switch extensions per host. A single virtual switch can have
up to 64 virtual ports, and all virtual switch policies —including QoS, VLAN, and ACLs —are configured
per virtual port. Any policy that is configured on a virtual port is preserved during a virtual switch state
transition, such as a live migration. Each virtual port can connect to one virtual network adapters; in
the case of External, each virtual port can connect to a team of virtual network adapters.
The extensible virtual switch framework allows for third-party extensions to extend and affect the
behavior of the Hyper-V switch. The extensibility stack comprises an extension miniport driver and anextension protocol driver that are bound to the virtual switch. Switch extensions are lightweight filter
drivers that bind between these drivers to form the extension stack.
There are three classes of extensions:
Capture: Sit on top of the stack and monitor switch traffic. Filter : Sit in the middle of the stack and can both monitor and modify switch traffic. Forwarding : Sit on the bottom of the stack and replace the virtual switch forwarding
behavior.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 57/112
Fabric Architecture Guide 57
Table 9 lists the various types of Hyper-V extensible switch extensions.
Feature Purpose Examples ExtensibilityComponent
Network PacketInspection Inspecting networkpackets, but does notchange them
sFlow and networkmonitoring NDIS filter driver
Network Packet Filter Injecting, modifying,and droppingnetwork packets
Security NDIS filter driver
Network Forwarding Third-partyforwarding thatbypasses defaultforwarding
OpenFlow, VirtualEthernet Port Aggregator(VEPA), and proprietarynetwork fabrics
NDIS filter driver
Firewall/ IntrusionDetection
Filtering andmodifying TCP/IPpackets, monitoring
or authorizingconnections, filteringIPsec-protectedtraffic, and filteringRPCs
Virtual firewall andconnection monitoring
WFP callout driver
Table 9 Windows Server 2012 Virtual Switch Extension Types
Only one forwarding extension can be installed per virtual switch, although multiple capture and
filtering extensions can be installed. In addition, by monitoring extensions, you can gather statistical
data by monitoring traffic at different layers of the switch. Multiple monitoring and filtering
extensions can be supported at the ingress and egress portions of the Hyper-V extensible switch. Onlyone instance of the forwarding extension can be used per switch instance, and it overrides the default
switching of the Hyper-V extensible switch. Figure 21 shows the architecture of the Hyper-V
extensible switch and the extensibility model.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 58/112
58 Fabric Architecture Guide
Figure 21 Hyper-V extension layers
The Hyper-V extensible virtual switch data path is bidirectional, which allows all extensions to see the
traffic as it enters and exits the virtual switch. The NDIS Send path is used as the ingress data path,
while the NDIS Receive path is used for egress traffic. Between ingress and egress, forwarding of
traffic occurs by the Hyper-V virtual switch or by a forwarding extension. Figure 22 outlines this
interaction.
Figure 22 Hyper-V Extension bi-directional filter Windows Server 2012 provides Windows PowerShell cmdlets for the Hyper-V extensible switch that
lets you build command-line tools or automated scripts for setup, configuration, monitoring, and
troubleshooting. These cmdlets can be run remotely. Windows PowerShell also helps third parties
build their own tools to manage the Hyper-V extensible switch.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 59/112
Fabric Architecture Guide 59
9.1.1.7 Network Isolation and SecurityWindows Server 2012 contains new security and isolation capabilities through the Hyper-V extensible
switch. With Windows Server 2012, you can configure Hyper-V servers to enforce network isolation
among any set of arbitrary isolation groups, which are typically defined for individual customers or
sets of workloads.Windows Server 2012 provides the isolation and security capabilities for multitenancy by offering the
following new features:
Multi-tenant virtual machine isolation through private virtual LANs (private VLANs) Protection from Address Resolution Protocol (ARP) and Neighbor Discovery protocol
spoofing Protection against Dynamic Host Configuration Protocol (DHCP) snooping and DHCP guard Isolation and metering by using virtual port access control lists (ACLs) The ability to trunk traditional VLANs to virtual machines
Monitoring Windows PowerShell and Windows Management Instrumentation (WMI)
VLANs9.1.1.7.1
Currently, VLANs are the mechanism that most organizations use to help support tenant isolation and
the reuse of address space. A VLAN uses explicit tagging (VLAN ID) in the Ethernet frame headers, and
it relies on Ethernet switches to enforce isolation and restrict traffic to network nodes that have the
same VLAN ID.
Private VLANs9.1.1.7.2
VLAN technology is traditionally used to subdivide a network and provide isolation for individualgroups that share a common physical infrastructure. Windows Server 2012 introduces support for
private VLANs, which is a technique that is used with VLANs that can be used to provide isolation
between two virtual machines that are on the same VLAN.
When a virtual machine does not have to communicate with other virtual machines, you can use
private VLANs to isolate it from other virtual machines that are in your data center. By assigning each
virtual machine in a PVLAN only one primary VLAN ID and one or more secondary VLAN IDs, you can
put the secondary private VLANs into one of three modes, as shown in the following table. These
PVLAN modes determine to which other virtual machines on the PVLAN a virtual machine can talk to.
To isolate a virtual machine, you should put it in isolated mode.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 60/112
60 Fabric Architecture Guide
PVLAN Mode DescriptionIsolated Isolated ports cannot exchange packets with each
other at Layer 2.
Promiscuous Promiscuous ports can exchange packets with anyother port that is on the same primary VLAN ID.
Community Community ports that are on the same VLAN IDcan exchange packets with each other at Layer 2.
Table 10 PVLAN modes
ARP and Neighbor Discovery Spoofing Protection 9.1.1.7.3
The Hyper-V extensible switch helps provide protection against a malicious virtual machine stealing IP
addresses from other virtual machines through ARP spoofing (also known as ARP poisoning in IPv4).
With this type of man-in-the-middle attack, a malicious virtual machine sends a fake ARP message,
which associates its own MAC address to an IP address that it does not own. Unsuspecting virtual
machines send network traffic that is targeted to that IP address to the MAC address of the malicious
virtual machine, instead of to the intended destination. For IPv6, Windows Server 2012 helps provide
equivalent protection for Neighbor Discovery spoofing. This is a mandatory scenario to consider
hosting companies where the virtual machine is not under control of the fabric or cloud
administrators.
Router Guard9.1.1.7.4
The Hyper-V extensible switch now helps protect against router advertisement and redirection
messages from an unauthorized virtual machine pretending to be a router. This will lead a maliciousvirtual machine to want to be a router for other virtual machines. After a virtual machine becomes the
next hope in the network routing path, it can do man-in-the-middle attacks, for example, to steal
passwords from SSL connections.
DHCP Guard9.1.1.7.5
In a DHCP environment, a rogue DHCP server could intercept client DHCP requests and provide
incorrect address information. The rogue DHCP server could cause traffic to be routed to a malicious
intermediary that sniffs all traffic before forwarding it to the legitimate destination. To protect against
this particular man-in-the-middle attack, the Hyper-V administrator can designate which Hyper-V
extensible switch ports can have DHCP servers connected to them. DHCP server traffic from other
Hyper-V extensible switch ports is automatically dropped. The Hyper-V extensible switch now helps
protect against a rogue DHCP server that is attempting to provide IP addresses that would cause
traffic to be rerouted.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 61/112
Fabric Architecture Guide 61
Virtual Port ACLs9.1.1.7.6
Port ACLs provide a mechanism for isolating networks and metering network traffic for a virtual port
on the Hyper-V extensible switch. By using port ACLs, you can meter the IP addresses or MAC
addresses that can (or cannot) communicate with a virtual machine. For example, you can use port
ACLs to enforce isolation of a virtual machine by letting it talk only to the Internet, or communicateonly with a predefined set of addresses. By using the metering capability, you can measure network
traffic that is going to or from a specific IP address or MAC address, which lets you report on traffic
that is sent or received from the Internet or from network storage arrays.
You also can configure multiple port ACLs for a virtual port. Each port ACL consists of a source or
destination network address, and a permit to deny or meter action. The metering capability also
supplies information about the number of instances where traffic was attempted to or from a virtual
machine from a restricted (―deny‖) address.
Trunk Mode to Virtual Machines9.1.1.7.7
A VLAN makes a set of host machines or virtual machines appear to be on the same LAN,
independent of their actual physical locations. By using the Hyper-V extensible switch trunk mode,
traffic from multiple VLANs can now be directed to a single network adapter in a virtual machine that
could previously receive traffic from only one VLAN. As a result, traffic from different VLANs is
consolidated, and a virtual machine can listen to multiple VLANs. This feature can help you shape
network traffic and enforce multi-tenant security in your data center.
Monitoring/Port Mirroring 9.1.1.7.8
Many physical switches can monitor the traffic from specific ports that is flowing through specific
virtual machines on the switch. The Hyper-V extensible switch also provides this port mirroring,helping you to designate which virtual ports should be monitored and to which virtual port the
monitored traffic should be delivered for further processing. For example, a security-monitoring
virtual machine can look for anomalous patterns in the traffic that flows through other specific virtual
machines on the switch. In addition, you can diagnose network connectivity issues by monitoring
traffic that is bound for a particular virtual switch port.
Network Virtualization9.1.1.7.9
Isolating the virtual machines of different departments or customers can be a challenge on a shared
network. When entire networks of virtual machines must be isolated, the challenge becomes even
greater. Traditionally, VLANs have been used to isolate networks, but VLANs are very complex to
manage on a large scale. The following are the primary drawbacks of VLANs:
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 62/112
62 Fabric Architecture Guide
Cumbersome reconfiguration of production switches is required whenever virtual machines or
isolation boundaries must be moved. Moreover, frequent reconfigurations of the physical
network for purposes of adding or modifying VLANs increases the risk of an outage.
VLANs have limited scalability because typical switches support no more than 1,000 VLAN IDs
(with a maximum of 4,095).
VLANs cannot span multiple subnets, which limits the number of nodes in a single VLAN and
restricts the placement of virtual machines based on physical location.
Windows Server 2012 introduces Hyper-V network virtualization, a new feature that enables you to
isolate network traffic from different business units or customers on a shared infrastructure, without
having to use VLANs. Network virtualization also lets you move virtual machines as needed within
your virtual infrastructure while preserving their virtual network assignments. You can even use
network virtualization to transparently integrate these private networks into a preexisting
infrastructure on another site.
Hyper-V network virtualization extends the concept of server virtualization to permit multiple virtual
networks, potentially with overlapping IP addresses, to be deployed on the same physical network. By
using network virtualization, you can set policies that isolate traffic in a dedicated virtual network,
independently of the physical infrastructure.
To virtualize the network, Hyper-V network virtualization uses the following elements:
Two IP addresses for each virtual machine
Generic Routing Encapsulation (GRE)
IP address rewrite
Policy management server
The potential benefits of network virtualization include the following:
Tenant network migration to the cloud with minimum reconfiguration or effect on isolation.
Customers can keep their internal IP addresses while they move workloads onto shared IaaS
clouds, thus minimizing the configuration changes that are needed for IP addresses, DNS
names, security policies, and virtual machine configurations. In software-defined, policy-based
data center networks, network traffic isolation does not depend on VLANs, but it is enforced
within Hyper-V hosts, based on multi-tenant isolation policies. Network administrators canstill use VLANs to manage traffic in the physical infrastructure if the topology is primarily
static.
Tenant virtual machine deployment anywhere in the data center. Services and workloads can
be placed or migrated to any server in the data center while keeping their IP addresses,
without being limited to a physical IP subnet hierarchy or VLAN configurations.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 63/112
Fabric Architecture Guide 63
Simplified network design and improved server and network resource use. The rigidity of
VLANs, along with the dependency of virtual machine placement on a physical network
infrastructure, results in overprovisioning and underuse. By breaking this dependency, virtual
networking increases the flexibility of virtual machine workload placement, thus simplifying
network management and improving the use of servers and network resources. Placement of
server workloads is simplified because migration and placement of workloads are
independent of the underlying physical network configurations. Server administrators can
focus on managing services and servers, while network administrators can focus on overall
network infrastructure and traffic management.
Works with present day hardware (servers, switches, appliances) to promote performance.
Network virtualization can be deployed in present day data centers, and yet it is compatible
with emerging data center ―flat network‖ technologies such as Transparent Interconnection of
Lots of Links (TRILL), which is an IETF-standard architecture that is intended to expand
Ethernet topologies. Full management through Windows PowerShell and WMI. You can use Windows PowerShell
to script and automate administrative tasks easily. Windows Server 2012 includes Windows
PowerShell cmdlets for network virtualization that let you build command-line tools or
automated scripts to configure, monitor, and troubleshoot network isolation policies.
Virtual Fibre Channel 9.1.1.7.10
Windows Server 2012 provides Fibre Channel ports within the Hyper-V guest operating system, this
lets you connect to Fibre Channel directly from virtual machines when virtualized workloads have to
connect to existing storage arrays. This protects your investments in Fibre Channel, lets you virtualize
workloads that use direct access to Fibre Channel storage, lets you cluster guest operating systems
over Fibre Channel, and offers an important new storage option for servers that are hosted in your
virtualization infrastructure.
Fibre Channel in Hyper-V requires:
One or more installations of Windows Server 2012 with the Hyper-V role installed. Hyper-V
requires a computer with processor support for hardware virtualization.
A computer that has one or more Fibre Channel host bus adapters (HBAs), each of which has
an updated HBA driver that supports virtual Fibre Channel. Updated HBA drivers are included
with the HBA drivers for some models.
Virtual machines that are configured to use a virtual Fibre Channel adapter, which must use
Windows Server 2012, Windows Server 2008 R2, or Windows Server 2008 as the guest
operating system.
Connection only to data logical unit numbers (LUNs). Storage that is accessed through a
virtual Fibre Channel connected to a LUN cannot be used as boot media.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 64/112
64 Fabric Architecture Guide
Virtual Fibre Channel for Hyper-V provides the guest operating system with unmediated access to a
SAN by using a standard World Wide Name (WWN) that is associated with a virtual machine. Hyper-V
users can use Fibre Channel SANs to virtualize workloads that require direct access to SAN LUNs.
Virtual Fibre Channel also allows you to operate in new scenarios, such as running the Windows
failover clustering feature inside the guest operating system of a virtual machine that is connected to
shared Fibre Channel storage.
Midrange and high-end storage arrays are capable of advanced storage functionality that helps
offload certain management tasks from the hosts to the SANs. Virtual Fibre Channel presents an
alternate hardware-based I/O path to the virtual hard disk stack in Windows software. This allows you
to use the advanced functionality offered by your SANs directly from virtual machines running Hyper-
V. For example, you can use Hyper-V to offload storage functionality (like taking a snapshot of a LUN)
on the SAN hardware by using a hardware Volume Shadow Copy Service (VSS) provider from within a
virtual machine running Hyper-V.
N_Port ID Virtualization (NPIV) support: Virtual Fibre Channel for Hyper-V guest operating systemsuses the existing (NPIV) T11 standard to map multiple virtual N_Port IDs to a single physical Fibre
Channel N_Port. A new NPIV port is created on the host each time a virtual machine is started that is
configured with a virtual HBA. When the virtual machine stops running on the host, the NPIV port is
removed.
9.1.1.8 VHDXHyper-V in Windows Server 2012 contains an update to the VHD format (called VHDX) that has much
larger capacity and built-in resiliency. The principal new features of the VHDX format are:
Support for virtual hard disk storage capacity of up to 64 TB Additional protection against data corruption during power failures by logging updates to the
VHDX metadata structures Improved alignment of the virtual hard disk format to work well on large sector physical disks
The VHDX format also has the following features: Larger block sizes for dynamic and differential disks, which allows these disks to attune to the
needs of the workload Four-kilobyte (4 KB) logical sector virtual disk that allows for increased performance when it is
used by applications and workloads that are designed for 4 KB sectors The ability to store custom metadata about the file that the user might want to record, such
as the operating system version or updates that have been applied Efficiency in representing data (called ―trim‖), which results in smaller files size and allows the
underlying physical storage device to reclaim unused space. (Trim requires directly attachedstorage or SCSI disks and trim compatible hardware.)
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 65/112
Fabric Architecture Guide 65
9.1.1.9 Guest Non-Uniform Memory AccessHyper-V in Windows Server 2012 supports Non-Uniform Memory Access (NUMA) in a virtual machine.
NUMA refers to a compute architecture in multiprocessor systems in which the required time for a
processor to access memory depends on the location of the memory, relative to the processor.
By using NUMA, a processor can access local memory (memory that is attached directly to theprocessor) faster than it can access remote memory (memory that is local to another processor in the
system). Modern operating systems and high-performance applications such as SQL Server have
developed optimizations to recognize the NUMA topology of the system, and they consider NUMA
when they schedule threads or allocate memory to increase performance.
Projecting a virtual NUMA topology into a virtual machine provides optimal performance and
workload scalability in large virtual machine configurations. It does so by letting the guest operating
system and applications such as SQL Server utilize their inherent NUMA performance optimizations.
The default virtual NUMA topology that is projected into a Hyper-V virtual machine is optimized to
match the NUMA topology of the host.
9.1.1.10 Dynamic MemoryDynamic Memory, which was introduced in Windows Server 2008 R2 SP1, helps you use physical
memory more efficiently. By using Dynamic Memory, Hyper-V treats memory as a shared resource
that can be automatically reallocated among running virtual machines. Dynamic Memory adjusts the
amount of memory that is available to a virtual machine, based on changes in memory demand and
on the values that you specify.
In Windows Server 2012, Dynamic Memory has a new configuration item called ―minimum memory.‖
Minimum memory lets Hyper-V reclaim the unused memory from the virtual machines. This can resultin increased virtual machine consolidation numbers, especially in VDI environments.
Windows Server 2012 also introduces Hyper-V Smart Paging for robust restart of virtual machines.
Although minimum memory increases virtual machine consolidation numbers, it also brings a
challenge. If a virtual machine has a smaller amount of memory than its startup memory and it is
restarted, Hyper-V needs additional memory to restart the virtual machine. Because of host memory
pressure or the states of the virtual machines, Hyper-V might not always have additional memory
available, which can cause sporadic virtual machine restart failures in customer environments. In
Windows Server 2012, Hyper-V Smart Paging is used to bridge the memory gap between minimum
memory and startup memory and to let virtual machines restart more reliably.
In Windows Server 2012, new functionality in Dynamic Memory for Hyper-V lets you:
Configure a lower minimum memory for virtual machines and have an effective restartexperience.
Increase the maximum memory and decrease the minimum memory on virtual machines thatare running.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 66/112
66 Fabric Architecture Guide
9.1.1.11 Hyper-V ReplicaYou can use failover clustering to create high availability virtual machines, but this does not protect
businesses from outage of an entire data center without the use of hardware-based SAN replication
across data centers. Hyper-V Replica fills an important need by providing an affordable failure
recovery solution from an entire site, down to a single virtual machine. It provides asynchronous,unlimited replication of your virtual machines over a network link from one Hyper-V host at a primary
site to another Hyper-V host at a replica site, without relying on storage arrays or other software
replication technologies.
Hyper-V Replica tracks the write operations on the primary virtual machine and replicates these
changes to the replica server efficiently over a WAN. The network connection between the two servers
uses the HTTP or HTTPS protocol and supports integrated and certificate-based authentication.
Connections configured to use integrated authentication are not encrypted. For an encrypted
connection, use certificate-based authentication. Hyper-V Replica is closely integrated with Windows
failover clustering and provides seamless replication across migration scenarios in the primary andreplica servers.
9.1.1.12 Resource MeteringIn Windows Server 2012, Hyper-V introduces resource metering, which is a technology that helps you
track historical data on the use of virtual machines and gain insight into the resource use of specific
servers. You can use this data to perform capacity planning, monitor consumption by different
business units or customers, or capture data that is necessary to help redistribute the costs of running
a workload. You could also use the information that this feature provides to help build a billing
solution, so that you can charge customers of your hosting services appropriately for their usage of
resources.
9.1.2 Windows Server 2012 Hyper-V Failover ClusteringA Hyper-V host failover cluster is a group of independent servers that work together to increase the
availability of applications and services. The clustered servers (which are called nodes) are connected
by physical cables and software. If one of the cluster nodes fails, another node begins to provide
service —a process that is known as failover. In the case of a planned live migration, users will
experience no perceptible service interruption.
The host servers are one critical component of a dynamic, virtual infrastructure. Consolidation of
multiple workloads onto the host servers requires that those servers be highly available. WindowsServer 2012 provides advances in failover clustering that enable high availability and live migration of
virtual machines between physical nodes.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 67/112
Fabric Architecture Guide 67
9.1.2.1 Host Failover-Cluster TopologyIt is recommended that the server topology consist of at least two Hyper-V host clusters. The first
needs at least two nodes, and it is referred to as the fabric management cluster. The second, plus any
additional clusters, is referred to as fabric host clusters.
In scenarios of smaller scale or specialized solutions, the management and fabric clusters can beconsolidated onto the fabric host cluster. Special care has to be taken to provide resource availability
for the virtual machines that host the various parts of the management stack.
Each host cluster can contain up to 64 nodes. Host clusters require some form of shared storage such
as a Scale-Out File Server cluster, Fibre Channel, or iSCSI SAN.
9.1.2.2 Cluster Quorum and Witness ConfigurationsIn quorum configurations, every cluster node has one vote, and a witness (disk or file share) also has
one vote. A witness (disk or file share) is recommended when the number of voting nodes is even, but
it is not required when the number of voting nodes is odd. It is always recommended to keep thetotal number of votes in a cluster as odd. Therefore, a cluster witness should be configured to support
Hyper-V cluster configurations when the number of failover cluster nodes is even.
Choices for a cluster witness include a shared disk witness and a file-share witness. There are distinct
differences between these two models. A disk witness consists of a dedicated LUN to serve as the
quorum disk that is used as an arbitration point. A disk witness stores a copy of cluster database for
all nodes to share. It is recommended that this disk consist of a small partition that is at least 512 MB
in size; however, it is commonly recommended to reserve a 1 GB disk for each cluster. This LUN can
be NTFS- or ReFS-formatted and does not require the assignment of a drive letter.
File-share witness configurations use a simple, unique file share that is located on a file server to
support one or more clusters. This file share must have write permissions for the cluster name object
(CNO), along with all of the nodes. It is highly recommended that this file share exist outside any of
the cluster nodes, and therefore, carry the requirement of additional physical or virtual servers outside
the Hyper-V compute cluster within the fabric. Writing to this share results in minimal network traffic,
because all nodes contain separate copies of the cluster database, and only cluster membership
changes are written to the share. The additional challenge that this creates is that file-share witness
configurations are susceptible to ―split‖ or ―partition in time‖ scenarios and could create situations in
which surviving nodes and starting nodes have different copies of the cluster database 3. File-share
witness disks should be used only in configurations in which no shared disk infrastructure exists.
3 http://technet.microsoft.com/en-us/library/cc770830(v=WS.10).aspx
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 68/112
68 Fabric Architecture Guide
9.1.2.3 Host Cluster NetworksA variety of host cluster networks are required for a Hyper-V failover cluster. The network
requirements help enable high availability and high performance. The specific requirements and
recommendations for network configuration are published on in the TechNet Library in the Hyper-V:
Live Migration Network Configuration Guide .
Network AccessType
Purpose of thenetwork-access type
Network-trafficrequirements
Recommendednetwork access
Storage Access storage throughSMB, iSCSI, or FibreChannel. (Fibre Channeldoes not need anetwork adapter.)
High bandwidth andlow latency.
Usually, dedicated andprivate access. Refer toyour storage vendor forguidelines.
Virtual machine
access
Workloads that run on
virtual machines usuallyrequire externalnetwork connectivity toservice client requests.
Varies. Public access, which
could be teamed forlink aggregation or tofail over the cluster.
Management Managing the Hyper-Vmanagement operatingsystem. This network isused by Hyper-VManager or SystemCenter Virtual Machine
Manager (VMM).
Low bandwidth. Public access, whichcould be teamed to failover the cluster.
Cluster andCluster SharedVolumes (CSV)
Preferred network thatis used by the clusterfor communications tomaintain cluster health.Also, used by CSV tosend data betweenowner and non-ownernodes. If storage accessis interrupted, thisnetwork is used toaccess the CSV or tomaintain and back upthe CSV.
The cluster should haveaccess to more than
Usually, low bandwidthand low latency.Occasionally, highbandwidth.
Private access.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 69/112
Fabric Architecture Guide 69
one network forcommunication to helpmake sure that it is ahigh availability cluster.
Live migration Transfer virtual machinememory and state.
High bandwidth andlow latency duringmigrations.
Private access
Table 11 Network access types
9.1.2.4 Management NetworkA dedicated management network is required so that hosts can be managed through a dedicated
network to prevent competition with guest traffic needs. A dedicated network provides a degree of
separation for the purposes of security and ease of management. A dedicated management network
typically implies dedicating a network adapter per host and port per network device to the
management network.
Additionally, many server manufacturers also provide a separate out-of-band (OOB) management
capability that enables remote management of server hardware outside the host operating system.
9.1.2.5 iSCSI NetworkIf using iSCSI, a dedicated iSCSI network is required, so that storage traffic is not in contention with
any other traffic. This typically implies dedicating two network adapters per host and two ports per
network device to the management network.
9.1.2.6 CSV/Cluster Communication NetworkUsually, when the cluster node that owns a VHD file in a CSV performs disk I/O, the node
communicates directly with the storage. However, storage connectivity failures sometimes prevent a
given node from communicating directly with the storage. To maintain functionality until the failure is
corrected, the node redirects the disk I/O through a cluster network (the preferred network for CSV)
to the node where the disk is currently mounted. This is called CSV redirected I/O mode.
9.1.2.7 Live-Migration Network
During live migration, the contents of the memory of the virtual machine that is running on thesource node must be transferred to the destination node over a LAN connection. To enable high-
speed transfer, a dedicated live-migration network is required.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 70/112
70 Fabric Architecture Guide
9.1.2.8 Virtual Machine Network(s)The virtual machine networks are dedicated to virtual machine LAN traffic. A virtual machine network
can be two or more 1 GbE networks, one or more networks that have been created through NIC
Teaming, or virtual networks that have been created from shared 10 GbE network adapters.
9.1.3 Hyper-V Guest Virtual Machine DesignStandardization is a key tenet of private cloud architectures and virtual machines. A standardized
collection of virtual machine templates can drive predictable performance and greatly improve
capacity planning capabilities. As an example, the following table illustrates the composition of a basic
virtual machine template library.
Template Specifications Network Operating system Unit costTemplate 1 —Small
2 vCPU, 4 GB memory,50 GB disk
VLAN 20 WindowsServer 2008 R2
1
Template 2 —Medium
8 vCPU, 16 GB memory,100 GB disk
VLAN 20 WindowsServer 2012
2
Template 3 —
X-Large
24 vCPU, 64 GB memory,200 GB disk
VLAN 20 WindowsServer 2012
4
Table 12 Template specification
9.1.3.1 Virtual Machine StorageDynamically Expanding Disks
Dynamically expanding VHDs provide storage capacity as needed to store data. The size of the VHD
file is small when the disk is created and grows as data is added to the disk. The size of the VHD file
does not shrink automatically when data is deleted from the virtual hard disk; however, you can use
the Edit Virtual Hard Disk Wizard to make the disk more compact and decrease the file size after data
is deleted.
Fixed-Size Disks
Fixed-size VHDs provide storage capacity by using a VHD file that is in the size that is specified for the
virtual hard disk when the disk is created. The size of the VHD file remains fixed, regardless of the
amount of data that is stored. However, you can use the Edit Virtual Hard Disk Wizard to increase thesize of the VHD, which in turn increases the size of the VHD file. By allocating the full capacity at the
time of creation, fragmentation at the host level is not an issue. (Fragmentation inside the VHD itself
must be managed within the guest.)
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 71/112
Fabric Architecture Guide 71
Differencing Disks
Differencing VHDs provide storage to help you make changes to a parent VHD without changing the
disk. The size of the VHD file for a differencing disk grows as changes are stored to the disk.
Pass-Through Disks
Hyper-V helps virtual machine guests directly access local disks or SAN LUNs that are attached to the
physical server, without requiring the volume to be presented to the host server. The virtual machine
guest accesses the disk directly (by utilizing the GUID of the disk) without having to utilize the file
system of the host. Given that the performance difference between fixed-disk and pass-through disks
is now negligible, the decision is based on manageability. For instance, a VHD is hardly portable if the
data on the volume will be very large (hundreds of gigabytes), given the extreme amounts of time it
takes to copy. For a backup scheme with pass-through disks the data can be backed up only from
within the guest.
When you are utilizing pass-through disks, no VHD file is created, because the LUN is used directly bythe guest. Because there is no VHD file, there is no dynamic sizing or snapshot capability.
In-guest iSCSI Initiator
Hyper-V can also utilize iSCSI storage by directly connecting to iSCSI LUNs that are utilizing the virtual
network adapters of the guest. This is mainly used for access to large volumes on SANs to which the
Hyper-V host itself is not connected or for guest clustering. Guests cannot boot from iSCSI LUNs that
are accessed through the virtual network adapters without utilizing a third-party iSCSI initiator.
9.1.3.2 Virtual Machine Networking
Hyper-V guests support two types of virtual network adapters: synthetic and emulated. Syntheticadapters make use of the Hyper-V VMBus architecture, and they are the high-performance, native
devices. Synthetic devices require that the Hyper-V integration services be installed within the guest.
Emulated adapters are available to all guests, even if integration services are not available. They
perform much more slowly and should be used only if synthetic devices are unavailable.
You can create many virtual networks on the server running Hyper-V to provide a variety of
communications channels. For example, you can create networks to provide the following:
Communications between virtual machines only. This type of virtual network is called a
private network.
Communications between the host server and virtual machines. This type of virtual network is
called an internal network.
Communications between a virtual machine and a physical network by creating an association
to a physical network adapter on the host server. This type of virtual network is called an
external network
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 72/112
72 Fabric Architecture Guide
9.1.3.3 Virtual Processors and Fabric DensityProviding sufficient CPU density is a matter of identifying the required number of physical cores to
serve as virtual processors for the fabric. When performing capacity planning, it is proper to plan
against cores and not hyperthreads or symmetric multithreads (SMT). Although SMT can boost
performance (approximately 10 to 20 percent), SMT threads are not equivalent to cores.A minimum of two CPU sockets is required for product line architecture (PLA) pattern configurations.
Combined with a minimum 6-core CPU model, 12 physical cores per scale unit host are available to
the virtualization layer of the fabric. Most modern server class processors support a minimum of six
cores, with some families supporting up to 10 cores. Given the average virtual machine requirement
of two virtual CPUs (vCPUs), a two-socket server that has midrange six-core CPUs provides 12 logical
CPUs. This provides a potential density of between 96 and 192 virtual CPUs on a single host.
As an example, table 13 outlines the estimated virtual machine density based on a two-socket, six-
core processor that uses a pre-determined processor ratio. Note, however, that CPU ration
assumptions are highly dependent on workload analysis and planning and should be factored into
any calculations. For this example, the processor ratio would have been defined through workload
testing and an estimation of potential density, and required reserve capacity, could then be
calculated.
Nodes Sockets Cores TotalCores
LogicalCPU
pCPU/vCPURatio
AvailablevCPU
AveragevCPU
Workload
EstimatedRawVirtualMachineDensity
VirtualMachineDensityLessReserveCapacity
1 2 6 12 12 8 96 2 48 N/A
4 2 6 12 48 8 384 2 192 144
8 2 6 12 96 8 768 2 384 336
12 2 6 12 144 8 1152 2 576 528
16 2 6 12 192 8 1536 2 768 720
32 2 6 12 384 8 3072 2 1536 1344
64 2 6 12 768 8 6144 2 3072 2688
1 2 8 16 16 8 128 2 64 N/A
4 2 8 16 64 8 512 2 256 192
8 2 8 16 128 8 1024 2 512 448
12 2 8 16 192 8 1536 2 768 704
16 2 8 16 256 8 2048 2 1024 960
32 2 8 16 512 8 4096 2 2048 1792
64 2 8 16 1024 8 8192 2 4096 3584
1 2 10 20 20 8 160 2 80 N/A
4 2 10 20 80 8 640 2 320 240
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 73/112
Fabric Architecture Guide 73
8 2 10 20 160 8 1280 2 640 560
12 2 10 20 240 8 1920 2 960 880
16 2 10 20 320 8 2560 2 1280 1200
32 2 10 20 640 8 5120 2 2560 2240
64 2 10 20 1280 8 10240 2 5120 4480
Table 13 Example virtual machine density chart
Note that there are also supported numbers of virtual processors in a specific Hyper-V guest
operating system. Improvements to the integration services for Hyper-V are periodically released,
adding support for additional operating systems. For more information, please see Hyper-V Overview
on Microsoft TechNet.
10 Fabric and Fabric Management
The PLA patterns at a high level include the concepts of compute, storage, and network fabrics. This islogically and physically independent from components, such as the components in System
Center 2012, that provide management of the underlying fabric.
Figure 23 : Fabric and fabric management components
10.1.1 FabricThe definition of the fabric is all of the physical and virtual resources under the scope of management
of the fabric management infrastructure. The fabric is typically the entire compute, storage, and
network infrastructure —usually implemented as Hyper-V host clusters —being managed by the
System Center infrastructure.
For private cloud infrastructures, the fabric constitutes a resource pool that consists of one of more
scale units. In a modular architecture, the concept of a scale unit refers to the point to which a module
in the architecture can scale before another module is required. For example, an individual server is a
scale unit, because it can be expanded to a certain point in terms of CPU and RAM; however, once it
reaches its maximum scalability, an additional server is required to continue scaling. Each scale unit
Fabric Management(System Center)
Fabric(Hyper-V/Compute/Storage/Network)
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 74/112
74 Fabric Architecture Guide
also has an associated amount of physical installation and configuration labor. With large scale units,
like a preconfigured full rack of servers, the labor overhead can be minimized.
It is critical to know the scale limits of all hardware and software components when determining the
optimum scale units for the overall architecture. Scale units enable the documentation of all the
requirements (for example, space, power, HVAC, or connectivity) needed for implementation.
10.1.2 Fabric ManagementFabric management is the concept of treating discrete capacity pools of servers, storage, and
networks as a single fabric. The fabric is then subdivided into capacity clouds, or resource pools, which
carry characteristics like delegation of access and administration, service-level agreements (SLAs), and
cost metering. Fabric management enables the centralization and automation of complex
management functions that can be carried out in a highly standardized, repeatable fashion to increase
availability and lower operational costs.
10.1.2.1 Fabric Management Host ArchitectureIn a private cloud infrastructure, it is recommended that the systems that make up the fabric resource
pools be physically separate from the systems that provide fabric management. Much like the concept
of having a top-of-rack (ToR) switch, it is recommended to provide separate fabric management hosts
to manage the underlying services that provide capacity to the private cloud infrastructure. This
model helps make sure that the availability of the fabric is separated from fabric management, and
regardless of the state of the underlying fabric resource pools, management of the infrastructure and
its workloads is maintained at all times.
To support this level of availability and separation, private cloud architectures should contain a
separate set of hosts (minimum of two) configured as a failover cluster in which the Hyper-V role isenabled. Furthermore, these hosts should contain high availability virtualized instances of the
management infrastructure (System Center) to support fabric management operations that are stored
on dedicated CSVs.
11 Non-Converged Architecture PatternThis section contains an architectural example that is based on the non-converged pattern validation
requirements that were outlined in the previous sections. This example provides guidance about the
hardware that is required to build the non-converged pattern reference architecture by using high-
level, non-OEM –specific system models.
As explained earlier, the non-converged pattern comprises traditional blade or non-blade servers that
utilize a standard network and storage-network infrastructure to support a high availability Hyper-V
failover-cluster fabric infrastructure. This infrastructure pattern provides the performance of a large-
scale Hyper-V host infrastructure and the flexibility of utilizing existing infrastructure investments at a
lower cost than a converged architecture.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 75/112
Fabric Architecture Guide 75
Figure 24 outlines a logical structure of components that follow this architectural pattern.
Figure 24 Non-converged architecture pattern
11.1.1 ComputeThe compute infrastructure is one of the primary elements that must scale to support a large number
of workloads. In a non-converged fabric infrastructure, a set of hosts that have the Hyper-V role
enabled provide the fabric with the capability to achieve scale in the form of a large-scale failover
cluster.Figure 25 provides an overview of the compute layer of the private cloud fabric infrastructure.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 76/112
76 Fabric Architecture Guide
Figure 25 Compute minimum configuration
11.1.1.1 Hyper-V Host InfrastructureThe server infrastructure is comprised of a minimum of four hosts and a maximum of 64 hosts in a
single Hyper-V failover-cluster instance. Although Windows Server 2012 failover clustering supports a
minimum of two nodes, a configuration at that scale does not provide a sufficient reserve capacity to
achieve cloud attributes such as elasticity and resource pooling.
As with any failover-cluster configuration, reserve capacity must be accounted for in the host
infrastructure. Adopting a simple n-1 methodology does not always provide a sufficient amount of
reserve capacity to support the workloads that are running on the fabric infrastructure. For true
resilience to outages, we recommend that you size the reserve capacity within a single scale unit to
one or more hosts. This is critical for delivering availability within a private cloud infrastructure and it
is a key consideration when you are advertising the potential workload capacity of the fabric
infrastructure.
Equally important to the overall density of the fabric is the amount of physical memory that is
available for each fabric host. For enterprise configurations, a minimum of 192 GB of memory is
required. As the demand for memory within workloads increases, this becomes the second largestfactor for scale and density in the compute fabric architecture.
As discussed earlier, Hyper-V provides Dynamic Memory to support higher densities of workloads
through a planned oversubscription model. Although it is safe to assume that this feature will provide
increased density for the fabric, a private cloud infrastructure should carefully consider the use of
Hyper-V Dynamic Memory as part of the compute design due to supportability limitations and
performance requirements in certain workloads. Always refer to the vendor workload
recommendations and support guidelines when enabling Hyper-V Dynamic Memory.
Additional considerations that should be accounted for in density calculations include:
The amount of startup RAM that is required for each operating system
The minimum RAM that is allocated to the virtual machine after startup for normal operations
The maximum RAM that is assigned to the system to prevent oversubscription scenarios
when memory demand is high
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 77/112
Fabric Architecture Guide 77
The Hyper-V parent partition (host) must have sufficient memory to provide services such as I/O
virtualization, snapshot, and management to support the child partitions (guests). Previous guidance
was provided to tune the parent partition reserve; however; when Dynamic Memory is used, root
reserve is calculated automatically (based on the root physical memory and NUMA architecture of the
hosts) and no longer requires manual configuration.
Although guidance about network connectivity that uses onboard network connections is provided in
the following section, you should help make sure that out-of-band (OOB) network-management
connectivity is provided to support the remote management and provisioning capabilities that are
found within System Center. To address these capabilities, the compute infrastructure should support
a minimum of one OOB management interface, with support for Intelligent Platform Management
Interface (IPMI) 1.5/Data Center Management Interface (DCMI) 1.0 or Systems Management
Architecture for Server Hardware (SMASH) 1.0 over WS-Man. Failure to include this component will
result in a compute infrastructure that cannot utilize automated provisioning and management
capabilities in the private cloud solution.It should be assumed that customers will also require multiple types (or classifications) of resource
pools to support a number of scenarios and associated workloads. These types of resource pool are
expected to be evaluated as part of the capabilities that the resulting fabric will be required to
provide. For example, a resource pool that is intended for VDI resources might have different
hardware, such as specialized graphics cards, to support RemoteFX capabilities within Hyper-V. For
these reasons, options for a compute infrastructure that provide advanced resource pool capabilities,
such as the RemoteFX resource pool, should be available to address these needs and provide a
complete solution.
11.1.2 NetworkWhen you are designing the fabric network for the Hyper-V failover cluster in Windows Server 2012, it
is important to provide the necessary hardware and network throughput to provide resiliency and
quality of service (QoS). Resiliency can be achieved through availability mechanisms, and QoS can be
provided through dedicated network interfaces or through a combination of hardware and software
QoS capabilities.
Figure 26 provides an overview of the network layer of the private cloud fabric infrastructure.
Figure 26 Network minimum configuration
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 78/112
78 Fabric Architecture Guide
11.1.2.1 Host ConnectivityDuring the design of the network topology and associated network components of the private cloud
infrastructure, the following key considerations apply:
Provide adequate network port density: Designs should contain top-of-rack switches with
sufficient density to support all host network interfaces.
Provide adequate interfaces to support network resiliency: Designs should contain a sufficient
number of network interfaces to establish redundancy through NIC Teaming.
Provide network quality of service: Although dedicated cluster networks is an acceptable way
to achieve quality of service, utilizing high-speed network connections in combination with
hardware- or software-defined network QoS policies provides a more flexible solution.
For PLA pattern designs, a minimum of two 10-GbE network interfaces and one OOB management
connection is assumed a minimum baseline of network connectivity for the fabric architecture. Two
interfaces are used for cluster traffic, and the third is available as a management interface. To provideresiliency, additional interfaces can be added and teamed by using the NIC Teaming feature in
Windows Server 2012.
It is recommended to have redundant network communication between all private cloud cluster
nodes. As previously described, host connectivity in a private cloud infrastructure should support the
following types of communication that are required by the Hyper-V clusters that make up the fabric:
Host management
Virtual machine
Live migration
iSCSI
Intracluster communication and CSV
Host management consists of isolated network traffic to manage the parent partition (host), and
virtual machine traffic is on an accessible network for clients to access the virtual machines. The usage
of the virtual machine traffic is highly dependent on the running workload and the interaction of the
client with that application or service.
Live migration traffic is intermittent and used during virtual machine mobility scenarios such as
planned failover events. This has the potential to generate a large amount of network traffic over
short periods during transition between nodes. Live migration will default to the second lowest metric
if three or more networks are configured in failover clustering. When iSCSI is used, a dedicated
storage network should be deployed within the fabric. These interfaces should be disabled for cluster
use, because cluster traffic can contribute to storage latency. Intracluster communication and CSV
traffic consist of the following traffic types:
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 79/112
Fabric Architecture Guide 79
Network health monitoring
Intracluster communication
CSV I/O redirection
Network health monitoring traffic consists of heartbeats that are sent to monitor the health status ofnetwork interfaces in a full mesh manner. This lightweight unicast traffic (approximately 134 bytes) is
sent between cluster nodes over all cluster-enabled networks. Because of its sensitivity to latency,
bandwidth is important, as opposed to quality of service, because if heartbeat traffic becomes
blocked due to network saturation, fabric nodes could be removed from cluster membership. By
default, nodes exchange these heartbeats every one second, and they are considered to be down if
they do not respond to five heartbeats.
Intracluster communication is variable (based on workload), and it is responsible for sending database
updates and state synchronization between the nodes in the fabric cluster. This lightweight traffic
communicates over a single interface. As with network health monitoring, bandwidth is the primaryconcern, because this type of traffic is sensitive to latency during state changes, such as failover.
CSV I/O redirection traffic consists of lightweight metadata updates, and it can communicate over the
same interface as intracluster communication mentioned previously. It requires a defined quality of
service to function properly. CSV routes I/O over the network between nodes over SMB during
failover events, so sufficient bandwidth is required to handle the forwarded I/O between cluster
nodes. Additionally, CSV traffic will utilize SMB multichannel and advanced network adapter
capabilities such as RDMA; however, use of Jumbo Frames has shown little increase in performance.
11.1.3 StorageStorage provides the final component for workload scaling, and as for any workload, storage must be
designed properly to provide the required performance and capacity for overall fabric scale. In a non-
converged fabric infrastructure, traditional SAN infrastructures that are connected over Fibre Channel,
or iSCSI provide fabric with sufficient capacity to achieve storage scale.
Figure 27 provides an overview of the storage infrastructure for the non-converged pattern.
Figure 27 Storage minimum configuration
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 80/112
80 Fabric Architecture Guide
11.1.3.1 Storage ConnectivityFor the operating system volume of the parent partition that is using direct attached storage to the
host, an internal Serial Advanced Technology Attachment (SATA) or SAS controller is required, unless
the design utilizes SAN for all system storage requirements, including boot from SAN for the host
operating system. (Fibre Channel and iSCSI boot are supported in Windows Server 2012.) Dependingon the storage protocol and devices that are used in the non-converged storage design, the following
adapters are required to allow shared storage access:
If using Fibre Channel SAN, two or more host bus adapters (HBAs)
If using iSCSI, two or more 10 GbE network adapters or HBAs
As described earlier, Hyper-V in Windows Server 2012 supports the ability to present SAN storage to
the guest workloads that are hosted on the fabric infrastructure by using virtual Fibre Channel
adapters. Virtual SANs are logical equivalents of virtual network switches within Hyper-V, and each
Virtual SAN maps to a single physical Fibre Channel uplink. To support multiple HBAs, a separateVirtual SAN must be created per physical Fibre Channel HBA and mapped exactly to its corresponding
physical topology.
When configurations use multiple HBAs, MPIO must be enabled within the virtual machine workload.
A virtual SAN assignment should follow a similar pattern as a Hyper-V virtual switch assignment, in
that if there are different classifications of service within the SAN, it should be reflected within the
fabric.
As discussed in earlier sections, all physical Fibre Channel equipment must support NPIV. Hardware
vendors must also provide WHQL-certified drivers for all Fibre Channel HBAs, unless the WHQL
drivers are provided in Windows Server 2012. If zoning that is based on physical Fibre Channel switchports is part of the fabric design, all physical ports must be added to allow for virtual machine
mobility scenarios across hosts in the fabric cluster. Although virtual machines can support iSCSI boot,
boot from SAN is not supported over the virtual Fibre Channel adapter and should not be considered
as part of workload design.
11.1.3.2 Storage InfrastructureThe key attribute of the storage infrastructure for the non-converged pattern is the use of a
traditional SAN infrastructure to provide access to storage to the fabric, fabric management, and
workload layers. As discussed earlier, the primary reasons to adopt or maintain this design are to
preserve existing investments in SAN or to maintain the current level of flexibility and capabilities that
a SAN-based storage-array architecture provides.
For Hyper-V failover cluster and workload operations in a non-converged infrastructure, the fabric
components utilize the following types of storage:
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 81/112
Fabric Architecture Guide 81
Operating system : Non-shared physical boot disks (DAS or SAN) for the fabric managementhost servers
Cluster witness : Shared witness disk or file share to support the failover cluster quorum Cluster Shared Volumes (CSV) : One or more shared CSV LUNs for virtual machines (Fibre
Channel or iSCSI), as presented by the SAN Guest clustering [optional]: Shared Fibre Channel or iSCSI LUNs for guest clustering
Figure 28 provides a conceptual view of the storage architecture for the non-converged pattern.
Figure 28 Non-converged architecture pattern
As outlined in the overview, fabric and fabric management host controllers require sufficient storage
to account for the operating system and paging files. To help determine the optimal page file
configuration for your environment, the article entitled How to determine the appropriate page file
size for 64-bit versions of Windows provides detailed guidance. However in Windows Server 2012, we
recommend that virtual memory be configured as ―Automatically manage paging file size for all
drives. ‖
Although boot from SAN from Fibre Channel or from iSCSI storage is supported in Windows
Server 2012, it is widely accepted to have onboard storage configured locally per server to provide
these capabilities for each server given the configuration of standard non-converged servers. In these
FC HBA FC HBA
Hyper-V Failover Cluster Node 1
Cluster Shared Volumes (CSV v2) + CSV Cache
VHDs
FCPort
FCPort
FCPort
FCPort
FC HBA FC HBA
FCPort
FCPort
FCPort
FCPort
VHDs
Hyper-V Failover Cluster Node n
SAN Disk Array
SANController
SANController
10Gb-EPort
10Gb-EPort
10Gb-EPort
10Gb-EPort
Fibre Channel Infrastructure
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 82/112
82 Fabric Architecture Guide
cases, local storage should include two disks that are configured as RAID —one mirror (as a minimum)
with an optional global hot spare is sufficient.
To provide quorum for the server infrastructure, it is recommended to utilize a quorum configuration
of Node and Disk Majority. To support this, a cluster witness disk is required to support this quorum
model. In non-converged pattern configurations, it is recommended that a 1 GB disk witnessformatted as NTFS be provided for all fabric and fabric management clusters to provide resiliency and
prevent partition in time scenarios within the cluster.
As described in earlier sections, Windows Server 2012 provides multiple host access to a shared disk
infrastructure through CSV 2.0. For non-converged patterns, the SAN should be configured to provide
adequate storage for virtual machine workloads. Given that workload, virtual disks often exceed
multiple gigabytes, so it is recommended that where supported by the workload, dynamically
expanding disks be used to provide higher density and more efficient use of storage. Additional SAN
capabilities such as thin provisioning of LUNs can assist with the consumption of physical space;
however, this functionality should be evaluated to help make sure that workload performance is notadequately affected.
CSVs must be configured in Windows as a basic disk formatted as NTFS (FAT, FAT32, and ReFS are not
supported for CSV), cannot be used as a witness disk, and cannot have Windows data deduplication
enabled. A CSV has no restrictions in the number of virtual machines that it can support on an
individual CSV volume because metadata updates on a CSV volume are orchestrated on the server
side, and they run in parallel to provide no interruption and increased scalability.
Performance considerations fall primarily on the IOPS that SAN provides, given that multiple servers
from the Hyper-V failover cluster stream I/O to a commonly shared LUN. Providing more than one
CSV to the Hyper-V failover cluster within the fabric can increase performance, depending on the SAN
configuration.
To support guest clustering, LUNs can be presented to the guest operating system through iSCSI or
Fibre Channel. Configurations for the non-converged pattern should include sufficient space on the
SAN to support the number of LUNs needed for workloads with high-availability requirements that
must be satisfied within the guest virtual machines and associated applications.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 83/112
Fabric Architecture Guide 83
12 Converged Architecture PatternThis section contains an architectural example that is based on the converged-pattern validation
requirements that were previously outlined. This example will provide guidance on the hardware that
is required to build the converged pattern reference architecture by using high-level, non-OEM –
specific system models.
As explained earlier, the converged pattern comprises advanced blade servers that utilize a
converged-network and storage-network infrastructure (often referred to as converged-network
architecture) to support a highly available Hyper-V failover-cluster fabric infrastructure. This
infrastructure pattern provides the performance of a large-scale Hyper-V host infrastructure and the
flexibility of utilizing software-defined networking capabilities at a higher system density than can be
achieved through traditional non-converged architectures.
Although many aspects of converged architectures are the same, this section will outline the key
differences between these two patterns. The following diagram outlines an example logical structureof components that follow the converged architectural pattern.
Figure 29 Converged architecture pattern
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 84/112
84 Fabric Architecture Guide
In the converged pattern, the physical converged-network adapters (CNAs) are teamed and present
NICs and Fibre Channel HBAs to the parent operating system. From the perspective of the parent
operating system, it appears that NICs and Fibre Channel HBAs are installed. The configuration of
teaming and other settings are performed at the hardware level.
12.1.1 ComputeAs identified in the non-converged pattern, compute infrastructure remains as the primary element
that provides fabric scale to support a large number of workloads. Identical to the non-converged
pattern, the converged fabric infrastructure consists of an array of hosts that have the Hyper-V role
enabled to provide the fabric with the capability to achieve scale in the form of a large-scale failover
cluster.
The diagram showed in Figure 30 provides an overview of the compute layer of the private cloud
fabric infrastructure.
Figure 30 Compute minimum configuration
With the exception of storage connectivity, the compute infrastructure of the converged pattern is
similar to the infrastructure of the non-converged pattern, because the Hyper-V host clusters utilize
FCoE or iSCSI to connect to storage over a high-speed converged-network architecture.
12.1.1.1 Hyper-V Host InfrastructureAs in non-converged infrastructures, the server infrastructure comprises a minimum of four hosts and
a maximum of 64 hosts in a single Hyper-V failover-cluster instance. Although Windows Server 2012
failover clustering supports a minimum of two nodes, a configuration at that scale does not provide
sufficient reserve capacity to achieve cloud attributes such as elasticity and resource pooling.
Converged infrastructures typically utilize blade servers and enclosures to provide compute capacity.
In large-scale deployments in which multiple resource pools exist across multiple blade enclosures, a
guideline of containing no more than 25 percent of a single cluster in a blade enclosure is
recommended.
12.1.2 NetworkWhen you are designing the fabric network for the Windows Server 2012 Hyper-V failover cluster, it is
important to provide the necessary hardware and network throughput to provide resiliency and
quality of service (QoS). Resiliency can be achieved through availability mechanisms, while QoS can be
COMPUTEMinimum Configuration:- Servers: 4-64 Servers (Cluster nodes)- CPU: Dual Socket: Minimum 6 core per socket, 12 cores total- RAM: Minimum of 192 GB RAM per node- On-Board Storage: Minimum of 2 - 300GB Local HDD (none if boot from SAN is utilized)- Host Connectivity:
- Storage and network connectivity as outlined in the Storage and Network sections below- 1 dedicated out-of-Band (OOB) management interface (IPMI/DCMI or SMASH over WS-Man)
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 85/112
Fabric Architecture Guide 85
provided either through dedicated network interfaces or through a combination of hardware and
software QoS capabilities.
The diagram showed in Figure 31 provides an overview of the network layer of the private cloud fabric
infrastructure.
Figure 31 Network minimum configuration
12.1.2.1 Host ConnectivityDuring the design of the network topology and associated network components of the private cloud
infrastructure, the following key considerations apply:
Provide adequate network port density —Designs should contain top-of-rack switches withsufficient density to support all host network interfaces.
Provide adequate interfaces to support network resiliency —Designs should contain asufficient number of network interfaces to establish redundancy through NIC teaming.
Provide network quality of service — Having dedicated cluster networks is an acceptableway to achieve QoS, however the use of high-speed network connections in combination witheither hardware-defined or software-defined network QoS policies provide a more flexiblesolution.
For PLA pattern designs, a minimum of two 10-GbE converged-network adapters (CNAs) and one
OOB management connection is assumed a minimum baseline of network connectivity for the fabric
architecture. Two interfaces are used for cluster traffic, and the third is available as a management
interface. To provide resiliency, additional interfaces can be added and teamed using the OEM
hardware NIC teaming solution. It is recommended to have redundant network communication
between all private cluster nodes. As previously described, host connectivity in a private cloud
infrastructure should support the following types of communication that are required by the Hyper-V
clusters that make up the fabric:
Host management Virtual machine Live migration FCoE or iSCSI Intra-cluster communication and CSV
In a converged-network architecture, LAN and storage traffic utilize Ethernet as the transport. Fibre
Channel and iSCSI are possible choices for the converged-infrastructure pattern. Although CA over
NETWORKMinimum Configuration:- Network Switch Infrastructure: Sufficient 10GbE connectivity/port density to support connectivity for all hosts with hostand switch redundancy, support for VLANs (tagging, trunking, etc)- Host Network Connectivity
- Minimum 2 converged network adapters (CNAs) per host
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 86/112
86 Fabric Architecture Guide
SMB could also be considered a converged architecture, it is broken out into a separate design
pattern.
The converged pattern refers to either FCoE or iSCSI approaches. Proper network planning is critical in
a converged design. Use of quality of service (QoS), VLANs, and other isolation or reservation
approaches is strongly recommended, so that storage and LAN traffic is appropriately balanced.
12.1.3 StorageStorage provides the final component for workload scaling and, as with any workload, must be
designed properly to provide the required performance and capacity for overall fabric scale. In a
converged fabric infrastructure, connectivity to the storage uses an Ethernet-based approach such as
iSCSI or FCoE.
The diagram showed in Figure 32 provides an overview of the storage infrastructure for the
converged pattern.
Figure 32 Network minimum configuration 12.1.3.1 Storage Connectivity
For the operating system volume of the parent partition that is using direct attached storage to the
host, an internal SATA or SAS controller is required, unless the design utilizes SAN for all system-
storage requirements, including boot from SAN for the host operating system. (Both Fibre Channel
and iSCSI boot are now supported in Windows Server 2012.) Depending on the storage protocol and
devices that are used in the converged storage design, the following adapters are required to allow
shared storage access:
If using Fibre Channel SAN, two or more converged-network adapters (CNAs) If using iSCSI, two or more 10-gigabit (GB) Ethernet NICs or iSCSI HBAs
As described earlier, Windows Server 2012 Hyper-V supports the ability to present SAN storage to the
guest workloads that are hosted on the fabric infrastructure by using virtual Fibre Channel adapters.
Virtual SANs are logical equivalents of virtual network switches within Hyper-V, and each Virtual SANmaps to a single physical Fibre Channel uplink. To support multiple CNAs, a separate Virtual SAN
must be created per physical Fibre Channel CNA and mapped exactly to its corresponding physical
topology. When configurations use multiple CNAs, MPIO must be enabled within the virtual machine
workload itself. Virtual SAN assignment should follow a similar pattern as Hyper-V virtual switch
assignment in that, if there are different classifications of service within the SAN, it should be reflected
within the fabric.
STORAGEMinimum Configuration:- Array: Shared storage array with cloud Fabric, with capacity to support all Fabric and F abric Management workloads
(2 TB minimum reserve for Fabric Management plus additional capacity for Fabric workloads)- Storage Connectivity: Sufficient connectivity for all hosts with host and switch redundancy
- Minimum 2 converged network adapters (CNAs) per host
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 87/112
Fabric Architecture Guide 87
As discussed in earlier sections, all physical Fibre Channel equipment must support NPIV. Hardware
vendors must also provide Windows Server 2012 WHQL-certified drivers for all Fibre Channel CNAs,
unless WHQL drivers are provided in-box. If zoning that is based on physical Fibre Channel switch
ports is part of the fabric design, all physical ports must be added to allow for virtual machine
mobility scenarios across hosts in the fabric cluster. Although virtual machines can support iSCSI boot,
boot from SAN is not supported over the virtual Fibre Channel adapter and should not be considered
as part of workload design.
12.1.3.2 Storage InfrastructureThe key attribute of the storage infrastructure for the converged pattern is the use of a traditional
SAN infrastructure but accessed through an Ethernet transport for the fabric, fabric management, and
workload layers. As discussed earlier, the primary reason to adopt or maintain this design is either to
preserve existing investments in SAN or to maintain the current level of flexibility and capabilities that
a SAN-based storage-array architecture provides, while consolidating to a single network
infrastructure: Ethernet.
For Hyper-V failover-cluster and workload operations in a converged infrastructure, the fabric
components utilize the following types of storage:
Operating system —Non-shared physical boot disks (DAS or SAN) for the fabricmanagement host servers (unless using boot from SAN)
Cluster witness —Shared witness disk or file share to support the failover-cluster quorum Cluster Shared Volumes (CSV) —One or more shared CSV LUN(s) for virtual machines (Fibre
Channel or iSCSI), as presented by the SAN Guest clustering [optional] —Shared Fibre Channel or iSCSI LUNs for guest clustering
The diagram showed in Figure 33 provides a conceptual view of this architecture for the converged
pattern.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 88/112
88 Fabric Architecture Guide
Figure 33 Converged architecture pattern
As outlined in the overview, fabric and fabric management host controllers require sufficient storage
to account for the operating system and paging files. However in Windows Server 2012, we
recommend that virtual memory be configured as ―Automatically manage paging file size for alldrives.‖
Although boot from SAN from either Fibre Channel and iSCSI storage is supported in Windows
Server 2012, it is widely accepted to have onboard storage configured locally per server to provide
these capabilities for each server, given the configuration of standard non-converged servers. In these
cases, local storage should include two disks that are configured as RAID 1 (mirror) as a minimum,
with an optional global hot spare being sufficient.
To provide quorum for the server infrastructure, it is recommended to utilize a quorum configuration
of Node and Disk Majority. To support this, a cluster witness disk is required to support this quorum
model. In converged pattern configurations, it is recommended that a 1-GB disk witness formatted as
NTFS be provided for all fabric and fabric management clusters to provide resiliency and prevent
partition in time scenarios within the cluster.
As described in earlier sections, Windows Server 2012 provides multiple-host access to a shared disk
infrastructure through CSV version 2. For converged patterns, the SAN should be configured to
provide adequate storage for virtual machine workloads. Given that workload virtual disks often
CNA CNA
Hyper-V Failover Cluster Node 1
Cluster Shared Volumes (CSV v2) + CSV Cache
VHDs
CNAPort
CNAPort
CNAPort
CNAPort
CNA CNA
CNAPort
CNAPort
CNAPort
CNAPort
VHDs
Hyper-V Failover Cluster Node n
SAN Disk Array
FC/iSCSI Gateway /SAN Controller
FC/iSCSI Gateway /SAN Controller
10Gb-EPort
10Gb-EPort
10Gb-EPort
10Gb-EPort
Ethernet Infrastructure
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 89/112
Fabric Architecture Guide 89
exceed multiple gigabytes, it is recommended that —where it is supported by the workload —
dynamically expanding disks be used to provide higher density and more efficient use of storage.
Additional SAN capabilities such as thin provisioning of LUNs can assist with the consumption of
physical space; however, this functionality should be evaluated to help make sure that workload
performance is not adequately affected.
A CSV must be configured in Windows as a basic disk formatted as NTFS (FAT, FAT32, and ReFS are
not supported for CSV), and it cannot be used as a witness disk or have Windows data deduplication
enabled. A CSV has no restrictions in the number of virtual machines that it can support on an
individual CSV volume, as metadata updates on a CSV volume are orchestrated on the server side and
parallelized for no interruption and increased scalability. Performance considerations fall primarily on
the IOPS that the SAN provides, given that multiple servers from the Hyper-V failover-cluster stream
I/O to a commonly shared LUN. Providing more than one CSV to the Hyper-V failover cluster within
the fabric can increase performance, depending on the SAN configuration.
To support guest clustering, LUNs can be presented to the guest operating system through iSCSI orFibre Channel. Configurations for the converged pattern should include sufficient space on the SAN to
support a small number of LUNs to support workloads with high availability requirements that must
be satisfied within the guest virtual machines and associated applications.
13 Continuous Availability over SMB Storage Architecture PatternKey attributes of the Continuous Availability over SMB Storage pattern include the use of the SMB 3.0
protocol, and in the case of Variation A, the implementation of the new Scale-Out File Server cluster
design pattern in Windows Server 2012.
This section outlines a finished example of a Continuous Availability over SMB Storage design that
uses Variation A. As illustrated previously, the following diagram shows the high-level architecture.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 90/112
90 Fabric Architecture Guide
SAS disks
SAScontroller
SAScontroller
Scale-Out File-Server cluster
Storage Spaces
Cluster Shared Volumes(CSV2) + CSV cache
RDMANIC
RDMANIC
Hyper-V Host cluster(s)
RDMANIC
RDMANIC
SMB3 DirectSMB3 Transparent Failover
SMB3 Multichannel
EthNIC
EthNIC
NICteaming
Hyper-Vextensible
switch
VMs
VMs
VHDs
LAN
Figure 34 Continuous availability over SMB Storage architecture pattern
The design consists of one or more Windows Server 2012 Scale-Out File Server clusters (left)
combined with one or more Hyper-V host clusters (right). In this sample design, a shared SAS storage
architecture is utilized by the Scale-Out File Server clusters, and the Hyper-V hosts store their virtualmachines on SMB shares on the file cluster, built on top of storage spaces and Cluster Shared
Volumes.
A key choice in the Continuous Availability over SMB pattern is whether to use InfiniBand or Ethernet
as the network transport between the Hyper-V clusters and the Scale-Out File Server clusters.
Currently, InfiniBand provides higher speeds per port than Ethernet (56 Gbps for InfiniBand, compared
to 10 or 40 GbE), but it requires a separate switching infrastructure, whereas a purely Ethernet-based
approach can utilize a single physical network infrastructure.
13.1.1
ComputeThe compute infrastructure is one of the primary elements that provides fabric scale to support a
large number of workloads. In a Continuous Availability over SMB fabric infrastructure, an array of
hosts that have the Hyper-V role enabled provide the fabric with the capability to achieve scale in the
form of a large-scale failover cluster.
Figure 35 provides an overview of the compute layer of the private cloud fabric infrastructure.
Figure : Compute minimum configuration
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 91/112
Fabric Architecture Guide 91
With the exception of storage connectivity, the compute infrastructure of the Continuous Availability
over SMB pattern is similar to the infrastructure of the converged and non-converged patterns,
because the Hyper-V host clusters utilized SMB over Ethernet or InfiniBand to connect to storage.
13.1.1.1 Hyper-V Host InfrastructureThe server infrastructure comprises a minimum of four hosts and a maximum of 64 hosts in a singleHyper-V failover cluster instance. Although a minimum of two nodes is supported by Windows
Server 2012 failover clustering, a configuration at that scale does not provide sufficient reserve
capacity to achieve cloud attributes such as elasticity and resource pooling.
Note The same sizing and availability guidance that is provided in the Hyper-V Host Infrastructure
subsection (in the Non-Converged Architecture Pattern section) applies to this pattern.
Figure 36 provides a conceptual view of this architecture for the continuous availability pattern.
Figure : Continuous availability pattern
10Gb-E
10Gb-E
NIC Teaming / LBFO / VLAN Trunk(s)
Hyper-V
LiveMigration
Cluster /CSV
VM VLANTrunk
iSCSI
Virtual Switches
OOB/Base
BoardMgmt
Non-Clustered VM
Guest ClusterNode
Guest ClusterNode
VLAN TrunkVM LAN(s), LM, CSV, iSCSI, Host Mgmt
Parent Partition (Host OS)
Scale-Out File Clusterwith Shared SAS JBOD
10Gb-E with RDMA
10Gb-E with RDMA
SMB Multi-Channel / Transparent Failover
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 92/112
92 Fabric Architecture Guide
A key factor in the computer infrastructure is a determination from the storage design as to whether
Ethernet or InfiniBand will be utilized as the transport between the Hyper-V host clusters and the
Scale-Out File Server clusters. The other consideration is whether RDMA (recommended) will be
supported by the design.
As outlined in previous sections, RDMA cannot be used in conjunction with NIC Teaming. Therefore,in this design, which utilizes a 10 GbE network fabric, each Hyper-V host server in the compute layer
contains four 10-GbE network adapters. One pair is for virtual machine and cluster traffic, and it
utilizes NIC Teaming. The other pair is for storage connectivity to the Scale-Out File Server clusters,
and it is RDMA-capable.
13.1.2 NetworkWhen you are designing the fabric network for the Windows Server 2012 Hyper-V failover cluster, it is
important to provide the necessary hardware and network throughput to provide resiliency and
quality of service (QoS). Resiliency can be achieved through availability mechanisms, while QoS can be
provided through dedicated network interfaces or through a combination of hardware and software
QoS capabilities.
Figure 37 provides an overview of the network layer of the private cloud fabric infrastructure.
Figure 35 Network minimum configuration
13.1.2.1 Host ConnectivityWhen you are designing the network topology and associated network components of the
private cloud infrastructure, certain key considerations apply. You should provide:
NETWORKMinimum Configuration:- Network Switch Infrastructure: Sufficient 1GbE or 10GbE connectivity/port density for all hosts withhost and switch redundancy, support for VLANs (tagging, trunking, etc) and support for RDMA.- Host Network Connectivity:
- If 10 GbE – 2 or 4 Dedicated Connections plus 1 Out of Band management connection- If Infiniband – 2 Infiniband Connections plus 2 10 GbE Connections, plus 1 O ut of Bandmanagement connection- If 1 GbE (not recommended) – 8 Total Dedicated Connections- RDMA required for all 10GbE adapters (if using 1GbE, the adapters for storage connectivity mustbe RDMA capable
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 93/112
Fabric Architecture Guide 93
Adequate network port density : Designs should contain top-of-rack switches that havesufficient density to support all host network interfaces.
Adequate interfaces to support network resiliency : Designs should contain a sufficientnumber of network interfaces to establish redundancy through NIC Teaming.
Network quality of service : Although the use of dedicated cluster networks is an
acceptable way to achieve quality of service, utilizing high-speed network connections incombination with hardware- or software-defined network QoS policies provides a moreflexible solution.
RDMA support : For the adapters (InfiniBand or Ethernet) that will be used for storage(SMB) traffic, RDMA support is required.
The network architecture for the Continuous Availability over SMB pattern is critical because all
storage traffic will traverse a network (Ethernet or InfiniBand) between the Hyper-V host clusters
and the Scale-Out File Server clusters.
13.1.3 Storage13.1.3.1 Storage Connectivity
For the operating system volume of the parent partition that is using direct attached storage to the
host, an internal SATA or SAS controller is required —unless the design utilizes SAN for all system-
storage requirements, including boot from SAN for the host operating system. (Fibre Channel and
iSCSI boot are supported in Windows Server 2012.)
Depending on the storage transport that is utilized for the Continuous Availability over SMB pattern,
the following adapters are required to allow shared storage access:
Hyper-V host clusters:
10 GbE adapters that support RDMA InfiniBand adapters that support RDMA
Scale-Out File Server clusters:
10 GbE adapters that support RDMA InfiniBand adapters that support RDMA SAS Controllers for access to shared SAS storage
The number of adapters and ports that are required for storage connectivity between the Hyper-V
host clusters and the Scale-Out File Server clusters depends on a variety of sizing- and density-
planning factors. The larger the clusters and the higher the number of virtual machines that are to be
hosted, the more bandwidth and IOPS capacity will be required between the clusters.
13.1.3.2 Scale-Out File Server Cluster ArchitectureThe key attribute of Variation A and B of the Continuous Availability over SMB Storage pattern is the
usage of Scale-Out File Server clusters in Windows Server 2012 as the ―front end‖ or access point to
storage. The Hyper-V host clusters that run virtual machines have no direct storage connectivity.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 94/112
94 Fabric Architecture Guide
Instead, they have SMB Direct (RDMA) –enabled network adapters, and they store their virtual
machines on file shares that are presented by the Scale-Out File Server clusters.
For the PLA patterns, there are two options for the Scale-Out File Server clusters that are required for
Variations A and B. The first is the Fast Track ―small‖ SKU, or ―Cluster -in-a- Box,‖ which can serve as the
storage cluster. Any validated ―small‖ SKU can be used as the storage tier for the ―medium‖ IaaS PLAContinuous Availability over SMB Storage pattern. The ―small‖ SKU would have to be combined with
one or more dedicated Hyper-V host clusters for the fabric.
The second option is a larger, dedicated Scale-Out File Server cluster that meets all of the validation
requirements that are outlined in the Continuous Availability over SMB Storage section. Figures 38,
39, and 40 illustrate these options.
Figure 36 Continuous availability options
In the preceding design, a dedicated fabric management cluster and one or more fabric clusters use a
Scale-Out File Server cluster as the storage infrastructure.
Figure 37 Another option for fabric management design
In the preceding design, a dedicated fabric management cluster and one or more fabric clusters use a
Fast Track ―small‖ (or ―Cluster-in-a- Box‖) SKU as the sto rage infrastructure.
Fabric ManagementCluster (2 – 4 nodes)
FabricHyper-V Host Clusters (x 2 – 64 nodes)
Scale-out File Server Cluster(2 – 8 nodes) Shared SAS Storage
Fabric ManagementCluster (2 – 4 nodes)
Fabric
Hyper-V Host Clusters (x 2 – 64 nodes)
Fast Track Small (Cluster in a Box)
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 95/112
Fabric Architecture Guide 95
Figure 38 One or more fabric cluster
In the preceding design, a Fast Track ―small‖ (or ―Cluster -in-a- Box‖) SKU is used as the storage
infrastructure and to run the fabric management virtual machines for one or more fabric clusters.
Fast Track “Small” SKU (“Cluster -in-a- Box”)
As part of the Fast Track program, Microsoft has been working with server industry customers to
create a new generation of simpler, high-availability solutions that deliver small implementations as a―Cluster-in-a- Box‖ or as consolidation appliance solutions at a lower pr ice.
In this scenario, the solution is designed as a storage ―building block‖ for the data center, such as a
dedicated storage appliance. Examples of this scenario are cloud solution builders and enterprise data
centers. For example, suppose that the solution supported Server Message Block (SMB) 3.0 file shares
for Hyper-V or SQL Server. In this case, the solution would enable the transfer of data from the drives
to the network at bus and wire speeds with CPU utilization that is comparable to Fibre Channel.
In this scenario, the file server is enabled in an office environment in an enterprise equipment room
that provides access to a switched network. As a high-performance file server, the solution can
support variable workloads, hosted line-of-business (LOB) applications, and data.
The ―Cluster-in-a- Box‖ design pattern requires a minimum of two clustered server nodes and shared
storage that can be housed within a single enclosure design or a multiple enclosure design, as shown
in Figure 41.
FabricHyper-V Host Clusters (x 2 – 64 nodes)
Fast Track Small (Cluster in a Box) +Fabric Management VMs
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 96/112
96 Fabric Architecture Guide
Figure 39 Cluster-in-box design pattern
Fast Track Medium Scale-Out File Server Cluster
For higher end scenarios in which larger capacity I/O or performance is required, larger multi-node
Scale-Out File Server clusters can be utilized. Higher performing networks (such as 10 GbE or 56 GbE
InfiniBand) between the file cluster and the Hyper-V clusters can be utilized.
The Scale-Out File Server cluster design is scaled out by adding additional file servers to the cluster.
By using CSV 2.0, administrators can create file shares that provide simultaneous access to data files,
with direct I/O, through all nodes in a file-server cluster. This provides better utilization of network
bandwidth and load balancing of the file server clients (Hyper-V hosts).Additional nodes also provide additional storage connectivity, which enables further load balancing
between a larger number of servers and disks.
In many cases, the scaling out of the file server cluster when you use SAS JBOD runs into limits in
terms of how many adapters and individual disk trays can be attached to the same cluster. You can
avoid these limitations and achieve additional scale by using a switched SAS infrastructure, as
described in previous sections.
Figure 42 illustrates this approach. For simplicity, only file-cluster nodes are diagrammed; however,
this could easily be four nodes or eight nodes for scale-out.
NetworkNetwork
CPUCPU
StorageController
StorageController
Shared Storage
Server EnclosureServer A Server B
1 GbE 1 GbE
1 GbE Ethernet Cluster Connect
x8 PCIe x8 PCIe
x8 PCIe x8 PCIe
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 97/112
Fabric Architecture Guide 97
Figure 40 Medium Scale-Out File Server cluster
Highlights of this design include the SAS switches, which allow a significantly larger number of disk
trays and paths between all hosts and the storage. This approach can enable hundreds of disks and
many connections per server (for instance, two or more four-port SAS cards per server).
To have resiliency against the failure of one SAS enclosure, you can use two-way mirroring (minimum
of three disks in the mirror for failover clustering/CSV) and Enclosure Awareness, which requires three
physical enclosures. Two-way mirror spaces must use three or more physical disks, therefore three
enclosures are required to have one disk in each enclosure and to have the storage pool be resilient
to one enclosure failure. For this design, the pool must be configured with the IsEnclosureAware flag,
and the enclosures must be certified to use the Storage Spaces feature in Windows Server 2012.
For enclosure awareness, Storage Spaces leverage the array’s failure and identify/locate lights to
indicate drive failure or a specific drive’s location within the disk tray. The array or enclosure mustsupport SCSI Enclosure Services (SES) 3.0. Enclosure Awareness is independent of an SAS switch or the
number of compute nodes.
This design also illustrates a 10 GbE with RDMA design for the file server cluster to provide high
bandwidth and low latency for SMB traffic. This could also be InfiniBand if requirements dictate that.
Balancing the available storage IO capacity through the SAS infrastructure to the demands of the
Hyper-V clusters that will be utilizing the file cluster for their storage is key to a good design. An
SAS Disks
SAS HBA SAS HBA
Scale-Out File Server Cluster Node
Storage Spaces
Cluster Shared Volumes (CSV v2) + CSV Cache
VHDs
SASPort
SASPort
SASPort
SASPort
SAS HBA SAS HBA
SASPort
SASPort
SASPort
SASPort
VHDs
Scale-Out File Server Cluster Node
SAS Switch SAS Switch
SAS JBOD Array with DualExpander/Dual Port Drives
Storage Pool(s)
SASExpander
SASExpander
SAS Disks
SAS JBOD Array with DualExpander/Dual Port Drives
SASExpander
SASExpander
10Gb-ERDMA Port
10Gb-ERDMA Port
10Gb-ERDMA Port
10Gb-ERDMA Port
SAS Disks
SAS JBOD Array with DualExpander/Dual Port Drives
SASExpander
SASExpander
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 98/112
98 Fabric Architecture Guide
extremely high-performance InfiniBand infrastructure does not make sense if the file servers will have
only two SAS connections to storage.
13.1.3.3 Storage InfrastructureFor Hyper-V failover-cluster and workload operations in a continuous availability infrastructure, the
fabric components utilize the following types of storage:
Operating system : Non-shared physical boot disks (DAS or SAN) for the file servers andHyper-V host servers.
Cluster witness : File share to support the failover cluster quorum for the file server clustersand the Hyper-V host clusters (a shared witness disk is also supported).
Cluster Shared Volumes (CSV) : One or more shared CSV LUNs for virtual machines onStorage Spaces that are backed by SAS JBOD.
Guest clustering [optional] : Requires iSCSI or guest Fibre Channel. For the CA pattern,adding the iSCSI target to the file server cluster nodes can enable iSCSI shared storage for
guest clustering.As outlined in the overview, fabric and fabric management host controllers require sufficient storage
to account for the operating system and paging files. However in Windows Server 2012, we
recommend that virtual memory be configured as ―Automatically manage paging file size for all
drives.‖
Sizing of the physical storage architecture for the CA over SMB pattern is highly dependent on the
quantity and type of virtual machine workloads that are to be hosted.
Given that workload virtual disks often exceed multiple gigabytes, where it is supported by the
workload, it is recommended to use dynamically expanding disks to provide higher density and moreefficient use of storage.
CSVs on the Scale-Out File Server clusters must be configured in Windows as a basic disk that is
formatted as NTFS (FAT, FAT32, and ReFS are not supported for CSV). They cannot be used as a
witness disk, and they cannot have Windows data deduplication enabled.
A CSV has no restrictions in the number of virtual machines that it can support on an individual CSV
volume, because metadata updates on a CSV volume are orchestrated on the server side, and they
run in parallel for no interruption and increased scalability.
Performance considerations fall primarily on the IOPS that the file cluster provides, given that multiple
servers from the Hyper-V failover cluster connect through SMB to a commonly shared CSV on the file
cluster. Providing more than one CSV to the Hyper-V failover cluster within the fabric can increase
performance, depending on the file cluster configuration.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 99/112
Fabric Architecture Guide 99
14 Multi-Tenant DesignsIn many private cloud scenarios, and nearly all hosting scenarios, a multi-tenant infrastructure is
required. This section illustrates how a multi-tenant fabric infrastructure can be created by using
Windows Server 2012 and the technologies described in the fabric architecture guide.
The term ―multi-tenant‖ is fairly general. In general, multi -tenancy implies multiple non-related
consumers or customers of a set of services. Within a single organization, this could be multiple
business units with resources and data that must remain separate for legal or compliance reasons.
Most hosting companies require multi-tenancy as a core attribute of their business model. This might
include a dedicated physical infrastructure for each hosted customer or a logical segmentation of a
shared infrastructure by using software-defined technologies.
14.1.1 Requirements GatheringThe design of a multi-tenant fabric must begin with a careful analysis of the business requirements,
which will drive the design. In many cases, legal or compliance requirements drive the designapproach, which means that a team of several disciplines (for example, business, technical, and legal)
should participate in the requirements gathering phase. If specific legal or compliance regimes are
required, a plan to ensure compliance and ongoing auditing (internal or third party) should be
implemented.
To organize the requirements gathering process, an ―outside in‖ approach can be helpful. For hosted
services, the end customer or consumer is outside of the hosting organization. Requirements
gathering can begin by taking on the persona of the consumer and determining how the consumer
will become aware of and be able to request access hosted services.
Then consider multiple consumers, and ask the following questions:
Will consumers use accounts that the host creates or accounts that they use internally to
access services?
Is one consumer allowed to be aware of other consumer’s identities, or is a separation
required?
Moving further into the ―outside in‖ process, determine whether legal or compliance concerns require
dedicated resources for each consumer:
Can multiple consumers share a physical infrastructure? Can traffic from multiple consumers share a common network?
Can software-defined isolation meet the requirements?
How far into the infrastructure must authentication, authorization, and accounting be
maintained for each consumer (for example, only at the database level, or including the disks
and LUNs that are used by the consumer in the infrastructure)?
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 100/112
100 Fabric Architecture Guide
The following list provides a sample of the types of design and segmentation options that might be
considered as part of a multi-tenant infrastructure:
Physical separation by customer (dedicated hosts, network, storage)
Logical separation by customer (shared physical infrastructure with logical segmentation)
Data separation (such as dedicated databases and LUNs)
Network separation (VLANs or private VLANs)
Performance separation by customer (shared infrastructure but guaranteed capacity or QoS)
The remainder of this section describes multi-tenancy options at the fabric level and how those
technologies can be combined to enable a multi-tenant fabric.
14.1.2 Infrastructure RequirementsThe aforementioned requirements gathering process should result in a clear direction and set of
mandatory attributes that the fabric architecture must contain. The first key decision is whether ashared storage infrastructure or dedicated storage per tenant is required. For a host, driving toward as
much shared infrastructure as possible is typically a business imperative, but there can be cases where
it is prohibited.
As mentioned in the previous storage sections in the fabric architecture guide, Windows Server 2012
supports a range of traditional storage technologies such as JBOD, iSCSI/Fiber Channel SAN, and
converged technologies such as FCOE. In addition, the new capabilities of storage spaces, cluster
shared volumes, and scale-out file clusters present a potentially lower cost solution for advanced
storage infrastructures.
The shared versus dedicated storage infrastructure requirement drives a significant portion of the
design process. If dedicated storage infrastructures per tenant are required, appropriate sizing and
minimization of cost are paramount. It can be difficult to scale down traditional SAN approaches to a
large number of small- or medium-sized tenants. In this case, the Scale-Out File Cluster and Storage
Spaces approach, which uses shared SAS JBOD, can scale down cost effectively to a pair of file servers
and a single SAS tray.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 101/112
Fabric Architecture Guide 101
Figure 41 Shared SAS storage
On the other end of the spectrum, if shared but logically segmented storage is an option, nearly all
storage options become potentially relevant. Traditional Fiber Channel and iSCSI SANs have evolved
to provide a range of capabilities to support multi-tenant environments through technologies such as
zoning, masking, and virtual SANs. With the scalability enhancements in Windows Server 2012 in the
storage stack, large-scale shared storage infrastructures that use the Scale-Out File Cluster and
Storage Spaces can also be a cost effective choice.
Although previous sections discussed architecture and scalability, the section highlights technologies
for storage security and isolation in multi-tenant environments.
14.1.3 Multi-Tenant Storage Considerations14.1.3.1 SMB 3.0The Server Message Block (SMB) protocol is a network file sharing protocol that allows applications
on a computer to read and write to files and to request services from server programs in a computer
network. The SMB protocol can be used on top of its TCP/IP protocol or other network protocols.
By using the SMB protocol, an application (or the user of an application) can access files or other
resources on a remote server. This allows users to read, create, and update files on the remote server.
The application can also communicate with any server program that is set up to receive an SMB client
request.
Windows Server 2012 provides the following practical ways to use the new SMB 3.0 protocol:
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 102/112
102 Fabric Architecture Guide
File storage for virtualization (Hyper-V over SMB) : Hyper-V can store virtual machine files(such as configuration files, virtual hard disk (VHD) files, and snapshots) in file shares over theSMB 3.0 protocol. This can be used for stand-alone file servers and for clustered file serversthat use Hyper-V with shared file storage for the cluster.
Microsoft SQL Server over SMB : SQL Server can store user database files on SMB file shares.
Currently, this is supported with SQL Server 2008 R2 for stand-alone servers. Traditional storage for end-user data : The SMB 3.0 protocol provides enhancements to the
information worker (client) workloads. These enhancements include reducing the applicationlatencies experienced by branch office users when accessing data over wide area networks(WAN) and protecting data from eavesdropping attacks.
SMB Encryption14.1.3.1.1
A security concern for data that traverses untrusted networks is that it is prone to eavesdropping
attacks. Existing solutions for this issue typically use IPsec, WAN accelerators, or other dedicated
hardware solutions. However, these solutions are expensive to set up and maintain.
Windows Server 2012 includes encryption that is built-in to the SMB protocol. This allows end-to-enddata protection from snooping attacks with no additional deployment costs. You have the flexibility to
decide whether the entire server or only specific shares should be enabled for encryption. SMB
Encryption is also relevant to server application workloads if the application data is on a file server
and it traverses untrusted networks. With this feature, data security is maintained while it is on the
wire.
Cluster Shared Volumes14.1.3.1.2
By using Cluster Shared Volumes (CSVs), you can unify storage access into a single namespace for
ease of management. A common namespace folder that contains all the CSVs in the failover cluster is
created at the path C:\ClusterStorage\. All cluster nodes can access a CSV at the same time, regardless
of the number of servers, the number of JBOD enclosures, or the number of provisioned virtual disks.
This unified namespace enables high availability workloads to transparently fail over to another server
if a server failure occurs. It also enables you to easily take a server offline for maintenance.
Clustered storage spaces can help protect against the following risks:
Physical disk failures : When you deploy a clustered storage space, protection againstphysical disk failures is provided by creating storage spaces with the mirror resiliency type.Additionally, mirror spaces use ―dirty region tracking‖ to track modifications to the disks in
the pool. When the system resumes from a power fault or a hard reset event and the spacesare brought back online, dirty region tracking creates consistency among the disks in thepool.
Data access failures : If you have redundancy at all levels, you can protect against failedcomponents, such as a failed cable from the enclosure to the server, a failed SAS adapter,power faults, or failure of a JBOD enclosure. For example, in an enterprise deployment, youshould have redundant SAS adapters, SAS I/O modules, and power supplies. To protectagainst complete disk enclosure failure, you can use redundant JBOD enclosures.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 103/112
Fabric Architecture Guide 103
Data corruptions and volume unavailability : The NTFS file system and the Resilient FileSystem (ReFS) help protect against corruption. For NTFS, improvements to the Chkdsk tool inWindows Server 2012 can greatly improve availability. If you deploy highly available fileservers (without using CSVs), you can use ReFS to enable high levels of scalability and dataintegrity regardless of hardware or software failures.
Server node failures : Through the Failover Clustering feature in Windows Server 2012, youcan provide high availability for the underlying storage and workloads. This helps protectagainst server failure and enables you to take a server offline for maintenance without serviceinterruption.
The following are some of the new technologies in Windows Server 2012 that can enable multi-tenant
architectures.
File storage for virtualization (Hyper-V over SMB) : Hyper-V can store virtual machine files(such as configuration files, virtual hard disk (VHD) files, and snapshots) in file shares over theSMB 3.0 protocol. This can be used for stand-alone file servers and for clustered file serversthat use Hyper-V with shared file storage for the cluster.
Microsoft SQL Server over SMB : SQL Server can store user database files on SMB file shares.Currently, this is supported with SQL Server 2008 R2 for stand-alone servers.
Storage can be made visible to only a subset of nodes : Enables cluster deployments thatcontain application and data nodes.
Integration with Storage Spaces : Allows virtualization of cluster storage on groups ofinexpensive disks. The Storage Spaces feature in Windows Server 2012 can integrate withCSVs to permit scale-out access to data.
Security and Storage Access Control14.1.3.1.3
A solution that uses file clusters, storage spaces, and SMB 3.0 in Windows Server 2012 eases the
management of large scale storage solutions because nearly all the setup and configuration is
Windows based with associated Windows PowerShell support.
If desired, particular storage can be made visible to only a subset of nodes in the file cluster. This can
be used in some scenarios to leverage the cost and management advantage of larger shared clusters,
and to segment those clusters for performance or access purposes.
Additionally, at various levels of the storage stack (for example, shares, CSVs, and storage spaces),
access control lists can be applied. In a multi-tenant scenario, this means that the full storage
infrastructure can be shared and managed centrally and that dedicated and controlled access tosegments of the storage infrastructure can be designed. A particular customer could have LUNs,
storage pools, storage spaces, cluster shared volumes, and shares dedicated to them, and access
control lists can ensure that only that tenant has access to them.
Additionally, by using SMB Encryption, all access to the file-based storage can be encrypted to protect
against tampering and eavesdropping attacks. The biggest benefit of using SMB Encryption over
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 104/112
104 Fabric Architecture Guide
more general solutions (such as IPSec) is that there are no deployment requirements or costs beyond
changing the SMB server settings. The encryption algorithm used is AES-CCM, which also provides
data integrity validation (signing).
14.1.4 Multi-Tenant Network ConsiderationsThe network infrastructure is one of the most common and critical layers of the fabric where multi-tenant design is implemented. It is also an area of rapid innovation because the traditional methods
of traffic segmentation, such as VLANs and port ACLs, are beginning to show their age, and they are
unable to keep up with highly virtualized, large scale hosting data centers and hybrid cloud scenarios.
The following sections describe the range of technologies that are provided in Windows Server 2012
for building modern, secure, multi-tenant network infrastructures.
14.1.4.1 Windows Network VirtualizationHyper-V Network Virtualization provides the concept of a virtual network that is independent of the
underlying physical network. With this concept of virtual networks, which are composed of one ormore virtual subnets, the exact physical location of an IP subnet is decoupled from the virtual network
topology.
As a result, customers can easily move their subnets to the cloud while preserving their existing IP
addresses and topology in the cloud so that existing services continue to work unaware of the
physical location of the subnets.
Hyper-V Network Virtualization provides policy-based, software-controlled network virtualization that
reduces the management overhead that is faced by enterprises when they expand dedicated IaaS
clouds, and it provides cloud hosts with better flexibility and scalability for managing virtual machines
to achieve higher resource utilization.
An IaaS scenario that has multiple virtual machines from different organizational divisions (dedicated
cloud) or different customers (hosted cloud) requires secure isolation. Virtual local area networks
(VLANs), can present significant disadvantages in this scenario.
For more information, see Hyper-V Network Virtualization Overview in the TechNet Library.
VLANs Currently, VLANs are the mechanism that most organizations use to support address space
reuse and tenant isolation. A VLAN uses explicit tagging (VLAN ID) in the Ethernet frame headers, and
it relies on Ethernet switches to enforce isolation and restrict traffic to network nodes with the same
VLAN ID. As described before, there are disadvantages with VLANs which introduce challenges in
large-scale multi-tenant environments.
IP address assignment In addition to the disadvantages that are presented by VLANs, virtual
machine IP address assignment presents issues, which include:
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 105/112
Fabric Architecture Guide 105
Physical locations in data center network infrastructure determine virtual machine IPaddresses. As a result, moving to the cloud typically requires rationalizing and possibly IPaddresses across workloads and tenants.
Policies are tied to IP addresses, such as firewall rules, resource discovery, and directoryservices. Changing IP addresses requires updating all the associated policies.
Virtual machine deployment and traffic isolation are dependent on the topology.
When data center network administrators plan the physical layout of the data center, they must make
decisions about where subnets will be physically placed and routed. These decisions are based on IP
and Ethernet technology that influence the potential IP addresses that are allowed for virtual
machines running on a given server or a blade that is connected to a particular rack in the data center.
When a virtual machine is provisioned and placed in the data center, it must adhere to these choices
and restrictions regarding the IP address. Therefore, the typical result is that the data center
administrators assign new IP addresses to the virtual machines.
The issue with this requirement is that in addition to being an address, there is semantic information
associated with an IP address. For instance, one subnet may contain given services or be in a distinct
physical location. Firewall rules, access control policies, and IPsec security associations are commonly
associated with IP addresses. Changing IP addresses forces the virtual machine owners to adjust all
their policies that were based on the original IP address. This renumbering overhead is so high that
many enterprises choose to deploy only new services to the cloud, leaving legacy applications alone.
Hyper-V Network Virtualization decouples virtual networks for customer virtual machines from the
physical network infrastructure. As a result, it enables customer virtual machines to maintain their
original IP addresses, while allowing data center administrators to provision customer virtual
machines anywhere in the data center without reconfiguring physical IP addresses or VLAN IDs.
Each virtual network adapter in Hyper-V Network Virtualization is associated with two IP addresses:
Customer Address (CA) : The IP address that is assigned by the customer, based on theirintranet infrastructure. This address enables the customer to exchange network traffic withthe virtual machine as if it had not been moved to a public or private cloud. The CA is visibleto the virtual machine and reachable by the customer.
Provider Address (PA) : The IP address that is assigned by the host or the data centeradministrators, based on their physical network infrastructure. The PA appears in the packets
on the network that are exchanged with the Hyper-V server that is hosting the virtualmachine. The PA is visible on the physical network, but not to the virtual machine.
The CAs maintain the customer's network topology, which is virtualized and decoupled from the
actual underlying physical network topology and addresses, as implemented by the PAs. Figure 44
shows the conceptual relationship between virtual machine CAs and network infrastructure PAs as a
result of network virtualization.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 106/112
106 Fabric Architecture Guide
Figure 42 Conceptual relationship between CAs and PAs
Key aspects of network virtualization in this scenario includes:
Each virtual machine CA is mapped to a physical host PA.
Virtual machines send data packets in the CA spaces, which are put into an ―envelope‖ with aPA source and destination pair based on the mapping.
The CA-PA mappings must allow the hosts to differentiate packets for different customervirtual machines.
As a result, the mechanism used to virtualize the network is to virtualize the network addresses used
by the virtual machines.
Hyper-V Network Virtualization supports the following modes to virtualize the IP address:Generic Routing Encapsulation The network virtualization generic routing encapsulation (NVGRE)
is part of the tunnel header. This mode is intended for the majority of data centers that deploy Hyper-
V Network Virtualization. In N VGRE, the virtual machine’s packet is encapsulated inside another
packet. The header of this new packet has the appropriate source and destination PA IP addresses in
addition to the virtual subnet ID, which is stored in the Key field of the GRE header.
IP Rewrite In this mode, the source and the destination CA IP addresses are rewritten with the
corresponding PA addresses as the packets leave the end host. Similarly, when virtual subnet packets
enter the end host, the PA IP addresses are rewritten with appropriate CA addresses before being
delivered to the virtual machines. IP Rewrite is targeted for special scenarios where the virtual
machine workloads require or consume very high bandwidth throughput (~10 Gbps) on existing
hardware. IP Rewrite is intended for special scenarios where virtual machines require ~10 Gbps
bandwidth today, and the customer cannot wait for NVGRE-aware hardware.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 107/112
Fabric Architecture Guide 107
14.1.4.2 Hyper-V Extensible SwitchThe Hyper-V Virtual Switch is a software-based, layer-2 network switch that is available in Hyper-V
Manager when you install the Hyper-V server role. The switch includes programmatically managed
and extensible capabilities to connect virtual machines to virtual networks and to the physical
network. In addition, Hyper-V Virtual Switch provides policy enforcement for security, isolation, andservice levels.
The Hyper-V Virtual Switch in Windows Server 2012 introduces several features and enhanced
capabilities for tenant isolation, traffic shaping, protection against malicious virtual machines, and
simplified troubleshooting.
With built-in support for Network Device Interface Specification (NDIS) filter drivers and Windows
Filtering Platform (WFP) callout drivers, the Hyper-V virtual switch enables independent software
vendors (ISVs) to create extensible plug-ins (known as virtual switch extensions) that can provide
enhanced networking and security capabilities. Virtual switch extensions that you add to the Hyper-V
virtual switch are listed in the Virtual Switch Manager feature of Hyper-V Manager.
The capabilities provided in the Hyper-V virtual switch mean that organizations have more options to
enforce tenant isolation, to shape and control network traffic, and to employ protective measures
against malicious virtual machines.
Some of the principal features that are included in the Hyper-V virtual switch are:
ARP and Neighbor Discovery spoofing protection : Provides protection against a maliciousvirtual machine by sing Address Resolution Protocol (ARP) spoofing to steal IP addresses fromother virtual machines. Provides protection against attacks that can be launched for IPv6 by
using Neighbor Discovery spoofing. DHCP Guard protection : Protects against a malicious virtual machine representing itself as a
Dynamic Host Configuration Protocol (DHCP) server for man-in-the-middle attacks.
Port ACLs : Provides traffic filtering, based on Media Access Control (MAC) or InternetProtocol (IP) addresses and ranges, which enables you to set up virtual network isolation.
Trunk mode to virtual machines : Enables administrators to set up a specific virtual machineas a virtual appliance, and then direct traffic from various VLANs to that virtual machine.
Network traffic monitoring : Enables administrators to review traffic that is traversing thenetwork switch.
Isolated (private) VLAN : Enables administrators to segregate traffic on multiple VLANs, tomore easily establish isolated tenant communities.
The features listed above can be combined to deliver a complete multi-tenant network design.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 108/112
108 Fabric Architecture Guide
14.1.4.3 Example Network DesignIn Hyper-V Network Virtualization, a customer is defined as the owner of a group of virtual machines
that are deployed in a data center. A customer can be a corporation or enterprise in a multi-tenant
public data center, or a division or business unit within a private data center. Each customer can have
one or more customer networks in the data center, and each customer network consists of one ormore customer networks with virtual subnets.
Customer network
Each customer network consists of one or more virtual subnets. A customer network forms anisolation boundary where the virtual machines within a customer network can communicatewith each other. As a result, virtual subnets in the same customer network must not useoverlapping IP address prefixes.
Each customer network has a routing domain, which identifies the customer network. Therouting domain ID, which identifies the customer network, is assigned by data center
administrators or data center management software, such as System Center Virtual MachineManag er (VMM). The routing domain ID has a GUID format, for example, ―{11111111 -2222-3333-4444- 000000000000}‖.
Virtual subnets
A virtual subnet implements the Layer 3 IP subnet semantics for the virtual machines in thesame virtual subnet. The virtual subnet is a broadcast domain (similar to a VLAN). Virtualmachines in the same virtual subnet must use the same IP prefix, although a single virtualsubnet can accommodate an IPv4 and an IPv6 prefix simultaneously.
Each virtual subnet belongs to a single customer network (with a routing domain ID), and it is
assigned a unique virtual subnet ID (VSID). The VSID is universally unique and may be in therange 4096 to 2^24-2).
A key advantage of the customer network and routing domain is that it allows customers to bring
their network topologies to the cloud. The following diagram shows an example where the Blue Corp
has two separate networks, the R&D Net and the Sales Net. Because these networks have different
routing domain IDs, they cannot interact with each other. That is, Blue R&D Net is isolated from Blue
Sales Net, even though both are owned by Blue Corp. Blue R&D Net contains three virtual subnets.
Note that the routing domain ID and VSID are unique within a data center.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 109/112
Fabric Architecture Guide 109
Figure 43 Example Hoster/Service Provider data center network design
In this example, the virtual machines with VSID 5001 can have their packets routed or forwarded by
Hyper-V Network Virtualization to virtual machines with VSID 5002 or VSID 5003. Before deliveringthe packet to the virtual switch, Hyper-V Network Virtualization will update the VSID of the incoming
packet to the VSID of the destination virtual machine. This will only happen if both VSIDs are in the
same routing domain ID. If the VSID that is associated with the packet does not match the VSID of the
destination virtual machine, the packet will be dropped. Therefore, virtual network adapters with
RDID1 cannot send packets to virtual network adapters with RDID2.
Each virtual subnet defines a Layer 3 IP subnet and a Layer 2 (L2) broadcast domain boundary similar
to a VLAN. When a virtual machine broadcasts a packet, this broadcast is limited to the virtual
machines that are attached to switch ports with the same VSID. Each VSID can be associated with a
multicast address in the PA. All broadcast traffic for a VSID is sent on this multicast address.
In addition to being a broadcast domain, the VSID provides isolation. A virtual network adapter in
Hyper-V Network Virtualization is connected to a Hyper-V switch port that has a VSID ACL. If a packet
arrives on this Hyper-V virtual switch port with a different VSID, the packet is dropped. Packets will
only be delivered on a Hyper-V virtual switch port if the VSID of the packet matches the VSID of the
virtual switch port. This is the reason that packets flowing from VSID 5001 to 5003 must modify the
VSID in the packet before delivery to the destination virtual machine.
If the Hyper-V virtual switch port does not have a VSID ACL, the virtual network adapter that is
attached to that virtual switch port is not part of a Hyper-V Network Virtualization virtual subnet.Packets that are sent from a virtual network adapter that does not have a VSID ACL will pass
unmodified through the Hyper-V Network Virtualization.
When a virtual machine sends a packet, the VSID of the Hyper-V virtual switch port is associated with
this packet in the out-of-band (OOB) data. If generic routing encapsulation (GRE) is the IP
virtualization mechanism, the GRE Key field of the encapsulated packet contains the VSID.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 110/112
110 Fabric Architecture Guide
On the receiving side, Hyper-V Network Virtualization delivers the VSID in the OOB and the de-
capsulated packet to the Hyper-V virtual switch. If IP Rewrite is the IP virtualization mechanism, and
the packet is destined for a different physical host, the IP addresses are changed from CA addresses
to PA addresses, and the VSID in the OOB is dropped. Hyper-V Network Virtualization verifies a policy
and adds the VSID to the OOB data before the packet is passed to the Hyper-V virtual switch.
14.1.5 Multi-Tenant Compute ConsiderationsSimilar to storage and network, the compute layer of the fabric can be dedicated per tenant or shared
across multiple tenants. That decision greatly impacts the design of the compute layer. Two primary
decisions are required to begin the design process:
Will the compute layer be shared between multiple tenants?
Will the compute infrastructure provide high availability by using failover clustering?
This leads to four high-level design options:
Dedicated stand-alone Hyper-V servers
Shared stand-along Hyper-V servers
Dedicated Hyper-V failover clusters
Shared Hyper-V failover clusters
The introduction of shared-nothing live migration in Windows Server 2012 enables stand-alone
Hyper-V servers to be a viable option when high availability of the running virtual machines is not a
requirement. Shared-nothing live migration enables virtual machines to be moved from any Hyper-V
host running Windows Server 2012 to another, with nothing but a network connection required —it
does not require shared storage.
For hosts that are delivering stateless application and web hosting services, this may be an option.
The shared-nothing live migration feature would enable the host to move virtual machines and
evacuate hosts for patching without causing downtime to the running virtual machines. However,
stand-alone hosts do not provide virtual machine high availability, so if the host fails, the virtual
machines are not automatically started on another host.
The decision of using a dedicated vs. a shared Hyper-V host is primarily driven by the compliance or
business model requirements discussed previously.
14.1.5.1 Hyper-VThe Hyper-V role enables you to create and manage a virtualized computing environment by using
the virtualization technology that is built in to Windows Server 2012. Installing the Hyper-V role
installs the required components and optionally installs management tools. The required components
include the Windows hypervisor, Hyper-V Virtual Machine Management Service, the virtualization
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 111/112
Fabric Architecture Guide 111
WMI provider, and other virtualization components such as the virtual machine bus (VMbus),
virtualization service provider (VSP) and virtual infrastructure driver (VID).
The management tools for the Hyper-V role consist of:
GUI-based management tools: Hyper-V Manager, a Microsoft Management Console (MMC)snap-in, and Virtual Machine Connection, which provides access to the video output of avirtual machine so you can interact with the virtual machine.
Hyper-V-specific cmdlets for Windows PowerShell. Windows Server 2012 includes a Hyper-Vmodule, which provides command-line access to all the functionality that is available in theGUI, in addition to functionality that is not available through the GUI.
The scalability and availability improvements in Hyper-V allow for significantly larger clusters and
greater consolidation ratios, which are key to the cost of ownership for enterprises and hosts. Hyper-V
in Windows Server 2012 supports significantly larger configurations of virtual and physical
components than in previous releases of Hyper-V. This increased capacity enables you to run Hyper-V
on large physical computers and to virtualize high-performance, scale-up workloads.
Hyper-V provides a multitude of options for segmentation and isolation of virtual machines that are
running on the same server. This is critical for shared Hyper-V server and cluster scenarios where
multiple tenants will host their virtual machines on the same servers. By design, Hyper-V ensures
isolation of memory, VMBus, and other system and hypervisor constructs between all virtual machines
on a host.
14.1.5.2 Failover ClusteringFailover clusters provide high availability and scalability to many server workloads. Failover Clustering
in Windows Server 2012 supports increased scalability, continuously available file-based serverapplication storage, easier management, faster failover, and more flexible architectures for failover
clusters.
For the purposes of a multi-tenant design, Hyper-V clusters can be used in conjunction with the
aforementioned scale-out file server clusters for an end-to-end Microsoft solution for storage,
network, and compute architectures.
14.1.5.3 Resource MeteringService Providers and enterprises deploying private clouds need tools to charge back business units
that they support while providing the business units with the right amount of resources to match theirneeds. For hosting providers, it is equally important to issue chargebacks based on the amount of
usage by each customer.
To implement advanced billing strategies that measure the assigned capacity of a resource and its
actual usage, earlier versions of Hyper-V required users to develop their own chargeback solutions
that polled and aggregated performance counters. These solutions could be expensive to develop
and sometimes led to loss of historical data.
8/12/2019 IaaS Product Line Architecture Fabric Architecture Guide
http://slidepdf.com/reader/full/iaas-product-line-architecture-fabric-architecture-guide 112/112
To assist with more accurate, streamlined chargebacks while protecting historical information, Hyper-
V in Windows Server 2012 introduces Resource Metering, a feature that allows customers to create
cost-effective, usage-based billing solutions. With this feature, service providers can choose the best
billing strategy for their business model, and independent software vendors can develop more
reliable, end-to-end chargeback solutions on top of Hyper-V.
14.1.5.4 ManagementAlthough this guide deals only with fabric architecture and not the more comprehensive topic of
fabric management by using System Center 2012, there are significant management and automation
of multi-server, multi-tenant environments enabled by Windows Server 2012 technologies.
Windows Server 2012 delivers significant management efficiency with broader automation of
common management tasks and a path toward full out-of-band management automation. For
example, Server Manager in Windows Server 2012 enables multiple servers on the network to be
managed effectively from a single computer. With the Windows PowerShell 3.0 command-line
interface, Windows Server 2012 provides a platform for robust, multi-machine automation for all
elements of a data center, including servers, Windows operating systems, storage, and networking. It
also provides centralized administration and management capabilities such as deploying roles and
features remotely to physical and virtual servers, and deploying roles and features to virtual hard
disks, even when they are offline.
top related