REFERENCE ARCHITECTURE

CITRIX XENSERVER® 6.5 AND EMC™ XTREMIO™ ALL-FLASH-ARRAY

ABSTRACT This reference architecture describes the performance and operational advantages of a XenServer 6.5 deployment on an EMC XtremIO All-Flash Array, and describes how the solution enhances consolidation and virtualization of XenServer 6.5 environments.

March 2016.

2

To learn more about how EMC products, services, and solutions can help solve your business and IT challenges, contact your local representative or authorized reseller, visit www.emc.com, or explore and compare products in the EMC Store

Copyright © 2016 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

The information in this publication is provided "as is". EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

VMware and VMware vSphere® are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions.

Citrix® and Citrix XenServer® are registered trademarks or trademarks of Citrix Systems, Inc. in the United States and/or other jurisdictions.

All other trademarks used herein are the property of their respective owners.

Part Number H14965

3

TABLE OF CONTENTS

EXECUTIVE SUMMARY .............................................................................. 4 Audience ........................................................................................................... 4

Document Purpose ............................................................................................. 4

Business Requirements ....................................................................................... 4

OVERVIEW ................................................................................................ 5 Solution ............................................................................................................ 5

Key Components ................................................................................................ 5

Virtualization ..................................................................................................... 6

Overview .................................................................................................................. 6

Citrix XenServer 6.5 ................................................................................................... 6

XenCenter 6.5 / XenServer CLI .................................................................................... 6

EMC XtremIO Storage ......................................................................................... 6

Cluster Design ........................................................................................................... 6

Deduplication Capacity Saving ..................................................................................... 7

Thin Provisioning ........................................................................................................ 7

Fault Protection.......................................................................................................... 7

Scalability ................................................................................................................. 7

In-Memory Metadata Operations .................................................................................. 7

SOLUTION ARCHITECTURE ....................................................................... 8 Overview .......................................................................................................... 8

Logical Architecture ............................................................................................ 8

Hardware Resources ........................................................................................... 9

Software Resources ............................................................................................ 9

REFERENCE ARCHITECTURE .................................................................... 10 Overview ........................................................................................................ 10

Multipath ........................................................................................................ 10

HBA and XtremIO Devices I/O Scheduler and Queue Depth Configuration ............... 13

Storage Repositories ........................................................................................ 14

Control Domain - Dom0 .................................................................................... 14

Space Reclamation ........................................................................................... 15

TESTS RESULTS ...................................................................................... 16

4

Executive Summary Server virtualization has been a driving force in data center efficiency gains for the past decade. However, mixing multiple virtual machine (VM) workloads on single physical servers creates a randomization of I/O for the storage array, hence stalling the virtualization of I/O-intensive workloads. EMC XtremIO All-Flash Arrays not only addresses this performance challenge in a cost-effective manner, but also provide new levels of speed and provisioning agility to virtualized environments.

This reference architecture is a comprehensive guide to specific technical aspects of the XenServer 6.5 virtualization solution. Server capacity is provided in generic terms for the required minimum of CPU, memory and network interfaces. You can select the server and networking hardware that meets, or exceeds, the tested minimums shown herein.

Audience This reference architecture is intended for EMC employees, partners and customers, including storage and XenServer administrators wanting to understand how an EMC XtremIO All-Flash Array and Citrix's XenServer 6.5 provide an easy-to-use, high-performance storage solution for VSI and VDI environments.

It is assumed that readers of this document are familiar with the following products:

• EMC XtremIO All-Flash Array

• Citrix XenServer 6.5 and XenCenter

Document Purpose This document describes the reference architecture of the Citrix XenServer 6.5 infrastructure for server/desktop virtualization with an EMC XtremIO All-Flash Array.

Business Requirements Business applications are moving into consolidated compute, network and storage environments. EMC XtremIO All-Flash Array and Citrix XenServer server virtualization reduces the complexity of configuring every single component of a traditional deployment model. Virtualization reduces the complexity of integration management, while maintaining application design and implementation options. Administration is unified while process separation is adequately controlled and monitored.

This solution addresses the complexity of deploying and managing complex, consolidated environments, by providing the following key benefits:

• End-to-end virtualization, utilizing the capabilities of unified infrastructure components

• A Citrix XenServer server virtualization solution that efficiently provides thousands of Virtual Machines (VMs) for various customer use cases, with a reliable, flexible and scalable reference design

• A simplified data center environment management interface

• Better support of service-level agreements and compliance initiatives

• Lower operational and maintenance costs

5

Overview

Solution This solution uses the EMC XtremIO All-Flash Array and the Citrix XenServer 6.5 to provide storage and server hardware consolidation for private cloud or server virtualizations.

The new virtualized infrastructure is centrally managed to provide the efficient deployment and management of a scalable number of VMs and the associated shared storage that is entailed.

Figure 1 shows the solution components.

Figure 1: Server Virtualization Components

Key Components The key components of this solution include:

• Virtualization - The virtualization layer decouples the physical implementation of resources, from the applications that use them. Therefore, the application view of the available resources is not directly tied to the hardware.

• Storage - The storage layer is critical for implementation of the server's virtualization. The EMC XtremIO All-Flash Array used in this solution provides extremely high performance and supports a number of capacity-efficient and data-service capabilities.

6

Virtualization

Overview The virtualization layer is a key component of any server virtualization and/or private cloud solution. This component decouples the application resource requirements from the underlying physical resources, thus enabling greater flexibility in the application layer. This greater flexibility is achieved by eliminating hardware downtime for maintenance and changes to the physical system without affecting the hosted applications. In a server virtualization or private cloud use case, the virtualization layer enables multiple independent VMs to share the same physical hardware, as opposed to being directly implemented with dedicated hardware.

Citrix XenServer 6.5 XenServer 6.5 is a complete server virtualization platform from Citrix. The XenServer package contains everything that is required in order to create and manage the deployment of virtual x86 computers running on Xen; the open-source paravirtualizing hypervisor with near-native performance.

XenServer is optimized for both Windows and Linux virtual servers, and runs directly on server hardware without the need of an underlying operating system, thus resulting in an efficient and scalable system. XenServer works by abstracting elements from the physical machine (such as hard drives, resources and ports) and allocating them to VMs running on it.

A VM behaves exactly like a physical computer, containing its own virtual (software-based) CPU, RAM, hard disk and Network Interface Card (NIC).

XenServer enables you to create VMs, create VM disk snapshots and manage VM workloads.

XenCenter 6.5 / XenServer CLI

The two methods used to administer XenServer are:

• XenCenter - a graphical Windows-based user interface that enables managing XenServer hosts, pools and shared storage, and enables the deployment, management and monitoring of VMs from Windows desktop machines.

• XenServer Command Line Interface (CLI) - an Online Help, providing a useful resource for getting started with XenCenter and for obtaining context-sensitive assistance. The XenServer CLI enables administering XenServer by using Linux-based XE commands.

EMC XtremIO Storage

Cluster Design XtremIO has a scale-out cluster design that delivers balanced capacity and performance in order to meet every storage requirement. Each cluster building block provides high-availability, fully active/active storage servers with no single point of failure. An XtremIO cluster automatically balances the workloads of all hosts, as well as performance, as the cluster is expanded.

The XtremIO operating system (XIOS) manages storage clusters and provides the following functionality:

• Ensures that the cluster's SSDs are evenly loaded, thus providing the highest possible performance and endurance required for high-demand workloads, throughout the life of the array.

• Eliminates the need to perform complex configuration steps, as needed with traditional arrays. XIOS negates the need to set RAID levels, determine drive-group sizes, set stripe widths, set caching policies, build aggregates, or perform any other manual configuration.

• Automatically and optimally configures volumes, by ensuring that I/O performance on existing volumes and data sets automatically increases when a cluster scales out.

• Manages the process of expanding clusters, and ensures that data remains balanced across newly-added X-Bricks (XtremIO's fundamental cluster building blocks). XIOS also ensures that I/O performance of existing volumes, and that of data sets, automatically increase when the cluster scales out. It eliminates the need to restripe data if application requirements change. Every volume receives the full performance potential of the entire XtremIO cluster.

7

Deduplication Capacity Saving The XtremIO All-Flash Array performs inline data deduplication that is based on an algorithm that checks to ensure that duplicate data blocks are not stored on SSDs. The result is that every storage I/O is deduplicated in real-time upon ingest, with only unique blocks ever being written to the flash storage, exclusively. Moreover, deduplication on XtremIO aids performance substantially, as all metadata is in memory, thus ensuring maximum host I/O performance.

Thin Provisioning In addition to delivering high performance, the XtremIO All-Flash Array natively offers thin provisioning capabilities in order to allocate on-demand capacity as application needs arise, without impacting the array or storage I/O performance. XtremIO thin provisioning is also granular, in that capacity is allocated in 4KB blocks, thus ensuring the thrifty use of flash capacity, maintaining consistency with VMware vSphere I/O block size usage.

Fault Protection The EMC XtremIO array delivers the utmost in reliability and availability, with completely redundant components and an ability to tolerate any component failure without loss of service.

XtremIO provides the following fault protection:

• Dual power supplies in Storage Controllers and disk array enclosures (DAEs) to compensate for any power supply loss, while keeping the Storage Controllers/DAE in service

• Redundant active/active Storage Controllers to compensate for controller failures

• Redundant Serial Attached SCSI (SAS) interconnect modules in the DAEs

• Redundant inter-controller communication links

• Multiple host connections with multipath capabilities to survive path failures

• XtremIO Data Protection (XDP) to tolerate SSD failures

• Multiple techniques to ensure initial and ongoing data integrity

Scalability

XtremIO clusters support a fully-distributed scale-out design to enable linear increases both in capacity and performance, for infrastructure agility. XtremIO employs a building-block approach in which the array is scaled out, by introducing additional X-Bricks. The system provides host access by incorporating N-way active/active controllers for performance and capacity linear scaling, thus simplifying support for growing virtualized environments. As a result, as the array's capacity grows, its performance grows in direct proportion with the addition of Storage Controllers.

In-Memory Metadata Operations An XtremIO cluster distributes metadata evenly across all Storage Controllers, hence maintaining metadata in memory during runtime. Metadata is affixed to SSDs in order to provide the array with failure and power-loss tolerance. However, all metadata retrieval is memory-based during normal operations, which is made possible by segmenting the metadata tables and distributing them evenly across all Storage Controllers. In contrast, dual-controller designs do not contain enough RAM to store all metadata in memory, require de-staging large amounts of metadata to flash, and have several associated performance drawbacks.

XtremIO's in-memory metadata and unique Inline Data Reduction model combine to deliver new and unprecedented virtualized data center capabilities.

8

Solution Architecture

Overview This section provides a summary and characterization of configurations that are to be performed in order to validate an EMC XtremIO All-Flash Array and Citrix XenServer 6.5. The validation procedure involves building a clustered, highly-available VM environment in XtremIO, and integrating the new platform features to provide a high-performance, compelling and cost-effective server virtualization platform.

The defined configuration forms the basis for creating a customized solution.

Logical Architecture Figure 2 shows the solution's logical architecture, and characterizes the validated infrastructure where an 8GB Fibre Channel carries storage traffic, and 10GbE carries management and application traffic.

Figure 2: Logical Architecture

9

Hardware Resources Table 1. Hardware Resources Component Configuration

XenServer

Servers

CPU 2 Physical CPUs per server

40 vCPUs per server

4 vCPUs per VM

Memory 256GB per server

8GB RAM per VM

Network 4 * 10 GbE NICs

Network infrastructure 2 Physical switches

3 * 10GbE used for MGMT and application traffic

1 * 10GbE port for VM migration and XenServer HA

EMC XtremIO All-Flash Array Single X-Brick with 25 * 400GB eMLC SSD drives

Software Resources Table 2. Software Resources Software Configuration

Citrix XenServer

XenServer XenServer 6.5 Free

XenCenter XenCenter 6.5.2

EMC XtremIO

XtremApp 4.0.2

Virtual Machines

Windows Microsoft Windows Server 2012 R2

Linux Centos 7.1

Workload Generator

btest 174.6

vdbench 5.0.4

10

Reference Architecture

Overview This section describes the steps required for configuring XenServer 6.5 for use with an EMC XtremIO All-Flash Array in order to achieve the best possible performance and highest possible storage capacity. Since XenServer is a Linux-based hypervisor that is based on XenSource Project 4.X, the configuration procedure is initiated from the underlying (Red Hat-based) Linux OS. Following configuration, the XenServer 6.5 hypervisor (Dom0) and Storage Repositories (SR) are configured for maximum utilization of the XtremIO cluster.

Multipath Once configured, prior to using the native multipath that accompanies the XenServer 6.5, XtremIO configuration is added to the /etc/multipath.conf.

To configure the XenServer for use with an XtremIO cluster:

1. Commence the configuration via Linux.

2. Configure the XenServer 6.5 hypervisor (Dom0) and SRs.

3. Add XtremIO configuration to the /etc/multipath.conf.

device { vendor XtremIO product XtremApp path_selector "queue-length 0" rr_min_io_rq 1 path_grouping_policy multibus path_checker tur failback immediate fast_io_fail_tmo 15 }

4. From XenCenter's GUI, right-click the XenServer, and from the menu select Enter Maintenance Mode, as shown in Figure 3.

Note: Existing VMs should be powered off or migrated prior to setting XenCenter to Maintenance Mode.

Figure 3: Setting the XenServer to Maintenance Mode

11

5. Click Enter Maintenance Mode, as shown in Figure 4.

Figure 4: Entering Maintenance Mode

6. Right-click the XenServer, and from the menu select Properties, as shown in Figure 5.

Figure 5: XenServer Properties

12

7. From the XenServer Properties window left pane, select Multipathing, and check Enable multipathing on this server, as shown in Figure 6.

Figure 6: Enabling Multipathing

8. Click OK.

9. From the right-click menu, select Exit Maintenance Mode, as shown in Figure 7.

Figure 7: Exiting Maintenance Mode

10. Repeat Steps 1 through 9 9 of the above procedure in order to configure the XenServers that are intended for use with the EMC XtremIO All-Flash Array.

For information on how to configure the Citrix XenServer 6.5 for use with an EMC XtremIO All-Flash Array via the CLI, and/or on an existing SR with running VMs, refer to the Citrix support article 'CTX118791' (at http://support.citrix.com/article/CTX118791).

13

HBA and XtremIO Devices I/O Scheduler and Queue Depth Configuration The XenServer HBA queue depth should be configured to 128 in accordance with the HBA manufacturer. Alternatively, refer to the Linux section of the EMC XtremIO Storage Array Host Configuration Guide for queue depth configuration information.

XenServer 6.5 supports the use of UDEV rules, which assist in automatically configuring XtremIO volumes as they are detected by the XenServer host.

• On the XenServer shell, add the following lines to /etc/udev/rules.d/99-XtremIO.rules:

# Use noop scheduler ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_VENDOR}=="XtremIO", ENV{ID_MODEL}=="XtremApp", ATTR{queue/scheduler}="noop" ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="dm*", ENV{DM_NAME}=="??14f0c5*" , ATTR{queue/scheduler}="noop" # Reduce CPU overhead due to disk entropy contribution ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_VENDOR}=="XtremIO", ENV{ID_MODEL}=="XtremApp", ATTR{queue/add_random}="0" ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="dm*", ENV{DM_NAME}=="??14f0c5*" , ATTR{queue/add_random}="0" # forces the IO processing completion to run on the requesting cpu ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_VENDOR}=="XtremIO", ENV{ID_MODEL}=="XtremApp", ATTR{queue/rq_affinity}="2" ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="dm*", ENV{DM_NAME}=="??14f0c5*" , ATTR{queue/rq_affinity}="2" # increase queue depth on the volume ACTION=="add|change", SUBSYSTEM=="scsi", ATTR{vendor}=="XtremIO ",ATTR{model}=="XtremApp", ATTR{queue_depth}="128" ACTION=="add|change", SUBSYSTEMS=="scsi", ATTRS{vendor}=="XtremIO ", PROGRAM="/bin/sh -c 'echo 128 > /sys$devpath/../../queue_depth'"

Noop Scheduler – The NOOP scheduler inserts all incoming I/O requests into a simple FIFO queue, and implements request merging. The scheduler is useful in cases where it is predetermined that the host will not attempt to re-order requests which are based on the sector numbers contained therein. In other words, the scheduler "assumes" that the host is definitionally "unaware" of how to productively re-order requests if I/O scheduling is handled at a lower layer of the I/O stack. For example, at the block device, by an intelligent RAID controller, network attached storage, or by an externally attached controller (such as a storage subsystem that is accessed via a switched storage area network). Since I/O requests are potentially re-scheduled at the lower level, resequencing IOPS at the host level can create a situation in which the host's CPU time is unnecessarily wasted on operations, only to become undone upon reaching the lower level, thus increasing latency and decreasing throughput without productivity.

Disk Entropy Contribution – Linux defaults to use all disk devices in order to generate random data. This line excludes XtremIO volumes from being accessed for random data generation.

Disk CPU Affinity – The Linux system's default for CPU affinity is "round-robin". Therefore, an I/O can start on one core and finish on another, thus causing unnecessary data transfer between cores and increasing the I/O's latency.

Volume Queue Depth – Only changing the queue depth on the HBA level to 128 does not affect the various paths used by the kernel. The queue depth remains on the default setting of 32. Using this rule, all XtremIO volumes are assigned a queue depth of 128.

14

Storage Repositories XenServer's external Storage Repositories (SR) are called "Hardware HBA". After creating SR, using the XenCenter (or XE CLI), the new SR uses the default queue depth (of 32). The scheduler (which is CFQ) requires a setting of 128 for queue depth and NOOP for the I/O scheduler.

Using the XE CLI, implement the change on the shell only, which is sufficient for it to run on one XenServer in the pool. The change is global.

xe sr-list|grep name-descrip -B2|grep XtremIO -B2 |grep uuid|awk -F" " '{print "xe sr-param-set uuid="$5" other-config:blkback-mem-pool-size-rings=8 other-config:scheduler=noop"}'>/tmp/write_param;chmod a+x /tmp/write_param;/tmp/write_param

Control Domain - Dom0 XenServer 6.5 introduces many architectural improvements that are designed to improve overall performance, and remove a number of XenServer 6.2's scalability limitations. For details of the new configuration limits, refer to the Citrix support article 'CTX141510' (at http://support.citrix.com/article/CTX141510).

The new 64-bit Control Domain (Dom0) enables XenServer to accommodate many more PCI devices per host (such as NICs, GPUs, etc.), and enables the use of devices that are only 64-bit (including many Solid State Drive solutions). The new 64-bit kernel removes the previous restrictive low/high memory division, previously limiting Dom0's maximum amount of usable memory. This limitation could result in sub-optimal memory performance within Dom0 (when in excess of 752MB of RAM is allocated). Additionally, Dom0 was upgraded from CentOS 5.7 to CentOS 5.10.

Dom0 memory is automatically configured (depending on amounts of free host memory that is available) and can be scaled and optimized in order to cope with the very latest vGPU, disk and network driver memory demands. This enables support for more VMs and for internal caches to speed up disk access.

XenServer 6.5 includes the latest Xen Project hypervisor that is available; version 4.4 delivers many improvements. XenServer 6.5 vastly increases the number of virtual event channels available for Dom0; from 1023 to 131071, which translates to a corresponding larger number of attached virtual devices. XenServer version 6.2 used a special interim solution that provided 4096 event channels, sufficient for supporting approximately 500 VMs per host, yet containing only a few virtual devices in each VM. With the support for extra event channels in version 4.4, XenServer 6.5 enables each of these VMs a far richer set of virtual devices.

Xen version 4.4 also handles grant-copy locking requests more efficiently, thus dramatically improving aggregate network and disk throughput.

The only configuration that Citrix still recommends, is performed by the “host-cpu-tune” command, which only appears if Dom0 uses the recommended CPU configuration for that specific HW.

[root@scvdi45 ~]# host-cpu-tune advise Citrix recommends assigning 8 vCPUs to dom0, not using pinning. This can be achieved by running: /opt/xensource/bin/host-cpu-tune set 8 nopin [root@scvdi45 ~]# host-cpu-tune show dom0's vCPU count: 8, not pinned

The first command (host-cpu-tune advise) displays the Citrix recommendation for Dom0 CPU configuration, and prints the command that is required to run in order to perform this configuration change. A reboot is necessary to enable the parameter.

15

Space Reclamation Space reclamation enables freeing up unused blocks (such as deleted VDIs in an SR) on a LUN that has been thinly provisioned by the storage array. It enables notifications of deletions within LVM to be communicated directly to the array. Once released, the reclaimed space is freely available for reuse by the array. Space reclamation is automatic when a XenServer VM snapshot is deleted.

If space reclamation is not issued on a weekly or monthly basis (depending on the amount of data being deleted or replaced in a specific environment), the array becomes populated with old, unused data, and the value of storage utilization reported is higher than it actually is.

To reclaim freed space using XenCenter:

1. From the View menu, select Infrastructure, and click the host or pool connected to the SR.

2. Click the Storage tab.

3. Select the SR from the list, and click Reclaim freed space.

4. Click Yes.

5. Click Notifications, Events; the Events window appears, showing the status of the operation.

16

Tests Results This section describes varying scenarios and the resulting benefits, as they are shown in the XtremIO cluster's graphic interface. Key results include:

• There is a 37% total performance improvement after implementing all tweaks.

• More is achieved from the same XenServer hardware.

• Less infrastructure to implement the same number of VMs results in a substantial reduction to the solution's over-all cost.

Figure 8 shows performance results when using multipath only.

Figure 8: Performance Using Multipath

Figure 9 shows performance results once HBA and XtremIO devices I/O scheduler and queue depth are configured.

Figure 9: Performance after HBA and XtremIO Devices I/O Scheduler and Queue Depth Configuration

17

Figure 10 shows the performance test result after configuring the XtremIO SR.

Figure 10: Performance after Configuring Storage Repositories for EMC XtremIO

Figure 11 shows performance results once space reclamation is implemented.

Figure 11: Performance after Space Reclamation