wmware storage vmotion

BEST PRACTICES FOR SUN STORAGE 6000 SERIESAND 2500 SERIES ARRAYS

VMware STORAGE VMotion

White PaperMarch 2009

Sun Microsystems, Inc.

Table of Contents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

VMware Storage VMotion Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

What is Moved by VMware Storage VMotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

How VMware Storage VMotion Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

VMware Storage VMotion Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

VMware Storage VMotion Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

VMware Storage VMotion Uses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Storage Tiering with Sun Storage Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Sun Storage Array Performance and Scalability Management . . . . . . . . . . . . . . . . 8

Storage Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Sun Storage 6000 and 2500 series architecture guidelines for

VMware Storage VMotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

IntroductionVMware Storage VMotion, allows the user to perform live online migration of a

running Virtual Machine’s disks, or disk files from ESX Server’s perspective, from

internal disk drives on a server, within and across storage arrays without the need

for downtime while maintaining transaction integrity. The key difference between

traditional live storage migration and Storage VMotion is that Storage VMotion can

perform live online migration of the entire operating system, including Swap or page

files and any virtual machine related configuration and log files, from one storage

device to another.

The purpose of this document is to provide guidelines and best practices for using

VMware Storage VMotion with Sun Storage 6000 series, including the new ST6580 and

ST6780, and the Sun Storage 2500 series storage arrays. Although all Fibre Channel or

iSCSI capable Sun Microsystems Storage arrays listed in the VMware Storage/SAN

Compatibility Guide are supported with VMware Storage VMotion. The guide is

updated on a regular basis with new storage arrays added so it is highly recommended

that the online version of the guide be referred to prior to providing any architectural

recommendations.

VMware Storage VMotion BasicsStorage VMotion was introduced in VMware Virtual Infrastructure 3 release with ESX

Server 3.5 and vCenter 2.5, initially supporting Fibre Channel storage. Release of ESX

Server 3.5 update 1 added support for iSCSI storage.

What is Moved by VMware Storage VMotion• Virtual machine’s home directory

• Configuration file (vmx)

• Logs

• Swap file (vswp)

• Any snapshots of the virtual machine

• Any other files related to the virtual machine

• Virtual machine disks or luns (vmdk)

• The operating system disk

• Data disk(s)

The disks can be moved independently as needed or required. The OS disk can be

moved without having to move data disks or luns and any and all disks can be placed

on any disk/lun configuration on the particular ESX server.

Sun Microsystems, Inc.1 VMware Storage VMotion

How VMware Storage VMotion WorksVMware Storage VMotion requires VMware vCenter 2.5 or later, VMware VMotion

license enabled on the host and the VMware provided Remote Command-Line

Interface application or Virtual Appliance which can be downloaded from

VMware.com. There are VMware vCenter plug-ins available online which allow for a

GUI-based Storage VMotion capability but none are officially support by VMware. For

more details, please refer to the VMware VI3 Basic System Administration Guide and

Remote Command-Line Interface Installation and Reference Guide.

VMware Storage VMotion consists of six steps outlined below:


Step 1. Copy virtual machine home directory to new location. This step uses NFCcopier to copy the files over to the new location

Step 2. Self-VMotion to new virtual machine home. A self-VMotion uses the sameprocess as a normal VMotion operation. This step enables the use of the new swapfile and reopen other copied files such as the virtual machine config file and othersuch files.


Step 3. Take a disk-only snapshot. This creates a child disk which will now be usedto maintain ALL changes being made to the Virtual Machine data.

Step 4. Copy virtual machine disk to destination (read-only).


Step 5. Consolidate child disk created in step 4 into copied disk.

Step 6. Delete original virtual machine home and disks and enable read/write onnew copied disk.

VMware Storage VMotion Requirements• VMware vCenter with VMware VMotion license installed

• VMware VMotion must be enabled and properly configured on the ESX server

(Refer to VMware Basic Administration Guide)

– The virtual machine must not be using any device preventing VMotion (e.g.

floppy disk, CD-ROM)

– A physical network link for VMotion is not required for Storage VMotion

• The ESX server must have sufficient resources to perform all of the steps

outlined above:

– At least twice the amount of memory and CPU resources to accommodate a

second instance of the virtual machine on which Storage VMotion is to be

performed (see Steps 1 and 2 above). For example, if the virtual machine uses

1 vCPU and 500 MB of memory, there needs to 1 vCPU and 500 MB readily

available beyond what is being used by all running virtual machines to be able

to perform Storage VMotion operation on this particular virtual machine. This

doubling of memory and CPU resources is temporary while Storage VMotion steps

above are taking place but required.

– At least twice the amount of storage needs to be available at the source storage

to allow for a snapshot to be taken prior to copying the virtual machine disk to

the destination storage (see Steps 3 to 5 above)

• Virtual machine’s disks must be snapshot eligible as a snapshot is taken in Step 3

to create a child disk prior to copying the disk to the new location. Without this

capability Storage VMotion will not work.

• When using RDMs, RDMs in Virtual Compatibility Mode are supported with Storage

VMotion as these are snapshot capable. Snapshots can not be taken for RDMs in

Physical Compatibility Mode (refer to VMware Fibre Channel SAN Configuration

Guide and VMware VM vs. Physical Server I/O Performance Comparison document

for more information on RDMs).

• The ESX server must have access to both the source and destination storage

or datastore

• A maximum of four concurrent Storage VMotion migrations can occur

simultaneously on the same datastore

• Only Fibre Channel and iSCSI storage arrays are supported

• Virtual machines requiring Storage VMotion capability must not have network

connection to any internal virtual switches



NOTE: Storage VMotion of an RDM does not move data from one physical raw lun to another.It only moves the mapping file over to the location where the virtual machine config file(vmx) is moved. To move data from Lun 1 physically, a different technology such as RVMcan be used. (See Figure 1).

• Virtual machines requiring Storage VMotion capability must not have any cluster

relationship with another virtual machine as is the case when using a clustering

technology such as Microsoft Cluster Server or Sun Cluster at the virtual

machine level

• Simultaneous Storage VMotion and VMotion on the same virtual machine is not

allowed. A virtual machine’s datastores can be moved to a different storage first

using Storage VMotion and the virtual machine can be moved to a different ESX

server second or vice-versa.

VMware Storage VMotion Best Practices• Although not required, it is best practice to use the VMware Remote CLI Virtual

Appliance to perform Storage VMotion tasks via command line. There are GUI-based

VMware vCenter plug-ins available but are not supported by VMware.com.

• A lot of pre-planning need to take place prior to any Storage VMotion operation

– Plan for Storage VMotion during initial design and architecture of the ESX servers,

the SAN and the storage arrays

• Free memory, CPU and storage space in source LUN need to be available

• Ideally, use different host bus adapter ports for the source and destination storage

on the ESX server for maximum performance

Figure 1. Data on RDM luns do not move as part of Storage VMotion.


• If moving virtual machine disks on datastores residing on the same storage array,

use different target ports on the array for the source and destination luns for

maximum performance

– Coordination between storage, server and VMware administrators is key as all

components need to work seemlessly for a successful Storage VMotion operation

– It is strongly recommended that Storage VMotion be performed during off peak

hours for the specific virtual machine in order to avoid any noticeable

performance impact

VMware Storage VMotion Uses

Storage Tiering with Sun Storage Storage ArraysThe most compelling case for Storage Storage VMotion is made when the need to

migrate from legacy to latest array technologies arises in a virtualized infrastructure

without a considerable impact to availability. Especially if there is no array-to-array

replication available, as may be the case with customers migrating from a non-Sun

storage array to the new Sun Storage 6780 or 6580 array for instance. There are many

alternative methods to migrate data from one storage to another, some of which are

listed below:

• Use of remote replication technologies within the arrays such as Remote Volume

Mirroring for Sun Storage 6000 series arrays or TrueCopy for Sun Storage 9000 series

arrays. However, this technology requires additional feature licenses to be enabled

on the arrays, which the customer may not need beyond the scope of migrating the

data from older to newer arrays. And, the remote replication technology works only

within the same series of arrays so replication between Sun Storage 6000 series and

Sun Storage 9000 series is not available without the use of additional software.

NOTE: Storage VMotion is not a technology replacement for Disaster Recovery implementa-tions requiring data services such as Remote Replication to remote sites. Remote Replicationof LUNs is still required for Storage Level data replication to remote sites and for maintainingdata redundancy at the same site.

• Snap Shot technologies within an array can be used to migrate data from one lun to

another due to any number of reasons such as a new raid layout, for example

moving from RAID-5 to RAID-6. These snapshot technologies such as ShadowImage

for Sun Storage 9000 series or SnapShot for Sun Storage 6000 or 2500 array series

also may require a license feature as is the case with ShadowImage for Sun Storage

9000 series and require further configuration tasks to setup a mirror, sync the luns,

break the mirror and then perform further administration tasks at the ESX server

level for the secondary lun to be mapped to the same ESX server.

– Storage VMotion is also not a technology replacement for In-Storage Replication

Data Services. Storage VMotion does not create a copy of the data, rather

migrates virtual machine and its associated disk files between a given set of LUNs.


• Stop all I/O to luns to be migrated, create a full backup and perform a restore to the

destination array

– Possible issues include unnecessary and prolonged downtime, errors during

backups and/or restores, possible overloading of the backup infrastructure which

may take resources away from regular day-to-day backup requirements or having

to create or add new backup resources such as tape drives, etc.

To avoid all of these and many other much more complicated, task-oriented methods

to perform a simple lun migration from one array to another or a full migration of all

ESX server luns from one array to anther or to a different set of luns with in the same

array, the use of Storage VMotion is the obvious choice as it maintains uptime for the

virtual machines in any situation without having go through complex tasks of setting

up remote replication, backup and recovery or an internal snapshot technology.

Virtual machine disks resident on an iSCSI array can be migrated to Fibre Channel or

vice-versa. For instance, a Test Virtual Machine moving into a Production status can

simply be migrated from a test Sun Storage 2510 to a new Fibre Channel Sun Storage

6780 Production array using Storage VMotion.

Sun Storage Array Performance and Scalability ManagementVMware Storage VMotion can be used to move virtual machine disks in order to

achieve optimal performance and scalability for established SAN environments as well.

Consider an ESX host where all of the virtual machines currently reside on the internal

disk storage. All of the virtual machines are exposed to down time due to the server

being the single point of failure. If the customer decides to scale their current

configuration to two ESX servers and shared SAN and/or iSCSI storage, the

configuration becomes much more scalable avoiding single points of failures at

the server layer. (See Figure 2).

Figure 2. VMs residing on internal drives of an ESX server are exposed to SPoF.Shared Storage offers enhanced redundancy and availability.


The configuration can be further enhanced by adding arrays which would then allow

for complete redundancy at all levels. (See Figure 3). Virtual machine disks can be

moved seamlessly from internal server disks to the single Sun Storage 6140 and then

can be further scaled and divided between the Sun Storage 6140 and the Sun Storage

6780 based on the degree of data and virtual machine criticality. Data requiring high

performance and redundancy can be moved independently to the higher performance

more scalable ST6780 and other data and virtual machine disks can stay on the

ST6140. All of this data migration can be performed using VMware Storage VMotion.

Storage MaintenanceAnother use of Storage VMotion could be storage maintenance which may have an

impact on production I/O. Storage space reclamation is a major factor and another

compelling reason for use of Storage VMotion.

Figure 3. Low $$/GB storage should be used for non-essential data and High $$/GB,high performing storage for critical data.


Sun Storage 6000 and 2500 Series Architecture Guidelinesfor VMware Storage VMotionWhen architecting a storage solution for VMware, it is important to consider all

aspects of the software including Storage VMotion, which can have a considerable

impact on storage I/O performance when active. (See Figure 4).

Given the illustration above, some important best practices to consider:

• It is strongly recommended to separate Storage VMotion traffic from Production I/O

traffic at the HBA level to the Storage array port to avoid any possible performance

bottle necks

– Another reason to design the solution as such is so data moves from Source Lun to

Destination Lun using a completely different I/O path which will allow Storage

VMotion to complete the process quicker than if the same path was to be used to

moved hence possible creating a bottleneck. The label “Storage VMotion” path

and “Production” path is generic in terms of the illustration above.

Figure 4. If possible, isolate Storage VMotion I/O traffic from Production I/O byusing different Initiators/Targets.


– Sun Storage Common Array Manager software can be used to change “Preferred”

Controller for each Lun or Volume on the array which allows the I/O load to be

spread across controllers. (See Figure 5).

• Avoid moving VMware datastores from a lun within the same volume on an array.

For example, in the illustration above, moving a datastore from Lun 2 to Lun 3 is

not recommended as it will put unnecessary I/O stress on that volume. The given

move also defeats the purpose of using Storage VMotion as a tool to disperse data

effectively and decrease performance bottlenecks unless there was a specific reason

to perform this task such as the need for more space at the lun level.

• For the Sun Storage 2510 array, it is strongly recommended to use dedicated host

network interface and array port to avoid severe I/O bottlenecks when moving

datastores from or to the array. Since the array ports run at 1 GigE speeds, moving

datastores from a Fibre Channel array such as the Sun Storage 2540 could result in

very slow Storage VMotion operation.

• Since Storage VMotion only currently supports Fibre Channel and iSCSI storage

arrays, Sun Storage 2530 array is not supported with Storage VMotion

• Follow all Sun Storage and VMware best practices and guidelines to configure the

ST6000 series and ST2500 series arrays for use with VMware ESX servers

Figure 5.


References• Sun Storage 6580 and 6780 Release Notes –

http://dlc.sun.com/pdf/820-5776-11/820-5776-11.pdf

• Sun Storage 6580 and 6780 Hardware Installation Guide –


• Sun Storage 2500 Series Array Hardware Installation Guide –


• Sun Storage 6140 Array Hardware Installation Guide –


• Sun Storage Common Array Manager Software Installation Guide –


• VMware VI3 and Virtual Infrastructure Basic System Administration Guide –

www.VMware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_admin_guide.pdf

• VMware Storage/SAN Compatibility Guide –

www.VMware.com/resources/compatibility/pdf/vi_san_guide.pdf

• VMware Remote Command-Line Interface Installation and Reference Guide –

www.VMware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_rcli.pdf

• VMware Fibre Channel SAN Configuration Guide –

www.VMware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_san_cfg.pdf

VMware Storage VMotion On the Web sun.com

© 2009 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, and the Sun logo are trademarks or registered trademarks of Sun Microsystems, Inc. or its subsidiaries in the United States and othercountries. Information subject to change without notice. SunWIN #559704 Lit. #SYWP14884-0 04/09

Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (9786) Web sun.com

wmware storage vmotion

Documents

storage vmotion operation

vmware vmotion license

iscsi storage

storage device

storage maintenance

storage tiering

microsystems storage

new storage arrays