EMC – Local Replication Solutions All the local replication solutions that were discussed in this module are available on EMC Symmetrix and CLARiiON arrays.

ü EMC TimeFinder/Mirror and EMC TimeFinder/Clone are full volume replication solutions on the Symmetrix arrays, while EMC TimeFinder/Snap is a pointer based replication solution on the Symmetrix. EMC SnapView on the CLARiiON arrays allows full volume replication via SnapView Clone and pointer based replication via SnapView Snapshot.

ü EMC TimeFinder/Mirror: Highly available, ultra-performance mirror images of Symmetrix volumes that can be non-disruptively split off and used as point-in-time copies for backups, restores, decision support, or contingency uses.

ü EMC TimeFinder/Clone: Highly functional, high-performance, full volume copies of Symmetrix volumes that can be used as point-in-time copies for data warehouse refreshes, backups, online restores, and volume migrations.

ü EMC SnapView Clone: Highly functional, high-performance, full volume copies of CLARiiON volumes that can be used as point-in-time copies for data warehouse refreshes, backups, online restores, and volume migrations.

ü EMC TimeFinder/Snap: High function, space-saving, pointer-based copies (logical images) of Symmetrix volumes that can be used for fast and efficient disk-based restores.

ü EMC SnapView Snapshot: High function, space-saving, pointer-based copies (logical images) of CLARiiON volumes that can be used for fast and efficient disk-based restores.

We will discuss EMC TimeFinder/Mirror and EMC SnapView Snapshot in more detail in the next few slides.

ü Array based local replication technology for Full Volume Mirroring on EMC Symmetrix Storage Arrays – Create Full Volume Mirrors of an EMC Symmetrix device within an Array

ü TimeFinder/Mirror uses special Symmetrix devices called Business Continuance Volumes (BCV). BCVs: – Are devices dedicated for Local Replication – Can be dynamically, non-disruptively established with a Standard device. They can be subsequently

split instantly to create a PIT copy of data. ü The PIT copy of data can be used in a number of ways:

– Instant restore – Use BCVs as standby data for recovery – Decision Support operations – Backup – Reduce application downtime to a minimum (offline backup) – Testing

ü TimeFinder/Mirror is available in both Open Systems and Mainframe environments

ü Establish – Synchronize the Standard volume to the BCV volume – BCV is set to a Not Ready state when established

Ø BCV cannot be independently addressed – Re-synchronization is incremental – BCVs cannot be established to other BCVs – Establish operation is non-disruptive to the Standard device – Operations to the Standard can proceed as normal during the establish

ü Split

– Time of Split is the Point-in-Time – BCV is made accessible for BC Operations – Consistency

Ø Consistent Split – Changes tracked

The TimeFinder/Mirror Consistent Split option ensures that the data on the BCVs is consistent with the data on the Standard devices. Consistent Split holds I/O across a group of devices using a single Consistent Split command, thus all the BCVs in the group are consistent point-in-time copies. Used to create a consistent point-in-time copy of an entire system, an entire database, or any associated set of volumes. The holding of I/Os can be either done by the EMC PowerPath multi-pathing software or by the Symmetrix Microcode (Enginuity Consistency Assist). PowerPath-based consistent split executed by the host doing the I/O, I/O is held at the host before the split. Enginuity Consistency Assist (ECA) based consistent split can be executed, by the host doing the I/O or by a control host in an environment where there are distributed and/or related databases. I/O held at the Symmetrix until the split operation is completed. Since I/O is held at the Symmetrix, ECA can be used to perform consistent splits on BCV pairs across multiple, heterogeneous hosts.

ü Restore – Synchronize contents of BCV volume to the Standard volume – Restore can be full or incremental – BCV is set to a Not Ready state – I/Os to the Standard and BCVs should be stopped before the restore is initiated

ü Query – Provide current status of BCV/Standard volume pairs

TimeFinder/Mirror allows a given Standard device to maintain incremental relationships with multiple BCVs. This means that different BCVs can be established and then split incrementally from a standard volume at different times of the day. For example a BCV that was split at 4:00 a.m. can be re-established incrementally even though another BCV was established and split at 5:00 a.m. In this way, a user can split and incrementally re-establish volumes throughout the day or night and still keep re-establish times to a minimum. Incremental information can be retained between a STD device and multiple BCV devices, provided the BCV devices have not been paired with different STD devices. The incremental relationship is maintained between each STD/BCV pairing by the Symmetrix Microcode.

ü Two BCVs can be established concurrently with the same Standard device ü Establish BCVs simultaneously or one after the other ü BCVs can be split individually or simultaneously. ü Simultaneous. “Concurrent Restores”, are not allowed ü

SnapView is software that runs on the CLARiiON Storage Processors, and is part of the CLARiiON Replication Software suite of products, which includes SnapView, MirrorView and SAN Copy. SnapView can be used to make point in time (PIT) copies in 2 different ways – Clones, also called BCVs or Business Continuity Volumes, are full copies, whereas Snapshots use a pointer-based mechanism. Full copies are covered later, when we look at Symmetrix TimeFinder; SnapView Snapshots will be covered here. The generic pointer-based mechanism has been discussed in a previous section, so we’ll concentrate on SnapView here. Snapshots require a save area, called the Reserved LUN Pool. The ‘Reserved’ part of the name implies that the LUNs are reserved for use by CLARiiON software, and can therefore not be assigned to a host. LUNs which cannot be assigned to a host are known as private LUNs in the CLARiiON environment.

To keep the number of pointers, and therefore the pointer map, at a reasonable size, SnapView divides the LUN to be snapped, called a Source LUN, into areas of 64 kB in size. Each of these areas is known as a chunk. Any change to data inside a chunk will cause that chunk to be written to the Reserved LUN Pool, if it is being modified for the first time. The 64 kB copied from the Source LUN must fit into a 64 kB area in the Reserved LUN, so Reserved LUNs are also divided into chunks for tracking purposes. The next 2 slides show more detail on the Reserved LUN Pool, and allocation of Reserved LUNs to a Source LUN.

The CLARiiON storage system must be configured with a Reserved LUN Pool in order to use SnapView Snapshot features. The Reserved LUN Pool consists of 2 parts: LUNs for use by SPA and LUNs for use by SPB. Each of those parts is made up of one or more Reserved LUNs. The LUNs used are bound in the normal manner. However, they are not placed in storage groups and allocated to hosts, they are used internally by the storage system software. These are known as private LUNs because they cannot be used, or seen, by attached hosts. Like any LUN, a Reserved LUN will be owned by only one SP at any time and they may be trespassed if the need should arise (i.e., if an SP should fail). Just as each storage system model has a maximum number of LUNs it will support, each also has a maximum number of LUNs which may be added to the Reserved LUN Pool. The first step in SnapView configuration will usually be the assignment of LUNs to the Reserved LUN Pool. Only then will SnapView Sessions be allowed to start. Remember that as snapable What is Business Continuity? * Business Continuity is the preparation for, response to, and recovery from an application outage that adversely affects business operations. * Business Continuity Solutions addresses systems unavailability, degraded application performance, or unacceptable recovery strategies.

Failures happen; hardware, software, natural disasters, etc. and downtime has a significant impact. the cost is more then just a financial loss. What can we do to avoid downtime or minimize the length of time we are down? EMC offers Business Continuity Solutions that help address common failures or outages: PowerPath: Host to Storage failures and performance bottleneck of a Host Bus Adapter (HBA) TimeFinder: Family of Products: Local Storage Protection (TimeFinder Mirror, Clone or Snap) SRDF: Family of Products: Remote Protection. (SRDF/S, SRDF/A..) Timefinder is a local replication solution that allows you to make copies of data locally. It is an array based local replication solution, runs on EMC Symmetrix Arrays, and it allows make PIT copies of data, so application can run on symm devices, and in the meantime have replicas for multiple purposes, such as backup, testing, reporting, etc. TimeFinder/Mirror (formerly called TimeFinder), has been the industry-leading local replication product since it was first announced in 1997. It is the only local replication product available that creates true mirror images of its source. Because TimeFinder/Mirror business continuance volumes (BCVs) are full physical copies and appear as a mirror of the standard device, they can be used to increase availability by servicing I/O if the source volume and the source volume’s protection are destroyed. It uses this BCVs vols allowing the user to create complete replicas of its original volumes. TimeFinder/Clone is the newest addition to the TimeFinder product family. Originally released as a feature of TimeFinder in Enginuity™ 5x68, TimeFinder/Clone provides significant configuration flexibility because clone copies do not use Symmetrix mirror positions. This option uses any symmetrix device of like size, (it must be of the same size). TimeFinder/Snap TimeFinder/Snap is different from TimeFinder/Clone and TimeFinder/Mirror as it does not actually create a full copy of the source data. The benefit to customers is that they only need to allocate a small percentage of additional capacity to support changes to the source volume. Since it is not a full copy of the data, TimeFinder/Snap creates pointers back to the original source data. It only uses 30% of the space needed for the other Timefinder options, as it uses save devices.

When managing Timefinder, the normal option is Solutions Enabler, but it can also be managed from Control Center for example. * Solutions Enabler Management tools: * EMC Control Center – Easy, point-and-click access – Excellent for ad hoc TimeFinder operations * SMC: Symmetrix Management Console – Web-based application for managing the Symmetrix – Works the same as the CLI/API – the same rules apply – Makes it easier to perform complex CLI tasks * EMC Replication Manager – Discovers replication environments – Automates replication process – Integrates replication technologies at the application level

* TimeFinder Clone/Base Product – New standard for full -copy – Formerly part of TimeFinder/Mirror – MF SNAP Facility included * TimeFinder/Mirror becomes add-on option – TimeFinder/Mirror recommended for ultra high performance and availability requirements * Existing TimeFinder/Mirror customers “grandfathered” – If on maintenance, will continue to receive TimeFinder/Mirror and TimeFinder/Clone (including MF SNAP) TimeFinder/Mirror allows companies to make more effective use of their most valuable resources by enabling parallel information access, as opposed to traditional sequential information access, thus eliminating the need to do things like quiesce an application for backups. BCV and standard devices should be configured on opposing Front End and Back End Directors for redundant paths. TimeFinder/Mirror Business Continuance is possible due to Business Continuance Volume (BCV) devices. These BCV devices are standard Symmetrix devices that are specially configured to be dynamic mirrors. Each BCV device has its own host address and is configured as a stand-alone device.

If you want to use Timefinder mirror, you will need devices with BCV Attribute given. This can be done using ECC, SymCli or SMC. Once we have BCV, usgin TF Controls, we can do an Establish Process. This process will synchronize all data and can be done online. Completed the Establish process, you can do a Split Operation (PIT), this will the point at when the data is completely replicated. Then the replicated data can be used for any purpose. To check the TF mirror status of the replciation, you can do Query operations, that will provide you the percentage of the process running. If you suffer logical corruption or any problem in your original copy of the data, you can do a Restore, of your PIT replica made in the last Split operation. When issuing the split command with ‘-instant’ option, as soon as given back the prompt, you will be able to utilize the replicated copy of the data, but in the meantime, background operations will continue to run.

When a split is initiated for each specified BCV pair in a device group, the following occurs: command validity is checked. For example, the Symmetrix array makes sure that the standard device has an active BCV mirror and that the standard and BCV devices comprise a BCV pair. − Any pending write transactions to the standard device and the BCV device are destaged. − The BCV device is split from the BCV pair. − If the device is a meta device, all meta members are implicitly split as well. − The BCV device state is changed to Ready, enabling host access through its separate address (BCV001). − Operation with the standard device is resumed and any tracks changed from write operations to the standard device are marked. (This is necessary for updating the BCV device if it is reestablished with the same standard device at a later time.) − If the BCV device has any of its own mirrors, the mirrors are synchronized. The instant (-instant) option improves the performance of a typical split operation by performing a quick foreground BCV split. This option can be made the continual default split mode by setting the following: SYMAPI_DEFAULT_BCV_SPLIT_TYPE = INSTANT in file: /var/symapi/config/option or: C:\Program Files\EMC\symapi\config\options. To remove the instant default option, remove the line entry or comment the line out by using a pound sign (#) at the beginning of the line.

This performs an Instant Split across a group of devices using a single Consistent Split command, thus all BCVs in the group are consistent point-in-time copies. It is used to create a consistent point-in-time copy of an entire system, entire database, or any associated set of volumes. It can create consistent splits on remote BCVs through SRDF.

PowerPath-based consistent split executed by the host doing the I/O: * -rdb option * -ppath option I/O is held at the host before the split. ECA-based consistent split can be executed: * By host doing the I/O, as long as there is a dedicated path for a gatekeeper to perform the control * By a control host (no database), in an environment where there are distributed and/or related databases * -cons option I/O held at the Symmetrix until the instant split operation is completed: * Approximately 3 seconds. Available on Solaris, HP-UX, AIX, Tru64, and W2K platforms Since I/O is held at the Symmetrix, ECA can be used to perform consistent splits on BCV pairs across multiple, heterogeneous hosts.

There are three types of consistent activation available; each offers a restartable copy on the BCV devices: 1. PowerPath 2. ECA 3. Veritas File System or -vxfs

Consistent split using PowerPath Consistent splits using PowerPath can be implemented with an instant (-instant) split where you must also specify either a database or PowerPath device(s) and any pre-action and post-action scripts: To target a database, use the following syntax: symmir -g DgName split –instant -rdb -dbtype DbType [-db DbName] [-preaction Script][-postaction Script] To target the PowerPath standard devices of the group, or just specific PowerPath device name(s) as a target, use the following syntax: symmir -g DgName split –instant -ppath STDDEVS|<PowerPathPdevName...> [-preaction Script][-postaction Script] PowerPath consistent split operations require Version 2.0.1 or higher PowerPath-connected devices on Symmetrix arrays. Through the assistance of PowerPath, a symmir consistent split (supplied with database or PowerPath parameters) initially suspends I/O to the devices that hold the database. This prevents the database application from proceeding. The consistency split command also lets you specify the name of scripts using the -preaction and -postaction script options. The script commands are executed prior to the freeze and after the thaw operation, respectively. TimeFinder/Mirror Restore * During a symmir restore operation, data on the standard is replaced with data from the BCV Volume. * Primary host still has full read/write access to the standard volume * If an application is using the standard volume during a restore, the results would be unpredictable y Applications must be stopped, file systems unmounted, and volume groups deactivated when a restore is initiated Remember that restore is a recovery operation. Applications should be stopped, files systems unmounted, and VG deactivated before you issue the restore command. You will be able tu utilize your STD Volumes (Original source of data) but you won’t like to do that.

TimeFinder/Mirror Extended Operations The extended BCV operations include protect establish, reverse split, and protected restore. These operations add flexibility to TimeFinder configurations and allow for added protection of gold BCV copies. Protected Restore: A protected restore feature allows the contents of a BCV to remain unchanged during and after a restore operation, even while the BCV and standard are joined. Reverse Split: In a reverse split operation, the direction of data flow between the BCV mirrors is reversed. During a reverse split, the fixed BCV mirror (M2) will refresh the moving mirror (M1) after the split operation. This behavior may be desirable when you need to revert to an older copy of the data that was on the BCV before it was established. Protected BCV Establish: In a 2-way BCV mirror configuration for a normal establish, M2 is fixed and can only be updated from M1 after a split. For a 2-way BCV mirror device for a protected establish, both M1 and M2 move to the standard device mirror set and become instantly synchronized and available for updates from I/O activity on the standard device. By default, TimeFinder can “remember” up to eight BCVs. This means that different BCVs can be established and then split from a standard volume at different times of the day. With the Multi-BCV function, up to eight BCVs will be remembered by the Symmetrix. In other words, a BCV that was split at 4:00 a.m. can be re-established, even though another BCV was established and split at 5:00 a.m. In this way, a user can split and incrementally re-establish volumes

throughout the day or night and still keep re-establish times to a minimum. Incremental information can be retained between a STD device and multiple BCV devices, provided the BCV devices have not been paired with different STD devices. Using the environment variable SYMCLI_MAX_BCV_PAIRS, the maximum number of pairs (established or restored) can be adjusted between 1 to 16 BCV devices. TimeFinder/Mirror Concurrent BCVs * Each BCV occupies a mirror position; locally protected R1 Volumes cannot have two BCVs established with them concurrently. * Establish two BCVs with the same standard simultaneously

or one after the other * BCVs can be split individually or simultaneously * Simultaneous “Concurrent Restores” are not allowed Concurrent BCVs is a TimeFinder/Mirror feature that allows two BCVs to be simultaneously attached to a standard volume. The BCV pair can be split, providing customers with two copies of the customer’s data. Each BCV can be mounted online and made available for processing. When you establish a BCV device as a mirror of a standard device, that relationship is known as a BCV pair. When you sequentially establish/split/establish a number of BCV devices over time with a specified standard, that is known as a multi-BCV relationship. You can establish two BCV devices (eight when using emulation mode) as concurrent mirrors of a single standard device all within the same symmir command line. This relationship is known as a concurrent BCV Pair. This feature allows you or an application script to instantly generate two synchronized copies of the standard data.

When the two BCVs are split from the standard, the BCV’s hosts can access the data on either BCV. When establishing concurrent BCV pairs, you can either specify the BCVs, or use the -concurrent option to allow the Symmetrix array to select suitable BCV(s) from the available BCV list. To have this concurrent BCV’s option, you must have at least 3 copies, and remember that you can only restore from one of the BCVs at the time. And if SRDF involved, the 3rd BCV will be on the remote Symmetrix Array. Querying Concurrent BCV Status * When concurrent BCVs are part of a device group, use the –multi option as part of the query command * Invalid track tables are maintained - future concurrent incremental establish operations are possible: symmir –g <testdg1> query –multi * To evaluate the background progress after initiating a split operation: symmir –g <testdg1> query –multi –bg * To verify the status of concurrent BCV pairs: symmir –g <testdg1> verify –concurrent To query the state of a split action on multi-BCVs or concurrent BCVs in a group prod, enter the following command. Note the use of the –multi flag. symmir -g <testdg1> query –multi TimeFinder/Mirror Host Considerations * While established, BCV is a mirror to the STD device and is not accessible by the host * All access to a BCV must be stopped before it is established – Stop application, un-mount filesystem, and deactivate volume group * Any attempt to access the BCV device while it is established results in an I/O error – inq command will hang and timeout – Activating a volume group or mounting a filesystem that resides on a BCV while the BCV is established results in an I/O error * If the host is re-booted or reconfigured while the BCV is established, the BCV is not detected and may cause it to be removed from the host device configuration TimeFinder/Mirror provides host-transparent, "splittable" mirrors that can be used in a variety of ways to enhance application availability and reliability. On the BCV host side, TimeFinder/Mirror operations are not transparent. When the BCV is established with the standard volume, it becomes unavailable to the BCV host system. Any attempt to access the BCV will fail. If the host is re-booted or reconfigured while the BCV is established, the BCV may not be detected and may cause it to be removed from the host device configuration. IBM addresses established BCV issues by automatically configuring the BCV as a Defined device in the AIX ODM. When a reboot occurs, the BCV is not automatically configured. BCV devices that are not established can be configured during reboot with the EMC command mkbcv. BCV devices that are established can be made available with mkbcv after they have been split.

* When a BCV is split, it can be used by the same host or by a different host * There are LVM issues when both BCV and Standard device are accessed from same host • UNIX Systems: Duplicate Physical Volume Ids and Volume Group Ids • Windows 2003: Duplication of disk signatures

Note: Dynamic disk requires a second host and Volume configuration information. A split BCV is an identical copy of the volume it was dynamically mirrored to. When accessing the BCV on the same host as it’s mirror duplicate, volume information must be changed.

Multiple Host Environment

* When a Standard volume and BCV are split, a different host can easily access the volume group by “importing” the volume group – Unix - export/importing – Windows - C:\diskmgmt.msc * There is no conflict with PVIDs and VGDA information because each host only accesses one copy of the data * There is also a performance benefit when BC applications use a copy of the data on a non- production host. Note – PVID = Physical Vol ID –VGDA = Volume Group Descriptor Area When accessing a BCV on a second host, there are no duplicate IDs to consider. The device can simply be brought online using standard host controls. 1. symmir establish # symmir –g <dgname> establis # symmir –g <dgname> query 2. symmir split # symmir –g <dgname> split # symmir –g <dgname> query 3. Mount the file system on Host B From Host B 4. Start your application on Host B This slide shows the basic steps to access a BCV on a secondary host in a TimeFinder environment. It performs a “clean” split operation which requires no form of recovery from the application when restarted at the secondary host. It does not take into account the use of consistent split that would perform a freeze/thaw on the application to allow for a consistent point-in-time copy of data at the secondary host, but could require a level of recovery. Host Considerations: Same Host * When the BCV and standard volumes are accessed by the same host, there is a conflict – The Standard and BCV volumes have identical PVIDs and VGDA information – Host expects the PVID and VGDA to be unique * Additional steps must be taken before the host can use the BCV and standard device. Accessing a BCV on the same host as the original production data requires addressing duplicate disk identifiers in order to access the device. Each operating system has unique methods for addressing duplicate identifiers. The SYMCLI TimeFinder/Mirror component extends the basic SYMCLI command set to include TimeFinder/Mirror or “Business Continuance Commands” that allow you to perform control operations on BCV device pairs within a TimeFinder/Mirror environment. The symmir command allows you to perform a wide spectrum of monitor and control operations on standard/BCV device pairs within a TimeFinder/Mirror environment. The symbcv command allows you to perform support operations on Symmetrix BCV devices.

The Device Group represented in the diagram is currently in a “Split” state. The BCV devices are now accessible via a target host. The data on the BCV devices represents a point in time copy. When the BCV’s are split, the source or production host will continue to read and write to its source devices. This will cause an out of sync state between the standards and all the BCV pairs within the device group. The changes made on the standard devices will be tracked for incremental synchronization at a later time. Identify Accessible Standard / BCV Volumes

• Syminq: Displays volume type and can be used to identify accessible BCV volumes.

The syminq command can be used to obtain SCSI disk device information for one or all locally attached physical devices. Options can be specified to control what data is returned and its presentation as specified in the syminq manpage, which can be found in the EMC Solutions Enabler Symmetrix CLI Command Reference guide. It should be noted here that data returned from issuing any syminq command is not stored in the configuration database or file. Identify BCV Volumes * C:\symbcv list pd displays summary information about all BCV volumes accessible to your host * C:\symbcv list dev displays summary information about all BCV volumes in the attached Symmetrix. * A host “does not need” access to a BCV in order to perform TimeFinder operations.

Configuration and status information is stored in the Symmetrix configuration database file for each device on every Symmetrix array, including BCV devices. You can find all BCV devices on a Symmetrix array and view their physical and Symmetrix device names. You can also display details about the BCV devices, including the BCV device name, Symmetrix device name of the paired standard device, number of invalid tracks for both the BCV device and standard device, and the BCV pair state. To list all BCV devices that are visible to your host, enter:

symbcv list pd To list all BCV devices, regardless of whether they are visible to your host, enter:

symbcv list dev Device Group Operations * Related devices are grouped into Device Groups * BCV devices are associated with Device Groups * All devices in a Device Group must be in the same Symmetrix * All devices in a group must be the same type (RDF1, RDF2, Regular) * By default, relationships of one STD device with up to eight BCV devices can be maintained. (increased up to sixteen - SYMCLI_MAX_BCV_PAIRS) * A device can belong to multiple Device Groups as of SE 6.3 - (Provided the SYMAPI_ALLOW_DEV_IN_MULT_GRPS parameter is enabled) A device group is a logical grouping of Symmetrix volumes. There are three types of device groups: regular, rdf1, and rdf2. A device group with type regular cannot contain RDF volumes. The device group definition will be stored in a file referred to as “SYMAPI database” on the host where the symdg create command was executed. Users can add regular devices and associate BCV devices with a device group by using either the physical device name, or the Symmetrix device name.

The above diagram is a review of what we just created. The TimeFinder Device Group contains two standard devices and two BCV devices. The device group name is “Testdg1”. Using the syminq and symbcv commands, the user has selected symm standard devices 0021 and 0022 and BCV devices 0061 and 0062 when creating this Device Group. When creating device groups for any production application, it is recommended that one documents the environment. With dozens of Device Groups, each containing many devices, the environment can become confusing. Also, careful consideration should be given to any production Device Group name. A Device Group name should represent what data is being mirrored.

Display SYMCLI Device Groups * symdg show displays detailed information about a device group * Group membership * Associated BCV devices The symdg show <Device Group> command is a quick way to show what devices belong to which Device Group. Outputting the results of this command to a file can be used to help create and maintain documentation for any production environment. symmir establish: All Devices * Establishes a BCV mirror to each volume in a device group * Default is to perform incremental establish * -full must be specified the first time a BCV is established to a volume * -exact assigns BCVs in order associated with DG After configuration and initialization of a Symmetrix array, BCV devices contain no data. The BCV devices, like standard devices, have unique host addresses and are online and ready to the hosts to which they are connected. The full establish must be used the first time the standard devices are paired with BCV devices. Optionally, you can target devices in a device group, composite group, or device file:

• symmir -g DgName -full establish • symmir -g CgName -full establish • symmir -f FileName -full establish

To initiate a full establish on all the BCV pairs in the testdg1 device group, enter: symmir -g testdg1 -full establish To initiate a full establish on more than one BCV pair (list) in the testdg1 device group with one command, enter: symmir -g testdg1 -full establish DEV001 BCV ld BCV001 DEV002 BCV ld BCV002

• to show full “Synchronization” has been achieved.

You can perform a query (symmir query) to determine the state of a BCV pair, or all BCV pairs, in the device group. The query is sent via the gatekeeper device to the Symmetrix array, returning with information about the state of the BCV pair(s). Because the Device Group is in an “Established” state, any “Write” activity to the standards will mirror to its BCV pair. During this “Establish” state, the BCV devices are “Not Ready” to any target host.

The verify option used with the –synched flag will enable script to confirm “synchronization”. The following command verifies whether the BCV device pair(s) are only in the Synchronized state. example C:\symmir -g testdg1 -synched verify symmir establish: selected devices * Allows you to override existing device pairing * Default is to perform incremental establish * -full must be specified the first time a BCV is established to a volume. When a full establish is initiated for each specified BCV pair in a device group: * Command validity is checked. For example, the Symmetrix array makes sure both the standard device and BCV device are the same size; the device specified as BCV has the BCV attribute; and the standard device does not already have a BCV device assigned to it. * If the standard device is a meta head device, then the BCV must also share the same meta device properties. All meta members will be implicitly established along with the meta head device. * The BCV device is set as Not Ready to the host. * The BCV device is assigned as the next available mirror of the standard device. * The contents of the standard device are copied to the BCV. To initiate a full establish on one BCV pair, (DEV001 to Sym DEV 0062) in the testdg1 device group, enter: symmir -g testdf -full establish DEV001 BCV dev 0062

The previous slide shows that within a Device Group, a user can select specific device pairs to mirror. In the above example, the storage administrator has selected a full mirror copy of device 0021 to device 0062. Careful consideration needs to be applied when performing this TimeFinder/Mirror functionality. Understanding the application that is being mirrored is a must. In the event that both BCV devices contain a database application, replicating one device at a different time would cause the mirrored environment to be out of sync from each other. If one device contains database system files and the second device contains redo logs, mirroring the two devices independently will result in out of sync database. Storage and Application administrators must fully understanding the application being replicated. To be safe, always create a TimeFinder Mirror replication environment on a “Test” platform before migrating to production.

symmir query * Setting the SYMCLI_DG variable eliminates having to specify the Device Group using the -g flag * The –i (Interval = 5) , -c (Count = 10) options will query the state of the Device Group (Testdg1) ten times every five seconds. Setting the SymCLI device group system variable (SYMCLI_DG) eliminates having to use the device group name. Example c:\Set Symcli_dg=testdg1 c:\Symmir –full establish –nop c:\symmir -synched verify The above symmir query command will display the status of the full establish between Standard Device 0021 and BCV Device 0062. With the –i (interval) and –c (count) flags invoked, the query command will be displayed ten times, every five seconds. Also notice that the –g <device group name> was not included in the above query/verify commands.

symmir split * All devices or specific devices within a device group can be split with a single operation.

The result of a split, changes the BCV device from “Not Ready” to “Ready” to the target host. After an establish operation, the standard device and BCV mirrors are synchronized and the BCV device becomes a mirror copy of the standard device. You can split the paired devices to where each holds separate valid copies of the data, but will no longer remain synchronized to changes when they occur. The target host can now use the BCV device. When a split is initiated for each specified BCV pair in a device group, the following occurs: * Command validity is checked. For example, the Symmetrix array makes sure the standard device has an active BCV mirror and the standard and BCV devices comprise a BCV pair. * I/O is suspended to the standard device until the split operation completes. * Any pending write transactions to the standard device and BCV device are destaged. * The BCV device is split from the BCV pair. * The BCV device state is changed to Ready, enabling host access through its separate address.

* Operation with the standard device is resumed and any tracks changed from write operations to the standard device are marked. (This is necessary for updating the BCV device if it is re- established with the same standard device at a later time.) * If the BCV device has any of its own mirrors, the mirrors are synchronized.

symmir –g testdg1 split Results in a split condition between all standard devices and all BCV devices within a Device Group symmir –g testdg1 split -consistent Results in a consistent split condition between all standard devices and all BCV devices within a Device Group Once the establish operation synchronizes the standard devices to its BCV pairs, one can issue a symmir split. The split will result in the BCV maintaining a point in time copy of its standard pair. Splitting the paired devices results into each holding separate valid copies of the data, but will no longer remain synchronized to changes when they occur. The BCV Devices can now be mounted to the Target Host. symmir restore * Data is restored from BCV to Standard Volume * Default is incremental restore. Specify -full if required The incremental restore process accomplishes the same thing as the restore, but with a major time- saving exception. The BCV device copies only new data that was updated on the BCV device while the BCV pair was split. Any changed tracks on the standard device are also overwritten by the data on corresponding tracks on the BCV device. This maximizes the efficiency of the synchronization process. This process is useful if the results from running a new application on the BCV device were desirable, and the user wants to port the data and new application to the standard device. To initiate an incremental restore on all the BCV pairs in the testdg1 device group, enter: symmir -g testdg1 restore

symmir –g testdg1 –full restore Results in a full track for track copy from the BCV to it’s Standard pair – within Device Group - (Testdg1) Like the full establish operation, a full restore operation copies the entire contents of the BCV devices to the standard device.

Multi-BCV’s Example * Identify two Regular Volumes and four BCVs * Create two SYMCLI Device Groups of type Regular - AMdg and PMdg * Add two Regular Volumes to AMdg. Associate first two BCVs in Amdg * Associate the next two BCVs with PMdg (as yet does not have any Regular Volumes added) * Full establish AMdg, verify synchronization, then split AMdg * Move Regular Volumes to PMdg, full establish, verify synchronization Let’s discuss a specific business example where Multi-BCV’s can be used. The above “Multi-BCV” example has been set up to enable application replication from the same standard devices to two BCV sets. Both AM and PM device groups will enable the creation of “test” and or “development” environments for the target host. The symld “moveall” command enables a set of production standard devices to be moved from one device group to another, giving the user production copies at specific points in time. Another way of accomplishing this would be to enable BCV’s to become members of multiple device groups. Associating a BCV device with multiple device groups - By default, a BCV device cannot be associated with more than one device group at the same time when you are using one SYMCLI configuration database file. However, you can change this default behavior by enabling the SYMAPI_ALLOW_DEV_IN_MULT_GRPS parameter in the options file. The following page identifies the symcli command set for the above business example. 1. Identify two regular volume and four BCV’s 2. C:\symdg -type regular create AMdg 3. C:\symdg -type regular create PMdg 4. C:\symld -g AMdg addall dev -range 092:093 5. C:\symbcv -g AMdg associateall dev -range 056:057 6. C:\symbcv -g PMdg associateall dev -range 058:059 7. c:\symmir -g AMdg establish -full -exact 8. c:\symmir -g AMdg query All devices in group 'AMdg' are in the 'Synchronized' state 9. c:\symmir -g AMdg split -instant –noprompt Move the standard devices to PMdg, - perform a full establish, and verify synchronization. 10. c:\symld -g AMdg moveall PMdg 11. c:\symmir -g PMdg est -full –exact 12. c:\symmir -g PMdg split -instant –noprompt 13. c:\symld -g PMdg moveall Amdg 14. c:\symmir -g AMdg restore 15. c:\symmir -g AMdg split –noprompt TimeFinder/Clones * A TimeFinder Clone is an instant point-in-time copy of a Symmetrix Device – Copying can be immediate or deferred (copy on access) – Copying can be controlled by copy sessions – Copy sessions are maintained on the Symmetrix unit – Copy sessions can be: Created, Recreated, Restored, Activated, Queried, Listed, and Terminated using the symclone command. – Data can be copied from a single source device to as many as sixteen target devices, with automatic background copying for up to eight target devices simultaneously – Does not take-up a “Mirror” position – Remote clone is supported with –rdf When creating a clone image, it is not necessary to first perform a full establish operation as you would with traditional TimeFinder BCVs. Upon activation of the cloned image, it is fully read/write enabled; data is copied in the background as it is accessed. The Symmetrix array is currently limited to sixteen sessions per source device, which can be used for TimeFinder Mirror, Clone or Snap operations. This limits the number of available clone copies that can be created. A total of sixteen concurrent CopyOnAccess copy sessions can be created from a standard device.

TimeFinder/Clone Operations * Create – Create relationship between standard and Clone * Activate – Clone is now active and available immediately for read/write access

– Production I/O is processed against standard * Re-Create – Clone is re-attached to Standard for new point-in-time copy (incremental/differential) * Establish – Re-create and activate * Restore – Re-attached to Standard and incremental or full restore is performed * Split – You can use the symclone split command to split a clone device pair that is in the Restored state. This command requires Enginuity Version 5671 or later * Terminate The create action defines the copy session requirements and sets the track protection bitmap on the source device to detect which tracks are being accessed by the target host or written to by the source host. The target device is made Not Ready to its host and placed on hold status for copy session activity. This prevents other control operations from using the device. The activate action makes the clone ready to the host. It also starts one of the background copying processes. The process, Copy-On-Access or Full Copy, depends on the argument used when the clone session was created. The recreate command allows you to incrementally copy all subsequent changes made to the source device. In other words, it allows you to refresh the contents of the clone with new contents of the source since the clone session has been activated. The restore action restores contents of the clone volume to a restore target. The establish operation creates and then immediately activates a clone session with a single command. The terminate action terminates a source clone relationship. TimeFinder/Clone - Options with Creation * CopyOnAccess – Default (Prior to 72 code) – Only modified tracks are copied to clone volume after activation * Full Device Copy – Full copy in the background starts after activation * Precopy – Full copy in the background starts after creation * Differential – Used with Full Copy or Precopy (implied Full Copy by default) – Required if recreate clone session is planned – Required if incremental restore is planned * Symclone Recreate Before Copied – No need to wait for a clone copy to complete, before activating a new point-in-time with recreate – Symclone recreate or establish is now permitted while the session is in CopyInProg mode – Recreate is a differential copy (modified tracks copied /unmodified not) * CopyOnWrite – Default behavior change utilizes CopyOnWrite instead of CopyOnAccess CopyOnAccess: By default, when a copy session is finally activated, the device pair state will be CopyOnAccess. This means that after activating the copy session, only those tracks that have been written to the source or written/read from the target, will be copied to the target device. A full data copy to the target device will not occur unless all of device tracks are accessed or written to while participating in the active session. CopyOnWrite: Before SE 6.4, the data is copied to the target device when there is change of data on the source or target. But it also copies the data even when the data is read from the target which is unnecessary since the data is not changed and there is no need to copy the data . This impacts the performance since even read operation to the target device destages the data. The symclone status was CopyOnAccess. One can modify this default device pair state to CopyOnWrite by setting the following parameter in the options file to ENABLE.SYMAPI_CLONE_COPY_ON_WRITE = ENABLE | DISABLE Once CopyOnWrite as been enabled and activated, all reads will be handled from the source device and writes to the source device or target device during the active copy session will result in the data being copied to a target device. Full Device Copy Option: When the copy session is eventually activated, data begins background copying so that a full copy of the data will become available on the target device. Precopy Option: The -precopy option can be used with the create argument to start copying tracks in the background, before the clone session is activated.

Differential Option: This option creates an SDDF (Symmetrix Data Differential Facility) session for maintaining changed track information. Creating a differential clone session automatically implies a Full Device Copy. Recreate Before Copied: This feature allows users to recreate the symclone session before the copy finishes. In prior releases the customer had a problem if there was change in the SRC after the initial clone session. The customer had to wait until the previous copy finished in order to make differential changes. Aborting the original session with terminate would destroy the differential information and require a full copy. TimeFinder/Clone - Actions and States of a Session

• A restore operation is permitted only when a clone is in a “copied” state The diagram depicts flow of actions and states of a clone session. The Split action when in a Restored state is new, as of Solutions Enabler 6.3. TimeFinder/Clone - Create a Copy Session * To create a CopyOnAccess session: symclone … create – To create a “CopyOnAccess” Clone session: when activated tracks that have been written to the source or written/read from the target will be copied to the target device * To create a Full “Background Copy” session: symclone … create … –copy – To create a full “copy” – when activated will invoke a background track for track copy of the source to the target clone device. * To create a Precopy session: symclone … create … –precopy – To create a –precopy which enable track for track copy from the source to the clone device before the clone session is activated * To create a Differential clone session: symclone … create … -differential – Creating a differential clone session automatically implies a Full Device Copy The symclone create command: * Makes the target not ready (NR) to its host * Creates a copy session * Sets up the track protection bitmap to provide user access to data on the target as soon as the clone pair is activated * Puts a hold on the target to prevent other control operations (for example, TimeFinder operations), or a copy over previously copied data * The -copy option makes an immediate copy of the entire source to the target. Copying begins upon issuing the symclone activate command * -precopy starts the copy immediately after the session has been created * -differential allows a clone session to be recreated and an incremental restore to be performed

TimeFinder/Clone - Multiple Clone Copies

Up to 16 clone copies of a standard source device can be created. The Symmetrix array is currently limited to 16 sessions per source device, which can be used for TimeFinder Mirror, Clone, Snap, or SDDF (Symmetrix Differential Data Facility) operations. This limits the number of available Clone copies that can be created. A total of 16 concurrent CopyOnAccess copy sessions can be created from a standard device (sessions created without using the -copy option). However, this number is decremented for each current TimeFinder Mirror, Clone, Snap, and SDDF session. For example, if you have a standard device with two paired BCV devices, you can only create up to 14 copy sessions for that standard device. Copy sessions created using the -copy option to create full data copies are limited to eight concurrent sessions, with up to an additional eight CopyOnAccess sessions, for a total of sixteen sessions. Note: The use of -differential uses up an additional SDDF session for a maximum of eight (8) copies. Major Activation Options * Not Ready – The clone volume is placed in NR state: symclone –g testdg2 activate DEV001 sym dev 0063 –not_ready * Consistent – Create a consistent image of a heterogeneous database environment: symclone –g testdg2 activate –consistent Three methods to execute Consistent Activation: * By specifying -ppath and PowerPath STD devices that hold the database * By specifying –consistent * By specifying –vxfs for VxFS filesystem (Solaris and HP only)

Not Ready Option: The -not_ready option can be used with the activate action to cause the target device to remain not ready to its host. The copy session will be activated and the target device will be placed in the Not Ready state. The Clone copy can later be read/write enabled to the host using the symld ready command. Consistent Option: The symclone activate command can be used with the -consistent option to invoke the Enginuity Consistency Assist (ECA) feature. This feature can be used to create Clone copies that are consistent with the database up to the point in time that the activation occurs. The feature suspends writes to the source devices during the activation. When the activation has completed, writes are resumed and the target device contains a consistent production database copy of the source device at the time of activation. Additional details for ECA technology are discussed in the Consistency Technology “SRDF Solutions” module. Refer to EMC Solutions Enabler Symmetrix CLI Version 6.5 COMMAND REFERENCE, P/N 300- 002-181, REV A05 for the complete option list.

Multiple Clone Copies of a BCV

Clone sessions can use BCV as a source volume. This helps isolate production volumes from clone I/O during the copy sessions. The BCV background split must be completed before a clone session creation.

Establish a Copy Session * The establish command combines create/recreate and activate into a single command – Makes behavior more similar to TimeFinder/Mirror * Execute recreate followed by activate . symclone establish * Execute create followed by activate . symclone –full establish * Activate options (-consistent, -preaction, etc.), are performed at activate (not create) time . symclone –full establish –consistent The symclone establish command sets the target device to Not Ready for a short time. Therefore, you may want to unmount the target volume before issuing the command. An establish operation creates and then activates a new clone session. The clone session created by the establish operation is by default equivalent to a session created with –copy and –differential options. The establish operation can also be performed on an existing differential clone session. In which case, the operation performs recreate and activate operations on the clone session. Terminate a Copy Session * The symclone terminate command: – Stops a copy session – Removes any hold on the target – Deletes copy session information from the Symmetrix unit * Normal termination is possible whenever a copy pair is in the Created, Copied, CopyOnAccess, Split or Restore state * The syntax is: . symclone –g testdg2 terminate DEV001 sym dev 0063 Terminating a copy session deletes the pairing information in the Symmetrix array and removes any hold on the target device. Terminating a session while the device pairs are in the CopyOnAccess or CopyInProg state causes the session to end. If the application has not finished accessing all of the data, the target copy is not a full copy. The symclone terminate command is allowed for all TimeFinder pair states. A created / activated copy session may be terminated, but the data on the target device becomes invalid (whether or not data was actually copied to the target). If the state is CopyInProg, then the -symforce option must be applied to terminate the session. This also leaves the target copy as an incomplete copy.

Restore from a TimeFinder/Clone symclone –g testdg2 restore DEV001 sym dev 0063 * Restore can be done to other standards or the source volume * Session must be created with –copy or –precopy * Session must be activated * Wait until the clone is in a “Copied” state * Issue a symclone restore command You can use the symclone restore command to copy target data to another device (full restore), or back to the original source device (incremental restore). In the case of a full restore (-full), the original session terminates and a copy session to the target of the restore starts. In the case of an incremental restore, the original session terminates and an incremental copy session back to the original source device starts. To support this operation, the session must have been created with the -differential option and the device must be in a fully copied state. TimeFinder/Clone Operations Using the “symclone list” command will show all the clone session in the Symmetrix. The above shows a clone session between device 0023 and 0063 has been created but is currently in a non-active state (Created). Also, a clone session between device 0023 and 0064 has been created and activated (CopyOnAccess). The symclone list command:

The symclone query command:

Above is the symclone query on device group testdg1 showing the clone relationship between BCV001, BCV002 to DEV001. One clone pair is in a “CopyOnAccess” state where the other is in a “Created” state. The following command will active the DEV001 to BCV001 pair: C:\ symclone –g testdg2 activate DEV001 sym dev 0063 -nop

The above diagram represents the SYMCLI command set which first creates the device group (testdg2). Next the devices are added t the device group. Next a clone session is created than activated. The last command is querying the state of the device group which shows that an active clone session between devices 0023 and 0064 is in a “CopyOnAccess” state. Create the symclone Session

The symclone create action defines the clone copy session requirements and sets the track protection bitmap on the source device to detect which tracks are being accessed by the target host or written to by the source host. The target device is made Not Ready to its host and placed on hold status for clone copy session activity. This prevents other control operations from using the device. The device pair state transitions from CreateInProgress to Created when complete. The clone copy does not become accessible to its host until the copy session is activated. Check the symclone create Session When performing a symclone query on testdg2, notice that the background copy is active for this pair. This is known as a “PreCopy” state.

With Enginuity Version 5671 and later, you can use the -PreCopy option with the create argument to start copying tracks in the background, before activating the copy session. This allows the early movement of data before the point-in-time clone copy is established.

Activate the symclone Session

This activates the copy operation from the source device to the target device. Activating the copy session places the target device in the Read/Write state and initiates the -precopy option. (Refer to the previous slide where the copy session was created with the –precopy flag). The target host can access the cloned data and has access to data on the source host until the copy session is terminated. Note: Cloned data is made available as a point-in-time copy at the time of activation and not at the time that the session was created. Performing the symclone –g testdg2 query command currently shows that “The background copy setting is active for this pair”, “The Target device is currently associated with this group”, “This Clone session is currently a “PreCopy” state ” and “The pre-copy operation has not completed one cycle”.

Query Clone Session State

Terminating the symclone Session

Terminating a copy session deletes the pairing information in the Symmetrix array and removes any hold on the target device. Terminating a session while the device pairs are in the CopyOnAccess or CopyInProg state causes the session to end. If the application has not finished accessing all of the data, the target copy is not a full copy. The symclone terminate command is allowed for all TimeFinder/Clone pair states. Note: A created and activated copy session may be terminated, but the data on the target device is not valid unless the state had previously been COPIED.

The symclone create –differential Flag

Subsequent cloning to the same target can be performed as differential copying. One can use the –differential option the first time you clone a full copy from the source to the target. All subsequent copying during that copy session automatically becomes incremental (that is, copying only new writes to the source device). A differential or incremental clone operation copies only those device tracks that have changed since the initial full clone was performed. To do this, however, the copy session that existed for the full clone must still exist. You must also set up the differential clone during a full clone operation. The –differential option, therefore, must be used here with the –copy (or –precopy) option. For example: C:\symclone –g testdg2 create DEV001 sym dev 0063 –copy –differential The –differential option creates an SDDF session for the source and target. Using the -differential option, utilizes two SDDF sessions. Query after Creation (-differential) Notice this clone session is a “differential” copy. With Enginuity Version 5671 and later, subsequent cloning to the same target can be performed as differential copying. You can use the –differential option the first time you create a clone copy session. All subsequent copying during that copy session automatically becomes incremental.

The symclone activate –consistent

The symclone activate command can be used with the -consistent option to invoke the Enginuity Consistency Assist (ECA) feature. This feature can be used to create clone copies that are consistent with the database up to the point in time that the activation occurs. The feature suspends writes to the source devices during the activation. When the activation has completed, writes are resumed and the target device contains a consistent production database copy of the source device at the time of activation. The symclone Recreate

Once a clone device is fully copied, the symclone recreate command allows you to incrementally copy all subsequent changes made to the source device (made after the point-in-time copy initiated) to the target device. To use this feature, the copy session must have been created with either the -copy or -precopy option, and the -differential option. In addition, the session must have been activated to establish the new point-in-time copy and to initiate the copy of the changed data. While in the Recreated state, the target device remains Not Ready to the host.

Query after Recreating

Notice the state of the clone session currently in a “Recreated” state. At this point, the target device is not available to the target host. With Enginuity Version 5671 and later, once a clone device is fully copied, you can use the symclone recreate command to incrementally copy all subsequent changes made to the source device (made after the point-in-time copy initiated) to the target device. The symclone Establish

To create and then immediately activate a copy session with a single command, you can use the symclone establish command. Note: The symclone establish command sets the target device to Not Ready for a short time. Therefore, you may want to unmount the target host before issuing the command.

Query of Multi Clone

Note – The “–multi” flag is required to see all the clone sessions. When using the query argument you can display the state of all devices involved in the clone operation by including the -multi option or you can use the verify argument to verify that the data has been fully copied to the target: c:\symclone -g testdg2 query –multi symclone Restore

You can use the symclone restore command to copy target data to another device (full restore), or back to the original source device (incremental restore). In the case of a full restore (-full), the original session terminates and a copy session to the target of the restore starts. In the case of an incremental restore, the original session terminates and an incremental copy session back to the original source device starts. To support this operation, the session must have been created with the -differential option and the device must be in a fully copied state.

Query after Restore

For example, to fully restore data from the original target device (BCV001) created in the example above: symclone -g testdg2 restore -full DEV001 sym ld BCV001 Note: When constructing a symclone restore command, the device receiving the data always appears first in the command, followed by the device from which the data is being copied. TimeFinder/Clone Emulation Mode * Emulation mode allows customers to leverage existing TimeFinder Mirror scripts – symmir commands are converted to symclone commands * Enables – Faster introduction of TimeFinder Clone into existing TimeFinder Mirror environments – Simplified conversion to RAID 5 protected BCVs * Support for all current TimeFinder Clone capabilities – Symmetrix mirror position not required – Immediately read and write accessible – NOCOPY option – TimeFinder Consistency Group support Starting with Enginuity™ Version 5671, TimeFinder supports the ability to map TimeFinder commands to Clone operations. With this feature, TimeFinder commands execute Clone operations. This feature is enabled by setting the environment variable: SYMCLI_CLONE_EMULATION to ENABLED, or if a RAID 5 BCV is encountered. * SE 6.0 added capabilities to match the missing TimeFinder/Mirror functionality: − Recreate and -differential (Ref Note – Page 10 - The use of -differential uses up an additional SRDF session for a maximum of 8 copies) to match the incremental capabilities. − Establish to remove the separation of activate and create − Copy mode -precopy to with the -cycled verify state to simulate the TimeFinder synchronization before split. − Restore to explicitly match restore support including incremental restore − Remote RDF operations for DGs, CGs and the Device File * Only Clones: − Can be protected using any type of supported protection scheme including RAID5 − Can be put into a NOCOPY mode fully suspending "synchronization" I/Os.

Operations Mappings: TimeFinder/Mirror to TimeFinder/Clone

The following limitations apply to this feature: * The TimeFinder reverse split feature is not allowed * Restores are always protected restores * Incremental Restore is only accepted if all tracks were originally copied from the source prior to the restore taking place * The SYMAPI_DEFAULT_BCV_SPLIT_TYPE, SYMAPI_DEFAULT_BCV_RESTORE_TYPE, SYMAPI_DEFAULT_BCV_ESTABLISH_TYPE option file settings are ignored. * Any Snap or Clone operations to an emulated device are rejected * The maximum number of BCVs that can be incrementally established with a Standard device is 8, instead of the 16 allowed by TimeFinder. SYMCLI_MAX_BCV_PAIRS can, at most, be eight * The BCV states Split-Before-Restore and Split-Before-Sync are not valid states for an emulation device. In both cases, a forced Split completes the operation. Clone Emulation in Action Setting the “SYMCLI_CLONE_EMULATION” variable The device 0063 is a BCV, which automatically triggers a TF/clone emulation. The device is paired with a standard as follows: symmir -g testdg1 establish DEV001 sym dev 0063 –full -nop 'Full Establish' operation execution is in progress for device ‘DEV001‘ paired with BCV device ‘0063' in device group ‘testdg1'. Please wait... 'Full Establish' operation successfully initiated for device ‘DEV001‘ in group ‘testdg1' paired with BCV device ‘0063'. The translation of TimeFinder / Mirror to TimeFinder / Clone command is done by the Symmetrix. The only way that users know the emulation is in effect is to perform a symdev show on the device, as displayed on this slide.

TimeFinder/Clone or TimeFinder/Mirror?

Clone advantages include: * Relieving the four-mirror limit on the STD device (since clones do not require mirror positions). Since no positions are taken up by BCVs, more are free for SRDF configurations, including SRDF/Star and Concurrent SRDF. * Using more than two Clone Copies Concurrently. Today we are limited by the mirror positions available and number of dynamic mirrors a device can have. Using clone emulation mode, we allow up to eight concurrent BCV devices to be established to the STD. * Using more than one protected BCV establish. Today, a protected BCV establish (for BCV devices that are mirrored) moves both mirrors of the BCV device to the STD. This uses up the mirror positions in a way that only a single BCV device can be "protected established" to the STD. Using clone emulation mode, the mirror restrictions are removed and you can perform a protected. BCV establish with more BCV devices. * For the sole purpose of using RAID-5 BCVs. * TimeFinder/Mirrors strengths are higher performance and availability needs. TimeFinder Command Line Interface:

TimeFinder/Snap for Symmetrix DMX * Snapshots create “logical” point-in-time images of a source volume * Requires only a fraction of the source volume’s capacity * Up to 15 snapshots can be created from a source volume and are available immediately * Setting the SymCli_Multi_Virtual_snap variable to enabled will provide capability of up to 128 snaps session * Complements TimeFinder and provides unmatched replication flexibility

TimeFinder Snap creates space-saving, logical point-in-time images or “snapshots.” The snapshots are not a full copies of data; they are logical images of the original information, based on the time the snapshot was created. It’s simply a view into the data. A set of pointers to the source volume data tracks is instantly created upon activation of the snapshot. This set of pointers is addressed as a logical volume and is made accessible to a secondary host that uses the point-in-time image of the underlying data. Some documents may indicate that the maximum copy session is sixteen. The fact is that users can create up to fifteen copy sessions and the Symmetrix reserves one session for restoring purpose. Solutions Enabler supports up to 128 snaps from a source device. This support requires that you enable the following SYMCLI environment variable: SYMCLI_MULTI_VIRTUAL_SNAP = ENABLED

Snap Operations Overview This slide illustrates a virtual copy session where the controlling host creates a set of pointers to reference where the original data resides. The pointers reside on the Virtual Device. TimeFinder/Snap functionality is managed via copy sessions, which pair the source and target devices. Sessions are maintained on the Symmetrix array and can be queried to verify the current state of devices. A copy session must first be Created that defines the Snap devices in the copy operation. Once the session is subsequently activated, the target virtual devices then become accessible to its host. When the information is no longer needed, the session can be terminated. Snap operations are controlled from the host by using the symsnap command to create, activate, terminate, and restore the Snap copy sessions.

TimeFinder/Snap * TimeFinder Snap captures a point-in-time view of a source volume by duplicating only the changed data: – The target is a “slim” virtual device that contains pointers to original source data – The target can be mounted like any other volume – Copying only occurs when there are writes to the source or target – The pre-image of data that has changed is copied to a save device, consuming a fraction of the space required to copy the entire source – Copying is controlled by copy sessions, which are created, activated, queried, listed, and terminated using the “symsnap” command. TimeFinder Snap allows you to make copies of data simultaneously on multiple target devices from a single source device. The data is available to a target host instantly. However, you can copy data from a single source device to as many as fifteen(15) target devices. This source device can be a Standard or a BCV device. (BCV in a Split State). A target device is a virtual device that consumes negligible physical storage through the use of pointers to track data. TimeFinder/Snap - Copying Data Once the TimeFinder Snap session is activated: – The first time a track on the source is written to, the original data on the source is copied to the save device, the VDEV pointer is changed to point to the save device; “then” the source is changed. – “Copy on First Write” applies to writes to the VDEV as well. TimeFinder Snap uses a process called “Copy on First Write” when handling writes to the production data at the same time a snapshot is running. When a host attempts to write to data on the production volume, the original track is first copied to the Save Area, then the write is processed against the production volume. This process of pointers maintains the consistent, point-in-time copy of data for the ongoing snapshot. Striping the Save Device * The copy-on-write is repeated every time a different track on the source is first written to. * Tracks are striped in a round-robin manner to the Save Devices to improve performance. The save pool device is not host accessible, but accessed only through the virtual devices that point to it. Save devices provide a pool of physical space to store snap-copy data to which virtual devices point. Tracks are striped by the microcode in a round-robin manner to save devices to improve performance. When savedevs are configured across the backend, they are separated first by director, then channel, and then by disk. Terminating a Copy Session

* When a Copy Session is Terminated: – The virtual device is made not ready. – Tracks on the save device(s) are reclaimed if they are not referenced by any other copy session. – The copy session structures are freed-up. As sessions are terminated, the space in the Save Area is released and available for re-use by other snap sessions.

Multiple Save Device Pools

Symmetrix SAVE devices are specially configured devices (not mapped to the host) that provide pooled physical storage space used to store pre-update images or changed tracks during a virtual copy session. Devices within a SAVE device pool act as a group to store data in striped form. Symmetrix supports the creation of multiple named SAVE device pools, allowing symsnap commands to use a particular pool. Save Area capacity can also be allocated based on application requirements, for example: * Allocate more space for Write-intensive applications. * Allocate more space for long duration snapshots. The -svp option can be used with the create action to specify which SAVE device pool to use for an operation. For example, to instruct the copy session to use the SAVE device pool Accounting, use the following symsnap command: symsnap -g BackupDB create DEV002 vdev ld VDEV005 –svp Accounting Operations and States

When a virtual device is in a “Not Ready” state, it can participate in a new snap session. The symsnap create argument creates a copy session, and sets the track protection bitmap on the source device to protect all tracks and detect which tracks are being accessed by the target host or written to by the source host. The target virtual device remains Not Ready to its host and placed on hold status for copy session usage. This prevents other control operations from using the device. The device pair state will transition from “Create In Progress” to “Created” when complete. When the copy session is activated, a point-in-time copy of the data becomes available to a target host. The state of the device pair is “CopyOnWrite”. This means that any first write to the source or target device during an active snap session will result in data being copied to an additional save device for storage. The target host accesses the copy through the use of device pointers stored on the virtual device. The pointers keep track of where the data is located in physical storage for accessing the point-in-time copy from the time of activation.

Symsnap restore command is used to restore changed tracks to the desired restore target, which can be either a Symmetrix standard or BCV device. After users issue the command, the status of a virtual device will change from Restore in Progress to Restored. The snap session in the Restored state must be terminated before the snap session of the source can be terminated. Terminating a session with the symsnap terminate command while the device pairs are in the CopyOnfirstWrite or Restored state, will cause the session to end. Once the virtual copy session is terminated, the information is no longer available on the virtual device. Consistent Activation * Four methods to execute “Consistent Activation” * Specify -ppath and PowerPath STD devices that hold the database – Specific devices; supply the device name – All devices using keyword SRCDEVS – Use –rdb for automatic mapping of DB devices * Specify –consistent * Specify –vxfs for VxFS filesystem (Solaris and HP only) * Consistent activate command -both_sides – For devices with an RDF State of Synchronized – Ability to generate a consistent activate for both the local and remote VDEVS at the same time. There are four types of consistent activation available; each offers a restartable copy on BCV devices: – PowerPath – ECA

– Veritas File System – -both_sides

symsnap Operations * Common Snap operations: – Create – Activate – Restore – Terminate C:\symsnap –g testdg3 create DEV001 sym DEV 00A1 –svp training C:\symsnap –g testdg3 activate DEV001 sym DEV 00A1 C:\symsnap –g testdg3 restore DEV001 sym DEV 00A1 C:\symsnap –g testdg3 terminate DEV001 sym DEV 00A1 –restored C:\symsnap –g testdg3 terminate DEV001 sym DEV 00A1 The SYMCLI symsnap – Create, Activate, Restore, and Terminate command set are displayed. Configuration Considerations * Cache – There is some additional cache required for TimeFinder Snap snapshots – Total number of VDEVs (snapshots) must be accounted for in cache calculations * VDEV (snapshots) – Are persistent, – On 56XX small physical devices must be allocated (1/600th the size of a source device) – Consume SYM device ID and count as a DA target – Are not convertible to / from other device types (Standard / BCV) Additional Cache is required for Snap snapshots. The number of VDEVs must be included in cache calculations. VDEVs are not customer configurable. Symmetrix volume limits must be respected, VDEVs consume device IDs and count as DA targets; VDEVs count towards hypervolume limits for physical devices. * SaveDev (Save Area) – Physical device(s) store original data copied from source on Copy on First Write – Recommendation of one SaveDev for every five snapshots that will be active at the same time (@20% change rate) – SaveDev should be spread over as many physical volumes as possible – SaveDev consumes SYM device IDs and count as DA targets – Data is striped across entire Save Pool * SaveDev (Save Area) Monitoring – SaveDev thresholds can be set and notifications can be sent – SaveDev can be added dynamically – When Save Area fills, all snapshots are terminated one by one

Symmetrix volume limits must be respected. It cannot exceed total Symmetrix budgets for either device ID limits or DA target limits. SaveDev’s consume device IDs and count as DA targets and cannot exceed hypervolume limits on physical devices since SaveDevs count towards hypervolume limits for physical devices. Use symsnap monitor to track the amount of available space of the save area. The symsnap monitor command can also be used to automatically run an action if a predetermined threshold for space has been crossed. Use -percent nn –action script_pathname to specify the percentage full threshold and script associated with it. Save Device Space Considerations * Virtual Devices do not pre-allocate space for all potential copy-on-write tracks * It is possible to run out of free space on save devices * If Writes can not complete due to no free space: – The target VDEV goes Not Ready (NR), applications beware! – Copy-on-write is disabled and the source track is changed * Draining Save Devices . Permits a disable command to work on an active save device . All active tracks copied (drained) to other devices in the save pool . Protections against disabling the last device or causing the pool to overflow and terminate snap session . Symconfigure syntax remains unchanged: . disable dev SymDevName[:SymDevName] in pool <poolname>, type = snap Monitoring Save Device Space * Use symsnap monitor to automatically run an action: C:\symsnap monitor -sid 24 -percent 1 –action one_percent_script.sh –svp DEFAULT_POOL -i 10 –c 5 * The above command will monitor the default save pool (“DEFAULT_POOL”) every 10 seconds for 5 iterations. If the save pool usage reaches 1 percent, then the command will execute the one_percent_script.sh script. You can monitor SAVE devices by using the symsnap monitor command to check the percentage full. When devices reach the specified percentage, an optional action script can be executed by the application to preserve the data or terminate sessions. The following is an example of the monitor command: symsnap monitor -percent 80 -action SaveScript -norepeat -i 60 –svp Accounting In the above example, the SAVE device pool “Accounting” will be monitored every minute for percentage full. When the percentage of the SAVE devices are 80 percent full, the associated action script SaveScript will be executed each time the threshold of 80 percent is met. The first command above will list all the Save Pools. The second command will list all the devices within a save pool The above example will list all the devices within the “training” save pool. C:\symsnap list –pools C:\symsnap list –svp training -savedevs Using the symconfigure command set the storage administrator can create save pools to be used against a specific application. * First use the Symconfigure to create a save pool * Next, disable save devices from the default save pool * Next, add the disables save devices to the new save pool * Than enable the devices within the new save pool For additional information on creating save pools for snap sessions, reference Symmetrix Array Controls CLI, version 6.5, Product Guide P/N 300-002-940, REV A05. List all the Save Pools and Devices within a Pool c:\symsnap list –pools c:\symsnap list –svp training –savedevs

View Save Pool’s Details c:\symsnap –sid 97 show pool training

symsnap –sid <id> show pool <poolname> <- Shows detailed information about the specified “save device” pool. c:\symsnap list –svp training –savedevs The above command shows detailed information about specified “save devices within a specific save pool. The symsnap list command with the –savedevs option displays any “save” devices that have been configured on this Symmetrix array to hold data for virtual devices. The display indicates there has not yet been any virtual device activity that caused tracks of data to be copied to these SAVE devices. Add VDEV to Device Group C:\symdg create –type regular testdg3 C:\symld –g testdg3 add dev 0025 C:\symld –g testdg3 add dev 00A1 C:\symdg list Reference the above set of commands. The first command creates a testdg3 device group. The second command adds device “0025” as the source from symm ID 97 to the testdg3 device group. The third command adds device “00A1” to the device group and names it “VDEV001”. The last command displays a list of all the device groups which now contain testdg3. symsnap create c:\symsnap –g testdg3 create DEV001 sym DEV 00A1 –svp training The symsnap create action defines the copy session requirements and sets the track protection bitmap on the source device to protect all tracks and detect which tracks are being accessed by the target host or written to by the source host. The target virtual device remains Not Ready to its host and placed on hold status for copy session usage. This prevents other control operations from using the device. The device pair state transitions from CreateInProg to Created when complete. The virtual data becomes accessible to its host when the copy session is activated. symsnap activate The symsnap activate command places the snap pair in the CopyOnWrite (copy-on-first-write) state and the target virtual device in a Read/Write (RW) state. The actual copying of data is deferred until you modify tracks on either the source or the target devices. c:\symsnap –g testdg3 activate DEV001 sym DEV 00A1

symsnap query / list c:\symsnap –g testdg3 query c:\symsnap list At this point, using the symsnap “query” or “list” commands will report the state of the device pair for the snap session as “CopyOnWrite”.

symsnap restore * Three types of restore operations can be performed for virtual device copy sessions: – Incremental restore back to the original source device – Incremental restore to a split BCV – Full restore to any standard or split BCV device Three types of restore operations can be performed for virtual device copy sessions: 1. Incremental restore back to the original source device. 2. Incremental restore to a BCV, which has been split from its original standard source device but maintains the incremental relationship with the source. 3. Full restore to any standard or split BCV device outside of the existing copy session. The target device of the restore must the same size and emulation type as the source device. Since Enginuity Version 5670, all original Snap copy sessions are maintained when performing a restore operation. A new restore copy session between the source device and restore target device is created. A restore operation can only be performed if an additional copy session is available for use. C:\symsnap -g testdg3 create DEV001 sym DEV 00A1 -svp training C:\symsnap -g testdg3 activate DEV001 sym DEV 00A1 C:\symsnap -g testdg3 restore DEV001 sym DEV 00A1 C:\symsnap -g testdg3 query

The above command initiates, by default, an incremental restore from the target device “00A1” to the source device “DEV001”. Note: Prior to performing a restore operation, the user should stop all activity on the file system. This can be accomplished by un-mounting the file system. The symsnap restore command initiates an incremental restore operation that restores from the target VDEV device to the source DEV001. Symsnap query with the –multi flag c:\symsnap –g testdg3 query –multi The symsnap query command with the –multi option flag displays all the current copy sessions associated to the testdg3 device group. c:\symsnap -g testdg3 terminate DEV001 sym DEV 00A1 -restored C:\symsnap -g testdg3 terminate DEV001 sym DEV 00A1 The restored copy session must be terminated first, before the original copy session can be terminated. To terminate the restored copy session, enter: symsnap –g <testdg3> terminate <DEV001> vdev DEV <00A1> -restored After the restore session has been terminated the original copy session can be now be terminated. symsnap –g <testdg3> terminate <DEV001> vdev DEV <00A1> When to Use TimeFinder/Snap The following table represents when TimeFinder Snap vs. TimeFinder Clones / Mirrors should be used, and is a general guideline that should be used to generate discussion regarding the appropriate TimeFinder solution. Snap Summary TimeFinder/Snap functionality is managed via copy sessions, which pair the source and target devices. Sessions are maintained on the Symmetrix array and can be queried to verify the current state of the devices. A copy session must first be created that defines the Snap devices in the copy operation. Once the session is subsequently activated, the target virtual devices then become accessible to its host. When the information is no longer needed, the session can be terminated. Snap operations are controlled from the host by using the symsnap command to create, activate, terminate, and restore the Snap copy sessions. The Snap operations described in this chapter explain how to manage the devices participating in a copy session through using the SYMCLI.

Symmetrix Management Console (SMC) * Independent, light-weight, web-based application – Simple and easy to use browser interface – Hosted on small Windows/Linux/Solaris server – Enables remote access and management from nearly any client * Enables access, configuration, and basic operation of Symmetrix arrays – Supports all configuration capabilities of Solutions Enabler/CLI * Supports multiple generations of Symmetrix – Enginuity version 5x68 and newer * Provides day-one support of new Symmetrix features when released * Adding full-feature ControlCenter does not require management data to be migrated EMC completes the device management offering with the addition of Symmetrix Management Console (SMC for short), enabling you to deploy a light-weight, web-based Graphical User Interface, or GUI, for performing device management in Symmetrix environments. You can now choose the right management product or products to meet your specific requirements. Almost anything you can do using the Solutions Enabler CLI can now be done using the SMC GUI. SMC will manage all Symmetrix systems running Enginuity version 5568 and newer, and will support new hardware and software features and functionality at the time of product release. Addressing customer demands for enhanced platform interoperability, SMC is an independent application which runs using its own lightweight Windows/Linux/Solaris server. The client runs in a browser window, therefore supporting nearly any client with remote access to the server. The SMC GUI features closely match Solutions Enabler CLI features to include all basic monitoring, configuration, and control of Symmetrix arrays. SMC has no Symmetrix-related database other than the Solutions Enabler database, so all data automatically transfers to the full-feature ControlCenter when discovered by the Symmetrix agent. Due to the combination of being light-weight yet feature rich, early response to SMC shows it to be a leader in this type of product. SMC Functionality * Access Management – Manage users, permissions/roles – Symmetrix Access Controls * Configuration Management – Create devices, map and mask devices, create device groups, set Symmetrix attributes * Replication Management – TF/Clone, TF/Mirror, TF/Snap, SRDF/S, SRDF/A, SRDF/DM, Open Replicator, Optimizer * Alerts and Monitoring – Monitor Device status, device attributes, operations status – Monitor array alerts The major components of the SMC Interface are highlighted here. To monitor and control operations, select a single object or folder of multiple objects in the Navigation Tree. The View Bar is used to switch between five different views: Properties, Config Session, Alerts, Command History, and Replication Monitor. The view currently selected displays in the view area. The View button and corresponding view display are color coded to match. The Alert counter in the top right, which also selects the Alert View, Tree Selection, View selection, and object selection within the View area, determines the current display. The View may be split horizontally into two or three areas, depending on the detail associated with the selection. Device Groups and Composite Groups * SRDF and TimeFinder operations in SMC require Device Groups or Composite Groups (DG/CG) * All DG/CGs in the default symapi_db.bin or GNS (if active) are available in SMC * DG/CGs can be created, deleted, renamed and devices can be added and removed as needed * SRDF and TimeFinder operations are invoked by selecting a DG or CG and using the Replication option * Device Masking operation can also be invoked by selecting a DG * Devices can belong to multiple DGs SRDF and TimeFinder operation in SMC requires the use of either Device Groups or Composite Groups. SMC has visibility to all the DG/CGs that exist in the symapi_db of the SMC Server host and to all device groups, if GNS is in use. The DG/CG Wizard allows the creation of new DGs and CGs. Groups can be deleted and renamed and devices can be added or removed as necessary. The Mapping and Masking dialogs can be launched directly from a Device group, making it convenient to quickly change the allocation configuration of a whole data set. The same device can be added to more than one device group or consistency group at the same time, making it easier to manage complex synchronization environments. For backwards compatibility, the SYMAPI_ALLOW_DEV_IN_MULT_GRPS option must be enabled in the SYMAPI configuration file (<install>/symapi/config/options) after install or upgrade.

Device Types in DG/CG

The Devices types supported in SMC are shown on the slide. The STD Devices can be Regular (non SRDF Devices), SRDF R1s, or SRDF R2. Devices can be added as Clone Target devices on the local or the remote array, and remote Virtual devices can be added. The device group dialog displays only valid devices when choosing a device type. For instance, when adding remote Clone Target devices, only eligible devices configured on the remote array display in the dialog. Creating Device Groups * Device Group Management à Create Device Group

SMC Device Group creation is a multi-step process, similar to creating SYMCLI device groups using symdg, symld, and symbcv. The wizard is launched by right clicking the Device Group Folder and choosing the Device Group Management Æ Create Device Group option. In this example, we create a Device Group of Type Regular and associate local VDEVSs, local BCVs and local clone targets. The first step is to choose a name for the Device Group (SMCDG in this example), then the Device Group Type (Regular in this example because we want to add STD devices). Click on STD in the Device Type Column and add the required STD devices from the Available box to the Group Members box. The bottom of the wizard gives a summary of the selections that have been made. In the next slide, the local BCVs, VDEVs and the local Clone target selections are made.

The local BCVs, VDEVs and the local clone targets (TGT) are added to the device group. Click OK to complete the creation of the device group. Device Group Created Successfully

A completion message is displayed. The completion message reports types of devices added, in this case STD, BCV, VDEV, and TGT. The new group device group, SMCDG, is displayed in the tree view with folders for the STD, BCV, VDEV, and TGT devices. Replication Operations Supported by SMC Almost all replication technologies available on the Symmetrix Arrays can be monitored and managed via SMC. At the present time, SRDF/Star cannot be managed via SMC. Management of TimeFinder operations is shown in this module.

TimeFinder/Snap Operations * Replication à TimeFinder/Snap TimeFinder/Snap operations are performed by right clicking a Device Group and choosing Replication à TimeFinder/Snap. The TimeFinder/Snap dialog has two pages. On Page 1, the device pairings need to be setup. Use the exact pairing option or alternately click on the Edit Pairs button to choose the device pairs from within the device group. In the example shown here, the exact pairing is used. The Snap action to be performed must be chosen on this page as well. The most logical action for the selected pairs is displayed as the default action. Choose a different action from the Action drop-down menu, or return to the default by selecting the Reset to default action radio button. In this example, the Snap session has not yet been created so the action defaults to Create. The SAVE pool that this Snap session uses should also be specified via the Pool pull down. Select the device pairs by highlighting the rows or clicking on Select all. On Page 2, the relevant options for a given action can be chosen then click Finish to execute the operation. A confirmation dialog displays, followed by a success/failure dialog. In a similar manner, the TimeFinder/Snap dialog can be used to Activate and Terminate Snap sessions. Although this example is for TimeFinder-Snap, the sequence of steps is generally applicable to most other Replication dialog sequences.

* Activate Snap Session After the Snap session has been created, it can be Activated. The slide shows the activation of the Snap session. The Consistent option is available when activating the Snap session. The “Snap Query” tab in the Properties view shows information about the Snap session.

TimeFinder/Clone Operations TimeFinder/Clone operations are performed by right clicking a device group and choosing Replication à TimeFinder/Clone. Choose the Clone Type: Local or Local TGT. Local allows you to pair the STD devices in the DG/CG with other STD devices or with any associated BCV devices. Local TGT allows you to pair STD device in the DG/CG with the associated TGT devices in the group. Device pairs can be defined by using the Edit Pairs button or by using Set Exact pair or Optimize pairs option.

With Edit pairs, the device pairs can be defined in the Add/Remove Clone pairs dialog.

The Clone dialog also has two pages, like most of the other replication dialogs. The possible clone actions are listed on the slide. The wizard will only show those actions that are appropriate based on the clone pair state. For example, you cannot Activate a clone session if it has not been created. * Clone Actions: – Create – Activate – Terminate – Establish (Full/Incremental) – Restore (Full/Incremental) – Recreate

TimeFinder/Mirror operations are performed by right clicking a device group and choosing Replication à TimeFinder/Mirror. Device pairs can be defined by using the Edit Pairs button or by using Set Exact pair or Optimize pairs option. With Edit pairs, the device pairs can be defined in the Add/Remove BCV pairs dialog.

The TimeFinder/Mirror dialog also has two pages, like most of the other replication dialogs. The possible TimeFinder/Mirror actions are listed on the slide. The wizard will only show those actions that are appropriate based on the pair state. For example, you cannot Split a STD/BCV pair if it has not been Established.

Enabling Replication Monitor * Administration à Replication Monitor à Configure DG/CG Monitor SMC Replication Monitor can be used to monitor SRDF and TimeFinder/Snap sessions. Replication Monitoring has to be enabled on a per Device/Composite Group basis and thresholds (on a per Symmetrix basis) have to be set for SRDF/A Cache usage, cycle time, DSE Pool Utilization and for the TimeFinder/Snap SAVE Pool Utilization. As shown on the slide, TimeFinder/Snap monitoring is enabled for the SMCDG Device Group. For SRDF Device groups SRDF/A or SRDF/S monitoring can be enabled.

Replication Monitor – Thresholds * Administration à Replication Monitor à Configure Thresholds Choose the metric from the Component pull down. The thresholds can be edited by clicking on the number and entering the desired value. In this example, the Snap Save Pool Utilization thresholds have been set. Three severity levels per metric – warning, critical and fatal à

Replication Monitor View * SRDF/S, SRDF/A, and TimeFinder/Snap Dashboard * Monitoring can be enabled by group and replication type * User-configurable threshold alerts The Replication Monitor view gives a dashboard overview of the SRDF/S, SRDF/A, and TimeFinder/Snap replication state of device groups. It quickly shows the state of synchronization, the device state, and displays user-configured threshold alerts. In this example, we see the TimeFinder Snap status of device group SMCDG is being monitored. The Dev State information is listed as Normal in this case. This indicates that the Snap pool utilization is below any of the configured thresholds.

SRDF Recovery Point Objectives (RPO)

Business Needs Drive the Technology Choice

Recovery Point Objective (RPO) is the maximum amount of data loss an application can tolerate as measured in time. In other words, the amount of data loss that can be tolerated (cost of transaction versus risk). Individual customer business needs drive the technology chosen to meet specific recovery point objectives. This is also known at RPO (Recovery Point Objective). Data Characteristics that Influence Data Storage Decisions Several factors affect the value of data, including: legislation, which can mandate how long the data must be accessible, and by whom; business processes that are tied to points in time (book closing, quarterly reports, tax deadlines, billing cycles, etc.), and business processes that are tied to customer satisfaction service levels associated with the data as its purpose changes. For example, data can start out as transactional, then migrate to billing, then reporting and customer service, then to scoring data for a marketing system, and finally to archival. The usefulness of data to the business will vary over time, hence the necessity to have immediate access to the data changes. The decisions about where data is placed in the storage infrastructure and the methods used to protect that data are fundamentally driven by three factors: The time required to access the data relative to the cost of the access (that is, usefulness to the business versus cost). Recovery Time Objective (RTO) refers to the maximum time a company budgets to bring an application back online in the event of a disaster. In other words, the time it takes to recover the data once a disaster or other recovery event is declared (risk versus cost). Each change in data placement and protection criteria represents a stage in the life cycle of the data and is directly related to the usefulness or importance of the data to keep the business functioning. What are SRDF Solution Sets:

• SRDF - Symmetrix Volumes are mirrored between geographically dispersed locations • SRDF - Maintains real-time physically separate mirrors • SRDF/A - (Asynchronous) Maintains near real-time physically separate mirrors of selected volumes • SRDF/AR - (Automated Replication) Utilizing SRDF and TimeFinder to insure Business Continuity Replication. • Mirror copy can be split and used for disaster recovery or business continuance applications •

The SRDF family of software is the most powerful suite of remote storage replication solutions available for disaster recovery and business continuity. It leverages high-end Symmetrix storage architecture to offer unmatched deployment flexibility and massive scalability so you can meet mixed service level requirements with minimal operational impact. The SRDF family is the most widely deployed set of high-end remote replication solutions, and is installed in tens of thousands of demanding environments worldwide. The SRDF family is the only product that provides cross volume and storage system consistency, tight integration with industry-leading applications, and automated management for simplified usage.

SRDF Solutions

EMC has several remote replication offerings for various service level requirements. For zero data exposure, EMC offers the industry leader for synchronous mirroring: SRDF. However, as with any synchronous solution, there are characteristics that must be understood. Distance is limited by application time-outs and bandwidth must be sized for peak workload at all times. SRDF/Asynchronous is a solution for service level requirements that need Recovery Point Objectives (RPO) in the seconds-to-minutes area. SRDF/AR delivers solutions that combine SRDF with TimeFinder to create single-hop and multi-hop environments for specialized needs.

SRDF Cascaded is a three-site disaster recovery and data mirroring solution that provides enhanced replication capabilities, greater interoperability and security, and a multiple ease-of-use improvement. SRDF/Star is a three-site one disaster recovery solution that uses concurrent RDF technology to replicate data from a primary production site (referred to as the workload site) to a nearby remote site and a distant remote site. Data is transferred in SRDF/Synchronous (SRDF/S) mode to the nearby remote site (referred to as the synchronous target site) and in SRDF/Asynchronous (SRDF/A) mode to the distant remote site (referred to as the asynchronous target site). SRDF/Star provides consistent data protection and incremental data recovery between target sites in the event of a workload site failure. These solutions offer different RPOs and have different requirements for bandwidth, supported distances, etc. No matter what your requirements are, EMC can help deliver the right Remote Replication Solution. SRDF Management Tools

In addition to SYMCLI (Solutions Enabler), SRDF can be managed by a variety of management tools such as EMC Control Center and Symmetrix Management Console. Through its graphical user interface, EMC Control Center and SMC software will organize related devices into device groups. SRDF operations may be performed on all devices in this device group by using a single command. The group information is maintained in the SYMAPI database. Another management tool is EMC Replication Manager. It is an application that automates, simplifies, and manages disk-based replications by using SRDF operations. * Solutions Enabler * EMC Control Center – Easy, point-and-click access – Excellent for ad hoc TimeFinder operations * SMC: Symmetrix Management Console – Web-based application for managing the Symmetrix – Works the same as the CLI/API – the same rules apply – Makes it easier to perform complex CLI tasks * EMC Replication Manager – Discovers replication environments – Automates replication process – Integrates replication technologies at the application level Symmetrix Remote Data Facility (SRDF) * Facility for maintaining real-time or near-real-time physically separate mirrors of selected volumes * Uses no host CPU resources • Mirroring done at the storage level * Operating system independent • Open Systems • Mainframe

Symmetrix Remote Data Facility (SRDF) is a Symmetrix system based business continuance, disaster recovery, restart, and data mobility solution. In the simplest terms, SRDF is a configuration of multiple Symmetrix units that maintains real time copies of logical volume data in more than one location. The Symmetrix units can be in the same room, in different buildings within the same campus, or hundreds and even thousands of miles apart.

SRDF Source and Target Volumes * Symmetrix Logical Volume types: • SRDF Source or R1 Volumes: Primary Volume with R/W access to local host • SRDF Target or R2 Volumes: Backup Volume used for DS or DR Applications * The attached host is unaware of SRDF protection

This slide displays the representation of the mirror positions when both the Source and the Target SRDF Logical Volumes have local protection (RAID-1).

In this diagram, the Target-R2 volume is also represented with 4 mirror positions and has local protection implemented. Three of the mirror positions are used. The first two mirror positions represent local mirrors and the third mirror is occupied by SRDF. If a BCV is established with the R2 volume, then it will occupy the next available mirror position. Under normal circumstances, the R1 volume presents a Read-Write (RW) status to the host which access it, and the R2 presents Write-Disabled (WD) to its host. Remote Link Director (RLD) * The symcfg list command will identify the storage array configurations * The symrdf list command will identify the RDF configured devices

A Remote Link Director is a hardware that provides communication and data paths between local and remote Symmetrix units. The Symmetrix can be configured with the following RLDs: Fibre Channel directors (RF) // ESCON directors (RA) Multiprotocol Channel Directors (MPCD) available with these channel connections: − FICON − iSCSI for host − GigE (RE) for SRDF SRDF Link Configuration SRDF offers three types of link configurations between source (local) and target (remote) Symmetrix systems: Uni-Directional, Bidirectional and Dual Configuration. SRDF Unidirectional Link Configuration If all primary (source or R1) volumes reside in one Symmetrix system and all secondary (target or R2) volumes reside in another Symmetrix system, write operations move in one direction, from primary to secondary. Data moves in the same direction over every link in the SRDF group. SRDF Bidirectional Link Configuration If an SRDF group contains both primary and secondary volumes, write operations move data in both directions over the SRDF links for that group. SRDF Dual-Directional Link Configuration With a dual-directional configuration, multiple SRDF groups are used; some groups send data in one direction, while other groups send data in the opposite direction. SRDF Modes of Operations Primary Modes of operation Primary Modes of operation • SRDF/S (Synchronous) • Secondary SRDF Mode • Adaptive Copy -Write Pending

- Disk Mode • SRDF/A (Asynchronous) • Domino Mode

• Operational Modes are set on Symmetrix Logical Volume level Using GUI or CLI and can be changed

Dynamically

1. c:\ symrdf –g <testdg> set mode sync 2. c:\ symrdf –g <testdg> set mode async 3. c:\ symrdf –g <testdg> set domino on/off Listed are the operational modes for SRDF operations: Synchronous mode, Adaptive Copy-Write Pending mode, Adaptive Copy-Disk Copy mode, and Asynchronous mode. These operational modes are selectable based on many requirements such as RPO, bandwidth, and performance. One of the two primary SRDF modes of operations is set at the source (R1) volume during Symmetrix configuration. All source (R1) volumes are configured for either the Synchronous or Semi-Synchronous mode. These two modes are considered to be pre-determined SRDF modes, which may be altered using SymCli. Adaptive copy is the secondary mode that facilitates data sharing and migration. Asynchronous mode continually collects and sends data to the remote Symmetrix. Asynchronous mode must be set for the entire RA group. Users can set SRDF to function in a secondary or Asynchronous mode. SRDF will revert to the pre-determined primary mode if it cannot maintain the criteria to remain in the secondary mode. Domino Mode could be classified as an SRDF attribute. Not necessarily a “Mode”. This attribute is set or used in conjunction with other SRDF modes except SRDF/A. It effectively stops all write operations to both source and target volumes if the target volume become unavailable, or if all SRDF links become unavailable. Synchronous Mode 1. Write I/O received from host/server at the source 2. I/O is transmitted to the target 3. An acknowledgment is provided by target back to the source 4. I/O is serviced to the host SRDF Synchronous Mode is used primarily in SRDF campus environments. In this mode of operation, Symmetrix maintains a real-time mirror image of the data of the remotely mirrored volumes. Data on the source (R1) volumes and target (R2) volumes are always fully synchronized at the completion of an I/O sequence. The sequence of operations is: * A write is received from the host/server at the source. * The write is transmitted to the target. * An acknowledgment is provided by the target back to the source. * The write is acknowledged to the Host. If step 3 never happens, the source SRDF services the I/O after a pre-determined timeout to keep the production machine running. Adaptive Copy Mode 1. An acknowledgment is provided by target back to the source 2. I/O accumulates in/on: - Symmetrix cache Write Pending Mode -R1 volume Disk Mode 3. Write I/O received from host/server at the source 4. I/O is serviced to the host 5. I/O is transmitted to the target SRDF Adaptive Copy Mode is used primarily for data migrations and data center moves. This operational mode is not recommended for use when mirroring for disaster recovery/restart purposes unless used with TimeFinder. This mode is very useful for initial synchronization, especially over long distances. (Used within a SRDF/Star configuration). SRDF Adaptive Copy Mode allows the source (R1) volumes and target (R2) volumes to be a out of synchronization by a number of I/O’s that users can define, a skew value. There are two types of adaptive copy: Write Pending Mode and Disk Mode. Adaptive Copy data movement is handled at the track level. The target data is only usable after a full synchronization. The sequence of operations is: * An I/O write is received from the host/server at the source. * I/O is accumulating. * I/O is serviced. * The I/O is transmitted to the target.

* An acknowledgment is provided by the target back to the source. In Write Pending Mode, the unit of transfer across the SRDF link is the updated blocks rather than an entire track, resulting in more efficient use of SRDF link bandwidth. Data is read from global memory than from disk, thus improving overall system performance. However, the global memory is temporarily consumed by the data until it is transferred across the link. In Disk Mode, while less global memory is consumed it is typically slower to read data from disk than from global memory, additionally, more bandwidth is used because the unit of transfer is the entire track. Additionally, because it is slower to read data from disk than global memory, device resynchronization time increases. Adaptive copy disk mode should not be used if the primary volumes are not RAID protected. Asynchronous Mode SRDF/A provides a long-distance replication solution with minimal impact on performance. This protection level is intended for customers requiring minimal host application impact, who need to maintain a restartable copy of data at the target site at all time. SRDF/A continually process Write I/O’s in batches. The interval between batches is referred to as a cycle. The sequence of operations is: 1. An I/O write is received from the host/server into the cache of the source. 2. I/O is accumulating. 3. I/O is serviced. 4. The I/O is transmitted to the target. Domino Mode (attribute) with SRDF/Synchronous

1. Write I/O received from host/server at the source 2. I/O fails to transmit to the target 3. Both Source and Target become unavailable

Domino Mode is used in conjunction with other SRDF modes except SRDF/A. It effectively stops all write operations to both source and target volumes if target volume become unavailable, or if all SRDF links become unavailable. User will need to manually re-enable the source volumes. While such a shutdown temporarily halts production processing, domino modes can prevent data integrity exposure that causes the inconsistent image on the target volume. SRDF Level of Synchronization * Synchronous Mode - Source = Target * Adaptive Copy - Source ≠ Target Source may be up to 65535 tracks per volume ahead of Target and Skew value set per logical volume * Asynchronous - SRDF/A - Source is minutes ahead of Target - SRDF/AR - Source is hours ahead of Target SRDF offers considerable flexibility for various levels of synchronization. To determine the level of synchronization, one must understand the required Recovery Point Objective. This is the amount of data that can be lost in the event of a site outage. There are other factors like distance, bandwidth, and response time latency that must be considered before determining a synchronization level. SRDF Devices

When configured for SRDF, the individual Symmetrix devices are designated as either a source mirror (R1 device), a target mirror (R2 device), or a cascaded mirror (R21). A cascaded mirrored device (R21) performs as both a source and target mirror. If the source device fails, the data on its corresponding target device can be accessed by the local host. Once the source device is replaced, it can be

resynchronized. SRDF configurations have at least one source (R1) device mirrored to one target (R2) device. For concurrent RDF systems, there can be two R2 targets. Cascaded RDF environments include R1 -> R21 -> R2 devices. Source (R1) devices can only belong to an RDF1 device group, target (R2) devices can only belong to an RDF2 device group, and R21 devices can only belong to an RDF21 device group. An RDF device group is a user-defined device group comprised of RDF devices from a single Symmetrix array. At the time of creation, a device group must be defined as type REGULAR, RDF1, RDF2, or RDF21 and may contain various device lists for standard, BCV, virtual (VDEV), and remote devices. If the group type is defined as RDF1, RDF2, or RDF21, the group is considered an RDF device group. By default, a device cannot belong to more than one device group. However, you can change this default behavior to allow a device to belong to multiple groups by enabling the SYMAPI_ALLOW_DEV_IN_MULT_GRPS parameter in the Symmetrix options file. RDF - RA Groups and Device Groups * RA Groups SRDF Groups are used to define relationships between DMX storage arrays - Static Device Groups SRDF groups that are explicitly configured in the DMX “bin” file are static SRDF groups - Dynamic Device Groups The ability to create SRDF Groups add dynamic devices and create their own SRDF environment without a “bin” file modification * SRDF Device Group A user defined object that is used to “view”and manage a number of DMX devices * SRDF Composite Group A DMX device group that contains devices that span multiple DMX arrays and multiple RA Groups * Consistency Group A Composite group that was created with the -rdf_consistency flag to insure dependent write consistency across the composite group EMC’s white paper titled “An Overview of Groups in Symmetrix and Solutions Enable Environments” (date Jan 08) provides extensive definitions for the above. RA Groups or SRDF Groups are used to define relationships between multiple DMX storage arrays RA groups fall into two categories, Static and Dynamic. Static Device Groups are configured in the DMX “bin” file. Where Dynamic Device Groups provides the ability for the storage administrator to create an RA Group without a “bin” file modification. An SRDF Device Group is a user defined object that enables the user to “view” and “manage” a number of DMX devices. A SRDF Composite Group is a DMX device group that contains devices that span multiple DMX arrays and multiple RA Groups. A Consistency Group is a Composite Group that was created with the -rdf_consistency flag to insure dependent write consistency across the composite group.

SRDF groups may be referred to as “RA Groups” or “RDF Groups” Both of these terms essentially mean the same thing. Performing a symdg show dev <Dev ID> will identify which RA group a device belongs to. A user can create a “Device Group” that contains some or all the devices in the RA group. If asynchronous replication will be used across the RDF link than all the devices in the RA group must be a member of the device group. More on this in the “SRDF/A Operation Module” (Module 4). A “Composite Group” follows the same rules as a Symmetrix device group except it contains devices that span multiple Symmetrix arrays and multiple RDF / RA group. A “Consistency Group” is a Composite Group that was created with the –rdf_consistency flag to enable RDF devices in separate RDF groups to maintain dependent write consistency. Write consistency is maintained by using either EMC PowerPath or Enginuity Consistency Assist (ECA) for SRDF/S or MSC (Multi Session Consistency) for SRDF/A. A complete overview of the above is presented in an EMC White Paper, titled “An Overview of Groups in Symmetrix and Solutions Enabler Environments”, date Jan 2008. Dynamic SRDF * Enables user to dynamically define relationships between R1 and R2 volumes * Provides flexibility for user to tailor SRDF configuration to their changing application requirements

Prior to Dynamic SRDF, the R1 and R2 pairings were static and defined in the configuration file (BIN File) on the Symmetrix. Any changes to SRDF device pairing required a new BIN file to be defined and loaded into the Source and Target Symmetrix. Dynamic SRDF available with 5x68 Enginuity code will provide the capability to change device pairings on the fly without requiring a BIN file configuration change to be performed by an EMC Customer Engineers. SRDF Dynamic Group Configuration C:\symrdf addgrp -label dyngrp2 -rdfg 2 -sid 80 -dir 12a -remote_rdfg 2 -remote_sid 40 -remote_dir 13a

An SRDF group, also known as RDF group or RA group, logically defines relationships between Symmetrix systems. An SRDF group is a set of SRDF director port connections configured to communicate with a another set of SRDF director ports in another Symmetrix system. Logical volumes (devices) are assigned to SRDF groups. Many SRDF groups can share a physical link between the Remote Link Directors. There are two ways to create an RDF group - static and dynamic. Both share the same features and functionality, the difference between the two types is how they are created. Static RDF groups are created during the Symmetrix configuration, and almost always by EMC personnel. Dynamic RDF groups are created and deleted by users through a set of Symmetrix command line interface (SYMCLI) commands.

Solutions Enabler has added support for 128 RDF groups and 32 RDF groups per director for arrays running Enginuity 5772. RDF Composite Groups * Similar to Device Groups – 3 Types: Regular, RDF1 and RDF2 – Can be used for all Control operations (e.g. TimeFinder, SRDF, etc.) – Can participate in consistency operations such as TimeFinder/Mirror consistent splits and TimeFinder/Snap activate – Composite Groups insures consistency of an application across multiple DMX environments * Different from Device Groups – Can span multiple Symmetrix arrays – Managed by RDF daemon if created with –rdf_consistency option and consistency is enabled – Required for SRDF/A Multi-session Consistency (MSC) Composite groups are similar to device groups. They can be type R1, R2 or Regular, and used for every action that are available with device groups. Composite groups can span Symmetrix arrays. Thus, a host running a database application spanning two Symmetrix arrays can use a composite group to perform a consistent TimeFinder split of the application data. If the type of the Composite Group is RDF1 or RDF2, the CG (Composite Group) can span RDF groups. Composite groups are a requirement for building SRDF consistency in SRDF/S and SRDF/A. When a “Composite Group” is created for SRDF consistency the “–rdf_consistency” option must be specified at the group creation time. Consistency: The Dependent Write I/O Principle * Logical dependency between write I/Os – Embedded in the logic of an application, operating system, or DBMS * A write I/O is not be issued by an application until a prior related write I/O is completed – A logical dependency, not a time dependency – Inherent in all Database Management Systems (DBMS) - Page (data) write is dependent write I/O based on a successful log write – Power failures create a dependent write consistent image – Restart transforms dependent write consistent to transactionally consistent * Ensuring ‘dependent write consistency’ is the basic principle behind all of EMC’s Consistency Technology solutions – SRDF Consistency Groups – TimeFinder Consistent Split – Consistent SNAPs and Consistent Clones – SRDF/A – Open Replicator for Symmetrix Almost all commercial applications, such as databases, are inherently consistent by design. EMC’s consistency technology makes it possible for the consistency to be maintained when replicas of the production data are made. All logging database management systems use the consistency principles described on this slide to maintain integrity. This is required for the protection against local power outages, loss of local channel connectivity, or storage devices. There is a logical dependency between I/Os built into database management systems, certain applications, middleware tools such as MQ Series, and operating systems. EMC can create a dependent write consistent local image with its TimeFinder family of products, whereas SRDF Consistency Groups and SRDF/A create a consistent image on one or more remote Symmetrix arrays. SRDF Serialization * Writes to Target volumes must happen in the same order as they are written to the Source in order to have an instance in time, consistent and recoverable copy * In Synchronous, and Asynchronous modes, writes are sent to the remote Symmetrix in the order received – If the remote Symmetrix is not accessible, writes are accumulated as invalid tracks – When the remote Symmetrix becomes available, invalid tracks are sent without regard to serialization

* Serialization is not maintained in Adaptive Copy mode – Typically used for data migrations Serialization maintains the order in which writes are received at the remote (target) Symmetrix. SRDF serialization must be maintained in order to have a recoverable/restartable copy of data at a target site. Through serialization, write fidelity is guaranteed. In normal operations, SRDF maintains order writes with Synchronous, Semi-synchronous, and Asynchronous modes. But when the link becomes unavailable for any reason, writes accumulate as invalid tracks which the application continues to function on the host. When the link is restored, the Adaptive Copy mode is used to propagate changes across the link. This introduces risk, since serialization is not maintained with Adaptive Copy. RDF-ECA Consistency Protection for SRDF/S * ECA is a feature that works inside the Symmetrix array * Used with SRDF/S to hold write I/Os to a “consistency group” until all relevant links are suspended * Interacts with RDF daemons on one or more control hosts to manage consistency * Stalls write I/Os to a user defined list of Symmetrix devices prior to splitting a source volume and its replica * Supports Concurrent SRDF RDF Enginuity Consistency Assist (RDF-ECA) provides consistency protection for synchronous mode devices by performing suspend operations across all SRDF/S devices in a consistency group or a named subset of all devices in a composite group. SRDF/S with RDF-ECA is supported by an RDF daemon that performs monitoring and cache recovery operations across all SRDF/S sessions in the group. If one or more source (R1) devices in an SRDF/S consistency group cannot propagate data to their corresponding target (R2) devices, the RDF daemon suspends data propagation from all R1 devices in the consistency group, halting all data flow to the R2 targets. This ensures that a consistent R2 data copy of the database exists at the point-in-time any interruption occurs. The RDF daemon monitors data copy operations and coordinates the suspension of R1 to R2 data propagation if the consistency protection is suspended (tripped). A composite group must be created using the RDF consistency protection option (-rdf_consistency) and must be enabled using the symcg enable command before the RDF daemon begins monitoring and managing the RDF-ECA consistency group. Concurrent SRDF * One R1 can be paired with two R2 devices, concurrently * Each of the two concurrent mirrors must belong to different RA groups Concurrent SRDF allows two remote SRDF mirrors of a single R1 device, e.g. use one remote copy for disaster recovery, and another for decision support or backup. Each Remote Link Director is assigned to an RA Group. With ESCON, only one RA group per RLD is allowed, but Fibre Channel SRDF RA Groups can be defined to the same RLD. Any mixture of SRDF modes is allowed, except for Sync and Semi-sync configuration and Async and Async configuration. A write IO from the host at the primary device side cannot be returned as completed until both remote Symmetrix’ signal the local Symmetrix that the SRDF IO is in cache at the remote side. 1 Sync and 1 Adaptive Copy remote mirror: The SRDF IO from the secondary device operating in Synchronous mode must present ending status to the sending Symmetrix before a second host IO can be accepted. The host I/O does not wait for the secondary device operating in Adaptive Copy mode. * One R1 can be paired with two R2 devices, one in each Symmetrix, concurrently * All combinations of Primary/Secondary modes for the R1- R2 pairs are allowed - except one pair in Sync and the other in semi-sync, both cannot be “Async” * Cannot restore from both R2 mirrors to the R1 Simultaneously * SRDF swap is not allowed. For example if the R1 is

changed to an R2 one will be left with R2->R1, R2- >R2@#! * Remote BCVs can be associated with only one of the R2 Mirrors A BCV can only be established with one of the Target volumes, not both. In case the source is locally protected, the BCV device cannot be established with it’s source, because all four(4) mirror positions will be occupied 2 Synchronous remote mirrors : * A write IO from the host at the primary device side cannot be returned as completed until both remote Symmetrix’ signal the local Symmetrix that the SRDF IO is in cache at the remote side. 1 Sync and 1 Adaptive Copy remote mirror: * The SRDF / IO from the secondary device operating in Synchronous mode must present ending status to the sending Symmetrix before a second host IO can be accepted. The host I/O does not wait for the secondary device operating in Adaptive Copy mode. RDF - R1/R2 Personality “Swap”

An R1/R2 personality swap (or R1/R2 swap) refers to when the RDF personality of the RDF device designations of a specified device group are swapped so that source R1 device(s) become target R2 device(s) and target R2 device(s) become source R1 device(s). Dynamic RDF swaps are available with Enginuity™version 5567 or later. To perform an R1/R2 swap, you must have an SRDF license with Symmetrix 5567 microcode or higher and Dynamic RDF must be enabled in your Symmetrix configuration.

Sample scenarios for R1/R2 Swap: - Symmetrix Load Balancing In today’s rapidly changing computing environments, it is often necessary to deploy applications and storage on a different Symmetrix without having to give up disaster protection. R1/R2 swap can enable this redeployment with minimal disruption, while offering the benefit of load balancing across two Symmetrix storage arrays. - Primary Data Center Relocation Sometimes a primary data center needs to be relocated to accommodate business practices. For example, several financial institutions in New York City routinely relocate their primary data center across the Hudson River to New Jersey as part of their disaster drills. R1/R2 swaps allow these customers to run their primary applications in their New Jersey data centers. The Manhattan data centers now act as the disaster protection site. - Post-Failover Temporary Protection Measure If the hosts on the source side are down for maintenance, R1/R2 swap permits the relocation of production computing to the target site without giving up the security of remote data protection. When all problems have been solved on the local Symmetrix, you have to failover again and swap the personality of the devices to go back to the original configuration. Identify Accessible SRDF Volumes

c:\symrdf list –rdfg 1 * symrdf list –rdfg <#> will display summary information about RDF volumes for a specific RA group, accessible to the host. The symrdf list command lists the SRDF devices that are visible to your host, or SRDF devices that are configured on a given Symmetrix array. For example, to list the SRDF devices that are visible to your host, enter: symrdf list DEV The results provide details about the SRDF devices, source (R1) and target (R2), and Cascaded RDF devices (R21), including: − Symmetrix device name − Remote Symmetrix device name − RDF type and RA number − Status of the SA, RA, and SRDF links − SRDF mode − Domino mode − Adaptive copy mode − Number of invalid tracks on R1 and R2 − SRDF link state c:\symdev list –r1 c:\symdev list –r1 à Will list all R1 devices available to the source host c:\symdev list –r2 c:\symdev list –r2 à Will list all the R2 devices available to the target host c:\symdev list –r21 c:\symdev list –r21 à Will identify all the R21 devices (Executed from the target host)

To display devices that have been configured as dynamic RDF_capable, use the symdev list command with the -dynamic option as follows: symdev list -dynamic [-R1] [-R2] [-R21] From the displayed list, determine which dynamic devices on the source Symmetrix array you would like to pair with dynamic devices on the target Symmetrix array. If -R1 or -R2 is not specified, all devices that are RDF-capable will be displayed. The above example will display R21 configured devices. Notice that this symdev list command was executed from the target host. Using a combination of “symrdf “and “symdev” list commands, storage administrators can quickly determine device configuration and device availability for both source and target hosts. It’s always a “Best Practice” to create a working diagram of all the devices that are available. Configuring SYMCLI SRDF Device Groups Devices can be “Grouped” into “Device Groups” * All devices in a device group must be in the same DMX array * All devices must be the same type (RDF1, RDF2, or Regular) A device group is a logical grouping of Symmetrix volumes. There are three types of device groups:

regular, rdf1, and rdf2. A device group, with the type “regular”, cannot contain RDF volumes. Therefore, users must create a device group with rdf1 or rdf2 for SRDF operations. The device group definition is stored in the SYMAPI database on the host where the symdg create command was executed. Displaying SYMCLI Device Groups: Part 1 * symdg show will display detailed information about a device group c:\symdg show testdg4 c:\symdg show testdg4

The symdg show command displays detailed group information for any specific device group. In the above device group (testdg4), the group contains 2 local standard devices (003F and 0040). The screen shot above also displays the source and target Symmetrix array ID’s. Also notice the device group type, which is type “RDF1” . The SRDF SYMCLI Command Syntax c:\symrdf -g <device group> <action> [options] Where action can be one of the following:

Users can perform a number of Symmetrix SRDF operations using host-based SYMCLI commands. Major SRDF operations or actions include: suspend, resume, failover, failback, update, split, establish, and restore. Refer EMC Solutions Enabler Symmetrix SRDF Family CLI Version 6.5 Product Guide P/N 300-000-877 - Rev A10 for a complete list of all capability supporting the symrdf command. Verifying the SRDF Link Status C:\symrdf ping C:\symcfg –RA all List à How to identify the current RA groups. The symrdf ping command checks if the RDF link communication is operational. The symcfg command

lists configuration information of all the remote directors. Notice in the above example the remote director on both the source and target arrays is “16D” Also notice that the above configuration identifies four RA groups. To list device volumes assigned to each RA group, perform the following command: c:\symrdf list –rdfg <Group Number> Suspending the SRDF Link • Logically suspends mirror relationship between source and target volumes c:\symrdf –g testdg4 suspend –nop c:\symrdf –g testdg4 suspend –nop • The symrdf suspend command stops data transfer between specified pairs. • New writes to source volume accumulate as invalid tracks. • The R1 devices remain “Read Write”, and the link becomes “Not Ready”. • The R2 devices become “Write Disabled”, and the RDF Pair is placed into a “Suspended” state. Resuming the SRDF Link • Logically resumes mirror relationship between source and the target volumes c:\symrdf –g testdg4 resume –nop c:\symrdf –g testdg4 resume –nop

The symrdf resume command resumes data transfer between the pairs. The Invalid tracks will start to synchronize. However, remember that the serialization is not maintained during the synchronization. Performing a query on the device group “testdg4” C:\ symrdf –g testdg4 query When invoking the resume command, the R1 devices remain “Read Write” and now the link becomes “Read Write” enabled. The R2 devices remain “Write Disabled ” and eventually the RDF Pair will become “Synchronized”. Changing SRDF Operational Modes

c:\symrdf –g testdg4 set mode acp_disk –nop The “symrdf set mode” command will change the SRDF operation mode. The –noprompt or –nop flag is used to bypass a confirmation question from the command line. This switch is applicable to most operations with the symrdf command. The –nop flag is primarily used for scripting to bypass user verification for any command execution. For the above example, the link is placed in a “Not Ready” state because of the split condition. The mode is now set to “Adaptive Copy / Disk Mode” also known as SRDF/DM. Adaptive Copy is primarily used for “Data Migration”. SRDF Volume Operational Status - “query” c:\set SYMCLI_DG=testdg4 c:\set SYMCLI_DG=testdg4 c:\symrdf query c:\symrdf query The name of the device group can be exported as a SYMCLI environment variable so that you would not have to type the device group name in each time. If a script needs to be generated, the application developer may want to set the SYMCLI_DG variable for a specific device group. Doing so will set this variable for this process only. Logging out and then back in again will require a reset of this variable. The above example “query” command shows the SRDF status of the device group “testdg4” because the SYMCL_DG variable was set to equal testdg4. SRDF Disaster Recovery Operations Failover - c:\symrdf –g testdg4 failover -nop • Will make a copy of the data on target array (R2-Volumes) available to the attached host. • Used for: Host Symmetrix or Site failure • Maintenance Operation: Will provide data availability during host, Symmetrix, or Site maintenance Update - c:\symrdf –g testdg4 update –until 1000 –nop • Begins synchronization back to the original production environment prior to resuming full operations on the source volumes Failback - c:\symrdf –g testdg4 failback -nop • Resumes operation back on the primary host, all changes made during failover will be updated back onto the primary source volumes The disaster recovery operations for SRDF devices are: * Failover from the source side to the target side, switching data processing to the target side * Failback from the target side to the source side by switching data processing to the source side * Update the source side after a failover while the target side may still be operational to its local host The “Update” function is used primarily to reduce the number of changed tracks owed to the original source volumes. The storage administrator would use this function prior to failing back to reduce the failback time span. SRDF - Failover, Update, Failback Summary

Production switches back to the R1 side and any additional invalids are copied to the R1 array

Summary / Review for Failover, Update, and Failback: Failover - Once a failover has been invoked, the R1 devices become “Write Disabled” and the link becomes “Not Ready” because of the “Failed Over” state. The target devices become “Read Write” and the RDF Pair is now in a “Failed Over” state. Update - When performing and update, the R1 devices are still “Write Disabled” the link becomes “Read Write” enabled because of the “Updated” state. The target devices (R2) will remain “Read Write” during the update process. Failback - When performing a failback, the R1 devices will become “Read Write” and the link will become “Read Write” as well. The target devices become “Write Disabled” and the RDF Pair will eventually become synchronized. SRDF Failover - c:\symrdf –g testdg4 failover –nop c:\symrdf –g testdg4 failover –nop In a period of scheduled downtime for maintenance, or after a serious system problem which has rendered either the host or Symmetrix unit containing the source (R1) devices unreachable, no read/write operations can occur on the source (R1) device. In this situation, the failover operation should be initiated to make the target (R2) devices read/write enabled to their local host(s). In the above example once a failover has been invoked, the R1 devices become “Write Disabled” and the link becomes “Not Ready” because of the “Failed Over” state. The target devices become “Read Write” and the RDF Pair is now in a “Failed Over” state. Updating Source Volumes - c:\symrdf –g testdg4 update –until 1000 While the target (R2) device is still operational (Write Enabled to its local host(s)), an incremental data copy from the target (R2) device to the source (R1) device can be initiated in order to update the R1 mirror with changed tracks from the target (R2) device. When performing and update, the R1 devices are still “Write Disabled”; the link becomes “Read Write” enabled because of the “Updated” state. The target devices (R2) remain “Read Write” during the update process. Note the –until flag, which represents a skew value assigned to the update process. SRDF Failback - c:\symrdf –g testdg4 failback –nop A failback, or source (R1) device takeover, is performed when you are ready to resume normal SRDF operations by initiating read/write operations on the source (R1) devices, and stopping read/write operations on the target (R2) devices. The target (R2) devices become write-disabled (WD) to their local host(s) while the source (R1) devices are read/write enabled to their local host(s). The Failback may be executed on either the source or target side. When performing a failback, the R1 devices become “Read Write” and the link becomes “Read Write” as well. The target devices will become “Write Disabled” and the RDF Pair will eventually become synchronized. Note - Host activity should be stopped prior to execution. This means stop any applications and unmount any and all file systems that would be impacted. SRDF Decision Support/Concurrent Operations 1) Establish - c:\symrdf –g testdg4 establish –full -nop • Save source data • Resume Normal SRDF operations • Preserves data on the source (R1) volumes, discarding changes to the target (R2) volumes.

• Establish an SRDF pair by initiating a data copy from the source side to target side. The operation can be a full or incremental. • Establish - When performing an establish operation, the R1 devices and the link become “Read Write” enabled. The target devices (R2) will become “Write Disabled” and the RDF pair will become fully synchronized. 2) Split - c:\symrdf –g testdg4 split -nop • Places the Symmetrix units in a state for concurrent access • Suspends link between source (R1) and target (R2) volumes • Enables read and write operations on both source and target volumes. • Split an SRDF pair which stops mirroring for the SRDF pairs in a device group. • Split - When performing a split the R1 and R2 devices will become “Read Write ” enabled and the link will become “Not Ready”. 3) Restore - c:\symrdf –g testdg4 restore -nop • Resumes SRDF operations • Preserves data on the target (R2) volumes, discarding changes to the source (R1) volumes. • Restore remote mirroring, which initiates a data copy from the target side to the source side. The operation can be full or incremental. • Restore - When in a restore state, the source volumes will become “Read Write” enabled and the link will become “Read Write” as well. The target devices (R2) will become “Write Disabled” and the state of the RDF Pair will become “Restore in Progress”, eventually becoming fully “Synchronized”.

Invalid tracks move across the RDF link from the R2 to the R1 pairs – “Restore in Progress to Synchronized Concurrent Operations: Establish An establish operation will resume SRDF operations, retaining data from source and overwriting data on target volumes. When performing an establish operation, the R1 devices and the link will become “Read Write” enabled. The target devices (R2) will become “Write Disabled” and the RDF pair will become fully synchronized. Concurrent Operations: Split

The split command suspends the link between source (R1) and Target (R2) volumes. It also enables read and write operations on both source and target volumes. Changes to the source volumes are kept as R2 invalids. Changes to target are kept track of as R1 invalids. When performing a split, the R1 and R2 devices will become “Read Write ” enabled and the link will become “Not Ready”. Concurrent Operations: Restore The restore operation resumes SRDF operation retaining data on target and overwriting data on the source volumes. When in a restore state, the source volumes become “Read Write” enabled and the link will become “Read Write” as well. The target devices (R2) will become “Write Disabled” and the state of the RDF Pair will become “Restore in Progress”, eventually becoming fully “Synchronized”. SRDF Device Group Continuous Monitoring

c:\symrdf –g testdg4 query –i 5 –c 5 -i Represents the “interval” time the command will run (every 5 seconds) -c Represent the “count” or the number of time the command will run Performing a query on a device group will show the status of RDF Volume. The –i or interval represents the time the command will run. In the above example, the command will run every five seconds. The –c or count represents the number of times the command will run. In the above example, the command will run five times.

Creating a Dynamic RDF Group * To create a Dynamic RDF group – first check to make sure both arrays have the “Dynamic RDF Configuration State” setting enabled. c:\symcfg –sid 0997 list –v Since Enginuity 5568, devices can be configured to be Dynamic RDF-capable devices. Dynamic RDF functionality enables you to create, delete, and swap SRDF pairs while the Symmetrix array is in operation. Using Dynamic RDF technology, one can establish SRDF device pairs from non-configured SRDF devices, then synchronize and manage them in the same way as configured SRDF pairs.

Running the “symcfg list” command to check for Dynamic RDF configuration. In order to create dynamic RA groups, the Symmetrix on both the source and target arrays need to have “Dynamic RDF Configuration State” enabled. Also, a number of device volumes need to be configured as dynamic as well. c:\symcfg list –RA all –sid 0997 1. To determine the RDF director and 2. The number of groups currently configured. 3. To determine RDF dynamic configured devices – Perform this on both Local and Remote hosts c:\symdev list –dynamic

Before creating a new RA group, some information needs to be gathered. First we need to know RDF director ID. This is the RA director link between the two Symmetrix arrays. We also need to know the current RA groups and their group ID numbers. In the above example, we can see RA groups 1 though 4 being identified. We also need to identify the dynamic volumes that we will eventually place into the new RA group. Performing a “symcfg” list and a “symdev” list will give us the required information to create a new RA group. c:\symrdf –v addgrp –label grp5 –rdfg 5 –sid 0997 –dir 16D –remote_rdfg 5 -remote_sid 0998 –remote_dir 16D –nop

c:\symrdf –v removegrp –label grp5 –nop The above command will remove RDF group 5. Note – The RDF group must contain no dynamic devices (The group must be empty) The symrdf addgrp command creates an empty dynamic RDF group that represents another RDF link between Symmetrix 0997 and Symmetrix 0998. It adds dynamic RDF group 5 on the local Symmetrix, and RDF group 5 on the remote Symmetrix. You must specify a group label (grp5 in this case) that can be used when modifying or deleting the group. Creation of the dynamic RDF group includes director 16D from the local Symmetrix and 16D from the remote Symmetrix as the director end points for this connection. The symrdf removegrp command deletes the dynamic group. The group must be empty to be deleted. It is important to be aware of your network topology when creating dynamic RDF groups between two Symmetrix arrays. To create a dynamic RDF link (a connection) between RA directors, the director end points must be able to see each other through the Fibre Channel fabric. For example, a dynamic RDF link can be created between local and remote directors only if the Fibre Channel zoning is set up so that the two directors can see each other through the fabric. Pairing Dynamic Devices c:\symrdf createpair –file grp5.txt –sid 0997 –type rdf1 –rdfg 5 –establish

* To determine the source and target dynamic devices for pairing, perform a c:\symdev list –dynamic on both the source and target * Create a device file and list source and target pairs. y Use the symrdf createpair command against this file. * Use the symrdf createpair command against this file.

Using Dynamic

RDF technology, you can establish SRDF device pairs from non - SRDF devices using the symrdf createpair command. Once established, the new SRDF pairs can be synchronized and managed in the same way as configured SRDF pairs. Prior to Enginuity version 5568, SRDF device pairing was limited to the static SRDF pairs set at Symmetrix configuration time. Dynamic RDF enables the creation and deletion of SRDF pairs while the Symmetrix array is in operation. The create SRDF pairs command creates SRDF pairs from devices listed in a device pairs file. For example, to create SRDF pairs from a device pairs file called grp5.txt, enter: symrdf createpair -sid 0997 -file grp5.txt -type rdf1 -rdfg 5 -establish Delete SRDF pairs The delete SRDF pairs command cancels SRDF pairs in the device file specified. For example, to delete the SRDF pairs in a RDF group 10, enter: symrdf deletepair -sid 0997 -file grp5.txt -rdfg 5 The deletepair command can also be executed on device groups (-g) instead of specifying the device text file. Dynamic SRDF Options * Establish option: - Invalidates R2s, merges track tables, brings up RDF links, starts copy process from R1 to R2 c:\symrdf createpair –file grp5.txt –sid 0997 –type rdf1 –rdfg 5 –establish * Restore option: - Invalidates R1s, merges track tables, brings up RDF links, starts copy process from R2 to R1 c:\symrdf createpair –file grp5.txt –sid 0997 –type rdf1 –rdfg 5 –restore * Invalidate option: - Allows creation of dynamic SRDF pairs, but does not bring up the RDF links and initiate data copy - To perform an establish, use -invalidate r2 - To perform a restore, use -invalidate r1 c:\symrdf createpair –file grp5.txt –sid 0997 –type rdf1 –rdfg 5 –invalidate r2 Creating dynamic SRDF pairs with establish - Optionally, you can include the establish operation in the createpair command line by replacing the -invalidate r2 option described earlier with the –establish option, where the default copy path is R1 to R2 for all the device pairs in the list as follows: Example Symrdf createpair -file <grp5.txt> -sid <97> -type <RDF1> -rdfg <5> -establish Creating dynamic SRDF pairs for a restore - One can perform a restore operation to copy data back to the R1 source devices by including the -restore option in the createpair command line as follows: Example Symrdf createpair -file <grp5.txt> -sid <97> -type <RDF1> -rdfg <5> -restore Creating dynamic SRDF pairs and not bring up the RDF link. Invalidate, allows creation of dynamic SRDF pairs, but does not bring up the RDF links and initiate data copy Example Symrdf createpair -file <grp5.txt> -sid <97> -type <RDF1> -rdfg <5> -invalidate Deleting Device Pairings * Removes the pairing information from the Symmetrix y Must suspend RDF links using before issuing symrdf deletepair command (link state must be NR and pair state is Suspended, Split, or FailedOver):

c:\symrdf suspend –file grp5.txt –sid 0997 –rdfg 5 ///// c:\symrdf suspend -sid 0997 -file grp5.txt -rdfg 5 c:\symrdf deletepair –file grp5.txt –sid 0997 –rdfg 5 ///// c:\symrdf deletepair -sid 0997 -file grp5.txt -rdfg 5 * Dynamic SRDF pairs can also be cancelled within the context of a device group c:\symrdf deletepair –g <device group Name> * Canceling dynamic SRDF pairings changes the type of the device group from RDFx to Regular * Devices in the device group are changed from DRx devices to RDF-capable standard devices and the SYMAPI database is updated. The delete SRDF pairs command cancels SRDF pairs in the device file specified. For example, to delete the SRDF pairs in an RDF group 5, enter: Before the delete pair can be invoked the pair must be suspend first. What is SRDF/A (Asynchronous) * Asynchronous remote mirroring • Minimal impact to production applications • Extended distance • Always consistent image on R2 * Efficient bandwidth usage * Supports Mainframe and Open Systems * Complements existing SRDF solutions • Meet a wide range of RPO and RTO service level requirements * Mixed SRDF and SRDF/A • Share links and directors SRDF/A’s unique architecture delivers a remote mirroring solution that has no impact on production applications over extended distance because I/O requests from the host are acknowledged locally. Changes made to the same data blocks are periodically sent only once to the remote Symmetrix. This enables significant operational savings through reduced bandwidth requirements. Moreover, SRDF/A provides an alternative Disaster Recovery solution, in addition to SRDF/S, by maintaining a consistent image of a RDBMS (Relational DataBase Management System) on the R2 volumes at all times. SRDF/A is a single solution supporting both Mainframe and Open Systems. It also compliments SRDF solutions to meet mixed service level requirements. In fact, it can also share the same communication links as SRDF. SRDF/A Supported Environments * All existing SRDF topologies – ESCON with Farpoint – Fibre Channel – IP * Any host – Mainframe – Open System * All emulation types supported by Symmetrix, including: – FBA (Fixed Block Architecture) – CKD (Count Key Data) * Key operating systems, including: – UNIX (Sun, HP, IBM) – Windows (NT, 2000, 2003) – IBM Mainframe (OS/390, z/OS) Note: Support begins with Symm 5670 microcode and carries forward to future generations of Symmetrix. An SRDF/A Asynchronous environment: * Will maintain a dependent write consistent copy on the R2 devices at all times * Supports all current SRDF topologies, including point-to-point and switched fabric * Requires no additional hardware, such as switches or routers * Has the ability to operate at any given distance without adding response time to the R1 host * Supports all hosts and data emulation types supported by the Symmetrix array (such as FBA, CKD, AS/400) * Minimizes the impact imposed on the back-end DA directors * Provides a performance response time equivalent to writing to local non-SRDF devices * Allows restore, failover, and failback capability between the R1 and the R2 sites * Dynamic RDF pair operations including the createpair, deletepair and swap commands are supported for SRDF/A-capable devices. Functionality requires Enginuity Version 5671. Refer to “Dynamic SRDF pair operations” on page 97 for additional information on dynamic pairing

* Concurrent RDF devices that are SRDF/A-capable are supported. Functionality requires Enginuity Version 5671. Only one RDF group in a concurrent configuration can be run in asynchronous mode at one time, the other RDF group must be in a mode other than asynchronous. Industry’s Traditional Write Ordering 1. Host I/O Writes to cache 2. Every write must be time stamped, ordered and sent to the target side 3. Writes must be re-ordered before de-staging to disk 4. Writes de-staged to disk Traditional approaches to asynchronous mirroring have their architectural shortfalls. * Uses a combination of cache and files to perform mirroring * Timestamps or sequence numbers are applied to each and every incoming write * Each write must have a timestamp applied to it before sending it to the remote side That means every single write MUST be sent to the remote side, and because they do not necessarily arrive in order, they must be re-sequenced before being applied to disk. If you have writes number 100-200 pending at the remote side, all waiting for write number 99 to arrive, the system must re-sequence and wait for number 99 to arrive before committing writes 100-200. This creates a significant amount of overhead and data management activity in both the source and target systems as they scramble to time-stamp, send, re-order, wait for a dependant time-stamp, and then eventually commit the writes to disk at the target side. SRDF/A Architecture / Delta Sets

SRDF/Asynchronous mode Dependent write consistency is achieved through the processing of the ordered SRDF/A delta sets between the source (R1) and the target (R2). Dependent write consistency ensures that all writes to the R2 are processed in sequential numbered sets to maintain a consistent copy of data between the R1 and the R2. When the first SRDF/A cycle (N) is active, it collects any new writes on the R1, overwriting any duplicate tracks intended for data transfer over the link. The cycle is active for a predetermined amount of time, which can be configured on the Symmetrix array. The default time is 30 seconds. After the set time has been reached, the delta set data moves into the next cycle position (N-1) and begins transferring the delta set over the link to the R2. A new cycle N then begins collecting new writes again for the next delta set transfer. In cycle N-1, the delta set is temporarily collected on the R2 side for destaging. When the N-1 cycle has finished transferring data into the R2 and the minimum cycle time has elapsed, the delta set data moves into the next cycle position (N-2) and begins destaging the data to the R2 storage devices. The delta set data is considered committed to the R2 in cycle N-2 as it is applied to disk. One delta set is dependent upon the other for achieving write consistency. No cycle can begin until the prior one has completed. All data is transferred at the block level. Dependent Write Consistency * Dependent write logic: – If ‘A’ is a predecessor and ‘B’ is a dependent write

– Any I/O ‘B’ that arrives after I/O ‘A’ has been acknowledged to the host, must be dependent on ‘A’ – Implemented by the application, such as RDBMS’s * SRDF/A ensures that: – ‘A’ and ‘B’ are in the same Delta Set or – ‘B’ is in later Delta Set All commonly used database management systems are inherently dependent write consistent. For instance, a DBMS (DataBase Management System) will not perform a log write, indicating that a transaction is complete, until it has received an acknowledgement from the storage subsystem that the log data pertaining to the transaction itself was completely written to disk. Symmetrix honors this logic in SRDF/A by treating any successor I/O, which arrives after a predecessor I/O as a dependent I/O. Verifying the SRDF/A Environment * List the configuration of the Symmetrix to verify that relevant System Attributes have been set C:\symcfg list –v -sid 0997 * List the Remote Adapters configured on the Symmetrix and verify their status C:\symcfg list –ra all * List the RDF Groups that have been created on the Symmetrix C:\symcfg list –rdfg all C:\symcfg list –rdfg 1 As noted earlier, SRDF/A is supported under all topologies. However, if switched RDF, Concurrent, RDF Dynamic RDF and Concurrent Dynamic RDF Configurations are enabled, then the end user can create Dynamic RDF Groups, using dynamic SRDF devices, and perform Concurrent RDF operations. Perform the symcfg list –ra command to identify the Remote Adapters the SRDF/A device group will be using for “delta set” data transmition. Performing the same command using the –rdfg flag will identify all the RDF groups, both static and dynamic. For additional review of the above commands refer back to module “Symmetrix Business Continuity: SRDF Operations. Transitioning to SRDF/A mode * From Synchronous – If the devices are in Synchronized state, then by definition the R2 devices have a consistent image. Enabling SRDF/A immediately provides consistent data on the R2 * From Adaptive Copy Disk – Any invalid tracks owed to the R2 are synchronized. Two cycle switches after Synchronization, SRDF/A provides consistent data on the R2 * From Adaptive Copy Write Pending – Write pending slots are merged into the Active SRDF/A cycles. When there are no more write pending slots, it takes an additional two cycle switches before R2 data is consistent SRDF/A can be enabled when the device pairs are operating in any of the listed modes. In the case of Adaptive Copy to SRDF/A transitions, it takes two additional cycle switches after resynchronization of data for the R2 devices to be consistent. SRDF/A Devices The SRDF/A-capable device option (-rdfa) allows you to list devices that are SRDF/A-capable. (Command was used prior to Enginuity 71) C:\symrdf –sid 0997 –rdfa –rdfg 5 list The command was used prior to Enginuity 71 to identify devices capable of operating in asynchronous mode. The above identifies all the volumes in RA group “5” that will be used in a device groups for asynchronous replication. To get to this point planning is required. Knowing what volumes to be used for asynchronous replication takes time and careful planning.

First identify those volumes that contain your application. Next create and RA group and add all identified volumes to the RA group. Create a device group and add all device pairs to the group. Notice that the current RDF mode for the device pairs is set to “Adaptive Copy Write Pending” mode at the moment. Review : - symcfg list to identify the RDF director - symrdf addgrp to create an new RA group - symrdf createpair to add all the device pairs to the new RA group (Need to create a device pair file ) - symdg create (type RDF1) device group - Symld add DEV to add all the devices to the device group to be used for asynchronous replication. Note: The key here is that all devices in the RA group must be managed together when applying asynchronous replication. Enabling SRDF/A C:\symrdf –g testdg6 set mode async 1. Note the Set Mode to “async” command – (SRDF/A) 2. Session Number and Cycle Number and RDFA session is “Active” 3. Mode is Asynchronous (A…) 4. All the device pairs within the Device Group are “Consistent” Transition to SRDF/A is immediate (A…) and the pair state is immediately Consistent. Note above the Set Mode to “async” command The Session Number being “4” and Cycle Number of “3” and the “RDFA” session being “Active”. The MDAC identifier indicates that the mode is Asynchronous (A…) and all the device pairs within the Device Group are “Consistent” at this time. Example: SRDF/A to ACP_Disk C:\symrdf –g testdg6 set mode acp_disk In this example, the device pairs are operating in SRDF Adaptive Copy Write Pending Mode (C.W). There are a number of invalid tracks owed to the R2 devices. This is governed by the skew value that was set earlier. Note the “symrdf disable” command above. During and active SRDF/A session enabling consistency between all device pairs within the device group is activated with the symrdf –g <device group name> enable command. During “sync” (SRDF/S) and “adaptive copy” modes, disabling consistency is performed with the symrdf –g <device group name> disable command. Consistent R2 Data C:\symrdf –g testdg6 query –rdfa In this example, all invalid tracks were synchronized within the second cycle. Consequently, by the third cycle, the RDF Pair STATE has become Consistent. Tracks not Committed to the R2 Side is a measure of data in Capture, Transmit, and Receive cycles at the time of the query.

Display Devices with SRDF/A Enabled C:\symdev show 0035 When devices are in SRDF/A enabled state, the display includes their RDFA information. Note that the above device is currently in an “Asynchronous” RDF mode. It’s also currently consistent with it’s RDF pair. Its consistency state is enabled, which is the result of the symrdf –g <device group name> enable command invoked earlier. Also notice the time interval the R2 device is behind it’s R1 pair. SRDF/A Consistent Deactivation from Async to Sync

The Consistent Deactivation process works in three phases. In the first phase, SRDF/A operates normally. When the request to transition from SRDF/A to synchronous is received, at the next cycle switch, which guarantees an empty active cycle at the R1 side, a transition to the second phase occurs. In the second phase, new writes at the R1 side are sent directly in synchronous mode to the R2 side, with one key exception. As these write arrive at the R2, they are kept in the inactive (receive) cycle at the R2 side. And, the inactive (transmit) cycle at the R1 side continues to send data to the inactive (receive) cycle at the R2. At the next cycle switch (two switches into the process), a transition to the third phase occurs. From phase 2, when the next cycle switch is received (two cycle switches into the process), the inactive cycle at the R2 side becomes the active cycle, and the SRDF/A restore process begins and ends. At the end of the restore, when all tracks are marked write pending to the R2 devices, the Consistent Deactivation is complete. SRDF/A Consistent Deactivation: Limitations * The transition is not immediate – It takes two cycle switches from request to completion * There is some performance loss during the transition – To host writes done during the transition – Similar to host writes done during re-sync copy * Cache requirements – During transition, all new writes are written to an existing inactive (receive) SRDF/A cycle at the R2 side – For consistency, the data must be committed (restored) to the R2 all at once – This may require an inactive (receive) cycle at the R2 which is ~2x the size as normal * Both Symmetrix systems must be running Enginuity 5x71 The limitations to this function are described above. Please note the additional cache which may be required at the R2 side. Because Enginuity changes are required at both R1 and R2 side Symmetrix systems, this is a 5x71 only feature. This function fails if attempted on a 5x71 to 5670 SRDF connected Symmetrix pair. SRDF/A: Loss of Link * Temporary Loss of Link:

– If all links are lost for a period less than 10 seconds, SRDF/A remains in an Active state. Data continues to accumulate in cache and consequently, the cycle time is elongated. However, R2 stays consistent, and the relationship between R1-R2 is not suspended * Permanent Loss of Link: – In this event, all data in the Capture and Transmit cycles on the R1 side are changed from write-pending for the remote mirror to invalid on the remote mirror. Any new data on the R1s is also be marked as invalid on the remote mirror. Likewise, all data in the Receive cycle on the R2 side are changed to invalid on the remote mirror. The Apply cycle on the R2 side completes the commit to the local devices. The devices themselves are set to Not Ready on the link Data on the R2 devices is always consistent in SRDF/A, even with the loss of link. However, during resynchronization, data is temporarily inconsistent on the R2 until all the invalid tracks have been sent over to R2. For this reason, it is preferable to have a point-in-time copy on a BCV, for example, prior to starting the resynchronization process. Resynchronization can be initiated by issuing symrdf establish command. The logical connection between R1 and R2 can be lost under several conditions. Some of them are listed below: * Network problems leading to loss of physical connection between source and target * Symmetrix dropping the links due to link saturation * User issued commands such as symrdf suspend/split SRDF/A Reserve Capacity – Transmit Idle * 5772 and later support “Transmit Idle” * After links fail - Data transmission from source to target stops - SRDF/A remains active and the capture cycle grows in cache - Session is suspended if cache fills up * After links are revived, cycle switching continues * Not supported with ESCON RAs * Should be enabled on both Source and Target side * Should be used if using Delta Set Extension SRDF/A Transmit Idle is a Reserve Capacity enhancement to EMC’s SRDF/A feature that provides SRDF/A with the capability of dynamically and transparently extending the Capture, Transmit, and Receive phases of the SRDF/A cycle while masking the effects of an “all SRDF links lost” event. Without the SRDF/A Transmit Idle enhancement, an “all SRDF links lost” event would normally result in the abnormal termination of SRDF/A with either a CACA.20 or CACA.40 error. The SRDF/A Transmit Idle enhancement has been specifically designed to prevent this event from occurring. The versions of 5x71 which support Transmit Idle are 5771 Minimum release level of 92.99 with Epack# 1016 and 5671 Minimum release level 59.64 with Epack# 1017. SRDF/A Reserve Capacity - Delta Set Extension * Allows offloading of SRDF/A delta sets from cache to specially configured device pools - Delta Set Extension Pools * Intended to make SRDF/A resilient to temporary increases in write workloads or loss of link - Should be used in conjunction with Transmit Idle SRDF/A Delta Set Extension (DSE) provides a mechanism for augmenting the cache-based Delta Set buffering mechanism of SRDF/A with a disk-based buffering ability. This extended Delta Set buffering ability may allow SRDF/A to ride through larger and/or longer SRDF/A throughput imbalances than would be possible with cache-based Delta Set buffering alone. When many SRDF/A groups run in parallel in the same Symmetrix array, complex I/O profiles and potential interruptions in link availability and bandwidth, make the task of calculating reasonable cache requirements difficult. The SRDF/A Reserve Capacity feature, Delta Set Extension (SRDF/A DSE), extends the cache space available for SRDF/A session cycles by off loading some or all of its cycle data from cache to preconfigured disk storage, or pools, which are similar to SAVE device pools. Using SRDF/A DSE pools provide a safety net for running SRDF/A sessions; however, SRDF/A will continue to operate normally with this feature disabled. * DSE Pools * Save Pools are designated as DSE pools at creation – Contains SAVE devices of a single emulation + CKD, FBA or AS400 * Pools can be associated (shared) with multiple SRDF/A RDF groups * Must have at least one DSE pool configured with a type that matches one of the device types in the SRDF/A

group in order to activate DSE * Pools can optionally start automatically when SRDF/A is enabled * SRDF/A DSE Pools and Save devices are managed in the same way as TimeFinder/Snap pools. * A RDF group can have at most one pool of each emulation. * A single rdfa_dse pool can be associated with more than one RDF group, similar to snap pools shared by multiple snap sessions. * SRDF/A DSE Threshold sets the percentage of cache used for SRDF/A that will start offloading cache to disk. * DSE must be enabled on both the source and target arrays. Extension on only one side of a link would lead to failure of the SRDF/A recovery with SRDF/A dropping because the R2 side would fail to have enough cache to hold the large and extended Transmit cycle. When Can DSE Help You? * DSE is not designed to solve any permanent and persistent problems: – Mis-configurations: - Unbalanced cache or backend on target side is smaller/weaker than source side - Not enough cache - Not enough bandwidth – Host writes consistently exceeding RDF link bandwidth – Prolonged link outages * SRDF/A DSE solves abnormal and temporary problems – Unexpected host load – Link bandwidth issues – Temporary link loss (use with Transmit Idle) * Increase resilience for SRDF/A SRDF/A DSE should be used with the Transmit Idle feature. Thus, SRDF/A can ride through a temporary link loss. Once the DSE threshold is reached, data is paged out to the DSE Pools. Recovering After Loss of Links * It is recommended to split a BCV copy of the R2 prior to starting any resynchronization * In the event of an extended loss of link, a large number of R2 invalid tracks can build up on the R1 side * It is advisable to enable SRDF/A after the two sides are synchronized * Resynchronization prior to enabling SRDF/A can be performed by: – Setting SRDF mode to Adaptive Copy Disk As noted earlier, during resynchronization, R2 does not have consistent data. A BCV copy of the consistent R2 data prior to resynchronization can safeguard against unexpected failures during the resynchronization process. When the link is resumed, if there are a large number of invalid tracks owed by the R1 to its R2, it is recommended that SRDF/A not be enabled right away. Enabling SRDF/A right after link resumption causes a surge of traffic on the link due to (a) shipping of accumulated invalid tracks, and (b) the new data added to the SRDF/A cycles. This would lead to SRDF/A consuming more cache and reaching the System Write Pending limit. If this happens, SRDF/A would drop again. Like with SRDF/A, resynchronization should be performed during periods of relatively low production activity. Resynchronization in Adaptive Copy Disk mode minimizes the impact on the production host. New writes are buffered and these, along with the R2 invalids, are sent across the link. The time it takes to resynchronize is elongated. Resynchronization in Synchronous mode impacts the production host. New writes have to be sent preferentially across the link while the R2 invalids are also shipped. Switching to Synchronous is possible only if the distances and other factors permit. For instance, if the norm is to run in SRDF/S and toggle into SRDF/A for batch processing (due to higher bandwidth requirement). In this case, if a loss of links occurs during the batch processing, it might be possible to resynchronize in SRDF/S. In either case, R2 data is inconsistent until all the invalid tracks are sent over. Therefore, it is advisable to enable SRDF/A after the two sides are completely synchronized. Recovery Example – Disabling RDFA Consistency C:\symrdf –g testdg6 disable –nop C:\symrdf –g testdg6 query As per recommendation, we will next place the devices in SRDF Adaptive Copy Disk mode (C.D) and

wait for the pair states to become Synchronized. But before change the mode to ACP_Disk, we have to disable consistency protection via symrdf –g testdg6 disable command. When consistency is enabled, one cannot switch out of SRDF/A without first disabling it. Once the synchronization is complete, we can then enable SRDF/A and enable consistency.

The session is still in asynchronous mode but consistency has been disabled. Recovery Example – set mode to Adaptive Copy C:\symrdf –g testdg6 set mode acp_disk –nop C:\symrdf –g testdg6 query

The session is now in adaptive copy – Disk mode but still in a suspended state As mentioned, we will next place the device group in Adaptive Copy Disk mode (C.D). Notice the device group is still in a “Suspended” state. Also notice the invalid tracked owed to the R2 side of the device group. Once we resume the link (symrdf –g testdg6 resume –nop), those invalids propagate across the link. The R1/R2 pairs will become “Synchronized”. Once full synchronization has accorded, the storage administrator can set mode to asynchronous replication mode if desired. SRDF Session Recovery Tool (C:\symrecover) * EMC SRDF session recovery utility is initiated by the symrecover command * Runs in the background and monitors the state of SRDF/A or SRDF/S sessions * If failure is detected, automatic recovery and restart is attempted based on symrecover options file (pre-configured) * The symrecover command can be run from the R1 or R2 side - As long as all the devices making up the group being monitored are fully viewable from the host - When concurrent RDF is used, this command must be run from the R1 side * Solutions Enabler 6.4 or higher must be installed The symrecover command can be run manually from the command line, but more commonly will be setup to run continuously in the background by using the Windows Scheduled Tasks, the UNIX CRON/scheduled. The symrecover command can be run from either the R1 or the R2 side, as long as all the devices in the monitored group are fully viewable from the execution host. However, when concurrent RDF is used, this command must be run from the R1 side. To manually start symrecover for an SRDF/A composite group named RDFAmon, using the options file named cg_mon_opts, enter: symrecover start -cg RDFAmon -mode async -options cg_mon_opts The following is an example of the cg_mon_opts option file: # Option file for symrecover ####################################################### goldcopy_type_r2 = bcv goldcopy_bcv_r2_mirror_state_startup = establish goldcopy_bcv_r2_mirror_state_post_restart = split goldcopy_max_wait_bcv = 3600 log_level = 3 monitor_cycle_time = 600 monitor_only = 0 rdfg = name:London ... Failover/Failback with SRDF/A

* If the primary site fails, data on R2 is consistent up to the last Apply cycle (N-2) - Partial data in the Receive cycle is discarded * SRDF failover procedure can then be executed and the workload can be started on the R2 devices - Consistency protection should be disabled prior to issuing symrdf failover without the –force option * Failback procedure after the primary site has been restored is identical to Synchronous SRDF - After symrdf failback command completion, workload can be restarted on the R1 devices. SRDF/A can be enabled Again, it is advisable to split off a BCV copy of the R2 prior to executing a failback operation. When workload is resumed on the R1 devices immediately after a failback, accumulated invalid tracks have to be synchronized from the R2 to the R1, and new writes must be shipped from the R1 to R2. If there is an interruption now, data on the R2 is not consistent. Even though SRDF/A can be enabled right after a failback, for reasons stated earlier, it should be enabled after the SRDF pairs entered the Synchronized state. Multiple Independent SRDF/A Groups C:\symcfg –RA all list

RA Group 6 uses RDF Director 2D where RA Group 7 is using RDF Director 15D and 2D. C:\symrdf –rdfg <#> list

Notice - At this point both RDF-Group “6” and “7” are in a “Consistent” state. RDF-Group 6 is using RA 2D where RDF-Group 7 is using RA 15D and 2D C:\symrdf –g testdg6 query * Because of a lost link, (RA director “2D” has been disconnected) the state of the RA-Group (Group “6”) becomes “Partitioned” * Another way of looking at it – The device pairs within device group “testdg6”are now in a “Partitioned” state

C:\symrdf –g testdg7 query * Notice that “Device Group testdg7” remains in a “Consistent” state where Device Group testdg6” is in a “Partitioned” state. * That’s because RA Group 7 also using RA director 15D and it’s link is valid. What is SRDF/A Multi Session Consistency (MSC) * Manages multiple SRDF/A sessions logically as if they were a single session – RDF Daemon for Open System support – Sessions can be within or across Symmetrix arrays – Ensures a complete, restartable point-in-time copy on the remote side Since Enginuity 5671, consistency protection for SRDF/Asynchronous devices is provided using Multi Session Consistency (MSC). If one or more source (R1) devices in an SRDF/A MSC enabled RDF consistency group cannot propagate data to their corresponding target (R2) devices. The MSC process suspends data propagation from all R1 devices in the consistency group, halting all data flow to the R2 targets SRDF/A with MSC supported by an RDF process daemon that performs cycle-switching and cache recovery operations across all SRDF/A sessions in the group. This ensures that a consistent R2 data copy of the database exists at the point-in-time any interruption occurs. A composite group must be created using the RDF consistency protection option (-rdf_consistency) and must be enabled using the symcg enable command before the RDF daemon begins monitoring and managing the MSC consistency group. * RDF Daemon coordinates cycle switching of the SRDF/A MSC group sessions as a single entity – Responsible for detecting ‘failure’conditions that would cause data on the R2 side to become inconsistent – When a ‘failure’ condition is detected, the cycle switching for all SRDF/A sessions in the group are stopped in a manner that leaves the R2 side with a consistent data image. The RDF process daemon maintains consistency for enabled composite groups across multiple Symmetrix arrays for SRDF/A with MSC. For the MSC option (-rdf_consistency) to work in an RDF consistency-enabled environment, each locally-attached host performing management operations must run an instance of the RDF daemon (storrdfd). Each host running storrdfd must also run an instance of the base daemon (storapid), which coordinates all Symmetrix locks and parallel application syscalls. Optionally, if the Group Naming Services (GNS) daemon is also running, it communicates the composite group definitions back to the RDF daemon. If the GNS daemon is not running, the composite group must be defined on each host individually. How does Multi Session Consistency work? In single session, SRDF/A cycle switch occurs when the Transmit cycle on the R1 side AND the Apply cycle on the R2 side are both empty. The switch is controlled by Enginuity. In MSC, the Transmit cycles on the R1 side of all participating sessions must be empty, and also the corresponding Apply cycles on the R2 side. The switch is coordinated and controlled by the RDF Daemon. All host writes are held for the duration of the cycle switch. This ensures dependent write consistency. SRDF/A MSC Operations * Set SYMAPI_USE_RDFD = ENABLE in options configuration file

* Create a Composite Group (CG) with the -rdf_consistency option • Group definition is passed to the RDF Daemon as a ‘candidate group’ • If the Daemon is not already running, it is started automatically * Add all of the devices in the multiple SRDF/A sessions to the CG * Put all CG devices into Async mode: symrdf -cg <CGname> set mode async * Enable CG devices for consistency protection: symcg -cg <CGname> enable • The RDF Daemon is notified that the group should now be monitored • Enable command must be done after the devices are put into Async mode * When the devices become RW on the link, the RDF Daemon: • Starts performing cycle switching • Actively monitors the health of the group to maintain R2 data consistency There are three ways the RDF daemon can be started. If the RDF daemon is enabled, the daemon is started automatically by the Solutions Enabler libraries the first time they attempt to connect with it, which can cause a slight delay in performance on that initial connection while the daemon starts and builds its cache. Note: Prior to starting storrdfd, ensure that your default SYMAPI configuration database is up-to-date, since storrdfd uses the information stored in it to establish contact with your Symmetrix arrays. Alternatively, the daemon can be started manually via the stordaemon command line utility as follows: - stordaemon start storrdfd [-wait Seconds] Note: The stordaemon command requires a path of /usr/storapi/storbin. By default, the stordaemon command waits 30 seconds to verify that the daemon is running. To override this, use the -wait option. Additionally, the daemon can be set to start automatically every time the local host is booted using the following command line: - stordaemon install storrdfd –autostart. Pre-starting the daemon, either manually or via the automatic option, is useful because the daemon may take a while to initially construct its cache - depending on the number of groups and Symmetrix arrays it has to load. If the daemon is stopped for some reason, it can optionally be restarted automatically by an internal Solutions Enabler watchdog mechanism. A combination of the watchdog mechanism and the auto-start option described above can be used to ensure that the daemon is always running. To stop the RDF daemon, use the following command: - stordaemon shutdown storrdfd|all [-wait Seconds] Applying the “all” option stops all of the daemons currently running, such as storapid, storgnsd, and storrdfd. Managing RDF Daemon * Modify the SYMAPI options file: SYMAPI_USE_RDFD=ENABLE * Start Daemon: stordaemon start storrdfd * Stop Daemon: stordaemon shutdown storrdfd Prior to starting storrdfd, ensure that your default SYMAPI configuration database is up-to-date, since storrdfd uses the stored information to establish contact with your Symmetrix arrays. There are three ways the RDF daemon can be started. First, if the RDF daemon is enabled, the daemon is started automatically by the Solutions Enabler libraries the first time they attempt to connect with it, which can cause a slight delay in performance on that initial connection while the daemon starts and builds its cache. Second, the daemon can be started manually via the stordaemon command line utility as follows: stordaemon start storrdfd [-wait Seconds] By default, the stordaemon command waits 30 seconds to verify the daemon is running. To override this, use the -wait option. Third, the daemon can be set to start automatically every time the local host is booted using the following command line: stordaemon install storrdfd -autostart Pre-starting the daemon, either manually or via the automatic option, is useful because the daemon may take a while to initially construct its cache - depending on the number of groups and Symmetrix arrays it has to load. MSC: Loss of Links for One RA Group C:\symrdf –cg testdg67 query We currently have a total loss of link for RA Group 6. The devices in this group go to a Partitioned state. However, note that the devices in RA Group 7 have been placed in Suspended state. Loss of links for one RA Group trips the Multi-session Consistency Group. This prevents the propagation of data for the other RA Groups in the MSC-CG, thus preserving a consistent image of data on ALL R2s at the time of link loss. As stated

earlier, the storrdfd daemon is responsible for coordinating cycle switches between the RA groups, to trip the MSC-CG and perform any recovery/cache cleanup that might be necessary when the links are resumed. Recovering from this state can be accomplished as usual: symrdf –cg testdg67 establish Once the invalid tracks are marked, merged, and synchronized, MSC protection is automatically re-instated; i.e., user does not have to issue symcg –cg testdg67 enable again. MSC Cleanup * There are three possible cleanup scenarios in case of a failure in MSC: 1 All Receive cycles are marked as complete. In this case, the Receive cycles are committed – i.e. promoted to Apply 2 Some Receive cycles are marked as complete and others are marked as incomplete. In this case, ALL Receive cycles are discarded 3 Some Receive cycles have been promoted to Apply, where as some of them have not. In this case, the promoted receive cycles and those not yet promoted are committed The first scenario is easy to understand. In this instance, the Receive cycles contain the most recent and consistent data. The second situation arises if there is a failure when some Receive cycles are complete while the others are in transit. In this case, clearly it is only the Apply cycles of all sessions that contain the consistent data. Therefore, ALL Receive cycles are discarded. To understand the third scenario, keep in mind the following: For a cycle switch to be initiated, ALL Transmits must be empty and all Apply’s must be empty. This means that the failure has occurred DURING the cycle switch process in this case. Receive can only be promoted to Apply on a cycle switch. In MSC, cycle switch is sent to all sessions at once. So each of them is in the process of executing a cycle switch. For example, failure occurred prior to promoting some Receives to Apply. Therefore, the Receives not yet promoted should be committed along with those that have already been promoted. * Cleanup is automatically performed by the RDF Daemon on the R1 side, if the link to the R2 side is available * If the link is unavailable (total site failure on the R1 side), then invocation of any SRDF command, such as symrdf failover or split, from the R2 side performs the automatic cleanup. SRDF/A Configuration Parameters * Two SRDF/A Symmetrix Array-wide parameters • Maximum SRDF/A Cache Usage • Maximum Host Throttle Time * Two RDF Group (aka SRDF/A Session) level settings • Minimum Cycle Time • Session Priority * Settable via the SYMCLI symconfigure command • Set Symmetrix Metrics for Symmetrix level attributes set symmetrix rdfa_cache_percent = 94; set symmetrix rdfa_host_throttle_time = 0; • Set RDF Group Metrics for Group level attributes set rdf group 3, session_priority = 33; set rdf group 3, minimum_cycle_time = 30; SRDF/A System Configuration Parameters * rdfa_cache_percent • Defaults to 94, with a range of valid values from 0 to 100 percent • It is the percentage of the Max# of System Write Pending Slots available to SRDF/A. The purpose is to ensure that other applications can utilize some of the WP limit • When SRDF/A hits its WP cache limit it will be forced to drop SRDF/A sessions to free up cache • Setting it lower reserves WP limit for non-SRDF/A cache usage. Setting it higher, allows SRDF/A to potentially use more of the cache WP limit, potentially creating performance problems for other applications * rdfa_host_throttle_time • Defaults to 0, with a range of valid values from 0 to 65535 • If >0, this value overrides the rdfa_cache_percent and session_priority settings • When the System WP Limit is reached, throttling will delay a write from the host until a cache slot becomes free • The value is the number of seconds to throttle host writes before dropping SRDF/A sessions. A value of 65535 means wait forever. SRDF/A Group/Session Configuration Parameters * SRDF/A group - minimum_cycle_time

– Defaults to 30, with a potential range of 5 to 59 seconds * SRDF/A group - session_priority – Defaults to 33, with a range of valid values from 1 to 64. – The number ranks the SRDF/A session relative to other sessions to determine the order for dropping sessions should the cache WP limit be reached – When SRDF/A needs to drop sessions when the cache WP limit is reached. The sessions are dropped, starting with priority values of 64. Those with a setting of 1 are last to be dropped. Monitoring SRDF/A * Using the symstat command options symstat –type cycle –reptype rdfa –rdfg all symstat –type cache –reptype rdfa –rdfg all symstat –type request –reptype rdfa –rdfg all * Using the symevent command symevent list –error C:\symstat -type cache –reptype rdfa –rdfg all

Note the RA Grp “5” has a default Priority value of 33. Also, this is where Cache Slots available is reported for all SRDF/A sessions A lso notice that RA Grp “5” is currently active. C:\symstat -type cycle –reptype rdfa –rdfg all

This output was captured from the R1 side. As displayed, the Minimum cycle time for each of the SRDF/A session is at the default of 30 seconds. The Active Cycle is the Capture and the Inactive is the Transmit, as this output is from the R1 (source) perspective. From the R2 prospective the Active Cycle is Apply and the Inactive is Receive. When Max Cache is used up… A Giving Preference to Sessions via Priority Setting… B (A) For this example, the maximum available cache slots for SRDF/A has been reduced to 50% (1.68/3.36). Now, cache utilization reaches a 100% of available cache, and RDFG 5 is dropped (Inact ive).

(B) In this instance, the Cache available for SRDF/A has been set to 80% (2.69/3.36). When 100% (103.2) of this is used, we see that the session with the least priority is dropped first – RA Group 4 with a priority of 33.

C:\symevent list –error The dropping of the SRDF/A session can also be displayed via the symevent command. Concurrent SRDF with SRDF/A * One source, two targets * One target sync (SRDF/S) • short distance, zero data lag * One target async (SRDF/A) • longer distance, variable data lag, no performance impact * SRDF: Remote Re-synch • Continued protection upon source failure In an SRDF configuration, a single source (R1) device can concurrently be remotely mirrored to two target (R2) devices. This allows you to have two identical remote copies available at any point in time. It is valuable for duplicate restarts or disaster recovery, or for increased flexibility in data mobility and migrating applications. Concurrent RDF technology can use two different RA adapters (RAs, RAFs, or RFs) in the interface link to achieve the connection between the R1 device and its two concurrent R2 mirrors. Each of the two concurrent mirrors must belong to a different RDF (RA) group. Enginuity 5671 supports Concurrent SRDF with SRDF/A. Only one of the SRDF mirrors is allowed to be in Asynchronous mode, regardless if SRDF/A is active or not. SRDF/A Restrictions * Each SRDF/A group must be unidirectional * Concurrent RDF with both RDF mirrors in SRDF/A mode is not allowed * A personality swap with SRDF/A running is not allowed * Cannot activate empty SRDF/A group * SRDF/A-capable devices that are enabled for consistency group protection must be disabled before attempting to change the mode from asynchronous * Symmetrix RDF Automated Replication (SRDF/AR) control operations are not supported for SRDF/A-capable devices running in asynchronous mode Symmetrix Remote Data Facility / Automated Replication * Allows business restart site to be any distance away from source * Collaboration of “SRDF” and “TimeFinder” commands * Minimizes network costs SRDF/AR allows users to automate the sequence of SRDF and TimeFinder mirror operations. The automated sequence -cycle - is performed on a user-defined interval called cycle time. The replication cycles automatically loop indefinitely or to the number of cycles specified by the users. Users perform all SRDF/AR operations, setup, start, stop, restart, and query, through the symreplicate command. Even though the SRDF link can be set to all SRDF operational mode, except Asynchronous, it is usually set to operate in Adaptive Copy mode due to the long distance between local and remote sites. This allows the users to save on network bandwidth, thus minimizing the network costs without compromising the integrity of the data. SRDF/AR Single-Hop Configuration * Uses 2 Symmetrix Arrays – Uses STD, R1-BCV, R2 and BCV device types – SRDF Mode of Operation . Adaptive copy – Controlled data loss . Remote BCV can be used for disaster restart

The copy path for a single-hop configuration is from the local R1/BCV pair (1) to the SRDF pair (2) to the remote BCV pair (3). The remotely associated BCV holds the DBMS restartable copy. The amount of data loss is a function of the replication cycle time (period of time between the start of one copy cycle and the start of another copy cycle). Copy cycle time is affected by distance, bandwidth, I/O update rate, and locality of reference for the updates. Update rate and locality of reference tend to equate to changed tracks. The maximum data loss would be one copy cycle, thus makes the RPO ~ One Cycle Time. Single Hop benefits include: * The ability to perform incremental resynchronization between the intermediate SRDF target site and the final SRDF target site, reducing required network bandwidth * Reduction in communication link cost and improved resynchronization time for long-distance SRDF implementations. SRDF/AR Multi-Hop Configuration * Uses 3 Symmetrix Arrays: – Local site, Hop1 bunker site, and Hop2 target site – Uses R1, R2, R1-BCV, and BCV (optional) device types – SRDF Mode of Operation . Local to Hop 1 Bunker: Synchronous . Hop 1 Bunker to Hop 2 Target: Adaptive Copy – Zero data loss at Bunker site – One cycle data loss at Target site The copy path for a multi-hop configuration is from the local SRDF pair (1) to the remote BCV pair (2) to the remote SRDF pair (3) to the Target BCV (4). If your configuration does not include Target BCVs, the path stops at (3). Automated replication with the BCVs at Target is applicable if you want a zero data loss solution but cannot risk the loss of both the Local site and Bunker site at the same time. With this configuration, there are two possible disaster restart possibilities: y If only the Local site is lost, the result is zero data loss at the Target restart site. y If both the Local and Bunker site are lost, the result is a DBMS restartable copy at the Target restart site with controlled data loss. The amount of data loss is a function of the replicate copy cycle time between the Bunker site and the Target restart site. Multi-Hop benefits include: y The ability to perform incremental resynchronization between the intermediate SRDF target site and the final SRDF target (Multi-Hop) site, reducing required network bandwidth y Reduction in communication link cost and improved resynchronization time for long-distance SRDF implementations y The ability to use the SRDF Multi-Hop site to provide disaster recovery testing, point-in-time backups, decision support operations, third-party software testing, and application upgrade testing or the testing of new applications Single Hop Set-up: Create a Device Group

Single Hop Set-up: Initialize Mirrors 1 Establish the STD and R1/BCV (Note1): symmir -g testdg8 est –full (Note - This step will automatically suspend the RDF link) 2 Split the STD and R1/BCV: symmir –g testdg8 split -consistent 3 Resume RDF link (Note above Step 1): symrdf –g testdg8 resume -bcv 4 Establish the R2 and remote BCV (Note1):symmir –g testdg8 establish –full –rdf –bcv

5 Split the R2 and remote BCV: symmir –g testdg8 split –full –rdf –bcv 6 Establish the STD and R1/ BCV: symmir -g testdg8 establish There are six steps to prepare the state of all the mirrors involved in SRDF/AR Single Hop prior to running the symreplicate command. To begin a replication session in the single-hop configuration, the mirrors must be in the following states: * The local STD/BCV pairs must be fully synchronized * The RDF pairs must be in a suspended state * The remote BCV pairs must be in a split state Note: The user must wait and check for full synchronization before proceeding to the next step and issuing the next command. The symreplicate Command * SYMCLI binary that integrates symmir and symrdf commands with the appropriate arguments and options depending on the configuration * Used to propagate automated data copies – Incremental (changed tracks only) – Behavior is based on pre-defined parameters (symreplicate options file) * When used with TimeFinder consistent split, ensures a remotely-associated BCV contains a DBMS restartable copy of data – Requires PowerPath or ECA * Used in both single-hop and multi-hop configurations * Used to start, stop, restart, or query SRDF/AR replication sessions * Executed on the local host The symreplicate command set invokes a session that generates automated, recurrent background copies of the standard data across an RDF link and cascading BCVs. The symreplicate command set will start, stop, and restart SRDF/AR session. The symreplicate Options File: User created, named, and configurable text file that defines and controls SRDF/AR replication behavior. The option file is a text file that contains parameters such as SRDF/AR hop type and cycle time. This file is required and is used in conjunction with the symreplicate command to start a replication session. This slide lists the variables available to be placed in the SRDF/AR options file. There are a minimum of two options that must exist for the symreplicate to be executed. The two required options are: SYMCLI_REPLICATE_HOP_TYPE and one of SYMCLI_REPLICATE_CYCLE or SYMCLI_REPLICATE_CYCLE_DELAY. As of Solutions Enable 6.2 additional SYMCLI REPLICATE variables have been added.. Ref the “SE” product guide for a complete detailed explanation for each option setting. SYMCLI_REPLICATE_CONS_SPLIT_RETRY, SYMCLI_REPLICATE_R1_BCV_EST_TYPE, SYMCLI_REPLICATE_R1_BCV_DELAY, SYMCLI_REPLICATE_FINAL_BCV_EST_TYPE, SYMCLI_REPLICATE_FINAL_BCV_DELAY, and SYMCLI_REPLICATE_PERSISTENT_LOCKS SYMCLI_REPLICATE_HOP_TYPE=[SINGLE|MULTI] ** Must be specified SYMCLI_REPLICATE_CYCLE=[minutes|hh:mm] - Defines the period to wait between copy operations if more than 1 cycle - Can be specified in total minutes or hours and minutes - Defaults to 0; when one cycle ends another begins **Either CycleTime or Delay is a required parameter SYMCLI_REPLICATE_CYCLE_DELAY=[minutes] - Specifies the minimum time to wait between the end of one cycle and the beginning of the next cycle - Defaults to 0 ** Either CycleTime or Delay is a required variable

* Sleep time parameters: - Specifies the minimum amount of time the symreplicate process will sleep: - Before checking again to see if the devices have entered a specific state - or - To retry an operation when a recoverable error occurs - symreplicate must wait for prior operations to complete before moving to subsequent operations - If the pair state is not reached, symreplicate sleeps for a period of time and then checks device state again - When checking device state, symreplicate calculates how long to sleep based on the number of invalid tracks and the rate the data is moving - The Sleep time value must be greater than 0 Cycle, Delay and Overflow Parameters

The three parameters SYMCLI_REPLICATE_CYCLE, SYMCLI_REPLICATE_CYCLE_OVERFLOW, and SYMCLI_REPLICATE_CYCLE_DELAY significantly contribute to how the next replication cycle starts after the current cycle. This slide show different combinations of cycle time, delay time, and overflow behavior to achieve various results. Reference points on the cycle time lines are marked ACT (actual completion time), NCS (next cycle start), and MDT (minimum delay time). Short cycle and delay times (in minutes) were chosen for illustration purposes only. Copy cycles #1 and #2 have the same cycle time (2) and no delay time. When copy cycle #1 runs longer than two minutes, the overflow setting of “Next” results in a new copy cycle beginning at the four-minute mark. Copy cycle #3 finishes before its scheduled cycle time of three minutes, waits two minutes, and begins copying again at the next scheduled copy cycle (the six-minute mark). Copy cycle #4 performs continuous copy cycles when its cycle time is set to zero. At the end of a cycle, the system waits three minutes and then begins another copy cycle, regardless of the overflow setting. Defining and Adjusting Parameters * EMC recommends starting with loose time constraints for cycle time parameters y Adjust parameters once basic information is gathered or determined from initial cycles ** Data size ** SRDF throughput ** Operation timings * Session progress can be monitored using the query argument for the symreplicate command and settings can be adjusted as required * The symreplicate stats command will provide statistics on your SRDF/AR sessions It is recommended to be generous with time parameters for cycles, and adjust once more information is collected for various cycles. The times configured in the options file should be configured for worst case scenarios. Beginning with Solutions Enabler 6.1, you can display statistical information for cycle time and invalid tracks by using the symreplicate stats command. The command can be issued by device group (-g) or composite group (-cg) for a specified Symmetrix (-sid) and information can optionally be written to a specified log file (-log).

The -all option is the default and will display both the cycle time and invalid tracks statistics. The following example will display both cycle time and invalid track for device group testdg8 on Symmetrix 23, and will out put it’s result to a log file symreplicate -g testdg8 -sid 23 -all stats –log c:\temp\abcdg.log Automatic Initialization to Required Mirror States

The setup command: * Sets-up required pair states − Options file indicates single or multi hop type setup − Same options file used for setup and subsequent start command * Executes one cycle, which may take some time, and then exits − Setup command executes just one cycle, regardless of the number of cycles specified in the options file The start command with the -setup option: * Sets-up the required pair states * If successful, begins the symreplicate session − Options file defines the hop type and the copy cycle parameters that you have chosen for the session Single Hop Replication Cycle

The symreplicate command automatically executes the above steps in each cycle during a replication session. These steps are similar to those used to initializing the mirror states. It is important to understand these inner working steps because users can choose to terminate a session after the current step instead of at the end of the cycle. The SYMCLI_REPLICATE_LOG_STEP parameter in the option file must set to TRUE for symreplicate to log the information after each step to the “SYMAPI log” file. The location of the SYMAPI varies from one platform to another. The format of the file name, symapi-YYYYMMDD.log, is identical across platforms. Single Hop Example: symreplicate “Setup” The below command runs a symreplicate session in the foreground (note the -foreground flag) so that the resulting output display illustrates the various steps involved in completing one copy cycle. One can also optimize (note the –optimize flag) the disk I/O on standard/BCV pairs in the device or composite group, using the -optimize option when you use the -setup option or the setup argument.

This will cause the setup action to split all pairs and perform an optimized STD-BCV pairing within the specified group. For device groups using this optimize option, the device pair selection attempts topair devices in the group that are not on the same disk adapter to distribute the I/O. C:\symreplicate –g testdg8 setup –optimize –options sar_options_v2.txt –foreground –nop * The –optimize flag will cause the “setup” action to split all pairs and perform an optimized STD-BCV pairing within the specified device/con group * The –foreground flag will output the results of the symreplicate “setup” action Single Hop Example: Query and Start c:\symreplicate –g testdg8 query c:\symreplicate –g testdg8 start –options sar_options_v2.txt –consistent –nop * The “-consistent” flag used with the “start” action, will consistently split all of the BCV pairs on the local Symmetrix array * Use the symreplicate “query” command to check the Session “type”and “statues” Query shows the setup has completed. We are now ready to start the single-hop replication cycles. When executing the symreplicate start command with the -setup option, the first cycle puts the devices in the required pair state. The following command line shows how to execute the symreplicate start command with the “-setup” flag. symreplicate -g testdg8 start -options sar_options_v2.txt –setup Using the -consistent split option flag: The “-consistent” flag used with the start action will consistently split all of the BCV pairs on the local Symmetrix array for a typical SRDF configuration, or on the Hop 1 remote Symmetrix array for a multi-hop configuration. Single Hop: “Stopping” a Session C:\symreplicate –g testdg8 stop

Note - Setting the “symcli_replication_num_cycles” to “0” will results in continuous cycling until the “symreplicate stop” command is issued. The symreplicate command supports both single-hop and multi-hop SRDF configurations. You can start, stop, or restart a replicate session without degradation of the data copy. During a replication session, you can have access. SYMCLI_REPLICATE_NUM_CYCLES=NumCycles “NumCycles” indicates the number replication cycles (copies) to perform before symreplicate exits. For example, a zero value (the default value) results in continuous cycling until the symreplicate stop command is issued. Single Hop: “Restarting” a Stopped Session C:\symreplicate –g testdg8 restart –recover

The “-recover” option is used to recover the device locks from a previously started session. If you begin a session and specify a user log file name (-log), you must specify the -log option for all other commands in the session sequence. If you begin a session specifying only the group name (-g, -cg), the log file will be named the same as the group and must be specified using only the -g option for all other commands in the session sequence. If the primary node fails at some point in the replication process, the SRDF/AR session can now be restarted from another local host using the following command: symreplicate restart -g testdg8 -log testdg8.log -sid 201 –recover The “-recover” option is used to recover the device locks from the previously started session. Failure/Recovery: Single Hop * When a failure occurs in the primary site, we can have one of the three situations on the target site 1. All BCVs on the target are split from their corresponding R2 devices 2. All BCVs on the target are established with their corresponding R2 devices 3. Some of the BCVs on the target are split from their corresponding R2 devices, while others are still established with their corresponding R2 devices * Recovery/Restart involves identifying devices with the most current and consistent copy of data Determination of the states of the devices and deducing the cycle step using the states can be performed from a host on the target side, using appropriate device files. These operations can be performed from a different host on the source side, if the failure only affected the primary host, and the array and site are still available and accessible. Clustered SRDF/AR C:\symreplicate start –g testdg8 –log srdfar1.log –sid 97 SFS = “Symmetrix File System”

Since Enginuity 5669, Symmetrix arrays support clustered SRDF/AR environments for multiple node (host) capability. Clustered SRDF/AR provides the capability to start, stop, and restart replication sessions from any host connected to any local Symmetrix array participating in the replication session. The clustered SRDF/AR environment allows the replication log file to be written directly to the Symmetrix File System (SFS) instead of the local host directory of the node that began the session. If the primary node should fail, then any locally attached host to the Symmetrix array containing the log file would then be able to restart the SRDF/AR session from where it left off. If you begin a session and specify a user log file name (-log), you must specify the -log option for all other commands in the session sequence. To write the log file to the SFS, you must specify the ID of the Symmetrix array (-sid) where the log file is to be stored at the start of the replication session, along with a group name (-g, -cg) and an optional user log filename (-log). For example: symreplicate start -g session1 -log srdfar1.log -sid 201 Note:

Not specifying the Symmetrix ID (-sid) at the start of the session causes the log file to be written to local disk using the default SYMAPI log directory, which is not restartable from another node. General Recovery Procedures * Once the R2 devices on the target site are populated with the most recent and consistent data, they must be RW enabled for the host * Application restart times must be factored in to the overall Recovery Time Objective * Availability of necessary device file definitions at the target site can help in reducing the RTO. Device files must be updated on any configuration changes – adding more Standard devices or R1s * It is preferable to restart/recover from the R2 devices – Facilitates easier return home – BCVs can be used to preserve previous cycle’s data RW enabling of R2s can be performed via the symrdf failover command with the appropriate device file definitions. What is “Cascaded” SRDF * Cascaded SRDF – No data loss – A three site configuration * New Symmetrix Device – A new device with a “dual” role – It’s called an “R Twenty One” - R21 Cascaded Symmetrix Remote Data Facility is a new, three site “No Data Loss” disaster recovery configuration where data from a Primary Site (Site A) is synchronously replicated to a Secondary Site (Site B), then asynchronously replicated to a Tertiary Site (Site C). Cascaded SRDF uses a new dual role SRDF device. The new device is referred to as an R twenty-one (R21). The R21 device is installed at the Secondary Site (Site B). This device can act as both an R2 to the Primary Site (Site A) and an R1 to the Tertiary Site (Site C), simultaneously. This can help reduce the number of devices required for a 3-site “No Data Loss” extended distance replication solution. Who Will Use Cascaded SRDF? * Customers > No Data loss >RPO >RTO * Largest EMC Customers >Worlds Largest Global Investment Banks >World’s Largest Financial Services Banks >World’s Largest Brokerage Firms Cascaded SRDF is another three site “No Data Loss” configuration targeted at our most discriminate customers with stringent, Recovery Point Objective and Recovery Time Objective targets. The RPO is the point in time to which systems and data must be recovered after an outage, for example, the end of the previous day’s processing. The RTO is the period of time within which systems, applications, and functions must be recovered after an outage, for example, one business day. Most of these customers, although small in numbers, are the largest EMC accounts. Benefits of Cascaded SRDF * Major Customer Benefits > Faster recovery > Zero data loss > Optimal RPO and RTO > Cost savings using R21 device > Integrated with TimeFinder products * Major EMC Benefits > Expand SRDF robustness > Remain competitive > Foundation for Cascaded SRDF/STAR Listed are some of the major customer benefits when using Cascaded SRDF. There can be faster recovery times at the Tertiary Site (Site C), which is enabled by the capability to continue replicating or quickly initiating replication from the Secondary Site (Site B) to the Tertiary Site (Site C) in the event that the Primary Site (Site A) goes down. Cascaded SRDF can facilitate and assist customers in achieving quicker RTO (Recovery Time Objectives).

There could be zero data loss achievable up to the point of the Primary Site (Site A) failure. Possible cost savings, due to the fact Cascaded SRDF only requires 3 full copies for operation compared to 4 or 5 with SRDF Automated Replication (SRDF/AR). The major benefit for EMC is it expands the robustness of SRDF, making it more attractive for customers to purchase. Hardware/Software Requirements * Hardware – DMX3 ; DMX4 ; Enginuity 5773 required at Secondary Site (Site B) ; 5772 ; 5671 * Software – Solutions Enabler 6.5 or later – Symmetrix Management Console 6.1 or later Enginuity 5773 is required to take advantage of the new SRDF R21 device. 5773 can be loaded on a DMX3 or DMX4. 5773 is required at the Secondary Site, Site B. The Primary and Tertiary sites are not required to be at 5773, however they can be. The Primary Site and Tertiary Site can be at 5772 or 5671. An example of this would be 5772 on the Primary Site connected to 5773 running on the Secondary Site to 5671 running on the Tertiary Site. There could be fixes needed to connect a 5773 DMX to Symmetrix running older Enginuity, however there are no fixes specifically for Cascaded SRDF. The starting version of Software for Open Systems are Solutions Enabler 6.5 and Symmetrix Management Console 6.1. For Mainframe, Host Component 5.6. There are no plans for making this R21 devices available at older code levels. SRDF/AR – vs. – SRDF Cascaded

In the figure above, (Diagram “A”) showing a multi-hop SRDF/AR configuration, a complete copy cycle, copies: From the local R1 to R2 standard device of the remote Hop 1 Symmetrix (path 1a). From the Hop 1 R2 standard device to its BCV (RBCV) (path 2a). From the Hop 1 RBCV device to the R2 standard device of Hop 2 Symmetrix (path 3a). From the Hop 2 R2 standard device to its BCV (RRBCV) (path 4a). This SRDF/AR configuration offers a disaster recovery solution. There can be four or five copies of data on SRDF and BCV devices. Cascaded SRDF offers another solution. However, Cascaded SRDF does not replace SRDF/AR. With the introduction of “Enginuity 5773” a new “R21” Device type has been introduced. The basic Cascaded SRDF configuration (Ref Diagram “B” above) consists of a Primary Site (A) replicating to a Secondary Site (B) and then replicating the same data to a Tertiary Site (C). Having the same device with both an R2 and R1 mirror, at the Secondary Site (B), means the data will pass automatically from mirror to mirror and onto the Tertiary Site (C). The core benefit behind a “Cascaded” configuration is its inherent capability to continue replicating from the Secondary Site (B) to the Tertiary Site (C) in the event that the Primary Site (A) goes down. This enables a faster recovery at the Tertiary Site (C), provided that is where the customer is looking to restart their operation. Cascaded SRDF uses the dual role R21 device on the Secondary Site (B). The R21 device acts as both an R2 to the Primary Site (A) and an R1 to the Tertiary Site (C). Data Modes of Operation for “Cascaded” RDF

Notice “Asynchronous” mode cannot be run on both sides of the link at the same time. Enginuity only supports asynchronous mode on one side. The R21 Device The R21 device acts as the R2 device for the R1 to R2 relationship for the primary to secondary sites. It also acts as the R1 device for the R1 to R2 relationship in the secondary to tertiary sites. The R21 device takes the place of the R2 and R1 BCV at the Secondary Site (Site B) in an SRDF/ Automated Replication configuration. There are two different RA groups required on the R21 device. RA group 01 contains the R1 device from the Primary Site (Site A) and the R2 side of the R21 device from the Secondary Site (Site B). RA group 02 contains the R1 side of the R21 device paired with the R2 side from the Tertiary Site (Site C). The R1 at the Primary Site (Site A) will only see the R2 side of the R21 device. The R2 at the Tertiary Site (Site C) will only see the R1 side of the R21 device. Solutions Enabler Output of R21 Device C:\symrdf list –R21 C:\symdev –sid 98 list –R21

Symcfg discover will discover the new R21 devices.

Symdev list, with the new –R21 flag, will display the R21 device. In this example output, the R21 devices are shown as not visible to the host and are not mapped to a front end director. The backend directors (DA) are listed as is the device protection (config). The state and capacity or each R21 device is also displayed. C:\symrdf list –cascade To display the R1, R2, or R21 devices separately, use the -R1, -R2, or -R21 option with the symrdf list command. For example, if you want to display

the SRDF R1 devices in Symmetrix array 123, enter “symrdf -sid 123 -R1 list”. The -cascade option will list all R21 devices and the R1 and R2 devices with which they are paired. Using the -cascade flag in conjunction with either the -R1, -R2, or -R21 flags will filter the display to only R1, R2, or R21 devices that are participating in a cascaded relationship. Using the -concurrent option with symrdf list, will display all the Symmetrix devices which are configured by SYMCLI as concurrent RDF devices. Since R21 devices are considered concurrent RDF devices, using the -concurrent flag will display these devices. In the example above (Which is executed from the Secondary Host) shows the R21 (003F) and it’s R1 Pair (003F). The command also shows it’s tertiary pair (0049). RDF Group 1 is currently in a synchronous state, where RDF Group 2 is in a suspend state. Always note the location of the –cascaded command. The information will be the same but its format/presentation will be slightly different. Setting up a Cascaded RDF Environment

Note the following screen shots will reference the above SRDF cascaded set up For the purposes of explaining SRDF “Cascaded”, the above example will be used from this point on. Before any storage administrator initiates any cascaded set up, a diagram should be created, identifying all the known entities. For our example, the “Primary” Symm array is 92. The “Secondary” Symm array is 94 and the “Tertiary” array is 34. RDF Group “21” (-RDFG 21) will be the controlling link between the “Primary/Secondary” sites, where RDF Group “61” (-RDFG 61) will be the controlling link between the “Secondary/Tertiary” sites. Most of our commands to create our SRDF “Cascaded” environment will be performed from the “Secondary” Host. We reference this host as our “Controlling Host” or “Win 2”. For the purposes of this example, we have selected device “F9” at each site. Performing a symdev show for all three devices will show device configuration for each to be a 2-Way Mirror with dynamic capability (RDF1 or RDF2). It is important to note at this time that knowing the location of command executio is key to understanding SRDF “Cascaded”. Identifying R21 devices 1. C:\symdev show F9 –sid 94 2. C:\symdev show F9 –sid 92 3. C:\symdev show F9 –sid 34 As mentioned, for the purposes of this example, we have selected device “F9” at each site. Performing a symdev show for all three devices will show device configuration for each to be a 2-Way Mirror with dynamic capability (RDF1 or RDF2). Performing a combination of symrdf –rdfg # list and symdev show , we see device F9 (-sid 92) is paired with F9 (-sid 94) via RDF group 21. This is what will become the “Primary to Secondary Leg” for our cascaded environment. Performing a second set of symrdf –rdfg # list and symdev show commands, we see that device F9 (-sid 94) is paired with F9 (-sid 34) via RDF group 61. This is what will become the “Secondary to Tertiary Leg” of our cascaded environment.

Creating the “Primary” to “Secondary” relationship C:\symrdf createpair –file cas_v1.txt –type R2 –rdfg 21 –establish –rdf_mode sync testdg10 –sid 94 The first step in setting up an SRDF cascaded environment is to perform a symrdf createpair command, establishing the “Primary to Secondary leg”. Because we are performing this command from the “secondary” host (Win2), the –type is an “R2”. The device pair file is “CAS_V1.Txt”. This file was created before the symrdf createpair command was executed. Also notice the device group name “testdg10”. At this point, all we have done is create an RDF device group (R1 to R2) and placed it into a full establish (SRDF/S) state, via RDF group 21. Performing a “Query” on the primary to secondary relationship C:\symrdf –g testdg10 query

Performing a symrdf query will show the R1 (-sid 92) to R2 (-sid 94) relationship. Note the Link is Read/Write. The R2 device is in a “Write Disabled” (WD) state and the R1 device is in a full “Read Write” (RW) enabled state. The relationship between the R1 to R2, at the moment, is fully synchronized. Also note that the query command was performed from the Secondary host (Win2). Creating the “Secondary” to “Tertiary” cascaded pair C:\symrdf createpair –file cas_v1.txt –type R1 –rdfg 61 –establish –rdf_mode async –sid 94

We are now set to create our “Secondary to Tertiary leg” of our cascaded environment. To do this we run the symrdf createpair command a second time. For our example, we are executing this command from the secondary host, but because we are creating an R1 to R2 relationship, the createpair type is now an R1 (-type R1). The createpair is now using RDF Group 61 and we are placing this leg of the cascaded environment into an asynchronous (SRDF/A) mode of operation. Reference the “Data Modes of Operation Table” o identify cascaded modes allowed across all three sites. Once this command executes successfully, we are in an “SRDF Cascaded” mode of operation.

Performing a “Query” on the secondary to tertiary cascaded pair C:\symrdf –g testdg10 query –rdfg 61 In the above example, the user tried to perform a query on device group testdg10 and was informed that testdg10 is an “RDF21 DG”. The testdg10 device group is in a full cascaded mode and the RDF group number must be provided to query which leg of the cascaded environment one wants to query. The command Symrdf –g testdg10 query –rdfg 21 will report the statues of the “Primary/ Secondary” leg of the cascaded environment, where the Symrdf –g testdg10 query –rdfg 61 will report the status of the “Secondary/Tertiary” leg of the cascaded environment. Looking at the cascaded pairs C:\symrdf list -cascade

Running the above (symrdf list –cascade) command from the secondary host shows the “Primary/Secondary” leg of our cascaded environment is in a “synchronous: state, were the “Secondary/Tertiary” leg in a “asynchronous” or “consistent” state. Enabling and verifying consistency on the “Secondary/Tertiary” leg can be accomplished with the following commands: symrdf –g testdg10 enable –rdfg 61 symrdf –g testdg10 query -rdfa –rdfg 61 RDFG 21 relationship from the primary site C:\symrdf query –g testdg10

Performing a symrdf query from the “Primary” host does not require an RDF group identifier. From the primary host, the “R21” is an R2. From the Secondary host the “R21” is an R1 The query from the primary host against “testdg10” is in fact an SRDF device group query. The above command will simply report out the R1 to R2 relationship. Notice the output will report the “Primary” sym to be 92 (-sid 92) and the “Secondary” sym being 94 (-sid 94). Again, as mentioned earlier note where the command is being executed. RDFG “61” relationship (–hop2) from the primary site C:\symrdf query –g testdg10 –hop2

Performing the query command with the –hop2 flag from the primary host will report out the “Secondary /Tertiary” leg of the cascaded environment. Note the “Primary” sym being 92 (-sid 92), “Secondary” sym being 94 (-sid 94) and the “Tertiary”sym being 34 (-sid 34). Performing this command on the Tertiary host will report the status of the “Primary/Secondary” leg of the cascaded environment. Splitting the RDFG “21” relationship from the secondary site C:\symrdf –g testdg10 split –rdfg 21

The above command executed from the secondary host will split the primary/secondary RDF link. The following command will perform the same function when executed from the primary host. c:\symrdf –g testdg10 split -nop

Splitting the RDFG “21” relationship from the secondary site C:\symrdf –g testdg10 query –rdfg 21 Te result of the primary/secondary cascaded link is now split. Both the R1 and R2 devices are “Read Write” (RW) enabled where the link is “Not Ready” (NR). Notice that the Device Group type is identified as an “RDF21”. What are the following commands accomplishing? 1. C c:\symrdf –g testdg10 query 2. C c:\symrdf –g testdg10 query –hop2 3. B c:\symrdf –g testdg10 –rdfg 21 split 4. A c:\symrdf –g testdg10 enable –rdfg 61 5. B c:\symrdf –g testdg10 split –rdfg 61 6. B c:\symrdf –g testdg10 query rdfa –rdfg 61 7. B c:\symrdf –g testdg10 set mode acp_disk –rdfg 61 8. C c:\symrdf –g testdg10 set mode acp_disk –hop2 9. B c:\symrdf –g testdg10 suspend –rdfg 21 10. B c:\symrdf –g testdg10 resume –rdfg 61 1. The status of the secondary/tertiary cascaded link run from the tertiary host 2. Query command run from the tertiary host reporting the status of the primary/secondary link 3. Split command run from the secondary host, splitting the primary/secondary link 4. Enabling consistency on the SRDF/A connection between the secondary/tertiary link, run from the primary host 5. Split command run from the secondary host, splitting the secondary/tertiary link 6. Query command run from the secondary host checking for SRDF/A consistency across the secondary/tertiary link 7. Setting the mode to adaptive copy disk across the secondary/tertiary link from the secondary host 8. Setting the mode to adaptive copy disk across the primary/secondary link from the tertiary host 9. Suspending the primary/secondary link from the secondary host 10. Resuming the secondary/tertiary link from the secondary host Solutions Enabler Support for R21 Devices * BCV can not be a R21 Device • BCV can be paired with a R21 device * Thin Devices • can not be R21s * Not Supported • Concurrent to Cascaded – (an R1 to two R21s) • Cascaded to Cascaded – (an R21 to an R21) A BCV cannot be a R21 device. However an R21 device can be paired with a BCV. Thin devices can not be R21 devices. An R11 device cannot be in an SRDF relationship with two R21 devices. An R21 device cannot be in an SRDF relationship with another R21 device. Note: R11 is also known as a concurrent device. Cascaded SRDF – RDF21 Group Types * New –type RDF21 • symdg –type RDF21 create R21_test_dg • symcg –type RDF21 create R21_test_cg * Restrictions: • RDF21 Dg Devices must use the same RA Groups • The RDF Mirror Types for a RDF Group must match • This rule applies to both RDF21 Dgs and Cgs Device groups and composite groups can be created to contain R21 devices. These groups can be identified with a new RDF group type: RDF21. The above syntax shows creating an RDF21 device group and composite group using: “–type RDF21”. There are some restrictions using “-type RDF21”. The RDF21 can not be of the type power path. The reason being power path RDF consistency never supported concurrent RDF. Next, RDF21 devices must use the same RA Groups.

In Example 1, the R21 device group, Testdg10, contains Dev001 with its R1 Mirror in RA group 5 and its R2 Mirror in RA group 8. Any more R21 devices added to Testdg10 must belong to the same RA groups. In Example 2, Dev002 cannot be added to Testdg10 since its R2 mirror belongs to RA group 9. If it belonged to RA group 8 it could be added. Example 3 shows the mirror types do not match. The R1 mirror belongs to RA group 8 and the R2 mirror belongs to RA group 5. The mirrors are reversed. The R1 mirror should belong to RA group 5 and the R2 mirror should belong to RA group 8. These rules apply to both device groups and composite groups. Cascaded: “-hop2” Flag Symdg create testdg10 –type rdf1 Symld –g testdg10 –sid 92 add dev 288 Symbcv –g testdg10 –hop2 –rdfg 21 –remote_rdfg 61 associate dev 097 Symclone –g testdg10 –hop2 create –precopy –diff Symclone –g testdg10 query –hop2 Using SRDF technology and the TimeFinder “-hop2” option, you can clone devices on a Symmetrix array located at the Tertiary Site (Site C) of a Cascaded SRDF configuration. Performing SYMCLI commands from the controlling host allows the tertiary target device to receive a copy of the data from the R2 device. The cloned data can then be accessed by the remote “Host 3”. The following steps outline an example of cloning devices on a Symmetrix array located at the Tertiary Site (Site C) of a Cascaded SRDF configuration. To create an RDF1 device group execute: symdg create testdg10 –type rdf1 To add an R1 device to the device group execute: symld -g testdg10 –sid 92 add dev 288 To associate a Hop 2 BCV with the RDF1 device group execute: symbcv -g testdg10 -hop2 -rdfg 21 -remote_rdfg 61 associate dev 097 The BCV device (097) will hold the clone copy. To clone an immediate full copy from the source device to the remote BCV target device: symclone -g testdg10 –hop2 create -precopy –diff To query the progress of the clone operation or verify when the copy is completed, issue the following commands that examine the clone pair. symclone -g testdg10 query -hop2 symclone -g testdg10 verify –copied -hop2