hp storage works xp cluster extension

HP StorageWorks

XP Cluster Extension Software AdministratorGuideThis guide contains detailed instructions for configuring and troubleshooting HP StorageWorks XP ClusterExtension Software in AIX, Windows, Solaris, and Linux environments. The intended audience has independentknowledge of related software and of the HP StorageWorks XP disk array and its software.

Part Number: T1656-96035Fifteenth edition: April 2010

Legal and notice information

© Copyright 2003-2010 Hewlett-Packard Development Company, L.P.

Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Itemsare licensed to the U.S. Government under vendor's standard commercial license.

The information contained herein is subject to change without notice. The only warranties for HP products and services are setforth in the express warranty statements accompanying such products and services. Nothing herein should be construed asconstituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation.

UNIX is a registered trademark of The Open Group.

Contents

1 XP Cluster Extension features ............................................................. 11Integration into cluster software ................................................................................................... 11Enhanced disaster tolerance ....................................................................................................... 11Automated monitoring and redirecting of XP Continuous Access Software pairs ................................ 11Rolling disaster protection .......................................................................................................... 12Command-line interface (CLI) ..................................................................................................... 12Fast Failback using XP Continuous Access Software ...................................................................... 12XP Cluster Extension configurations ............................................................................................. 12

One-to-one configurations ................................................................................................... 12Consolidated-site configuration ............................................................................................ 13Supported XP Continuous Access Software configurations and fence levels ................................ 14XP Cluster Extension server configurations .............................................................................. 14

Planning for XP Cluster Extension ................................................................................................ 15Before configuring XP Cluster Extension resources ................................................................... 15Cluster setup considerations ................................................................................................. 15

MNS quorum clusters (MSCS) ........................................................................................ 15SLE HA cluster setup considerations ................................................................................ 15RHCS cluster setup considerations .................................................................................. 18

Setting up XP RAID Manager ............................................................................................... 20Creating XP RAID Manager command devices ................................................................ 20Creating XP RAID Manager instances ............................................................................. 20Creating XP RAID Manager device groups ...................................................................... 21Network considerations ................................................................................................ 21Starting and stopping the XP RAID Manager instances ..................................................... 22Test takeover function ................................................................................................... 22

2 Configuring XP Cluster Extension for AIX ............................................. 25Configuring resources ................................................................................................................ 25

Configuring the pair/resync monitor ..................................................................................... 25Updating the remote access hosts file ............................................................................. 25Configuring the pair/resync monitor port ........................................................................ 26

Integrating XP Cluster Extension into HACMP ................................................................................ 26User configuration file for HACMP .............................................................................................. 28Bringing a resource group online ................................................................................................ 30Taking a resource group offline .................................................................................................. 31Pair/resync monitor integration ................................................................................................... 32

Adding a Custom Cluster Event for pair/resync monitor integration ........................................... 32Redefining the Custom Cluster Event as a pre-event of the standard HACMP event ...................... 33

Timing considerations ................................................................................................................ 34Failure behavior ....................................................................................................................... 35Restrictions for IBM HACMP with XP Cluster Extension ................................................................... 35

3 Configuring XP Cluster Extension for Windows ..................................... 37Integrating XP Cluster Extension with MSCS ................................................................................. 37

XP Cluster Extension Software Administrator Guide 3

Configuring XP Cluster Extension ................................................................................................ 37Starting the XP Cluster Extension configuration tool ................................................................. 37Defining XP Cluster Extension configuration information using the GUI ....................................... 38Defining XP Cluster Extension configuration information using the CLI ........................................ 40Importing and exporting configuration information ................................................................. 41

Exporting configuration settings using the GUI ................................................................. 41Exporting configuration settings using the CLI .................................................................. 42Importing configuration settings using the GUI ................................................................. 42Importing configuration settings using the CLI .................................................................. 42

Adding an XP Cluster Extension resource ..................................................................................... 42Adding an XP Cluster Extension resource using the Cluster Administrator GUI (Windows Server2003) ............................................................................................................................... 43Adding an XP Cluster Extension resource using the Failover Cluster Management GUI (WindowsServer 2008/2008 R2) ...................................................................................................... 44Adding an XP Cluster Extension resource using the Microsoft CLI cluster commands .................... 44

Changing an XP Cluster Extension resource name ......................................................................... 44Changing an XP Cluster Extension resource name (Windows Server 2003) ................................ 44Changing an XP Cluster Extension resource name (Windows Server 2008/2008 R2) ................. 45

Configuring XP Cluster Extension resources ................................................................................... 45Setting Microsoft cluster-specific resource and service or application properties ........................... 46Setting XP Cluster Extension-specific resource properties .......................................................... 49

Setting XP Cluster Extension resource properties using the Cluster Administrator GUI (WindowsServer 2003) .............................................................................................................. 49Setting XP Cluster Extension resource properties using the GUI (Windows Server 2008/2008R2, Server Core, and Hyper-V Server) ............................................................................. 54Setting XP Cluster Extension resource properties using the MMC ........................................ 62Setting XP Cluster Extension resource properties using the CLI ............................................ 63Setting XP Cluster Extension properties using a UCF ......................................................... 64

Adding dependencies on an XP Cluster Extension resource ............................................................ 64Adding dependencies using Cluster Administrator (Windows Server 2003) ............................... 65Adding dependencies using Failover Cluster Management (Windows Server 2008/2008 R2) ..... 65Adding dependencies using the CLI ...................................................................................... 66

Disaster-tolerant configuration example using a file share ............................................................... 66Managing XP Cluster Extension resources .................................................................................... 70

Bringing a resource online ................................................................................................... 70Taking a resource offline ..................................................................................................... 70Deleting a resource ............................................................................................................ 70

Using Hyper-V Live Migration with XP Cluster Extension .................................................................. 71Timing considerations for MSCS ................................................................................................. 72Bouncing service or application .................................................................................................. 72Administration .......................................................................................................................... 73

Remote management of XP Cluster Extension resources in a cluster (Windows Server 2008/2008R2) ................................................................................................................................... 73Remote management of XP Cluster Extension resources in a cluster (Windows Server 2003) ......... 73System resources ................................................................................................................ 74Logs ................................................................................................................................. 74

Hyper-V Live Migration log entries .................................................................................. 74

4 Configuring XP Cluster Extension for Solaris ......................................... 75Configuration of the XP Cluster Extension agent ............................................................................ 75

Disaster tolerant configuration example using a web server ...................................................... 75Configuring the pair/resync monitor ............................................................................................ 78

Updating the remote access hosts file .................................................................................... 78Configuring the pair/resync monitor port .............................................................................. 78

4

Including the XP Cluster Extension resource type ........................................................................... 79Configuring the XP Cluster Extension resource ............................................................................... 79

XP Cluster Extension resource types ....................................................................................... 79Resource type definition ...................................................................................................... 79

Adding an XP Cluster Extension resource ..................................................................................... 80Adding an XP Cluster Extension resource using the VCS CLI ..................................................... 80Adding an XP Cluster Extension resource using the VCS Cluster Manager GUI ........................... 80

Changing XP Cluster Extension attributes ..................................................................................... 83Changing an attribute value using the VCS CLI ...................................................................... 83Changing an attribute value using the VCS Cluster Manager GUI ............................................. 83

Linking an XP Cluster Extension resource ...................................................................................... 84Linking other resources to the XP Cluster Extension resource ..................................................... 84Linking other resources using the VCS Cluster Manager GUI .................................................... 85

Bringing an XP Cluster Extension resource online .......................................................................... 85Enabling and bringing an XP Cluster Extension resource online using the CLI ............................. 85Enabling and bringing an XP Cluster Extension resource online using the VCS Cluster ManagerGUI .................................................................................................................................. 85

Taking an XP Cluster Extension resource offline ............................................................................. 86Taking an XP Cluster Extension resource offline using the VCS Cluster Manager GUI ................... 86

Deleting an XP Cluster Extension resource .................................................................................... 87Deleting a resource using the VCS CLI .................................................................................. 87Deleting a resource using the VCS Cluster Manager GUI ......................................................... 87

Disabling the XP Cluster Extension agent ...................................................................................... 87Pair/resync monitor integration ................................................................................................... 88

Log-level reporting .............................................................................................................. 88Timing considerations for VCS .................................................................................................... 88Enabling/disabling service groups .............................................................................................. 89Restrictions for VCS with XP Cluster Extension ............................................................................... 90Unexpected offline conditions ..................................................................................................... 90

Bringing the XP Cluster Extension resources online .................................................................. 91

5 Configuring XP Cluster Extension for Linux ........................................... 93XP Cluster Extension for Linux: Sample configuration ...................................................................... 93Configuring XP Cluster Extension with RHCS ................................................................................. 95

Configuration overview ....................................................................................................... 95Creating an RHCS XP Cluster Extension shared resource ......................................................... 95

Using Conga to create a shared resource ....................................................................... 95Using system-config-cluster to create a shared resource ..................................................... 96

Creating an RHCS service using the XP Cluster Extension shared resource ................................. 97Using Conga to create a service .................................................................................... 97Using system-config-cluster to create a service .................................................................. 98Creating the XP Cluster Extension resource configuration file .............................................. 99Testing the service configuration .................................................................................. 100

Managing XP Cluster Extension services (RHCS) ................................................................... 101Starting an RHCS service ............................................................................................ 101Stopping or disabling an RHCS service ........................................................................ 101

Configuring XP Cluster Extension with SLE HA ............................................................................ 102Configuration overview ..................................................................................................... 102Creating and configuring an XP Cluster Extension resource .................................................... 102

Creating the XP Cluster Extension resource configuration file ............................................ 102Creating an XP Cluster Extension resource for Pacemaker ................................................ 103Creating an XP Cluster Extension resource for Heartbeat ................................................. 105Testing the configuration ............................................................................................. 106

Managing XP Cluster Extension services (SLE HA) ................................................................. 106


Rescanning multipath devices ................................................................................................... 106Configuring the rescan script ............................................................................................. 106Finding the user-friendly name of a multipath device ............................................................. 107

Configuring the pair/resync monitor .......................................................................................... 108Updating the remote access hosts file .................................................................................. 108Configuring the pair/resync monitor port ............................................................................ 108

Activating the pair/resync monitor ............................................................................................ 109Timing considerations .............................................................................................................. 109

6 XP Cluster Extension and CLI ........................................................... 111Configuring the CLI ................................................................................................................. 111

Creating the Continuous Access environment and configuring XP RAID Manager ...................... 111Timing considerations ....................................................................................................... 111Restrictions for customized XP Cluster Extension implementations ............................................. 112Creating and configuring the user configuration file .............................................................. 112

CLI commands ....................................................................................................................... 113clxrun ............................................................................................................................. 113clxchkmon ....................................................................................................................... 114

Displaying resources .................................................................................................. 115Removing resources .................................................................................................... 115Stopping the pair/resync monitor ................................................................................ 115

7 XP Cluster Extension recovery procedures .......................................... 119XP disk pair states ................................................................................................................... 119Recovery sequence ................................................................................................................. 120

8 User configuration file and XP Cluster Extension objects ...................... 123User configuration file location ................................................................................................. 123

File structure .................................................................................................................... 124Specifying object values .................................................................................................... 124

COMMON objects ................................................................................................................. 126APPLICATION objects ............................................................................................................. 128

APPLICATION objects ....................................................................................................... 128Basic configuration example ..................................................................................................... 139

9 Advanced XP Cluster Extension configuration ..................................... 141Implementing rolling disaster protection ..................................................................................... 141

Using XP RAID Manager with rolling disaster protection ........................................................ 141Setting XP Cluster Extension objects to enable rolling disaster protection .................................. 141Setting automatic recovery for rolling disaster protection ........................................................ 142Using the pair/resync monitor with rolling disaster protection ................................................. 142Restoring server operation for rolling disaster protection ........................................................ 142

Monitoring and resynchronizing device groups ........................................................................... 143Enabling write access regardless of disk pair state ...................................................................... 144Executing programs before and after an XP Cluster Extension takeover .......................................... 145

Arguments ....................................................................................................................... 145Pre-executable return codes ............................................................................................... 146Post-executable return codes ............................................................................................... 147

10 Troubleshooting ........................................................................... 149XP Cluster Extension log facility ................................................................................................ 149Start errors ............................................................................................................................. 150

6

Failover error handling ............................................................................................................ 150HACMP-specific error handling ................................................................................................. 150

Start errors ...................................................................................................................... 151Failover errors .................................................................................................................. 151

MSCS-specific error handling ................................................................................................... 153Resource start errors ......................................................................................................... 153Failover errors .................................................................................................................. 153Using the Domain user account (Windows Server 2008/2008 R2 only) ................................. 154

VCS-specific error handling ...................................................................................................... 154Start errors ...................................................................................................................... 155Failover errors .................................................................................................................. 155

Linux-specific error handling ..................................................................................................... 156Failover errors .................................................................................................................. 157The FC link is down (RHCS) ............................................................................................... 157A storage replication link is down (RHCS) ............................................................................ 158A data center is down (SLE HA and RHCS) .......................................................................... 158

Pair/resync monitor messages in syslog/errorlog/messages/event log ........................................... 159

11 Support and other resources .......................................................... 161Contacting HP ........................................................................................................................ 161

Subscription service .......................................................................................................... 161New and changed information in this edition ............................................................................. 161Related information ................................................................................................................. 161

White papers .................................................................................................................. 162HP websites ..................................................................................................................... 162

Typographic conventions ......................................................................................................... 162HP product documentation survey ............................................................................................. 163

Glossary .......................................................................................... 165

Index ............................................................................................... 169


Figures

One-to-one (1:1) configuration ................................................................................. 131

Consolidated-site configuration ................................................................................. 142

HACMP configuration example ................................................................................. 293

Service or application example (quorum service control disks not shown) ....................... 674

CLX_FILESHARE resource sample .............................................................................. 675

XP Cluster Extension resource tree for CLX_SHARE ....................................................... 686

VERITAS Cluster Service configuration example ........................................................... 767

Sample resource graph of the CLX_WEB_SERVER service group .................................... 778

Sample configuration .............................................................................................. 939

Disaster-tolerant configuration with rolling disaster protection ...................................... 14310

Incompatible XP disk pair state .............................................................................. 15411

Incompatible XP disk pair state (VCS Cluster Manager Log Desk window) .................... 15612

Detailed information of the XP disk pair state (VCS Log Desk) ..................................... 15613

8

Tables

Setting resource properties and values in the GUI ........................................................ 471

Service or application properties and values .............................................................. 482

XP disk pair states ................................................................................................. 1193

Cluster software supported objects .......................................................................... 1254

Document conventions ........................................................................................... 1625


1 XP Cluster Extension features

HP StorageWorks XP Cluster Extension Software monitors HP StorageWorks XP Continuous AccessSoftware disk pairs and enables automatic access to remote data copies when clustered applicationsbecome unavailable locally. XP Cluster Extension integrates with popular cluster software to ensurethat consistent and concurrent data copies on HP disk arrays can be accessed when needed.

Integration into cluster softwareXP Cluster Extension integrates with the following cluster software products:

• High Availability Cluster Multi-Processing (HACMP)• Microsoft Cluster Service (MSCS)• VERITAS Cluster Server (VCS)• SUSE Linux Enterprise High Availability Extension (SLE HA)• Red Hat Cluster Suite (RHCS)

Integrating XP Cluster Extension with cluster software allows you to manage a disk array as if it werea disk or volume group of the clustered application.

For supported cluster software versions, see the HP SPOCK website: http://www.hp.com/storage/spock.

Enhanced disaster toleranceHP XP Continuous Access Software copies valuable data to a remote data center so that you canrestore application service after a local server, storage, or data center failure. Disk arrays with XPContinuous Access Software can change mirroring direction, swapping the primary/secondaryrelationship of disk pairs almost instantaneously if the application must access the secondary disk.This feature ensures that the failback process is as fast as the failover process. If the links betweenyour primary and secondary disk arrays are broken, each array maintains a bitmap table tosynchronize the changes when the links become available again.

Because cluster software requires the application service to have read/write access to data disks andbecause the secondary volume of an XP Continuous Access Software disk pair is normally read-only,the failover process using cluster software alone typically involves manual intervention. With XP ClusterExtension Software, manual intervention is required only if the current disk array states and usersettings conflict with the rules stored in the XP Cluster Extension database.

Automated monitoring and redirecting of XP Continuous AccessSoftware pairs

XP Cluster Extension monitors the health of the XP Continuous Access Software links between yourarrays. When it detects a lost and later re-established link, it automatically resynchronizes the suspendeddisk pairs, ensuring that the most current data is available on either site. For information on configuringresynchronization parameters, see “Monitoring and resynchronizing device groups” on page 143.


http://www.hp.com/storage/spock


Rolling disaster protectionA rolling disaster is a catastrophic event that affects the remote site after an outage at the local site.In a rolling disaster, data stored on remote disks can be entirely lost during a recovery attempt.

To ensure the survival of critical data during a resynchronization/restore operation, HP StorageWorksXP Business Copy Software pairs can be associated with the local data disks. XP Cluster Extensionrecovers automatically, provided that a local XP Business Copy Software mirror can be suspended.

Although the local copy can be out of date, it represents the best starting point for the recovery.

XP Cluster Extension also resumes local XP Business Copy Software mirrors automatically, if specified,to allow the local site to keep an up-to-date image of the primary data. To implement rolling disasterprotection, see “Implementing rolling disaster protection” on page 141.

Command-line interface (CLI)XP Cluster Extension Software provides a CLI to enable disaster tolerance without cluster software.The CLI is convenient if you use in-house software to migrate application services from one system toanother or if you want XP Cluster Extension to check disk states to make sure you can automaticallystart an application service on the local disk array.

Fast Failback using XP Continuous Access SoftwareXP Cluster Extension supports the XP RAID Manager Fast Failback feature. This feature allows XPContinuous Access Asynchronous Software to automatically redirect the mirroring direction of a diskpair even if the remote XP RAID Manager instance is not available. This ensures the fastest possiblerecovery to the original site in case of an application service failover at the alternate site.

XP Cluster Extension configurationsXP Cluster Extension is an array-based solution. It requires at least two HP XP disk arrays with XPContinuous Access Software providing remote mirroring. XP Cluster Extension connects XP softwarewith cluster software and uses the ability of cluster software to react to system hardware and applicationfailures.

Servers are members of the same cluster dispersed over two or more sites.

XP Cluster Extension supports one-to-one and consolidated-site configurations.

One-to-one configurationsIn one-to-one (1:1) configurations, cluster host nodes are split between two geographically separatedata centers and use redundant, diversely routed network connections for intra-cluster communications.(See Figure 1 on page 13.) These links must be as reliable as possible to prevent false failoveroperations or split-brain situations.

XP Cluster Extension features12

Figure 1 One-to-one (1:1) configuration.

Each cluster host node needs redundant FC or SCSI I/O paths to the XP disk array. Individual hostscannot be connected to both the primary (P-VOL) and the secondary (S-VOL) copy of the applicationdisk set.

HP recommends a minimum of two cluster host nodes per site. This allows for a preferred local failoverin case of a system failure. Local failover operations are faster than a remote failover between XPdisk arrays because the mirroring direction of the XP disks does not need to be changed.

XP Cluster Extension can be deployed in environments where several clusters use the same pair of XPdisk arrays. Although XP disk arrays can be mirrored in various configurations, XP Cluster Extensiondoes not support multiple disk arrays as both primary and secondary disk arrays. XP Cluster Extensionsupports configurations where two or more disk arrays use one remote disk array in a logical one-to-oneconfiguration.

CAUTION:XP Cluster Extension can operate with only one system at each site, with a single I/O path betweenthe server system and the disk array and a single link in each direction between disk arrays.However, those configurations are not considered highly available, nor are they disaster tolerant. XPCluster Extension configurations with single points of failure are not supported by HP.

Consolidated-site configurationIn consolidated-site configurations, a single XP disk array in the secondary (remote) data center isconnected to up to four other primary XP disk arrays (see Figure 2 on page 14.) The restrictionsoutlined in “One-to-one configurations” on page 12 also apply to consolidated configurations. XP


Cluster Extension does not support configurations in which the application service's data disk set isspread over two or more XP disk arrays.

Figure 2 Consolidated-site configuration.

Supported XP Continuous Access Software configurations and fence levelsXP Continuous Access Software offers three modes of replication:

• Synchronous replication• Asynchronous replication• Journal replication

For the replication modes supported by specific versions of XP Cluster Extension, see the HP SPOCKwebsite: http://www.hp.com/storage/spock. For information about synchronous and asynchronousreplication modes, see the HP StorageWorks XP Continuous Access Software User Guide. Forinformation about journal replication, see the HP StorageWorks XP Continuous Access Journal SoftwareUser Guide.

The XP Continuous Access Software fence level is used to configure the remote replication feature ofan XP disk array based on needs for application service availability, data concurrency, and replicationperformance. XP Cluster Extension supports all XP Continuous Access Software fence levels: NEVER,DATA, and ASYNC (includes JOURNAL).

XP Cluster Extension is supported with XP Continuous Access Software in the configurations describedin the HP StorageWorks SAN Design Reference Guide, available at http://www.hp.com/go/sdgmanuals.

XP Cluster Extension server configurationsThe ideal cluster configuration for XP Cluster Extension consists of at least four servers (two at eachsite) and separate redundant communications links for cluster heartbeats, client access, and XPContinuous Access Software. Installing communications interfaces in pairs allows failover and preventssingle points of failure (SPOFs). Using four servers provides faster recovery from a system failure byallowing local application services to fail over to a local cluster system instead of the remote system.On the remote site, HP recommends that two systems be available in case one system experiences ahardware or power failure. In addition to at least four servers, the MNS configuration requires anadditional node per cluster, located at a third site so that whenever a disaster affects either the local



http://www.hp.com/go/sdgmanuals

http://www.hp.com/go/sdgmanuals

or remote site, the other site together with the added node would have a majority. In a MNS withFile Share Witness configuration, the, file share should be located at the third site.

TIP:To upgrade XP firmware while the application service is running, use host load balancing andmultipathing software, such as Auto Path, HP MPIO Full Featured Device Specific Module (DSM) forXP family of Disk Arrays (HP MPIO XP DSM), or VERITAS for Sun Solaris.

XP Cluster Extension allows you to configure the failover behavior so that the application servicestartup is stopped if no remote cluster members can be reached. The default configuration of XP ClusterExtension expects the cluster software to deal with the split-brain syndrome.

Planning for XP Cluster Extension

Before configuring XP Cluster Extension resourcesBefore configuring XP Cluster Extension resources for the Windows CLI implementation or Unixenvironments, review the XP Cluster Extension objects in the UCF.cfg file. For more information aboutXP Cluster Extension objects, see Chapter 8 on page 123.

Cluster setup considerationsFor cluster setup considerations that apply to Windows and Linux, see:

• MNS quorum clusters (MSCS), page 15• SLE HA cluster setup considerations, page 15• RHCS cluster setup considerations, page 18

MNS quorum clusters (MSCS)In an MNS cluster, the cluster service is allowed to start or run only if it has access to the majority ofthe configured nodes.

This means that losing half the nodes in a 2-, 4-, 6-, or 8-node cluster or losing the communicationlinks with 50% of the nodes on each site forces every node to terminate the cluster services becausenone of them have access to a majority of the configured nodes.

Therefore, a geographically dispersed MNS-based cluster requires an additional node per clusterlocated at a third site so that whenever a disaster affects either the local or remote site, the other sitetogether with the added node has a majority.

SLE HA cluster setup considerationsFollow the guidelines in this section when you configure clusters for use with XP Cluster Extension.

Quorum

In an SLE HA cluster, quorum is defined as a strict majority of the defined cluster (more than 50%).With certain failures, a cluster might be divided into two subclusters. In an SLE HA cluster, a subclusterwith more than 50% of the nodes wins the quorum. The subcluster that wins the quorum re-forms thecluster and fences the subcluster that lost the quorum. The behavior of the subcluster that lost the


quorum depends on the defined no-quorum policy. This behavior is in effect until the cluster is fenced.When the cluster is fenced, the resources owned by the fenced nodes fail over to active cluster nodes.

STONITH

STONITH is an SLE HA cluster fencing method. SLE HA cluster provides STONITH plug-ins for devicessuch as UPS, PDU, Blade power control devices, and lights out devices. Some plug-ins can STONITHmore than one node (for example, Split Brain Detector STONITH) and some can STONITH only onenode (for example, HP iLO STONITH).

HP iLO STONITH uses the power control functions of an HP iLO device to STONITH a node that haslost quorum and needs to be fenced.

IMPORTANT:If all of the iLO devices in a cluster are connected using a single network, a single switch failure mightdisable iLO, preventing nodes from being fenced. This failure might be difficult to detect, especiallybefore a node failure where iLO features would be required.

The STONITH action can be set to power off or reset, depending on the environment requirements.

• Power off: The STONITH agent powers off the nodes in the errant subcluster.• Reset: The STONITH agent resets the nodes in the errant subcluster, and the nodes try to automat-

ically rejoin the cluster.

NOTE:IPMI fencing can be used for Integrity servers that do not support RIBCL scripting.

Networking in an SLE HA cluster

Configuring redundant and independent cluster communication paths is a good way to avoid SplitBrain conditions. With redundancy in communication paths, the loss of a single interface or switchdoes not break the communication between nodes and prevents Split Brain conditions.

Administrators can configure multiple independent communication paths. HP recommends usingbonded Ethernet channels.

Resource constraints

Resource constraints allow administrators to specify which cluster nodes resources can run on, theorder resources are loaded, and the other resources a specific resource is dependent on.

There are three types of resource constraints:

• Resource location: Defines the nodes on which a resource can run, cannot run, or is preferred tobe run.

• Resource colocation: Defines which resources can or cannot run together on a node.• Resource order: Defines the sequence of actions for resources running on a node.

Resource operation attribute

SLE HA does not monitor resource health by default. To enable this feature, add the monitor operationto the resource definition. You can specify the interval attribute and the timeout attribute for a monitor


operation. The interval attribute defines the time interval in which the monitor operation is executed.The timeout attribute determines how long to wait before considering the resource as failed.

Define start, stop, and monitor operations for the XP Cluster Extension resource.

XP Cluster Extension resource dependency

A Group resource in an SLE HA cluster ensures that the member resource agents are started andstopped in the required order. An XP Cluster Extension resource must be added as the first memberof the group. This way, all primitive resources added after the XP Cluster Extension resource aredependent on XP Cluster Extension.

Since the primitive resources within a resource group can be failed over independently, set a collocationconstraint for each resource group ID with the last resource in the group to achieve the failover of theentire group when any primitive resource fails.

Failover order

Use location constraints to define the failover order for a resource group.

For each node, define a location constraint with the appropriate score to prioritize the resource groupon that particular node. During failover, the cluster calculates the score of the resource group on theavailable nodes, and the node with the highest score is considered the next preferred owner. Formore information, see the SLE HA documentation.

Failback option

HP does not recommend auto failback in configurations with XP Cluster Extension because the resourcefailovers due to storage failure can cause resources to go into an unstable state (failover/failbackmight toggle the resource between the nodes).

SLE HA provides the meta-attribute resource-stickiness to determine how much a resource agent prefersto stay where it is. To disable auto failback, set resource-stickiness to the lowest value compared tothe other resource location constraints.

Migration-threshold

A resource is automatically restarted if it fails. If a restart cannot be achieved on the current node orit fails to start a certain number of times on the current node, it tries to fail over to another node. Youcan define the number of failures for resources (a migration-threshold) after which they migrate to anew node. If you have more than two nodes in your cluster, the high availability software choosesthe node a particular resource fails over to.

When an XP Cluster Extension resource fails, HP recommends configuring your cluster to fail over theresource without restarting on the local node. To set this preference, set the migration-threshold to 1.

Disk monitoring

For the situations in which disk access is lost or read/write protection is in effect due to storage fencing,application monitoring agents, file system agents, or LVM resource agents detect the IO failure. XPCluster Extension does not monitor the disk access status.


RHCS cluster setup considerations

Quorum

In RHCS, the quorum is based on a simple voting majority of the defined nodes in a cluster. To re-formsuccessfully, a majority of all possible votes is required.

Each cluster node is assigned a number of votes, and they contribute to the cluster while they aremembers. If the cluster has a majority of all possible votes, it has quorum (also called quorate);otherwise, it does not have quorum.

Fencing

Cluster software adjusts the node membership based on various failure scenarios. The concept ofquorum defines which set of nodes continue to define the cluster. To protect data, nodes that do nothave quorum are removed from the cluster. The non-quorate nodes that are removed must be preventedfrom accessing the shared resources. This process is called fencing.

HP iLO fencing is one method that can be used with RHCS to restrict cluster node access to sharedresources.

Observe the following guidelines when using HP iLO network configurations with RHCS clusters:

• HP iLO can be connected to the client access network or to a different network, but the networkmust be routable.

• HP iLO should not be on the network that is used for cluster communication.• The HP iLO of each cluster system must be accessible over the network from every other cluster

system.

To handle infrequent failures of the HP iLO fencing (such as a switch failure), you can set up a backupfence method for redundancy.

HP iLO fencing can be used on HP Proliant systems with built-in iLO hardware. For third-party systems,other power control fencing methods can be used.

NOTE:IPMI fencing can be used for Integrity servers that do not support RIBCL scripting.

Qdisk configuration

Red Hat recommends the use of a Qdisk configuration to bolster quorum to handle failures such ashalf (or more) of the members failing, a tie-breaker in equal split partition, and a SAN failure.

In an XP Cluster Extension configuration with multiple storage arrays, a Qdisk configuration is notsupported.

Failover domains

A cluster service is associated with a failover domain, which is a subset of cluster nodes that areeligible to run a particular cluster service. To maintain data integrity, each cluster service can run ononly one cluster node at a time. By assigning a cluster service to a restricted failover domain, youcan limit the nodes that are eligible to run a cluster service in the event of a failover, and you canorder the nodes by preference to ensure that a particular node runs the cluster service (as long as thatnode is active).


A failover domain can have the following characteristics:

• Unrestricted: Specifies that the subset of members is preferred, but the cluster service assigned tothis domain can run on any available member.

• Restricted: The cluster service is allowed to run only on a subset of failover domain members.• Unordered: The member on which the cluster service runs is chosen from the available list of failover

domain members with no preference order.• Ordered: The failover domain member on which the cluster service runs is selected based on

preference order. The member at the top of the list (as specified in /etc/cluster/cluster.conf) is the most preferred, followed by the second member, and so on.

For an orderly failover, HP recommends using the Ordered and Restricted options for your failoverdomains.

Failback policy

HP does not recommend auto failback in configurations with XP Cluster Extension because the resourcefailovers due to storage failure can cause resources to go into an unstable state (failover/failbackmight toggle the resource between the nodes). In this situation, HP recommends correcting the failureand then manually failing back to the intended data center or server.

To disable the auto failback, set the nofailback flag for the failover domain.

Enabling this option for an ordered failover domain prevents automated failback after a more-preferrednode rejoins the cluster.

Recovery policy

When a resource inside the service fails, the default action is to restart the service on the local nodebefore the failover. In an XP Cluster Extension environment, it is always expected to relocate theservice during restart. To enable this functionality, set the service recovery policy to relocate.

Service hierarchical structure and resource dependency

In RHCS, a service is a collection of cluster resources configured into a single entity that is managed(started, stopped, or relocated) for high availability. A service is represented as a resource tree thatspecifies each resource, its attributes, and its relationship among other resources in the resource tree.The relationships can be parent, child, or sibling. Even though a service is seen as a single entity, thehierarchy of the resources determines the order in which each resource within the service is startedand stopped.

In the case of a child-parent relationship, the startup or shutdown is simple. All parents are startedbefore children, and children must all stop cleanly before a parent can be stopped. For a resourceto be considered in good health, all of its children must be in good health.

A service is considered failed if any of its resources fail. In this case, the expected course of actionis to restart the entire service, including the failed resource and the other resources that did not fail.

In an XP Cluster Extension environment, configure the XP Cluster Extension resource as the parentresource in the service so that XP Cluster Extension can control the service behavior based on the userconfiguration and storage device status. This means that the XP Cluster Extension resource must beconfigured at the highest level in the dependency hierarchy.


Disk monitoring

For the situations in which disk access is lost or read/write protection is in effect due to storage fencing,application monitoring agents, file system agents, or LVM resource agents detect the IO failure. XPCluster Extension does not monitor the disk access status.

Setting up XP RAID ManagerIn addition to the cluster software it integrates with, XP Cluster Extension depends on HP StorageWorksXP RAID Manager. Before configuring XP Cluster Extension, verify that XP RAID Manager is installedand configured, and that the host and disk array systems are properly configured as described in thefollowing topics.

Creating XP RAID Manager command devicesTo control XP Continuous Access Software mirrored disks from a clustered server, install XP RAIDManager on the server and configure a special disk, called a command device. The command devicemust not be an MSCS resource, cannot be paired, and is assigned to a 36 MB or greater CVS volume.The command device is identified by CM appended to the emulation type.

XP RAID Manager command devices can be accessed by redundant paths. HP recommends redundantpaths to prevent XP Cluster Extension from aborting if one path to the command device is missing.See the HP StorageWorks XP RAID Manager User's Guide for more information.

NOTE:If you use Auto Path to enable alternate pathing on IBM AIX together with the XP disk array, XP RAIDManager does not support Auto Path virtual paths for command devices.

Creating XP RAID Manager instancesXP Cluster Extension requires at least one instance of XP RAID Manager. XP Cluster Extension startsthe configured XP RAID Manager instance if it is not running. However, if the XP RAID Managerinstance cannot be started or returns an error, XP Cluster Extension can switch to an alternate XP RAIDManager instance.

• Ensure that the path to the XP RAID Manager binary files is included in the PATH environmentvariable.

• Create an XP RAID Manager instance to control pair operations and to gather disk array statusinformation. Because XP Cluster Extension switches to the next available instance when a currentinstance becomes unavailable, HP recommends that you create several XP RAID Manager instancesto provide redundancy. Bear in mind, however, that the XP RAID Manager instance numbers usedfor the RaidManagerInstances object must be the same among all servers using XP Cluster Extension.

• HP recommends that the XP RAID Manager instances be running at all times to provide the fastestfailover capability. XP Cluster Extension provides scripts to include the XP RAID Manager startupprocedure in the system startup file (for example, /etc/inittab for non-Windows systems). See“Starting and stopping the XP RAID Manager instances” on page 22 for more information. XPCluster Extension starts the configured XP RAID Manager instances if it cannot find any runninginstance.


Creating XP RAID Manager device groupsA device group is the unit in which the failover/failback operation is performed. A device group cancontain several volume groups.

Configure a single device group for a service group (VCS), resource group (HACMP), cluster group(MSCS), or resource (SLE HA, RHCS). This device group must include all disks being used for theapplication service.

The XP RAID Manager configuration file (horcmX.conf) is used to map device groups to the internaldisk array disks. A device group is the common unit for failover operations initiated from the serverside.

Network considerationsBecause XP RAID Manager is essential to XP Cluster Extension, HP recommends that you use theheartbeat network (a private network) for XP RAID Manager communications. Alternative networkpaths are highly recommended. Configure the networks XP RAID Manager uses for each device groupin the HORCM_INST part of the XP RAID Manager configuration file.


Starting and stopping the XP RAID Manager instancesStart the XP RAID Manager instances for XP Cluster Extension at system boot time to provide the fastestaccess to disk status information.

XP Cluster Extension provides scripts (Linux/UNIX) or a service (Windows) to integrate the XP RAIDManager instance startup into the system startup process. This feature reduces resource group failovertimes because the XP Cluster Extension resource does not need to start the XP RAID Manager instances.If the system cannot automatically start and monitor XP RAID Manager instances, you can start andstop XP RAID Manager with the following commands:

Linux/UNIX

horcmstart.sh instance numbers

horcmshutdown.sh instance_numbers

Windows

horcmstart instance_numbers

horcmshutdown instance_numbers

Starting XP RAID Manager without specifying an instance number will start instance 0 with theassociated horcm.conf file. For this reason, zero (0) is not recommended as an instance numberfor an XP Cluster Extension RAID Manager instance.

Test takeover functionAfter configuring XP RAID Manager for the device groups used by XP Cluster Extension, verify thateach device fails over correctly from each server in the cluster. The device group must be in PAIRstate.

CAUTION:XP RAID Manager keeps configuration data of the XP disk array in system memory. Therefore, youmust stop and restart XP RAID Manager instances on all servers if a configuration change is appliedto any XP disk array.

To test the correct failover and failback behavior, log in to each server used with XP Cluster Extensionand invoke the following commands if the local disk is the secondary (S-VOL) disk:

Linux/UNIX

export HORCMINST=instance_number

pairdisplay –g device_group_name –fx –CLI

horctakeover –g device_group_name –t timeout

Windows

set HORCMINST=instance_number

pairdisplay –g device_group_name –fx –CLI

horctakeover –g device_group_name –t timeout


The output of the pairdisplay command indicates whether the local disk is the secondary (S-VOL)disk and if so, the horctakeover command shows a SWAP-takeover as a result. If pairdisplayshows the local disk as the primary (P-VOL) disk, log in to a system connected to the secondary (S-VOL)disk and invoke the horctakeover command there. If the horctakeover command does not resultin a SWAP-takeover, see “Recovery sequence” on page 120 to resolve the issue.

The –t option of the horctakeover command is used only for fence level ASYNC (both Async andJournal).


2 Configuring XP Cluster Extension for AIX

XP Cluster Extension is integrated with the HACMP cluster software using HACMP customizationfunctions. Cluster administrators configure disk array failover as a pre-event of the standard HACMPevent get_disk_vg_fs.

For information about how to install XP Cluster Extension in an IBM HACMP environment, see the HPStorageWorks XP Cluster Extension Software installation guide.

For supported operating system versions, see the HP SPOCK website: http://www.hp.com/storage/spock.

Configuring resourcesThe XP Cluster Extension resource gathers all necessary information about the disk arrays andconfigured device groups whenever a resource group is brought online.

If configured, a pair/resync monitor is started to monitor the XP Cluster Extension resource. To usethis monitor, set the standard HACMP event release_vg_fs to call a pre-event.

Call the XP Cluster Extension binary clxhacmp as a pre-event of the standard HACMP eventget_disk_vg_fs to check the status of the XP RAID Manager device group and, if necessary, toallow access to the device group before HACMP tries to access the disks of the resource group.

Configure XP Cluster Extension parameters with the user configuration file: /etc/opt/hpclx/conf/UCF.cfg. See Chapter 8 on page 123 for more information about the user configuration file.

Configuring the pair/resync monitorThe pair/resync monitor is a service that verifies that disks are in the pair state, and resyncs themwhen necessary. The pair/resync monitor determines whether the requesting server is allowed accessto the pair/resync monitor. To access the pair/resync monitor, you must update the remote accesshosts file and configure the pair/resync monitor port.

Updating the remote access hosts fileEnter the names of the remote systems in a remote access hosts file.

1. Open the /etc/opt/hpclx/conf/clxhosts file.

2. Enter each host name on a separate line.

You can leave blank lines, but do not enter comments. For example:

# cat /etc/opt/hpclx/conf/clxhostsdcBserverdcAserver




Configuring the pair/resync monitor portEnter the port that the pair/resync monitor will monitor.

1. Open the /etc/services file.

2. Choose the port that the pair/resync monitor will use, and then add the following line to theservices file:

clxmonitor nnnnn /tcp

where nnnnn is the port number. For example:

clxmonitor 22222/udp # CLX Pair/Resync Monitorclxmonitor 22222/tcp # CLX Pair/Resync Monitor

Integrating XP Cluster Extension into HACMP1. Create a new Custom Cluster Event:

#smitty hacmp

2. Select the following options:

• Extended Configuration• Extended Event Configuration• Configure Pre/Post-Event Commands• Add a Custom Cluster Event

3. Enter the following values:

• Cluster Event Name: get_disk_vg_fs_pre• Cluster Event Description: Cluster Extension XP

• Cluster Event Script File: /opt/hpclx/bin/clxhacmp

Configuring XP Cluster Extension for AIX26

4. Configure the newly created Custom Cluster Event as a pre-event of get_disk_vg_fs:

#smitty hacmp

5. Select the following options:

• Extended Configuration• Extended Event Configuration• Change/Show Pre-Defined HACMP Events

6. Select the event get_disk_vg_fs.

7. Define the previously defined custom event get_disk_vg_fs_pre as a pre-event ofget_disk_vg_fs.

NOTE:With HACMP 5.2 and later, to have the get_disk_vg_fs event called, you must specifySerial Acquisition Order. If Serial Acquisition Order is not specified, AIX will use the defaultParallel Acquisition Order.

8. Use SMIT and select the following options:

• Extended Configuration• Extended Resource Configuration• Configure Resource Group Run-Time Policies• Configure Resource Group Processing Ordering


9. Specify the resource groups configured to run with XP Cluster Extension.

10. Once the resource groups have been specified, press Enter to complete the configuration process.

XP Cluster Extension controls the disk pairs based on XP RAID Manager device groups. The volumegroup definition of the HACMP resource group is used to determine the corresponding XP RAIDManager device group. The mapping of the HACMP volume group configuration and the correspondingXP RAID Manager device group is done by the XP Cluster Extension user configuration file /etc/opt/hpclx/conf/UCF.cfg. Because of this mapping mechanism, you must specify the volumegroups owned by the HACMP resource groups in the user configuration file.

User configuration file for HACMPBefore configuring the objects in the user configuration file, review the XP Cluster Extension objectsdescribed in Chapter 8 on page 123.

• Set the ApplicationStartup object to RESYNCWAIT, because HACMP does not offer the feature fordisabling resource groups on a particular system to move the resource group back to the mostcurrent copy of your data. If the ApplicationStartup object is set to FASTFAILBACK (default), theresource group will fail to be brought online in cases where the most current copy of your dataresides in the disk array on the remote site. If you set the ApplicationStartup object to FASTFAIL-BACK, you must stop the resource group online process and either resynchronize your disk fromthe remote site or manually bring your resource group online at the remote site.

• The Vgs object in the user configuration file is used to map the volume groups of the HACMP re-source group to the corresponding APPLICATION object within the user configuration file. TheDeviceGroup object within each APPLICATION section determines the XP RAID Manager devicegroup needed to control all the shared disks of the HACMP resource group.


Figure 3 on page 29 and the examples that follow show two possible mappings.

Figure 3 HACMP configuration example.

Example 1The application OracleRG corresponds to an HACMP resource group OracleRG, which consists ofthe volume groups ora1vg and ora2vg. The corresponding XP RAID Manager device group oraclecontrols all disks which form the volume groups of the HACMP resource group. The resource groupis configured to wait for a pair resynchronization in case you have not done any disk pair recoveryafter the resource group has been moved to an alternate system. The resource group will be broughtonline on the local system again (ApplicationStartup object is set to RESYNCWAIT). The AutoRecoverobject is set to NO, which means that you will not utilize XP Cluster Extension capabilities toautomatically recover suspended disk pair states. The DataLoseMirror object and DataLoseDataCenterobject are set to NO, which means XP Cluster Extension will not allow you to bring the resource grouponline if the disk pair is suspended or a takeover operation leads to a suspended disk pair.

Example 2The application SapRG uses the device group sap to control all the disks of the corresponding HACMPresource group SapRG, which uses the volume group sap1vg and sap2vg. The resource group isconfigured to fail back to the remote system rather than waiting for a pair resynchronization, in caseyou have not done any disk pair recovery after the resource group has been moved to an alternatesystem. The resource group will be brought online on the local system again (ApplicationStartup objectis set to FASTFAILBACK per default). This setup will lead to an error loop, as HACMP does notprovide the feature to automatically failback after an error has been reported. The AutoRecover objectis set to NO by default, which means that you will not utilize XP Cluster Extension capabilities toautomatically recover suspended disk pair states.

The following example shows the configuration file for examples 1 and 2:


COMMONLogDir /var/opt/hpclx/log/ #default (optional)LogLevel error # error|info default: error (optional)

APPLICATION OracleRG # package/service group test_applicationVgs ora1vg ora2vg # HACMP specific, to map vg to OracleRGApplicationDir /etc/opt/hpclxXPSerialNumbers 30368 30380RaidManagerInstances 11DeviceGroup oracle # raid manager device groupFenceLevel data # values: data | never | asyncApplicationStartup resyncwait # values: fastfailback | resyncwaitAutoRecover no # possible values: yes | noDataLoseMirror no # possible values: yes | noDataLoseDataCenter no # possible values: yes | noPreExecScript /etc/opt/hpclx/ora_pre.shPostExecScript /etc/opt/hpclx/ora_post.sh

APPLICATION SapRG # package/service group test_applicationVgs sap1vg sap2vg # HACMP specific, to map vg to SapRGXPSerialNumbers 30368 30380RaidManagerInstances 11DeviceGroup sap # raid manager device groupFenceLevel never # possible values: data | never | async

Bringing a resource group onlineResource groups are usually brought online automatically when the cluster is started on a particularserver. If necessary, a resource group can be brought online manually:

1. Run SMIT (HACMP section):

#smitty hacmp

2. Select the following:

• System Management (C-SPOC)• HACMP Resource Group and Application Management• Bring a Resource Group Online


3. Select a resource group from the list of available groups.

Taking a resource group offlineResource groups will usually be taken offline automatically when the cluster is stopped on a particularsystem. If necessary, a resource group can be brought offline manually:

1. Run SMIT (HACMP section):

#smitty hacmp

2. Select the following:

• System Management (C-SPOC)• HACMP Resource Group and Application Management• Bring a Resource Group Offline

3. Select a resource group from the list of available groups.


Pair/resync monitor integrationThe pair/resync monitor is used to detect and react to suspended XP Continuous Access links. It isactivated by setting the ResyncMonitor object to YES. Additionally, the automatic disk pairresynchronization feature is activated if the ResyncMonitorAutoRecover object value is YES.

When the HACMP resource group is taken offline, disable the monitor for the XP RAID Managerdevice group used for this resource group.

CAUTION:If the resource group cannot be taken offline, disable monitoring of the device group for this HACMPresource group. To avoid data corruption, this must be part of the recovery procedure when XP ClusterExtension is deployed in the HACMP environment. See “Stopping the pair/resync monitor” on page 115.Ensure that the pair/resync monitor does not monitor and resynchronize the disk pair (device group)from both disk array sites.

To use the pair/resync monitor you must create a pre-event for the release_vg_fs event. If theresource group is taken offline on a cluster system, the corresponding application/device group istaken from the list of monitored device groups and the monitoring is disabled.

To adjust the event handling:

1. Create an additional Custom Cluster Event. See “Adding a Custom Cluster Event for pair/resyncmonitor integration” on page 32.

2. Configure the new Custom Cluster Event as a pre-event for the standard event release_vg_fs.See “Redefining the Custom Cluster Event as a pre-event of the standard HACMPevent” on page 33.

Adding a Custom Cluster Event for pair/resync monitor integration1. Run SMIT (HACMP section):

#smitty hacmp

2. Select the following:• Extended Configuration• Extended Event Configuration• Configure Pre/Post-Event Commands• Add a Custom Cluster Event


3. Enter the following values:

• Cluster Event Name: release_vg_fs_pre• Cluster Event Description: XP Cluster Extension Pre-Event

• Cluster Event Script Filename: /opt/hpclx/bin/clxstopmonhacmp

Redefining the Custom Cluster Event as a pre-event of the standard HACMP event1. Run SMIT (HACMP section):

#smitty hacmp

2. Select the following:• Extended Configuration• Extended Event Configuration• Change/Show Pre-Defined HACMP Events

3. Select event release_vg_fs.


4. Define the previously defined Custom Cluster Event release_vg_fs_pre as a pre-event ofrelease_vg_fs.

Timing considerationsXP Cluster Extension is designed to prefer XP disk array operations over cluster software operations.If XP Cluster Extension invokes disk pair resynchronization operations or gathers information aboutthe remote XP disk array, XP Cluster Extension will wait until the requested status information is reported.This assumption has been made to clearly prioritize data integrity over the cluster software's failoverbehavior.

In some cases, however, this behavior could lead to an HACMP error event (config_too_long).The default timeout value is 6 minutes. Use the chssys command to increase the timeout. For example:

#chssys -s clstrmgr -a "-u 60000"

NOTE:You must stop the cluster to run the chssys command.

The described timeouts can occur in the following situations:

• XP Cluster Extension uses XP RAID Manager instances to communicate with the remote XP diskarray. Depending on the settings of the XP RAID Manager instance timeout parameter and thenumber of remote instances, the online operation could time out. This can happen if the local XPRAID Manager instance cannot reach the remote XP RAID Manager instance. See “Setting up XPRAID Manager” on page 20 for more details.

• XP Cluster Extension tries to resynchronize disk pairs and waits until the XP RAID Manager devicegroup is in PAIR state if the ApplicationStartup attribute is set to RESYNCWAIT. Depending on theXP RAID Manager version and the XP firmware version, this could be a full resynchronization,which can take much longer than the online timeout interval. Even if the XP RAID Manager versionand the XP firmware version allow a delta resynchronization, the delta between the primary andthe secondary could be big enough for the copy process to exceed the online timeout value.


• If running in fence level ASYNC, the default value of the AsyncTakeoverTimeout can cause theresource group online process to fail because its value is set very high. This is because the takeoverprocess for fence level ASYNC can take longer when slow communication links are in place.To prevent takeover commands from being terminated prematurely by the takeover timeout,measure the time to copy the installed XP disk array cache. To measure the copy time, use onlythe slowest XP Continuous Access Software link. This ensures that the XP disk array cache can betransferred from the remote XP disk array, even in the event of a single surviving replication linkbetween the XP disk arrays.

Because the failover environment is dispersed into two (or more) data centers, the failover time cannotbe expected to be the same as it would be in a single data center with a single shared disk device.

Failure behaviorXP Cluster Extension will run in an endless loop if either of the following is discovered:

• A configuration error• An XP disk pair state that does not allow automated actions

This event is logged in the log files:

/var/opt/hpclx/log/clxhacmp.log and /tmp/hacmp.out.

To return control to the HACMP cluster software, you must remove the lock file:application_dir/application_name.LOCK.

For example: /etc/opt/hpclx/OracleRG.LOCK

This process has been adopted from HACMP's behavior. In the case of a failure, HACMP will alsorun in an endless loop until you recover all errors and manually start the application. After all errorshave been recovered, invoke the command clruncmd to return control to the cluster software.

Restrictions for IBM HACMP with XP Cluster ExtensionThe following is a summary of restrictions that apply for HACMP configurations when XP ClusterExtension is used to enable failover between two XP disk arrays:

• The FastFailbackEnabled object is not used by the XP Cluster Extension integration with HACMP.• XP Cluster Extension must not be used with concurrent resource group configurations (for example,

parallel databases).• XP Cluster Extension must not be used with raw devices without volume groups.• Target mode SCSI: Serial networks within HACMP based on target mode SCSI (TMSCSI) are not

supported.


3 Configuring XP Cluster Extension forWindows

After installing XP Cluster Extension, use the configuration tool to define the XP Cluster Extension setupconfiguration. After you configure XP Cluster Extension, use Cluster Administrator (Windows Server2003), Failover Cluster Management (Windows Server 2008/2008 R2), or cluster commands in theCLI to add and configure resources.

For information about how to install XP Cluster Extension, see the HP StorageWorks XP Cluster Extensioninstallation guide.

Integrating XP Cluster Extension with MSCSTo integrate XP Cluster Extension with MSCS:

1. Configure the XP Cluster Extension application. For instructions, see “Configuring XP ClusterExtension” on page 37.

2. Add an XP Cluster Extension resource. For instructions, see “Adding an XP Cluster Extensionresource” on page 42.

3. Configure the XP Cluster Extension resource. For more information, see “Configuring XP ClusterExtension resources” on page 45.

4. Add dependencies on the XP Cluster Extension resource. For instructions, see “Addingdependencies on an XP Cluster Extension resource” on page 64.

Configuring XP Cluster ExtensionAfter installation, you must define the setup configuration using the XP Cluster Extension configurationtool, and then copy the configuration information to all of the cluster nodes that use XP Cluster Extension.

You can configure XP Cluster Extension with the GUI or the CLI. Use the following instructions for theGUI. For instructions on performing XP Cluster Extension configuration tasks with the CLI, see “DefiningXP Cluster Extension configuration information using the CLI” on page 40.

Starting the XP Cluster Extension configuration toolTo start the XP Cluster Extension configuration tool:

• For Windows Server 2003 or Windows Server 2008/2008 R2: Double-click the HP StorageWorksXP CLX Configuration Tool icon on the desktop, or select Start > Programs > Hewlett-Packard >HP StorageWorks XP CLX Configuration Tool.

• For Server Core or Hyper-V Server: Open a command window and enter CLXXPCONFIG -I.

The XP CLX Configuration Tool window appears.


NOTE:The service name clxmonitor is appended with the text “(not configured)” unless the port numberis configured in the configuration tool.

Defining XP Cluster Extension configuration information using the GUI1. Open the configuration tool. For instructions, see “Starting the XP Cluster Extension configuration

tool” on page 37.

2. The pair/resync monitor monitors the disk pair status if the ResyncMonitor attribute is set to YES,and resynchronizes disk pairs if the ResyncMonitorAutoRecover attribute is set to YES. By default,the pair/resync monitor uses port 5307.

To use a port other than 5307, enter the desired port number in the Port box. The range ofavailable ports is 1024 through 65535. The pair/resync monitor port value must be the sameon all cluster nodes.

Configuring XP Cluster Extension for Windows38

3. Specify the XP RAID Manager instances that define the device groups you want to manage withXP Cluster Extension. For more information about XP Cluster Extension and XP RAID Manager,see “Setting up XP RAID Manager” on page 20.

a. Click Add in the XP RAID Manager Instance Configuration section to open the Add XP RAIDManager instances window.

b. Select the XP RAID Manager instances to use, and then click OK.

4. Specify the servers that are possible owners for the XP Cluster Extension-managed disks. A serveris a possible owner of a disk if it is capable of managing the disk when failover occurs.

a. Click Add in the Server Configuration section to display the Add Servers window.

b. Select the servers that are possible owners of the XP Cluster Extension-managed disks, andthen click OK.

NOTE:See the Microsoft Cluster Administrator (Windows Server 2003) or Failover ClusterManagement (Windows Server 2008/2008 R2) documentation for more informationabout possible owners.


5. Click OK to save the information and close the configuration tool. The configuration informationis saved to the ClxXPCfg file.

NOTE:XP Cluster Extension provides an XP RAID Manager service, which automatically starts XPRAID Manager instances at system boot time. This feature reduces resource group andservice and application failover times because the XP Cluster Extension resource does notneed to start the XP RAID Manager instances. When you click Apply or OK in theconfiguration tool, the XP RAID Manager service is started.

6. Use the procedures in “Importing and exporting configuration information” on page 41 to copythe ClxXpCfg file to the other cluster nodes.

Defining XP Cluster Extension configuration information using the CLIYou can configure XP Cluster Extension with the CLI command CLXXPCONFIG. Enter CLXXPCONFIG-?, CLXXPCONFIG /?, or CLXXPCONFIG /help to view usage information.

1. Enter CLXXPCONFIG -I to open the configuration tool.

2. The pair/resync monitor monitors the disk pair status if the ResyncMonitor attribute is set to YES,and resynchronizes disk pairs if the ResyncMonitorAutoRecover attribute is set to YES. By default,the pair/resync monitor uses port 5307.

To change the pair/resync monitor port, enter CLXXPCONFIG PRM /PORT=value, wherevalue is the port number you want to use.

NOTE:To view the pair/resync monitor port, enter CLXXPCONFIG PRM.


3. Specify the XP RAID Manager instances that define the device groups you want to manage withXP Cluster Extension. For more information about XP Cluster Extension and XP RAID Manager,see “Setting up XP RAID Manager” on page 20.

• To view the available XP RAID Manager instances, enter CLXXPCONFIG RM.• To add an XP RAID Manager instance, enter CLXXPCONFIG RM /ADDVAL=value, where

value is the XP RAID Manager instance you want to add. For example:Enter CLXXPCONFIG RM /ADDVAL=101 to add XP RAID Manager instance number 101.

• To remove an XP RAID Manager instance, enter CLXXPCONFIG RM /REMOVEVAL=value,where value is the XP RAID Manager instance you want to remove. For example:Enter CLXXPCONFIG RM /REMOVEVAL=101 to remove XP RAID Manager instance number101.

NOTE:XP Cluster Extension provides an XP RAID Manager service, which automatically starts XPRAID Manager instances at system boot time. This feature reduces resource group andservice or application failover times because the XP Cluster Extension resource does notneed to start the XP RAID Manager instances. Adding or removing XP RAID Managerinstances will start or restart the XP RAID Manager service.

4. Specify the servers that are possible owners for the XP Cluster Extension-managed disks. A serveris a possible owner of a disk if it is capable of managing the disk when failover occurs.

• To determine whether cluster nodes have been configured for XP Cluster Extension, enterCLXXPCONFIG SERVER.

• To add a server, enter CLXXPCONFIG SERVER /ADD /NAME=servername, whereservername is the server to add.

• To remove a server, enter CLXXPCONFIG SERVER /REMOVE /NAME=servername, whereservername is the server to remove.

5. Use the procedures in “Importing and exporting configuration information” on page 41 to copythe configuration information to the other cluster nodes.

Importing and exporting configuration informationThe import feature allows you to define the setup configuration using an existing configuration file.The export feature allows you to save a copy of an existing configuration file.

Use the import and export features to copy the XP Cluster Extension configuration file (ClxXpCfg)from one cluster node to another.

Exporting configuration settings using the GUI1. Open the configuration tool.

For instructions, see “Starting the XP Cluster Extension configuration tool” on page 37.

2. Click Export.

3. When prompted, choose a save location, enter a file name, and then click Save.

4. Click OK to save and close the configuration tool.


Exporting configuration settings using the CLI1. Open a command window.

2. Enter CLXXPCONFIG EXPORT /FILE=filepath, where filepath specifies the save locationand file name.

Importing configuration settings using the GUI1. Open the configuration tool.

For instructions, see “Starting the XP Cluster Extension configuration tool” on page 37.

2. Click Import.

3. When prompted, choose the configuration file, and then click Open.

4. Click OK to save and close the configuration tool.

Importing configuration settings using the CLI1. Open a command window.

2. Enter CLXXPCONFIG IMPORT /FILE=filepath, where filepath specifies the file locationand name.

Adding an XP Cluster Extension resource

IMPORTANT:In Cluster Administrator (Windows Server 2003), resources are added to resource groups. In FailoverCluster Management (Windows Server 2008/2008 R2), the term resource groups changed to servicesand applications. In this guide, the term services and applications refers to resource groups forWindows Server 2003 and services and applications for Windows Server 2008/2008 R2.

To use XP Cluster Extension, you must add an XP Cluster Extension resource. Use one of the followingprocedures to add an XP Cluster Extension resource:

• For Windows Server 2003, use the Cluster Administrator GUI or cluster commands in the CLI. Forinstructions, see “Adding an XP Cluster Extension resource using the Cluster Administrator GUI(Windows Server 2003)” on page 43 or “Adding an XP Cluster Extension resource using theMicrosoft CLI cluster commands” on page 44.

• For Windows Server 2008/2008 R2, use the Failover Cluster Management GUI or cluster com-mands in the CLI. For instructions, see “Adding an XP Cluster Extension resource using the FailoverCluster Management GUI (Windows Server 2008/2008 R2)” on page 44 or “Adding an XPCluster Extension resource using the Microsoft CLI cluster commands” on page 44.

• For Server Core or Hyper-V Server, use the Failover Cluster Management GUI on the remotemanagement station or cluster commands in the CLI. For instructions, see “Adding an XP ClusterExtension resource using the Failover Cluster Management GUI (Windows Server 2008/2008R2)” on page 44 or “Adding an XP Cluster Extension resource using the Microsoft CLI clustercommands” on page 44.

XP Cluster Extension resource names and service and application names must consist of only oneword. The pair/resync monitor cannot interface with a resource or a service and application that hasspace characters in its name.


CAUTION:Do not use the following characters in XP Cluster Extension resource names: \ / : * ? " < >|. Using these characters might affect the creation of the resourcename.online file, which isused for the XP Cluster Extension resource health check mechanism. If the file creation fails and thepair/resync monitor is not configured, the cluster will report a failed state for the XP Cluster Extensionresource.

Adding an XP Cluster Extension resource using the Cluster Administrator GUI(Windows Server 2003)

Use the procedure in this section to add a resource using the Cluster Administrator GUI. For instructionson using the CLI, see “Adding an XP Cluster Extension resource using the Microsoft CLI clustercommands” on page 44.

1. Add a resource group in the Cluster Administrator GUI, as described in your Microsoftdocumentation.

2. From the File menu, select File > New > Resource.

3. Enter the following values, and then click Next:

Name: The name of the resource.

Description: As appropriate for the resource.

Resource type: Select Cluster Extension XP from the list.

Group: Select a group to associate with the resource.

4. Add or remove possible resource owners, and then click Next.

The Dependencies window appears.

5. Do not add any dependencies. Click Next to open the Parameters window.

The Parameters window contains values entered during the XP Cluster Extension configurationsteps.

6. Modify the resource property values of the new XP Cluster Extension resource as needed, andthen click Finish.


Adding an XP Cluster Extension resource using the Failover Cluster ManagementGUI (Windows Server 2008/2008 R2)

Use the procedure in this section to add a resource using the Failover Cluster Management GUI. Forinstructions on using the CLI, see “Adding an XP Cluster Extension resource using the Microsoft CLIcluster commands” on page 44.

1. Add a service or application in the Failover Cluster Management GUI, as described in yourMicrosoft documentation.

2. Right-click the service or application and select Add a resource > More resources > Add ClusterExtension XP.

Adding an XP Cluster Extension resource using the Microsoft CLI cluster commandsYou can use the cluster commands in this section with Windows Server 2003, Windows Server 2008/2008 R2, Server Core, and Hyper-V Server.

Use the following command to add an XP Cluster Extension resource:

cluster resource resource_name /create /group:service_or_application_name/type:"Cluster Extension XP"

Example

This command adds an XP Cluster Extension resource called clx_fileshare to the CLX_SHAREservice or application.

cluster resource clx_fileshare /create /group:CLX_SHARE /type:"Cluster Extension XP"

Changing an XP Cluster Extension resource nameWhen changing resource names, observe the following rules:

• Do not change an XP Cluster Extension resource name when the resource is online and thepair/resync monitor is enabled for the resource. Changing the name when the resource is onlinemight cause problems with the pair/resync monitor functionality.Changing the resource name is allowed when a resource is offline, or when a resource is onlineand the pair/resync monitor is disabled.

• XP Cluster Extension resource names and service and application names must consist of only oneword. The pair/resync monitor cannot interface with a resource or a service and application thathas space characters in its name.

• Do not use the following characters in XP Cluster Extension resource names: \ / : * ? " <> |. Using these characters might affect the creation of the resourcename.online file, whichis used for the XP Cluster Extension resource health check mechanism. If the file creation fails andthe pair/resync monitor is not configured, the cluster will report a failed state for the XP ClusterExtension resource.

Changing an XP Cluster Extension resource name (Windows Server 2003)In this procedure, you use the Cluster Administrator GUI to change a resource name. For instructionson using CLI commands to change a resource name, see “Setting XP Cluster Extension resourceproperties using the CLI” on page 63.


1. Open Cluster Administrator.

2. Open the resource Properties window and click the General tab.

3. Enter a new name in the Name field.

4. Click OK to save your changes and close the window.

Changing an XP Cluster Extension resource name (Windows Server 2008/2008R2)

In this procedure, you use the Failover Cluster Management GUI to change a resource name. ForServer Core or Hyper-V Server, use the MMC to run the Failover Cluster Management GUI from aremote node or use cluster commands in the CLI to change the resource name. See “Setting XP ClusterExtension resource properties using the CLI” on page 63 for instructions.

1. Open Failover Cluster Management.

2. Open the resource Properties window and click the General tab.

3. Enter a new name in the Resource Name field.


Configuring XP Cluster Extension resources

IMPORTANT:In Cluster Administrator (Windows Server 2003), resources are added to resource groups. In FailoverCluster Management (Windows Server 2008/2008 R2), the term resource groups changed to servicesand applications. In this guide, the term services and applications refers to resource groups forWindows Server 2003 and services and applications for Windows Server 2008/2008 R2.

XP Cluster Extension resource properties are configured using Cluster Administrator (Windows Server2003), Failover Cluster Management (Windows Server 2008/2008 R2), or cluster commands in theCLI.

• For information about MSCS and Microsoft Failover Cluster Service properties that affect XP ClusterExtension, see “Setting Microsoft cluster-specific resource and service or application proper-ties” on page 46.

• For information about XP Cluster Extension-specific properties, see “Setting XP Cluster Extension-specific resource properties” on page 49.

Before configuring XP Cluster Extension resources, review the XP Cluster Extension objects describedin Chapter 8 on page 123.

When configuring XP Cluster Extension resources, note the following:

• If the Cluster Administrator or Failover Cluster Management GUI is used to configure an XP ClusterExtension resource, configuring the resource using a user configuration file (UCF) is not required.

• If the resource is not configured to use the pair/resync monitor, XP Cluster Extension creates a filecalled resource_name.online in the directory specified by the ApplicationDir resourceproperty.If the resource is taken offline, this file will be removed, or the device group associated with theservice or application will be removed from the pair/resync monitor list, if the pair/resync monitor


is configured for that resource. If the device group is the last monitored disk pair, and you takethe resource offline, the pair/resync monitor will be stopped.

• Windows Server 2008 only: If an XP Cluster Extension resource is not configured, the resourceicon in the Failover Cluster Management GUI shows the message “not configured” next to the re-source status.

• The XP Cluster Extension resource must be the first resource for all disk resources in the dependencylist of a resource cluster group.

• If you have an application in a cluster that uses more than one physical disk from the same devicegroup, configure a single XP Cluster Extension resource, and ensure that all of the applicationdisks depend on that resource. If the disks are split into different device groups, you must configuremultiple XP Cluster Extension resources since an XP Cluster Extension resource operates at thedevice-group level.

• The PendingTimeout value must be greater than the ResyncWaitTimeout value.• The PendingTimeout must be greater than twice the wait time of all remote XP RAID Manager in-

stances multiplied by the number of remote systems. Otherwise, the XP Cluster Extension resourcewill fail to go online if there is a complete remote data center failure.tonline > nremote systems x 2 x tWT

where:tonline = resource online timeout

nremote systems = number of remote systems configured to run XP RAID Manager instances

tWT = wait time until remote error will be reported by local XP RAID Manager instance

If a post-executable is specified, the PendingTimeout must be greater than the number of remotesystems multiplied by three times tWT.

Setting Microsoft cluster-specific resource and service or application propertiesMicrosoft allows you to set specific failover parameter and threshold values for a service or application,as well as for a resource. Some of these values must be changed for XP Cluster Extension to enablemanual recovery actions in case of a disaster.

To set Microsoft cluster-specific resource properties:

• For Windows Server 2003, use the Cluster Administrator GUI or cluster commands in the CLI.• For Windows Server 2008/2008 R2, use the Failover Cluster Management GUI or cluster com-

mands in the CLI.• For Server Core or Hyper-V Server, use cluster commands in the CLI.

TIP:You can use the GUI option for Server Core or Hyper-V Server by using the MMC to manage acluster remotely. For more information about using the MMC, see your Microsoft documentation.

XP Cluster Extension requirements for Cluster Administrator and Failover Cluster Management resourceproperties are described in Table 1 on page 47. If there is no required value for a property, the validand/or default values are specified. Set these properties in the resource properties window or theCLI. If you use the CLI, use the following command:

cluster.exe resource ResourceName /prop PropertyName="PropertyValue".


For more information about setting resource properties, see your Microsoft documentation.

Table 1 Setting resource properties and values in the GUI

ValueDescriptionFormatProperty

Windows Server 2008/2008R2 GUI: 01:00 mm:ss (Default)Windows Server 2003 GUI:60000 milliseconds (Default)CLI: 60000 milliseconds(Default)

Used to poll “Alive” state for the resource.Decreasing this value allows faster resourcefailure detection but also consumes moresystem resources.Set this value in the Advanced Policies tabof the resource properties window inFailover Cluster Management, or in theAdvanced tab of the resource propertieswindow in Cluster Administrator.

IntegerThorough Resource HealthCheck Interval (WindowsServer 2008/2008 R2)“Is Alive” poll interval(Windows Server 2003)IsAlivePollInterval (CLI)

Windows Server 2008/2008R2 GUI: 00:05 mm:ss (Default)Windows Server 2008/2008R2 CLI: 5000 milliseconds(Default)Windows Server 2003 GUI:60000 milliseconds (Default)Windows Server 2003 CLI:60000 milliseconds (Default)

Used to poll “Alive” state for the resource.Decreasing this value allows for fasterresource failure detection but alsoconsumes more system resources.Set this value in the Advanced Policies tabof the resource properties window inFailover Cluster Management, or in theAdvanced tab of the resource propertieswindow in Cluster Administrator.

IntegerBasic Resource Health CheckInterval (Windows Server2008/2008 R2)“Looks Alive” poll interval(Windows Server 2003)LooksAlivePollInterval (CLI)

0 (Required)Defines whether a resource can beautomatically restarted after it has failed.Set this value in the Policies tab of theresource properties window in FailoverCluster Management, or in the Advancedtab of the resource properties window inCluster Administrator.

IntegerIf a resource fails, attemptrestart on current node –Maximum restarts in thespecified period (WindowsServer 2008/2008 R2)Restart Threshold (WindowsServer 2003)RestartThreshold (CLI)

Windows Server 2008/2008R2: Check (Required)Windows Server 2003: Restartand affect the group (Required,Default)CLI: 2 restart andaffect the group(Required)

Defines whether resources will be failedover if a restart is unsuccessful.Set this value in the Policies tab of theresource properties window in FailoverCluster Management, or in the Advancedtab of the resource properties window inCluster Administrator.Windows Server 2003 only: This valuemust “affect the group.” This ensures thatthe resource group fails over to anothersystem if a resource is reported FAILED.

IntegerIf restart is unsuccessful, failover all resources in this serviceor application (Windows Server2008/2008 R2)RestartAction (Windows Server2003)RestartAction (CLI)

Windows Server 2008/2008R2: 15:00 mm:ss (Default)Windows Server 2003: 900seconds (Default)CLI: 900000 milliseconds(Default)

Determines the amount of time for restart.Set this value in the Policies tab of theresource properties window in FailoverCluster Management, or in the Advancedtab of the resource properties window inCluster Administrator.

IntegerIf a resource fails, attemptrestart on current node – Periodfor restarts (Windows Server2008/2008 R2)RestartPeriod (Windows Server2003)RestartPeriod (CLI)



Windows Server 2008/2008R2: 03:00 mm:ssWindows Server 2003: 180seconds (Default)CLI: 180000 milliseconds(Default)

Used to specify the timeout for statusresolution. For more information, see“Timing considerations for Microsoft ClusterService ” on page 72.Set this value in the Policies tab of theresource properties window in FailoverCluster Management, or in the Advancedtab of the resource properties window inCluster Administrator.

IntegerPending timeout (GUI)PendingTimeout (CLI)

XP Cluster Extension requirements for service or application properties are described in Table2 on page 48. If no specific value is required, the default value is listed. Set these values in the Failovertab of the service or application properties window (Windows Server 2008/2008 R2), the resourcegroup properties window (Windows Server 2003), or in the CLI. For more information about settingservice or application properties, see your Microsoft documentation.

TIP:To change the properties in Table 2 on page 48 with the CLI, use the following command: clustergroup groupname /prop propertyname="propertyvalue".

Table 2 Service or application properties and values


GUI: PreventfailbackCLI: 0(required)

Prevents automatic fail back of a service or applic-ation to its primary system. Transfer the service orapplication back manually after the failure hasbeen recovered. This allows for recovery of allpossible failure sources and pair resynchronization(if necessary) while the application service is stillrunning.

IntegerGUI: Failback (Preventfailback or Allowfailback)CLI: AutoFailbackType

6 (Default)Determines the time (in hours) over which thecluster service attempts to fail over a service orapplication. See “Timing considerations for Mi-crosoft Cluster Service ” on page 72 for more in-formation.

StringGUI: PeriodCLI: FailoverPeriod

Windows Server2008/2008 R2:Number of nodesin the clusterminus 1.Windows Server2003, CLI: 10(Default)

Determines the number of failover attempts. Thedefault value allows the cluster service to transferthe service or application to each system once incase of subsequent system failure. Due to thenature of this parameter, it is possible that theservice or application automatically restarts on asystem several times if all cluster systems are notmembers of the cluster at that time.If this value is set to a number higher than thecurrent number of clustered systems for the clustergroup, the service or application will continue torestart until either the FailoverThreshold value orthe FailoverPeriod timeout value is reached.

IntegerGUI: Maximum failuresin the specified period(Windows Server2008/2008 R2),Threshold (WindowsServer 2003)CLI: FailoverThreshold


Setting XP Cluster Extension-specific resource propertiesChanges to resource properties take effect when the resource is brought online again. For instructionson changing resource properties, see:

• Setting XP Cluster Extension resource properties using the Cluster Administrator GUI (WindowsServer 2003), page 49

• Setting XP Cluster Extension resource properties using the GUI (Windows Server 2008/2008 R2,Server Core, and Hyper-V Server), page 54

• Setting XP Cluster Extension resource properties using the MMC, page 62• Setting XP Cluster Extension resource properties using the CLI, page 63• Setting XP Cluster Extension properties using a UCF, page 64

Setting XP Cluster Extension resource properties using the Cluster Administrator GUI (WindowsServer 2003)

You can set XP Cluster Extension properties by using the Parameters tab in the Cluster AdministratorGUI.

Configuring XP RAID Manager instance numbers for XP RAID Manager service

Use the Cluster Administrator Properties window to change XP RAID Manager instance numbers.

1. Open Cluster Administrator and double-click the resource you want to edit.

2. Click the Parameters tab.

3. To remove an instance, select it and click Remove.

4. To add an instance:

a. Click Add to open the Add RAID Manager instances window.

b. Select one or more instances, and then click OK.


Configuring the XP RAID Manager device group details1. Open Cluster Administrator and double-click the resource you want to edit.



3. To change the device group details, select a new value in the RM XP device group menu.


Configuring XP RAID Manager device group advanced properties

The Parameters tab of the XP Cluster Extension resource offers basic settings and is used to enterenvironment data, such as XP RAID Manager instances. The more advanced settings can be accessedthrough additional buttons in the Parameters tab.



3. Click the Advanced button to open the Advanced Fence Level Failover Behavior window. Theavailable settings in this window depend on the fence level used with your device groups.

• For the DATA fence level, you can update the Data lose mirror and DATA lose data centervalues. See DataLoseDataCenter on page 133 and DataLoseMirror on page 134 for more in-formation about these values.

• For the ASYNC fence level, you can update the ASYNC takeover timeout value. SeeAsyncTakeoverTimeout on page 130 for more information about this value.


• For the journal fence level, you can update the Journal data currency on S-VOL and ASYNCtakeover timeout values. See JournalDataCurrency on page 136 and AsyncTakeover-Timeout on page 130 for more information about these values.

4. Update the settings as needed, and then click OK to close the window.


Notes

• After a device group is configured in the resource configuration utility, do not change the devicegroup name or swap the name with another device group name in the HORCM file. If you do this,restart the HORCM manager instance and reconfigure the XP Cluster Extension resource.

• Do not use HORCM commands to change the device group property for a device group that isconfigured for an XP Cluster Extension resource. If you do this, the changed property is not reflectedimmediately in the Parameters tab. To work around this situation, re-select the device group fromthe XP RM device group menu in the Parameters tab.

Configuring server data center assignments1. Open Cluster Administrator and double-click the resource you want to edit.


3. To remove a data center assignment, select the assignment, and then click Remove.

4. To modify a data center assignment, select the assignment, and then click Modify. Enter the newData center name in the Modify Node in Data Center List window, and then click OK.

5. To add a data center assignment, click Add. Select a host and a data center, and then click OK.


Changing failover and failback behavior1. Open Cluster Administrator and double-click the resource you want to edit.



3. Click Failover/Failback to display the Failover/Failback window.

4. Update the ApplicationStartup and AutoRecover values as needed, and then click OK.


Activating the pair/resync monitor

The pair/resync monitor detects and responds to suspended XP Continuous Access links if theResyncMonitor object is set to YES. If the ResyncMonitorAutoRecover object is set to YES, automaticdisk pair resynchronization is also activated.

When the resource is taken offline, the monitor is stopped for the XP RAID Manager device groupused for this resource.

CAUTION:If a resource cannot be taken offline manually, and goes into a failed state, the cluster administratormust disable monitoring of the device group for this resource. To avoid data corruption, this task mustbe part of the recovery procedure when XP Cluster Extension is deployed in an MSCS/Failover ClusterService environment. See “Stopping the pair/resync monitor” on page 115.You must ensure that the pair/resync monitor does not monitor and resynchronize the disk pair (devicegroup) from both disk array sites.

To use the pair/resync monitor with an XP Cluster Extension resource:



3. Click Pair ResyncMon to open the Pair/Resync Monitor Properties window.

4. Select the Use pair/resync monitor check box to set the ResyncMonitor object to YES.


5. Select the Pair/resync monitor autoRecovery check box to set the ResyncMonitorAutoRecoverobject is to YES.

6. If you want to change the monitoring interval (ResyncMonitorInterval), enter a value in the Monitorinterval box.

7. Click OK to save your changes and close the Pair/Resync Monitor Properties window.

8. Click OK to save your changes and close the Properties window.

TIP:You can activate ResyncMonitor from cluster commands in the CLI. For example, if your XP ClusterExtension resource is clx_fileshare, enter the following command: cluster resourceclx_fileshare /privprop ResyncMonitor=yes.

Configuring takeover actions

Pre-executables and post-executables can be defined to be executed before or after XP Cluster Extensioninvokes its takeover functions.



3. Click Pre/Post Exec to display the Pre/Post Executable Properties window.

4. Update the PreExecScript, PostExecScript, and PostExecCheck values as needed, and then clickOK.

When configuring pre/post takeover executable paths, enter the full path to the script. If a scriptfails, the XP Cluster Extension resource will fail.

5. Click OK to save your changes and close the Properties window or Resource Configuration tool.

Configuring rolling disaster protection

To configure rolling disaster protection for an XP Cluster Extension resource:




3. Click Rolling Disaster to display the Rolling Disaster Protection window.

4. Add mirror units to each data center:

a. Click Add MU # to DC A.

b. Select mirror units from the list, and click OK.

c. Repeat the previous steps for Data Center B.

5. Update the BCResyncEnabledA, BCResyncEnabledB, BCResyncMuListA, and BCResyncMuListBvalues as needed, and then click OK.

6. Click OK to save your changes and close the Properties window.

NOTE:For more information, see “Setting objects to enable rolling disaster protection” on page 141.

Setting XP Cluster Extension resource properties using the GUI (Windows Server 2008/2008R2, Server Core, and Hyper-V Server)

This section describes the procedures for setting XP Cluster Extension resource properties with a GUI.You can perform these procedures through the resource configuration utility using the Failover ClusterManagement GUI or the standalone resource configuration tool. For instructions on using the two GUIoptions, see the following sections:• Using Failover Cluster Management to set resource properties (Windows Server 2008/2008

R2), page 55• Using the resource configuration tool to set resource properties (Server Core and Hyper-V Serv-

er), page 55


TIP:For information on managing XP Cluster Extension resources from a remote management stationthrough the MMC, see “Setting XP Cluster Extension resource properties using the MMC” on page 62.

Using Failover Cluster Management to set resource properties (Windows Server 2008/2008 R2)

For Windows Server 2008/2008 R2, use the Failover Cluster Management GUI to set resourceproperties.


2. Double-click the XP Cluster Extension resource in the summary pane to open the Properties window.


4. Make the necessary parameter changes, and then click OK.

Using the resource configuration tool to set resource properties (Server Core and Hyper-V Server)

For Server Core or Hyper-V Server, use the XP Cluster Extension resource configuration tool to setresource properties. When using the resource configuration tool:

• You must run the tool on a Server Core or Hyper-V cluster node. You cannot run the tool on a remotemanagement station.

• You cannot use the resource configuration tool to add or delete a resource.


• You can use the tool to configure multiple resources at one time. This saves time because you canswitch resources from the tool menu.

• The resource configuration tool is recommended for Hyper-V and Server Core environments becausethe properties you enter are validated. When you configure XP Cluster Extension resource propertiesfrom a remote management station or through the CLI, the properties you enter are not validated.


To use the resource configuration tool:

1. Open a command window and enter ClxXpResConfig.exe.

2. Select the resource you want to change in the XP CLX resource menu.

3. Make the necessary parameter changes, and then click OK.

Configuring XP RAID Manager instance numbers for XP RAID Manager service

To configure XP RAID Manager instance numbers from the Failover Cluster Management Parameterstab or the resource configuration tool:

1. To add an instance:

a. Click Add to open the Add RAID Manager instances window.

b. Select one or more instances and click OK.

2. To remove an instance, select it and click Remove.



Configuring the XP RAID Manager device group details

To configure XP RAID Manager device group details from the Failover Cluster Management Parameterstab or the resource configuration tool:

1. Select a value in the RM XP device group menu.


Configuring XP RAID Manager device group advanced properties

The Parameters tab of the XP Cluster Extension resource offers basic settings and is used to enterenvironment data, such as XP RAID Manager instances. The more advanced settings can be accessedthrough additional buttons in the Parameters tab.

To configure XP RAID Manager advanced properties from the Failover Cluster Management Parameterstab or the resource configuration tool:

1. Click the Advanced button to open the Advanced Fence Level Failover Behavior window. Theavailable settings in this window depend on the fence level used with your device groups.

• For the DATA fence level, you can update the Data lose mirror and DATA lose data centervalues. See DataLoseDataCenter on page 133 and DataLoseMirror on page 134 for more in-formation about these values.


• For the ASYNC fence level, you can update the ASYNC takeover timeout value. SeeAsyncTakeoverTimeout on page 130 for more information about this value.

• For the journal fence level, you can update the Journal data currency on S-VOL and ASYNCtakeover timeout values. See JournalDataCurrency on page 136 and AsyncTakeover-Timeout on page 130 for more information about these values.

2. Update the settings as needed, and then click OK to close the window.


Notes• After a device group is configured in the resource configuration utility, do not change the device

group name or swap the name with another device group name in the HORCM file. If you do this,restart the HORCM manager instance and reconfigure the XP Cluster Extension resource.

• Do not use HORCM commands to change the device group property for a device group that isconfigured for an XP Cluster Extension resource. If you do this, the changed property is not reflectedimmediately in the Parameters tab. To work around this situation, re-select the device group fromthe XP RM device group menu in the Parameters tab.

Configuring server data center assignments

To configure server data center assignments from the Failover Cluster Management Parameters tabor the resource configuration tool:

1. To remove a data center assignment, select the assignment, and then click Remove.

2. To modify a data center assignment, select the assignment, and then click Modify. Enter the newData center name in the Modify Node in Data Center List window, and then click OK.

3. To add a data center assignment, click Add. Select a host and a data center, and then click OK.



Changing failover and failback behavior

To configure failover and failback behavior from the Failover Cluster Management Parameters tab orthe resource configuration tool:

1. Click Failover/Failback to display the Failover/Failback window.

2. Update the ApplicationStartup and AutoRecover values as needed, and then click OK.


Activating the pair/resync monitor

The pair/resync monitor detects and responds to suspended XP Continuous Access links if theResyncMonitor object is set to YES. If the ResyncMonitorAutoRecover object is set to YES, automaticdisk pair resynchronization is also activated.


CAUTION:If a resource cannot be taken offline manually, and goes into a failed state, the cluster administratormust disable monitoring of the device group for this resource. To avoid data corruption, this task mustbe part of the recovery procedure when XP Cluster Extension is deployed in an MSCS environment.See “Stopping the pair/resync monitor” on page 115.You must ensure that the pair/resync monitor does not monitor and resynchronize the disk pair (devicegroup) from both disk array sites.

To activate the pair/resync monitor from the Failover Cluster Management Parameters tab or theresource configuration tool:


1. Click Pair ResyncMon to open the Pair/Resync Monitor Properties window.

2. Select the Use pair/resync monitor check box to set the ResyncMonitor object to YES.

3. Select the Pair/resync monitor autoRecovery check box to set the ResyncMonitorAutoRecoverobject is to YES.

4. If you want to change the monitoring interval (ResyncMonitorInterval), enter a value in the Monitorinterval box.

5. Click OK to save your changes and close the Pair/Resync Monitor Properties window.


TIP:You can activate ResyncMonitor from the Microsoft CLI. For example, if your XP Cluster Extensionresource is clx_fileshare, enter the following command: C:\>cluster resourceclx_fileshare /privprop ResyncMonitor=yes.

Configuring takeover actions

Pre-executables and post-executables can be defined to be executed before or after XP Cluster Extensioninvokes its takeover functions.

To configure takeover actions from the Failover Cluster Management Parameters tab or the resourceconfiguration tool:

1. Click Pre/Post Exec to display the Pre/Post Executable Properties window.


2. Update the PreExecScript, PostExecScript, and PostExecCheck values as needed, and then clickOK.

When configuring pre/post takeover executable paths, enter the full path to the script. If a scriptfails, the XP Cluster Extension resource will fail.


Configuring Rolling Disaster Protection

To configure rolling disaster protection from the Failover Cluster Management Parameters tab or theresource configuration tool:

1. Click Rolling Disaster to display the Rolling Disaster Protection window.

2. Add mirror units to each data center:

a. Click Add MU # to DC A.

b. Select mirror units from the list, and click OK.

c. Repeat the previous steps for Data Center B.

3. Update the BCResyncEnabledA, BCResyncEnabledB, BCResyncMuListA, and BCResyncMuListBvalues as needed, and then click OK.


NOTE:For more information, see “Setting objects to enable rolling disaster protection” on page 141.

Setting XP Cluster Extension resource properties using the MMCIf you are using Server Core or Hyper-V Server, you can manage a cluster remotely by using the MMCto run Failover Cluster Management.


NOTE:When you configure XP Cluster Extension resource properties from a remote management stationthrough the MMC, which uses the standard Microsoft Properties tab, the properties you enter are notvalidated, so you must enter the property values accurately, and verify them against the XP ClusterExtension documentation.

When you use this option, you will see the default Microsoft properties page instead of the XP ClusterExtension Parameters tab. For more information about using the MMC, see “Remote management ofXP Cluster Extension resources in a cluster (Windows Server 2008/2008 R2)” on page 73 and yourMicrosoft documentation.

Setting XP Cluster Extension resource properties using the CLIThe cluster commands in this section can be used with Windows Server 2003, Windows Server2008/2008 R2, Server Core, or Hyper-V Server.

The MSCS default properties for a resource can be changed using the following command: clusterresource resource_name /privprop [object_name=value|"value1 value2 ..."].

NOTE:When you configure XP Cluster Extension resource properties using the CLI, the properties you enterare not validated, so you must enter the property values accurately, and verify them against the XPCluster Extension documentation.

You can display all attributes of the XP Cluster Extension resource clx_fileshare with the followingcommand:

cluster resource clx_fileshare /privprop


The following example changes the FenceLevel property of the XP Cluster Extension resourceclx_fileshare:

C:\>cluster resource clx_fileshare /privprop FenceLevel=data

The following example changes the XP RAID Manager instance used for the XP Cluster Extensionresource clx_fileshare from 10 to 99, and then adds instance 22 to provide redundancy:

C:\>cluster resource clx_fileshare /privprop RaidManagerInstances="99 22"

The following example changes the name of XP Cluster Extension resource XP Cluster Extensionresource1 to XP Cluster Extension resource2:

cluster resource "XP Cluster Extension resource1" /ren:"XP Cluster Extension resource2"

Setting XP Cluster Extension properties using a UCFYou can use a UCF to configure certain XP Cluster Extension properties for Windows.

Properties that you can configure in a UCF include:

• LogLevel• ClusterNotifyCheckTime• ClusterNotifyWaitTime• LocalDCLMForNonPAIRDG• StatusRefreshInterval

IMPORTANT:If you plan to use the default values for these properties, no UCF is required.

To configure properties using a UCF:

1. Take the XP Cluster Extension resource offline.

2. Open the sample UCF.cfg file located in %HPCLX_PATH%\sample.

3. Update the file with the property values you want to use.

For more information on the available properties, see Chapter 8 on page 123.

4. Save the file and copy it to the following directory on all cluster nodes: %HPCLX_PATH%\conf.

5. Bring the XP Cluster Extension resource online.

Adding dependencies on an XP Cluster Extension resourceXP Cluster Extension Software must be the first resource in the resource chain of a MSCS service orapplication. All resources that depend on the disk resource, such as a file share, and all disk resources(physical disks), must be configured for dependency on the XP Cluster Extension resource.

When adding dependencies:

• For Windows Server 2003, use the Cluster Administrator GUI or cluster commands in the CLI.


• For Windows Server 2008/2008 R2, use the Failover Cluster Management GUI, cluster commandsin the CLI, or the MMC for remote management.

• For Server Core or Hyper-V Server, use cluster commands in the CLI or the MMC.

Adding dependencies using Cluster Administrator (Windows Server 2003)1. Open Cluster Administrator.

2. Select the Resources folder in the console-tree.

3. Double-click the disk resource you want to edit.

4. Click the Dependencies tab, then click Modify.

5. Add the XP Cluster Extension resource to the Dependencies of the disk resource.

6. Click OK to finish your modifications.

Adding dependencies using Failover Cluster Management (Windows Server2008/2008 R2)

You can add dependencies with the GUI on a local node or by using the MMC to run the FailoverCluster Management application.


2. Select a service or application that has an XP Cluster Extension resource.

3. Double-click a disk in the summary pane.

4. Click the Dependencies tab, and then click Insert.

5. Select the XP Cluster Extension resource in the Resource menu.

6. Click OK to add the selected dependency.


Adding dependencies using the CLIThe cluster commands in this section can be used with Windows Server 2003, Windows Server2008/2008 R2, Server Core, or Hyper-V Server.

To add a dependency on an XP Cluster Extension resource using the CLI, use the following command:

cluster resource physical_disk_resource /adddependency:Cluster_Extension_XP_resource

The following command adds a dependency on the XP Cluster Extension clx_fileshare resourceto the physical disk resource Disk_32b_00b:

cluster resource Disk_32b_00b /adddependency:clx_fileshare

Disaster-tolerant configuration example using a file shareThe following example describes a configuration in which:

• Your environment consists of four systems (host1_DCA, host2_DCA, host3_DCB andhost4_DCB).

• Your environment includes two XP disk arrays with serial numbers 35014 and 35013.• You have configured clxfileshare as device group in the XP RAID Manager c:\windows\

horcm101.conf file and in the c:\windows\horcm102.conf file.• A pre-executable clxpre.exe will be invoked by XP Cluster Extension.• You use the default failover behavior for the cluster group.• The resource CLX_FILESHARE is part of the service group CLX_SHARE and must be brought online

before the physical disk resource Disk_32b_00b.

Figure 4 on page 67 illustrates failover options and shows a second cluster group CLX_IIS. Figure5 on page 67 is a sample CLX_FILESHARE resource screen shot, and Figure 6 on page 68 is anexample of the resource tree for service or application CLX_SHARE.


Figure 4 Service or application example (quorum service control disks not shown).

Figure 5 CLX_FILESHARE resource sample.


Figure 6 XP Cluster Extension resource tree for CLX_SHARE.

XP Cluster Extension is configured as a single resource to enable read/write access to the physicaldisk resource used for the CLX_SHARE cluster group. The physical disk resource depends on the XPCluster Extension resource and can be brought online only when the XP Cluster Extension resource isalready online. Independent of this resource tree, the network card will be configured with theCLX_SHARE service or application's (resource group's) IP address and network name.

If all those resources have been brought online, the file share can be started.

To configure the XP Cluster Extension resource according to the configuration in Figure 6 on page 68:

1. Log in to the host3_DCB system with the Administrator account.

2. Create the file share service or application with all previously mentioned resources and itsdependencies, except the XP Cluster Extension resource on host3_DCB.

3. Create a new resource of type XP Cluster Extension and add systems host2_DCA, host3_DCB,and host4_DCB to its possible owners.

4. Change the restart behavior of the XP Cluster Extension resource so that the resource can berestarted and so that the restart affects the group. Set the number of restarts to 0.

5. Edit the resource properties, including the following information:• XP RAID Manager instances• XP RAID Manager device group details• Server data center assignments

6. Click the Pre/Post Exec button and add clxpre.exe with its full path. (The clxpre.exeprogram is an example. It is not included in the XP Cluster Extension product.)

7. Add a dependency on the XP Cluster Extension resource CLX_FILESHARE to the physical diskresource Disk_32b_00b.

8. Check the cluster service, group, and resource settings with the following commands:

C:\>cluster group CLX_SHARE /prop

C:\>cluster resource CLX_FILESHARE /prop


9. For Windows Server 2003 only: Set the XP Cluster Extension resource property RestartAction tozero (0), or check the Do not restart check box in the resource's Advanced tab window, andthen use the following commands to check if the value has changed. For example:

C:\>cluster resource CLX_FILESHARE /prop RestartAction=0


If you are using the CLI to set resource properties, the equivalent command is cluster resCLX_FILESHARE /prop RestartAction=0.

10. For Windows Server 2008/2008 R2 only: Enable the XP Cluster Extension resource property Ifrestart is unsuccessful, fail over all resources in this service or application. This value is set in thePolicies tab in the Failover Cluster Management Properties window.

If you are using the CLI to set resource properties, the equivalent command is cluster resCLX_FILESHARE /prop RestartAction=0.

11. Bring the service or application online on host3_DCB by using the Failover Cluster ManagementGUI, Cluster Administrator GUI, or the following cluster command in the CLI:

C:\>cluster group CLX_SHARE /online:host3_DCB

12. Verify that the XP Cluster Extension resource and all other CLX_SHARE application resources arebrought online:

C:\>cluster group CLX_SHARE

13. Take the service or application offline, and verify that all resources are stopped:

C:\>cluster group CLX_SHARE /offline


14. Bring the service or application online again and verify that all resources are available:

C:\>cluster group CLX_SHARE /online:host3_DCB


15. Check the cluster service settings of system host4_DCB, and the group and resource settings.

16. Move the service or application to system host4_DCB and verify that all resources are available:

C:\>cluster group CLX_SHARE /moveto:host4_DCB


17. Check the cluster service settings of system host2_DCA, and the group and resource settings.

18. Move the service or application to system host2_DCA and verify that all resources are available:

C:\>cluster group CLX_SHARE /moveto:host2_DCA


19. Check the cluster service settings of system host1_DCA, and the group and resource settings.

20. Take the service or application offline, and verify that all resources are stopped:

C:\>cluster group CLX_SHARE /offline



21. Change the XP Cluster Extension resource to be able to restart on another system:

C:\>cluster resource CLX_FILESHARE /prop RestartAction=2


Managing XP Cluster Extension resourcesYou can manage resources by bringing them online and offline, or by deleting them.

Bringing a resource onlineResources are usually brought online automatically when the service or application is brought online.You might need to move the service or application to the node where you want to bring the resourceonline.

When bringing resources online:

• For Windows Server 2008/2008 R2, use the GUI, MMC, or CLI.• For Server Core or Hyper-V Server, use the CLI or the MMC.• For Windows Server 2003, use the GUI or CLI.

For more information on using this command, see your Microsoft documentation.

Taking a resource offlineResources are usually taken offline automatically when the service or application is taken offline.Taking a resource offline causes resources that depend on that resource to go offline.

When taking resources offline:

• For Windows Server 2008/2008 R2, use the GUI, MMC, or CLI.• For Server Core or Hyper-V Server, use the CLI or the MMC.• For Windows Server 2003, use the GUI or CLI.

For more information on using this command, see your Microsoft documentation.

Deleting a resourceDeleting a running resource causes the resource and its dependents to go offline.

CAUTION:Deleting a running XP Cluster Extension resource does not remove the resource_name.onlinefile and does not remove the device group from the list of monitored device groups if the pair/resyncmonitor is used to monitor the XP Continuous Access Software link. Therefore, the device group mustbe deleted from the list of monitored device groups manually using the clxchkmon command afterdeleting the XP Cluster Extension resource. See “Stopping the pair/resync monitor” on page 115.


CAUTION:Failure to delete the monitored device group from the list of monitored device groups can cause datacorruption if the ResyncMonitorAutoRecover attribute is set to YES.

When deleting resources:

• For Windows Server 2008/2008 R2, use the GUI or CLI.• For Server Core or Hyper-V Server, use the CLI or the MMC.• For Windows Server 2003, use the GUI or CLI.

For more information on deleting resources, see your Microsoft documentation.

Using Hyper-V Live Migration with XP Cluster ExtensionLive migration is a managed failover of VM resources. Live migration should be performed when allof the solution constituents are in a healthy state, all the servers and systems are running, and all thelinks are up. Ensure that the underlying infrastructure is in a healthy state before performing livemigration.

XP Cluster Extension has the capability of discovering unfavorable storage-level conditions forperforming live migration. In response to these conditions, XP Cluster Extension will stop or cancelthe live migration process and inform the user. This is accomplished with no VM downtime. Forexample, if live migration is initiated while VM data residing on the storage arrays is still mergingand not in sync, XP Cluster Extension will proactively cancel the live migration and inform the user towait until the merge is in progress. Without this feature, live migration might fail or the VM mightcome online in the remote data center with inconsistent data.

The XP Cluster Extension StatusRefreshInterval property, which you can configure in a UCF for eachapplication, specifies the time interval between consecutive array status gathering operations beforethe live migration to the target cluster node occurs. By adjusting this property, you can increase theprobability of getting the correct XP array status to ensure a successful live migration. The defaultStatusRefreshInterval value is 300 seconds. For more information about configuring this property, see“Setting XP Cluster Extension properties using a UCF” on page 64.

XP Cluster Extension cancels live migration operations within the local data center when the devicegroup is not in PAIR status. Use the LocalDCLMForNonPAIRDG property, which can be configured ina UCF for each application, to change the setting to allow live migration to occur within the localdata center even if the device group is not in PAIR status.

Hyper-V Live Migration is supported with XP Cluster Extension for Windows Server 2008 R2 usingonly the synchronous fence levels DATA and NEVER. The asynchronous and journal fence levels arenot supported.

Using Hyper-V Live Migration with Cluster Shared Volumes is not supported with XP Cluster Extension.

TIP:For more information about using Hyper-V Live Migration with XP, see the white paper Live Migrationacross data centers and disaster tolerant virtualization architecture with HP StorageWorks ClusterExtension and Microsoft Hyper-VTM on the white papers website: www.hp.com/storage/whitepapers.


www.hp.com/storage/whitepapers

Timing considerations for MSCSXP Cluster Extension gives priority to XP disk array operations over cluster software operations. If XPCluster Extension invokes a disk pair resynchronization operation or gathers information about theremote XP disk array, XP Cluster Extension waits until the requested status information is reported.This ensures the priority of data integrity over cluster software failover processes. This behavior canlead to failed XP Cluster Extension resources as described below:

• XP Cluster Extension uses XP RAID Manager instances to communicate with the remote XP diskarray. Depending on the setting of the XP RAID Manager instance timeout parameter and thenumber of remote instances, the online operation could time out. This can occur if the local XPRAID Manager instance cannot reach the remote XP RAID Manager instance.

• XP Cluster Extension tries to resynchronize disk pairs and waits until the XP RAID Manager devicegroup is in PAIR state if the ApplicationStartup resource property is set to RESYNCWAIT. XP RAIDManager and the XP firmware fully support delta resynchronization; however, the delta betweenthe primary and secondary disks could be large enough for the copy process to exceed the resourcePendingTimeout value.

• The ResyncWaitTimeout object can cause XP Cluster Extension resources to fail if its value ishigher than the resource PendingTimeout value.

• If running in fence level ASYNC, the default value of AsyncTakeoverTimeout can cause the resourceto fail because its value exceeds the resource PendingTimeout value. The takeover process forfence level ASYNC can take much longer when slow communications links are in place.To prevent takeover commands from being terminated by the resource PendingTimeout, measurethe time required to copy the installed XP disk array cache and adjust the resource PendingTimeoutvalue. When measuring the copy time, measure only the slowest link used for XP Continuous AccessSoftware. This ensures that the XP disk array cache can be transferred from the remote XP diskarray, even in the event of a single surviving replication link between the XP disk arrays.

In general, because the failover environment is dispersed into two (or more) data centers, the failovertime cannot be expected to be the same as that in a single data center with a single shared diskdevice. Therefore, the following values of the XP Cluster Extension resource and the service andapplication using that resource must be adjusted, based on failover tests performed to verify the properconfiguration setup: FailoverPeriod, RestartPeriod, PendingTimeout, LookAlive, and IsAlive.

In addition, the service or application's FailoverPeriod value must be higher than the resource’sRestartPeriod value, and both must be higher than the resource’s PendingTimeout value.

MSCS provides two parameters to adjust state change recognition/resolution:

• IsAlive• LookAlive

XP Cluster Extension automatically calls the IsAlive function whenever the cluster service calls theLookAlive function. Therefore, both functions must be set to the same value.

Bouncing service or applicationXP Cluster Extension will alternate (start and fail) between local nodes if the ApplicationStartup propertyhas been set to FASTFAILBACK and no remote system is available until the service or applicationrestart limit has been reached. For more information, see ApplicationStartup on page 129.

The FastFailbackEnabled property is not used by the XP Cluster Extension integration with MSCS.


AdministrationXP Cluster Extension administration includes remote management of resources and monitoring ofsystem resources and logs.

Remote management of XP Cluster Extension resources in a cluster (WindowsServer 2008/2008 R2)

You can use the MMC with Failover Cluster Management to manage clusters and configure XP ClusterExtension resources. Note the following when configuring XP Cluster Extension resources by usingthe MMC from a remote management station:

• When you use the MMC to remotely configure XP Cluster Extension resource properties in aServer Core or Hyper-V Server cluster node, the Failover Cluster Management GUI on the remotemanagement station displays the standard Microsoft Properties tab instead of the customized XPCluster Extension Parameters tab. For more information about the Properties tab, see “Setting XPCluster Extension resource properties using the MMC” on page 62.

• When you install XP Cluster Extension into a Windows Server 2008/2008 R2 environment, theresource extension DLL is registered by default, which prevents you from configuring an XP ClusterExtension resource from a remote management station. If you need to remotely configure an XPCluster Extension resource in a Windows Server 2008/2008 R2-based cluster, unregister clxm-scsex.dll from the cluster node, which allows you to configure the XP Cluster Extension resourceusing the standard Microsoft Properties tab. Use the command cluster /UNREGADMINEXT:clxm-scsex.dll to unregister the DLL.

CAUTION:Configuring XP Cluster Extension resources using the MMC from a remote management stationis supported using only the standard Microsoft Properties tab. Do not try to use the customizedXP Cluster Extension Parameters tab for this purpose.

• If you see the customized XP Cluster Extension Parameters tab when you try to configure an XPCluster Extension resource from a remote management station using the MMC, you must unregisterclxmscsex.dll from the cluster node. Use the command cluster /UNREGADMINEXT:clxm-scsex.dll to unregister the DLL. Unregistering the DLL allows you to configure the resource usingthe standard Microsoft Properties tab. This situation might occur if you have a cluster with bothServer Core or Hyper-V Server and Windows Server 2008/2008 R2 cluster nodes.

• When you configure XP Cluster Extension resource properties from a remote management stationthrough the MMC, which uses the standard Microsoft Properties tab, the properties you enter arenot validated, so you must enter the property values accurately, and verify them against the XPCluster Extension documentation.

Remote management of XP Cluster Extension resources in a cluster (WindowsServer 2003)

You can use Cluster Administrator to manage clusters and configure XP Cluster Extension resources.When using Cluster Administrator to configure XP Cluster Extension resources from a remotemanagement station, note the following:

• In a Windows Server 2003 cluster with XP Cluster Extension installed, when you try to use ClusterAdministrator to configure an XP Cluster Extension resource from a remote management station,


you will see the customized XP Cluster Extension Parameters tab. The customized tab is displayedbecause the resource extension DLL is registered by default on Windows Server 2003 clusternodes, which prevents you from configuring the XP Cluster Extension resource from a remotemanagement station.If you need to configure an XP Cluster Extension resource remotely for a Windows Server 2003-based cluster, unregister clxmscsex.dll from the cluster node, which allows you to remotelyconfigure an XP Cluster Extension resource using the standard Microsoft Properties tab. Use thecommand cluster /UNREGADMINEXT:clxmscsex.dll to unregister the DLL.

NOTE:Configuring XP Cluster Extension resources by using Cluster Administrator from a remotemanagement station is supported using only the standard Microsoft Properties tab. Do not try touse the customized XP Cluster Extension Parameters tab for this purpose.

• When you configure XP Cluster Extension resource properties from a remote management stationthrough the Cluster Administrator, which uses the standard Microsoft Properties tab, the propertiesyou enter are not validated, so you must enter the property values accurately, and verify themagainst the XP Cluster Extension documentation.

System resourcesMonitor the system resources on a regular basis as part of Windows administration. If any systemresource usage by the cluster service is reaching maximum levels, stop and then restart the clusterservice. This action automatically fails over the resources and resets system resources. See the MSCSdocumentation for information about how to stop a cluster service.

An alternate method is to manually move all resources to another node in the cluster before stoppingthe cluster service. After all resources are successfully moved to another node, stop and then restartthe cluster service; then, manually move back all resources.

LogsIf the XP Cluster Extension log files need to be cleared and reset (for example, to reduce disk spaceusage), you can delete the files. XP Cluster Extension automatically creates new log files.

TIP:Archive the log files before deleting them.

Hyper-V Live Migration log entriesIn the XP Cluster Extension log file (clxmscs.log), live migration messages include the prefix CLX_LMto help you differentiate live migration issues from XP Cluster Extension log messages.

For example:

[10/12/09 20:13:02][2136][CLX_LM: CLXVMDISK04-App01][INFO] CLX detectedthat Live Migration for VM "Virtual Machine VM04" has begun.

[10/12/09 20:13:02][2136][CLX_LM: CLXVMDISK04-App01][INFO] CLX startedgathering VM "Virtual Machine VM04" specific storage information.


4 Configuring XP Cluster Extension forSolaris

HP StorageWorks XP Cluster Extension for VCS provides a resource agent to VCS. This allows clusteradministrators to configure the XP disk array-specific failover behavior as easily as any other resourcein VCS. XP Cluster Extension objects are configured as attributes of a resource in VCS.

For information about how to install XP Cluster Extension, see the HP StorageWorks XP Cluster Extensioninstallation guide.

For supported configurations, see the HP SPOCK website: http://www.hp.com/storage/spock.

Configuration of the XP Cluster Extension agentThe XP Cluster Extension agent is preconfigured to fit most of your cluster configurations. It comes witha sample configuration that can be modified to fit your VCS and disk array environments. Beforeconfiguring the XP Cluster Extension agent, review VCS resource attributes of the XP Cluster Extensionresource type.

Disaster tolerant configuration example using a web serverThe following example describes a configuration in which:

• There are four systems: sunrise, dawn, sunset, and dusk.• There are two disk arrays with serial numbers 35014 and 35013.• web is configured as a device group in the XP RAID Manager /etc/horcm11.conf file.• A pre-executable web_pre.sh and a post-executable web_post.sh will be invoked by XP

Cluster Extension.• You are using the default failover behavior for the service group.

The resource clx_web is part of the service group CLX_WEB_SERVER, and must be brought onlinebefore the DiskGroup resources webdg and httpddg. The XP RAID Manager device group webincludes all disks for the VxVM disk groups webdg and httpddg in the following example. Figure7 on page 76 illustrates failover options and shows a second service group, CLX_ORACLE.



Example of the clx_web resource

ClusterExtensionXP clx_web (XPSerialNumbers = { 35014, 35013 }RaidManagerInstances = { 11 }DeviceGroup = webPreExecScript = "/etc/opt/hpclx/web_pre.sh"PostExecScript = "/etc/opt/hpclx/web_post.sh"DC_A_Hosts = { sunrise, dawn }DC_B_Hosts = { sunset, dusk })

Figure 7 VERITAS Cluster Service configuration example.

Figure 8 on page 77 shows an example resource graph of the CLX_WEB_SERVER service group.

XP Cluster Extension is configured as a single resource to enable read/write access to the disk groupsused for the web server service group. The DiskGroup resources depend on the XP Cluster Extensionresource, and the Mount resources can be brought online only when the DiskGroup resources andthe XP Cluster Extension resource are already online. Independent of this resource tree, the networkcard will be configured with the web server service group IP address.

When all these resources have been brought online, the web server can be started.

Configuring XP Cluster Extension for Solaris76

Figure 8 Sample resource graph of the CLX_WEB_SERVER service group.

Configuring the XP Cluster Extension agent according to Figure 71. Log in to system sunrise as root.

2. Create the XP Cluster Extension resource (for example, clx_web) in the $VCS_CONF/config/main.cf file, using the previous example.

3. Link the new resource as a child resource to all disk resources in the service group.

4. Edit the attributes in the file $VCS_CONF/config/main.cf to configure your XP Cluster Extensionresource. Enter the XP RAID Manager instances, the XP RAID Manager device group, the XPserial numbers, DC_A_Hosts, and the DC_B_Hosts.

5. Verify the syntax of the file $VCS_CONF/config/main.cf:

#hacf –verify $VCS_CONF/config

6. Start the VCS engine (had) on sunrise:

#hastart

7. Verify that the XP Cluster Extension and all other web server service group resources are broughtonline:

#hagrp -display

8. Take the service group offline, and verify that all resources are stopped:

#hagrp –offline CLX_WEB_SERVER –sys sunrise

#hagrp –display

9. Bring the service group online again, and verify that all resources are available:

#hagrp –online CLX_WEB_SERVER –sys sunrise

#hagrp –display


10. Start the VCS engine on dawn:

#hastart

11. Start the VCS engine on sunset and dusk, and switch the web server service group to dawnand later to sunset and dusk. Before you switch the service group to the remote data center,make sure that the XP Continuous Access Software links are configured for bidirectional mirroringand that XP RAID Manager instances include the device group, configured for the web serverservice group. To switch the web server service group, enter:

#hagrp –switch CLX_WEB_SERVER –to system_name

12. Verify that all XP Cluster Extension and web server service group resources are brought online:

#hagrp -display














Including the XP Cluster Extension resource typeSee “Importing the XP Cluster Extension resource types configuration file” in the HP StorageWorksXP Cluster Extension Software Installation Guide.

Configuring the XP Cluster Extension resourceFor VCS, you can configure an XP Cluster Extension resource using either the VCS CLI or the VCSCluster Manager GUI. The XP Cluster Extension resource gathers all necessary information about theservice group and the XP disk arrays if the XP Cluster Extension resource is brought online. Considerthe following:

• If you use the default values for XP Cluster Extension COMMON objects, no user configurationfile is required.

• If configured, a pair/resync monitor is started to monitor the XP Cluster Extension resource. If theresource is not configured to use the pair/resync monitor, a file is created in the directory specifiedby the ApplicationDir attribute: resource_name.online.If the resource is taken offline, the file is removed or the device group associated with the servicegroup is removed from the pair/resync monitor list. If the device group is the last monitored diskpair, the monitor is stopped also.

• The resource type definition file, ClusterExtensionXPTypes.cf, must be included in the VCSconfiguration file main.cf. The XP Cluster Extension resource type definition is preconfigured forthe most typical cluster configurations. It comes with a sample configuration that can be modifiedto fit your VCS and disk array environment.

XP Cluster Extension resource typesBefore configuring the objects in the user configuration file, review the XP Cluster Extension objectsdescribed in Chapter 8 on page 123.

Resource type definitionTo configure an XP Cluster Extension resource, use the following object definitions:

type ClusterExtensionXP (static int MonitorInterval = 15static int OfflineMonitorInterval = 3600static int OnlineTimeout = 100static str ArgList[] = { ApplicationDir, DeviceGroup,

ResyncMonitorInterval, ResyncMonitor,ResyncMonitorAutoRecover,RaidManagerInstances,XPSerialNumbers,FenceLevel,DataLoseMirror, DataLoseDataCenter,AsyncTakeoverTimeout, JournalDataCurrency,AutoRecover, ApplicationStartup,ResyncWaitTimeout,FastFailbackEnabled, PostExecCheck,PreExecScript, PostExecScript,DC_A_Hosts, DC_B_Hosts,BCMuListA, BCMuListB, BCResyncMuListA,BCResyncMuListB,


BCEnabledA, BCEnabledBBCResyncEnabledA, BCResyncEnabledB }

str ApplicationDir = "/etc/opt/hpclx/"str XPSerialNumbers[]str RaidManagerInstances[]str DeviceGroupstr DC_A_Hosts[]str DC_B_Hosts[]str FenceLevel = neverstr DataLoseMirror = nostr DataLoseDataCenter = yesstr JournalDataCurrency = yes

int AsyncTakeoverTimeout = 1800str ApplicationStartup = fastfailbackint ResyncWaitTimeout = 300str FastFailbackEnabled = yesstr AutoRecover = nostr ResyncMonitor = nostr ResyncMonitorAutoRecover = nostr ResyncMonitorInterval = 60str PreExecScriptstr PostExecScriptstr PostExecCheck = nostr BCMuListA[]str BCMuListB[]str BCResyncMuListA[]str BCResyncMuListB[]str BCEnabledA = nostr BCEnabledB = nostr BCResyncEnabledA = nostr BCResyncEnabledB = no)

Adding an XP Cluster Extension resourceThese procedures add a resource to an existing service group.

Adding an XP Cluster Extension resource using the VCS CLI

Syntaxhares –add resource_name ClusterExtensionXP service_group

The following example adds an XP Cluster Extension resource called clx_web to service groupCLX_WEB_SERVER:

hares -add clx_web ClusterExtensionXP CLX_WEB_SERVER

Adding an XP Cluster Extension resource using the VCS Cluster Manager GUI1. Open the Cluster Explorer.


2. Select the service group, right-click, and choose Add Resource; or, click Add Resource in theCluster Explorer toolbar.

3. Enter the resource name in the Resource name box.

4. Select ClusterExtensionXP from the Resource Type list.


5. To change attribute values of the new XP Cluster Extension resource, click the button in the Editcolumn of the value you want to change, and modify the values as desired in the Edit Attributewindow.

6. Select the Critical and Enabled boxes in the Add Resource window.

7. Click OK.


Changing XP Cluster Extension attributesXP Cluster Extension resource attributes can be changed after the configuration has been write-enabled.

To change attribute values of the XP Cluster Extension resource the resource must be taken offline.Changing an attribute of the XP Cluster Extension resource while the resource is running is notsupported.

Changing an attribute value using the VCS CLI

Syntaxhares –modify ClusterExtensionXP_resource [ –add | –update ] attributevalue

The following example changes an XP Cluster Extension resource called clx_web to change thedefault FenceLevel attribute:

# hares -modify clx_web FenceLevel data

The following commands change the XP RAID Manager instance used for the XP Cluster Extensionresource clx_web, and then add an additional instance to provide redundancy:

# hares -display clx_web -attribute RaidManagerInstances# hares -modify clx_web RaidManagerInstances -update 90# hares -modify clx_web RaidManagerInstances -add 22

The following example displays all attributes of the XP Cluster Extension resource clx_web:

# hares -display clx_web

Changing an attribute value using the VCS Cluster Manager GUI1. Open the Cluster Explorer.

2. Click the resource name.


3. Click the Properties tab in the View Panel.

4. Click Edit for the attribute you want to change.

5. Enter changes to the attribute value. For nonscalar attributes, use the + and x buttons to add orremove elements. Do not change the attribute's scope to local; all XP Cluster Extension attributesare global in scope.

6. Click OK to accept the change.

Linking an XP Cluster Extension resourceXP Cluster Extension must be the first resource in the resource chain of a VCS service group.

All resources depending on the disk resource (for example, Mount), including the disk resources(DiskGroup, Disk, DiskReservation), must be parent resources to the XP Cluster Extension resource.

CAUTION:XP Cluster Extension does not support ServiceGroupHB resources in XP Continuous Access Softwareconfigurations because of the read/write mode differences between the primary and secondary diskin an XP disk array.

Linking other resources to the XP Cluster Extension resource

Syntaxhares –link disk_group_resource ClusterExtensionXP_resource

The following example links the disk group resource netscapedg to the clx_web_server resource:

# hares -link netscapedg clx_web_server


Linking other resources using the VCS Cluster Manager GUI1. Open the Cluster Explorer

2. Click the Resources View tab on the View Panel.

3. Click the resource icon of the resource that is to be the parent resource.

4. Move the yellow line to the resource that is to be the (child) XP Cluster Extension resource andclick to link the child to the parent resource selected in step 3.

5. Click YES in the dialog box to confirm the dependency.

Bringing an XP Cluster Extension resource onlineResources are usually brought online automatically when the service group is brought online.

To bring a resource group online manually, the service group must be enabled on the system andauto-enabled in the cluster. Finally, the resource must be enabled.

Enabling and bringing an XP Cluster Extension resource online using the CLI

Syntaxhares –modify ClusterExtensionXP_resource Enabled 1

hares –online ClusterExtensionXP_resource –sys system_name

The following example enables and brings the XP Cluster Extension resource clx_web online:

# hares -modify clx_web Enabled 1# hares -online clx_web -sys sunrise

Enabling and bringing an XP Cluster Extension resource online using the VCSCluster Manager GUI

1. Open the Cluster Explorer

2. Right-click the resource name.

3. Select online, and then select the system where you want to bring the resource online.


4. Click YES in the dialog box to confirm your selection.

Taking an XP Cluster Extension resource offlineResources are usually taken offline automatically when the service group is taken offline.

There are two ways to manually take a resource group offline:

• Take only the specified resource offline.• Propagate the offline request to all parent resources, which takes all parent resources offline before

the specified resource.

Syntaxhares –offline ClusterExtensionXP_resource –sys system_name

hares –offprop ClusterExtensionXP_resource –sys system_name

The following example takes the XP Cluster Extension resource clx_web offline or propagates theoffline request to all its parent resources before taking it offline:

# hares -offline clx_web -sys sunrise# hares -offprop clx_web -sys sunrise

Taking an XP Cluster Extension resource offline using the VCS Cluster ManagerGUI

1. Open the Cluster Explorer

2. Right-click the resource name.

3. Select Offline or Offline Prop, and then the select system where you want to bring the resourceoffline.

4. Click YES in the dialog box to confirm.


Deleting an XP Cluster Extension resourceThese procedures remove an XP Cluster Extension resource from an existing service group.

Deleting a resource using the VCS CLI

Syntaxhares –delete ClusterExtensionXP_resource

CAUTION:Deleting a running XP Cluster Extension resource does not remove the resource_name.onlinefile and does not remove the device group from the list of monitored device groups if the pair/resyncmonitor is used to monitor the XP Continuous Access Software link. Therefore, the device group mustbe deleted from the list of monitored device groups manually using the clxchkmon command afterdeleting the XP Cluster Extension resource. See “Stopping the pair/resync monitor” on page 115.

CAUTION:Failure to delete the monitored device group from the list of monitored device groups can cause datacorruption if the ResyncMonitorAutoRecover attribute is set to YES.

Deleting a resource using the VCS Cluster Manager GUI1. Open the Cluster Explorer.

2. Right-click the name of the resource you want to delete.

3. Select Delete.

4. Click YES in the dialog box to confirm.

The resource is deleted.

Disabling the XP Cluster Extension agentBefore you can disable the agent, you must first stop the service group or switch the service group toanother system.

To remove the XP Cluster Extension resource from the service group, you must first confirm whetherthe service group is online:

• If the service group is online, take the service group offline or switch the service group using oneof the following commands from the VCS command line:#hagrp –state service_group –sys system_name

#hagrp –switch service_group –to system_name

or#hagrp –offline service_group –sys system_name


• If the service group is offline, you can remove the XP Cluster Extension resource from the servicegroup.

To remove the resource, see “Deleting an XP Cluster Extension resource” on page 87.

Pair/resync monitor integrationThe pair/resync monitor is used to detect and react to suspended XP Continuous Access Softwarelinks. It is activated if the ResyncMonitor attribute value is set to YES. The automatic disk pairresynchronization feature is also activated if the ResyncMonitorAutoRecover attribute value is YES.


The pair/resync monitor is not started if the ResyncMonitor attribute is changed to YES while theresource is online. However, a running ResyncMonitor is disabled for the resource if the ResyncMonitorattribute is changed to NO while the resource is online.

CAUTION:If the resource group cannot be taken offline gracefully, the cluster administrator must disablemonitoring of the device group for this resource. To avoid data corruption, this task must be part ofthe recovery procedure when XP Cluster Extension is deployed in the VCS environment. See “Stoppingthe pair/resync monitor” on page 115.

VCS automatically attempts to stop the pair/resync monitor for the resource if it is running on morethan one system.

CAUTION:Ensure that the pair/resync monitor does not monitor and resynchronize the disk pair (device group)from both disk array sites.

Log-level reportingThe default setting for the pair/resync monitor's log facility is log level WARNING in the syslog. Solarisdoes not log warning messages to syslog by default.

To receive messages from the pair/resync monitor in case of XP Continuous Access Software linkfailures, add the following line to the /etc/syslog.conf file:

user.warning /var/adm/messages

This line ensures that you will be notified of XP Continuous Access Software link failures if you use thepair/resync monitor.

Timing considerations for VCSXP Cluster Extension gives priority to XP disk array operations over cluster software operations; if XPCluster Extension invokes disk pair resynchronization operations or gathers information about theremote XP disk array, XP Cluster Extension waits until the requested status information is reported.This feature prioritizes data integrity over the cluster software's failover behavior.


In some cases, this behavior could lead to failed XP Cluster Extension resources:

• XP Cluster Extension uses XP RAID Manager instances to communicate with the remote XP diskarray. Depending on the settings of the XP RAID Manager instance timeout parameter and thenumber of remote instances, the online operation could time out. This can happen if the local XPRAID Manager instance cannot reach the remote XP RAID Manager instance.

• XP Cluster Extension tries to resynchronize disk pairs and waits until the XP RAID Manager devicegroup is in PAIR state if the ApplicationStartup attribute is set to RESYNCWAIT. Depending on theXP RAID Manager version and the XP firmware version this could be a full resynchronization andmay take longer than the online timeout interval. Even if the XP RAID Manager version and the XPfirmware version allow a delta resynchronization, the delta between the primary and the secondarycould be big enough for the copy process to exceed the online timeout value.

• The ResyncWaitTimeout attribute can automatically lead to failed XP Cluster Extension resourceswhen set higher than the online timeout interval.

• If running in fence level ASYNC, the default value of the AsyncTakeoverTimeout can cause theresource to fail because its value is set beyond the resource online timeout interval. This is donebecause the takeover process for fence level ASYNC can take much longer when slow communic-ations links are in place.To prevent takeover commands from being terminated by the takeover timeout before finishing,measure the time to copy the installed XP disk array cache and adjust the resource online timeoutinterval according to the measured copy time. When measuring the copy time, measure only theslowest link used for XP Continuous Access Software. This ensures that the XP disk array cachecan be transferred from the remote XP disk array, even in the event of a single surviving replicationlink between the XP disk arrays.

Because the failover environment is dispersed into two (or more) data centers, the failover time cannotbe expected to be the same as it would be in a single data center with a single shared disk device.Therefore, adjust the online timeout values, the monitor interval of the XP Cluster Extension resource,and the service group using the XP Cluster Extension resource based on failover tests performed toverify the proper configuration setup.

Enabling/disabling service groupsBased on the XP disk array status information, XP Cluster Extension can change the cluster softwarebehavior to automatically failover (or failback) the service group faster to the remote data center.

For example, when the remote disk state is S-VOL_SSUS and the SSWS option has been set to indicatea prior takeover to the secondary disk set, if you have set the ApplicationStartup object toFASTFAILBACK, XP Cluster Extension would disable the service group for all systems in the respectivedata center(s) and VCS would transfer the service group back to the remote site rather than waitingfor a pair resynchronization to be finished before the service group could start on the local site.

This could happen only if you have not recovered the suspended disk pair after a prior takeover,where the PAIR state could not be maintained because of, for example, an XP Continuous AccessSoftware link failure.

This feature reduces application downtime because the service group (and the application) are notbrought online on each system in the service group's system list. It is moved to the first available systemlisted in the service group's system list, which is connected to the remote XP disk array, which is doneby enabling the VCS configuration file (main.cf) to be writable. The service group is disabled forall systems contained in either the DC_A_Hosts object or DC_B_Hosts object. Then, the VCSconfiguration file is saved (dumped).

This feature can be disabled. If the FastFailbackEnabled object is set to NO, the standard VCS processis used and the XP Cluster Extension resource will fail on the local system (and so would the servicegroup). VCS then tries to bring the service group online on the next system in the service group's


system list (which should be a local system). This fails because the state of the local XP disk array hasnot changed. The service group fails until the service group is brought online on a system connectedto the remote XP disk array. The service group online process takes longer and it does not access theVCS configuration file.

Restrictions for VCS with XP Cluster ExtensionThe following restrictions apply for VERITAS Cluster Server configurations when XP Cluster Extensionis used to enable failover between two XP disk arrays:

• The XP Cluster Extension resource must be the first (child) resource for all other disk resources.• Heartbeat disks cannot be used because of the P/S-VOL read/write behavior of XP Continuous

Access Software.• No service group heartbeat disks are allowed in the service group. The ServiceGroupHB resource

is not supported in XP Cluster Extension configurations because of the P/S-VOL read/write beha-vior of XP Continuous Access Software.

• Only one XP Cluster Extension resource is allowed to be configured per service group.• XP Cluster Extension must not be used with parallel service groups. If XP Cluster Extension is used

in a parallel service group, all systems configured for this service group must be connected to thesame XP disk arrays. A failover operation to the secondary XP disk array must be done manuallyonly. In such a case, all active service groups must be brought offline before any of those servicegroups can be brought online on the secondary XP disk array.

• The ApplicationDir attribute value must not be changed when the resource is online.The ApplicationDir attribute defines the location of the application_dir/resource_name.on-line file. This file is created when the resource is brought online (if ResyncMonitor attribute is setto NO). The XP Cluster Extension resource monitors the file located in the location specified byApplicationDir. Changing this attribute can cause the XP Cluster Extension resource to fail.

• The resource online timeout must be greater than the value specified for the ResyncWaitTimeoutattribute.

• The resource online timeout should be greater than twice the wait time of all remote XP RAIDManager instances times the number of remote systems. Otherwise, the XP Cluster Extension resourcefails to go online when there is a complete remote data center failure. If a post-executable is spe-cified, the resource online timeout should be greater than the number of remote systems multipliedby three times tWT.

tonline > nremote systems x 2 x tWT

where:tonline = resource online timeout

nremote systems = number of remote systems configured to run XP RAID Manager instances

tWT = wait time until remote error will be reported by local XP RAID Manager instance

Unexpected offline conditionsIn rare cases, XP Cluster Extension resources go offline after the following conditions all occur at thesame time:

• The cluster has been stopped forcibly (without taking the resources offline).• The XP Continuous Access links have failed.• A remote XP RAID Manager instance is not available due to a full network outage.


XP Cluster Extension resources go offline because the primary volume state changes from P-VOL_PAIRto P-VOL_PSUE and the secondary volume state changes from S-VOL_PAIR to EX_NORMT.

The state combination P-VOL_PSUE and EX_NORMT is not designed to be handled automaticallybecause the remote side (remote XP RAID Manager/ disk array), which has no status informationavailable, could have more current data then the primary (P-VOL_PSUE) site. In this particular case,you are required to investigate data currency and determine the appropriate action to be taken.

Bringing the XP Cluster Extension resources onlineUse one of the following procedures to bring XP Cluster Extension resources online:

1. Recover the XP Continuous Access link error and the network error, and restart XP RAID Manageron the remote site.

2. Manually resynchronize the affected disk pairs.

3. Bring the XP resources online. See “Bringing an XP Cluster Extension resource online” on page85 for instructions.

or

1. Create the forceflag resource_name.forceflag in the ApplicationDir path.

Default: /etc/opt/hpclx/

2. Bring the XP Cluster Extension resources online.

3. Depending on the attributes set for the resources, you might need to manually resynchronize theXP Continuous Access disk pairs.


5 Configuring XP Cluster Extension for Linux

XP Cluster Extension supports integration with the following cluster software for Linux:

• Configuring XP Cluster Extension with RHCS, page 95• Configuring XP Cluster Extension with SLE HA, page 102

NOTE:For a list of XP Cluster Extension versions and the cluster software versions they support, see the HPSPOCK website: http://www.hp.com/storage/spock.

XP Cluster Extension for Linux: Sample configurationFigure 9 on page 93 shows a sample configuration with RHCS or SLE HA and XP Cluster Extension.

Figure 9 Sample configuration.



The configuration example in Figure 9 on page 93 assumes the following information about the cluster:

• There are four nodes in the cluster: Host1, Host2, Host3, and Host4.• There are two XP disk arrays with serial numbers 30047 and 30053.• The device group clxwebvgs is configured in the XP RAID Manager /etc/horcm101.conf

file.• XP Cluster Extension invokes the pre-executable script clxweb_pre_takeover.sh and the post-

executable script clxweb_post_takeover.sh. These files can be an executable script or aprogram of your choice.

• For RHCS, the configuration file /etc/opt/hpclx/conf/CLXXP.config is associated withthe RHCS service CLXWEB that is configured to use the XP Cluster Extension resource agent script.RHCS invokes the resource agent script to start the CLXWEB service, which checks the disk pairstates before the volume groups vgweb and vghtdocs are activated and the web server is started.The XP RAID Manager device group clxwebvgs includes all disks for the LVM volume groupsvgweb and vghtdocs. The sample CLXXP.config file shows the contents of the configurationfile with the described failover behavior.For SLE HA, the XP Cluster Extension resource configuration file /etc/opt/hpclx/conf/CLXXP.config is associated with the SLE HA resource CLXWEB. SLE HA invokes the resourceagent script, /usr/lib/ocf/resource.d/heartbeat/CLXXP, which checks the disk pairstates before the volume groups vgweb and vghtdocs are activated and the web server is started.The XP RAID Manager device group clxwebvgs includes all disks for the LVM volume groupsvgweb and vghtdocs. The sample CLXXP.config file shows the contents of the configurationfile with the described failover behavior.Sample configuration file:

COMMONLogLevel info # values: error|info (optional)

APPLICATION CLXWEB # == service (RHCS) or resource group (SLE HA)XPSerialNumbers 30047 30053RaidManagerInstances 101DeviceGroup clxwebvgs # raid manager device groupDC_A_Hosts Host1 Host2# systems in data center ADC_B_Hosts Host3 Host4# systems in data center B

#optional parameter (only necessary if other than default)FenceLevel data # values: data | never | asyncApplicationStartup resyncwait # values: fastfailback | resyncwaitAutoRecover yes # possible values: yes | noDataLoseMirror no # possible values: yes | noDataLoseDataCenter no # possible values: yes | noPreExecScript /etc/opt/hpclx/clxweb_pre_takeover.shPostExecScript /etc/opt/hpclx/clxweb_post_takeover.sh

• The ApplicationStartup object is set to RESYNCWAIT to configure the service (RHCS) or resourcegroup (SLE HA) to wait for a pair resynchronization in the event that the service (RHCS) or resourcegroup (SLE HA) fails over to an adoptive node.

• The AutoRecover object is set to YES, which means that you use XP Cluster Extension capabilitiesto automatically recover suspended disk pair states.

• The DataLoseMirror object and DataLoseDataCenter object are set to NO, which means XP ClusterExtension does not allow you to start the service (RHCS) or resource group (SLE HA) automaticallyif the disk pair is suspended or a takeover operation leads to a suspended disk pair.

Configuring XP Cluster Extension for Linux94

• XP Cluster Extension enables read/write access to the disk groups used for the web server serviceor resource group. Activation of the volume groups depends on a successful return code from XPCluster Extension. The logical volumes can be mounted only when their volume groups are activeand XP Cluster Extension allows read/write access to the disk group. After the file system for theweb server's executables and content data is mounted and checked, the NIC is configured withthe web server's IP address.

Configuring XP Cluster Extension with RHCSXP Cluster Extension Software is integrated with the RHCS using an RHCS shared resource. XP ClusterExtension provides a resource agent script (clxxp.sh) that allows you to manage XP Cluster Extensionresources.

The executable clxxplxcs is called by XP Cluster Extension before volume group activation. Thischecks the status of the XP RAID Manager device group. If necessary, XP Cluster Extension takesappropriate actions to allow access to the disks before the cluster software accesses them.

Configuration overview1. Create an RHCS shared resource. For instructions, see “Creating an RHCS XP Cluster Extension

shared resource” on page 95.

2. Create an RHCS service using the XP Cluster Extension shared resource. For instructions, see“Creating an RHCS service using the XP Cluster Extension shared resource” on page 97.

3. Configure the pair/resync monitor if you plan to use the pair/resync feature (optional). Forinstructions, see “Configuring the pair/resync monitor” on page 108.

4. Activate the pair/resync monitor (optional). For instructions, see “Activating the pair/resyncmonitor” on page 109.

Creating an RHCS XP Cluster Extension shared resourceAfter XP Cluster Extension is installed, as described in the XP Cluster Extension Installation Guide, useConga or the Cluster Configuration Tool (system-config-cluster) to create an XP Cluster Extension sharedresource. This procedure is required as part of the initial XP Cluster Extension configuration procedure.After you complete this procedure, you do not need to repeat it when you add services. Use one ofthe following procedures:

• Using Conga to create a shared resource, page 95• Using system-config-cluster to create a shared resource, page 96

Using Conga to create a shared resourceTo create an XP cluster extension shared resource using Conga:

1. Log in to Conga.

2. Click the Cluster tab, and then select Cluster List.

3. Click the name of the cluster you want to administer.

4. Click Resources.

5. Click Add a Resource.

6. Select Script in the Select a Resource Type box.

7. Enter a name for the XP Cluster Extension shared resource in the Name box. For example: CLXXP.


8. Enter /usr/share/cluster/clxxp.sh in the Full path to script file box.

9. Click Submit.

Using system-config-cluster to create a shared resourceTo create an XP cluster extension shared resource using system-config-cluster:

1. Start system-config-cluster.

2. Click the Cluster Configuration tab.

3. Expand the Managed Resources tree.

4. Select the Resources tree.

5. Click Create a Resource to open the Resource Configuration dialog box.

6. Select Script in the Select a Resource Type box.

7. Enter CLXXP in the Name box.

8. Enter /usr/share/cluster/clxxp.sh in the File (with path) box.

9. Click OK.

10. Select File > Save to save the configuration changes.

The service configuration in /etc/cluster/cluster.conf is updated.

11. Click Send to Cluster to propagate the cluster configuration to the other cluster nodes.


Creating an RHCS service using the XP Cluster Extension shared resourceAfter you create a shared resource, create an RHCS service using the XP Cluster Extension sharedresource.

Configuration overview1. Create a service at the root of the dependency tree using the XP Cluster Extension shared resource

created in “Creating an RHCS XP Cluster Extension shared resource” on page 95. This ensuresthat the XP Cluster Extension resource is the first resource to start in a service. All other resourcesin this service should be configured as child resources to XP Cluster Extension. Use one of thefollowing procedures:

• Using Conga to create a service, page 97• Using system-config-cluster to create a service, page 98

2. Create a configuration file. For instructions, see Creating the XP Cluster Extension resourceconfiguration file, page 99.

3. Test the service configuration. For instructions, see Testing the service configuration, page 100.

Using Conga to create a serviceTo create an XP Cluster Extension service using Conga:

1. Log in to Conga.2. Click the Cluster tab, and then select Cluster List.3. Click the name of the cluster you want to administer.4. Click Services.5. Click Add a Service.

The Add a Service page appears.

6. Enter the service name in the Service name box.

IMPORTANT:The service name must match the name that is defined for the APPLICATION property in the XPCluster Extension configuration file CLXXP.config.

7. Select a failover domain. For information about the failover domain requirements, see “Failoverdomains” on page 18.

8. Select Relocate for the recovery policy.9. Click Add a resource to this service to add the XP Cluster Extension shared resource.

The Add a resource page appears.

10. Select an XP Cluster Extension shared resource from the Use an existing global resource menu.


11. Click Submit.Conga saves the configuration information and updates all of the other cluster nodes.

NOTE:To add additional resources to the service, use the Add a child feature.

Using system-config-cluster to create a serviceTo create an XP Cluster Extension service using system-config-cluster:

1. Start the Cluster Configuration tool.

2. Click the Cluster Configuration tab.

3. Expand the Managed Resources tree.

4. Select Services.

The Service properties page appears.

5. Click Create a Service.

The Add a Service dialog box appears.


6. Enter the service name in the Name box, and then click OK.

IMPORTANT:The service name must match the name that is defined for the APPLICATION property in theconfiguration file CLXXP.config.

The Service Management dialog box appears.

7. Click Add a Shared Resource to this service.

The Resource Configuration dialog box appears.

8. Select CLXXP in the Select a Resource Type menu, and then click OK.

9. To add additional resources to the service, select the XP Cluster Extension resource and clickAttach a new Private Resource to the Selection. Select the resource to be configured and providethe required resource agent parameters.

10. Click Close to close the Service Management window.

11. Select File > Save to save the configuration changes.

The service configuration in /etc/cluster/cluster.conf is updated.

12. Click Send to Cluster to propagate the cluster configuration to the other cluster nodes.

Creating the XP Cluster Extension resource configuration fileThe procedure in this section is based the sample configuration in “XP Cluster Extension for Linux:Sample configuration” on page 93. Use this procedure as a guide for configuring your environment.

1. Log in to system Host1 as root.

2. Create the configuration file CLXXP.config in the /etc/opt/hpclx/conf directory, bycopying and editing the sample file CLXXP.config provided in the /opt/hpclx/sampledirectory.

$cp /opt/hpclx/sample/CLXXP.config /etc/opt/hpclx/conf/CLXXP.config


3. In the configuration file (CLXXP.config), enter the appropriate values for:

• XPSerialNumbers• RaidManagerInstances• DeviceGroup• DC_A_Hosts• DC_B_Hosts• ResyncMonitor• FenceLevel• DataLoseMirror• DataLoseDataCenter

NOTE:For more information about these values, see Chapter 8 on page 123.

For example:

APPLICATION CLXWEBXPSerialNumbers 30060 30080RaidManagerInstances 101DeviceGroup vgnetscapeDC_A_Hosts sys1A sys2ADC_B_Hosts sys1B sys2BResyncMonitor yesFenceLevel neverDataLoseMirror yesDataLoseDataCenter yes

IMPORTANT:If you are using Device Mapper Multipath, configure the multipath_rescan.sh scriptas a PostExecScript. For more information, see “Rescanning multipath devices” on page 106.

4. Copy the updated CLXXP.config file to the other cluster nodes.

Testing the service configurationThe procedure and commands in this section are based the sample configuration in “XP ClusterExtension for Linux: Sample configuration” on page 93. Use this procedure as a guide for configuringyour environment.

1. Use the Cluster User Service Administration Utility (clusvcadm) to start the service on Host1.

#clusvcadm -e CLXWEB –m Host1

2. Verify that the service started successfully.

#clustat –s CLXWEB

3. Stop the service and verify that the service stopped successfully.

#clusvcadm –s CLXWEB Or #clusvcadm –d CLXWEB


4. Start the service on Host2.

#clusvcadm –e CLXWEB –m Host2

5. Relocate the service to a remote data center node.

a. Verify that the disks CLXWEB uses are in the PAIR state:

#export HORCMINST=101 #pairdisplay –fcx –g clxwebvgs

b. Move the service CLXWEB to Host3. Verify that service has successfully moved and startedon Host3:

#clusvcadm -r CLXWEB -m Host3

#clustat -s CLXWEB

c. Verify that the disk pairs are now in read/write mode on the remote storage system:

#pairdisplay –fcx –g clxwebvgs

d. After verifying that the service CLXWEB, including XP Cluster Extension, can be run on eachsystem in the cluster, move the service back to its primary system:

#clusvcadm -r CLXWEB -m Host1

#clustat -s CLXWEB

#pairdisplay –fcx –g clxwebvgs

Managing XP Cluster Extension services (RHCS)This section includes the instructions for starting or stopping an RHCS service.

• Starting an RHCS service, page 101• Stopping or disabling an RHCS service, page 101

Starting an RHCS serviceTo start an XP Cluster Extension service using Cluster User Service Administration (clusvcadm), enterthe following commands:

#clusvcadm –e service name

#clusvcadm -e service name -m cluster node

For instructions on starting an XP Cluster Extension service using Conga or the Cluster ConfigurationTool, see the RHCS documentation.

Stopping or disabling an RHCS serviceTo stop an XP Cluster Extension service using Cluster User Service Administration (clusvcadm), enterthe following command:

#clusvcadm –d service name

To disable an XP Cluster Extension service using Cluster User Service Administration (clusvcadm),enter the following command:

clusvcadm -d service name


NOTE:For instructions on stopping or disabling an XP Cluster Extension service using Conga or the ClusterConfiguration Tool, see the RHCS documentation.

Configuring XP Cluster Extension with SLE HAXP Cluster Extension Software is integrated with SLE HA using a configuration file and a customresource agent.

The executable clxxplxcs is called by XP Cluster Extension before volume group activation. Thischecks the status of a device group configured for use with an SLE HA agent. If necessary, XP ClusterExtension takes appropriate actions to allow access to the volume group's physical disks beforeattempting to activate and mount the logical volume on a cluster node.

Configuration overview1. Create and configure an XP Cluster Extension resource. For instructions, see “Creating and

configuring an XP Cluster Extension resource” on page 102.

2. Configure the pair/resync monitor if you plan to use the pair/resync feature (optional). Forinstructions, see “Configuring the pair/resync monitor” on page 108

3. Activate the pair/resync monitor (optional). For instructions, see “Activating the pair/resyncmonitor” on page 109.

Creating and configuring an XP Cluster Extension resourceUse the following procedure to create an XP Cluster Extension SLE HA resource:

1. Create the configuration file. For instructions, see “Creating the XP Cluster Extension resourceconfiguration file” on page 102.

2. Create an XP Cluster Extension resource using the SLE HA GUI. Use one of the followingprocedures:

• Creating an XP Cluster Extension resource for Pacemaker, page 103• Creating an XP Cluster Extension resource for Heartbeat, page 105

3. Test the configuration. For instructions, see Testing the configuration, page 106.

Creating the XP Cluster Extension resource configuration fileThe procedure in this section is based the sample configuration in “XP Cluster Extension for Linux:Sample configuration” on page 93. Use this procedure as a guide for configuring your environment.

1. Log in to system Host1 as root.

2. Create the XP Cluster Extension resource configuration file CLXXP.config in the /etc/opt/hpclx/conf directory by copying and editing the sample file CLXXP.config provided in the/opt/hpclx/sample directory.

$cp /opt/hpclx/sample/CLXXP.config /etc/opt/hpclx/conf/CLXXP.config


3. In the configuration file (CLXXP.config), enter the appropriate values for:

• XPSerialNumbers• RaidManagerInstances• DeviceGroup• DC_A_Hosts• DC_B_Hosts• ResyncMonitor• FenceLevel• DataLoseMirror• DataLoseDataCenter

NOTE:For more information about these values, see Chapter 8 on page 123.

For example:

APPLICATION CLXWEBXPSerialNumbers 30060 30080RaidManagerInstances 101DeviceGroup vgnetscapeDC_A_Hosts sys1A sys2ADC_B_Hosts sys1B sys2BResyncMonitor yesFenceLevel neverDataLoseMirror yesDataLoseDataCenter yes

IMPORTANT:If you are using Device Mapper Multipath, configure the multipath_rescan.sh scriptas a PostExecScript. For more information, see “Rescanning multipath devices” on page 106.

4. Copy the updated file to the other cluster nodes.

Creating an XP Cluster Extension resource for PacemakerThis procedure uses the Linux HA Management Client and Pacemaker. For specific instructions onusing the GUI, see the SuSE Linux Enterprise High Availability Extension documentation.

1. Start the Linux HA Management Client.

2. Select Add group from the Resources menu, and enter a group ID.

3. Add XP Cluster Extension as a primitive group's first resource.

NOTE:The resource hierarchy depends on the order in which resources are added. Always addXP Cluster Extension resources as the first resource in a group.


4. Select the following options for the XP Cluster Extension resource:

ValueName

ocfClass

heartbeatProvider

CLXXPType

5. Configure the instance attributes for the resource by selecting the app parameter. In the Valuebox, enter the APPLICATION tag name configured in the XP Cluster Extension configuration file(/etc/opt/hpclx/CLXXP.config).

6. Configure the start, stop, and monitor operations for the XP Cluster Extension resource.

7. Add additional primitive resources to the group.

For example: If LVM and File System are used as the second and third resources of the group,the Summary dialog box is similar to the following:

8. Add a resource colocation constraint between the resource group ID assigned in Step 2 and thelast resource in the group hierarchy.

9. Set location constraints for the group ID to achieve the required failover order for the group.

10. Set the operation defaults to control failover behavior. To specify that when a resource fails, theresources attempts to restart on the same node or another node in the cluster, use the followingsettings:

ValueName

nothingrequires

restarton-fail

30timeout

11. Set the migration-threshold value. This value defines the number of failures that can occur on anode before the node becomes ineligible to host the resource and the resource fails over toanother node. Set this value to 1 for XP Cluster Extension.

12. Disable automatic failback by using resource constraints and setting resource-stickiness to thelowest value compared with the other resource location constraints.


Creating an XP Cluster Extension resource for HeartbeatThis procedure uses the Linux HA Management Client and Heartbeat. For specific instructions on usingthe GUI, see the Linux HA Management Client documentation.

1. Start the Linux HA Management Client.

2. Add a group resource for XP Cluster Extension with the following settings:

ValueName

Enter a resource group ID.ID

trueOrdered

trueCollocated

The Linux HA Management Client prompts you to enter the resource type details.

3. Set the value of the app parameter to the APPLICATION tag name configured in the XP ClusterExtension resource configuration file (/etc/opt/hpclx/conf/CLXXP.config).

NOTE:The resource hierarchy depends on the order in which resources are added. Always addXP Cluster Extension resources as the first resource in a group.

4. Add an LVM resource to the group created in Step 2. Set the value of the volgrpname parameterto the name of the volume group managed by the XP Cluster Extension resource.

5. Add a Filesystem resource. Set the following values as appropriate for your environment:

• device• directory• fstype

6. Configure the start, stop, and monitor operations for the XP Cluster Extension resource and allother resources.

7. Add a location constraint to the resource group ID assigned in Step 2.

8. Add an Expression to the location constraint. For information on the settings to enter, see the SLEHA documentation.

9. Select the Expression you added in Step 8 and enter a value in the Score box. A high scoreindicates a high priority for the selected location constraint.

For example, if there are three nodes N1, N2, and N3 and if N1 has highest priority followedby N2, and then N3, create three location constraints for the same resource, and assign thescores as 1000, 500, and 200, respectively.

10. Add a resource colocation constraint between the resource group ID assigned in Step 2 and thelast resource in the group hierarchy.

11. Right-click the resource in the Linux HA Management Client GUI, and then select Start.


Testing the configurationThe procedure and commands in this section are based the sample configuration in “XP ClusterExtension for Linux: Sample configuration” on page 93. Use this procedure as a guide for configuringyour environment.

Test the configuration by migrating the resource group to the remote data center nodes:

1. Verify that the disk pairs are in read-only mode on the remote storage system.

2. In the SLE HA GUI, click Management in the left pane. Right-click the XP Cluster Extension resourceand select Start.

This will bring the resource group online on one of the cluster hosts, based the configured resourceconstraints.

3. To migrate the resource, click Management in the left pane. Right-click the XP Cluster Extensionresource, and then select Migrate Resource. Select a target node in the remote data center in theMigrate Resource dialog box, and then click OK.

4. Verify that the disk pair is in source mode on the remote storage system.

5. Migrate the resource to a node in the same data center and verify that the disk pair status hasnot changed.

Managing XP Cluster Extension services (SLE HA)To manage an XP Cluster Extension resource:

1. Click Management in the left pane of the Linux HA Management Client.

2. Right-click the XP Cluster Extension resource and select Start or Stop to automatically initiate therequested operation on the each resource in the dependency tree. For more information, see theSLE HA documentation.

Rescanning multipath devices

IMPORTANT:The information in this section applies to Device Mapper Multipath Software users only.

When a device group takeover occurs, the permission settings of the LUs in the device group changefrom read-only to read-write at the destination site. In Linux configurations with the Device MapperMultipath Software, the hosts do not dynamically detect the LU permission change. In this situation,the disks used in the XP Cluster Extension setup fail to come online when the host OS does not detectthe LU permission change. As a workaround, configure the XP Cluster Extension scriptmultipath_rescan.sh as a PostExecScript to rescan the disks before they are brought online.

Configuring the rescan scriptThe multipath rescan script is available in the directory /opt/hpclx/sample/multipath_rescan.sh. To configure the script to run as a PostExecScript:


1. Copy the multipath_rescan.sh script to the /etc/opt/hpclx/conf folder, and renamethe file as follows:

• RHCS: multipath_rescan_ServiceName.sh• SLE HA: multipath_rescan_ResourceGroupName.sh

2. Open the script file and enter the user-friendly names of all multipath devices that are in thevolume groups configured for the RHCS service or SLE HA resource group.

For instructions on finding the user-friendly name of a multipath device, see “Finding theuser-friendly name of a multipath device” on page 107.

In the following example, you specify the user-friendly names (mpathab, mpathac, and mpathad)for the variable MULTIPATH_DEVICES:

MULTIPATH_DEVICES=( mpathab mpathac mpathad )

3. Enter the multipath_rescan.sh script for the PostExecScript object in the Cluster Extensionresource configuration file.

You must specify the full path name of the multipath_rescan.sh script. For example:

Finding the user-friendly name of a multipath deviceThe multipath_rescan.sh script requires that you enter the user-friendly names of the multipathdevices. To obtain the user-friendly name of a multipath device:

1. Run the pvs command to view the multipath device names for your volume groups.

In the following example, dm-21 and dm-23 are the multipath devices for the volume groupvg01:

[root@node1 ]# pvsPV VG Fmt Attr PSize PFree/dev/dm-21 vg01 lvm2 a- 1.82G 0/dev/dm-23 vg01 lvm2 a- 1.82G 0/dev/dm-24 vg02 lvm2 a- 1.82G 0


2. Obtain the SCSI ID for a multipath device. Use the scsi_id command for SUSE Linux EnterpriseServer, and the hp_scsi_id command for Red Hat Enterprise Linux.

SUSE Linux Enterprise Server:

[root@node1 ]# scsi_id -guns /block/dm-21360060e8014424600000142460000039d

Red Hat Enterprise Linux:

[root@node1]# hp_scsi_id -guns /block/dm-14360060e8014424600000142460000039d

3. Use the multipath command to obtain the user-friendly name for the multipath device's generatedSCSI ID. In the following example, mpathq is the user-friendly name of a multipath device:

[root@node1]# multipath -ll | grep 360060e8014424600000142460000039d | awk'{print $1}' mpathq














Activating the pair/resync monitorThe pair/resync monitor detects and reacts to suspended XP Continuous Access Software links. Toactivate the pair/resync monitor, set the ResyncMonitor object to YES. To activate automatic disk pairresynchronization, set the ResyncMonitorAutoRecover object to YES.

When a RHCS service or SLE HA resource group is stopped, the pair/resync monitor is stopped forthe XP RAID Manager device group the service or resource group uses.

The pair/resync monitor does not allow online changes in the XP Cluster Extension resourceconfiguration file when the corresponding RHCS service or SLE HA resource group is online.• If the ResyncMonitor object is changed to YES while the RHCS service or SLE HA resource group

is running, the pair/resync monitor is not started.• If the ResyncMonitor object is changed to NO while the RHCS service or SLE HA resource group

is running, a running pair/resync monitor is not stopped.

CAUTION:If a RHCS service or SLE HA resource group cannot be stopped gracefully, disable monitoring of thedevice group for the service or resource group. To avoid data corruption, this task must be part ofthe recovery procedure when XP Cluster Extension is deployed in the RHCS or SLE HA environment.See “Stopping the pair/resync monitor” on page 115.Ensure that the pair/resync monitor does not monitor and resynchronize the disk pair (device group)from both disk array sites.

Timing considerationsXP Cluster Extension gives priority to XP disk array operations over cluster software operations. If XPCluster Extension invokes disk pair resynchronization or gathers information about the remote XP diskarray, XP Cluster Extension waits until the requested status information is reported. This ensures thepriority of data integrity over cluster software failover processes. This behavior can lead to failedresources, as follows:

• XP Cluster Extension uses XP RAID Manager instances to communicate with the remote XP diskarray. Depending on the setting of the XP RAID Manager instance timeout parameter and thenumber of remote instances, the service or resource group start operation can time out. This canoccur if the local XP RAID Manager instance cannot reach the remote XP RAID Manager instance.In an SLE HA environment, the timeout value defined for the start operation can be adjusted to theappropriate value to avoid this situation. In an RHCS environment, the timeout value depends onthe timeout value specified in script resource agent (/usr/share/cluster/script.sh).

• XP Cluster Extension tries to resynchronize disk pairs and waits until the XP RAID Manager devicegroup is in the PAIR state if the ApplicationStartup object is set to RESYNCWAIT. XP RAID Manager


and the XP firmware fully support delta resynchronization; however, the delta between the primaryand secondary disks can be large enough for the copy process to exceed the service or resourcegroup startup timeout value.

• The ResyncWaitTimeout object can cause the resource to fail if its value is higher than the resourcestartup timeout value.

• If running in fence-level ASYNC, the default value of AsyncTakeoverTimeout can cause the resourceto fail if its value is set beyond the recommended startup timeout value. This is done because thetakeover process for fence-level ASYNC can take longer when communication links are slow.To prevent the takeover timeout from terminating the takeover commands, measure the time requiredto copy the installed XP disk array cache and adjust the resource startup timeout interval. Whenmeasuring the copy time, measure only the slowest link used for XP Continuous Access Software.This ensures that the XP disk array cache can be transferred from the remote XP disk array, evenin the event of a single surviving replication link between the XP disk arrays.

NOTE:Because the failover environment is dispersed over two or more data centers, the failover time cannotbe expected to be the same as that of a single data center with a single shared disk device. Therefore,you must adjust the service or resource group startup timeout value and the monitor interval of theXP RAID Manager device group based on failover tests you perform to verify the proper configurationsetup.


6 XP Cluster Extension and CLI

XP Cluster Extension allows integration into almost any cluster software for commercial UNIX, Linux,and Windows operating systems. Use the XP Cluster Extension clxrun command to check properfunctionality of XP Cluster Extension prior to integration with the cluster software.

The CLI also allows integration of XP Continuous Access Software. For information on supportedplatforms, see the HP SPOCK website: http://www.hp.com/storage/spock.

Configuring the CLIUsing the XP Cluster Extension CLI requires the following configuration steps:

1. Create the XP Continuous Access Software environment.

2. Create the XP RAID Manager configuration.

3. Create and configure the user configuration file.

Creating the Continuous Access environment and configuring XP RAID ManagerHP support personnel are trained and authorized to set up XP Continuous Access Software. You can,however, configure and change XP disk pairs and XP RAID Manager instances using HP StorageWorksXP LUN Manager, HP StorageWorks Command View XP, XP Remote Web Console, Command ViewXP Advanced Edition Software, and XP RAID Manager. For detailed information on using theseprograms, see the HP StorageWorks XP LUN Manager User's Guide, HP StorageWorks CommandView XP User's Guide, HP StorageWorks XP Remote Web Console User's Guide, HP StorageWorksCommand View XP Advanced Edition Software Device Manager Web Client User's Guide, and HPStorageWorks XP RAID Manager User's Guide.

Timing considerationsXP Cluster Extension is designed to prioritize XP disk array operations over application service startupoperations. If XP Cluster Extension invokes disk pair resynchronization operations or gathers informationabout the remote XP disk array, XP Cluster Extension waits until the requested status information isreported. This prioritizes data integrity over application service startup and failover behavior.

Because the takeover timing depends on the configuration of your XP RAID Manager environmentand the settings in UCF.cfg, these considerations must be evaluated:

• XP Cluster Extension uses XP RAID Manager instances to communicate with the remote XP diskarray. Depending on the settings of the XP RAID Manager instance timeout parameter and thenumber of remote instances, the online operation could time out. This can also happen if clxrunis used in a script or called by another program and the local XP RAID Manager instance cannotreach the remote XP RAID Manager instance. See “Setting up XP RAID Manager” on page 20 formore information.

• If the ApplicationStartup attribute is set to RESYNCWAIT, XP Cluster Extension tries to resynchronizedisk pairs and waits until the XP RAID Manager device group is in PAIR state. In some versions ofXP RAID Manager and XP firmware, a full resynchronization is done. Depending on the amount



of data to be transferred, it could take hours to resynchronize. If this is the case, clxrun maytake some time to return. Do not stop clxrun; use it to check the status of the associated XP RAIDManager device groups.Even if the XP RAID Manager version and the XP firmware version allow a delta resynchronization,the amount of delta data to be transferred between the primary and the secondary could be longenough for the copy process to take a while.

• If running in fence level ASYNC, the default value of the AsyncTakeoverTimeout is set to a veryhigh number. This is done because the takeover process for fence level ASYNC can take muchlonger when slow communications links are in place; adjust this value after measuring the XPContinuous Access Software environment. See AsyncTakeoverTimeout on page 130 for more details.To prevent premature termination of the takeover commands by the takeover timeout, measure thetime to copy the installed XP family disk array cache and adjust the resource online timeout intervalaccording to the measured copy time. Use only the slowest link XP Continuous Access Softwarelink to measure the copy time. This ensures that the XP disk array cache can be transferred fromthe remote XP disk array, even in the event of a single surviving replication link between the XPfamily disk arrays.

In general, because the failover environment is dispersed into two (or more) data centers, the failovertime cannot be expected to be the same as it would be in a single data center with a single shareddisk device.

Restrictions for customized XP Cluster Extension implementationsThe following are some restrictions that apply when using the XP Cluster Extension CLI:

• The XP Cluster Extension CLI call clxrun must be invoked before the associated disk resourcesare activated.

• Associated disk resources must not be activated on any other system. If other disk resources areactivated, XP Cluster Extension may remove write-access rights for those disk devices (putting themin read-only mode).

Creating and configuring the user configuration fileThe CLI expects as an argument the name configured as the APPLICATION tag value. You do notneed to specify the SearchObject object.

The following is an example of a customized user configuration file when using clxrun:

# /etc/opt/hpclx/conf/UCF.cfg# This is the XP Cluster Extension User Configuration File (UCF.cfg).# The COMMON tag specifies the configuration for the# XP Cluster Extension core environment#COMMONLogLevel info #show disk state info in the logs# The APPLICATION tag specifies the configuration for the# XP Cluster Extension failover behaviorAPPLICATION sap #the application serviceDeviceGroup sapdg #RM dev group for the app serviceRaidManagerInstances 22 90 #RM instance number for dev groupXPSerialNumbers 34001 34005 #local and remote XP Serial NumbersDC_A_Hosts eserv1 eserv2 #data center A hostnamesDC_B_Hosts eserv3 eserv4 #data center B hostnamesFenceLevel data #FenceLevel changed from default

XP Cluster Extension and CLI112

APPLICATION netscape #the application serviceDeviceGroup netscapedg #RM dev group for the app serviceRaidManagerInstances 22 90#RM instance number for dev groupXPSerialNumbers 34001 34005 #local and remote XP Serial NumbersDC_A_Hosts eserv1 eserv2 #data center A hostnamesDC_B_Hosts eserv3 eserv4 #data center B hostnames

CLI commandsThis section describes the following CLI commands:

• clxrun, page 113• clxchkmon, page 114

clxrunCheck disk set

Descriptionclxrun can be used to manually prepare the application service's disk set before an existingapplication service start procedure is invoked. When using clxrun, the status of the associated XPRAID Manager device group is checked to ensure that access to the disk set will occur under dataconsistency and concurrency situations only.

clxrun must be invoked before the application service disk set can be activated; it is considered anonline-only program. However, the CLI features provide the same disaster tolerance features as theintegrated versions of XP Cluster Extension.

NOTE:Execution of clxrun does not start the pair/resync monitor.

Syntaxclxrun [-version] [-forceflag] app_name

Arguments

–version Displays the XP Cluster Extension version

–forceflag Forces startup

app_name The application name configured in the user configuration file (UCF.cfg)

The clxrun program expects only one parameter as the default setting. This parameter is used touniquely identify the application service in the APPLICATION section of the user configuration file.

clxrun first checks for the –forceflag option. When using clxrun, it is not necessary to createan application_name.forceflag file. This option, however, must be specified first if used.


CAUTION:The forceflag option is implemented as an emergency switch to manually activate your XP diskset. If the forceflag option has been specified, XP Cluster Extension will not check any consistencyor concurrency rules before activating the XP disk set.

Return codesclxrun exits with one of the following return codes:

0 OKApplication service can be started.

1 ERROR_GLOBALApplication service should not start on any system in either site on either disk array.

2 ERROR_DCApplication service should not start on any system in the local site on the local disk array.

3 ERROR_LOCALApplication service should not start on this system.

Example 1

# clxrun sap

Example 1 is based on the assumption that you have defined an APPLICATION tag named sap inthe UCF.cfg file and you have specified all necessary objects, including the DeviceGroup object,to map the XP disk set to the application service sap. XP Cluster Extension will check the disk setmapped to the application service sap, run the necessary takeover procedure and return one of thereturn codes mentioned in the return code table.

Example 2

# clxrun -forceflag sap

Example 2 is based on the assumption that you have defined an APPLICATION tag named sap inthe UCF.cfg file and you have specified all necessary objects, including the DeviceGroup object,to map the XP disk set to the application service sap. XP Cluster Extension will check the XP disk setmapped to the application service sap, and run the necessary takeover procedure to enable read/writeaccess to the XP disk set.

clxchkmonPair/resync monitor access program

DescriptionThe clxchkmon utility program allows starting and stopping of the resynchronization features andqueries to gather state information of the monitored device groups.


To update or remove a specific resource, use clxchkmon –n resource_name –g device_group.If –clx is not specified, the command is applied only to non-XP Cluster Extension resources.

To update all non-XP Cluster Extension resources, use clxchkmon –t. To update XP Cluster Extensionresources, use clxchkmon –clx –t.

Displaying resourcesThe following command displays all resources:

clxchkmon –show

The following command displays XP Cluster Extension resources only:

clxchkmon –clx –show

Removing resourcesThe following command removes only non-XP Cluster Extension resources:

clxchkmon –remove

The following command removes all XP Cluster Extension resources:

clxchkmon –clx –remove

Stopping the pair/resync monitorThe pair/resync monitor is stopped when all resources are removed from monitoring.

1. To check whether the pair/resync monitor is running, execute the following command:

clxchkmon –show

2. Select the application and device group combination you want to remove from the pair/resyncmonitor and remove it with the following command:

clxchkmon -n [application_name | resource_group_name | resource_name] -gdevice_group_name –remove

where

application_name|resource_group_name|resource_name is the resource name (asdefined by the APPLICATION tag in the UCF.cfg file) of the XP Cluster Extension resource andshould match the clxchkmon output. If the –clx option is not specified, the command is executedonly for non-XP Cluster Extension resources.

CAUTION:If you respond Y (yes) to remove the combination, the resource will be removed from the list of resourcesto be monitored in the pair/resync monitor. If this is not an emergency removal attempt and the XPCluster Extension resource is online, the previous procedure will lead to a failed resource, which willtake all dependent resources offline and eventually force your application offline.

Do not use this command to take your XP Cluster Extension resources offline.


Syntax

clxchkmon [-clx] [-s host name] [-n resource_name —g device_group] [[-tmonitor_interval | -autorecover mode | -remove [-force] | -show | -pid | -stopsrv| -log [error | warning | info | trace]]] [-p port number]

where:

-s hostname Specifies the name of a host.

–n resource_name Specifies the resource (application) name as used in XP ClusterExtension.

–g device_group Specifies an XP RAID Manager group name.

–t monitor_interval Specifies interval in seconds to update registered monitor resources.

–autorecover mode Specify YES to enable autorecovery, or NO to disable autorecoveryfor registered monitor resource.

–clx Executes the command only for XP Cluster Extension resources.

–remove Removes the resource from the monitor list.

–force Disables user confirmation to remove resource.

–show Displays monitored resources.

—pid Returns the process ID of the pair/resync monitor.

–stopsrv Stops the pair/resync monitor socket server.

–log Sets the log level for the pair/resync monitor.

–p port_number Specifies the port number to be used.

Return codes

clxchkmon exits with one of the following return codes:

0 Successful, or device group is in PAIR state.

1 Device group is not in PAIR state.

2 Resource/device group is not registered with the pair/resync monitor.

3 Pair/resync monitor (clxchkd) is not running.

4 Device group's pair status is pending.

10 Pair/resync monitor internal error.


11 Invalid argument to pair/resync monitor.

12 Pair/resync monitor received signal (control-c) interrupt.

13 Unknown status for device group.

14 No port number is specified in services file for clxmonitor.

16 Invalid use of the –clx option on a non-XP Cluster Extension resource or XP Cluster Extensionresource specified without the –clx option.

100 XP RAID Manager error.

Related information

For more information, see “Monitoring and resynchronizing device groups” on page 143.


7 XP Cluster Extension recovery procedures

XP disk pair statesTable 3 on page 119 provides basic XP disk pair state information. The XP disk pair state transitionprocess is complex; see the HP StorageWorks XP Continuous Access and HP StorageWorks XPContinuous Access XP Journal user guides for more information.

Table 3 XP disk pair states

DescriptionState

The primary (master) disk of a disk pairP-VOL

The secondary (slave) disk of a disk pairS-VOL

A disk with no pair affinity to any other disk(This could be shown in pairdisplay outputs for your XP Continuous Access Software disk if you accidentallyexported the XP Business Copy Software environment variable HORCC_MRCF. In such a case, the MUnumber field will not be empty.)

SMPL

The disk is either a primary disk or a secondary disk. If both (P-VOL and S-VOL) disks are in PAIR state, XPContinuous Access Software updates the secondary disk based on the primary disk.If you see only one disk in PAIR state (while the second disk is in another state), one of the following hasoccurred:• The pair affinity on only one site of the disk pair was deleted.• A takeover command has been invoked on the secondary site, while no data has been written to the

primary site and the XP Continuous Access Software link was down.• A takeover command has been invoked on the primary site with the fence level configured to DATA to

release the fenced disk, while the XP Continuous Access Software link was down. (The secondary diskwould stay in PAIR state.)

PAIR

The pair affinity has been manually suspended or a takeover operation has been invoked on the secondarysite with the fence level configured to NEVER. (In this case, the secondary disk would have the stateSSUS-SSWS.)

PSUS

The pair affinity has been manually suspended or a takeover operation has been invoked on the secondarysite. In this case, the secondary disk would have the state SSWS if you invoke pairdisplay with the –fcoption. In fence level ASYNC, the disk could also show PFUL or PFUS when using the –fc option.

SSUS

Only the secondary disk could show SSUS. With the –fc option of pairdisplay, you can check whethersomebody manually suspended the pair or a takeover command had been invoked. A prior takeovercommand is indicated by the SSWS state. In this case, the secondary disk is mandatory and aresynchronization can be done only from the S-VOL site.

SSUS - SSWS

The disk is in a failure mode. Either the XP Continuous Access Software link is down, or the disk must bereplaced.

PSUE


DescriptionState

The disk is in a failure mode. Either the XP Continuous Access Software link is down, or the disk must bereplaced. This is a special state of PSUE. If you have configured several disks into a LUSE configuration,where several LDEVS are combined to create an extended size disk and one or more disks are in an errorcondition, this state will be shown.

PDUB

This state is used to indicate that a threshold of the side file area in the XP disk array cache has been reached.This state can be seen with fence level ASYNC only. See the HP XP Continuous Access Software documentationfor more information.

PFUL

This state is used to indicate that the side file is full and the XP disk array was not able to transfer the cachecontent to the remote XP disk array for a certain time. The XP disk pair has been suspended to continueprocessing host I/O. This state can be seen with fence level ASYNC only. See the HP XP Continuous AccessSoftware documentation for more information.

PFUS

Recovery sequenceTo recover from a certain server or XP Continuous Access Software link failure:

1. Start the XP RAID Manager instances on both local and remote servers:

Linux/UNIX

export HORCMINST=instance_number

horcmstart.sh instance_number

Windows

set HORCMINST=instance_number

HORCMSTART instance_number

2. Gather general pair status information:

pairdisplay –g device_group

3. Display the pair status information after a failed swap-takeover (the S-VOL state is SSWS):

pairdisplay –g device_group –fc

4. To recover from these states, invoke the following command from the S-VOL side:

pairresync –swaps –c 15 –g device_group

If the pair needs to be used on the old primary side, the following commands must be invokedfrom the primary side:

pairresync –swapp –c 15 –g device_group

horctakeover –g device_group

XP Cluster Extension recovery procedures120

5. Display the pair status information after a P-VOL takeover (local P-VOL PSUS; remote S-VOL PAIR):

pairdisplay –g device_group –fc

To recover from these states, invoke the following command from the P-VOL side:

pairresync –c 15 –g device_group

CAUTION:The application must be shut down and the file systems unmounted before a fenced disk infence level DATA can be set in read/write mode again. After the P-VOL takeover, the filesystem must be checked before it can be mounted. Any other recovery procedure couldlead to unrecoverable file systems.

• If a horctakeover command results in S-VOL, or P-VOL becomes SMPL and none of thedisks in the device group has been written to, you can recover from the situation by splittingthe remaining P-VOL or S-VOL to SMPL:pairsplit [-S | -R] -g device_group

After splitting the pair, the pair can be re-created without copying its content using:paircreate -nocopy –c 15 -f fence_level -g device_group -v [r | l]

• If a horctakeover command results in S-VOL, or P-VOL becomes SMPL and data waswritten to one of the disks in the device group, you can recover from the situation by splittingthe remaining P-VOL or S-VOL to SMPL:pairsplit [-S | -R] -g device_group

After being split, the pair can be re-created with a full copy using:paircreate –c 15 -f fence_level -g device_group -v [r | l]To ensure that a certain pair state has been established, invoke the event wait command:pairevtwait -g device_group -t time_to_wait -s pair_state


XP Cluster Extension recovery procedures122

8 User configuration file and XP ClusterExtension objects

Objects (also called properties in this document) define the disk array environment and failover/failbackbehavior. Information comes directly from the cluster software, indirectly from the disk array throughXP RAID Manager, and from a configuration file created by users. This file describes the dependenciesbetween application services and XP RAID Manager device groups in one file for all applicationservices in the cluster.

The user configuration file provides customized and default values for supported parameters. You canspecify all customizable XP Cluster Extension objects in the file, and a copy must exist on all nodesusing XP Cluster Extension. XP Cluster Extension uses the information objects to match current diskstates and configuration parameters and to invoke actions, including preparing disks to be activatedor stopping the application startup.

User configuration file locationThe user configuration file is placed in the configuration directory:

UNIX (AIX, Solaris, and Linux)

/etc/opt/hpclx/conf

Windows

%ProgramFiles%\Hewlett-Packard\Cluster Extension XP\conf

For more information, see:

• “Basic configuration example” on page 139• “Creating and configuring the user configuration file” on page 112

HACMP

The UCF.cfg file is required for IBM HACMP. You must maintain and copy the UCF.cfg file to allsystems running XP Cluster Extension. The UCF.cfg file includes a COMMON section to configurethe XP Cluster Extension environment and an APPLICATION section to configure the applicationservice-dependent failover/failback behavior. The APPLICATION section is a multitag component;the APPLICATION tag and application-related objects can appear numerous times in the UCF.cfg.

For more information, see “User configuration file for HACMP” on page 28.

MSCS

XP Cluster Extension integration with MSCS does not require a user configuration file when the standardenvironment for XP Cluster Extension is used. The XP Cluster Extension objects that are integrated withMSCS can be configured as resource-specific properties in the cluster software.

For more information, see “Configuring XP Cluster Extension resources” on page 45.


RHCS and SLE HA

XP Cluster Extension integration with RHCS and SLE HA uses an XP Cluster Extension resourceconfiguration file. The objects and format in the configuration file are the same as the UCF.cfg file.

For more information, see Chapter 5 on page 93.

VERITAS Cluster Server

Integrating XP Cluster Extension with VERITAS Cluster Server does not require a user configurationfile when the standard environment for XP Cluster Extension is used. The XP Cluster Extension objectsthat are integrated with VERITAS Cluster Server are configurable as resource attributes in the clustersoftware.

For more information, see Chapter 4 on page 75.

File structureThe configuration file consists of a COMMON section and an APPLICATION section. These sectionsare distinguished by control tags. XP Cluster Extension uses the following objects as control tags:

• COMMON• APPLICATION

Objects have one of the following formats:

A definition of an object; for example, COMMON or APPLICATIONtag

A number; for example, a timeout valueinteger

A name, which can include alphabetic and numeric characters and underscores; for example,an application startup valuestring

A list of space-separated strings, for example, a list of host names (lists of numbers are storedas lists of strings)list

Text that is a comment starts with the pound (#) symbol and continues until the end of the line.Comments can start on a new line or be part of a line specifying an object.

Specifying object valuesWhen using the default configuration, you must provide values for the following objects:

• DeviceGroup: An XP RAID Manager device group• DC_A_Hosts: A list of the cluster nodes in data center A• DC_B_Hosts: A list of the cluster nodes in data center B• RaidManagerInstances: A list of XP RAID Manager instances that XP Cluster Extension can use to

communicate with the disk array• XPSerialNumbers: The serial numbers of the primary and secondary XP disk arrays

You do not need to change the default settings unless you want to change the degree of protectionfor your paired disks. If you change an object, you may need to change additional related objects.For example, if you change the FenceLevel object to DATA, you might need to change theDataLoseMirror object.

User configuration file and XP Cluster Extension objects124

Objects are supported according to the requirements or capabilities of the cluster software, as shownin Table 4 on page 125.

Table 4 Cluster software supported objects

System

RHCS,SLE HAVCSMSCSHACMPCLIObject

•••••COMMON

•••••LogDir

•••••LogLevel

•SearchObject

•VcsBinPath

•••••APPLICATION

••••ApplicationDir

•••••ApplicationStartup

•••••AsyncTakeoverTimeout

•••••AutoRecover

•••••BCEnabledA

•••••BCEnabledB

•••••BCMuListA

•••••BCMuListB

•••••BCResyncEnabledA

•••••BCResyncEnabledB

•••••BCResyncMuListA

•••••BCResyncMuListB

•ClusterNotifyCheckTime

•ClusterNotifyWaitTime

•••••DataLoseDataCenter

•••••DataLoseMirror

•••••DC_A_Hosts


System

RHCS,SLE HAVCSMSCSHACMPCLIObject

•••••DC_B_Hosts

•••••DeviceGroup

•FastFailbackEnabled

•••••FenceLevel

••Filesystems

•••••JournalDataCurrency

•LocalDCLMForNonPAIRDG

•••••PostExecCheck

•••••PostExecScript

•••••PreExecScript

•••••RaidManagerInstances

••••ResyncMonitor

••••ResyncMonitorAutoRecover

••••ResyncMonitorInterval

•••••ResyncWaitTimeout

•StatusRefreshInterval

•••••Vgs

•••••XPSerialNumbers

• Supported

COMMON objectsThe COMMON section is used to set the environment of XP Cluster Extension.

The COMMON tag can appear in the configuration file only once. The COMMON object does notrequire any value.

Objects of the type COMMON can appear only one time. Those objects must be placed after theCOMMON tag in the configuration file.

If the default values fit your environment, there is no need to specify them in the file.


COMMON

Format tag

Description Distinguishes between general (common) and application-specific objects.

LogDir

Format String

Description (Optional) Defines the path to the XP Cluster Extension log file.

Default value Linux/Unix/var/opt/hpclx/log

Windows%ProgramFiles%\Hewlett-Packard\Cluster Extension XP\log

LogLevel

Format String

Description (Optional) Defines the logging level used by XP Cluster Extension.

Valid values error (default): Logs only error messages for events that are unrecoverable.warning: Logs error messages and warning messages for events that are recover-able.info: Logs error messages, warning messages, and additional information, suchas disk status.debug: Logs error messages, warning messages, info messages, and messagesthat report on execution status; useful for troubleshooting.

SearchObject (HACMP only)

Format String

Description (Optional) Searches for the application service if the user configuration file specifiesmultiple applications. This object is used for HACMP only.

Default value Vgs

VcsBinPath (VCS only)

Format String

Description (Optional) Defines the path to the VCS binaries. This object is used for VCS only.

Default value /opt/VRTSvcs/bin


APPLICATION objectsThe APPLICATION section defines the failover and failback behavior of XP Cluster Extension for eachapplication service. APPLICATION is a multitag that can appear in the configuration file for eachapplication service using XP Cluster Extension.

The APPLICATION object requires the name of the application service as its value. The objects specifiedafter an APPLICATION tag must appear only once per application. As with the COMMON objects,the APPLICATION objects have predefined default values.

XP Cluster Extension uses the following rules to define objects:

• If you use the default value, you do not have to specify the object.• XP Cluster Extension uses objects depending on the setting of other objects. For example, if you

set the FenceLevel object to DATA, XP Cluster Extension uses the values specified for theDataLoseMirror or DataLoseDataCenter object. However, these objects are ignored if the FenceLevelobject is set to NEVER.

• The pre-execution and post-execution functions in XP Cluster Extension are not processed if theassociated object values are empty. (This is the default setting.)

When setting APPLICATION object values:

• Use the VCS GUI for VCS.• Use a user configuration file for the CLI and HACMP.• Use the Microsoft Cluster Administrator GUI (Windows Server 2003) or the Failover Cluster

Management GUI (Windows Server 2008/2008 R2) for MSCS..• Use an XP Cluster Extension configuration file for RHCS and SLE HA.

APPLICATION objectsThis section describes the available APPLICATION objects for XP Cluster Extension.

APPLICATION

Format Tag

Description Distinguishes between general and application-specific objects. Specify the nameof the application service. The format of its value is equivalent to a string value.

ApplicationDir

Format String

Description Specifies the directory where XP Cluster Extension searches for application-specificfiles, such as the force flag or online file.If ApplicationDir is set to a nonexistent drive and PairResyncMonitor is not enabled,XP Cluster Extension is unable to create the online file and cannot put the resourceonline.WindowsIf ApplicationDir is not set, XP Cluster Extension uses the local %HPCLX_PATH%values as defined in the registry.


Default values Linux/UNIXonline file: /etc/opt/hpclxforce flag file: etc/opt/hpclx/confWindows%HPCLX_PATH%

Files resource_name.createsplitbrain

resource_name.forceflag

resource_name.online

If specified in a user configuration file, resource_name is the value of the APPLIC-ATION tag; otherwise, resource_name is the value of the XP Cluster Extensionresource name.

ApplicationStartup

Format String

Description (Optional) Specifies where a cluster group should be brought online.The ApplicationStartup object can be customized to determine whether an applic-ation service starts locally or is transferred back to the remote data center (if pos-sible) to start immediately without waiting for resynchronization. This object is usedonly if an application service has already been transferred to the secondary siteand no recovery procedure has been applied to the disk set (the disk pair has notbeen recovered and is not in PAIR state). This process is considered a failback at-tempt without prior disk pair recovery.XP Cluster Extension can detect the most current copy of your data based on thedisk state information. If XP Cluster Extension detects that the remote XP disk arrayhas the most current data, it orders a resynchronization of the local disk from theremote disk, or it stops the startup process to enable the cluster software to fail backto the remote XP disk array.If a resynchronization is ordered, XP Cluster Extension monitors the progress of thecopy process. If the application service was running on a secondary XP disk arraywithout a replication link, a large number of records may need to be copied. If thecopy process takes longer than the configured application startup timeout value,the application startup will fail.MSCSIf the ApplicationStartup resource property is set to FASTFAILBACK and the Fail-overThreshold value is set to a number higher than the current number of clusteredsystems for the service or application, the service or application will restart onconfigured nodes until one of the following conditions is met:• The resource is brought online in the remote data center.• The resource failed because the FailoverThreshold value has been reached.• The resource failed because the FailoverPeriod timeout value has been reached.

CAUTION:Disable subsequent automated failover procedures for recoveryfailback operations.


Valid values FASTFAILBACK (default)The cluster group is brought online in the remote data center (if possible) withoutwaiting for resynchronization. The application startup process is stopped locallyand XP Cluster Extension reports a data center error. Depending on the clustersoftware, the application service cannot start on any system in the local data center,and the cluster software transfers the application service back to the remote datacenter. Use this value to provide the highest level of application service availability.Depending on the value configured for the AutoRecover object, XP Cluster Extensionattempts to update the former primary disk based on the secondary disk and swapsthe personalities of the disk pair so that the local disk will become the primary disk.In a two-node cluster, this process does not work because the target failback systemis not available. In this case, the application service must be started manually, orthe ApplicationStartup object must be set to RESYNCWAIT.In an XP Cluster Extension for MSCS integration, XP Cluster Extension can detectwhen there is no target failback system available in the remote data center. In thiscase, XP Cluster Extension behaves as if the ApplicationStartup resource propertyis set to RESYNCWAIT.RESYNCWAIT

The online local cluster group must wait until the disk status is PAIR. XP Cluster Ex-tension initiates a resynchronization of the local disk based on the remote disk.The copy process is monitored; if no copy progress is made after a monitoring in-terval expires, the copy process is considered failed and XP Cluster Extension returnsa global error. If RESYNCWAIT has been specified for the ApplicationStartup object,the ResyncWaitTimeout object must be specified, in case XP Cluster Extensionshould wait for resynchronization changes for more or less than 90 seconds, whichis the default.

AsyncTakeoverTimeout

Format Integer

Description (Optional) Specifies the horctakeover command timeout in seconds. Must beadjusted based on disk mirroring link speed.This object is used only if the FenceLevel object value is ASYNC.The takeover operation for fence level ASYNC (XP Continuous Access Software)offers the option to stop the data transfer process after a specified time value. Thisis used to allow access to the remote copy if the data transfer process is stoppeddue to an XP Continuous Access Software link failure. All data that has been copiedup to the moment the timeout value is reached is consistent and available to accessat the secondary site.


CAUTION:Measure or calculate the full XP disk array cache copy time to usethe gathered information for the AsyncTakeoverTimeout object. Aftera takeover command has been invoked, XP Continuous AccessSoftware copies the side file area residing in the XP disk array cacheto the site where the takeover command has been issued (thesecondary disks). The side file area cannot exceed the installed cachesize. The maximum time for the AsyncTakeoverTimeout object is thetime to fully copy the amount of cache size data. The takeover timeoutvalue is used to terminate the copy process to provide access to thesecondary disks; for example, if all links or the primary XP disk arrayare unavailable to copy the side file area. The copy time depends onthe performance of the XP Continuous Access Software link betweenyour sites. The takeover or resynchronization operation could takelonger than the timeout value for application service startup in thecluster software. The application service startup might fail in this case.However, the takeover or resynchronization command will continuein the background.

Default value 3600

AutoRecover

Format String

Description (Optional) Recovers a suspended or deleted disk pair when the resource is broughtonline at application service startup time.If the AutoRecover object is set to YES, XP Cluster Extension will try to resynchronizethe remote disk at application startup time. XP Cluster Extension will ignore the returncode of the resynchronization command and allow access to the disk ensuringhighest application availability.If the resynchronization attempt fails, XP Cluster Extension will not fail. The internallogic will first apply the concurrency and consistency rules to allow access to thedisk set.If you configure fence level DATA for the device group and set the FenceLevel objectto DATA, the AutoRecover object will change XP Cluster Extension's behavior. XPCluster Extension will attempt to re-establish the PAIR state and wait for the PAIRstate before it allows access to the disk. If the resynchronization or takeover processfails, XP Cluster Extension returns a global error.

Valid values YES (default)NO

BCEnabledA

Format String

Description (Optional) Enables rolling disaster protection for data center A.


Valid values YES

NO (default)

BCEnabledB

Format String

Description (Optional) Enables rolling disaster protection for data center B.

Valid values YES

NO (default)

BCMuListA

Format List

Description (Optional) Space-separated list defines the MU number of the XP Business CopySoftware disk pairs in data center A.

BCMuListB

Format List

Description (Optional) Space-separated list defines the MU number of the XP Business CopySoftware disk pairs in data center B.

BCResyncEnabledA

Format String

Description (Optional) Enables automatic resynchronization of XP Business Copy Software diskpairs in data center A. The automatic resynchronization function is supported onlywhen the split XP Business Copy Software pair is located in the same data centerwhere XP Cluster Extension is started.

Valid values YES

NO (default)

BCResyncEnabledB

Format String

Description (Optional) Enables automatic resynchronization of XP Business Copy Software diskpairs in data center B. The automatic resynchronization function is supported onlywhen the split XP Business Copy Software pair is located in the same data centerwhere XP Cluster Extension is started.

Valid values YES

NO (default)


BCResyncMuListA

Format List

Description (Optional) Space-separated list defines the MU number of the XP Business CopySoftware disk pairs in data center A.

BCResyncMuListB

Format List

Description (Optional) Space-separated list defines the MU number of the XP Business CopySoftware disk pairs in data center B.

ClusterNotifyCheckTime

Format Integer

Description Specifies how often XP Cluster Extension will check for VM live migration statechanges.

Default value 10 seconds

ClusterNotifyWaitTime

Format Integer

Description Specifies the amount of time that XP Cluster Extension will monitor for VM live mi-gration state changes.


DataLoseDataCenter

Format String

Description (Optional) Specifies whether a resource should be brought online while the diskpair is (or will be) suspended or deleted and there is no connection (XP ContinuousAccess and IP network) to the remote data center.Used only if the FenceLevel object value is DATA.XP RAID Manager is able to access its remote peer to invoke takeover actions forXP Continuous Access Software device groups. It is also able to invoke a swap-takeover operation of the device group from the secondary site. If no configuredremote XP RAID Manager instance replies to a request of the local XP RAID Managerinstance (remote status EX_ENORMT), all network connections between the localand the remote data center are considered DOWN. If the swap-takeover operationleads to a suspended state for the device group, the XP Continuous Access Softwarelinks are considered DOWN.Because redundant networks and XP Continuous Access Software links are necessaryto build a disaster-tolerant environment, this situation can be considered as a data


center failure. The DataLoseDataCenter object is used to allow/prohibit automaticapplication service startup in this particular case.The combination of setting the DataLoseMirror object to YES and theDataLoseDataCenter object to NO are contradictory.


DataLoseMirror

Format String

Description (Optional) Specifies whether a resource should be brought online while the diskpair is suspended or deleted.Used only if the FenceLevel object value is DATA and local and remote XP diskstatus information can be gathered. If the remote XP disk state information is notavailable (remote state EX_ENORMT), the setting of the DataLoseDataCenter objectwill be used.Depending on the value configured for the AutoRecover object, XP Cluster Extensionwill attempt to recover the PAIR state for the device group. XP Cluster Extensionwaits until the PAIR state has been established. If this operation fails, XP ClusterExtension returns a global error. Because the DATA fence level ensures no loss ofconcurrency, manual intervention is required to recover the PAIR state. The PAIRstate must be re-established for all disks in the device group before you can startthe application service.The combination of setting the DataLoseMirror object to YES and theDataLoseDataCenter object to NO are contradictory.

Valid values YES

NO (default)

DC_A_Hosts (Required)

Format List

Description This space-separated list defines the cluster nodes in data center A.VCSThis object is a string-vector element. Add a new element to the list for each systemname.

DC_B_Hosts (Required)

Format List

Description This space-separated list defines the cluster nodes in data center B.VCSThis object is a string-vector element. Add a new element to the list for each systemname.


DeviceGroup (Required)

Format String

Description XP RAID Manager device group, containing the application service disk set.

Files Linux/UNIX/etc/horcmX.conf

Windows:\winnt\horcmX.conf %system_root%\horcmX.conf

where X is the XP RAID Manager instance number.

FastFailbackEnabled (VCS only)

Format String

Description (Optional) Disables VCS service groups for the data center. This allows the imme-diate transferring of the service group back to the remote data center. To allow thisoperation, the VCS configuration file (main.cf) will be write-enabled and savedlater.The service group will be disabled for all systems contained in either theDC_A_Hosts object or DC_B_Hosts object. Then, the VCS configuration file willbe saved (dumped).


FenceLevel

Format String

Description (Optional) The FenceLevel object specifies the fence level configured for the devicegroup. XP Cluster Extension checks whether the current fence level reported by theXP disk array is the same as the configured (expected) fence level. This object isalso used to make sure your configurations are supported based on consistencyconsiderations. Different failover and recovery procedures are used for differentfence levels.If you change the FenceLevel object value, also review the values of these objects:DataLoseMirror, DataLoseDataCenter, and AsyncTakeoverTimeout.

Valid values DATA

NEVER (default)ASYNC (includes JOURNAL)

Filesystems (CLI and HACMP only)

Format List


Description Space-separated list of file systems.

JournalDataCurrency

Format String

Description (Optional) Specifies whether a resource should be brought online while there couldstill potentially be a large amount of data on P-VOL Journal that cannot be transmit-ted to the secondary site due to the XP Continuous Access Software link beingdown.Used only if the FenceLevel object value is ASYNC and the local device is an S-VOL.XP Cluster Extension checks whether the current XP Continuous Access Softwarelink status is >0 using the minimum active paths (MINAP) value returned by the XPRAID Manager pairvolchk command. If the minimum active paths equals 0, thisindicates that the XP Continuous Access Software link is unavailable and that anydata still located in the primary journal will not be replicated to the secondaryvolume. If JournalDataCurrency is set to YES then XP Cluster Extension will notperform the takeover operation and will not allow the application to access thedata.


LocalDCLMForNonPAIRDG

Format String

Descrip-tion

Specifies whether a live migration operation within the local data center is allowedwhen the device group is not in PAIR state.Set this property to YES to allow live migration operations in the local data center whenthe device group is not in PAIR state, the latest data is in the local data center, and theXP Cluster Extension resource can come online.For example, if the device group state is PVOL_COPY in the local data center andSVOL_COPY in the remote data center, setting this property to YES allows you to performlive migration to nodes within the local data center.Set this property to NO if you want to cancel live migration operations within the localdata center when the device group is not in PAIR state.

NOTE:Configure this parameter for each XP Cluster Extension resource associatedwith the VM cluster resource and the corresponding application clusterresource in the UCF file. If the VM group contains more than one XP ClusterExtension resource, and you want to use this parameter, you must set thisparameter to the same value for each XP Cluster Extension resource. Ifyou do not set the parameter to the same value, this parameter will defaultto a value of NO.


Valid val-ues

YES

NO (default)

PostExecCheck

Format String

Description (Optional) The PostExecCheck object is used to configure XP Cluster Extension togather XP disk pair status information after the takeover procedure. That informationwill be passed to the post-executable. In case of a remote data center failure, itcould be time consuming to gather that information, especially if your post-execut-able does not need any XP status information. The arguments passed to the post-executable will include only the local disk status if the PostExecCheck object is setto NO. See “Setting up XP RAID Manager” on page 20.

Valid values YES

NO (default)

PostExecScript

Format String

Description (Optional) Specifies an executable with its full path name to be invoked after thetakeover action or failover procedure.

PreExecScript

Format String

Descrip-tion

(Optional) Specifies an executable with its full path name to be invoked before thetakeover action or failover procedure.

RaidManagerInstances (Required)

Format List

Description A space-separated list of XP RAID Manager instances that XP Cluster Extension canuse to communicate with the disk array. The instance numbers must be the sameamong all cluster systems. XP Cluster Extension can alternate between the specifiedinstances.VCSThis object is a string-vector element. Add a new element to the list for each systemname.

Files Linux/UNIX/etc/horcmX.conf

Windows%systemroot%\horcmX.conf

where X is the XP RAID Manager instance number.


ResyncMonitor

Format String

Description (Optional) Starts the pair/resync monitor to monitor the disk pair status and resyn-chronize disk pairs if the ResyncMonitorAutoRecover attribute is set to YES.

Valid values YES

NO (default)

ResyncMonitorAutoRecover

Format String

Description (Optional) Automatically recovers disk pairs states if the disk pairs are monitoredby the pair/resync monitor.

Valid values YES

NO (default)

ResyncMonitorInterval

Format Integer

Description (Optional) Specifies the monitor interval (in seconds) that the pair/resync monitorchecks the disk pair status.

Default value 60

ResyncWaitTimeout

Format Integer

Description (Optional) Specifies the timeout value (in seconds) for a disk pair resynchronization.It may take some time to resynchronize disks. The timer times out if there is nochange in the percentage value of the copy status for the device group in the spe-cified time interval. The timeout value is used if the ApplicationStartup object is setto RESYNCWAIT.

Default value 90

StatusRefreshInterval

Format Integer

Description Specifies how often XP Cluster Extension will gather XP storage array information.



Vgs (CLI and HACMP only)

Format List

Description List of volume groups.

XPSerialNumbers (Required)

Format List

Description A space-separated list of at least two serial numbers must be specified: the serialnumbers of the primary and secondary XP disk arrays. XP Cluster Extension checkswhether the local disk array is contained in this list. Serial numbers of the disk arraysof the connected cluster nodes (at least two).VCSThis object is a string-vector element. Add a new element to the list for each systemname.

Basic configuration exampleThe following is an example of a UCF.cfg file:

#/etc/opt/hpclx/conf/UCF.cfg#This is the XP Cluster Extension User Configuration File (UCF.cfg).#The COMMON tag specifies the configuration for the#XP Cluster Extension core environmentCOMMONLogLevel info #default (not necessary)APPLICATION sap #the application serviceVgs sapdatavg saptmpvg #the volume groups (not necessary)Filesystems /sapdata /saptmp #the filesystemsDeviceGroup sapdg #RM dev group for the app serviceRaidManagerInstances 22 #RM instance number for dev groupDC_A_Hosts host1a host2a #Data center ADC_B_Hosts host3b host4b #Data center B


9 Advanced XP Cluster Extensionconfiguration

This chapter describes advanced XP Cluster Extension configuration procedures.

Implementing rolling disaster protectionTo implement rolling disaster protection, create XP Business Copy Software disk pairs for the local XPContinuous Access Software disks.• Create the XP Business Copy Software disk pairs using the –m noread option of the paircreate

command. This option ensures that XP Business Copy Software disks are unavailable to otherservices and reserved for rolling disaster protection only.

• Map the XP Business Copy Software S-VOLs to a backup server, not to the local cluster node.When XP Cluster Extension suspends the XP Business Copy Software pairs, they become availableto the local server, which could result in duplicated volumes, disk group IDs, or signatures.

CAUTION:You must ensure that at least one XP Business Copy Software disk pair is in PAIR state. If rollingdisaster protection is enabled and none of the XP Continuous Access Software mirrored disk pairshave an XP Business Copy Software disk pair that is in PAIR state, XP Cluster Extension returns aglobal error, and you will not be able to activate the application service.You can use forceflag to start the application service. See “Enabling write access regardless ofdisk pair state” on page 144. In this case, XP Cluster Extension disables rolling disaster protection.

Using XP RAID Manager with rolling disaster protectionRolling disaster protection does not require that you define XP Business Copy Software disk pairs inthe XP RAID Manager horcmX.conf files. XP Cluster Extension uses the MU number to monitor andcontrol associated XP Business Copy Software pairs. You must create an XP RAID Managerconfiguration file to control the XP Business Copy Software disk pairs that are outside XP ClusterExtension control.

XP Cluster Extension Software cannot suspend XP Business Copy Software disk pairs on the remoteXP disk array in the remote data center if the XP RAID Manager instance in the remote data center isnot running or not reachable.

Setting XP Cluster Extension objects to enable rolling disaster protectionTo enable rolling disaster protection with XP Business Copy Software, set the BCEnabledA andBCEnabledB objects for data centers A and B. When these objects are set to YES, rolling disasterprotection is enabled and XP Cluster Extension checks whether the configured XP Business CopySoftware disk pairs are in PAIR state. Before initiating the resynchronization operation, XP Cluster


Extension suspends specified XP Business Copy Software disk pairs that are in PAIR state. Forinformation on setting XP Cluster Extension objects, see Chapter 8 on page 123.

When using rolling disaster protection, note the following:

• If the BCEnabledA and BCEnabledB objects are set to YES, you must configure specific XP BusinessCopy Software disk pairs using MU numbers. The MU number defines one of the many disk pairrelationships you can create with XP Business Copy Software disk pairs. You can specify as manyMU numbers as the XP Business Copy Software supports. Disk pair MU numbers are specified bythe BCMuListA and BCMuListB objects for data centers A and B.

• To enable resynchronization of XP Business Copy Software disk pairs that have been split by XPCluster Extension, use the BCResyncEnabledA and BCResyncEnabledB objects for data centers Aand B.

• XP Cluster Extension maintains a list of all associated XP Business Copy Software disk pairs thatwere in PAIR state before a resynchronization attempt. If pairs were suspended, XP Cluster Extensionautomatically resynchronizes those disk pairs after the XP Continuous Access Software remotemirrored disk pairs have been paired. This feature supports automatic resynchronization of locallysplit XP Business Copy Software disk pairs only. You must specify MU numbers for resynchronizationby using the BCResyncMuListA and BCResyncMuListB objects for data centers A and B.

Setting automatic recovery for rolling disaster protectionIf the AutoRecover object is set to YES, XP Cluster Extension automatically resynchronizes the XPContinuous Access Software disk pairs to update the remote disks. If rolling disaster protection is alsoenabled, it suspends the XP Business Copy Software disk pair that is attached to the remote XPContinuous Access Software disk.

If the remote XP RAID Manager instance is not running or cannot be reached, the remote XP BusinessCopy Software disk pair cannot be suspended. If this occurs, XP Cluster Extension continues theapplication service activation without automatic resynchronization of the XP Continuous AccessSoftware disk pair and without the suspending of the XP Business Copy Software disk pair. In thiscase, the XP Continuous Access Software disk pair must be recovered manually.

Using the pair/resync monitor with rolling disaster protectionIf the ResyncMonitor object is set to YES, the pair/resync monitor does not use XP Business CopySoftware pairs to recover suspended or failed XP Continuous Access Software disk pairs.

To protect the remote volume of an out-of-sync XP Continuous Access Software disk pair against rollingdisasters, use the pair/resync monitor's default settings. Resynchronize the XP Continuous AccessSoftware disk pair manually after splitting off the XP Business Copy Software disk pair.

Restoring server operation for rolling disaster protectionRolling disaster protection automatically recovers the PAIR state of the XP Continuous Access Softwaredisk pair of an application service. Before you fail over (or fail back) an application service from onedata center to the other, you must restore the server operation. After you restart the server, also startthe XP RAID Manager instance used to manage the XP Continuous Access Software disk pairs onthose servers. This enables rolling disaster protection to work correctly during a recoveryfailover/failback operation.

Figure 10 on page 143 depicts a fully configured XP Cluster Extension environment that uses rollingdisaster protection. The XP Business Copy Software disk pairs are specified as 0 in the XP ClusterExtension BCMuListA and BCMuListB objects. See “APPLICATION objects ” on page 128 for moreinformation about these objects.

Advanced XP Cluster Extension configuration142

Figure 10 Disaster-tolerant configuration with rolling disaster protection.

Monitoring and resynchronizing device groupsThe pair/resync monitor can either only monitor or both monitor and resynchronize the state of theXP RAID Manager device group for an application service.


CAUTION:If the application service stops, the cluster software or your customized solution must be able to stopthe monitoring or resynchronization utility. Without this ability, the use of the pair/resync monitor isnot supported. HP recommends that you disable application service failover during a disk pair recovery(resynchronization). When the pair/resync monitor is enabled, XP Cluster Extension takes immediateaction to recover any reported suspended disk pair. If, at any time, the resynchronization process isrunning on both disk array sites, data corruption might occur.

Turn the pair/resync monitor (clxchkd) on or off using the ResyncMonitor object. For informationon setting XP Cluster Extension objects, see Chapter 8 on page 123.

If the ResyncMonitorAutoRecover object is set to YES, the monitor tries to resynchronize the remotedisk based on the local disk. Resynchronization occurs only if the disks are in a P-VOL/S-VOL orS-VOL/P-VOL relationship. If one or both disk pairs are in the SMPL state or the device group state ismixed, automatic resynchronization is not attempted. The ResyncMonitorAutoRecover object set toYES is supported only if the minimum disk array firmware version is 01-11-xx (XP512/XP48) or21.01.xx (XP128/XP1024), and the minimum XP RAID Manager version is 01.04.00.

The monitor interval is specified with the ResyncMonitorInterval object. Do not set the monitor intervalbelow the XP RAID Manager timeout parameter (HORCM_MON in the horcmX.conf file).

If the link for the device group is broken, the pair/resync monitor notifies you by using the syslogfacility (Linux/UNIX) and the Event Log (Windows). The monitor recognizes a broken link only whendata is to be written to disk; otherwise, the data is the same on the primary and secondary disk, andthe device group state is reported as PAIR.

Enabling write access regardless of disk pair stateThe force flag forces XP Cluster Extension to skip the internal logic and enables write access to thelocal volume, regardless of the disk pair state. This flag can be set when you are sure that the localvolume contains the latest data, even though a previous application service startup process failedbecause XP Cluster Extension discovered a disk pair status that could not be handled automatically.

To use the force flag:

1. Ensure that the application service is not running.2. Create a file called application_name.forceflag in the directory specified by the Applic-

ationDir object.3. Start the application service. XP Cluster Extension removes the forceflag file after detecting

it.

You cannot use the force flag if the local disk state is S-VOL_COPY, which indicates that a copyoperation is in progress. When a copy operation is in progress, a disk cannot be activated, and XPCluster Extension returns a global error.

Using the force flag does not enable the automatic recovery features of XP Cluster Extension. Afterusing the force flag, you must recover the suspended or broken disk pairs using XP RAID Managercommands as described in “Recovery sequence” on page 120.


Executing programs before and after an XP Cluster Extensiontakeover

XP Cluster Extension can invoke other programs, such as Perl scripts, before or after an XP ClusterExtension takeover. These programs can be any executable, and must be able to provide return codesto XP Cluster Extension. If the programs add significant execution time to the application servicestartup, the timeout values for the startup must be adjusted in the cluster software.

XP Cluster Extension transfers information as command-line arguments to the pre-execution andpost-execution programs. Pre-executables and post-executables must be specified by full path in thePreExecScript and PostExecScript objects. If no executable is specified (empty value for the object),no preprocessing or postprocessing, is done.

The pre-executable and post-executable path names can include spaces and environment variables.The environment variables will be expanded to form the full path name for the executable.

To use Perl scripting with MSCS, the Perl script must be called from a Windows batch file; therefore,two scripts are needed: the calling batch file and the called Perl script.

In the following example, c:\tmp\preExec.bat is the calling batch file, and c:\tmp\preExec.plis the called Perl script:

Windows batch file: c:\tmp\preExec.bat@echo offc:\perl\bin\perl.exe c:\tmp\preExec.pl %3 %4 %5exit /B %ERRORLEVEL%

ArgumentsThe following arguments are transferred to the scripts in this order:

1. Name

2. Vgs (HACMP only)

3. RaidManagerInstances

4. DeviceGroup

5. local device group state (check)

Pre-executable status before failover and post-executable status after failover

6. local device group state (display)


IMPORTANT:An empty string is returned if parameter #5 is not SSWS, PSUE, or PDUB.

7. remote device group state (check)



8. remote device group state (display)


IMPORTANT:An empty string is returned if parameter #7 is not SSWS, PSUE, or PDUB.

9. current fence level

10. disk array serial numbers (local)

11. reserved

12. reserved

13. disk array firmware version (local)

14. XP RAID Manager version (local)

15. application directory path (ApplicationDir object)

16. log file location (LogDir object)

17. DC_A_Hosts node names

18. DC_B_Hosts node names

Pre-executable return codesPre-executables must give a return code. These return codes determine whether a takeover functionmust be called.

0 PRE_OK_TAKEOVER

Pre-executable OK and takeover action allowed.

1 PRE_ERROR_GLOBAL

Pre-executable failed; no takeover; stop application service cluster-wide.

2 PRE_ERROR_DC

Pre-executable failed; no takeover; stop application service in this data center.

3 PRE_ERROR_LOCAL

Pre-executable failed; no takeover; stop application service on this system.

4 PRE_ERROR_TAKEOVER

Pre-executable failed; takeover action allowed.

5 PRE_OK_NOTKVR_NOPST

Pre-executable ok; no takeover; no post-exec.


CAUTION:If the pre-execution program returns 1, 2, 3, or 5, a post-executable will not be executed. If a takeoverfunction fails, the post-executable will not be executed.

Post-executable return codesPost-executables must give a return code. These return codes determine whether the application isstopped.

0 POST_OK

Post-executable OK; continue.

1 POST_ERROR_GLOBAL

Post-executable failed; stop application service cluster-wide.

2 POST_ERROR_DC

Post-executable failed; stop application service in this data center.

3 POST_ERROR_LOCAL

Post-executable failed; stop application service on this system.

4 POST_ERROR_CONTINUE

Post-executable failed; continue without error.


10 Troubleshooting

To troubleshoot problems with XP Cluster Extension, you must understand XP Continuous AccessSoftware environments. Many issues can be attributed to incompatible disk pair states. See the XPContinuous Access Software and XP RAID Manager documentation before assuming that a problemhas been caused by XP Cluster Extension. For more information on XP Continuous Access Software,see the HP StorageWorks XP Continuous Access Software user guide.

XP Cluster Extension logs messages to the cluster-specific log location. However, it always keeps itsown log file in its default log location.

CAUTION:XP Cluster Extension is not able to handle XP device group states automatically and correctly whenthey result from manual manipulations. XP Cluster Extension will try to automatically recover suspendedXP RAID Manager device group states if the AutoRecover object is set to YES. However, if the recoveryprocedure experiences a problem, XP Cluster Extension will not stop unless fence level DATA is usedor the ApplicationStartup object is set to RESYNCWAIT. Therefore, ensure that the device group PAIRstate has been recovered before the next failure occurs.

Always disable automatic application service failover when resynchronizing disk pairs. A failure ofthe resynchronization source while resynchronizing can lead to unrecoverable data on theresynchronization target. The resynchronization process does not copy data in transactional order.For more information, see “Implementing rolling disaster protection” on page 141.

XP Cluster Extension log facilityThe logging module of XP Cluster Extension provides log messages to the cluster software as well asto the XP Cluster Extension log file. The XP Cluster Extension log file includes disk status information.

• The XP Cluster Extension log file is located in the following directories:Linux/UNIX/var/opt/hpclx/log

WindowsBy default, the location is defined as:%ProgramFiles%\Hewlett-Packard\Cluster Extension XP\log\

• For the configuration tool, the clxXPcfg.log file resides in:%ProgramFiles%\Hewlett-Packard\Cluster Extension XP\log\

If the log file needs to be cleared and reset, for example, to reduce disk space usage, archive the logfile and then delete it. A new log file is generated automatically.

For information about log levels, see LogLevel on page 127.


Start errorsStart errors can occur when the path to the XP RAID Manager binaries has not been set in the PATHenvironment variable. If a user configuration file is not found in the correct directory location, XPCluster Extension returns a local error.

A start error occurs if the APPLICATION name tag value in the XP Cluster Extension resourceconfiguration file does not match the service name (RHCS) or the App value of XP Cluster Extensionresource (SLE HA).

XP Cluster Extension returns a local error if it does not find the XP Cluster Extension resourceconfiguration file for in the correct directory location (RHCS and SLE HA).

Failover error handlingXP Cluster Extension automatically fails over application services if the system the application serviceis running on becomes unavailable. This also means that if a problem with the XP disk array stateoccurs, an application service startup process will be stopped. The behavior of XP Cluster Extensionis highly configurable.

Depending on the customer setting, XP Cluster Extension is used to prevent application services fromstarting automatically under the wrong conditions. Therefore, XP Cluster Extension will return local,data center-wide, or even cluster-wide errors to prevent accidental access to the XP disk array diskset.

XP Cluster Extension provides the following error return codes for failover operations:

Prohibits an application service startup on the local system. This can becaused by the inability of XP Cluster Extension to enable disk access, ormisconfiguration of the disk array environment.

local error

Prohibits an application service startup on any system in the local datacenter. This error is returned if the disk state indicates that it makes nosense to allow any other system connected to the same disk array to ac-cess the disks.

data center error

A global error is returned if the configuration or the disk state does notallow an automatic application service startup process. In such cases,manual intervention is required.

global error

When XP Cluster Extension is integrated, an error message string and integer value are displayed.For the CLI, a return code is displayed. For more information, see “CLI commands” on page 113.

HACMP-specific error handlingXP Cluster Extension related messages are logged by HACMP to the following locations:

• /usr/adm/cluster.log

This is the general HACMP log file, which gives an overview of all events processed and whetherthey were successful or unsuccessful.

• /tmp/hacmp.out

This is a detailed HACMP log file containing process logs of all event scripts. The output of XPCluster Extension can also be found in this file.

Troubleshooting150

The XP Cluster Extension log file is named clxhacmp.log.

Start errorsHACMP will go into a loop and wait until the problem is solved and until the file /etc/opt/hpclx/application_name.LOCK has been removed. This process has been adopted from HACMP, whichwill also run in an endless loop if there is a failure and until you recover all errors and start theapplication manually. After all errors have been recovered, you can invoke the command clruncmdto return control back to the cluster software.

If the program is in a very early state of processing and experiences a problem before resolution ofthe application name, it may return an error return code. The /etc/opt/hpclx/UNKNOWN.LOCKfile is created and must be removed after the problem has been resolved.

Failover errorsAs mentioned previously, the HACMP error handling of the XP Cluster Extension will create a .LOCKfile for the resource group (for example, /etc/opt/hpclx/OracleRG.LOCK). Messages are loggedto the log files /var/opt/hpclx/log/clxhacmp.log and /tmp/hacmp.out. The file can beremoved after the problem has been solved. HACMP can then continue to start the resource group.

This file will be created for any error XP Cluster Extension returns. However, XP Cluster Extension willspecify whether the error is a local, data center, or cluster-wide error.

The following example demonstrates the behavior of XP Cluster Extension for HACMP if a pair stateis discovered (which does not allow for an automatic takeover operation by XP Cluster Extension). Inthis case, the pairs have been manually suspended. It is impossible for XP Cluster Extension to determinewhich copy of the mirrored data is the most current.

The output in /tmp/hacmp.out will be similar to the following example:

clxHACMP: > Fri Dec 15 16:35:19 NFT 2000clxHACMP: > Arguments: oracle ora1vg ora2vg 0 oracle PVOL_PSUS PSUS SVOL_SSUS SSUSDATA 30368 30380 01-11-22/00 01.04.01clxHACMP: > number of arguments: 14clxHACMP: > 1: oracleclxHACMP: > 2: ora1vg ora2vgclxHACMP: > 3: 0clxHACMP: > 4: oracleclxHACMP: > 5: PVOL_PSUSclxHACMP: > 6: PSUSclxHACMP: > 7: SVOL_SSUSclxHACMP: > 8: SSUSclxHACMP: > 9: DATAclxHACMP: > 10: 30368clxHACMP: > 11: 30380clxHACMP: > 12:clxHACMP: > 13: 01-11-22/00clxHACMP: > 14: 01.04.01clxHACMP > ===PRE===============================================clxHACMP: pre-exec script successful (rc=0).clxHACMP: ERROR - no takeover action found.clxHACMP: ERROR - global cluster failure occurred - waiting!clxHACMP: ERROR -clxHACMP: ERROR - ================================================================clxHACMP: ERROR - XP Cluster Extension takeover procedure FAILED.clxHACMP: ERROR -


clxHACMP: ERROR - Pair state of device group "oracle" might beclxHACMP: ERROR - incorrect. Manual checking and correction withinclxHACMP: ERROR - Continuous Access XP is required.clxHACMP: ERROR - Remove file "/etc/opt/hpclx/OracleRG.LOCK" in orderclxHACMP: ERROR - to continue with HACMP specific recovery actions.=================================================================

The last message is repeated every 5 minutes. XP Cluster Extension will stop any further processinguntil the you remove the application_name.LOCK file to transfer control back to HACMP. Thisenables you to check the status of the data on each copy and decide whether it is safe to continueor not.

Depending on the amount of time needed for checking the configuration and the XP disk pair status,the HACMP timeout could be reached. This will automatically cause the event config_too_longto be called by HACMP. The following message will appear in the log file /tmp/hacmp.out:

WARNING: Cluster MYCLUSTER has been running recovery program '/usr/es/sbin/cluster/events/node_up.rp' for 1110 seconds.Please check cluster status.

If you think the XP Cluster Extension configuration is correct, and the XP disk pair status allows youto manually continue the process for starting the application, remove the application lock file /etc/opt/hpclx/oracle.LOCK mentioned in the previous error message.

When this file has been removed, XP Cluster Extension transfers control back to HACMP. The eventget_disk_vg_fs and all the subsequent events within the main event node_up_local will beprocessed. Because XP Cluster Extension as a pre-event of get_disk_vg_fs has produced an error,the main event node_up_local will fail as well. The following HACMP event event_error will becalled:

node_up_local[30] [ 0 -ne 0 ]node_up_local[8] exit 1Dec 15 17:07:17 EVENT FAILED:1: node_up_localnode_up[326] [ 1 -ne 0 ]node_up[328] cl_log 650 node_up: Failure occurred while processing Resource GroupOracleRG. Manual intervention required. node_up OracleRG***************************Dec 15 2000 17:07:17 !!!!!!!!!! ERROR !!!!!!!!!!***************************Dec 15 2000 17:07:17 node_up: Failure occurred while processing Resource GroupOracleRG. Manual intervention required.node_up[329] STATUS=1node_up[337] [ AIX1 != AIX1 ]node_up[356] exit 1Dec 15 17:07:18 EVENT FAILED:1: node_up AIX1

To continue any further processing of HACMP, you must invoke the HACMP command clruncmd torecover from the status event_error.

Example

# clruncmd aix1

This will bring the cluster into normal status again. All subsequent events (for example,node_up_complete) will be processed.

Troubleshooting152

MSCS-specific error handlingXP Cluster Extension related messages are logged by MSCS to the following locations:

• %ClusterLog%\cluster.log.

• The XP Cluster Extension log file is named clxmscs.log.• The XP Cluster Extension configuration tool log resides in the %ProgramFiles%\Hewlett-

Packard\Cluster Extension XP\log\ directory.

Resource start errorsMSCS configurations do not require a UCF.cfg file if the default COMMON objects are used(recommended).

MSCS will fail the XP Cluster Extension resource on the local system if the clxpcf file is not present.If the program is in a very early state of processing, the operation might fail and XP Cluster Extensionwill not show the resource name in the error message.

Failover errorsXP Cluster Extension's integration with MSCS returns a local error and fails the resource if aconfiguration error occurs. This could be a problem with the XP RAID Manager instance configurationor an error, which will probably require starting the resource group on another system.

XP Cluster Extension resources return a data center error and fail the resource if the XP disk arraystatus indicates that the problem experienced locally would not be solved on another system connectedto the same XP disk array. This means all systems specified in the DC_A_Hosts resource property orthe DC_B_Hosts resource property would fail to bring the resource group online.

Depending on the resource group and resource property values, the resource tries to start on differentnodes several times. If the remote data center is down, this would look like the resource group isalternating between the surviving systems. This happens until the previously mentioned resource andresource group property values are reached or you disable the restarting of the resource. This couldbe also the case if the ApplicationStartup resource property has been set to FASTFAILBACK.

If an XP disk array state has been discovered that does not allow bringing the resource group onlineon any system in the cluster, a cluster error would be reported and the resource would fail on allsystems. This could lead to the same behavior as described for an XP Cluster Extension data centererror.

Examples of such a state could be a SMPL state on both primary and secondary disks, a suspended(PSUS/SSUS) state on either site, or a state mismatch in the device group for this resource group.None of the previously mentioned scenarios will allow automatic recovery because the XP ClusterExtension resource cannot decide which copy of the data is the most current copy. In those cases, astorage or cluster administrator must investigate what happened to the environment.

In any case, restarting a failed resource group without investigating the problem is not recommended.A failed XP Cluster Extension resource indicates the need to check the status of the XP disk pair oneach copy and decide whether it is safe to continue or not.

Figure 11 on page 154 shows examples of an incompatible XP disk pair state shown in theclxmscs.log file. The same messages can be found in the MSCS cluster log file if the XP ClusterExtension LogLevel object is set to INFO; this, however, requires creating a UCF.cfg file.


Figure 11 Incompatible XP disk pair state.

Using the Domain user account (Windows Server 2008/2008 R2 only)When using the Domain user account to manage the cluster, modifying HORCM files might not bepossible, and XP Cluster Extension tools might not run as expected. If you experience any of theseissues, turn off UAC.

To turn off UAC, select Control panel > User Accounts, and click Turn User Account Control on or off.Clear the User Account Control (UAC) to help protect your computer check box. This might resolve theissue and allow you to use the XP Cluster Extension tools with the Domain user account.

VCS-specific error handlingXP Cluster Extension related messages are logged by VCS to the following locations:

• VCS 1.3.0 or later:/var/VRTSvcs/log/engine_A.log

• VCS 1.1.2:/var/VRTSvcs/log/engine.A_log

Troubleshooting154

This is a general VCS engine log file, which gives an overview of all cluster-related activities andwhether they were successful or unsuccessful.

• VCS 1.3.0 or later:/var/VRTSvcs/log/ClusterExtensionXP_A.log

• VCS 1.1.2:/var/VRTSvcs/log/ClusterExtensionXP.log_A

This XP Cluster Extension agent log file of VCS shows agent-related error information. The XP ClusterExtension log file is named clxvcs.log.

Start errorsVCS will fail the resource and disable the service group on the local system if it the clxpcf file isnot present. If the program is in a very early state of processing, this operation might fail, and XPCluster Extension will not show the service group in the error message. However, VCS will fail theresource.

Failover errorsXP Cluster Extension's integration with VCS disables service groups on the local system if a configurationerror occurs. In this case, XP Cluster Extension will return a local error.

The service group is disabled in the data center if the XP disk array status indicates the problemexperienced locally cannot be solved on another system connected to the same XP disk array. Allsystems specified in the DC_A_Hosts object or DC_B_Hosts object are disabled to bring the servicegroup online.

This could be also the case if the ApplicationStartup object has been set to FASTFAILBACK.

If an XP disk array state has been discovered (which does not allow bringing the service group onlineon any system in the cluster), a cluster error is reported and all systems are disabled to bring theservice group online. Such state could be a SMLP state on both primary and secondary disks, asuspended (PSUS/SSUS) state on either site, or a state mismatch in the device group for this servicegroup. None of the scenarios allows automatic recovery because XP Cluster Extension cannot determinewhich copy of the data is the most current. In these cases, a storage or cluster administrator mustinvestigate what happened to the environment.

CAUTION:HP does not recommend that you enable the service group again and try to bring the prior failedservice group online without investigating the problem. When a failed XP Cluster Extension resourceoccurs, check the status of the XP disk pair on each copy, and decide whether it is safe to continue.

Examples

Figure 12 on page 156 and Figure 13 on page 156 show examples of an incompatible XP disk pairstate shown in the VCS Cluster Manager Log Desk window.


Figure 12 Incompatible XP disk pair state (VCS Cluster Manager Log Desk window).

Figure 13 on page 156 shows detailed information for the current XP disk pair state, which will bedisplayed in the VCS Log Desk only if the XP Cluster Extension LogLevel object is set to INFO.

Figure 13 Detailed information of the XP disk pair state (VCS Log Desk).

Linux-specific error handlingXP Cluster Extension messages are logged by RHCS and SLE HA to the following location: /var/log/messages. The XP Cluster Extension log file is called clxxplxcs.log.

Troubleshooting156

Failover errorsXP Cluster Extension will fail to bring an RHCS service or SLE HA resource group online on the localsystem if a configuration error occurs. In this case, XP Cluster Extension returns a local error.

The RHCS service or SLE HA resource group will go into a failed state after a startup attempt on anysystem in the same data center if the disk array status indicates that a problem experienced locallywould not be solved on another system connected to the same disk array. In this case, XP ClusterExtension returns a data center error. This error could also occur if the ApplicationStartup object isset to FASTFAILBACK.

If a disk array state that does not allow starting the RHCS service or SLE HA resource group on anysystem in the cluster is discovered, a cluster error is reported and none of the systems will be allowedto run the service or resource group. Such a state could be an SMLP state on both primary andsecondary disks, a suspended (PSUS/SSUS) state on either site, or a state mismatch in the devicegroup for this RHCS service or SLE HA resource group. None of these scenarios allows automaticrecovery because XP Cluster Extension cannot determine which copy of the data is the most current.In these cases, a storage or cluster administrator must investigate the problem.

CAUTION:Do not start the RHCS service or SLE HA resource group again or try to start the failed RHCS serviceor SLE HA resource group without investigating the problem. When an RHCS service or SLE HAresource group using XP Cluster Extension fails, check the status of the XP disk pair on each copy anddecide whether it is safe to continue.

The FC link is down (RHCS)In RHCS, the detection of a storage outage due to failure of all paths to the storage depends on themonitoring capability of resources configured in the RHCS service. For example, the LVM and filesystemresource agents distributed with RHCS can detect the loss of storage and take appropriate actions.The stop operation on a service might fail due to the inability to stop individual resources cleanly.This may be caused by the loss of paths to the storage. When the stop operation on a service fails,RHCS marks the service as failed and the service does not automatically fail over to another node.

To recover from this situation, use the following procedure:

1. Remove the node that lost access to the storage by shutting down the node.

2. Follow the steps required to bring up a service in a failed state, as documented in the RHCSadministration guide. This process involves disabling the service, and then enabling it on thenode where the service is allowed to come online.


3. Restart the node that was shut down.

NOTE:The time to detect a storage outage due to failure of all paths to storage depends on thesetting for no_path_retry in the multipath software configuration. A value of fail doesnot queue I/O in the event of a failure in all paths and returns an immediate failure. Forinformation about the recommended value for your environment, see the DM-Multipathdocumentation.Some resource agents, such as LVM, offer a mechanism called self_fence to take themselvesout of a cluster through node reboot when an underlying logical volume can no longer beaccessed. For supported options, see the RHCS documentation.

A storage replication link is down (RHCS)If an Cluster Extension configuration uses DR groups with failsafemode enabled, the array disablesaccess to the disk when it cannot replicate the I/O to the remote array.

In this situation, if a replication link is broken, the resource agents of configured resources, such aslvm or fs, may be able to detect and take appropriate actions. The stop operation on a service mightfail due to the inability to stop individual resources cleanly because the disk is no longer accessiblefor read/write operations. When the stop operation on a service fails, RHCS marks the service asfailed and the service does not automatically fail over to another node.

To recover from this situation, use the following procedure:

1. Remove the node that lost access to the storage by shutting down the node.

2. Follow the steps required to bring up a service in a failed state, as documented in the RHCSadministration guide. This process involves disabling the service, and then enabling it on thenode where the service is allowed to come online.

3. Restart the node that was shut down.

NOTE:The time to detect a storage outage due to failure of all paths to storage depends on thesetting for no_path_retry in the multipath software configuration. A value of fail doesnot queue I/O in the event of a failure in all paths and returns an immediate failure. Forinformation about the recommended value for your environment, see the DM-Multipathdocumentation.Some resource agents, such as LVM, offer a mechanism called self_fence to take themselvesout of a cluster through node reboot when an underlying logical volume can no longer beaccessed. For supported options, see the RHCS documentation.

A data center is down (SLE HA and RHCS)RHCS and SLE HA expect an acknowledgement from the fencing device before services are failedover to another node. In the event of complete site failure, including fencing devices, clusters do notautomatically fail over services to surviving cluster nodes at the remote site. Manual intervention isrequired in this situation. For instructions on bringing a service online, see the cluster softwaredocumentation.

Troubleshooting158

Pair/resync monitor messages insyslog/errorlog/messages/event log

Using the pair/resync monitor will cause a message in the system log file of your operating system(for any non-PAIR state of the device group being monitored). Those messages might indicate thefollowing:

• The XP RAID Manager instance is not running or cannot be used to gather device group state in-formation.

• The device group is not in the PAIR state.

This could be caused by XP Continuous Access Software link failures or manual manipulation of thedisk pair state.

TIP:Recover the PAIR state immediately, because replication of your data is not possible.

Check monitored XP disk pairs by invoking the following command from the command line:

clxchkmon –n application_name –g device_group –show

TIP:Disable application service failover for the time of the XP disk pair recovery (resynchronization). XPCluster Extension's logic is based on the assumption that if the monitor is enabled, immediate actionwill be taken to recover a suspended XP disk pair.

Problem

Resource XYZ: XP Cluster Extension: device group XYZ is not in PAIR state.

This message appears even though the device group is in PAIR state.

Solution

If you are using the pair/resync monitor, the ResyncMonitorInterval must be less than or equal to theresource monitor interval for the XP Cluster Extension resource to prevent erroneous logging.

The ResyncMonitorInterval in XP Cluster Extension defines when the pair/resync monitor checks theactual device group state. This state will be valid and shown until the next update(ResyncMonitorInterval) occurs. If the actual XP disk pair state changes between twoResyncMonitorInterval(s), the PAIR state shown by the pair/resync monitor will not be correct.

The resource monitor checks the status of the XP Cluster Extension resource at the resource monitorinterval of the cluster software. The XP Cluster Extension resource reports the status of the device groupat that interval based on the current state in the pair/resync monitor.

If the ResyncMonitorInterval is set to a higher value than the resource monitor interval for the XP ClusterExtension resource, the pair/resync monitor will update the device group state less often.

However, the XP Cluster Extension resource logs messages only if the device group is not in PAIR stateor if an XP RAID Manager error occurred (for example, if XP RAID Manager is not running).


Example

Set the XP Cluster Extension agent's MonitorInterval attribute to 60 seconds (the default value); thenset the XP Cluster Extension resource ResyncMonitorInterval attribute to less than 60 seconds.

Troubleshooting160

11 Support and other resources

Contacting HPFor worldwide technical support information, see the HP support website:

http://www.hp.com/support

Before contacting HP, collect the following information:

• Product model names and numbers• Technical support registration number (if applicable)• Product serial numbers• Error messages• Operating system type and revision level• Detailed questions

Subscription serviceHP recommends that you register your product at the Subscriber's Choice for Business website:

http://www.hp.com/go/e-updates

After registering, you will receive e-mail notification of product enhancements, new driver versions,firmware updates, and other product resources.

New and changed information in this editionThe following additions and changes have been made for this edition:

• The following information has been updated:• Configuring XP Cluster Extension for Linux

Related informationThe following documents and websites provide related information:

• HP StorageWorks XP Cluster Extension Software Installation Guide• HP StorageWorks XP RAID Manager User's Guide• HP StorageWorks XP Continuous Access Software User Guide• HP StorageWorks XP Continuous Access Software Journal User Guide• HP StorageWorks XP Business Copy Software User Guide• HP StorageWorks SAN Design Reference Guide

You can find these documents on the Manuals page of the HP Business Support Center website:

http://www.hp.com/support/manuals


http://www.hp.com/support

http://www.hp.com/go/e-updates


In the Storage section, click Storage Software, and then select your product.

White papersThe following white papers are available at www.hp.com/storage/whitepapers:

• Live Migration across data centers and disaster tolerant virtualization architecture with HP Storage-Works Cluster Extension and Microsoft Hyper-VTM

• Considerations in HP StorageWorks XP Cluster Extension configurations to stop automatic XP CAdisk pair resynchronization when CA link is suspended

• Implementing HP StorageWorks Cluster Extension for Windows in a VMware Virtual Machine• Migrating HP StorageWorks XP Cluster Extension Quorum Filter Service Implementations to Microsoft

Majority Node Set Quorum Configurations• Migrating an HP Serviceguard for Linux Cluster to a Novell SUSE Linux Enterprise High Availability

Extension Cluster• Migrating an HP Serviceguard for Linux Cluster to Red Hat Cluster Suite in Red Hat Enterprise

Linux 5 Advanced Platform

HP websitesFor additional information, see the following HP websites:

• http://www.hp.com• http://www.hp.com/go/storage• http://www.hp.com/service_locator• http://www.hp.com/support/manuals• http://www.hp.com/storage/spock• www.hp.com/storage/whitepapers• http://docs.hp.com/en/ha.html

Typographic conventionsTable 5 Document conventions

ElementConvention

Cross-reference links and e-mail addressesBlue text: Table 5

Website addressesBlue, underlined text: http://www.hp.com

• Keys that are pressed• Text typed into a GUI element, such as a box• GUI elements that are clicked or selected, such as menu

and list items, buttons, tabs, and check boxes

Bold text

Text emphasisItalic text

• File and directory names• System output• Code• Commands, their arguments, and argument values

Monospace text

Support and other resources162


http://www.hp.com

http://www.hp.com/go/storage

http://www.hp.com/service_locator




http://docs.hp.com/en/ha.html

http://www.hp.com

ElementConvention

• Code variables• Command variablesMonospace, italic text

Emphasized monospace textMonospace, bold text

CAUTION:Indicates that failure to follow directions could result in damage to equipment or data.

IMPORTANT:Provides clarifying information or specific instructions.

NOTE:Provides additional information.

TIP:Provides helpful hints and shortcuts.

HP product documentation surveyAre you the person who installs, maintains, or uses this HP storage product? If so, we would like toknow more about your experience using the product documentation. If not, please pass this notice tothe person who is responsible for these activities.

Our goal is to provide you with documentation that makes our storage hardware and software productseasy to install, operate, and maintain. Your feedback is invaluable in letting us know how we canimprove your experience with HP documentation.

Please take 10 minutes to visit the following web site and complete our online survey. This will provideus with valuable information that we will use to improve your experience in the future.

http://www.hp.com/support/storagedocsurvey

Thank you for your time and your investment in HP storage products.


http://www.hp.com/support/storagedocsurvey

Support and other resources164

Glossary

CHA Channel adapter. A device that provides the interface between the array andthe external host system. Occasionally, this term is used synonymously with theterm channel host interface processor (CHIP).

CLI Command-line interface. An interface comprised of various commands whichare used to control operating system responses.

command device A volume on the disk array that accepts HP StorageWorks Continuous Accessor HP StorageWorks Business Copy control operations which are then executedby the array.

CU Control unit.

CVS Custom volume size. CVS devices (OPEN-x CVS) are custom volumes configuredusing array management software to be smaller than normal fixed-size OPENsystem volumes. Synonymous with volume size customization (VCS).

DLL Dynamic-link library.

DSM Device Specific Module.

failover Disconnecting a failed unit or path and replacing it with an alternative unit orpath to continue functioning.

FC Fibre Channel. A network technology primarily used for storage networks.

fence level A method of setting rejection of XP Continuous Access Software write I/O requestsfrom the host according to the condition of mirroring consistency.

GUI Graphical User Interface.

GUID Globally unique identifier.

HACMP High Availability Cluster Multi-Processing. An IBM application for AIX software.

HBA Host bus adapter.

heartbeat A periodic synchronization signal issued by cluster software or hardware toindicate that a node is an active member of the cluster.

LD, LDEV Logical device. An LDEV is created when a RAID group is carved into piecesaccording to the selected host emulation mode (that is, OPEN-3, OPEN-8,OPEN-9). The number of resulting LDEVs depends on the selected emulationmode. The term LDEV is also known as term volume.

LU Logical unit.

LUN Logical unit number.


LUSE The LUSE feature is available when the HP StorageWorks LUN Manager productis installed, and allows a LUN, normally associated with only a single LDEV, tobe associated with 1 to 36 LDEVs. Essentially, LUSE makes it possible forapplications to access a single large pool of storage.See also LD, LDEV

LVM Logical Volume Manager.

MINAP Minimum active paths.

MMC Microsoft Management Console.

MNS Majority node set.

MSCS Microsoft Cluster Service.

MU Mirror unit.

NIC Network interface card. A device that handles communication between a deviceand other devices on a network.

path A path is created by associating a port, a target, and a LUN ID with one or moreLDEVs. Also known as a LUN.

PCF Product Configuration File.

port A physical connection that allows data to pass between a host and the diskarray. The number of ports on an XP disk array depends on the number ofsupported I/O slots and the number of ports available per I/O adapter. The XPfamily of disk arrays supports FC ports as well as other port types. Ports arenamed by port group and port letter, such as CL1-A, in which “CL1” is the group,and “A” is the port letter.

primary site The data center location that owns the cluster group (quorum resource).

PSUS Pair suspended-split.

P-VOL Primary volume.

quorum In MSCS, a cluster resource that has been configured to control the cluster,maintaining essential cluster data and recovery information. In the event of anode failure, the quorum acts as a tie-breaker and is transferred to a survivingnode to ensure that data remains consistent within the cluster.

RAID Redundant array of independent disks.

SCSI Small Computer Systems Interface. A standard, intelligent parallel interface forattaching peripheral devices to computers, based on a device-independentprotocol.

secondary site The data center location with the mirror copy of the quorum disk pair.

SMIT System Manager Information Tool.

SMPL Simplex.

Glossary166

split-brainsyndrome

A state of data corruption that can occur if a cluster is re-formed as subclustersof nodes at each site, and each subcluster assumes authority, starting the sameset of applications and modifying the same data.

SPOCK Single Point of Connectivity Knowledge website. SPOCK is the primary portalused to obtain detailed information about supported HP StorageWorks productconfigurations.

SPOF Single point of failure.

S-VOL Secondary or remote volume. The copy volume that receives the data from theprimary volume.

SVP Service processor. A notebook computer built into the disk array. The SVPprovides a direct interface to the disk array and used only by the HP servicerepresentative.

TMSCSI Target mode SCSI.

UAC User account control.

UCF User configuration file.

VCS VERITAS Cluster Server.

volume On the XP array, a volume is a uniquely identified virtual storage devicecomposed of a control unit (CU) component and a logical device (LDEV)component separated by a colon. For example 00:00 and 01:00 are twouniquely identified volumes; one is identified as CU = 00 and LDEV = 00, andthe other as CU = 01 and LDEV = 00; they are two unique separate virtualstorage devices within the XP array.,

VSC Volume size customization. Also known as CVS.

VM Virtual Machine.

XP Business CopySoftware

HP StorageWorks XP Business Copy Software. XP Business Copy Software letsyou maintain up to nine local copies of logical volumes on the disk array.

XP ContinuousAccess Software

HP StorageWorks XP Continuous Access Software. XP Continuous Access Softwarelets you create and maintain duplicate copies of local logical volumes on aremote disk array.


Glossary168

Index

Aagent

configuring, 75disabling, 87

APPLICATION sectiondescription, 128

application service failover, 159ApplicationDir object

description, 128ApplicationStartup object

description, 129AsyncTakeoverTimeout object

description, 130AutoFailbackType

description, 48automatic recovery, 149AutoRecover object

description, 131rolling disaster protection, 142

BBasic Resource Health Check Interval

description, 47BCEnabledA object

description, 131BCEnabledB object

description, 132BCMuListA object

description, 132BCMuListB object

description, 132BCResyncEnabledA object

description, 132BCResyncEnabledB object

description, 132BCResyncMuListA object

description, 133BCResyncMuListB object

description, 133

CCLI

XP Cluster Extension, 12

cluster softwareintegration with XP Cluster Extension, 11

ClusterNotifyCheckTimedescription, 133UCF requirement, 64

ClusterNotifyWaitTimedescription, 133UCF requirement, 64

clxhostsupdating, 108

COMMON sectiondescription, 127

configurationconfiguration tool Windows, 37consolidated site, 13Linux, 97, 102Microsoft Cluster Service, 37one to one, 12

configuration informationexporting, 41importing, 41

configuration toolWindows, 37

contacting HP, 161conventions

document, 162

DDataLoseDataCenter object

description, 133DataLoseMirror object

description, 134DC_A_Hosts object

description, 134DC_B_Hosts object

description, 134deleting a device group, 87dependencies

adding (CLI), 66adding (Windows Server 2008/2008 R2),65

Device Mapper Multipath SoftwareRescanning devices, 106

DeviceGroup objectdescription, 135


disaster tolerance, 11disk pairs

XP Continuous Access, 11document

conventions, 162related documentation, 161

documentationHP website, 161providing feedback, 163

Eenabling a service group, 155error return codes

failover, 150exporting

configuration information, 41

FFailoverPeriod

description, 48FailoverThreshold

description, 48fast failback

XP Continuous Access AsynchronousSoftware, 12

FASTFAILBACK valuedescription, 130

FastFailbackEnabled objectdescription, 135VCS setting, 89

featuresXP Cluster Extension, 11

fence levelsXP Continuous Access, 14

FenceLevel objectdescription, 135

filesclxhosts, 108event log, 159force flag, 144services, 26, 78, 108

Filesystems objectdescription, 135

force flagfile, 144

forceflag option, 114

Ggroup names

Microsoft Cluster Service, 42, 44

HHACMP

bringing resource groups online, 30configuring resources, 25custom cluster events, 32, 33failover error handling, 150failure, 35integrating XP Cluster Extension, 26integration with XP Cluster Extension, 25pair/resync monitor, 32restrictions, 35taking resource groups offline, 31timing considerations, 34

helpobtaining, 161

HPtechnical support, 161

Hyper-V Live Migration, 71, 74

Iimporting

configuration information, 41instances

starting and stopping, 22IsAlivePollInterval

description, 47

JJournalDataCurrency object

description, 136

LLInux

timing considerations, 109Linux

Device Mapper Multipath Software, 106pair/resync monitor, 108pair/resync monitor integration, 109

live migration, 64, 71, 74LocalDCLMForNonPAIRDG

description, 136log file

location, 149log files

Microsoft Cluster Service, 73MSCS, 74

LogDir objectdescription, 127

LogLevelUCF requirement, 64

LogLevel objectdescription, 127

170

LooksAlivePollIntervaldescription, 47

Mmajority node set

Microsoft Cluster Service, 15MergeCheckInterval

UCF requirement, 64Microsoft Cluster Service

adding dependencies, 64administration, 73, 74changing resource names, 44, 45configuration example, 66configuration file, 123configuring XP RAID Manager advancedproperties, 58configuring XP RAID Manager device groupdetails, 58configuring XP RAID Manager instances, 57data center assignments, 59group names, 42, 44integration with XP Cluster Extension, 37majority node set, 15resource names, 42, 44

Microsoft Management Console, 46mounting a file system, 121multipath_rescan script, 106

Nnames

changing (Microsoft Cluster Service), 45changing (MSCS), 44

network considerationsXP RAID Manager, 21

Oobjects

APPLICATION section, 128COMMON section, 126XP Cluster Extension, 123

Ppair/resync monitor

configuring for Linux, 108configuring for Microsoft Cluster Service, 38HACMP integration, 32integration with Microsoft Cluster Service,52, 60invoking, 143port, 26, 38, 40, 78, 108troubleshooting, 159

PendingTimeoutdescription, 48

post-execution programsinvoking, 145return codes, 147

PostExecCheck objectdescription, 137

PostExecScript objectdescription, 137

pre-execution programsinvoking, 145return codes, 146

PreExecScript objectdescription, 137

programspost-execution, 145pre-execution, 145

RRaidManagerInstances object

description, 137recommendations

log files, 74recovering PAIR state, 159

recoverydisk pair states, 119procedures, 119sequence, 120

recovery procedure, 60Red Hat Cluster Service, 97, 102

environment file, 97, 102related documentation, 161remote management, 46, 62

Windows Server 2003, 73Windows Server 2008/2008 R2, 73

removing a combination, 115resource groups

HACMP, 30, 31resource names

Microsoft Cluster Service, 42, 44


resourcesadding for Microsoft Cluster Service, 42adding for VCS, 80adding for Windows Server 2008/2008 R2,44adding with the CLI, 44bringing online, 70changing attributes for VCS, 83changing properties for Microsoft ClusterService, 49configuring for Microsoft Cluster Service, 45configuring for VCS, 79deleting for MSCS, 70linking for VCS, 84Microsoft Cluster Service, 45properties (CLI), 63properties (UCF), 64taking offline, 70

Response to resource failuredescription, 47

RestartActiondescription, 47

RestartPerioddescription, 47

RestartThresholddescription, 47

resynchronizing a disk pair, 88, 109ResyncMonitor object

description, 138rolling disaster protection, 142

ResyncMonitorAutoRecover attribute, 71, 87ResyncMonitorAutoRecover object

description, 138ResyncMonitorInterval object

description, 138RESYNCWAIT value

description, 130ResyncWaitTimeout object

description, 138return codes

post-execution, 147pre-executable, 146

RHCSconfiguration file, 124

rolling disaster protection, 12automatic recovery, 142configuration with XP Business CopySoftware, 141pair/resync monitor, 142restoring server operation, 142setting in user configuration file, 141

SSearchObject object

description, 127server

restoring operation, 142service or application

bouncing, 72ServiceGroupHB resources, 84SG recovery, 109starting

errors, 150StatusRefreshInterval

description, 138UCF requirement, 64

Subscriber's Choice, HP, 161

Ttakeover function failure, 147technical support

HP, 161Thorough Resource Health Check Interval

description, 47timing

HACMP considerations, 34Microsoft Cluster Service considerations, 72

timing considerationsLinux, 109

troubleshootingoffline condition (VCS), 90XP Cluster Extension problems, 149

typographic conventions, 162

Uuser configuraiton file

LocalDCLMForNonPAIRDG, 136

172

user configuration fileAPPLICATION section, 128ApplicationDir object, 128ApplicationStartup object, 129AsyncTakeoverTimeout object, 130AutoRecover object, 131BCEnabledA object, 131BCMuListA object, 132BCMuListB object, 132BCResyncEnabledA object, 132BCResyncEnabledB object, 132BCResyncMuListA object, 133BCResyncMuListB object, 133COMMON section, 126, 127DataLoseDataCenter object, 133DataLoseMirror object, 134DC_A_Hosts object, 134DC_B_Hosts object, 134DeviceGroup object, 135FASTFAILBACK value, 130FastFailbackEnabled object, 135FenceLevel object, 135Filesystems object, 135HACMP, 123HACMP customization, 28JournalDataCurrency object, 136LogDir object, 127LogLevel object, 127object formats, 124objects, 123PostExecCheck object, 137PostExecScript object, 137PreExecScript object, 137RaidManagerInstances object, 137ResyncMonitor object, 138ResyncMonitorAutoRecover object, 138ResyncMonitorInterval object, 138RESYNCWAIT value, 130ResyncWaitTimeout object, 138sample, 139SearchObject object, 127specifying object values, 124structure, 124VcsBinPath object, 127Vgs object, 139XPSerialNumbers object, 139

VVCS recovery, 88VcsBinPath object

description, 127VERITAS Cluster Server (VCS)

configuration file, 124integration with XP Cluster Extension, 75

Vgs objectdescription, 139

Wwebsites

HP ,HP Subscriber's Choice for Business, 161product manuals, 161

XXP Business Copy Software

rolling disaster protection, 141XP Cluster Extension

CLI, 12cluster software, 11configurations

consolidated-site, 13one-to-one, 12

configuring with Microsoft Cluster Service,37dependency on XP RAID Manager, 20environments, 14features, 11

XP Continuous Accessconfigurations, 14fence levels, 14pairs, 11

XP Continuous Access Asynchronous Softwarefast failback, 12

XP RAID Manager, 20creating instances, 20device groups, 21network considerations, 21rolling disaster protection, 141setting up, 20starting and stopping instances, 22testing failover and failback, 22

XPSerialNumbers objectdescription, 139


hp storage works xp cluster extension

Documents