powerha systemmirror for linux version 7 - ibm · what's new in powerha systemmirror 7.2 for...

IBM PowerHA SystemMirror for Linux Version 7.2

IBM

Note

Before using this information and the product it supports, read the information in “Notices” on page97.

This edition applies to IBM® PowerHA® SystemMirror® 7.2 for Linux and to all subsequent releases and modifications untilotherwise indicated in new editions.© Copyright International Business Machines Corporation 2017, 2019.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract withIBM Corp.

Contents

About this document..............................................................................................vHighlighting...................................................................................................................................................vCase-sensitivity in Linux...............................................................................................................................vISO 9000.......................................................................................................................................................v

What's new........................................................................................................... 7

Concepts............................................................................................................. 11Compare PowerHA SystemMirror for AIX to PowerHA SystemMirror for Linux...................................... 11High availability clustering for Linux......................................................................................................... 13

High availability and hardware availability.......................................................................................... 13Benefits of PowerHA SystemMirror..................................................................................................... 13High availability clusters...................................................................................................................... 14

Physical components of a PowerHA SystemMirror cluster...................................................................... 14Nodes....................................................................................................................................................15Networks.............................................................................................................................................. 15Clients...................................................................................................................................................15

Managing users and user groups...............................................................................................................15Cluster nodes, networks, and heartbeating..............................................................................................18

Nodes....................................................................................................................................................18Cluster networks.................................................................................................................................. 18Configuring the netmon.cf file .............................................................................................................19IP address takeover............................................................................................................................. 20IP address takeover by using IP aliases..............................................................................................21Heartbeating over TCP/IP and disk......................................................................................................21Split policy............................................................................................................................................ 22Resources and resource groups.......................................................................................................... 25

Cluster configurations................................................................................................................................29Standby configurations........................................................................................................................ 29Takeover configurations.......................................................................................................................30

Planning..............................................................................................................35

Installing............................................................................................................ 39Planning the installation............................................................................................................................ 39Installing PowerHA SystemMirror ............................................................................................................ 40Cluster snapshot........................................................................................................................................ 41Upgrading PowerHA SystemMirror for Linux ........................................................................................... 41

Performing an upgrade operation on an offline cluster...................................................................... 42Performing a rolling upgrade operation ..............................................................................................42

Configuring......................................................................................................... 45Creating a cluster.......................................................................................................................................45Add a cluster.............................................................................................................................................. 46Configuring resources................................................................................................................................48Configuring resource group....................................................................................................................... 50Configuring a Shared File System resource ............................................................................................. 52Verifying the standard configuration......................................................................................................... 53Configuring dependencies between resource groups.............................................................................. 53

iii

Troubleshooting PowerHA SystemMirror.............................................................. 59Troubleshooting PowerHA SystemMirror clusters................................................................................... 59Using PowerHA SystemMirror cluster log files......................................................................................... 59Using the Linux log collection utility..........................................................................................................60Solving common problems........................................................................................................................ 60

PowerHA SystemMirror startup issues................................................................................................60PowerHA SystemMirror disk issues.....................................................................................................62PowerHA SystemMirror resource and resource group issues............................................................ 63PowerHA SystemMirror Fallover issues...............................................................................................64PowerHA SystemMirror additional issues........................................................................................... 65

PowerHA SystemMirror GUI.................................................................................67Planning......................................................................................................................................................67Installing.................................................................................................................................................... 69Logging in to the PowerHA SystemMirror GUI ......................................................................................... 70Navigating.................................................................................................................................................. 71Importing multiple clusters.......................................................................................................................73Smart Assist for SAP HANA....................................................................................................................... 74Troubleshooting.........................................................................................................................................75Configuring................................................................................................................................................. 77

Adding users and assigning roles........................................................................................................ 77Logging in as a non-root user...............................................................................................................78Configuring the PowerHA SystemMirror GUI server to be highly available........................................79Limiting access to the PowerHA SystemMirror GUI............................................................................80Restricting the use of IP address in the GUI....................................................................................... 81

Smart Assist for PowerHA SystemMirror Smart Assist for PowerHASystemMirror...................................................................................................83PowerHA SystemMirror for SAP HANA..................................................................................................... 83

Planning for SAP HANA........................................................................................................................ 84Configuring the Smart Assist application for SAP HANA.....................................................................85

PowerHA SystemMirror for SAP NetWeaver............................................................................................. 87Planning for SAP NetWeaver................................................................................................................87Installing an SAP system..................................................................................................................... 88Configuring SAP NetWeaver.................................................................................................................92

Troubleshooting PowerHA SystemMirror Smart Assist issues.................................................................95PowerHA SystemMirror is not able to harvest some values during Wizard execution.......................95Replication Site Name that is configured in Smart Assist wizard is different from Replication

Site Name shown in SAP HANA setup wizard................................................................................ 95Smart Assist policy fails to activate..................................................................................................... 95Smart Wizard does not detect or show one or more Ethernet interfaces in the list.......................... 95

Notices................................................................................................................97Privacy policy considerations.................................................................................................................... 98Trademarks................................................................................................................................................ 99

Index................................................................................................................ 101

iv

About this document

This document provides information about how you can configure, maintain, and monitor clusters withPowerHA SystemMirror for Linux.

HighlightingThe following highlighting conventions are used in this document:

Bold Identifies commands, subroutines, keywords, files, structures, directories, and otheritems whose names are predefined by the system. Bold highlighting also identifiesgraphical objects, such as buttons, labels, and icons that the you select.

Italics Identifies parameters for actual names or values that you supply.

MonospaceIdentifies examples of specific data values, examples of text similar to what youmight see displayed, examples of portions of program code similar to what you mightwrite as a programmer, messages from the system, or text that you must type.

Case-sensitivity in LinuxEverything in the Linux operating system is case-sensitive, which means that it distinguishes betweenuppercase and lowercase letters. For example, you can use the ls command to list files. If you type LS,the system responds that the command is not found. Likewise, FILEA, FiLea, and filea are threedistinct file names, even if they reside in the same directory. To avoid causing undesirable actions to beperformed, always ensure that you use the correct case.

ISO 9000ISO 9000 registered quality systems were used in the development and manufacturing of this product.

© Copyright IBM Corp. 2017, 2019 v

vi PowerHA SystemMirror for Linux Version 7.2

What's new in PowerHA SystemMirror 7.2 for Linux

Read about new or significantly changed information in PowerHA SystemMirror 7.2 for Linux.

How to see what's new or changed

To help you see where technical changes have been made, the information center uses:

• The image to mark where new or changed information begins.

• The image to mark where new or changed information ends.

What's new in PowerHA SystemMirror Version 7.2.3 for Linux with Service Pack 1PowerHA SystemMirror support on RHEL 7.7 and RHEL 8.0

PowerHA SystemMirror Version 7.2.3 for Linux with Service Pack 1 is supported on Red Hat EnterpriseLinux (RHEL) 7.7 and RHEL 8.0

PowerHA SystemMirror support on SLES 15 Service Pack 1PowerHA SystemMirror Version 7.2.3 for Linux with Service Pack 1 is supported on SUSE LinuxEnterprise Server (SLES) 15 Service Pack 1.

Support for Cost Optimized configuration of SAP HANAThe Cost Optimized configuration mode uses the secondary node for running a non-productive SAPHANA instance. The maximum amount of memory that is used by the productive SAP HANA instanceon the secondary node is limited by configuring a value for the Global Allocation Limitparameter and by disabling the table preload. Whenever takeover must occur, PowerHA SystemMirrorstops the non-productive SAP HANA instance first and then takeover occurs. For more information,see PowerHA SystemMirror for SAP HANA.

Multinode disk heartbeatPowerHA SystemMirror Version 7.2.3 for Linux with Service Pack 1, or later supports up to four nodesfor disk heartbeat. For more information, see Heartbeating over disk.

Enable cluster services to start or stop automatically on a node after system restartYou can bring the cluster services online or offline on a node after the system restarts by configuringthe START_ON_BOOT parameter. For more infraction see, Adding a node to a cluster.

Enhancements in Smart Assist

• If you specify the value for the DEL_POLICY parameter as YES in the Delete Smart Assist operation,the configured srHook files and configuration that is related to PowerHA SystemMirror are deleted.For more information, see SAP HANA operations.

• Enhanced the Setup Smart Assist operation for a Smart Assist policy by adding thePREFER_TAKEOVER parameter. The PREFER_TAKEOVER parameter defines whether PowerHASystemMirror must switch over the SAP HANA database to the secondary node when the primarySAP HANA instance fails instead of restarting the SAP HANA instance locally on the primary node.For more information, see PowerHA SystemMirror for SAP HANA.

Enhancements in PowerHA SystemMirror graphical user interface (GUI)The following updates are new in PowerHA SystemMirror GUI:

• Support for Cost Optimized configuration of SAP HANA.• Create and manage resource group dependencies.• Non-root support for creating clusters and cloning clusters from snapshots.• Configure PowerHA SystemMirror GUI server to be highly available by using the High Availability

wizard option in the PowerHA SystemMirror GUI.• Import multiple clusters by using the Add Multiple Clusters wizard.

© Copyright IBM Corp. 2017, 2019 7

• Enhanced activity log to display the activity ID, start time and end time of the activity, and theduration of the activity. You can also view details such as the number of successful login attemptsand failed login attempts, the number of new activities, and the number activities that are notviewed.

• Enhanced security features with options to disable anonymous login and global access.• Automatic download and installation of the remaining files that are required to complete the

PowerHA SystemMirror GUI installation process from the IBM website by using the smuiinst.kshcommand.

For more information, see PowerHA SystemMirror graphical user interface (GUI).

June 2019

The following information is a summary of the updates that are made to the PowerHA SystemMirror forLinux documentation:

• Smart Assist for SAP NetWeaver updates:

– Added information about Standalone Enqueue Server 2 support and updated the prerequisites forconfiguring SAP NetWeaver in the Planning for SAP NetWeaver topic.

– Added information about the INSTALL_TYPE and MODE parameters in the Configuring SAPNetWeaver topic.

– Updated information about the Delete Smart Assist operation in the SAP NetWeaver operations topic.– Added information about High availability of SAP NetWeaver on up to four-nodes in the Planning for

SAP NetWeaver topic.• Smart Assist for SAP HANA updates:

– Added information about SAP HANA Active/Active (read enabled) secondary configuration modesupport through Smart Assist in the PowerHA SystemMirror for SAP HANA topic.

– Updated information about the Delete Smart Assist operation in the SAP HANA operations topic.– Added information about the MODE and CONFIG parameters in the Configuring the Smart Assist

application for SAP HANA topic.• Added the Smart Assist for SAP HANA topic that describes the Smart Assist wizard that is available in

the PowerHA SystemMirror graphical user interface.• Added information about the Smart Assist wizard in the Navigating the PowerHA SystemMirror GUI

topic.• Updated information about the support for the following operating systems in the Planning for PowerHA

SystemMirror for Linux topic:

– SUSE Linux Enterprise Server (SLES) for SAP 12 SP4– SUSE Linux Enterprise Server (SLES) for SAP 15– Red Hat Enterprise Linux (RHEL) 7.6

• Updated migration information in the following topics:

– Upgrading PowerHA SystemMirror for Linux– Performing an upgrade operation on an offline cluster– Performing a rolling upgrade operation

• Updated the syntax for the file_collection, snapshot, and smartassist classes in the clmgrcommand topic.

June 2018

The following information is a summary of the updates that are made to the PowerHA SystemMirror forLinux documentation:

8 PowerHA SystemMirror for Linux Version 7.2

• Updated the following topics with information that is related to StartAfter relationship between theSAP NetWeaver system (application server) and the SAP HANA High Availability resources:

– Verifying the SAP NetWeaver High Availability• Added the Managing users and user groups topic that describes users and user groups management.• Added the /var/pha/log/appmon.log log file in the Using cluster log files topic.• Updated the following topics for the Shared Storage functionality:

– Highly available resources are in the Failed offline state– Resource group has Stuck Online state– Planning the installation of PowerHA SystemMirror for Linux– Installing PowerHA SystemMirror for Linux– PowerHA SystemMirror failed to create a cluster– Configuring a Shared File System resource– Verifying the standard configuration– Configuring resource groups for PowerHA SystemMirror for Linux– Configuring resources for PowerHA SystemMirror for Linux– Creating a cluster for PowerHA SystemMirror for Linux– File system

• Update the clmgr command with two new operations.• Updated information about resource groups in the clRGinfo command topic.

What's new in PowerHA SystemMirror 7.2 for Linux 9

PowerHA SystemMirror for Linux conceptsThe following information introduces important concepts you must understand before you can use thePowerHA SystemMirror for Linux.

Compare PowerHA SystemMirror for AIX to PowerHA SystemMirror for LinuxThe following table compares different functions that are shared between PowerHA SystemMirror forAIX® and PowerHA SystemMirror for Linux.

Table 1. Compare PowerHA SystemMirror for AIX to PowerHA SystemMirror for Linux

Functions PowerHA SystemMirror for AIX PowerHA SystemMirror for Linux

Installation The SP release contains only themodified filesets, so the base buildneeds to be installed beforeinstalling a service pack.

The PowerHA Linux release packaging forthe SP build is similar to the base build. So,there is no difference in the installation ofthe base and the SP build and for the SPbuild installation, you do not need to installthe base build first.

Configuration Any configuration change that isdone by the user is saved only inthe local Configuration Database(ODM) and is applied to thedifferent cluster nodes aftersynchronization procedure.

If any configuration changes are done bythe user, no separate synchronizationprocedure is needed and the configurationchange is applied immediately to all clusternodes.

Split policy(manual)

For the 2-node cluster in PowerHASystemMirror for AIX, if one of thenodes goes down, the other nodetakes over the resources of theresource group even when splitpolicy is set to the Manual state.

In PowerHA SystemMirror for Linux, whensplit policy is set to the Manual state for the2-node cluster, and if one node goes down,then manual intervention is needed byusing the runact command so that theresources of the resource group areacquired by the other node.

Split policy(manual)

When the split operation happens,the PowerHA SystemMirror for AIXsoftware broadcasts a message toall terminals that indicates splitoperation and starts the relevantscript to continue or to getrebooted as a recovery action.

When the split operation happens, the userneeds to verify the split operation by usingthe following command lssrc -lsIBM.RecoveryRM |grep"Operational Quorum State". Thevalue PENDING_QUORUM indicates splitoperation and the user needs to run therunact command with the relevant valueto continue or to get rebooted as a recoveryaction.

Split handling(tiebreaker)

The Tiebreaker policy can be usedwith any number of cluster nodes.

PowerHA SystemMirror for Linux usestiebreaker only when there is a tie and thetotal number of nodes that are configuredin a cluster is even.


Table 1. Compare PowerHA SystemMirror for AIX to PowerHA SystemMirror for Linux (continued)

Functions PowerHA SystemMirror for AIX PowerHA SystemMirror for Linux

Split handling(tiebreaker)

When a tiebreaker is configuredand more than half of the nodes arein the shutdown state (for example,two nodes are down in a 3-nodecluster), then also PowerHASystemMirror for AIX can start theresources on a node that is up andrunning.

PowerHA SystemMirror for Linux uses theNormal quorum type of Reliable ScalableCluster Technology (RSCT) to create a peerdomain cluster. This operation requires thatat least half-of-the nodes must be up. So iftwo nodes are down in a 3-node cluster,then even if a tiebreaker is configured, thenode that is up would not be able to acquirethe resources.

Stop behavior ofStartAfterdependency

The StartAfter dependency doesnot impact the stop behavior of theresource groups.

If two resource groups have StartAfterdependency, the target resource groupcannot be stopped while the sourceresource group is online.

Dependency (anti-collocated)

The resource groups with anti-collocated dependency are neveronline on the same node.

If the target resource group with anti-collocated dependency is brought onlinewhen the source resource group is alreadyonline on a node, and the intended node ofthe target resource group is not available,the resource group becomes online on thesame node where source resource group isalso active.

Dependency(collocated)

For two resource groups withcollocated dependency, failure ofany one of the resource groupstriggers movement of both theresource groups to another node.

For two resource groups with collocateddependency, if any one of the resourcegroups fails, then the PowerHASystemMirror software does not move boththe resource groups to another node.

unmanageparameter (clusteror node-levelparameter)

The unmanage parameter isavailable in the PowerHASystemMirror for AIX. Thisparameter allows the workload torun without being managed byPowerHA SystemMirror.

This parameter is not available in thePowerHA SystemMirror for Linux Version7.2.

START_ON_BOOTparameter (node-level parameter)

Using the START_ON_BOOTparameter, you can automaticallybring the cluster services online oroffline on a node after the nodereboot.

Using the START_ON_BOOT parameter, youcan automatically bring the cluster servicesonline or offline on a node after the nodereboot. The START_ON_BOOT parameter isavailable in PowerHA SystemMirror Version7.2.3 for Linux with Service Pack 1 or later.

Applicationconfiguration

The application controller and theapplication monitor are separatelycreated by using the clmgrfunction.

A single application class in clmgr is usedto configure the start, stop, and monitorscripts for a specific application.


High availability clustering for LinuxThe IBM PowerHA SystemMirror software provides a low-cost commercial computing environment thatensures quick recovery of mission-critical applications from hardware and software failures.

PowerHA SystemMirror monitors the cluster resources for failures, and when a problem is detected,PowerHA SystemMirror moves the application (along with resources that ensure access to theapplication) to another node in the cluster.

High availability and hardware availabilityHigh availability software is sometimes confused with simple hardware availability. Fault tolerant,redundant systems (such as RAID) and dynamic switching technologies (such as DLPAR) provide recoveryof certain hardware failures, but do not provide the full scope of error detection and recovery required tokeep a complex application highly available.

A modern, complex application requires access to all of these components:

• Nodes (CPU, memory)• Network interfaces (including external devices in the network topology)• Disk or storage devices• Application software

Surveys of the causes of downtime show that actual hardware failures account for only a smallpercentage of unplanned outages. Other contributing factors include:

• Operator errors• Environmental problems• Application and operating system errors.

Reliable and recoverable hardware simply cannot protect against failures of all these different aspects ofthe configuration. Keeping these varied elements, and therefore the application, highly available requires:

• Thorough and complete planning of the physical and logical procedures for access and operation of theresources on which the application depends. These procedures help to avoid failures in the first place.

• A monitoring and recovery package that automates the detection and recovery from errors.• A well-controlled process for maintaining the hardware and software aspects of the clusterconfiguration while keeping the application available.

Benefits of PowerHA SystemMirrorPowerHA SystemMirror has many benefits.

PowerHA SystemMirror helps you with the following:

• The PowerHA SystemMirror planning process and documentation include tips and advice on the bestpractices for installing and maintaining a highly available PowerHA SystemMirror cluster.

• Once the cluster is operational, PowerHA SystemMirror provides the automated monitoring andrecovery for all the resources on which the application depends.

• PowerHA SystemMirror provides a full set of tools for maintaining the cluster while keeping theapplication available to clients.

PowerHA SystemMirror allows you to:

• Quickly and easily setup a cluster by using the clmgr command-line interface or the applicationconfiguration assistants (Smart Assists).

• Ensure high availability of applications by eliminating single points of failure in a PowerHA SystemMirrorenvironment.

• Manage how a cluster handles component failures.

PowerHA SystemMirror for Linux concepts 13

• Secure cluster communications.• Monitor PowerHA SystemMirror components and diagnose problems that might occur.

High availability clustersAn application cluster is a group of loosely coupled machines networked together, sharing disk networkand other resources.

In a high availability cluster, multiple server machines cooperate to provide a set of services or resourcesto clients.

Physical components of a PowerHA SystemMirror clusterPowerHA SystemMirror provides a highly available environment by identifying a set of resources essentialto uninterrupted processing of an application. It also defines a protocol that nodes use to collaborate toensure that these resources are available.

PowerHA SystemMirror extends the clustering model by defining relationships among cooperatingprocessors where one processor provides the service offered by a peer should the peer be unable to doso. As shown in the following figure, a PowerHA SystemMirror cluster is made up of the following physicalcomponents:

• Nodes• Networks• Clients.

The PowerHA SystemMirror software allows you to combine physical components into a wide range ofcluster configurations, providing you with flexibility in building a cluster that meets your processing andavailability requirements. This figure shows one example of a PowerHA SystemMirror cluster. OtherPowerHA SystemMirror clusters could look very different - depending on the number of processors, thechoice of networking and disk technologies, and so on.

Figure 1. Example of a PowerHA SystemMirror cluster


PowerHA SystemMirror nodesNodes form the core of a PowerHA SystemMirror cluster. A node is a processor that runs Linux, thePowerHA SystemMirror software, and the application software.

In a PowerHA SystemMirror cluster, each node is identified by a unique name. A node has access to a setof resources: networks, network addresses, and applications. Typically, a node runs a server or a backend application that accesses data on the shared external disks.

NetworksAs an independent, layered component of the Linux operating system, the PowerHA SystemMirrorsoftware is designed to work with any TCP/IP-based network.

Nodes in a cluster use the network to:

• Allow clients to access the cluster nodes• Enable cluster nodes to exchange heartbeat messages

Types of networks

The PowerHA SystemMirror software defines two types of communication networks, characterized bywhether these networks use communication interfaces based on the TCP/IP subsystem (TCP/IP-based)or disk-based networks.

TCP/IP-based networkConnects two or more server nodes, and optionally allows client access to these cluster nodes, usingthe TCP/IP protocol.PowerHA SystemMirror uses only unicast communication for heartbeat.

Disk heartbeatProvides communication between PowerHA SystemMirror cluster nodes to monitor the health of thenodes, networks and network interfaces, and to prevent cluster partitioning

ClientsA client is a processor that can access the nodes in a cluster over a local area network.

Clients each run a "front end" or client application that queries the server application running on thecluster node. The PowerHA SystemMirror software provides a highly available environment for criticaldata and applications on the cluster nodes. The PowerHA SystemMirror software does not make theclients themselves highly available.

Managing users and user groupsThe clmgr command manages user accounts and user groups on all nodes in a cluster by makingconfiguration changes on any node in a cluster. The clmgr command also supports Lightweight DirectoryAccess Protocol (LDAP) for users and user groups management.

PowerHA SystemMirror manages Linux and LDAP users and user groups across all PowerHA SystemMirrorclusters. PowerHA SystemMirror user groups provide an additional level of security and enable thesystem administrators to change the configuration settings of a group of users as a single entity.

Managing Linux and LDAP user accounts in a PowerHA SystemMirror cluster

You can use the clmgr command to manage users and user groups from any node in a cluster. If youcreate a user that already exist on any node in the cluster, the operation might fail.

The following Linux files, which store user account information, must be consistent across all clusternodes.

• The /etc/passwd system file• Other system files in the /etc/security directory


Note: If a cluster node fails, users can log on to the operating nodes without experiencing any issues thatare caused by mismatched user or group IDs.

As the system administrator of a PowerHA SystemMirror, you can use the clmgr command to manageuser and user groups from any node in a cluster. The clmgr command propagates new and updatedinformation to all of the other nodes in the cluster.

To configure user accounts on an LDAP server, you must have an installation of the following software:

• openLDAP Server• openLDAP Client

To add a user on the local system, enter the following command:

clmgr add user Local_User

where,

• The Local_User parameter is the user name of the user account that must be added on the local systemand on the other nodes of the PowerHA SystemMirror cluster.

Note: If a node in a cluster is in an Offline state, you cannot create a user on that node.

To add an LDAP user on the local system, enter the following command:

clmgr add user LDAP_User registry=ldap DN=<distinguished_Name>

where,

• The DN parameter is the distinguished name, which is created when you set up the openLDAP server.The clmgr command prompts for an LDAP password, which is the LDAP administrator password usedwhen you set up the openLDAP server.

The clmgr add user command includes the following attributes for the Local user or an LDAP user:

Table 2. Local user and LDAP user attributes

Field name Description

user_name A login name that identifies this user account onthe system.

ID Defines a unique decimal integer string toassociate with this user account on the system.

ADMINSTRATIVE Select True if this user will be an administrativeuser. Otherwise, select False.

CHANGE_ON_NEXT_LOGIN Forces user to change the password on next login.

PRIMARY Specifies the group name of the user to which theuser belongs when the user logs in for the firsttime.

GROUPS Specifies name of all user groups to which a userbelongs. A user can access authority to theprotected resources.

ROLES Creates and manages role-based access control(RBAC) roles for users and user groups. You canuse these roles to control which commands can beexecuted by different sets of users of PowerHASystemMirror.


Table 2. Local user and LDAP user attributes (continued)


REGISTRY Indicates where the information about the newuser must be stored. If the user is being definedlocally and within the Linux environment, the valueLOCAL must be specified. If the user is defined ona remote server, such as the LDAP server, the valueLDAP must be used.

DN A DN is a sequence of relative distinguished names(RDN) separated by commas. The DN attribute isrelevant when the registry attribute is set to LDAP.

HOME Specifies the home directory of a user.

SHELL Specifies the default shell for user.

EXPIRATION Specifies the expiration date of the password foruser.

LOCKED Locks the password of the specified account.

DAYS_TO_WARN Defines the number of days before the systemissues a warning that a password change isrequired.

LOCKOUT_DELAY Specifies the time period, in weeks, betweenpassword expiration and lockout.

MAX_PASSWORD_AGE Defines the maximum age (in weeks) for the userpassword.

MIN_PASSWORD_AGE Defines the minimum age (in weeks) for the userpassword.

Managing user group accounts in a PowerHA SystemMirror cluster

Similar to users, the user groups can be configured on PowerHA SystemMirror Linux setup on the localsystem (files) and on an LDAP server.

To configure user accounts on an LDAP server, you must have an installation of the following software:

• openLDAP Server• openLDAP Client

To add a group on the local system, enter the following command:

clmgr add group User_group

The clmgr` command adds a group on the local system and on the other nodes of the cluster.

Note: The user groups are not created on the nodes, which are in an Offline state in the cluster.

To add a group in LDAP, enter the following command:

clmgr add group User_group registry=ldap DN=<distinguished_name>

The following attributes can be used to add a group:


Table 3. Group attributes


ID Defines a unique decimal integer string toassociate with the user account on the system.

ADMINSTRATIVE Select True if this user will be an administrativeuser. Otherwise select False.

USERS Specifies the names of the users that belong touser group. The members of a group can access(that is, read, write, or run) a resource or file that isowned by another member of the group asspecified by the access control list of the resource.

REGISTRY Indicates where the information about the newuser must be stored. If the user is being definedlocally and within the Linux environment, the valueLOCAL must be specified. If the user is defined ona remote server, such as the LDAP server, the valueLDAP must be used.

DN A DN is a sequence of relative distinguished names(RDN) separated by commas. The DN attribute isrelevant when the registry attribute is set to LDAP.

PowerHA SystemMirror cluster nodes, networks, and heartbeating conceptsThis section introduces major cluster topology-related concepts and definitions that are used throughoutthe documentation and in the PowerHA SystemMirror user interface.

NodesA node is a processor that runs both Linux and the PowerHA SystemMirror software.

Nodes might share a set of resources such as, networks, network IP addresses, and applications. ThePowerHA SystemMirror software supports up to 4 nodes in a cluster. In a PowerHA SystemMirror cluster,each node is identified by a unique name. In PowerHA SystemMirror, a node name and a hostname mustbe the same. Nodes serve as core physical components of a PowerHA SystemMirror cluster.

Cluster networksCluster nodes communicate with each other over communication networks.

If one of the physical network interface cards on a node on a network fails, PowerHA SystemMirrorpreserves the communication to the node by transferring the traffic to another physical network interfacecard on the same node. If a "connection" to the node fails, PowerHA SystemMirror transfers resources toanother node to which it has access.

In addition, the clustering software sends heartbeats between the nodes over the cluster networks toperiodically check on the health of the cluster nodes themselves. If the clustering software detects noheartbeats from a node, a node is considered as failed and resources are automatically transferred toanother node.

It is highly recommend configuring multiple communication paths between the nodes in the cluster.Having multiple communication networks prevents cluster partitioning, in which the nodes within eachpartition form their own entity. In a partitioned cluster, it is possible that nodes in each partition couldallow simultaneous non-synchronized access to the same data. This can potentially lead to differentviews of data from different nodes.


Physical and logical networksA physical network connects two or more physical network interfaces.

Note: If you are considering a cluster where the physical networks use external networking devices toroute packets from one network to another, consider the following: When you configure a PowerHASystemMirror cluster, PowerHA SystemMirror verifies the connectivity and access to all interfaces definedon a particular physical network. However, PowerHA SystemMirror cannot determine the presence ofexternal network devices such as bridges and routers in the network path between cluster nodes. If thenetworks have external networking devices, ensure that you are using devices that are highly availableand redundant so that they do not create a single point of failure in the PowerHA SystemMirror cluster.

A logical network is a portion of a physical network that connects two or more logical network interfacesor devices. A logical network interface or device is the software entity that is known by an operatingsystem. There is a one-to-one mapping between a physical network interface/device and a logicalnetwork interface/device. Each logical network interface can exchange packets with each logical networkinterface on the same logical network.

If a subset of logical network interfaces on the logical network needs to communicate with each other(but with no one else) while sharing the same physical network, subnets are used. A subnet mask definesthe part of the IP address that determines whether one logical network interface can send packets toanother logical network interface on the same logical network.

Logical networks in PowerHA SystemMirrorPowerHA SystemMirror has its own, similar concept of a logical network.

All logical network interfaces in a PowerHA SystemMirror network can communicate PowerHASystemMirror packets with each other directly. Each logical network is identified by a unique name. APowerHA SystemMirror logical network might contain one or more subnets.

PowerHA SystemMirror communication interfacesA PowerHA SystemMirror communication interface is a grouping of a logical network interface, a serviceIP address and a service IP label that you defined in PowerHA SystemMirror.

PowerHA SystemMirror communication interfaces combine to create IP-based networks. A PowerHASystemMirror communication interface is a combination of:

• A logical network interface is the name to which Linux resolves a port (for example, eth0) of a physicalnetwork interface card.

• A service IP address is an IP address (for example, 129.9.201.1) over which services, such as anapplication, are provided, and over which client nodes communicate.

• A service IP label is a label (for example, a hostname in the /etc/hosts file, or a logical equivalent of aservice IP address, such as node_A_en_service) that maps to the service IP address.

Communication interfaces in PowerHA SystemMirror are used in the following ways:

• A communication interface refers to IP-based networks and network interface cards (NIC). The NICsthat are connected to a common physical network are combined into logical networks that are used byPowerHA SystemMirror.

• Each NIC is capable of hosting several TCP/IP addresses. When configuring a cluster, you must definethe IP addresses that PowerHA SystemMirror monitors (base or boot IP addresses), and the IPaddresses that PowerHA SystemMirror keeps highly available (the service IP addresses).

• Heartbeating in PowerHA SystemMirror occurs over communication interfaces. PowerHA SystemMirroruses the heartbeating facility of RSCT. For more information, see the “Heartbeating over TCP/IP anddisk” on page 21 topic.

Configuring the netmon.cf fileThe netmon.cf file is an optional configuration file that customers can use to augment the pingoperation on the available target hosts on the network. The target hosts that are not defined to be part of


the cluster are not available from the cluster nodes. The target hosts can be accessed from the IPaddresses being monitored by topology services.

If you are running a single-node or two-node cluster, you must configure netmon.cf file to detect thenetwork interface failures.

PowerHA SystemMirror periodically attempts to contact each network interface in the cluster. If theattempt to contact an interface fails on one node of a two-node cluster, the corresponding interface onthe other node is also flagged as an Offline node. The other node is flagged as an Offline node because itdoes not receive a response from the peer cluster node. To avoid such behavior, the PowerHASystemMirror must be configured to contact a network instance outside of the cluster. You can use thedefault gateway of the subnet from which the PowerHA SystemMirror GUI is used.

To configure the netmon.cf file, complete the following steps:

1. Configure the netmon.cf file to check the status of the network monitored by the virtual switch.2. On each node, create the following file: /var/ct/cfg/netmon.cf

Note: Each line of netmon.cf file contains the system name or the IP address of the external networkinstance. IP addresses can be specified in the dotted decimal notation.

Example of the netmon.cf file

The following example shows the configuration result for the netmon.cf file.

#This is default gateway for all interfaces in the subnet 192.168.1.0 192.168.1.1

If you are using the Virtual I/O Server (VIOS), the configuration test becomes unreliable because thenetmon.cf file cannot determine whether the inbound traffic is received from the VIOS or a client. TheLPAR cannot distinguish a virtual adapter and a real adapter. To address this problem, the netmon librarysupports up to 32 targets for each local network adapter. If the ping operation for any of these targets issuccessful, the local adapter is considered to be in an Online state. The targets can be specified in thenetmon.cf file with the !REQD keyword. For example:

!REQD <owner><target>

The targets can be also specified in the netmon.cf file by entering the following command:

!IBQPORTONLY !ALL

Location/var/ct/cfg/netmon.cf

Location of the netmon.cf file in a PowerHA SystemMirror environment.

IP address takeoverIP address takeover is a mechanism for recovering a service IP label by moving it to another networkinterface card (NIC) on another node, when the initial NIC fails.

IPAT occurs if the physical network interface card on one node fails and if there are no other accessiblephysical network interface cards on the same network on the same node. Therefore, swapping IP labelsof these NICs within the same node cannot be performed and PowerHA SystemMirror will use IPAT torecover the service IP address by using a NIC on a backup node. IP address takeover keeps the IPaddress highly available by recovering the IP address after failures. PowerHA SystemMirror uses amethod called IPAT via IP aliases.


IP address takeover by using IP aliasesDefining IP aliases to network interfaces allows creation of more than one IP label and address on thesame network interface.

When a resource group containing the service IP label falls over from the primary node to the target node,the service IP labels are added (and removed) as alias addresses on top of the base IP addresses on anavailable NIC. This allows a single NIC to support more than one service IP label placed on it as an alias.Therefore, the same node can host more than one resource group at the same time.

When there a multiple interfaces on the same node connected to the same network, and those interfacesare not combined into a Ethernet Aggregation, all boot addresses must all be on different subnets. Also,any persistent addresses or service addresses must be on different subnets than the boot addresses.

Because IP aliasing allows coexistence of multiple service labels on the same network interface, you canuse fewer physical network interface cards in your cluster. Upon fallover, PowerHA SystemMirror equallydistributes aliases between available network interface cards.

Heartbeating over TCP/IP and diskA heartbeat is a type of a communication packet that is sent between nodes. Heartbeats are used tomonitor the health of the nodes, networks and network interfaces, and to prevent cluster partitioning.

In order for a PowerHA SystemMirror cluster to recognize and respond to failures, it must continuallycheck the health of the cluster. Some of these checks are provided by the heartbeat function.

Each cluster node sends heartbeat messages at specific intervals to other cluster nodes, and expects toreceive heartbeat messages from the nodes at specific intervals. If messages stop being received,PowerHA SystemMirror recognizes that a failure has occurred. Heartbeats can be sent over:

• TCP/IP networks• A physical volume (disk) which is accessible from all clusters nodes

The heartbeat function is configured to use specific paths between nodes. This allows heartbeats tomonitor the health of all PowerHA SystemMirror networks and network interfaces, as well as the clusternodes themselves.

The heartbeat paths are set up automatically by RSCT; you have the option to configure disk paths as partof PowerHA SystemMirror configuration.

Heartbeating over Internet Protocol networksPowerHA SystemMirror relies on RSCT to provide heartbeating between cluster nodes over InternetProtocol networks.

Heartbeating between cluster nodes can be configured by using the following command:

clmgr add network netw_0 TYPE=ether pvid=qoPDC7-G169-UJv3-zHI5-KePI-Whjy-K73nrb nodes= nodeA, nodeB

Heartbeating over diskYou can configure a disk heartbeat optionally. After configuration, the Reliable Scalable ClusterTechnology (RSCT) automatically exchanges heartbeat over the shared disk. These connections canprovide an alternative heartbeat path for a cluster that uses a single TCP/IP-based network.

You can configure a disk heartbeat by using following command:

clmgr add network netw_0 TYPE=disk pvid=qoPDC7-G169-UJv3-zHI5-KePI-Whjy-K73nrb nodes= nodeA, nodeB

Once the disk heartbeat configuration is complete, you can verify by using following command:

clmgr query network netw_0

To check whether the heartbeat interface is properly functioning, run the following command:


lssrc -ls cthats

The following output is generated:

Subsystem Group PID Statuscthats cthats 372762 active

Network Name Indx Defd Mbrs St Adapter ID Group IDdhbTEST_CG [1] 2 2 S 255.255.10.1 255.255.10.1dhbTEST_CG [1] dhb-1-2 0x01c8f4fe 0x01c8f511

In the specified sample command output, find the disk heartbeat stanza by locating the name of the diskheartbeat network, which is obtained by using the clmgr query network command output. The nameof the disk heartbeat interface resource is also displayed.

Positive result: Disk heartbeat is occurring if the Defd and Mbrs parameters are the same number:

Defd Mbrs2 2

Negative or error result: Disk heartbeat is not occurring if the Defd and Mbrs parameters are different:

Defd Mbrs2 1

If you get negative or error result for the disk heartbeat process, recheck the following options:

• Port VLAN ID (PVID) on which disk heartbeat network is configured is available on both the clusternodes.

• Storage connectivity with both the cluster nodes is working.• Physical volume on which disk heartbeat is configured is not being used by any other process, which

might block the IO on the device.

To remove the existing heartbeat network, enter the following command:

clmgr delete network netw_0

To modify an existing heartbeat network, enter the following command:

clmgr modify network netw_0 node=nodeA, nodeB, nodeC

Where, the node parameter lists the node that must be modified. PowerHA SystemMirror supports diskheartbeat for a four-node cluster.

You can use the clmgr modify network command only for the disk type of network.

Split policyA cluster split event can occur when a group of nodes cannot communicate with the remaining nodes in acluster. A cluster split event splits the cluster into two or more partitions.

After a cluster split event, the subdomain that has most of nodes are retained and the other subdomainsare deleted. If exactly half of the nodes of a domain are online and the remaining nodes are inaccessible,PowerHA SystemMirror must determine which subdomain has operational quorum by using ReliableScalable Cluster Technology (RSCT). The subdomain with an operational quorum is retained and the othersubdomains are deleted.

You can use PowerHA SystemMirror to configure a split policy that specifies the response to a cluster splitevent.

To configure a split policy, the following options are available:


None

The None split policy allows each partition that is created by the cluster split event to become anindependent cluster and each partition is started independently of the other partition. A user cancheck the quorum status by using the following command: lssrc -ls IBM.RecoveryRM

Before a split event, operational quorum state is in HAS_QUORUM state and the configuration quorumis TRUE. A user can do both operational change such as moving the cluster online or offline andconfiguration change such as adding resource group or deleting resource group into the cluster.

However, after the spilt event, for each partition, the operational quorum state will be HAS_QUORUMbut configurational quorum state will be FALSE. Hence, PowerHA SystemMirror and the user canperform operational change but not the configurational change.

For example, in two-nodes cluster, if the cluster split event occurs, all the resources are online onboth the nodes. As both the nodes have quorum, if the merge event occurs, lower priority node isrebooted if a critical resource is running on it and only RSCT subsystems is restarted if a non-criticalresource or no resource is running on it. You must use the None split policy if a user prefers multipleinstances of an application that runs simultaneously after split event.

To set the cluster policy as None, enter the following command:

clmgr modify cluster SPLIT_POLICY=none

Manual

The Manual split policy allows each partition that is created by the cluster split event to become anindependent cluster. However, each partition cannot start a workload, until the user allots theownership to the subcluster. Till then the Operational quorum state is in PENDING_QUORUM for eachsubcluster. Hence, PowerHA SystemMirror does not perform any action. The user can provideownership by using the following command:

runact -c IBM.PeerDomain ResolveOpQuorumTie Ownership=0/1

where1

Changes the quorum state from PENDING_QUORUM to HAS_QUORUM0

Changes the quorum state from PENDING_QUORUM to NO_QUORUM

The subcluster with NO_QUORUM state will have no authority to do any operational change on thenodes. If critical resource is running on NO_QUORUM node, the node is rebooted to avoid corruption ofcritical resource. The subcluster with HAS_QUORUM state has authority to do any operational changeon the subcluster.

Before a split event, the operational quorum state is HAS_QUORUM and configuration quorum is TRUE.PowerHA SystemMirror allows both operational change and configurational change into the cluster.

But after the split event, for each operation of the partition, the quorum state will bePENDING_QUORUM and configurational quorum will be FALSE. PowerHA SystemMirror does not allowthe user to do any operational change not even the configurational changes.

If a cluster splits, all the resources are in same state as they were before the split event. The user cangive permission as 1 to one subcluster and 0 to another cluster, so that the resources run only on onesub cluster. After giving permission, the PENDING_QUORUM will change to either HAS_QUORUM orNO_QUORUM.

After the split event, the node that has critical resource running on it is rebooted if 0 permission isgiven. If the non-critical resource or no resource is running on that subcluster, it remains same.

Manual Policy can be used if a user to give permission after split event before creation of anotherinstance of application.


The subcluster with HAS_QUORUM state automatically allows a user to perform any operationalchanges if the merge event occurs,. The lower priority node is rebooted if a critical resource is runningon it with an operational quorum state as HAS_QUORUM before merge and restarts the subsystemswhen the non-critical resource or no resource is running on it.

To set the cluster policy as Manual, enter the following command:

clmgr modify cluster SPLIT_POLICY=manual

Tiebreaker

You can use the tiebreaker option to specify a SCSI disk or a Network File System (NFS) file that isused by the split and merge policies.

A tiebreaker disk or an NFS file is used when the sites in the cluster can no longer communicate witheach other. This communication failure results in the cluster splitting the sites into two, independentpartitions. If failure occurs because the cluster communication links are not responding, bothpartitions attempt to lock the tiebreaker disk or the NFS file. The partition that acquires the tiebreakerdisk continues to function, while the other partition reboots, or has cluster services restarted,depending on if any critical resources are configured.

The disk or NFS-mounted file that is identified as the tiebreaker must be accessible to all nodes in thecluster.

Split Policy Tie Breaker allow one of the partitions that is created by the cluster split event to becomean independent cluster. After winning the tiebreaker, subcluster can start the workload or can allowthe user to perform an operational change. Whenever a split event occurs, the operational quorumstate remains in the PENDING_QUORUM state till one of the subclusters wins the tie breaker. Thesubcluster, which won the tie breaker starts the workload or allow the user to do operational changes.

The following are types of tiebreaker in PowerHA SystemMirror for Linux:NFS tiebreaker

When Network File System (NFS) type of tiebreaker is configured, one directory of the NFS servergets mounted on all nodes of the cluster. When a split event occurs due to network failure on onenode, that node loses the mounting of the NFS server directory and becomes the losing node. Theother node is the winning node.

Before you configure the NFS tiebreaker, a user must have proper understating like permission,accessibility of the NFS server machine and must check the directory on the NFS server that isgoing to be mounted on the local directory of the nodes. To configure the NFS tiebreaker inPowerHA SystemMirror for Linux, enter the following command:

clmgr modify cluster clMain SPLIT_POLICY=tiebreaker TIEBREAKER=nfs NFS_SERVER=192.2.2.5 NFS_SERVER_MOUNT_POINT=/test_nfs_tie NFS_LOCAL_MOUNT_POINT=/test_nfs_local NFS_FILE_NAME=nfsTestConfig

where,

• SPLIT_POLICY=tiebreaker is a type of split policy.• TIEBREAKER=nfs is type of tiebreaker.• NFS_SERVER=192.2.2.5 is the IP of nfs-server machine.• NFS_SERVER_MOUNT_POINT=/test_nfs_tie is the directory on the nfs-server machine.• NFS_LOCAL_MOUNT_POINT=/test_nfs_local is the directory on the nodes.• NFS_FILE_NAME=nfsTestConfig is any file name given by then user.

Before the split event, the quorum state can be seen by using the following command:

lssrc -ls IBM.RecoveryRMOperational Quorum State : HAS_QUORUMIn Config Quorum : TRUE


After the split event occurs, the quorum status will be:

Operational Quorum State : PENDING_QUORUMIn Config Quorum : FALSE

The quorum shows the following state after winning the tiebreaker on the winning node:

Operational Quorum State : HAS_QUORUMIn Config Quorum : FALSE

The quorum shows the following state after losing the tiebreaker on the losing node:

Operational Quorum State : NO_QUORUM In Config Quorum : FALSE

The losing node is rebooted if the critical resource is running on it. Otherwise, it remains as it is.

The winning node starts the workload if not running before otherwise it is running as it is and usercan perform operational configuration changes. The losing node will merge automatically to thecluster after reboot if a critical resource is running on it. If a non-critical resource is running on thenode, the RSCT subsystem is restarted on that node after the merge event.

The NFS tiebreaker must be used if a single-node interface fails. If two-nodes are connected totwo different switches and if one switch goes down, the node that is connected to that switch willnot be able to communicate. However, the second node can still communicate because the otherswitch is working. The second node wins as it can reach the NFS server.

Disk tiebreaker

Disk tiebreaker uses disk to resolve the tie. When the cluster split event occurs, each node tries tomake a lock on the disk. The node, which locks the disk first becomes the winning node. The othernode is the losing node.

Before you configure the disk tiebreaker, the user must find out about the shared disk among thenodes by using the following command:

lsrsrc IBM.Disk

To configure the disk tiebreaker in PowerHA SystemMirror for Linux, enter the followingcommand:

clmgr modify cluster SPLIT_POLICY=tiebreaker TIEBREAKER=disk DISK_WWID=36017595807eed37r0000000000000045

Where

SPLIT_POLICY=tiebreaker is type of split policy TIEBREAKER=disk is type of tiebreakerDISK_WWID=36017595807eed37r0000000000000045 is the shared disk used for tiebreaker.

The disk tiebreaker must be used if interface failure occurs on both the nodes. If both the nodesare connected to the same switch and if the switch goes down, the two nodes cannot reach theNFS serve r. The winning node is decided by the locking system on the disk. The node that locksthe disk first is the winning node.

PowerHA SystemMirror resources and resource groupsThis topic includes resource-related concepts and definitions that are used throughout thedocumentation and also in the PowerHA SystemMirror user interface.

Identifying and keeping your cluster resourcesThe PowerHA SystemMirror software provides a highly available environment.

The PowerHA SystemMirror software does this by:


• Identifying the set of cluster resources that are essential to the operation of an application, andcombining those resources into a resource group.

• Defining the resource group policies and attributes that dictate how PowerHA SystemMirror managesresources to keep them highly available during different cluster events like startup, fallover, andfallback.

By identifying resources and defining resource group policies, the PowerHA SystemMirror software makesnumerous cluster configurations possible, providing tremendous flexibility in defining a clusterenvironment tailored to individual requirements.

Identifying cluster resources

Cluster resources can include both hardware resources and software resources:

• Service IP labels or addresses• Applications

The PowerHA SystemMirror software handles the resource group as a unit, thus keeping theinterdependent resources together on one node and keeping them highly available.

Types of cluster resourcesThis section provides a brief overview of the resources that you can configure in PowerHA SystemMirrorand include into resource groups to let PowerHA SystemMirror keep them highly available.

ApplicationsThe purpose of a highly available system is to ensure that critical services are accessible to users.Applications usually need no modification to run in the PowerHA SystemMirror environment. Anyapplication that can be successfully restarted after an unexpected shutdown is a candidate for PowerHASystemMirror.

For example, all commercial DBMS products provide a checkpoint on the state of the disk in some sort oftransaction journal. In the event of a server failure, the fallover server restarts the DBMS, whichreestablishes database consistency and then resumes processing.

Note: The start and stop scripts are the main points of control for PowerHA SystemMirror over anapplication. It is very important that the scripts you specify operate correctly to start and stop all aspectsof the application. If the scripts fail to properly control the application, other parts of the applicationrecovery might be affected. For example, if the stop script you use fails to completely stop the applicationand a process continues to access a disk, PowerHA SystemMirror will not be able to recover it on thebackup node.

Add your application to a PowerHA SystemMirror resource group only after you have thoroughly testedyour application start and stop scripts.

The resource group that contains the application should also contain all the resources that the applicationdepends on, including service IP addresses. Once such a resource group is created, PowerHASystemMirror manages the entire resource group and, therefore, all the interdependent resources in it asa single entity. PowerHA SystemMirror coordinates the application recovery and manages the resourcesin the order that ensures activating all interdependent resources before other resources.

PowerHA SystemMirror includes application monitoring capability, whereby you can define a monitor todetect the unexpected termination of a process or to periodically poll the termination of an applicationand take automatic action upon detection of a problem.

Service IP labels and IP addressesA service IP label is used to establish communication between client nodes and the server node. Services,such as a database application, are provided using the connection made over the service IP label.

A service IP label can be placed in a resource group as a resource that allows PowerHA SystemMirror tomonitor its health and keep it highly available, either within a node or between the cluster nodes bytransferring it to another node in the event of a failure.

Note: Certain subnet requirements apply for configuring service IP labels.


Persistent node IP labelsA persistent node IP label is a useful administrative tool that lets you contact a node even if the PowerHASystemMirror cluster services are down on that node.

When you define persistent node IP labels PowerHA SystemMirror attempts to put an IP address on thenode. Assigning a persistent node IP label to a network on a node allows you to have a node-bound IPaddress on a cluster network that you can use for administrative purposes to access a specific node in thecluster. A persistent node IP label is an IP alias that can be assigned to a specific node on a clusternetwork and that:

• Always stays on the same node (is node-bound )• Co-exists on a network interface card that already has a service IP label defined• Does not require installing an additional physical network interface card on that node

There can be one persistent node IP label per node.

File systemA file system is a simple database for storing files and directories. A file system in Linux is stored on asingle logical volume.

The main components of the file system are the logical volume that holds the data and the file system log.PowerHA SystemMirror supports both ext3 and xfs as shared file systems.

Support of ext4 file system has been added in PowerHA SystemMirror version 7.2.2 for Linux with ServicePack 2 release.

Cluster resource groupsTo be made highly available by the PowerHA SystemMirror software, each resource must be included in aresource group. Resource groups allow you to combine related resources into a single logical entity foreasier management.

You can configure the cluster so that certain applications stay on the same node, or on different nodes notonly at startup, but during fallover and fallback events. To do this, you configure the selected resourcegroups as part of a location dependency set.

Participating node listA participating node list defines a list of nodes that can host a particular resource group.

You define a node list when you configure a resource group. The participating node list can contain someor all nodes in the cluster.

Typically, this list contains all nodes sharing the same data and disks.

Default node priorityDefault node priority is identified by the position of a node in the node list for a particular resource group.

The first node in the node list has the highest node priority. This node is also called the home node for aresource group. The node that is listed before another node has a higher node priority than the currentnode.

Home nodeThe home node (the highest priority node for this resource group) is the first node that is listed in theparticipating node list for a nonconcurrent resource group.

The home node is a node that normally owns the resource group.

The term home node is not used for concurrent resource groups because they are owned by multiplenodes.


Startup, fallover, and fallbackPowerHA SystemMirror ensures the availability of cluster resources by moving resource groups from onenode to another when the conditions in the cluster change.Cluster startup

When cluster services are started. resource groups are activated on different cluster nodes accordingthe resource group startup policy you selected.

Node failureResource groups that are active on this node fall over to another node.

Node recoveryWhen cluster services are started on the home node after a failure, the node reintegrates into thecluster, and may acquire resource groups from other nodes depending on the fallback policy for thegroup.

Resource failure and recoveryA resource group might fall over to another node, and be reacquired, when the resource becomesavailable.

Cluster shutdownWhen you stop cluster services you can choose to have the resource groups move to a backup node orbe taken offline on the current node.

Resource group policies

The following table describes the specific options for each policy:Startup

Startup is the activation of a resource group on a node (or multiple nodes). Resource group startupoccurs when cluster services are started.

Fallover

Fallover is the movement of a resource group from the node that currently owns the resource group toanother active node after the current node experiences a failure.

Fallover only occurs with nonconcurrent resource groups. concurrent resource groups are active on allnodes concurrently, so the failure of a single nodes means only that instance of the resource group onthat node is affected.

Fallback

Fallback is the movement of resources to the home node when it is reintegrated in the cluster afterfailure. No fallback means that resources continue to run on the same node, even after thereintegration of home node post failure.

Each combination of these policies allows you to specify varying degrees of control over which node,or nodes, control a resource group.

Startup, fallover, and fallback are specific behaviors that describe how resource groups behave atdifferent cluster events. It is important to keep in mind the difference between fallover and fallback.These terms appear frequently in discussion of the various resource group policies.


Networks and resource groupsA service IP label can be included in any nonconcurrent resource group - that resource group could haveany of the allowed startup policies except Online on All Available Nodes.

PowerHA SystemMirror cluster configurationsThis chapter provides examples of the types of cluster configurations supported by the PowerHASystemMirror software.

This list is by no means an exhaustive catalog of the possible configurations you can define using thePowerHA SystemMirror software. Rather, use them as a starting point for thinking about the clusterconfiguration best suited to your environment.

Standby configurationsStandby configurations are the traditional redundant hardware configurations where one or more standbynodes stand idle or running a less critical application, and will takeover the critical application should aserver or primary node fail, waiting for a server node to leave the cluster.

Concurrent resource groups are activated on all nodes concurrently and therefore cannot be used in astandby configuration.

Example: Standby configurations with online on first available node startup policy

Figure 2. One-for-one standby configuration where IP label returns to the home node

In this setup, the cluster resources are defined as part of a single resource group. A node list is thendefined as consisting of two nodes. The first node, Node A, is assigned a takeover (ownership) priority of1. The second node, Node B, is assigned a takeover priority of 2.

At cluster startup, Node A (which has a priority of 1) assumes ownership of the resource group. Node A isthe "server" node. Node B (which has a priority of 2) stands idle, ready should Node A fail or leave thecluster. Node B is, in effect, the "standby".

If the server node leaves the cluster, the standby node assumes control of the resource groups owned bythe server, starts the highly available applications, and services clients. The standby node remains activeuntil the home node rejoins the cluster (based on the fallback policy configured). At that point, thestandby node releases the resource groups it has taken over, and the server node reclaims them. Thestandby node then returns to an idle state.

Extending standby configurations

The standby configuration from the previously described example can be easily extended to largerclusters. The advantage of this configuration is that it makes better use of the hardware. Thedisadvantage is that the cluster can suffer severe performance degradation if more than one server nodeleaves the cluster.


The following figure illustrates a three-node standby configuration using the resource groups with thesepolicies:

• Startup policy: Online on First Available Node• Fallover policy: Fallover to Next Priority Node in the List• Fallback policy: Fallback to Home Node

Figure 3. One-for-two standby configuration with three resource groups

In this configuration, two separate resource groups (A and B) and a separate node list for each resourcegroup exist. The node list for Resource Group A consists of Node A and Node C. Node A has a takeoverpriority of 1, while Node C has a takeover priority of 2. The node list for Resource Group B consists ofNode B and Node C. Node B has a takeover priority of 1; Node C again has a takeover priority of 2. (Aresource group can be owned by only a single node in a nonconcurrent configuration.)

Since each resource group has a different node at the head of its node list, the cluster's workload isdivided, or partitioned, between these two resource groups. Both resource groups, however, have thesame node as the standby in their node lists. If either server node leaves the cluster, the standby nodeassumes control of that server node's resource group and functions as the departed node.

In this example, the standby node has three network interfaces (not shown) and separate physicalconnections to each server node's external disk. Therefore, the standby node can, if necessary, take overfor both server nodes concurrently. The cluster's performance, however, would most likely degrade whilethe standby node was functioning as both server nodes.

Takeover configurationsIn the takeover configurations, all cluster nodes do useful work, processing part of the cluster's workload.There are no standby nodes. Takeover configurations use hardware resources more efficiently thanstandby configurations since there is no idle processor. Performance can degrade after node failure,however, since the load on remaining nodes increases.

One-sided takeoverThis configuration has two nodes actively processing work, but only one node providing highly availableservices to cluster clients. That is, although there are two sets of resources within the cluster (forexample, two server applications that handle client requests), only one set of resources needs to behighly available.

The following figure illustrates a two-node, one-sided takeover configuration. In the figure, a lowernumber indicates a higher priority.


Figure 4. One-sided takeover configuration with resource groups in which IP label returns to the home node

This set of resources is defined as a PowerHA SystemMirror resource group and has a node list thatincludes both nodes. The second set of resources is not defined as a resource group and, therefore, is nothighly available.

At cluster startup, Node A (which has a priority of 1) assumes ownership of Resource Group A. Node A, ineffect, “owns” Resource Group A. Node B (which has a priority of 2 for Resource Group A) processes itsown workload independently of this resource group.

If Node A leaves the cluster, Node B takes control of the shared resources. When Node A rejoins thecluster, Node B releases the shared resources.

If Node B leaves the cluster, however, Node A does not take over any of its resources, since Node B'sresources are not defined as part of a highly available resource group in whose chain this nodeparticipates.

This configuration is appropriate when a single node is able to run all the critical applications that need tobe highly available to cluster clients.

Mutual takeoverThe mutual takeover for nonconcurrent access configuration has multiple nodes, each of which providesdistinct highly available services to cluster clients. For example, each node might run its own instance of adatabase and access its own disk.

Furthermore, each node has takeover capacity. If a node leaves the cluster, a surviving node takes overthe resource groups owned by the departed node.

The mutual takeover for nonconcurrent access configuration is appropriate when each node in the clusteris running critical applications that need to be highly available and when each processor is able to handlethe load of more than one node.

The following figure illustrates a two-node mutual takeover configuration for nonconcurrent access. Inthe figure, a lower number indicates a higher priority.


Figure 5. Mutual takeover configuration for nonconcurrent access

The key feature of this configuration is that the cluster's workload is divided, or partitioned, between thenodes. Two resource groups exist, in addition to a separate resource chain for each resource group. Thenodes that participate in the resource chains are the same. It is the differing priorities within the chainsthat designate this configuration as mutual takeover.

The chains for both resource groups consist of Node A and Node B. For Resource Group A, Node A has atakeover priority of 1 and Node B has a takeover priority of 2. For Resource Group B, the takeoverpriorities are reversed. Here, Node B has a takeover priority of 1 and Node A has a takeover priority of 2.

At cluster startup, Node A assumes ownership of the Resource Group A, while Node B assumesownership of Resource Group B.

If either node leaves the cluster, its peer node takes control of the departed node's resource group. Whenthe "owner" node for that resource group rejoins the cluster, the takeover node relinquishes theassociated resources; they are reacquired by the integrating home node.

Two-node mutual takeover configurationIn this configuration, both nodes have simultaneous access to the shared disks and own the same diskresources.

The following figure illustrates a two-node mutual takeover configuration for concurrent access:


Figure 6. Two-node mutual takeover configuration for concurrent access

In this example, both nodes are running an instance of a server application that accesses the database onthe shared disk. The application's proprietary locking model is used to arbitrate application requests fordisk resources.

Running multiple instances of the same server application allows the cluster to distribute the processingload. As the load increases, additional nodes can be added to further distribute the load.


Planning for PowerHA SystemMirror for LinuxAll the relevant Red Hat Package Manager (RPM) are installed automatically on starting the PowerHASystemMirror installation script.

To install the PowerHA SystemMirror for Linux, the following RPMs are used internally:

• powerhasystemmirror• powerhasystemmirror.adapter• powerhasystemmirror.policies• powerhasystemmirror.policies.one• powerhasystemmirror.policies.two• powerhasystemmirror.sappolicy

PowerHA SystemMirror internally uses Reliable Scalable Cluster Technology (RSCT) for the clusteringtechnology. The required versions of the RSCT RPM are installed automatically by default when you installthe PowerHA SystemMirror. During installation of the PowerHA SystemMirror, if RSCT is detected, and thelevel of RSCT is lower than the required RSCT package, then the currently installed RSCT package isupgraded. PowerHA SystemMirror for Linux installs the following RSCT RPMs:

• rsct.basic• rsct.core• rsct.core.utils• rsct.opt.storagerm• src

• If you define the disk tiebreaker resources, the disk on which IBM. TieBreaker resources are storedmust not be used to store file systems.

• Internet Protocol version 6 (IPv6) configuration is not supported in the PowerHA SystemMirror forLinux.

• You can check the firewall status by running the systemctl status firewalld.servicecommand. Firewall must be disabled or you must open the below ports:

657/tcp16191/tcp657/udp12143/udp12347/udp 12348/udp

When a node is configured with multiple connections to a single network, the network interfaces servedifferent functions in the PowerHA SystemMirror.

A service interface is a network interface that is configured with the PowerHA SystemMirror service IPlabel. The service IP label is used by clients to access application programs. The service IP is onlyavailable when the corresponding resource group is online.

A persistent node IP label is an IP alias that can be assigned to a specific node on a cluster network. Apersistent node IP label always stays on the same node (node-bound), and coexists on an NIC thatalready has a service or boot IP label defined. A persistent node IP label does not require installing anextra physical NIC on that node.

If you assign a persistent node IP label, it provides a node-bound address that you can use foradministrative purposes because a connection to a persistent node IP label always goes to a specificnode in the cluster. You can have one persistent node IP label per node.

For PowerHA SystemMirror, you must configure a persistent IP label for each cluster node. This is usefulto access a particular node in a PowerHA SystemMirror cluster for running reports or for diagnostics. This


https://www.ibm.com/support/knowledgecenter/en/SGVKBA

provides advantage that the PowerHA SystemMirror can access the persistent IP label on the nodedespite individual NIC failures, provided spare NICs are present on the network.

If you assign IP aliases to NICs, it allows you to create more than one IP label on the same networkinterface. During an IP address takeover by using the IP aliases, when an IP label moves from one NIC toanother, the target NIC receives the new IP label as an IP alias and keeps the original IP label andhardware address.

Configuring networks for IP address takeover (IPAT) by using IP aliases simplifies the networkconfiguration in the PowerHA SystemMirror. You can configure a service address and one or more bootaddresses for NICs.

PowerHA SystemMirror uses a technology referred to as IPAT by using IP aliases for keeping IP addresseshighly available.

If you are planning for IP address takeover by using IP aliases, review the following information:

• Each network interface must have a boot IP label that is defined in the PowerHA SystemMirror. Theinterfaces that are defined in the PowerHA SystemMirror are used to keep the service IP addresseshighly available.

• The following subnet requirements apply if multiple interfaces are present on a node that is attached tothe same network:

– All boot addresses must be defined on different subnets.– Service addresses must be on a different subnet from all boot addresses and persistent addresses.

• Service address labels that are configured for IP address takeover by using IP aliases can be included inall nonconcurrent resource groups.

• The netmask for all IP labels in a PowerHA SystemMirror for Linux network must be the same.

During a node fallover event, the service IP label that is moved is placed as an alias on the NIC of targetnode in addition to any other service labels configured on that NIC.

If your environment has multiple adapters on the same subnet, all the adapters must have the samenetwork configuration and the adapters must be part of the PowerHA SystemMirror configuration.

Linux operating system requirements

The cluster node on which you want to install PowerHA SystemMirror for Linux must be running on eitherone of the following versions of the Linux operating system:

• SUSE Linux Enterprise Server (SLES) for SAP 12 SP1 (64-bit)• SUSE Linux Enterprise Server (SLES) for SAP 12 SP2 (64-bit)• SUSE Linux Enterprise Server (SLES) for SAP 12 SP3 (64-bit)• SUSE Linux Enterprise Server (SLES) for SAP 12 SP4 (64-bit)• SUSE Linux Enterprise Server (SLES) for SAP 15 (GA) (64-bit)

• SUSE Linux Enterprise Server (SLES) for SAP 15 Service Pack 1 (64-bit)• Red Hat Enterprise Linux (RHEL) 7.2 (64-bit) (Little Endian)• Red Hat Enterprise Linux (RHEL) 7.3 (64-bit) (Little Endian)• Red Hat Enterprise Linux (RHEL) 7.4 (64-bit) (Little Endian)• Red Hat Enterprise Linux (RHEL) 7.5 (64-bit) (Little Endian)• Red Hat Enterprise Linux (RHEL) 7.6 (64-bit) (Little Endian)

• Red Hat Enterprise Linux (RHEL) 7.7 (64-bit) (Little Endian)


Note: PowerHA SystemMirror Version 7.2.2 for Linux is not supported on SLES 11 for SAP.


Planning for PowerHA SystemMirror for Linux 37

Installing PowerHA SystemMirror for Linux

You must understand the planning and prerequisite information about PowerHA SystemMirror for Linux,before you install PowerHA SystemMirror for Linux.

Planning the installation of PowerHA SystemMirror for LinuxBefore you install PowerHA SystemMirror in your Linux environments, you must ensure that allprerequisites are met.

Packaging

You can download PowerHA SystemMirror for Linux from the IBM website.

The PowerHA Linux release package for the Service Pack is similar to the base build. As there is nodifference between the installation of the SP build with respect to the base build and fresh installation ofSP build, you do not need to install the base build initially.

Prerequisites

You must fulfill the software and hardware requirements for PowerHA SystemMirror for Linux. Before youinstall the PowerHA SystemMirror on a Linux system, you must meet the following prerequisites:

• Root authority is required to install PowerHA SystemMirror.• The following scripting package is required in each SUSE Linux Enterprise Server (SLES) and Red Hat

Enterprise Linux (RHEL) system:

– KSH93 (ksh-93vu-18.1.ppc64le) for SLES and KSH93 (ksh-20120801) for RHEL– PERL– lvm2

You must use the Korn shell version ksh93vu suggested by SUSE. Download the KSH93 package fromthe SUSE website. The downloaded files contain the source package for theksh-93vu-18.1.src.rpm files that must be compiled to get the ksh-93vu-18.1.ppc64le.rpmfiles. To know more about RPM packages, refer to the SUSE technical notes (Section 6.2.6 Installingand Compiling Source Packages) in the SUSE Documentation website.

• The following packages are required in the RHEL system:

– bzip2– nfs-utils– perl-Pod-Parser– bind-utils

• Additionally, the following packages must be installed in RHEL systems for disk heartbeat configuration:

– lsscsi– sg3_utils

• The servicelog package is required in SLES15 systems.

Checking prerequisites

To verify whether all prerequisites are met, complete the following steps:

1. Log in as root user.


https://www.ibm.com/servers/eserver/ess/ProtectedServlet.wss

https://download.suse.com/Download?buildid=rGDRqNAcHLU~

https://www.suse.com/documentation/sles-12/singlehtml/book_sle_admin/book_sle_admin.html

2. After you have downloaded the tar file from the IBM website, extract the tar file by entering thefollowing command:

tar -xvf <tar file>

3. Enter the following command:

cd PHAxxxxLinux64

where, xxxx is the PowerHA SystemMirror for Linux version. For example, 7230 for PowerHASystemMirror Version 7.2.3 for Linux.

4. To start the prerequisites check, enter the following command:

./installPHA --onlyprereqcheck

5. When the check is complete, check the following log file for information about missing prerequisites:

/tmp/installPHA.<#>.log

The hashtag (<#>) is a number; the highest number identifies the most recent log file.6. If your system did not pass the prerequisites check, correct any problems before you start the

installation.

Installing PowerHA SystemMirror for LinuxAfter you complete the planning steps, you are ready to install PowerHA SystemMirror for Linux in yourLinux environment.

Running the installation

To install PowerHA SystemMirror for Linux, you must use the installation script. The installation scriptruns the following actions:Prerequisites check

The installation script runs a complete prerequisite check to verify that all required software areavailable and are at the required level. If your system does not pass the prerequisite check, theinstallation process does not start. To continue with the installation process, you must install therequired software.

Installing PowerHA SystemMirror for LinuxIf an IBM Reliable Scalable Cluster Technology (RSCT) peer domain exists, ensure that the node onwhich you are running the script is offline in the domain. Otherwise, the installation is canceled. Toinstall the PowerHA SystemMirror for Linux, complete the following steps:

1. Log in as root user.2. Download the tar file from the Entitled Systems Support website and extract the tar file by

entering the following command:

tar -xvf <tar file>

3. Enter the following command:

cd PHAxxxxLinux64

where, xxxx is the PowerHA SystemMirror for Linux version. For example, 7230 for PowerHASystemMirror Version 7.2.3 for Linux.

4. Run the following installation script:

./installPHA


https://www.ibm.com/servers/eserver/ess/ProtectedServlet.wss

http://www.ibm.com/servers/eserver/ess/index.wss

Note: You do not need to specify any of the options that are available for the installPHAcommand.

5. The installation program checks prerequisites to verify that all the required software are availableand are at the required level. If your system does not pass the prerequisites check, the installationdoes not start, and you must correct any problems before you restart the installation. Informationabout the results of the prerequisites check is available in the following log file:


6. After the system passes the prerequisite check, read the information in the license agreement andthe license information. You can scroll forward line-by-line by using the Enter key, and page-by-page with the space bar. After reviewing the license information, to indicate acceptance of thelicense terms and conditions, press the Y key. Any other input cancels the installation.

7. After you accept the license agreement, installation proceeds. You must check the following logfile for information about the installation:


The hashtag (<#>) is a number; the highest number identifies the most recent log file.8. You can verify the installation process by running the lssrc -a command that will list the

available subsystems. After installation, you must ensure that the clcomd subsystem is availableand active in the list.

Related informationVideo: Installing PowerHA SystemMirror for Linux

Cluster snapshot for PowerHA SystemMirror for LinuxIn PowerHA SystemMirror for Linux, you can use the cluster snapshot utility to save and restore clusterconfigurations. Snapshots provide useful information about troubleshooting cluster problems and can beused to duplicate a cluster configuration on different hardware configuration.

The cluster snapshot utility saves record of all data that defines a specific cluster configuration. Snapshotrestoration is the process of recreating a specific cluster configuration by using the cluster snapshotutility. You can restore a snapshot on nodes even if a cluster does not exist.

You can use the clmgr snapshot command with actions such as add, modify, query, or delete toutilize the snapshot utility feature.

Related informationclmgr command

Upgrading PowerHA SystemMirror for LinuxYou can upgrade PowerHA SystemMirror for Linux by performing a rolling upgrade or an offline upgradeoperation.

Prerequisites

Before you upgrade PowerHA SystemMirror for Linux, you must have the basic knowledge about thefollowing PowerHA SystemMirror for Linux information: concepts, planning, maintenance, andtroubleshooting.

Note: Offline upgrade and rolling upgrade with SAP NetWeaver workload is not supported when youupgrade from PowerHA SystemMirror Version 7.2.2 for Linux Service Pack 1 to PowerHA SystemMirrorVersion 7.2.2 for Linux Service Pack 2.

Installing PowerHA SystemMirror for Linux 41

https://www.ibm.com/developerworks/community/forums/html/topic?id=a49084f2-0f82-44d1-8726-bdf5853b2a20&ps=25

When you migrate from an earlier version of PowerHA SystemMirror for Linux to PowerHA SystemMirrorVersion 7.2.3 for Linux with Service Pack 1, the following configurations are affected:

• The value of the START_ON_BOOT parameter is set to TRUE.• The value of the PREFER_TAKEOVER parameter for the existing SAP HANA configuration is set to TRUE.• The disk heartbeat configuration from the earlier version is retained. You must delete the old disk

heartbeat network and create the disk heartbeat network again to use multi-node disk heartbeatfeature.

Performing an upgrade operation on an offline clusterYou can take cluster services offline on all the nodes and then update the cluster definitions individuallyon each node.

You can upgrade clusters from PowerHA SystemMirror Version 7.2.2 for Linux Service Pack 1, or later(source release) to PowerHA SystemMirror Version 7.2.2 for Linux Service Pack 2, or later (target release).

Note: A source release is the release that is installed on cluster nodes. A target release is the release towhich you want to upgrade.

To upgrade an offline cluster, complete the following steps:

1. Download the target release to which you want to migrate. Extract the downloaded package to a localdirectory and change the directory to the extracted PowerHA SystemMirror directory.

2. Navigate to the PowerHA SystemMirror directory that contains the release to which you plan toupgrade. Run the following script to check whether the upgrade operation can be performed:

./installPHA --migcheck

Note: You must run the installPHA script of the target PowerHA SystemMirror release to which youplan to upgrade.

3. Bring the cluster offline from any node of the cluster by running the following command:

clmgr offline cluster STOP_RSCT=yes

Note: When the cluster is offline with the STOP_RSCT parameter set to yes, you cannot run the clmgrcommand operations on any nodes in the cluster.

4. To upgrade all nodes, download and extract the target PowerHA SystemMirror release on each nodethat you plan to upgrade and run the following PowerHA SystemMirror installation command:

./installPHA -y

5. After all nodes are upgraded to the target release, the cluster can be brought online by running thefollowing command on any node:

clmgr online cluster manage=auto|manual

Note: The clmgr online cluster command fails if all the nodes are not upgraded to the targetPowerHA SystemMirror release.

Performing a rolling upgrade operationDuring a rolling upgrade operation, the cluster remains online even though the nodes in the cluster run ondifferent versions of PowerHA SystemMirror for Linux.

You can upgrade clusters from PowerHA SystemMirror Version 7.2.2 for Linux Service Pack 1, or later(source release) to PowerHA SystemMirror Version 7.2.3 for Linux, and above (target release).

Notes:


• A source release is the release that is installed on the cluster nodes. A target release is the release towhich you want to upgrade.

• You do not need to remove the cluster when you perform a rolling upgrade operation on a cluster.

To perform a rolling upgrade operation, complete the following steps:

1. Download the target release to which you want to migrate. Extract the downloaded package to a localdirectory and change the directory to the extracted PowerHA SystemMirror directory.

2. Navigate to the PowerHA SystemMirror directory that contains the release to which you plan toupgrade. Run the following script to check whether the upgrade operation is allowed:

./installPHA --migcheck

Note: You must run the installPHA script of the target PowerHA SystemMirror release to which youplan to migrate.

3. Stop the cluster services on one node in the cluster by running the following command:

clmgr offline node <node> STOP_RSCT=yes manage=move

Notes:

• When the cluster is offline with the STOP_RSCT parameter set to yes, you cannot run clmgrcommand operations from that node in the cluster.

• If the manage parameter is set to move, all resources that are managed by the resource group arebrought offline on the current node. The resources are brought online on the other node in thecluster.

4. To upgrade the node, extract the downloaded target PowerHA SystemMirror release on the node andrun the following PowerHA SystemMirror installation command:

./installPHA -y

5. After the node is upgraded to the target release, the cluster can be brought online by running thefollowing command:

clmgr online node <node>

6. Repeat steps 3 - 5 on all the nodes in the cluster.

Installing PowerHA SystemMirror for Linux 43

Configuring PowerHA SystemMirror for LinuxAfter you install PowerHA SystemMirror for Linux, you can configure the product by creating a cluster andby adding nodes to the cluster.

Creating a cluster for PowerHA SystemMirror for LinuxYou can configure the basic components of a cluster by using the options of the clmgr command.

The configuration path significantly automates the discovery and selection of configuration informationand chooses default behaviors.

Prerequisite tasks for configuring the cluster

Before you configure a cluster, PowerHA SystemMirror for Linux must be installed on all nodes andconnectivity must exist between the node where you are performing the configuration and all other nodesthat need to be included in the cluster.

Network interfaces must be both physically and logically configured with the Linux operating system sothat communication occurs from one node to each of the other nodes. The host name and IP addressmust be configured on each interface.

Note: All host name, service IP address, persistent IP address, and labels must be configured inthe /etc/hosts file.

All node IP addresses must be added to the /etc/cluster/rhosts file before you configure the clusterto verify that information is collected from the systems that belong to the cluster. PowerHA SystemMirroruses all configured interfaces on the cluster nodes for cluster communication and monitoring. Allconfigured interfaces are used to keep cluster IP addresses highly available.

Configuring a cluster

To configure a typical cluster component, complete the following steps:Configure a cluster

To configure a cluster, use clmgr add cluster command.Configure additional topology components

To configure additional components for a cluster, you can perform the following actions:

• Add or delete additional nodes from the cluster. For instructions, see “Adding a node to a cluster”on page 46.

• Configure PowerHA SystemMirror resources that are used to configure service IP and persistent IPaddress. For instructions, see “Configuring resources for PowerHA SystemMirror for Linux” on page48.

Configure the cluster resourcesConfigure the resources to be made highly available. For instructions, see “Configuring resources forPowerHA SystemMirror for Linux” on page 48 to configure resources that must be shared among thenodes in the cluster. You can configure the following resources:

• Application (scripts to start, stop, and monitor applications)• Service IP• Persistent IP• File system


Configure the resource groupsTo create the resource groups for each set of related resources, use the clmgr addresource_group command. Additionally, you can add resources to a resource group when youcreate the resource group.

Assign resources to the respective resource groupTo assign resources to each resource groups while creating a resource group, use the clmgr addresource_group command. To modify an existing resource group, use the clmgr modifyresource_group command.

Manage and view log dataTo adjust the log file view and log file management, use the clmgr modify log or clmgr viewlog command. This step is an optional step.

Display the cluster configurationTo view the cluster topology and resources configuration, use the clmgr query cluster or clmgrquery resource_group command. This step is optional.

Configure disk heartbeat configurationTo configure disk heartbeat, use the clmgr add network command. This step is optional.

Perform additional configurationsThe following optional cluster configuration steps can be performed depending on applicationrequirements:

• Configuring runtime policies of a resource group• Configuring dependencies between resource groups• Adding more applications• Configuring file collections• Configuring a cluster user or user group

Related informationclmgr commandVideo: Creating a cluster

Adding a node to a clusterYou can use the clmgr command to dynamically add a node to an existing cluster.

Consider a scenario in which you are creating a cluster that is named clMain, and if the participatingnodes are nodeA and nodeB, enter the following command to add these nodes to the clMain cluster:

clmgr create cluster clMain NODES=nodeA,nodeB

By default, the value of the START_ON_BOOT parameter for nodeA and nodeB is FALSE. If the value ofthe START_ON_BOOT parameter is TRUE, the cluster services on a node are always brought onlineautomatically and the value of STATE parameter in the query node is set to ONLINE after the systemreboot. If the value of the START_ON_BOOT parameter is FALSE, you must manually bring the clusterservices online on a node after the system reboot. The value of the STATE parameter in the query node isset to OFFLINE after the system reboot.

After you create the clMain cluster with nodeA and nodeB, you can also add nodeC to the clMain clusterby running the following command:

clmgr add node nodeC


https://www.ibm.com/developerworks/community/forums/html/topic?id=a49084f2-0f82-44d1-8726-bdf5853b2a20&ps=25

By default, the value of the START_ON_BOOT parameter for nodeC is FALSE. You can set the value ofthe START_ON_BOOT parameter to TRUE when you add nodeC to the clMain cluster by running thefollowing command:

clmgr add node nodeC START_ON_BOOT=TRUE

You can change the configured value of the START_ON_BOOT parameter on the existing node. If you wantto change the value of the START_ON_BOOT parameter to TRUE, run the following command:

clmgr online node nodeC when=reboot

The clmgr online node command includes the following attributes:MANAGE

You can set the value of this attribute to auto or manual. When you set the value of this attribute toauto, the clmgr command automatically brings the resource group online on bringing the clusternode online. When you set the value of this attribute to manual, you must manually bring theresource groups online. The default value of the MANAGE attribute is auto.

WHENYou can set this attribute to one of the following values:

• now: The clmgr command brings the cluster node online instantly and the value of theSTART_ON_BOOT parameter remains unchanged.

• reboot: The clmgr command brings the cluster node online after the system reboot and the valueof the START_ON_BOOT parameter is set to TRUE.

• both: The clmgr command brings the cluster node online instantly and the value of theSTART_ON_BOOT parameter is set to TRUE after the system reboot.

You can change the configured value of the START_ON_BOOT parameter to FALSE by running thefollowing command:

clmgr offline node nodeC when=reboot

The clmgr offline node command includes the following attributes:MANAGE

You can set the value of this attribute to offline or move. When you set the value of this attribute tooffline, the clmgr command automatically brings the resource groups on that node to an offlinestate before it brings the node to an offline state. When you set value of this attribute to move, theclmgr command moves the resource groups from that node to other node as specified in resourcegroup policies before the cluster node are brought to an offline state. The default value of the manageattribute is offline.

STOP_RSCTYou can set this attribute to one of the following values:

• yes: The clmgr command stops the RSCT peer domain cluster on the cluster node. The value of theRSCT_CLUSTER_STATE parameter is set to OFFLINE in the query node.

• no: The RSCT peer domain cluster remains online on the cluster node, but the node cannot host anyresource groups. The value of the RSCT_CLUSTER_STATE parameter is ONLINE but the value of theSTATE parameter is OFFLINE in the query node.

The default value of the STOP_RSCT parameter is no. The value of the STOP_RSCT attribute is set toyes during PowerHA SystemMirror for Linux migration operations.

WHENYou can set this attribute to one of the following values:

• now: The clmgr command brings the cluster node to an offline state instantly and the value of theSTART_ON_BOOT parameter remains unchanged.

• reboot: The clmgr command brings the cluster node to an offline state after system reboot and thevalue of the START_ON_BOOT parameter is set to FALSE.

Configuring PowerHA SystemMirror for Linux 47

• both: The clmgr command brings the cluster node to an offline state instantly and the value of theSTART_ON_BOOT parameter is set to FALSE after the system reboot.

Related informationclmgr command

Configuring resources for PowerHA SystemMirror for LinuxYou can configure resources that are required by the cluster application by using the clmgr command.

You must first define resources that are made available by the PowerHA SystemMirror for Linux for anapplication and then group them together in a resource group. You can add all the resources at once orseparately.

The following types of resources can be configured in a cluster:Application

The scripts that are used to start, stop, and monitor the application.PowerHA SystemMirror service IP labels or IP addresses

The service IP label or IP address is the IP label or IP address over which services are provided andwhich is kept highly available by PowerHA SystemMirror.

PowerHA SystemMirror persistent IP labels or IP addressesThe persistent IP label or IP address is the IP label or IP address that allows you to have a node-bound address on a cluster network that you can use for administrative purposes to access a specificnode in the cluster.

File systemA file system is a simple database for storing files and directories. A file system in Linux is stored on asingle logical volume. The main components of the file system are the logical volume that holds thedata and the file system log. PowerHA SystemMirror supports both ext3 and xfs as shared filesystems.

Configuring a persistent node IP labels or IP address

A persistent node IP label is an IP alias that can be assigned to a specified node on a network. Apersistent IP label has the following features:

• Always remains on the same node.• Co-exists with other IP labels that are present in an interface.• Does not require an additional physical interface in a node.• Always remains in an interface of the PowerHA SystemMirror network.

You can assign a persistent node IP label to a network on a node that allows you to have a node-boundaddress on a cluster network that you can use for administrative purposes to access a specific node in thecluster.

To add a persistent node IP label, enter the following command:

clmgr add persistent_ip pip_node1 NETWORK=net_ether_01 NODE=node1 NETMASK=<255.255.255.0>

In this example, the persistent node IP label named pip_node1 is assigned to the node1 node and usesinterfaces that are defined in the net_ether_01 network.

Consider the following limitations for using the persistent node IP label or IP address:

• You can define only one persistent IP label on each node.• Persistent node IP labels are available at the boot time of a node.


• You must configure persistent node IP labels individually on each node.• To change or show persistent node IP labels, you must use the Modify and Query ACTION and

persistent node IP label in the clmgr command.

Configuring the service IP labels or IP address

Service IP labels and IP addresses are used to establish communication between client nodes and theserver node. Services such as a database application are provided by using the connection that is madeover the service IP label. The /etc/hosts file on all nodes must contain all IP labels and associated IPaddresses that you define for the cluster that includes service IP labels and addresses.

To add a service IP label, enter the following command:

clmgr add service_ip sip NETWORK=net_ether_01 NETMASK=<255.255.255.0>

In this example, the service IP label named sip is established and is made available by using thenet_ether_01 network.

Note: Enter the service IP label that you want to keep highly available. The name of the service IP label orIP address must be unique within the cluster and distinct from the resource group names and it mustrelate to the application and also to any corresponding device. For example, sap_service_address.

Configuring a PowerHA SystemMirror application

A PowerHA SystemMirror application is a cluster resource that is used to control an application that userwant to make highly available. It contains scripts for starting, stopping, and monitoring an application.

To configure an application, enter the following command:

clmgr add application app_1 TYPE=custom STARTSCRIPT="/path/to/startscript" STOPSCRIPT="/path/to/stopscript" MONITORMETHOD="/path/to/monitorscript" RESOURCETYPE=1

In this example, an application named app_1 will be a non-concurrent application because theRESOURCETYPE flag is set to 1. The app_1 application uses the STARTSCRIPT flag for startingapplication, the MONITORMETHOD flag for monitoring application, and the STOPSCRIPT flag for stoppingapplication.

The application process which is invoked from the STARTSCRIPT must be detached from the callingscript by using either of the following method:

• Redirect all file handles to a file and start the application process in the background. For example /path/to/application >/outputfile 2>&1 &

• Create a wrapper application that uses the setsid() C-function to detach the application process fromthe calling STARTSCRIPT.

When you configure an application, the application performs the following actions:

• Associates a meaningful name with the application. For example, the application you are using withPowerHA SystemMirror is named sapinst1. You use this name to refer to the application when youdefine it as a resource. When you set up the resource group that contains this resource, you define anapplication as a resource.

• Allows you to configure application start, stop, and monitoring scripts for that application.• Reviews the vendor documentation for specific product information about starting and stopping a

particular application.• Verifies that the scripts exist and has an executable permissions on all nodes that participate as

possible owners of the resource group where this application is defined.

The clmgr add application command includes the following attributes:


ApplicationEnter an ASCII text string that identifies the application. You use this name to refer to the applicationwhen you add it to the resource group. The application name can include alphanumeric charactersand underscores.

STARTSCRIPTEnter the full path name of the script followed by arguments that are called by the event scripts of thecluster to start the application. Although, this script must have the same name and location on everynode, the content and function of the script can be different. You can use the same script and runtimeconditions to modify the runtime behavior of a node.

STOPSCRIPTEnter the full path name of the script that is called by the cluster event scripts to stop the application.This script must be in the same location on each cluster node that can start the application. Although,this script must have the same name and location on every node, the content and function of thescript can be different. You can use the same script and runtime conditions to modify runtimebehavior of a node.

MONITORMETHODEnter a script or run an executable file to customize how you want to monitor the health of thespecified application. This is a mandatory field. The program name must be specified as an absolutepath in the file system of the target nodes.

TYPEYou can select either the process application monitoring or custom application monitoring method.The process application monitoring method detects the termination of one or more processes of anapplication. The custom application monitoring method checks the health of an application by usingthe monitor script at user-specified polling intervals. The custom monitor script returns a value of 1for ONLINE or 2 for OFFINE status.

RESOURCETYPEThis attribute reflects the type of application resource. The resource can be either non-concurrent,which means that the resources are serially reusable across multiple nodes, or the resource can beconcurrently accessible on multiple nodes. By default, the resource type is non-concurrent.

Configuring resource groups for PowerHA SystemMirror for LinuxYou can configure resource groups that use different startup, fallover, and fallback policies.

To configure a resource group, enter the following command:

clmgr add resource_group RG1 SERVICE_IP=sip APPLICATIONS=app_1

In this example, the RG1 resource group is non-concurrent resource group that uses default policies. Theresource group contains the sip IP address and the app_1 application.

Configuring a resource group includes the following tasks:

• Configuring the resource group name, startup, fallover, and fallback policies, and the nodes that canown it (use the nodelist attribute for a resource group).

• Adding resources and additional attributes to the resource group.

If you are defining the resources in your resource groups, ensure that you are aware of followinginformation:

• A resource group might include multiple service IP addresses. According to the resource groupmanagement policies in PowerHA SystemMirror, when a resource group is moved, all service labels inthe resource group are moved as aliases to the available interfaces.

• When you define a service IP label or IP address on a cluster node, the service IP label can be used inany non-concurrent resource group.

The clmgr add resource_group command includes the following attributes:


Table 4. Add resource group fields

Field Value

Resource Group Enter the name for this group. The name of the resource group must be uniquewithin the cluster and distinct from the service IP label and an applicationname. It is helpful to create the resource group name that is related to theapplication it serves, and also to any corresponding device. For example,sap_service_address. Do not use reserved words as resource name. You cancheck the list of reserved words. You cannot add duplicate entries.

NODES Enter the names of the nodes that can own or take over this resource group.Enter the node with the highest priority for ownership first, followed by thenodes with the lower priorities, in the specific order.

STARTUP Defines the startup policy of the resource group:ONLINE ON FIRST AVAILABLE NODE

The resource group activates on the first node that becomes available.ONLINE ON ALL AVAILABLE NODES

The resource group is made online on all nodes. This is similar to thebehavior of concurrent resource group.

If you select this option for the resource group, ensure that resources in thisgroup can be made online on multiple nodes simultaneously.

FALLOVER Select a value from the list that defines the fallover policy of the resource group:

FALLOVER TO NEXT PRIORITY NODE IN THE LISTIn the case of a fallover, the resource group that is online on only one nodeat a time follows the default node priority order that is specified in thenodelist attribute of the resource group (it moves to the highest prioritynode this is currently available).

BRING OFFLINESelect this option to make the resource group offline.

FALLBACK Select a value from the list that defines the fallback policy of the resourcegroup:

NEVER FALLBACKA resource group does not fall back when a higher priority node joins thecluster.

FALLBACK TO Home NodeA resource group falls back to the Home Node.

SERVICE_LABEL This option is applicable only if you are adding resources to a non-concurrentresource.

List the service IP labels to be taken over when this resource group is takenover. These include addresses that are available but were previously taken overor that might be taken over.

APPLICATION Specify the application to include in the resource group.

FILESYSTEM Specify the file system resource to include in the resource group.

You can modify resource group attributes or the different resources such as application, service IP label,or IP address by using the clmgr modify resource_group command.

When a resource group is modified while it is an Online state, it is temporarily brought down and thenbrought back to original state by using the clmgr command.


Note: The maximum number of resource groups that are supported is 32.

Configuring a Shared File System resourceYou can configure file system resources by using the clmgr command.

Prerequisites

To verify the Shared storage functionality, complete the following steps:

1. Before you configure a filesystem, verify that disk is shared between the relevant nodes of cluster. Filesystem resource cannot be added into resource group, which have some nodes in the node list wherethe disk of that filesystem is not shared. You can check the shared disk by using one of followingcommand:

a. lsrsrc -a -s "ResourceType==1" IBM.Disk2. Create a partition on the shared disk by using the fdisk /dev/<device> command. Complete the

following steps:

a. Use the variable n for new and d for delete option.b. Change the fd partition type by using the t option.c. In the end, save all updates by using the wq option.

3. Create a physical volume on the partition by using the pvcreate <partition> command. You canlist all physical volumes by using the pvdisplay option.

4. Create a volume group on the partition by using the vgcreate <name of VG> <PV> [PV2….]command. You can list all volume groups by using the vgdisplay option.

5. Create a logical volume on the partition by using the lvcreate –L <LV size with G or Mdesignation> /dev/<name of VG> command. You can list all logical volumes by using thelvdisplay option.

Configuring

To configure the Shared storage functionality, complete the following steps:

1. After you complete the prerequisite, check whether the logical volume is available on all the nodes ofcluster by using the lvdisplay option or by using the clmgr query logical_volume commandquery.

2. If the logical volume is not visible on all the nodes, run the partprobe command on all the nodes ofcluster or try rebooting the nodes where the logical volumes are not visible.

After the logical volume is visible on all the nodes of cluster, configure a file system by using the followingcommand:

clmgr add file_system /home/fs_1 TYPE=ext3 LOGICAL_VOLUME="vg_1-lv_1"

Where, type is ext3, which is File System type, /home/fs_1 is name for the file system resource, andvg_1-lv_1 is the logical volume, which is shared among the nodes and must be present in clmgr querylogical_volume command query.

After the file system is created, you can add it to resource group along with application. Filesystem alonecan be configured in the RG in PowerHA SystemMirror Version 7.2.2 for Linux Service Pack 2 and later.

Then, change the resource group to an Online state.

Restrictions

Following are the restrictions to be followed when filesystem resource is added to RG

1. FBHN RG policy must not be used if filesystem resource is configured in RG.


2. If multiple filesystem resources are created on respective LVs on the same disk, then those resourcesmust not be added in different Resource Groups.

Verifying the standard configurationPowerHA SystemMirror for Linux verification process checks whether specific PowerHA modifications tothe Linux system files are correct, the cluster and the corresponding resources are configured correctly,security is configured correctly, all nodes agree on the cluster topology, network configuration, and theownership and takeover of PowerHA resources.

After you have configured, reconfigured, or updated a cluster, you must run the cluster verificationprocedure. The following checks are run for the verification for PowerHA SystemMirror for Linux:

1. Determine whether there is enough free space to run PowerHA SystemMirror for Linux.

• Minimum 100 MB of free space must be available in the /usr/sbin, /opt, and /var directories.• On each node of the cluster, m 128 MB RAM must be available.

2. Check installed cluster filesets on all nodes.

• To check if Tivoli System Automation (TSA) filesets are correctly installed on all the cluster nodes,run the Linux – rpm command.

3. Check the installed Reliable Scalable Cluster Technology (RSCT) filesets on all nodes.

• To check if the RSCT filesets are correctly installed on all the cluster nodes, run the Linux – rpmcommand.

4. Verify whether the start, stop, or monitor scripts that are used for an application are executable on allnodes.

• Verify scripts that are provided by the user to start or an application.• Verify the scripts that are provided by the user to stop the application.• Verify the scripts that are provided by the user to monitor the application.

5. Verify the availability of the logical volumes on all cluster nodes.6. If the file system is mounted, verify the size of logical volumes to know that how much of the logical

volume free.7. Verify whether the same disk is part of multiple file system resources to give an appropriate warning.8. Verify the availability of volume groups on all cluster nodes.9. Verify the size of logical volumes on all the cluster nodes.

To start the verification process, run the following commands:

clmgr verify cl -h clmgr verify cluster verify => validate

Configuring dependencies between resource groupsIn PowerHA SystemMirror, a dependency or a relationship exists between a source resource group andone or more target resources groups.

Dependency

To configure a dependency between two resource groups, enter the following command:

clmgr add dependency NAME=SA TYPE=STARTAFTER SOURCE=RG1 TARGET=RG2

Where,


SACustomized name that is used to define a dependency.

RG1Source resource group.

RG2Target resource group.

STARTAFTERType of a dependency or a relationship you want to configure between both resource groups.

The configurable dependencies have the following attributes:Name

Specifies the dependency name defined by the user.Source

Specifies the source resource group name.Target

Specifies the list of target resource group names.Type

Specifies the dependency that must be applied between source and target resource groups.

Types of dependencies

You can configure the following types of dependencies between the source and target resource groups.

Start and Stop dependencies

In Start and Stop dependencies, the resource groups have parent-child relationship. The following typesof Start and Stop dependencies can be configured:STARTAFTER

The STARTAFTER dependency ensures that the source resource group is started only when the targetresource groups are in the Online state. The STARTAFTER dependency provides the followingbehavior scheme:

Figure 7. STARTAFTER dependency

The STARTAFTER dependency defines a start sequence for the resource group A and resource groupB. To start the source resource group A, the target resource group B must be started first.

Note: The resource group A and resource group B can be started on different nodes.

The following rules apply to the STARTAFTER dependency:

• The STARTAFTER dependency must not conflict with an existing DEPENDSON dependency.• The STARTAFTER dependency does not define the location dependency between the managed

resource groups. If you must define a location dependency, you must create an additionaldependency.

• If PowerHA SystemMirror must start the source resource group, PowerHA SystemMirror does notattempt to start the target resource groups internally. You must start the target resource groups


before you attempt to bring the source resource group to an Online state, otherwise PowerHASystemMirror displays an error.

• If PowerHA SystemMirror must stop the target resource group, PowerHA SystemMirror does notattempt to stop the source resource groups internally. You must stop the source resource groupsbefore you attempt to bring the target resource group to an Offline state, otherwise PowerHASystemMirror displays an error.

• If the target resource group fails, the source resource group is not stopped.

STOPAFTERThe STOPAFTER dependency ensures that the source resource group can be stopped only when thetarget resource groups are stopped. The STOPAFTER dependency provides the following behaviorscheme:

Figure 8. STOPAFTER dependency

The resource group A is not stopped unless the target resource group is brought offline.

Note: The STOPAFTER dependency must not provide a start dependency and a force-downdependency.

DEPENDSONThe DEPENDSON dependency ensures that the source resource group is stopped when the targetresource groups fail. The DEPENDSON dependency also includes an implicit collocation between thesource and target resource groups. The DEPENDSON dependency provides the following behaviorscheme:

Figure 9. DEPENDSON dependency

The resource group A depends on the functionality of resource group B. Hence, the resource group Acannot function without the resource group B.

The DEPENDSON dependency provides the following behavior schemes:

• By using the start behavior scheme, the DEPENDSON dependency defines a start sequence for theresources group A and resource group B with an implicit collocation dependency: To start thesource resource group A, the target resource group B must be started first. After the target resourcegroup B is in the Online state, the source resource group A can be started on the same node.

• By using the stop behavior scheme, the DEPENDSON dependency defines a stop sequence forresource groups A and resource group B: To stop the target resource group B, the source resourcegroup A must be stopped first. After the source resource group A is offline, the target resourcegroup B can be stopped.


DEPENDSONANYThe behavior of the DEPENDSONANY dependency is identical to the DEPENDSON dependency exceptthat it does not provide the collocated constraint for the start sequence. Therefore, source and targetresource groups can be started either on the same node or on different nodes. The DEPENDSONANYdependency provides the following behavior schemes:

Figure 10. DEPENDSONANY dependency

• By using the start behavior, the DEPENDSONANY dependency defines a start sequence for theresource group A and resource group B without using the location dependency. The target resourcegroup B must be started first to start the source resource group A. After the target resource group Bis in the Online state, the source resource group A can be started.

• By using the stop behavior, the DEPENDSONANY dependency defines a stop sequence for theresource group A and resource group B. The source resource group A must be stopped first to stopthe target resource group B. After the source resource group A is offline, the target resource group Bcan be stopped.

FORCEDDOWNBYThe FORCEDDOWNBY dependency ensures that the source resource group is moved to the Offlinestate if the target resource group is in the Offline state.

The FORCEDDOWNBY dependency provides the following behavior scheme:

Figure 11. FORCEDDOWNBY dependency

The resource group A must be moved to an Offline state when the target resource group moves to anOffline state unexpectedly. The resource group A and resource group B can be stoppedsimultaneously.

Note: The FORCEDDOWNBY dependency does not provide a start behavior scheme and a stopbehavior scheme.

Location dependencies

The Location dependencies ensures that the source resource group and target resource groups are in anOnline state on either same node or different nodes. The location dependencies are of the followingtypes:COLLOCATED

The COLLOCATED dependency ensures that the source resource group and the corresponding targetresource groups are in an Online state on the same node. The COLLOCATED dependency provides thefollowing behavior scheme:


Figure 12. COLLOCATED dependency

The COLLOCATED dependency ensure that when the resource group A starts, move to an Online stateonly on the node where the resource group B is already running.

ANTICOLLOCATEDThe ANTICOLLOCATED dependency ensures that the source resource group and the correspondingtarget resource groups are in the Online state on different nodes. The ANTICOLLOCATED dependencyprovides the following behavior scheme:

Figure 13. ANTICOLLOCATED dependency

The ANTICOLLOCATED dependency ensures that the resource group A can be started only on a node,where the resource group B is already running.

Note: If the target resource group B cannot be started on a node other than the source node, thetarget resource group B can be started where the source resource group A is already running.

ISSTARTABLEThe ISSTARTABLE dependency provides the following behavior scheme:

Figure 14. ISSTARTABLE dependency

The ISSTARTABLE dependency indicates that the resource group A can be placed only on a nodewhere the resource group B can be started when the resources group A and resource group B are inthe Online state.


Troubleshooting PowerHA SystemMirror for LinuxUse this information to troubleshoot the PowerHA SystemMirror software for the Linux operating system.

Troubleshooting PowerHA SystemMirror clustersTypically, a functioning PowerHA SystemMirror cluster requires minimal intervention. If a problem doesoccur, diagnostic and recovery skills are essential. Therefore, troubleshooting requires that you identifythe problem quickly and apply your understanding of the PowerHA SystemMirror software to restore thecluster to be completely operational.

Tuning a cluster for optimal performance, can help you to avoid common PowerHA SystemMirror clusterproblems.

Using PowerHA SystemMirror cluster log filesYou can troubleshoot a cluster by using the PowerHA SystemMirror cluster log files.

Reviewing cluster message log files

PowerHA SystemMirror writes messages that it generates to the system console and to several log files.Each log file contains a different subset of messages that are generated by the PowerHA SystemMirrorsoftware. When viewed as a group, the log files provide a detailed view of all cluster activity.

The following list describes the log files into which the PowerHA SystemMirror software writes messagesand the types of cluster messages they contain. The list also provides recommendations for usingdifferent log files.

Note: Only default directories are listed in the following table. If you redirect any log files, you must checkthe appropriate location.

system log messagesContains time-stamped and formatted messages from all subsystems, including scripts and daemons.

The log file name is /var/log/messages.

/var/pha/log/clcomd/clcomddiag.logContains time-stamped, formatted, diagnostic messages generated by the clcomd daemon.

Recommended Use: Information in this file is for IBM Support personnel.

/var/pha/log/hacmp.outContains time-stamped, formatted messages generated by PowerHA SystemMirror events.

/var/pha/log/clmgr/clutils.logContains information about the date, time, commands generated by using the clmgr command.

/var/pha/log/appmon.logContains information about the exit code and failure counts generated when you run the applicationmonitor script.

/var/pha/log/salite/salite.logContains information specific to Smart Assist.

/var/pha/log/salite.stderrContains information about the reason for failure of resource creation or deletion when you run SmartAssist.


Using the Linux log collection utilityYou can use the clsnap Linux command to collect data from a PowerHA SystemMirror cluster.

clsnap commandThe clsnap command collects the log files from all nodes of the cluster and saves the log data as acompressed .tar file in the /tmp/ibmsupt/hacmp directory for every cluster node. The logs of theclsnap command are created in the /tmp/ibmsupt/hacmp.snap.log file. The clsnap commandperforms the following operations:

• To collect log information from the local node, enter the following command:

clsnap -L

• To collect log information from the specified node, enter the following command:

clsnap -n <node-name>

• To collect log information in a specified directory, enter the following command:

clsnap –d <dir>

Solving common problemsYou can interact with PowerHA SystemMirror by using relevant clmgr command arguments. If an erroroccurs when you run the clmgr command, and if the error reported on the console is not clear, you mustcheck the /var/pha/log/clmgr/clutils.log log file.

PowerHA SystemMirror startup issuesThese topics describe potential PowerHA SystemMirror startup issues.

PowerHA SystemMirror failed to create the clusterThis topic discusses a possible cause for failure in cluster creation.

Problem

Cluster creation fails with a reason that it cannot connect to other cluster nodes.

Solution

A number of situations can cause this problem. Review the following information to identify a possiblesolution for this problem:

• Check whether the IPv4 address entry exists in the /etc/cluster/rhosts file on all nodes of cluster.• If any entry is recently added or updated, refresh the Cluster Communication Daemon subsystem

(clcomd) by using the refresh -s clcomd command on all nodes of a cluster and then create thecluster again.

• Check whether the Cluster Communications (clcomd) daemon process is running by using the ps -aef| grep clcomd command. If the clcomd daemon process is not listed in the process table, start theclcomd daemon manually by using the startsrc -s clcomd command.

• Check and ensure that other nodes are not part of any cluster.• Check that the hostname of the different nodes are there in the /etc/hosts file.• Check the Firewall status by running the systemctl status firewalld.service command.

Firewall should be disabled and the following ports must be opened:

657/tcp16191/tcp


657/udp12143/udp12347/udp 12348/udp

– To open the port, enter the following command:

firewall-cmd --permanent --zone=public --add-port=<port no>/<protocol>

Example: firewall-cmd --permanent --zone=public --add-port=16191/tcp– To list all open ports on a firewall, enter the following command: firewall-cmd --list-all

• Check whether the subnet configuration is correct on all interfaces for all nodes that are part of thecluster. The subnet of all the interfaces on the same node must be different. For example, the subneteth1, eth2 and eth3 of node1 must be 10.10.10.0/24, 10.10.11.0/24, 10.10.12.0/24.

Highly available resource failed to startThis topic examines a situation where a highly available resource fails to start.

Problem

A highly available resource fails to start.

Solution

You can review the following information to identify possible solutions for this problem:

1. Check for messages that are generated when you run the StartCommand command for that resourcein the system log file (/var/log/messages), and in the ps –ef process table. If the StartCommandcommand is not run, proceed with next step, otherwise investigate why the application is online.

2. Either more than half of the nodes in the cluster are online or exactly half of the nodes are online andthe tiebreaker function is reserved. If less than half of the nodes are online, start the additional nodes.If exactly half of the nodes are online, check the attribute of the active tiebreaker. You can check theactive tiebreaker by running the clmgr query cluster command.

3. In some scenarios, a resource moves to the Sacrificed state when the PowerHA SystemMirror mightnot find a placement for the resource. PowerHA SystemMirror cannot start this resource because thereis no single node on which this resource might be started.

To resolve this problem, ensure that the Network, which the Service IP resource in the resource groupuses, does have at least one of the nodes included. This node must be part of the resource groupnodelist. To check whether different nodes are assigned on a network, run the clmgr querynetwork command. If different nodes are assigned on the network, delete the network and add itagain with correct entries of the nodes by using the clmgr add interface command. To displaythe detailed information about resource groups and the resources, run the clRGinfo -e command.This solution might resolve the issues.

If the application resource is in Sacrificed state, check whether the resource groups are not on thesame node when they have AntiCollocated relationship between them. To resolve this issue, moveone of the resource groups to the other node.

Resource group does not startThis topic examines a situation where a resource group fails to start.

Problem

A resource group does not start.

Solution

A resource group is composed of several resources. If none of resources of the resource group is starting,perform the following steps:

Troubleshooting PowerHA SystemMirror for Linux 61

1. Identify which of the resources must start first by evaluating the relationship status between them.2. Check all requests against the resource group, and evaluate all relationships in which the resource

group is defined as a source.

PowerHA SystemMirror goes to Warning StateThis topic discusses a possible cause for a cluster to be in the Warning state.

Problem

A cluster goes to the Warning state.

Solution

If a cluster goes in the Warning state, you can check the following scenario and resolve it.

• All the nodes of a cluster are not in the Online state.• All the nodes of a cluster are not reachable.

PowerHA SystemMirror fails to add nodeThis topic discusses a possible cause for failure while adding a node to the cluster.

Problem

Unable to add a node to the cluster. For example,

2632-077 The following problems were detected while adding nodes to the domain. As a result, no nodes will be added to the domain.rhel72node2: 2632-068 This node has the same internal identifier as rhel72node1 and cannot be included in the domain definition.

Solution

This error occurs if you use the cloned operating system images. To fix this issue, you must reset thecluster configuration by running the /opt/rsct/install/bin/recfgct -F command on the nodethat is specified in the error message. This action resets the RSCT node ID.

PowerHA SystemMirror disk issuesThese topics describe potential disk and file system issues.

PowerHA SystemMirror failed to configure the disk tiebreakerThis topic describes the situations where PowerHA SystemMirror failed to configure the disk tiebreaker.

Problem

PowerHA SystemMirror is unable to configure the disk tiebreaker.

Solution

Check if the disk is shared among all the nodes by comparing the Universally Unique Identifier (UUID) ofthe disk on all the nodes of the cluster. You can additionally perform the following steps:

• Use the lsscsi command to list the SCSI devices in the system.• Check the /usr/lib/udev/scsi_id -gu <SCSI_disk#> file for all nodes to check the Disk ID

attribute, and ensure that the Disk ID attribute is same across all nodes of the cluster


PowerHA SystemMirror failed to configure the disk heartbeatThis topic describes the situations where PowerHA SystemMirror failed to configure the disk heartbeat.

Problem

PowerHA SystemMirror is unable to configure the disk heartbeat.

Solution

Check whether the disk is a shared disk among all nodes by comparing the Port VLAN Identification(PVID) of the disk on all nodes of the cluster. You must ensure that the PVID of that disk is same on allnodes.

PowerHA SystemMirror is not able to detect common diskThis topic describes the situations where PowerHA SystemMirror software is not able to detect the shareddisk across nodes of the cluster.

Problem

PowerHA SystemMirror software is not able to detect the shared disk across nodes of the cluster.

Solution

• Use the lsrsrc IBM.Disk command to view the common disk between two nodes and ensure thatthe cluster is present. The lsrsrc command works only if the cluster is located in the common disk.

• You must choose the DeviceName attribute that corresponds to the nodelist attribute, which has thetotal number for nodes of the cluster.

PowerHA SystemMirror resource and resource group issuesThese topics describe potential resource and resource group issues.

Highly available resources are in the Failed offline stateThis topic examines situations when highly available resources fail.

Problem

PowerHA SystemMirror sets the resource to the Failed Offline state because the previous attempt tostart the resources failed.

Solution

This problem has two possible causes:Cluster node is not online

If a cluster node is not online, all resources that are defined on the node have Failed Offline state. Inthis case, the problem is not related to the resource but to a node.

Monitor command of resource returns failureTo determine the issue in this case, run the Monitor command by using the following steps:

1. Run the Monitor command.2. Get the return code by entering the echo $? command.

If the return code is 3 (Failed Offline state), determine the reason for the Monitor command failure.To investigate this problem, check the system log files for all messages that are located inthe /var/log/messages file that indicates a timeout for this resource or check whether all thescripts or binary file that are internally started by using the start scripts, stop scripts, or monitorscripts are present at the correct path and are having the required permissions.


Finally, resource can be reset by using the clmgr command:

clmgr reset application <application> [NODE=<node_name>]clmgr reset service_ip <service_ip> [NODE=<node_name>] clmgr reset file_system <file_system> [NODE=<node_name>]

After this step, PowerHA SystemMirror initially stops this resource, and if the corresponding resourcegroup is in an Online state, it starts that resource group again.

Resource group has Failed Offline stateThis topic examines situations where a resource group is in the Failed Offline state.

Problem

PowerHA SystemMirror sets the resource group to the Failed Offline state because previous attempt tostart the resource group failed.

Solution

If the resources of a resource group do not start and the resource group shows Failed Offline state, itindicates that the binder was unable to find a placement for the resources and now the resource groupshows Sacrificed state.

Also, check whether other resources have Failed Offline state and set the resource Online or Offlinestate.

Resource group has Stuck Online stateThis topic examines situations where a resource group is in the Stuck Online state.

Problem

PowerHA SystemMirror shows that the state of the resource group is Stuck Online.

Solution

The possible causes follow:

• The Monitor command of a resource group returns with error code. This can be checked by runningthe Monitor command manually and checking the return code by entering the echo $? option. Ifreturn code is 4 (Stuck Online state), determine the reason for the Monitor command failure.

• In case where the PowerHA SystemMirror cannot stop a resource, it is set to the Stuck Online state.This issue occurs if you run the Stop script for this resource against that resource, the failed to bring theresource in an Offline state.

This error requires manual intervention and the stop script must be corrected. The state of the resource isevaluated as Offline when you run the monitor command again or reset the resource group by using thefollowing command.

clmgr reset resource_group <resource_group>

In case the resource group state is still Stuck Online after stop script is corrected, then reset theresource group.

PowerHA SystemMirror Fallover issuesThis topic describes the potential fallover issues.

PowerHA SystemMirror fallover is not triggered after a node crash or reboot

Problem

PowerHA SystemMirror fails to selectively move the affected resource group to another cluster nodewhen a node crashes or restarts.


Solution

Check if either more than half of the nodes in the cluster are online or exactly half of the nodes are onlineand the tiebreaker is reserved. If less than half of the nodes are online, start additional nodes. If exactlyhalf of the nodes are online, check the attribute of the active tiebreaker.

The possible causes follow:

• Either more than half of the nodes in the cluster are online or exactly half of the nodes are online andthe tiebreaker is reserved. If less than half of the nodes are online, start additional nodes. If exactly halfof the nodes are online, check the attribute of the active tiebreaker by using the clmgr querycluster command.

• The split policy must be set as None and if it is set as Manual, the user intervention is required forfallover after node reboots.

• The policy of resource group must be FBHN, which can be checked by using the clmgr query rgcommand.

PowerHA SystemMirror additional issuesThis topics describes any additional issues of PowerHA SystemMirror.

Application is not working

Problem

The start, stop, or monitor scripts are not working properly.

Solution

• Ensure that the path and arguments of all the scripts are correct.• Try to manually run the start script and redirect all output to a file and observe the behavior of scripts.

For example, run the /usr/bin/application >/outputfile script.• Ensure that all the return codes of start, stop, and monitor scripts are returning correct values. The

monitor script shall return the value 1 for ONLINE or 2 for OFFLINE status.• The application process must be detached from the calling script by using either of the following

methods.

– Redirect all file handles to a file and start the application process in the background, for example /path/to/application >/outputfile 2>&1 &

– Create a wrapper application that uses the setsid() C-function to detach the application process fromthe calling StartScript.


PowerHA SystemMirror graphical user interface(GUI)

In PowerHA SystemMirror Version 7.2 for Linux, you can use a graphical user interface (GUI) to monitoryour cluster environment.

The PowerHA SystemMirror GUI provides the following advantages over the PowerHA SystemMirrorcommand line:

• A secure, single access point to all clusters• A health-at-a-glance dashboard that highlights any problem that you need to be aware of• Cluster zoning provides unprecedented organization and security• Management features are provided, allowing authorized users to perform actions on their clusters, such

as starting and stopping cluster services and resource groups, move resource groups to new nodes,create new clusters, create new resource groups with resources, and more. Cluster zoning providesunprecedented organization and security.

• Scan event summaries and read a detailed description for each event. If the event occurred because ofan error or issue in your environment, you can read suggested solutions to fix the problem.

• Role-based access controls allow safe and secure privilege elevation for non-root users• A clean view of critical cluster and system logs• View properties for a cluster such as the PowerHA SystemMirror version and name of nodes• Create, modify, and delete policies for SAP HANA by using the Smart Assist wizard. Also, view and

manage the saved and activated SAP HANA policies.

Planning for PowerHA SystemMirror GUIBefore you can install PowerHA SystemMirror GUI, your environment must meet certain requirements.

Linux operating system requirements

The nodes in the clusters on where you are running the install scripts must be running one of the followingversions of the Linux operating system:

• SUSE Linux Enterprise Server (SLES) for SAP 12 SP1 (64-bit)• SUSE Linux Enterprise Server (SLES) for SAP 12 SP2 (64-bit)• SUSE Linux Enterprise Server (SLES) for SAP 12 SP3 (64-bit)• SUSE Linux Enterprise Server (SLES) for SAP 12 SP4 (64-bit)• SUSE Linux Enterprise Server (SLES) for SAP 15 (GA) (64-bit)

• SUSE Linux Enterprise Server (SLES) for SAP 15 Service Pack 1 (64-bit)• Red Hat Enterprise Linux (RHEL) 7.2 (64-bit) (Little Endian)• Red Hat Enterprise Linux (RHEL) 7.3 (64-bit) (Little Endian)• Red Hat Enterprise Linux (RHEL) 7.4 (64-bit) (Little Endian)• Red Hat Enterprise Linux (RHEL) 7.5 (64-bit) (Little Endian)• Red Hat Enterprise Linux (RHEL) 7.6 (64-bit) (Little Endian)



Note:


• Before using the PowerHA SystemMirror GUI, you must install and configure secure shell (SSH) on eachnode.

• OpenSSL and OpenSSH must be installed on the system that is used as the PowerHA SystemMirror GUIserver.

You can install the PowerHA SystemMirror GUI on Linux operating system by using the installPHAGUIscript. For more information, see “Installing PowerHA SystemMirror GUI” on page 69.

Planning for SSH

OpenSSL and OpenSSH must be installed on the system that is used at the PowerHA SystemMirror GUIserver. OpenSSH is used to create secure communication between PowerHA SystemMirror GUI serverand nodes in the cluster. OpenSSH is needed only for the cluster addition process. After completion ofcluster addition, OpenSSH is no longer used for communication between the server and the agents. Formore information, see the OpenSSL website and the OpenSSH website.

The SSH File Transfer Protocol (SFTP) subsystem must be configured to work between the PowerHASystemMirror GUI server and nodes in the cluster. You can verify that the SFTP subsystem is configuredcorrectly in the /etc/ssh/sshd_config file. Verify that following path is correct:

Subsystem sftp /usr/libexec/openssh/sftp-server

If the path is not correct, you must enter the correct path in the /etc/ssh/sshd_config file, and thenrestart the sshd subsystem.

PowerHA SystemMirror support both the key-based and password-based form of SSH authentication.PowerHA SystemMirror for Linux disable the password-based authentication after SSH installation.Hence, only key-based authentication is available on the system. To enable the password authentication,you must change the configuration that allows the password authentication. To enable the passwordauthentication in SSH, edit the /etc/ssh/sshd_config file by adding the following line:

PasswordAuthentication yes

After you add the PasswordAuthentication yes line to the /etc/ssh/sshd_config file, restartSSH by entering the following command:

systemctl restart sshd

After you enable the password authentication on your system, check for the root authority access to add acluster to PowerHA GUI. If the root access is disabled by SSH, edit the /etc/ssh/sshd_config file byadding the following line:

PermitRootLogin yes

After you add the PermitRootLogin yes line to the /etc/ssh/sshd_config file, restart SSH byentering the following command:

systemctl restart sshd

You will need the following information to properly configure SSH for the PowerHA SystemMirror GUI:

Note: You need to connect to only one node in the cluster. After the node is connected, the PowerHASystemMirror GUI automatically adds all other nodes in the cluster.

• Host name or IP address of at least one node of the cluster.• User ID and corresponding password which will be used for SSH authentication on that node.• SSH password or SSH key location.

Supported web browsers

PowerHA SystemMirror GUI is supported in the following web browsers:


https://www.openssl.org/

http://www.openssh.com/

• Google Chrome Version 57, or later• Firefox Version 54, or later

Installing PowerHA SystemMirror GUIBefore you install the PowerHA SystemMirror GUI in your Linux environments, you must ensure that allprerequisites are met.

Prerequisites

The following prerequisites must be met before you install the PowerHA SystemMirror GUI on a Linuxsystem:

• The PowerHA SystemMirror package must be installed on your system.• The KSH93 scripting package is required on each SUSE Linux Enterprise Server and Red Hat Enterprise

Linux (RHEL) system.• The make utility is required on a RHEL 8 system where GUI server is to be installed.

Installing the PowerHA SystemMirror GUI

The PowerHA SystemMirror GUI software is delivered as a single tar file, which includes all RPMs andinstallation scripts. The PowerHA SystemMirror GUI installation script is available in thePHA723Linux64/Gui/installPHAGUI directory. The installation script includes of all steps forinstalling the PowerHA SystemMirror GUI software such as checking for errors and other setup.

1. Untar the tar file to access the installation script.2. To install the agent and server, run the following script:

./installPHAGUI

3. To install the agent only, run the following script:

./installPHAGUI –a -c

Note: If any previous installation of the agent exists, uninstall it by running the following script beforeyou install the new version:

./installPHAGUI -u -a -c

4. To install the server only, run the following script:

./installPHAGUI –s -c

Notes:

• You must run the ./installPHAGUI command only on one node of a cluster or multi-clusterenvironment that will be designated as server for the PowerHA SystemMirror GUI.

• Before you install the PowerHA SystemMirror GUI on the SUSE or RHEL environment, enter thefollowing commands on all nodes of cluster:

– Change the Password Authentication parameter to yes in the /etc/ssh/sshd_config file.– Restart the sshd daemon by entering the rcsshd restart command.

After the PowerHA SystemMirror GUI is installed, you can log in to the PowerHA SystemMirror GUI from aweb browser. In hybrid environments, if you want to support the AIX clusters on Linux server, you mustrun the /usr/es/sbin/cluster/ui/server/bin/smuiinst.ksh command to complete theinstallation process. The smuiinst.ksh command automatically downloads and installs the remainingfiles that are required to complete the PowerHA SystemMirror GUI installation process. Thesedownloaded files are not shipped in the RPMs because the files are licensed under the General PublicLicense (GPL).

PowerHA SystemMirror graphical user interface (GUI) 69

The PowerHA SystemMirror GUI server must have internet access or an HTTP proxy that is configured toallow access to the internet to run the smuiinst.ksh command. If you are using an HTTP proxy, youmust run the smuiinst.ksh -p command to specify the proxy information, or you must specify theproxy information by using the http_proxy environment variable. If the PowerHA SystemMirror GUI serverdoes not have internet access, complete the following steps:

1. Copy the smuiinst.ksh file from the PowerHA SystemMirror GUI server to a system that is running theUNIX compatible operating system that has internet access.

2. Run the smuiinst.ksh -d /directory command where /directory is the location where youwant to the download the files. For example, /smuiinst.ksh –d /tmp/smui_rpms.

3. Copy the downloaded files (/tmp/smui_rpms) to a directory on the PowerHA SystemMirror GUIserver.

4. From the PowerHA SystemMirror GUI server, run the smuiinst.ksh -i /directory commandwhere /directory is the location where you copied the downloaded files (/tmp/smui_rpms).

Migrating the PowerHA SystemMirror GUI database

When you are migrating a PowerHA SystemMirror GUI server that is not configured for high availability,the PowerHA SystemMirror GUI database is migrated automatically. For a highly available PowerHASystemMirror GUI server, you must manually migrate the database as automatic migration of thedatabase is not possible. You can migrate the PowerHA SystemMirror GUI database after all nodes in thecluster are migrated to the new version of PowerHA SystemMirror. You cannot migrate the PowerHASystemMirror GUI database during offline or rolling migration due to conflicts over concurrent access tothe shared file system that contains the PowerHA SystemMirror GUI database. The PowerHASystemMirror GUI database cannot be migrated until all the nodes in the cluster are migrated to the newversion of PowerHA SystemMirror.

To migrate the PowerHA SystemMirror GUI database, perform the following steps on the active PowerHASystemMirror GUI server node after all the nodes are migrated:

1. Stop the PowerHA SystemMirror GUI server by running the following command:

systemctl stop phauiserver && sleep 11

2. Mount the file system pha_gui_fs by running following command:

mount/opt/pha_gui_fs

3. Verify that the smui.db file is available in the /opt/pha_gui_fs directory by running the followingcommand:

ls -l /opt/pha_gui_fs/smui.db

4. Complete the GUI database migration by running the following command:

/usr/es/sbin/cluster/ui/server/bin/sqlite/opt/pha_gui_fs/smui.db < /usr/es/sbin/cluster/ui/server/node_modules/smui-server/resources/ddl-upgrade.sql

5. Restart the GUI server by running the following command:

systemctl start phauiserver

Logging in to the PowerHA SystemMirror GUIAfter you install the PowerHA SystemMirror GUI, you can log in to the PowerHA SystemMirror GUI from aweb browser.

To log in to the PowerHA SystemMirror GUI, complete the following steps:

1. Open a supported web browser, and enter https://HostName:8080/#/login, where HostName is thesystem on which you installed the cluster.es.smui.server RPM.


2. On the login page, enter the user name and password and click Log In. You can use the existing usernames and passwords that exist on the system to login.

Note: The first time you log in to the PowerHA SystemMirror GUI, you must add clusters to the GUI orcreate new clusters.

To add existing clusters to the PowerHA SystemMirror GUI, complete the following steps:

1. In the navigation pane, click the icon.2. Select Add cluster.3. Complete all required information.4. Click Discover clusters.

To create new clusters for the PowerHA SystemMirror GUI, complete the following steps:

1. In the navigation pane, click the icon.2. Select Create cluster.3. Complete all required information.4. Click Complete.

Navigating the PowerHA SystemMirror GUIThe PowerHA SystemMirror graphical user interface (GUI) provides you with a web browser interface thatcan monitor your PowerHA SystemMirror environment.

Health summary

In the PowerHA SystemMirror GUI, you can quickly view all events for a cluster in your environment. Thefollowing figure identifies the different areas of the PowerHA SystemMirror GUI that are used to viewevents and status.

Figure 15. Health summary

Navigation paneThis area displays all the zones, clusters, sites, nodes, and resource groups in a hierarchy that wasdiscovered by the PowerHA SystemMirror GUI. You can click to view resources for each cluster.


Notes:

• The clusters are displayed in alphabetic order. However, any clusters that are in a Critical orWarning state are listed at the top of the list.

• The Network tab is not included for the PowerHA SystemMirror for Linux Version 7.2.

Health SummaryThis menu provides cluster administrative features for the selected item. You can create a cluster,add an existing cluster, remove an existing cluster, import multiple clusters, or create a zone from the

Health Summary menu.

ScoreboardThis area displays the number of zones, clusters, nodes, and resource groups that are in Critical,Warning, or Maintenance state. You can click Critical, Warning, or Maintenance to view all themessages for a specified resource. For example, in Figure 1, there are 2 resource groups identified. Ifthe warning icon was highlighted and you clicked the warning icon, all messages (critical, warning,and normal) for the 2 resource groups would be displayed.

Event filterIn this area, you can click the icons to display all events in your environment that correspond to aspecific state. You can also search for specific event names.

Event timeline

This area displays events across a timeline of when the event occurred. This area allows you to viewthe progression of events that lead to a problem. You can zoom in and out of the time range by usingthe + or – keys or by using the mouse scroll wheel.

Event listThis area displays the name of the event, the time when each event occurred, and a description of theevent. The information that is displayed in this area corresponds to the events you selected from theevent timeline area. The most recent event that occurred is displayed first. You can click this area todisplay more detailed information about the event such as possible causes and suggested actions.

Action MenuThis area displays the following menus options:Users

PowerHA SystemMirror GUI allows an admin to create and manage users by using Users menu.The admin can assign built-in roles to new users.

Note: You can only add user names that are defined on the host running the PowerHASystemMirror GUI server.

RolesThe Roles tab displays information about available roles for each user. An admin can createcustom roles and provide permission to different users. PowerHA SystemMirror GUI provides thefollowing roles:

• ha_root• ha_mon• ha_op• ha_admin


SnapshotsAs an admin can create any number of snapshots and restore the existing cluster configuration astaken in snapshot. You can also edit existing snapshot and view details of the snapshot based ondifferent filter options available.

ZonesYou can create zones, which are groups of clusters. An admin can create zones and assign anynumber of clusters to a zone. You can also add new zones or edit existing zones.

Activity LogsYou can view information about all activities performed in the PowerHA SystemMirror GUI thatresulted in a change by using the View Activity Log tab. This view provides various filters tosearch for specific activities for the cluster, roles, zone, or user management changes.

Smart AssistA Smart Assist is a wizard that provides product-specific knowledge and assistance for configuring anapplication to be highly available within a PowerHA SystemMirror cluster. In PowerHA SystemMirrorVersion 7.2.3, or later, the necessary PowerHA SystemMirror resources, such as resource groups,service addresses, application start and stop scripts, and application monitors, are created for youautomatically by the Smart Assist after you provide the required information through the wizard.

GUI Server High AvailabilityYou can configure the PowerHA SystemMirror GUI server to make it highly available within the clusterby setting the GUI Server High Availability option to Enabled.

Additional Information

• Customers who move the host name of a node when their application moves must be aware that thiscan cause some complications with the GUI. We’ve added code in PowerHA to automatically handlethis, as long as the host name is changed and /etc/hosts is updated appropriately. However, short-term and temporary host name changes are not handled at this time.

• To add a cluster to the GUI successfully, make sure that SSH is not prompting for any unexpected input(except prompting for a password or passphrase). To test this on the GUI server you can run thefollowing script ssh <REMOTE_NODE> hostname, where <REMOTE_NODE> is the node that you areusing to add the cluster to the GUI.

• The default port that the GUI server uses is 8080, which is used to connect you to the browser. Thedefault port for the agents, that the server uses to communicate with them is 8081. If the firewall willnot allow these ports, you will get communication errors.

• Clusters may only be managed by one GUI server.• If you install the GUI server within a cluster, only install it on one of the nodes unless you are planning

to use the new GUI high availability configuration utility to make the GUI server highly available in thatcluster.

• When the Smart Assist SAP HANA operations delete all resource groups, the resource group that iscreated to keep the GUI server highly available is also deleted. You must repeat the process of makingthe GUI server highly available.

• The migration operation must be performed only after removing the high availability of the GUI server.The process of making the GUI server highly available must be repeated after the migration operation.

Importing multiple clustersYou can add multiple clusters to the PowerHA SystemMirror GUI at a time by importing a template filethat contains details about the cluster. Adding large number of clusters in a single operation not onlysimplifies the task but also saves time.

The template file is in the comma-separated value (CSV) file format. An example template file follows.


DEFAULT LOGIN ID, DEFAULT PASSWORD/SSH KEY<UserName>,<Password>IP ADDRESS/HOSTNAME,USERNAME,PASSWORD,ZONE<IP Address/Hostname>,<UserName>,<Password>,<ZoneName><IP Address/Hostname>,<UserName>,<Password>,<ZoneName><IP Address/Hostname>,<UserName>,<Password>,<ZoneName>

You can edit the template file by using a text editor or a spreadsheet tool. Ensure that you save the file inthe plain-text CSV format after you make any changes.

You can specify a hostname or the IP address for one node of each cluster that you want to import intothe PowerHA SystemMirror GUI.

You must specify the user ID and the corresponding password that is used for Secure Shell (SSH)authentication on that cluster node. The user ID that you specify must have the root authority to addclusters. To add clusters as a non-root user, a root user must configure the sudo command to be usedon one of the nodes in the cluster. For more information about how to configure the sudo command, see“Logging in as a non-root user” on page 78.

Users can use an SSH password or a public SSH key for authentication of their user ID. Ensure that theSSH key is contained within a single cell. The SSH key must not span across multiple cells. An SSH keythat requires a passphrase is not supported.

If the login credentials for all your clusters is the same, you can set the login credentials for all yourclusters by specifying the login credentials in the DEFAULT LOGIN ID, DEFAULT PASSWORD/SSH KEYline of the template. Also, if you want to use the default credentials, you can specify the login credentialsin the DEFAULT LOGIN ID, DEFAULT PASSWORD/SSH KEY line of the template. For example, to addall clusters as a root user, you can enter root in the default user name field. The default credentials canbe overridden by specifying a different user name, password, or SSH key for the hostname in thesubsequent rows.

Specifying cluster zones are optional. If you specify a cluster zone and if that cluster zone does not exist,the cluster zone is automatically created when you import clusters. If the cluster zone that you specifiedalready exits, the cluster is imported and added to the cluster zone. If you do not specify a cluster zone,the cluster is added to the Unassigned clusters list. The unassigned clusters are visible to all PowerHASystemMirror GUI users with the ha_mon role. Users must have the required privileges to performoperation on these clusters. For more information about roles and privileges, see “Adding users andassigning roles ” on page 77.

To import multiple clusters to the PowerHA SystemMirror GUI, perform the following steps:

1. On the navigation pane, click corresponding to Health Summary.2. Click Add Multiple Clusters.3. Download and update the template file to add information about each cluster that you want to import.4. Upload the template file and click Continue to add clusters.

If you want to download the detailed information about clusters that could not be added, click Exportas Excel.

Smart Assist for SAP HANAA Smart Assist is a wizard that offers product-specific knowledge and assistance for configuring anapplication to be highly available within a PowerHA SystemMirror cluster.

In PowerHA SystemMirror Version 7.2.3, or later, the Smart Assist for SAP HANA simplifies the process ofcreating and managing the policies for SAP HANA. The necessary PowerHA SystemMirror resources, suchas resource groups, service addresses, application start and stop scripts, and application monitors, arecreated automatically by the Smart Assist after you provide the required information through the wizard.


You can create more than one SAP HANA policy, and view and manage your saved SAP HANA policies.You can also modify, delete, or activate an SAP HANA policy. Only one SAP HANA policy can be activatedat a time.

When you activate an SAP HANA policy, you can specify how you want to activate the policy. You canchoose to bring the resource groups online automatically. If you choose to delete the existing resources,all the existing resources on the cluster are deleted. If you choose not to delete the existing resources,and if there are resources present for the SAP HANA policy that you are activating, PowerHASystemMirror automatically deletes the resources that are associated with the SAP HANA policy.

The following configuration modes are supported for SAP HANA:

• Performance optimized: The Performance optimized mode uses SAP HANA system replication. Thesecondary system, which is configured identically to the primary system, and is in hot standby modeuntil a takeover occurs.

• Active/Active (read enabled) secondary: The Active/Active (read enabled) secondary mode enablesthe secondary system for read-only queries. PowerHA SystemMirror configures a secondary virtual IPv4address to establish a direct connection to the secondary system. PowerHA SystemMirror alreadysupports a virtual IPv4 address, which is activated on the primary system. When the secondary systemis not available, PowerHA SystemMirror activates the secondary virtual IPv4 address along with the firstvirtual IPv4 address on the primary system.

• Cost optimized: The Cost optimized mode uses the secondary node for a non-productive SAP HANAinstance. If the primary SAP HANA instance fails, PowerHA SystemMirror tries to restart the SAP HANAinstance locally on the primary node when the value of the PREFER_TAKEOVER parameter is FALSE. Ifthe primary node fails, takeover occurs. During the takeover, the non-productive SAP HANA is stoppedfirst and the SAP HANA instance on the secondary node is started.

The Smart Assist also displays the replication status and the replication direction. The replication status isindicated by the following icons:

• Green: Indicates that the data replication is active.• Yellow: Indicates that the synchronization is in progress.• Grey: Indicates that the replication is inactive.• Red: Indicates a replication error.

You can specify how the log information must be sent to the secondary system by configuring thereplication mode. The following log replication modes can be configured for SAP HANA:

• Synchronous: In this mode, the primary system waits for an acknowledegment that confirms that thelog information is received and persisted by the secondary system before committing any datatransactions.

• Synchronous in-memory: In this mode, the primary system waits for an acknowledegement thatconfirms that the log information is received by the secondary system before committing any datatransactions.

• Asynchronous: In this mode, the primary system does not wait for any acknowledegement orconfirmation from the secondary system.

Troubleshooting PowerHA SystemMirror GUIYou can view log files to help you troubleshoot PowerHA SystemMirror GUI.

The information in this section is only a reference guide for different techniques and log files you may beable to use to troubleshoot problems with thePowerHA SystemMirror GUI. You should always contactIBM support if you are uncertain about the information here or how to solve your problem.

Log files

You can use the following log files to troubleshoot PowerHA SystemMirror GUI:


smui-server.logThis log file is located in the /usr/es/sbin/cluster/ui/server/logs/ directory. The smui-server.log file contains information about the PowerHA SystemMirror GUI server.

smui-agent.logThis log file is located in the /usr/es/sbin/cluster/ui/agent/logs/ directory. The smui-agent.log file contains information about the agent that is installed on each PowerHA SystemMirrornode.

notify-event.logThis log file is located in the /usr/es/sbin/cluster/ui/agent/logs/ directory. The notify-event.log file contains information about all PowerHA SystemMirror events that are sent from theagent to the PowerHA SystemMirror server.

Problems logging in to PowerHA SystemMirror GUI

If you are experiencing problems logging in to the PowerHA SystemMirror GUI, complete the followingsteps:

1. Check for issues in the /usr/es/sbin/cluster/ui/server/logs/smui-server.log file.2. Verify that the smuiauth command is installed correctly. Also, verify that the smuiauth command

has the correct permissions by running the ls -l command from the /usr/es/sbin/cluster/ui/server/node_modules/smui-server/lib/auth/smuiauth directory. An output that is similarto the following example is displayed when you run the ls -l command:

-r-x------ 1 root system 21183 Aug 31 21:48

3. Verify that you can run the smuiauth command by running the smuiauth -h command.4. Verify that the pluggable authentication module (PAM) framework is configured correctly by locating

the following lines in the /etc/pam.d/smuiauth file:

RHEL:auth requisite pam_nologin.soauth required pam_env.soauth optional pam_gnome_keyring.soauth required pam_unix.so try_first_passaccount requisite pam_nologin.soaccount required pam_unix.so try_first_pass

SUSE:auth requisite pam_nologin.soauth include common-authaccount requisite pam_nologin.soaccount include common-account

Note: The PAM configuration occurs when you install the PowerHA SystemMirror GUI server.

Problem adding clusters to the PowerHA SystemMirror GUI

If you are not able to add clusters to the PowerHA SystemMirror GUI, complete the following steps:

1. Check for issues in the /usr/es/sbin/cluster/ui/server/logs/smui-server.log file.

a. If sftp-related signatures exist in the log file, such as Received exit code 127 whileestablishing SFTP session, a problem exists with the SSH communication between thePowerHA SystemMirror GUI server and the cluster you are trying to add.

b. From the command line, verify that you can connect to the target system by using SSH File TransferProtocol (SFTP). If you cannot connect, verify that the daemon is running on the PowerHASystemMirror GUI server and the target node by running the ps –ef | grep –w sshd | grep–v grep command. You can also check the SFTP subsystem configuration in the /etc/ssh/sshd_config file and verify that following path is correct:

Subsystem sftp /usr/libexec/openssh/sftp-server


If the path is not correct, you must enter the correct path in the /etc/ssh/sshd_config file, andthen restart the sshd subsystem.

2. Check for issues in the /usr/es/sbin/cluster/ui/agent/logs/agent_deploy.log file on thetarget cluster.

3. Check for issues in the /usr/es/sbin/cluster/ui/agent/logs/agent_distribution.logfile on the target cluster.

The PowerHA SystemMirror GUI is not updating status

If the PowerHA SystemMirror GUI is not updating the cluster status or displaying new events, completethe following steps:

1. Check for issues in the /usr/es/sbin/cluster/ui/server/logs/smui-server.log file.2. Check for issues in the /usr/es/sbin/cluster/ui/agent/logs/smui-agent.log file. If

certificate-related problem exists in the log file, the certificate on the target cluster and the certificateon the server do not match. An example of a certificate error follows:

WebSocket server - Agent authentication failed, remoteAddress:::ffff:10.40.20.186, Reason:SELF_SIGNED_CERT_IN_CHAIN

Migrating the PowerHA SystemMirror GUI database

See “Migrating the PowerHA SystemMirror GUI database” on page 70.

Related informationInstalling PowerHA SystemMirror GUI

Configuring PowerHA SystemMirror GUIUse this information to learn how to use Role-Based Access Control (RBAC), change the default locationof log files, and change ports for the PowerHA SystemMirror GUI.

Adding users and assigning rolesYou can create users in the PowerHA SystemMirror graphical user interface (GUI) and use the Role-BasedAccess Control (RBAC) system to assign elevated privileges to the users.

Defining users

In PowerHA SystemMirror GUI, you can create and manage users by clicking Actions > Users.

Note: Users that are defined in the PowerHA SystemMirror GUI map to user accounts that are defined onthe host running the PowerHA SystemMirror GUI server. The login ID of a PowerHA SystemMirror GUIuser must match the login ID on the PowerHA SystemMirror GUI server host. If a user is defined in thePowerHA SystemMirror GUI and if the user ID does not have a matching login ID on the PowerHASystemMirror GUI server host, the user cannot log in to the PowerHA SystemMirror GUI.

After you define users, you can restrict their access to the clusters that are managed within the PowerHASystemMirror GUI by using clusters zones. Cluster zones provide a method to limit the scope of a userand can be used to implement multi-tenancy.

By default, all users that are defined on the PowerHA SystemMirror GUI server host are allowed to log into the PowerHA SystemMirror GUI. However, you can restrict specific users from logging into thePowerHA SystemMirror GUI, by changing the configuration file of the PowerHA SystemMirror GUI server.By default, the monitor role ha_mon is assigned automatically to a user who logs in to the PowerHASystemMirror GUI. Elevated privileges and capabilities can be assigned to users that are explicitly definedin the PowerHA SystemMirror GUI by using the predefined roles.


Role-based access control

In PowerHA SystemMirror 7.2.2, or later, the PowerHA SystemMirror GUI has a built-in Role-BasedAccess Control (RBAC) system that is independent from the AIX operating system RBAC system and it isalso easy to access.

Roles

The permission system of the GUI is based on roles. Permissions are allocated to a role and then the roleis allocated to one or more users. A user with no role is assigned a default role with view-only monitoringcapabilities. The permission system of the GUI is based on roles. Permissions are allocated to a role andthen the role is allocated to one or more users.

The PowerHA GUI provides the following predefined roles:ha_root

Users with this role can access every zone that is defined in the GUI without any restrictions. This roleis equivalent to the root access. When you log in as the root user, you are granted ha_rootpermissions.

Note: This role can be used while setting up the GUI, which includes setting up user access andcreating zones (if zones will be used).

ha_adminIn this role, you are an administrator and can perform all actions except defining users and zones.

ha_opThe ha_op, or operator, role can access only a subset of cluster management capabilities such asstarting and stopping cluster services, starting and stopping resource groups, moving resource groupsto another node, creating snapshots, and performing cluster verification.

ha_monThe ha_mon, or monitor, role is the default role that is assigned automatically to a user that logs in tothe GUI as a non-root user without being added to the GUI’s user list.

Note: In this role, you cannot perform any actions on a cluster and you have view-only access. If theusers log in that do not have a user account created for them in the GUI, they will not be grantedaccess to any of the zones and they are able to see unassigned clusters only (clusters that are notassigned to any zones)

All non-ha_root users might only access to zones that they have been assigned to by an ha_root user.They can also access clusters that are not assigned to any zones.

Note: Custom roles can be defined when none of the pre-defined roles are suitable for your needs.

Logging in as a non-root userThe PowerHA SystemMirror graphical user interface (GUI) uses encrypted communication to monitor andto manage clusters through a PowerHA SystemMirror GUI agent. The PowerHA SystemMirror GUI agent isconfigured and started when a cluster is added to the PowerHA SystemMirror GUI.

Before a cluster is added to the PowerHA SystemMirror GUI, PowerHA SystemMirror uses Secure Shell(SSH) for secure remote communication.

The following tasks use SSH and you must have the root authority to perform the following tasks:

• Add an existing cluster to the PowerHA SystemMirror GUI• Create a cluster that is automatically added to the PowerHA SystemMirror GUI• Clone a cluster from a snapshot and add the cluster to the PowerHA SystemMirror GUI

A non-root user must be provided root authority to perform these tasks. A root user must use the sudocommand to provide root access to a non-root user. The sudo command must be preconfigured to allowthe specific commands to be run as root user.


Note: After the PowerHA SystemMirror GUI agent is started, the communication switches from SSH to thePowerHA SystemMirror GUI agent. The PowerHA SystemMirror GUI agent provides the necessaryauthority to perform the specific tasks.

To use the sudo command, you must install the following RPMs from the AIX Toolbox for LinuxApplications website:

• cyrus-sasl• db• gettext• libgcc• ncurses• openldap• sudo• zlib

A root user must complete the following steps to configure the sudo command to create a user login, andto provide the created user the ability to discover clusters:

1. Run the visudo command or directly edit the /etc/sudoers file with a text editor.2. Add the following text that corresponds to your operating system to the end of the /etc/sudoers

file:

User_Alias POWERHA_GUI_USERS = user1,user2,user3,user4Cmnd_Alias POWERHA_GUI_CMDS = /usr/bin/clmgr -v query nodes, /usr/bin/clmgr query cluster, \ /usr/bin/clmgr list hosts TYPE=$TYPE, \ /usr/bin/clmgr -g SMUI list physical_volume NODES=$NODES, \ /usr/bin/clmgr query cluster, \ /usr/bin/clmgr -T $ACTIVITY_ID add cluster $NAME NODES=$NODES, \ /usr/bin/clmgr -T $ACTIVITY_ID manage snapshot restore *, \ /bin/mkdir -p /usr/es/sbin/cluster/ui/security, \ /bin/tar -xf /tmp/smui-security.tar, \ /bin/ksh93 ./deployment.sh,/bin/ksh93 ./distribute.sh, \ /bin/rm -f ./deployment.sh ./distribute.sh ./configuration-agent.json ./smui-security.tarPOWERHA_GUI_USERS ALL= NOPASSWD:SETENV: POWERHA_GUI_CMDS

3. Create a user login and password.4. Add the login ID of the user to the /etc/sudoers configuration file that was represented as the useruser1 in step 2.

Configuring the PowerHA SystemMirror GUI server to be highly availableYou can configure PowerHA SystemMirror GUI server to be highly available either by using the highavailability wizard for the PowerHA SystemMirror GUI server or by using the PowerHA SystemMirrorcommand line.

Configuring the PowerHA SystemMirror GUI server to be highly available by using the PowerHASystemMirror GUI

You can use the GUI Server High Availability option that is available in the cluster view of the PowerHASystemMirror GUI to enable or to disable the high availability of the PowerHA SystemMirror GUI server.

Before you enable the PowerHA SystemMirror GUI server for high availability, ensure that the server fileof the PowerHA SystemMirror GUI is installed on all nodes of the cluster. The cluster must be in an onlinestate and stable, or in an offline state.

When you set the GUI Server High Availability option to Enabled, the Enable the GUI Server for HighAvailability wizard is displayed. You must specify the volume group that must be used for shared storage,the service IP address to access to the PowerHA SystemMirror GUI server, and network on which theservice IP address must be configured to make the PowerHA SystemMirror GUI server highly available.The volume group that you specify must be created before you enable the PowerHA SystemMirror GUIserver for high availability . You cannot assign the volume group that is created by using the PowerHA


https://www.ibm.com/developerworks/aix/library/aix-toolbox/alpha.html

https://www.ibm.com/developerworks/aix/library/aix-toolbox/alpha.html

SystemMirror GUI, because the volume group is already assigned to a resource group. You can create thevolume group by using SMIT or by running the clmgr command. Create the volume group across all thecluster nodes and ensure that the volume group is not assigned to any resource group.

Configuring the PowerHA SystemMirror GUI server to be highly available by using the command line

IBM PowerHA GUI server can be made highly available either manually or by using the available utilities.

/usr/es/sbin/cluster/ui/server/bin/server_high_availability.sh

Syntax:server_high_availability.sh

<Volume_Group> <Service_IP> <Network> [<Netmask>]

server_high_availability.sh –-remove

server_high_availability.sh --help

After successful running of this utility, GUI server will behave like any other application that PowerHAmonitors. For example, the server can be started and stopped via clmgr or SMIT (for AIX), movedto another node, and automatically failover to another node when a problem is detected.

This utility also creates the following objects:

• Resource group (named pha_gui_server_rg)• Service IP• File system (created at /opt/pha_gui_fs)• Application controller (named pha_gui_server_app)• Application monitor (named pha_gui_server_mon)

The names and locations of the objects that the utility creates should not be changed. If any are changed,the removal feature of the utility will no longer function. If the default names and locations are leftunchanged, the --remove flag will successfully remove them all.

Additional actions taken by this utility follows:

• Copies the server database to the new shared filesystem• Copies the security files from the server where the utility is being run to the other server node• This will overwrite any preexisting security files on that remote node

Warning: --remove flag does not undo this. The overwritten files will be gone for good.

Limiting access to the PowerHA SystemMirror GUIYou can increase security of the PowerHA SystemMirror GUI by restricting users access to PowerHASystemMirror GUI and by preventing anonymous login.

To prevent anonymous login operations and to restrict user access to the PowerHA SystemMirror GUI,you must manually modify the server configuration file, /usr/es/sbin/cluster/ui/server/configuration-server.json, on the PowerHA SystemMirror GUI server host.

Note: Before you modify the server configuration file, ensure that you back up the server configurationfile.

Restricting user access to the PowerHA SystemMirror GUIWhen the PowerHA SystemMirror GUI server or a PowerHA SystemMirror GUI agent is initially set up,any users, who are defined on the GUI server host, can log in to the PowerHA SystemMirror GUI withonly monitoring capabilities.To allow access to users that are explicitly defined in the PowerHA SystemMirror GUI, set the value ofthe permitAllSystemUsers property to no in the server configuration file. The default value for thepermitAllSystemUsers property is yes to preserve compatibility with earlier versions.


Preventing anonymous loginAnonymous login operation is permitted for root users. Anonymous users are tagged as root in thePowerHA SystemMirror activity log file. The activity log file does not display the identity of the userand hence the login information is anonymous.To ensure that a login ID maps to a specific user in PowerHA SystemMirror GUI and to preventanonymous root login to the PowerHA SystemMirror GUI, set the value of the permitRootLoginproperty to no in the server configuration file. When you set the property to no, a root user cannotlog in to the PowerHA SystemMirror GUI. The default value for the permitRootLogin property isyes to preserve compatibility with earlier versions and to allow initial setup of the PowerHASystemMirror GUI.

Note: If you block the root user, ensure that at least one user is defined with the ha_root role. If nousers have the ha_root role, PowerHA SystemMirror ignores the permitRootLogin=no setting,and permits the root user to log in to the PowerHA SystemMirror GUI until a user with the ha_rootrole is defined.

After you modify the server configuration file, you must restart the PowerHA SystemMirror GUI server forthe new configuration to take effect. To restart the PowerHA SystemMirror GUI server and the PowerHASystemMirror GUI agent, enter the following command:

stopsrc -cs phauiserver; sleep 11; startsrc -s phauiserver

Restricting the use of IP address in the PowerHA SystemMirror GUIBy default, the PowerHA SystemMirror GUI server uses the IP address that is associated with thehostname command. However, you can specify a different IP address that is used by the PowerHASystemMirror graphical user interface (GUI) to communicate with the PowerHA SystemMirror GUI server.

You must manually modify the /usr/es/sbin/cluster/ui/server/configuration-server.json configuration file on the PowerHA SystemMirror GUI server host to specify a different IPaddress.

Note: Ensure that you back up the /usr/es/sbin/cluster/ui/server/configuration-server.json configuration file before you modify the configuration file.

When a cluster is added to the PowerHA SystemMirror GUI server for monitoring and management, thePowerHA SystemMirror GUI agents that are installed on each cluster node are configured and started.The configuration process pushes the PowerHA SystemMirror GUI agent configuration file,configuration-agent.json, to each cluster node. The PowerHA SystemMirror GUI agentconfiguration file is created dynamically on the PowerHA SystemMirror GUI server at run time. ThePowerHA SystemMirror GUI agent configuration file contains the IP address and port number that areused by the PowerHA SystemMirror GUI agent to communicate with the PowerHA SystemMirror GUIserver.

If you want to use a different IP address for the PowerHA SystemMirror GUI server communication, in theserver configuration file, /usr/es/sbin/cluster/ui/server/configuration-server.json,specify the new IP address in the serverAddress property.

After you modify the server configuration file, you must restart the PowerHA SystemMirror GUI server forthe new configuration to take effect. To restart the PowerHA SystemMirror GUI server and the PowerHASystemMirror GUI agent, enter the following command:

stopsrc -cs phauiserver; sleep 11; startsrc -s phauiserver

The specified IP address is used only for those clusters that are added to the PowerHA SystemMirror GUIafter you updated the IP address in the serverAddress property. If PowerHA SystemMirror GUI isalready managing clusters and if you want these clusters to switch to the new IP address, you must eitherremove all your clusters from the PowerHA SystemMirror GUI and add them again, or you must manuallyupdate the PowerHA SystemMirror GUI agent configuration file on each node for every cluster.


Removing existing clusters and adding them again

To remove the existing clusters and to add them again, perform the following steps:

1. Remove clusters that are already managed by PowerHA SystemMirror GUI.

a. On the navigation pane, click corresponding to Health Summary.b. Click Remove Clusters.c. Select the cluster that you want to remove.d. Click Remove.

2. Add the clusters back to the PowerHA SystemMirror GUI.

• If you want to add few clusters, perform the following steps:

a. On the navigation pane, click corresponding to Health Summary.b. Click Add an Existing Cluster.c. Specify the hostname of the cluster node, authentication method to reach the cluster node,

and the user ID and password for authentication.d. Click Add.e. Click Close.f. Repeat steps “2.a” on page 82 - “2.e” on page 82 for each cluster that you want to add.

• If you want to add many clusters, perform the following steps:

a. On the navigation pane, click corresponding to Health Summary.b. Click Add Multiple Clusters.c. Download and update the template file to add information about each cluster that you want

to import.d. Upload the template file and click Continue to add clusters. Click Close.

Updating the agent configuration file of the PowerHA SystemMirror GUIOn every cluster node in each cluster, modify the /usr/es/sbin/cluster/ui/agent/configuration-agent.json file, and replace the old server IP address in the serverURIproperty with the new IP address that you specified in the serverAddress property of the PowerHASystemMirror GUI server configuration file. After modifying the configuration file of the PowerHASystemMirror GUI agent, you must restart the PowerHA SystemMirror GUI agent for the newconfiguration to take effect. To restart the PowerHA SystemMirror GUI agent, enter the followingcommand:

stopsrc -cs phauiagent; sleep 11; startsrc -s phauiagent


Smart Assist for PowerHA SystemMirrorSmart Assist manages a collection of PowerHA SystemMirror components that you identify to support aparticular application. You can view these collections of PowerHA SystemMirror components as a singleentity, and in PowerHA SystemMirror that entity is represented by an application name.

PowerHA SystemMirror for SAP HANAThe high availability concepts of SAP HANA reduce the downtime of a system in case of any software orhardware failures.

The SAP HANA high availability policy defines all SAP HANA components as resources and starts or stopsthem in a pre-defined sequence to provide high availability for your SAP HANA system. The SAP HANAcomponents must be specified as an automated resource in the PowerHA SystemMirror by using SmartAssist.

The following components of the software stack in an SAP HANA installation must be highly available:Primary and secondary host with the following processes on each host:

• The hdbdaemon daemon manages the following subprocesses:

– hdbindexserver– hdbnameserver– hdbxsengine– hdbwebdispatcher– hdbcompileserver– hdbpreprocessor

• The sapstartsrv start, stop, and monitor processes for the hdbdaemon daemon.• IP address to access the primary host

PowerHA SystemMirror Version 7.2.3, or later, supports the SAP HANA Active/Active (read enabled)secondary configuration mode through Smart Assist. The Active/Active (read enabled) secondaryconfiguration mode reduces the load on the primary system and extends read capabilities. In this mode,the primary system remains active and it supports read and write operations and the secondary systemwill be in the standby mode and it supports only the read operation.

When you specify the CONFIG=ACTIVE_ACTIVE parameter, PowerHA SystemMirror configures asecondary virtual IPv4 address for establishing a direct connection to the secondary system. PowerHASystemMirror already supports a virtual IPv4 address, which is active on the primary system. When thesecondary system is not available, PowerHA SystemMirror activates the secondary virtual IPv4 addressalong with the first virtual IPv4 address on the primary system.

In PowerHA SystemMirror Version 7.2.3 for Linux with Service Pack 1, or later, you must specify thePREFER_TAKEOVER parameter when you configure SAP HANA through Smart Assist. ThePREFER_TAKEOVER parameter defines whether PowerHA SystemMirror switch over the SAP HANAdatabase to the secondary node when the primary SAP HANA instance fails instead of restarting the SAPHANA instance on the primary node locally. By default the value of the PREFER_TAKEOVER parameter isTRUE for the Performance Optimized and Active-Active SAP HANA configuration modes becauseswitching over to another node in these configuration modes is faster than restarting the node locally.

PowerHA SystemMirror Version 7.2.3 for Linux with Service Pack 1, or later, supports the SAP HANACost Optimized configuration mode through Smart Assist. When you specify the CONFIG=COST_OPTparameter, the secondary node is used for a non-productive SAP HANA and the secondary node mustlimit usage of system resources. The table preload for productive SAP HANA is turned off. Whenever


takeover must occur, the non-productive system stops first and then takeover occurs. ThePREFER_TAKEOVER parameter value is FALSE for Cost Optimized configuration mode. If the PowerHAmonitoring script detects failure in SAP HANA database, script tries to restart SAP HANA on the samenode if the PREFER_TAKEOVER parameter value is FALSE. You must specify the following additionaldetails for the Cost Optimized configuration mode.

• Non-productive SAP HANA SID and instance: PowerHA SystemMirror creates the non-productionresource group with a fixed node that is secondary node and manages the non-production SAP HANAdatabase by using the new resource group.

• Global allocation limit: PowerHA SystemMirror uses the global allocation limit value to specify themaximum amount of memory that is used by production SAP HANA memory manager. PowerHASystemMirror also turns off the table preload of the production SAP HANA database on the secondarynode.

• srHook script password and port number: PowerHA SystemMirror configures the srHook script on thesecondary node to cancel the memory limit of the production SAP HANA database and enables tablepreload when the standby node takes over and becomes the primary node of production SAP HANAdatabase.

To configure the srHook script, PowerHA SystemMirror updates the following details in the hook script:

– <SYSTEM password>: The SAP HANA system database password that was defined by user duringSAP HANA installation.

– <Port number>: The port number at which SAP HANA system database listens. For example,3NN13 or 3NN15, where NN is the instance number of the SAP HANA system database. You mustreplace NN with the production SAP HANA system database instance number. 30013 is the portnumber for scenarios with multiple tenants, and 30015 is the port number for scenarios with notenants.

Planning for SAP HANASAP HANA High Availability feature is supported for the following configurations:

• SAP HANA Version 2.0• Two node cluster deployment

Review the following information before you configure Smart Assist for SAP HANA:

• You must install the SAP HANA database before configuring resources by using the Smart Assistapplication.

• The user-defined resources such as Persistent IP can be deleted after configuring the Smart Assistpolicies.

• You must configure the NFS tiebreaker by using the clmgr command.

Prerequisites

Consider the following SAP HANA prerequisites for the System Replication scenario having data preload:

• The SAP HANA software version of the secondary system must be equal to the primary system.• The secondary system must have the same SAP system ID (SID) and same instance number as the

primary system.• System replication between two systems on the same host is not supported.• System Replication must be set up before configuring the Smart Assist application by using the clmgr

command.• The required ports must be available. The same <instance number> is used for the primary and

secondary systems. The <instance number>+1 must be free on both systems because this port range isused for system replication communication.


• An initial data backup or snapshot must be performed on the primary system before you activate thesystem replication.

• The SAP HANA database must be in an offline state before the Smart Assist policy is activated.Otherwise, the database stops if the Smart Assist policy is activated.

• You must ensure that the IP address that is used for the service IP label is deactivated, if it wasmanually added for testing, before you activate the new Smart Assist policy.

Configuring the Smart Assist application for SAP HANAYou can invoke the SAP HANA Smart Assist Policy Setup Wizard by using the clmgr setupsmart_assist command.

clmgr setup smart_assist APPLICATION=SAP_HANA SID=TS2 INSTANCE=HDB02 CONFIG=PERFORM_OPT MODE=UPDATE

The SAP HANA High Availability feature is defined for the SAP ID TS2 and for the instance name asHDB02.

The SAP ID TS2 and the instance name HDB02 are provided during the SAP HANA installation.

To configure an SAP HANA Smart Assist, complete the following steps:

1. From the command line, enter the clmgr setup smart_assist command.2. Enter the following values for different flags:

APPLICATIONEnter the name of application as SAP_HANA for the application you want to configure by using theSmart Assist application.

SIDEnter the SAP HANA system ID that you configured during the SAP installation.

INSTANCEThe instance name is used for the instance directory that contains all necessary files for SAPHANA.

CONFIGIndicates the configuration mode that is used when you run Smart Assist. The CONFIG parameteris available in PowerHA SystemMirror Version 7.2.3, or later. You can specify the following valuesfor the CONFIG parameter:

• PERFORM_OPT: In this mode, the secondary system, which is configured identically to theprimary system is in the standby mode until a takeover occurs.

• ACTIVE_ACTIVE: The ACTIVE_ACTIVE mode is similar to the PERFORM_OPT mode. However, inthis mode, the secondary system has read access.

• COST_OPT: In this mode, the maximum amount of memory used by production SAP HANAsystem on the secondary node is limited by configuring the value of Global AllocationLimit parameter.

The CONFIG parameter is optional and the default value for this parameter is PERFORM_OPT.

MODEIndicates the mode that is used by Smart Assist if the specified Smart Assist policy is alreadyavailable. You can specify the following values for the MODE parameter:

• NEW: In this mode, Smart Assist uses the default policy template to configure the new SmartAssist policy.

• UPDATE: In this mode, Smart Assist uses the specified saved policy for configuring theparameters of the saved Smart Assist policy. If a saved policy is not available, a new Smart Assistpolicy is configured by using the default Smart Assist policy template.

The MODE parameter is optional, and the default value for this parameter is UPDATE.

Smart Assist for PowerHA SystemMirror 85

Closing the Smart Assist wizard

You can close the Smart Assist wizard by choosing one of the following options:

1. Use the 0 option to close the wizard. The user input is saved by using the clmgr smart_assistcommand. If all parameter values are specified correctly which is indicated by the overall parameterstatus OK, you are prompted for the Smart Assist policy activation.

2. Use the X option to cancel the wizard. Confirm your cancel request to quit the wizard without savingany changes. The parameter summary is not created and you cannot activate the policy. The Finishand Cancel options are available in the Overview page and also in each parameter pane. The twoactions are run independently from the page or pane in which the option is selected.

Activating the Smart Assist wizard

If you used the 0 option to close the wizard and if all parameter values are specified correctly with overallparameter status OK, you can activate the policy.

Depending on the option that you select, one of the following actions is performed:Yes, activate as new policy

The Smart Assist policy is activated as a new policy. If you choose this option, the command that issimilar to the following example is invoked:

clmgr add smart_assist APPLICATION =SAP_HANA SID=<value> INSTANCE=<instance value>

Yes, activate by updating currently active policyThe Smart Assist policy is activated by updating the active policy. If you choose this option, thecommand that is similar to the following example is invoked:

clmgr update smart_assist APPLICATION=SAP_HANA SID=<value> INSTANCE=<instance value>

No, save modifications and exitThe Smart Assist policy is not activated. Your modifications are saved and the wizard is closed.

No, return to parameter overviewThe Smart Assist policy is not activated. Your modifications are not saved and the wizard displays theOverview page.

SAP HANA operations

The following operations are performed by SAP HANA:

Setup Smart AssistConfigures a Smart Assist policy. You can also save, activate, and modify a Smart Assist policy.

Add Smart AssistActivates a Smart Assist policy and deletes the existing resources.

Update Smart AssistUpdates an activated Smart Assist policy. The resources that are associated with the Smart Assistpolicy are deleted and the resources are created again. All other resources are not changed.

Modify Smart AssistModifies an activated Smart Assist policy and deletes the existing resources.

Note: The Modify Smart Assist operation is similar to the Add Smart Assist operation and thisoperation is removed from PowerHA SystemMirror Version 7.2.2 Service Pack 2 and later.

Delete Smart AssistPowerHA SystemMirror Version 7.2.2 Service Pack 1 and earlier: Deactivates the activated policy andthe existing resources are deleted.PowerHA SystemMirror Version 7.2.2 Service Pack 2 and later: Deactivates the requested activatedpolicy without deleting any other resources except the resources of the requested policy.


PowerHA SystemMirror Version 7.2.3 or later: When you configure a policy by using the Smart Assist,the parameters of Smart Assist policy are saved. If you specify the DEL_POLICY=YES parameterwhile running the clmgr delete smart_assist command, the Smart Assist policy is deleted.

PowerHA SystemMirror Version 7.2.3 for Linux with Service Pack 1, or later, the DEL_POLICY=YESparameter deletes the configured srHook files and PowerHA SystemMirror configuration.

Query Smart AssistLists the parameters that are associated with a Smart Assist policy.

Verifying the SAP HANA High Availability feature

You can verify the start and stop function of the SAP HANA High Availability feature if the installation runssuccessfully. You can also verify the failover scenario for planned and unplanned outages.

To verify the start and stop function of the SAP HANA High Availability feature, complete the followingsteps:

• To start your SAP HANA system, enter the clmgr online resource_group ALL command.• To display the different resource groups created by the wizard and their corresponding state, enter theclRGinfo command.

• To stop your SAP HANA system, enter the clmgr offline resource_group ALL command.

PowerHA SystemMirror for SAP NetWeaverThe high availability concepts for a SAP system provides information about choosing the Smart Assistpolicy (SAP Central Services) depending on the planned SAP installation.

The high availability setup reduces the downtime of a SAP system in case of any software or hardwarefailures. The high availability solution for SAP uses PowerHA SystemMirror to automate all SAPcomponents. PowerHA SystemMirror detects failed components and restarts or initiates a failover. Thissetup also helps to reduce the operational complexity of a SAP environment and to avoid operator errorsresulting from this complexity. PowerHA SystemMirror supports high availability for ABAP and JAVA SAPCentral Services installations.

Planning for SAP NetWeaverSAP NetWeaver High Availability feature is supported for the following configurations:

• Two-node cluster deployment• PowerHA SystemMirror 7.2.3 or later: Four-node cluster deployment

Review the following information before you configure Smart Assist for SAP NetWeaver:

• The following SAP Central Services high availability configurations are supported through the wizard:

– Advanced Business Application Programming (ABAP) System Central Services (ASCS) HighAvailability

– JAVA System Central Service instance (SCS) High Availability– SAP Application Server only High Availability– SAP router failover– SAP Web Dispatcher failover– Start after dependency for SAP HANA database

• You must install SAP NetWeaver components before you configure resources by using Smart Assist.• User-defined resources such as Persistent IP are deleted when a Smart Assist policy is activated. You

must configure the user-defined resources again.• You must configure the Network File System (NFS) tie breaker by using the clmgr command if youconfigure even number of nodes in the cluster.


Standalone Enqueue Server 2 support

The Standalone Enqueue Server 2 (ENSA2) in SAP NetWeaver Application Server for an SAP AdvancedBusiness Application Programming (ABAP) system provides improved scalability and advanced SMENQtransaction. You can use the advanced SMENQ transaction instead of the previous SM12 transaction tomonitor and administer ENSA2. The standalone ENSA2 provides a simplified high availability architectureand the Enqueue Replicator 2 (ERS2) provides an improved failover process.

In PowerHA SystemMirror Version 7.2.3, or later, Smart Assist for SAP NetWeaver discovers and supportsEnqueue Server (ENSA) by using Enqueue Replicator Server (ERS) and ENSA2 with ERS2. The ENSA2 canbe configured either in a two-node cluster configuration or a multi-node cluster configuration. In a two-node cluster configuration, ABAP SAP Central Services (ASCS) fails over to the node where ERS is running.In a multi-node cluster configuration, ASCS fails over to any online node in the cluster where ERS is notrunning.

Installing an SAP systemInstall an SAP system by using the Software Provisioning Manager tool. The tool is also called as thesapinst installation tool. After the installation is complete, configure the SAP profile parameters andverify the installation.

PrerequisitesThe following prerequisites must be met before you start the sapinst tool to install the SAP system:

• Use unique instance numbers for each instance that you install on a single host for an SAP System ID(SID). The SAP installation does not work if you do not use unique instance numbers.

• The SAP system ID (SID) and the instance number of SAP NetWeaver must be the same on all clusternodes.

• Create a separate directory for each component that you want to install for saving installation log filesand trace files. You must switch to the installation directory for that component before you start thesapinst tool.

• Ensure that you configure the Linux automounter daemon on all the nodes to connect to the NFS serverand to automatically mount the default SAP directory, /sapmnt.

• Register permanent entries for the virtual host names either in the DNS server or in the /etc/hostsfile on all nodes.

• Ensure that the network interfaces that you want to use have the same name on each SAP system. Foreach virtual IP address that is defined in the high availability policy, an equivalency of network interfaceis created. Only those network interfaces that have the same name on each node can be part of anequivalency. An equivalency is a group of resources that provide the same function. For example, whennetwork interfaces are defined as equivalencies, if one network interface is offline, another networkinterface with the same name takes over.

• The SAP instance directory /usr/sap/<sapsid>/<instancename> must be present in the local filesystem.

• The directory /usr/sap/trans must be located on the shared file system.

Installing a high availability SAP systemYou can install the mandatory and optional SAP system components on the nodes where you want toprovide high availability of SAP Netweaver by using the sapinst tool.

To install SAP system components on a node, perform the following steps:

1. Activate all virtual IP addresses that correspond to the virtual host names before you start theinstallation process.

2. Use the sapinst tool to install the following components with the installation option SAP Systems >High-Availability System. For some installation tasks, you must start the sapinst tool with a virtualhost name.

• Central Services Instance for ABAP (ASCS) or Central Services Instance for Java™ (SCS)• Enqueue Replication Server instance (ERS) for ASCS or Enqueue Replication Server instance for SCS


• Database instance• Primary application server instance on one node of the cluster• Additional application server instance on other nodes of the cluster• Web Dispatcher instance (optional)• SAP router (optional)

3. Remove all the virtual IP addresses that are activated.

Note: If you do not remove the virtual IP addresses after the installation and complete the initialmanual testing, you might experience incorrect behavior of the SAP high availability solution.

Configuring SAP profile parametersAfter you install an SAP system on all the nodes, you must set up the profile parameters for the newlycreated server profile files for high availability.

Ensure that you set up the profile parameters according to the Standalone Enqueue Server (ENSA) orStandalone Enqueue Server 2 (ENSA2) configuration. The profile files are present in the sapmnt/<SID>/profile and /usr/sap/<SID>/ERS<ID>/profile/ directories.

To configure SAP profile parameters, complete the following steps:

1. Disable the Autostart parameter on all SAP instances in each profile by commenting the lineAutostart = 1.

2. Ensure that PowerHA SystemMirror is already installed on cluster nodes. Add the following entries tothe default profile file DEFAULT.PFL.

#-----------------------------------------------------------------------# SAP high availability connector#-----------------------------------------------------------------------service/halib=/usr/sap/<SID>/SYS/exe/dbg/saphascriptco.soservice/halib_cluster_connector=/usr/sbin/rsct/sapolicies/sap/bin/sap_tsamp_cluster_connector

If the /usr/sap/<SID>/SYS/exe/dbg/saphascriptco.so file is not found on your local system,use the /usr/sap/<SAPSID>/SYS/exe/uc/<your_platform>/saphascriptco.so file.

3. Disable the SAP restart capability for Enqueue Server and Enqueue Replication Server in thecorresponding profiles. Otherwise, the automatic restart capability of SAP that was enabled by usingthe startsapsrv command mismatches with the PowerHA SystemMirror start function and mightcause problems with high availability.Set the SAP profile parameter for Enqueue Server (ENSA) and Enqueue Replicator Server (ERS) toStart_Program_<NR> in the ENSA and ERS profiles that are present in the /sapmnt/<SID>/profile directory. For servers other than ENSA and ERS, set the SAP profile parameter toRestart_Program_<NR>.

If the SAP profile parameter is set to Start_Program_<NR>, then

• When you start the SAP application instances for the first time, the startup operation is performed bythe startsapsrv framework.

• When a node failure occurs, during the recovery process, the SAP application instances are startedby PowerHA SystemMirror. PowerHA SystemMirror resources can start automatically or resourcescan failover.

If the SAP profile parameter is set to Restart_Program_<NR>, then

• When you start the SAP application instances for the first time, the startup operation is performed bythe startsapsrv framework.

• When a node failure occurs, during the recovery process, the SAP application instances are startedby PowerHA SystemMirror. PowerHA SystemMirror resources can start automatically.

4. Save the enqueue backup file in the NFS-mounted /sapmnt/<SID>/global directory so that it canbe accessed from all nodes in the cluster. The enqueue backup file is created when the enqueue


server is started. Add the following parameter to the ABAP SAP Central Services (ASCS) profile toshare the enqueue backup files across all nodes in the cluster:

enque/backup_file = $(DIR_GLOBAL)/ENQBCK(A)SCS

5. Verify the /usr/sap/sapservices file on all nodes.

The /usr/sap/sapservices file must contain commands to start the SAP start servicesapstartsrv for each SAP instance installed on the server. The /usr/sap/sapservices file musthave permission of 0755 (rwxr-xr-x) as the /usr/sap/sapservices file is read by the sapinitservice.

If there is no entry for the SAP start service in the /usr/sap/sapservices file after installation, youmust add an entry for starting the SAP start service in a single line without line breaks inthe /usr/sap/sapservices file. An example entry follows:

#!/bin/shLD_LIBRARY_PATH= /usr/sap/<sid>/<instance>/exe:$LD_LIBRARY_PATH;export LD_LIBRARY_PATH; /usr/sap/<sid>/<instance>/exe /sapstartsrv pf=<DIR_PROFILE>/<INSTANCE_START_profile> -D -u <sid>adm>

Verifying the installationTo verify the setup for the ERS replication operation, the SAP system is started and a manual failover ofthe ABAP SAP Central Services (ASCS) and standalone Enqueue Replicator Server (ERS) occurs. First, theASCS instance fails over from the first node to the second node. Then, the ASCS instance fails over fromthe second node back to the first node.

Ensure that the following prerequisites are met before you verify the installation:

• All ASCS, ERS, primary application server, and additional application server instances are stopped.• You have the authority to perform all steps as a <sid>adm user.

1. Start the ASCS instance on the first node and the ERS instance on the second node.a) Start the ERS instance on the second node by using the following commands:

ifconfig <interface_name> <ERS_IP_alias> netmask <IP_netmask> alias upstartsap r3 ERS<ID>

b) Start the ASCS instance and the primary application server instance on the first node by using thefollowing commands:

ifconfig <interface_name> <ASCS_IP_alias> netmask <IP_netmask> alias upstartsap r3 ASCS<ID>startsap r3 DVEBMGS<ID>

c) Start the additional application server instance on the second node by using the followingcommand:

startsap r3 D<ID>

d) Check the replication status for each ERS instance on the second node by using the ensmon (ENSA)utility or the enq_admin (ENSA2) utility.

• To check the replication status by using the ensmon utility, run the following command:

ensmon pf=/usr/sap/<SID>/ERS<ID>/profile/<SID>_ERS<ID>_<node2>

Select the task Get replication information. An output that is similar to the following example isdisplayed:

.

..Replication is enabled in server, replication server is connected. Replication is active...


• To check the replication status by using the enq_admin utility, run the following command:

enq_admin --replication_state pf=/usr/sap/<SID>/SYS/profile/<SID>_ASCS<ID>_<ASCS_HOSTNAME>

An output that is similar to the following example is displayed:

Enqueue Server 2

Replication-State = ON2019-03-19 01:17:09; OK; 'Replication State'; Response=294 usec

e) Verify whether you can start all application servers successfully.

• Log on to the primary application server for Advanced Business Application Programming (ABAP)by using the SAP graphical user interface (GUI).

• Log on to the additional application server for ABAP by using the SAP GUI.• Log on to the primary application server for Java by using a web browser. The default address ishttp://node1:5<ID>00/index.html.

• Log on to the additional application server for Java by using a web browser. The default isaddress is http://node2:5<ID>00/index.html.

2. Change the direction of replication from the second node to the first node. Start the ASCS instance onthe second node and the ERS instance on the first node.a) Stop the additional application server instance on the second node by using the following

commands:

stopsap r3 ERS<ID>stopsap r3 D<ID>ifconfig <interface_name> <ERS_IP_alias> delete

b) Stop ASCS, ERS, and primary application server instances on the first node by using the followingcommands:

stopsap r3 DVEBMGS<ID>stopsap r3 ASCS<ID>ifconfig <interface_name> <ASCS_IP_alias> delete

c) Start ASCS IP instances on the second node by using the following commands:

ifconfig <interface_name> <ASCS_IP_alias> netmask <IP_netmask> alias upstartsap r3 ASCS<ID>

d) Start ERS instances on the first node by using the following commands:

ifconfig <interface_name> <ERS_IP_alias> netmask <IP_netmask> alias upstartsap r3 ERS<ID>

e) Check the replication status for each ERS instance on the second node by using the ensmon (ENSA)utility or the enq_admin (ENSA2) utility.

• To check the replication status by using the ensmon utility, run the following command:

ensmon pf=/usr/sap/<SID>/ERS<ID>/profile/<SID>_ERS<ID>_<node2>

Select task Get replication information. An output that is similar to the following example isdisplayed:

.

..Replication is enabled in server, replication server is connected. Replication is active...


• To check the replication status by using the enq_admin utility, run the following command:

enq_admin --replication_state pf=/usr/sap/<SID>/SYS/profile/<SID>_ASCS<ID>_<ASCS_HOSTNAME>

An output that is similar to the following example is displayed:

Enqueue Server 2

Replication-State = ON2019-03-19 01:17:09; OK; 'Replication State'; Response=294 usec

f) Start the primary application server instance on the first node by using the following command:

startsap r3 DVEBMGS<ID>

g) Start the additional application server instance on the second node by using the followingcommand:

startsap r3 D<ID>

h) Verify whether you can start all application servers successfully.

• Log on to the primary application server for ABAP by using the SAP graphical user interface (GUI).• Log on to the additional application server for ABAP by using the SAP GUI.• Log on to the primary application server for Java by using a web browser. The default address ishttp://node1:5<ID>00/index.html.

• Log on to the additional application server for Java by using a web browser. The default isaddress is http://node2:5<ID>00/index.html.

After successful verification, all ASCS, ERS, primary application server, and additional application serverinstances must be stopped and the virtual IP addresses must be deactivated.

Configuring SAP NetWeaverYou can start the Smart Assist Policy Setup wizard by using the clmgr setup smart_assistcommand.

Examples

1. To configure the ABAP SAP Central Services (ASCS) HA solution, enter the following command:

clmgr setup smart_assist APPLICATION=SAP_ABAP SID=TS2 INSTANCE=SCS02 MODE=NEW

2. To configure the SAP JAVA (SCS) HA solution, enter the following command:

clmgr setup smart_assist APPLICATION=SAP_JAVA SID=TS2 INSTANCE=SCS02 MODE=NEW

3. To configure the SAP APPSERVER HA solution, enter the following command:

clmgr setup smart_assist APPLICATION=SAP_APPSERVER SID=TS2 INSTALL_TYPE={ABAP|JAVA} MODE=NEW

SAP NetWeaver High Availability

SAP NetWeaver High Availability is defined for SAP ID (TS2 in above example) and instance name (SCS02in above example). The SAP ID and the instance name are provided during SAP NetWeaver installation.

To setup SAP NetWeaver Smart Assist, complete the following steps:

1. From the command line, run the clmgr setup smart_assist command.2. Enter the flag value as follows:


APPLICATIONEnter the name of an application to configure by using Smart Assist, for example, SAP_ABAP,SAP_JAVA, or SAP_APPSERVER.

SIDEnter the SAP system ID configured during SAP installation.

INSTANCEInstance name is used for the instance directory that contains all necessary files for the SAPCentral Services instance.

INSTALL_TYPEThe INSTALL_TYPE is used only for SAP_APPSERVER configuration. The value for this parameteris either SAP_JAVA or SAP_ABAP.

MODEIndicates the mode that is used by Smart Assist if the specified Smart Assist policy is alreadyavailable. You can specify the following values for the MODE parameter:

• NEW: In this mode, Smart Assist uses the default policy template to configure the new SmartAssist policy.

• UPDATE: In this mode, Smart Assist uses the specified saved policy for configuring theparameters of the saved Smart Assist policy. If a saved policy is not available, a new Smart Assistpolicy is configured by using the default Smart Assist policy template.

The MODE parameter is optional, and the default value for this parameter is UPDATE.

Closing the Smart Assist wizard

You can close the Smart Assist wizard by choosing one of the following options:

1. Use the 0 option to close the wizard. The user input is saved by using the clmgr smart_assistcommand. If all parameter values are specified correctly which is indicated by the overall parameterstatus OK, you are prompted for the Smart Assist policy activation.

2. Use the X option to cancel the wizard. Confirm your cancel request to quit the wizard without savingthe changes. The parameter summary is not created and you cannot activate the policy. The Finish andCancel options are available in the Overview page and also in each parameter pane. The two actionsare run independently from the page or pane in which the option is selected.

Activating the Smart Assist wizard

If you used the 0 option to close the wizard and if all parameter values are specified correctly with overallparameter status OK, you can activate the policy.

Depending on the option that you select, one of the following actions is performed:Yes, activate as new policy

The Smart Assist policy is activated as a new policy. If you choose this option, the command that issimilar to the following example is invoked:

clmgr add smart_assist APPLICATION =SAP_JAVA|SAP_ABAP SID=<value> INSTANCE=<instance value>

Yes, activate by updating currently active policyThe Smart Assist policy is activated by updating the currently active policy. If you choose this option,the command that is similar to the following example is invoked:

clmgr update smart_assist APPLICATION=SAP_JAVA|SAP_ABAP SID=<value> INSTANCE=<instance value>

No, save modifications and exitThe Smart Assist policy is not activated. Your modifications are saved and the wizard is closed.

No, return to parameter overviewThe Smart Assist policy is not activated. Your modifications are not saved and the wizard displays theOverview page.


SAP NetWeaver operations

The following operations are performed by SAP NetWeaver:Setup Smart Assist

Configures a Smart Assist policy. You can also save, activate, and modify a Smart Assist policy.Add Smart Assist

Activates the Smart Assist policy. All existing resources are deleted.Update Smart Assist

Updates the active policy without stopping any resource. All existing resources are either modified orkept unchanged. The new resources are added to the policy.

Modify Smart AssistModify the active policy and remove all the existing resources. All resources that are not deleted arenot stopped.

Note: Modify operation is similar to Add operation so this operation is removed from PowerHASystemMirror Version 7.2.2 Service Pack 2 and later.

Delete Smart AssistPowerHA SystemMirror Version 7.2.2 Service Pack 1 and earlier: Deactivates the active policy. Allexisting resources are deleted.PowerHA SystemMirror Version 7.2.2 Service Pack 2 and later: Deactivates the requested activepolicy without deleting any other resources except the resources of requested policy.PowerHA SystemMirror Version 7.2.3 or later: When you configure a policy by using Smart Assist, theparameters of the Smart Assist policy are saved. If you specify the DEL_POLICY=YES parameterwhile running the clmgr delete smart_assist command, the Smart Assist policy is deleted.

Query Smart AssistLists the parameters that are associated with a Smart Assist policy.

Verifying the SAP NetWeaver High Availability

You can verify the start and stop function of the SAP HANA High Availability feature if the installation runssuccessfully. You can also verify the failover scenario for planned and unplanned outages.

To verify the start and stop function of the SAP HANA High Availability feature, completes the followingsteps:

• To start your SAP NetWeaver system, enter the clmgr online resource_group ALL command.• To display the different resource groups created by the wizard and their corresponding state, enter theclRGinfo command.

• To stop your SAP NetWeaver system, enter the clmgr offline resource_group ALL command.• You can verify whether the StartAfter relationship is configured between the application server and

SAP HANA by checking whether the application server resource groups change to an Online state onlyafter SAP HANA resource groups are in an Online state after running the clRGinfo command.

Note:

The StartAfter relationship between the SAP NetWeaver system (application server) and the SAP HANAHigh Availability feature is supported only when the SAP NetWeaver system and the SAP HANA HighAvailability feature are configured on the same cluster.


Troubleshooting PowerHA SystemMirror Smart Assist issuesThese topics describe how to troubleshoot PowerHA SystemMirror Smart Assist issues.

PowerHA SystemMirror is not able to harvest some values during Wizard executionThis topic examines the situations when PowerHA SystemMirror is not able to harvest some values duringwizard execution such as an instance name or a service IP address by using the specified host name.

Problem

PowerHA SystemMirror is not able to harvest some values during Wizard execution.

Solution

This problem has the following possible causes:

• Check that all the nodes of the cluster are online. This can be checked by using the clmgr querynode -v command.

• Check and run the harvest command that is mentioned in the Smart Assist Wizard manually in theLinux system. If the harvest command works, it is probably due to the delay in system responsebecause Smart Assist waits for 10 seconds to get a response from the harvest command.

Replication Site Name that is configured in Smart Assist wizard is different fromReplication Site Name shown in SAP HANA setup wizard

This topic examines the situations when the replication site name that is configured in Smart Assistwizard is different from what is configured manually during SAP HANA replication setup.

Solution

The SAP HANA Replication Site Name that is configured while running the Smart Assist Wizard must bethe same as specified during SAP HANA Replication setup.

Smart Assist policy fails to activate

Problem

PowerHA SystemMirror policy fails to activate with a reason Not able to find some file orpath.

Solution

This problem has the following possible causes:

• Check that all nodes of a cluster are online by using the clmgr query node -v command.• Check whether the other nodes of the cluster are reachable. An error might occur if one of the cluster

nodes associated with the Smart Assist policy is not reachable.

Smart Wizard does not detect or show one or more Ethernet interfaces in the list

Solution

Check whether the Ether interfaces are running. If interfaces are just brought up, it may take sometimefor Smart Assist to detect the interfaces.


Notices

This information was developed for products and services offered in the US.

IBM may not offer the products, services, or features discussed in this document in other countries.Consult your local IBM representative for information on the products and services currently available inyour area. Any reference to an IBM product, program, or service is not intended to state or imply that onlythat IBM product, program, or service may be used. Any functionally equivalent product, program, orservice that does not infringe any IBM intellectual property right may be used instead. However, it is theuser's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in thisdocument. The furnishing of this document does not grant you any license to these patents. You can sendlicense inquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle Drive, MD-NC119Armonk, NY 10504-1785US

For license inquiries regarding double-byte character set (DBCS) information, contact the IBM IntellectualProperty Department in your country or send inquiries, in writing, to:

Intellectual Property LicensingLegal and Intellectual Property LawIBM Japan Ltd.19-21, Nihonbashi-Hakozakicho, Chuo-kuTokyo 103-8510, Japan

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS"WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR APARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties incertain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodicallymade to the information herein; these changes will be incorporated in new editions of the publication.IBM may make improvements and/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM websites are provided for convenience only and do not inany manner serve as an endorsement of those websites. The materials at those websites are not part ofthe materials for this IBM product and use of those websites is at your own risk.

IBM may use or distribute any of the information you provide in any way it believes appropriate withoutincurring any obligation to you.

Licensees of this program who wish to have information about it for the purpose of enabling: (i) theexchange of information between independently created programs and other programs (including thisone) and (ii) the mutual use of the information which has been exchanged, should contact:

IBM Director of LicensingIBM CorporationNorth Castle Drive, MD-NC119Armonk, NY 10504-1785US

Such information may be available, subject to appropriate terms and conditions, including in some cases,payment of a fee.


The licensed program described in this document and all licensed material available for it are provided byIBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or anyequivalent agreement between us.

The performance data and client examples cited are presented for illustrative purposes only. Actualperformance results may vary depending on specific configurations and operating conditions.

Information concerning non-IBM products was obtained from the suppliers of those products, theirpublished announcements or other publicly available sources. IBM has not tested those products andcannot confirm the accuracy of performance, compatibility or any other claims related to non-IBMproducts. Questions on the capabilities of non-IBM products should be addressed to the suppliers ofthose products.

Statements regarding IBM's future direction or intent are subject to change or withdrawal without notice,and represent goals and objectives only.

All IBM prices shown are IBM's suggested retail prices, are current and are subject to change withoutnotice. Dealer prices may vary.

This information is for planning purposes only. The information herein is subject to change before theproducts described become available.

This information contains examples of data and reports used in daily business operations. To illustratethem as completely as possible, the examples include the names of individuals, companies, brands, andproducts. All of these names are fictitious and any similarity to actual people or business enterprises isentirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programmingtechniques on various operating platforms. You may copy, modify, and distribute these sample programsin any form without payment to IBM, for the purposes of developing, using, marketing or distributingapplication programs conforming to the application programming interface for the operating platform forwhich the sample programs are written. These examples have not been thoroughly tested under allconditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of theseprograms. The sample programs are provided "AS IS", without warranty of any kind. IBM shall not beliable for any damages arising out of your use of the sample programs.

Each copy or any portion of these sample programs or any derivative work must include a copyright noticeas follows:© (your company name) (year).

Portions of this code are derived from IBM Corp. Sample Programs.© Copyright IBM Corp. _enter the year or years_.

Privacy policy considerationsIBM Software products, including software as a service solutions, (“Software Offerings”) may use cookiesor other technologies to collect product usage information, to help improve the end user experience, totailor interactions with the end user or for other purposes. In many cases no personally identifiableinformation is collected by the Software Offerings. Some of our Software Offerings can help enable you tocollect personally identifiable information. If this Software Offering uses cookies to collect personallyidentifiable information, specific information about this offering’s use of cookies is set forth below.

This Software Offering does not use cookies or other technologies to collect personally identifiableinformation.

If the configurations deployed for this Software Offering provide you as the customer the ability to collectpersonally identifiable information from end users via cookies and other technologies, you should seekyour own legal advice about any laws applicable to such data collection, including any requirements fornotice and consent.

98 Notices

For more information about the use of various technologies, including cookies, for these purposes, seeIBM’s Privacy Policy at http://www.ibm.com/privacy and IBM’s Online Privacy Statement at http://www.ibm.com/privacy/details the section entitled “Cookies, Web Beacons and Other Technologies” andthe “IBM Software Products and Software-as-a-Service Privacy Statement” at http://www.ibm.com/software/info/product-privacy.

TrademarksIBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International BusinessMachines Corp., registered in many jurisdictions worldwide. Other product and service names might betrademarks of IBM or other companies. A current list of IBM trademarks is available on the web atCopyright and trademark information at www.ibm.com/legal/copytrade.shtml.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/orits affiliates.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Red Hat, the Red Hat "Shadow Man" logo, and all Red Hat-based trademarks and logos are trademarks orregistered trademarks of Red Hat, Inc., in the United States and other countries.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Notices 99

http://www.ibm.com/privacy

http://www.ibm.com/privacy/details

http://www.ibm.com/privacy/details

http://www.ibm.com/software/info/product-privacy

http://www.ibm.com/software/info/product-privacy

http://www.ibm.com/legal/us/en/copytrade.shtml

Index

Special Characters/var/pha/log/clmgr/clutils.log 59/var/pha/log/hacmp.out 59

Aapplication 26

Cclient 15cluster

client 15IP address takeover 20network 15, 18node 15, 18physical components 14resource groups 27reviewing message log files 59

communication interface 19configuration

standbyexample 29

takeovermutual 31one-sided 30two-node mutual 32

Configuring dependencies between resource groups 53Configuring netmon.cf file 19

EEvents 71example

standby configuration 29

Hheartbeating

point-to-point network 21TCP/IP network 21

IInstalling 69IP address takeover 20issues

PowerHA SystemMirror startup 60, 62–65, 95

Llog

reviewing cluster message 59Log files 71Logging in 70

logical network 19

MManaging users and user groups 15message log

reviewing 59monitoring

persistent node IP labels 27

NNavigating 71network

communication interfaces 19heartbeating 21logical 19physical 19

node 15, 18

Ppersistent node IP label 27physical network 19Planning 67PowerHA SystemMirror 15PowerHA SystemMirror startup issues 60, 62–65, 95

Rresource

applications 26service IP address 26, 27service IP label 26, 27

resource groupfallback 28fallover 28startup 28

reviewingmessage log files 59

Sservice IP address 26, 27service IP label 26, 27Split configurations 22

TTroubleshooting 75

Index 101

powerha systemmirror for linux version 7 - ibm · what's new in powerha systemmirror 7.2 for...

Documents