load balancing and high availability for bmc remedy action request system

February 1, 2008

Load Balancing and High Availability for BMC Remedy Action Request System

Kelly Deaver, BMC Technical Marketing

© Copyright February 1, 2008 BMC Software, Inc.2

Agenda

› Why use server group and load balancing

› How server group enhances BMC Business Service Management

› Products covered

› Examples of possible environment

› Licensing

› Installing – What to do and what not to do

› Configuring

› Managing

› BMC Remedy Knowledge Management and load balancing


Why Use Server Groups with Load Balancing

› Symptoms

– Screens taking too long to build/refresh

– Users not able to login (server busy or down)

› Server group as the solution

– Increased performance

• Administrative tasks can be assigned to a specific AR System Server

• User interactions are distributed between servers

– Redundancy for fail over

• If one AR System Server or AR System Mid-tier fails, others pick up the load and continue


Powered

By

AR System

BSM and AR System

Powered

By

AR System


Products Covered in this Presentation

› BMC Remedy Action Request System

› BMC Atrium CMDB

› BMC Remedy FTS (Full Text Search)

› BMC Remedy Flashboards

› BMC Remedy Asset Management

› BMC Remedy Change Management

› BMC Remedy Service Desk

› BMC Service Level Management

› BMC Knowledge Management


Configuration - Basic

Load

Balancer

AR System Servers

DatabaseClients

Administrative Operations

Escalation, Reconciliation, etc.

Top server is primary for Administration and

Escalation operations. Only one at a time can do this.

The other two are for back up of Admin operations.

All three are used for User interaction.


Configuration – With Firewall and Load Balancer

Load

Balancer

AR System Servers

DatabaseClients Firewall

“BMCRemedy”

(virtual server)

11.40.34.94:3030

“BMC1”

(real server)

11.40.35.47:2020

“BMC2”

11.40.35.49:2020

“BMC3”

11.40.35.51:2020

Here the Client comes in through a firewall to port 3030.

The load balance then acts as in the earlier example but the port for each server is 2020


Create a hosts file entry for each AR System

Server on the companion web server -

BMCRemedy 11.40.35.51

Configuration – Firewall, Web Servers & 1 Load Balancer

Load

Balancer

AR System

Servers


Web

1

Web

2

Web

3

Web Servers

“BMCRemedy”

(virtual server)

11.40.34.94:3030

“BMC3”

11.40.35.51:2020

As in the previous slide the client requests passes through

the firewall and into the load balancer but this time it is a web

client.

The load balancer distributes the load to the 3 web servers.

Each web server is paired with an AR System server by

making an IP address entry in the hosts file of the web server


Configuration – Firewall, Web Servers & 2 Load Balancers

Load

Balancer

AR System

Servers


Web Servers

Once a connection has been made through

the 2nd load balancer, session affinity must

be maintained.

Only

For

Failover

After Initial connection is made between the mid-tier and the

AR Server it remains a connection until either server fails.

When one server fails another pairing is made. Even if the

other server comes back up, doesn’t rebalance.


Session Affinity

› Why

– User instance information is cached on the AR System server

– Large data transfers may involve “chunking”. All blocks of data need to follow the same path.

– If encryption is used, encryption is set between the two servers.

› How

– Determine the “sticky” affinity timeout period

• AR System default user session timeout is 60 minutes

• The Sticky Timeout must be longer than the user session timeout

– In AR.cfg / AR.conf

• Internal-User-Instance-Timeout (the session timeout)

– The default setting is 1 hour, the maximum is 2 hours, and the minimum is 30 minutes.

– On Your Load Balancer

• Set the “sticky” affinity timeout


Configuration – DMZ, Firewall, Web Servers & 2 Load Balancers (Higher Security, WANs, etc)

Load Balancer

AR System Servers

Database

Clients FirewallWeb Servers

Load

Balancer

DMZ

LAN

1

2

3

4

5

6

7

10

11

12

13

16Q?A

8

9

1415


Configuration – DMZ, 2 Firewalls, Web Servers & 2 Load Balancers (Better Performance)

Load Balancer

AR System Servers

Database

Clients FirewallWeb Servers

Load

Balancer

DMZ

LAN

Firewall

2 Network

Pipes

To Reduce

Bottlenecks


Configuration - NOT

Load

Balancer

AR System

Servers

DatabaseClients Load

Balancer

For

Failover

Use A

Disk

Array

Instead

Load balancing is for spreading the load, not

condensing it. Plus you need to have a sticky

connection between the db and the AR Server.


Purchasing and Licensing

› Purchases required per environment (dev, test, production)

– 1 copy of AR System per server running it

– 1 copy of each application (all servers in the group use the same application)

› License 6.x or 7.0.x – on each server

– You must specify server group when ordering license keys

• You have to order through BMC Support-Licensing, not online key generation

– License is stored in license file on each server

• 1 AR Server

• Application (server group name must be consistent)

• Fixed User (*SITE*)

• Floating User (server group must be consistent)

› Licensing 7.1 is easier…

– For in-depth info see the session

BMC Remedy Customer Enforced Licensing Simplification – Thursday 2PM


Licensing 7.1 Servers

› License is stored in shared database

– Create an entry for each AR System Server (with required key matching each host)

– Create one shared entry for each application

– Create one shared entry for Fixed Users (system or application)

– Create one shared entry for Floating Users (system or application)


Install Order

› Install on the initial server– AR System, Approval Server, Assignment Engine, Email Engine– Mid-Tier Server (same or separate physical server)– Flashboards, CMDB– Applications

› For subsequent AR System Servers– All install paths must be consistent on each server in the group– Install AR System Server

• Choose the same database when installing the servers in a server group• Select the ‘Share’ database option

– License the servers appropriately– Install CMDB

• Specify ‘Share Server Group’ – only executables and config entries are created

– Install Flashboards• Forms are only installed on the first install/primary server by default

– Applications• Copy the application\bin sub-directories from the initial server• Configure ar.cfg/.conf and armonitor.cfg/.conf

› Install subsequent Mid-Tier Servers


Configure Each Server

› Server group name (shared)

– Name shared by all servers in the group (BMCRemedy)

– Must match name of group applied with licenses

› Check Interval

– How often other servers check if it is online

› Full Text Search indexing

– Put index directory on a shared file system and point all

servers there.

› Email Engine

– Port needs to match on all servers

› BMC Remedy Flashboards

– Port needs to match on all servers

› BMC Service Level Management

– Import the public key of the Data Collection Point security certificate from the initial server

› BMC Remedy Distributed Server Option (DSO)

– Specify any server in the group as the source

– Must specify explicitly a server for distributed returns, deletes and updates

Don’t configure

through the

load balancer

Don’t configure

through the

load balancer


Configure Each Mid-Tier Server

› Host file on each web server

<Load Balancer Virtual Server Name> <Actual Server IP>

BMCRemedy 11.40.35.51


Configure Each Server AR.cfg

› Server Name (unique)

– by default, the fully qualified name of the server

• bmc.mycompany.com

– You can specify a name in the ar.cfg file

• Server-Connect-Name : bmc1

– Used by the other servers to talk to the server

› Copy from ar.cfg on initial server to all others

– Plugin <full path>omfobjiefilapi.dll"

– Plugin <full path>arfslasetup.dll"

– Plugin <full path>arfcbdata.dll"

– Plugin-Path <concatenation of paths>

– Alternate-Approval-Reg: T


Configure Each Server ARMonitor.cfg

› Copy from armonitor.cfg on initial server to all others

– <full path>/brsvcbrie.exe" -d <AR System path> -m

– <full path>/arrecond.exe" -i <AR System path> -m

– <full path>/arcmdbd.exe" -i <AR System path> -m


Configure – Operation Ranking

The number of times the server does not respond

before the next server for the operation takes over

Servers are used based on ranking, lowest to highest.

Null excludes the server from the operation

Create 1 entry for each server/operation combination.

Business Rule Engine, Approval Server and CMDB must

share ranking with Administration. They have to run on

the same server.


Managing : Log - arsrvgrp

› Tracks

– starting/stopping operations

– evaluation of other servers

– recache requests

– Event time


Managing : Server Status Signaling

You can review the status of your server group from the

servgrp_board table in your database.


Configuring Knowledge Management

Load

Balancer

AR System

Servers Application

ServerHummingbird

Search Server

Clients

KM

Servers

Data

ServerKnowledge

Solutions

Top server is primary for Administration and Escalation

operations. Only one at a time can do this.

The other two are for back up of Admin operations and

primary for User interaction.


Configuring Knowledge Management Servers

› You must copy and edit files on all servers

› Details can be found in the White Paper –

Configuring BMC Remedy Knowledge Management in a

Load Balanced Environment

load balancing and high availability for bmc remedy action request system

Documents

web servers

ar system

ar system

ar system

ar server

web servers

load balancer

session affinity