configuring a tibco enterprise message service storage · 2019-10-11 · 4.2.3 nfs mount via fstab...
TRANSCRIPT
![Page 1: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/1.jpg)
TIBCO Software Inc.
Global Headquarters
3307 Hillview Avenue
Palo Alto, CA 94304
Tel: +1 650-846-1000
Toll Free: 1 800-420-8450
Fax: +1 650-846-1005
www.tibco.com
TIBCO fuels digital business by enabling better decisions and faster, smarter actions through the TIBCO Connected Intelligence Cloud. From APIs and systems to devices and people, we interconnect everything, capture data in real time wherever it is, and augment the intelligence of your business through analytical insights. Thousands of customers around the globe rely on us to build compelling experiences, energize operations, and propel innovation. Learn how TIBCO makes digital smarter at www.tibco.com.
Configuring a TIBCO Enterprise Message ServiceTM Fault Tolerant Environment on AWS with Zadara Storage This document provides the steps for configuring and testing TIBCO Enterprise Message Service Fault Tolerance with a Linux operating environment on AWS utilizing Zadara Storage
Version 1.0 October 2019 Initial Document
![Page 2: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/2.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 2
Copyright Notice COPYRIGHT© 2019 TIBCO Software Inc. All rights reserved.
Trademarks TIBCO, the TIBCO logo, and TIBCO Enterprise Message Service are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries. All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only.
Content Warranty The information in this document is subject to change without notice. THIS DOCUMENT IS PROVIDED "AS IS" AND TIBCO MAKES NO WARRANTY, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING BUT NOT LIMITED TO ALL WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. TIBCO Software Inc. shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance or use of this material.
For more information, please contact:
TIBCO Software Inc. 3303 Hillview Avenue Palo Alto, CA 94304 USA
![Page 3: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/3.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 3
Table of Contents
Configuring a TIBCO Enterprise Message ServiceTM Fault Tolerant Environment on AWS with Zadara Storage .......................................................................................................................................... 1
1 Overview ................................................................................................................................ 5 1.1 Zadara Virtual Private Storage Array Support .................................................................................. 5 1.2 Linux Kernel Support ...................................................................................................................... 5 1.3 EMS Version Support ...................................................................................................................... 6 1.4 Prerequisites .................................................................................................................................. 6
2 EC2 Instance Creation in AWS................................................................................................. 7
3 Zadara Virtual Private Storage Array Configuration ............................................................... 10
4 Configuring the AWS EC2 Instances ...................................................................................... 14 4.1 Software updates ......................................................................................................................... 14 4.2 Configuring NFS ............................................................................................................................ 14
4.2.1 IPsec installation and Configuration .......................................................................................... 14 4.2.2 Create the Zadara NFS Mount Configuration ............................................................................ 15 4.2.3 NFS Mount via Fstab ................................................................................................................. 16 4.2.4 Linux Kernel Changes ................................................................................................................ 16
5 Enterprise Message Service Installation and Configuration ................................................... 18 5.1 EMS Installation ............................................................................................................................ 18 5.2 EMS Configuration ........................................................................................................................ 18
5.2.1 Stores.conf ............................................................................................................................... 18 5.2.2 Factories.conf ........................................................................................................................... 19 5.2.3 Tibemsd.conf ............................................................................................................................ 19 5.2.4 Starting the EMS Instances ....................................................................................................... 21
6 Testing EMS Fault Tolerance on AWS with Zadara Storage .................................................... 22 6.1 EMS Client Application Setup ........................................................................................................ 22 6.2 Performing the EMS Fault Tolerant Test Cases .............................................................................. 23
6.2.1 EMS Process Failure Test .......................................................................................................... 23 6.2.2 Network Failure Test ................................................................................................................ 26 6.2.3 System Failure Test ................................................................................................................... 30
![Page 4: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/4.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 4
Table of Figures FIGURE 1 - CREATE AWS EC2 INSTANCE .............................................................................................................................. 7 FIGURE 2 - CONFIGURE EC2 INSTANCE DETAILS ..................................................................................................................... 8 FIGURE 3 - SECURITY GROUP INPUTS ................................................................................................................................... 8 FIGURE 4 - NEW EC2 INSTANCES ........................................................................................................................................ 9 FIGURE 5 - VPSA GUI ................................................................................................................................................... 10 FIGURE 6 - CONFIGURING A SERVER IN THE VPSA ................................................................................................................. 11 FIGURE 7 - CREATE SHARED VOLUME EXAMPLE .................................................................................................................... 12 FIGURE 8 – EXAMPLE OF A CREATED VOLUME ...................................................................................................................... 13 FIGURE 9 - ATTACHED SERVERS TO THE VPSA...................................................................................................................... 13 FIGURE 10 - MNT-NFS.MOUNT SERVICE EXAMPLE.................................................................................................................. 16 FIGURE 11 - FSTAB EXAMPLE ........................................................................................................................................... 16 FIGURE 12 - SYSCTL CHANGE ........................................................................................................................................... 17 FIGURE 13 - STORES.CONF EXAMPLE ................................................................................................................................. 19 FIGURE 14 - FACTORIES CONFIGURATION EXAMPLE ............................................................................................................... 19 FIGURE 15 - STARTING AN EMS INSTANCE EXAMPLE ............................................................................................................. 21 FIGURE 16 - CREATE THE SYNC QUEUE ............................................................................................................................... 23 FIGURE 17 - RUNNING THE TIBJMSMSGPRODUCERPERF APPLICATION ....................................................................................... 24 FIGURE 18 - THE STANDBY EMS INSTANCE BECOMING ACTIVE ON SERVER2 ................................................................................ 25 FIGURE 19 - PURGE THE SYNC QUEUE FOR TIBEMSADMIN ....................................................................................................... 26 FIGURE 20 - DROP_NFS.SH SCRIPT .................................................................................................................................... 27 FIGURE 21 - RUNNING DROP_NFS.SH SCRIPT ....................................................................................................................... 28 FIGURE 22 - DISK WRITE ERROR ON SERVER1 ...................................................................................................................... 28 FIGURE 23 - SERVER2 BECOMING ACTIVE AFTER NETWORK FAILURE .......................................................................................... 29 FIGURE 24 - AWS COMPUTE ENGINE/EC2 INSTANCES PAGE................................................................................................... 30 FIGURE 25 – STANDBY EMS INSTANCE RECOVERING FROM SYSTEM FAILURE OF PRIMARY ............................................................... 31
![Page 5: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/5.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 5
1 Overview
The purpose of the document is to provide a guide to install, configure, and run TIBCO Enterprise Message ServiceTM (EMS) in a fault-tolerant configuration on Amazon Web Service (AWS) utilizing a Zadara Storage System solution for shared storage. In addition, the document will provide the steps and expected results for testing EMS F/T on AWS with Zadara.
The document will outline:
• Setting up Linux EC2 instance on AWS • General outline for setting up the Zadara VPSA for the shared file system • Setting up the NFS4 mount on the Linux AWS EC2 instance • Installing and configuring EMS for F/T • Tuning EMS for the AWS environment • Running tests for:
o EMS process failure o Network failure between the AWS EC2 instance running EMS and the shared
storage o Accidental reboot of the AWS EC2 instance from the AWS Dashboard
1.1 Zadara Virtual Private Storage Array Support
Currently, Amazon Web Services provides EFS as a shared storage platform capable of supporting the shared storage requirements required for TIBCO Enterprise Message Service. However, there are alternatives. Zadara provides an Enterprise Storage as a Service Solution (STaaS). Zadara provides a fully managed on premise or in the cloud storage solution that meets the shared storage requirements for EMS when utilizing NFSv4. See https://www.zadara.com and http://guides.zadarastorage.com/vpsa-guide/1711/ for details on accessing a Zadara Virtual Private Storage Array (VPSA). Note: This document does not provide guidance on creating or accessing the VPSA, but will provide guidance on configuring a NAS/NFSv4 for EMS.
1.2 Linux Kernel Support
This document covers the installation and configuration of EMS on Amazon’s Linux2 kernel, which is based on Linux kernel 4.14.143-118. All Linux distributions that are materially equivalent to the above listed Linux version are supported. See https://docs.tibco.com/pub/ems/8.5.0/TIB_ems_8.5.0_readme.txt?id=1 for details.
![Page 6: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/6.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 6
1.3 EMS Version Support
EMS version 8.5 or later should be used for EMS server installations running in a AWS environment. Any available EMS hot fixes should also be applied. The Enterprise Edition or the Community Edition of EMS can be used. Download the EE from edelivery.tibco.com , and the CE at https://www.tibco.com/products/tibco-enterprise-message-service.
1.4 Prerequisites
The following prerequisites must be meet before configuring EMS on AWS:
• An AWS Account is required. An AWS Account can be created/accessed at https://console.aws.amazon.com.
• The TIBCO Enterprise Message Service Software version 8.5.0 has been downloaded and is available to be uploaded to the AWS.
• The Zadara VPSA onboarding has been completed, and the VPSA is available for configuration. See section 1.1 for details.
• The reader of this document is familiar with the following concepts: o The use of AWS Console for configuring EC2 instances o TIBCO EMS installation and configuration o Linux configuration o NFSv4 o Zadara VPSA o Configuration of IPsec for encryption
![Page 7: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/7.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 7
2 EC2 Instance Creation in AWS
Three AWS Linux EC2 instances are required. The following steps will outline creating the EC2 instances in the AWS console. Amazon Linux2 was used for the EC2 instances. Other operating systems or versions can be used, but are not covered in this document. For the following examples, the AWS region US-EAST is used. Note: TIBCO Enterprise Message Service 8.5 is currently not supported on CentOS/RHEL 8 distributions.
To create a new AWS EC2 Linux instance, use the following:
• In the AWS console, select Services, EC2, and then Launch Instance to create a new Amazon EC2 instance. Three EC2 instances are needed: two for EMS F/T and one for the client application.
Figure 1 - Create AWS EC2 Instance
• Select Amazon Linux 2 AMI (HVM), SSD Volume Type - ami-0b69ea66ff7391e80 (64-bit x86) for the AMI
• Select the appropriate size instance (vCPUs and Memory) for the environment • Click on Next: Configure Instance Details • Leave the number instances to 1. Use the default VPC, and defaults for the other settings,
except for the subnet. Select a subnet in the AWS Region Zone where the EC2 instance will be created. The EC2 instances used for the EMS Servers should be in separate zones.
![Page 8: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/8.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 8
Figure 2 - Configure EC2 Instance Details
• Click Next: Storage • Configure the Volume Type and the Size in GB. These will depend on what else is be
installed other than EMS. If just EMS, 20GB is sufficient. • Click on Next: Add Tags, and then Next: Configure Security Group • Either use a new security group, or an existing security group. Ensure the security group
used has rules to support SSH, TCP, NFS, and ICMP-IPv4 at the minimum. Must consider Source, Port Range, Protocol, and Type.
Figure 3 - Security Group Inputs
• Click on Review and Launch • Repeat the above steps for the two other EC2 instances
![Page 9: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/9.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 9
Figure 4 - New EC2 Instances
• Once the EC2 Linux instances have started, Putty (on Windows) or ssh on Mac/Linux/UNIX can be used to access the EC2 instances using the public IP address created by AWS.
• Click on the EC2 instance, and then on the Connect button. The Connect to Your Instance instructions will be displayed. Follow the instructions to connect to the EC2 instance.
• Once completed, note the Name, Internal, and External IP Address of the new compute engines.
![Page 10: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/10.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 10
3 Zadara Virtual Private Storage Array Configuration
The Zadara Virtual Private Storage Array (VPSA) must be configured before completing the setup of the AWS EC2 instances for EMS. See section 1.1 on setting up access to the Zadara Virtual Private Storage Array (VPSA). Zadara will provide the IP Address of the VPSA, as part of the onboarding procedure. This will also be the IP address used on the AWS EC2 instances for the NFS mount. Once there is access to the Zadara VPSA, follow this section to create the basic configuration for the Zadara VPSA with NFSv4 for use as the shared storage for EMS.
• Log into the VPSA using the username and password configured during the initial VPSA setup with Zadara. In the following examples, EMSAWS is the name of the VPSA. Click on Drives under resources, and verify there is at least one drive to configure.
Figure 5 - VPSA GUI
• Next, click on Servers under Resources. Click on Add and then Manual. This will need to be done for the two AWS EC2s used as the EMS servers in AWS. Provide:
o The name of the AWS EC2 instance used for EMS server o The OS type (Linux) o The internal IP Address of the AWS EC2 instance o Click on Enable IPsec if end to end encryption is required (recommended). o Click on Create
• The newly created server should look similar to the following example.
![Page 11: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/11.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 11
Figure 6 - Configuring a Server in the VPSA
• Create the volume in the VPSA for the NFSv4 mount • Click on Volumes under Resources, and then on Create, and then click on Create NAS
Share. Provide the: o The name for the new volume o The capacity for the volume (in GiB). Must be below the capacity of the drive o Export name for the NFS server. Note the export name, a this will also be used as
part of the NFS mount on the AWS EC2 instances. o Click on encrypted, if data at rest encryption is required. If IPsec was enabled
previously, it recommended to enable encryption at rest also o Ensure 512 KiB read ahead is selected o The rest of the values can be at their defaults if desired o Click on submit
![Page 12: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/12.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 12
Figure 7 - Create Shared Volume Example
• Once the new volume is created, click on the details for the newly created volume • Ensure that the all settings defined previously have been enabled
![Page 13: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/13.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 13
Figure 8 – Example of a created Volume
• Click on Servers from the top section of the VPSA GUI, while still on volumes • Next, click on Attach to Server(s), and click on the two servers previously created, and click
submit. This will allow the two AWS EC2 instances to be attached to the VPSA via the NFS mount.
• Click on Servers in the details for the volume. The two EC2 instances should be listed.
Figure 9 - Attached Servers to the VPSA
• The VPSA is now ready for use as the shared storage device.
![Page 14: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/14.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 14
4 Configuring the AWS EC2 Instances
The individual AWS EC2 instances can now be configured. The following sections will outline additional software and changes needed for EMS installation and configuration.
4.1 Software updates
Additional software is required to install and configure EMS and NFSv4 on the EC2 instances created on AWS. Not all software needs to be installed all machines, but it simplifies the installation.
• Log into each of the newly created AWS EC2 instances. Follow the AWS documentation for using ssh to connect to each EC2.
• There are several packages to install, but all can be installed at once. Use o sudo yum install -y nfs-utils java-devel unzip to install all packages o sudo yum update -y to update all packages
• Repeat on the other EC2s
4.2 Configuring NFS
On the two AWS EC2 instances which will be used as the EMS servers, NFSv4 must be configured and the shared storage mounted. If IPsec is being used, it must be installed and configured first. If IPsec is not being used, sections 4.2.1 and 4.2.2 can be skipped.
4.2.1 IPsec installation and Configuration IPsec will provide “end to end” encryption of all EMS messages between the EMS server and the Zadara VPSA. Follow this section to install and configure IPsec.
• An additional software package must be downloaded and installed. Openswan is utilized to connect to Zadara VPSAs over IPsec. Use sudo yum install –y openswan to install openswan
• Add the following to /etc/ipsec.conf:
conn zadara-VPSA-NFS type=transport auto=start authby=secret ike=aes128-sha1-modp1024 phase2=esp phase2alg=aes128-sha1 ikev2=never dpddelay=5 dpdtimeout=5 dpdaction=restart pfs=no
![Page 15: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/15.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 15
salifetime=1h rekey=yes rekeymargin=9m keyingtries=%forever # Always try to connect to VPSA ikelifetime=3h left=<Internal IP Address of the EC2 instance> leftprotoport=tcp right=<IP Address of the VPSA> rightprotoport=tcp/2049
• The file /etc/ipsec.secrets needs to contain the shared secret. This is the IPSEC key from the VPSA:
o From the Main Dashboard panel of the VPSA on the left hand side, click on System, then Settings.
o In the main screen, click on Security. The IPSEC key is at the bottom of the screen. This is the PSK.
o Edit /etc/ipsec.sec, and modify as shown below:
################### <IP Address of the EC2 instance> <IP Address of the VPSA>: PSK "<PSK>" ###################
For example: 10.150.0.3 172.24.140.200: PSK "D12340348GF34E26A481D74A1F456789"
• Start IPsec: sudo systemctl restart ipsec • To start after reboots: sudo systemctl enable ipsec • To verify IPsec, use: sudo ipsec whack –status - The output should contain information
that the connection to the Zadara VPSA has been established. Note: If IPsec is being used, Section 4.2.1 must be completed on both AWS instances where the EMS servers will run before continuing! See https://support.zadarastorage.com/hc/en-us/articles/213024926-How-to-enable-IPsec-on-Linux-instances for additional information.
4.2.2 Create the Zadara NFS Mount Configuration By default, with Linux systemd, the IPsec service may stop before the Zadara NFS storage can be completely dismounted. This will cause time-outs for several minutes while trying to stop the EC2 instance. To ensure the NFS dismount occurs before IPsec is stopped on a more consistent basis, an NFS mount service must be created.
Note: If IPsec is not being used, see section 4.2.3 for mounting the Zadara Storage via fstab.
• Create /usr/lib/systemd/system/mnt-nfs.mount. Note: The following example shows 172.32.160.200:/export/emsaws as the NFS Server, and /mnt/nfs as the mount point. Change accordingly. If /mnt/nfs is not the mount point, then the name of the mount service must also be changed. For example, if the mount point
![Page 16: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/16.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 16
is /opt/aws, then the name of the mount file would be opt-aws.mount. DO NOT modify the options shown below!
[Unit] Description=Zadara NFS /mnt/nfs
Requires=ipsec.service After=ipsec.service
[Mount] What=172.32.160.200:/export/emsaws/ Where=/mnt/nfs
Type=nfs Options=nfsvers=4.0,soft,retrans=2,actimeo=1,timeo=150,_netdev
[Install]
WantedBy=multi-user.target
Figure 10 - mnt-nfs.mount service example
• Enable the new mount service to run on startup: sudo systemctl enable mnt-nfs.mount • Create a new mount point for the storage: sudo mkdir /mnt/nfs • Mount the shared storage using sudo systemctl start mnt-nfs.mount
4.2.3 NFS Mount via Fstab If IPsec is not being used, creating a mount service is not required, and the Zadara Storage can be mounted via /etc/fstab. The following example is an example of the Zadara Storage being installed via fstab. As with the mount service, DO NOT modify the options!
172.32.160.200:/export/emsaws/ /mnt/nfs nfs nfsvers=4.0,soft,retrans=2,actimeo=1,timeo=150,_netdev
Figure 11 - Fstab example
• Create a new mount point for the storage: sudo mkdir /mnt/nfs • Mount the shared storage using sudo mount /mnt/nfs
4.2.4 Linux Kernel Changes The Linux kernel by default can keep the tcp_keepalives for up to twenty minutes. This can have a delayed affect on EMS fail-over. To shorten this time, the Linux kernel property tcp_retries2 can be modified. To modify this property, do the following:
• Edit /etc/sysctl.conf, and add the following value: net.ipv4.tcp_retries2 = 4 • Following is an example of /etc/sysctl.conf with the change:
sysctl settings are defined through files in # /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/. # # Vendors settings live in /usr/lib/sysctl.d/.
![Page 17: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/17.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 17
# To override a whole file, create a new file with the same in # /etc/sysctl.d/ and put new settings there. To override # only specific settings, add a file with a lexically later # name in /etc/sysctl.d/ and put new settings there. # # For more information, see sysctl.conf(5) and sysctl.d(5). # Update to shorten tcp keep alives net.ipv4.tcp_retries2 = 4
Figure 12 - Sysctl change
• Reboot the AWS EC2 instance. After the reboot, check: o The shared file system has mounted o That tcp_retries2 =4. Use cat /proc/sys/net/ipv4/tcp_retries2 to verify
Note: Resolve any issues before continuing!
![Page 18: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/18.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 18
5 Enterprise Message Service Installation and Configuration
This section will outline the installation and configuration of EMS on the AWS EC2 instances.
5.1 EMS Installation
EMS 8.5 was used for all testing, and is recommended. Either the Enterprise or Community edition of EMS may be used. Install EMS on all of the of the AWS EC2 instances. Nothing specific or custom is required to the base configuration of EMS. Follow the EMS 8.5 Installation guide for installing EMS.
Once EMS is installed, use the following to configure EMS on all three EC2s.
• EMS 8.5 may have been installed as the root user. If EMS was installed as the root user, change the ownership to the username created by AWS, and used to login into the AWS EC2 instances.
• Use sudo chown –R <username>:<username> /opt/tibco to change the ownership.
On the EMS server EC2s:
• Create /opt/tibco/ems/8.5/bin/logs directory
On one of the EMS server EC2s:
• Create the configuration directory on the shared storage:
mkdir –parents /mnt/nfs/tibco/cfgmgmt/ems/data/datastore
• Copy /opt/tibco/ems/8.5/samples/config/*.conf /mnt/nfs/tibco/cfgmgmt/ems/data/
5.2 EMS Configuration
There are specific configuration changes which must be made to provide better write performance and reliability of EMS F/T. This section will discuss these changes. See the EMS User Guide for additional information on setting or the use of, any properties discussed. Note: EMS 8.5 no longer pre-configures the main tibemsd.conf configuration file as part of the installation. All configuration modifications discussed must be completed. Copy tibemsd.conf to tibemsd1.conf and tibemsd2.conf.
5.2.1 Stores.conf In stores.conf, modify/add the following:
The file_minimum=xxGB can be added to each data store. By adding this property, EMS will pre-allocate the space on the shared storage the data store. Note: This is an optional setting, and is not required, but will enhance the message write throughput on disk. The minimum should be at least 1GB. Expect the initial startup of EMS to take longer as it creates and allocates the space for the store file.
![Page 19: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/19.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 19
The file_crc=true should be added, if not already present. This enables EMS to check for data integrity of the data store. This is the default, but ensures it is set.
The following is an example of stores.conf with the changes.
[$sys.failsafe] type=file file=sync-msgs.db mode=sync file_minimum=2GB file_crc=true [sync2] type=file file=sync2-msgs.db mode=sync file_crc=true
Figure 13 - Stores.conf example
5.2.2 Factories.conf The EMS client reconnect properties must be set to enable the EMS client to reconnect to the EMS server in the event of an EMS server failure in an F/T configuration. The reconnect properties can be defined in a number of ways, including in the Java/C code, TIBCO application’s configuration file, and/or through the connection factory when they are used. The default values may be too low in AWS to reliably allow the EMS client to reconnect to the EMS server after a fail-over, especially with network or system failure. It is recommended to set the reconnect_attempt_count to 100, and the reconnect_attempt_delay to 5000. With these values, the EMS client will attempt to reconnect 100 times, every 5 seconds.
The following example shows the values for the FTConnectionFactory in factories.conf. Note: In the following example for the url, <server1> is 10.150.0.7 and the <port1> is 7224, and <server2> is 10.150.0.3 and the <port2> is 7224. Substitute with the appropriate values for the environment.
[FTConnectionFactory] type = generic url = tcp://10.150.0.5:7224,tcp://10.150.0.3:7224 reconnect_attempt_count=100 reconnect_attempt_delay=5000
Figure 14 - Factories Configuration Example
5.2.3 Tibemsd.conf The tibemsd.conf for both EMS Servers needs to be updated for multiple properties.
These include:
• Location of all configuration files – The location must be on the shared storage device.
######################################################################## # Configuration files. ########################################################################
![Page 20: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/20.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 20
users = "/mnt/nfs/tibco/cfgmgmt/ems/data/users.conf" groups = "/mnt/nfs/tibco/cfgmgmt/ems/data/groups.conf" topics = "/mnt/nfs/tibco/cfgmgmt/ems/data/topics.conf" queues = "/mnt/nfs/tibco/cfgmgmt/ems/data/queues.conf" acl_list = "/mnt/nfs/tibco/cfgmgmt/ems/data/acl.conf" factories = "/mnt/nfs/tibco/cfgmgmt/ems/data/factories.conf" routes = "/mnt/nfs/tibco/cfgmgmt/ems/data/routes.conf" bridges = "/mnt/nfs/tibco/cfgmgmt/ems/data/bridges.conf" transports = "/mnt/nfs/tibco/cfgmgmt/ems/data/transports.conf" tibrvcm = "/mnt/nfs/tibco/cfgmgmt/ems/data/tibrvcm.conf" durables = "/mnt/nfs/tibco/cfgmgmt/ems/data/durables.conf" channels = "/mnt/nfs/tibco/cfgmgmt/ems/data/channels.conf" stores = "/mnt/nfs/tibco/cfgmgmt/ems/data/stores.conf" ######################################################################## # Persistent Storage.
# # store: directory to store persistent messages. ######################################################################## store = "/mnt/nfs/tibco/cfgmgmt/ems/data/datastore"
• Log File location – The location must be on the local disk if the AWS EC2 instance. The following example locates in the /opt/tibco/ems/8.5/bin/logs directory.
logfile = "/opt/tibco/ems/8.5/bin/logs/tibemsd1.log"
Server and Client Heartbeat and timeout values – These properties determine how long the client/server listen for the heartbeat from the the client/server, before disconnecting. The values shown below have been tested, and work well on AWS with the Zadara NFS storage. The connection_timeout can be increased if necessary, but the heartbeat must remain at 5.
server_heartbeat_client = 5 server_timeout_client_connection = 30 client_heartbeat_server = 5 client_timeout_server_connection = 30
Enabling exiting disk error property – New property since EMS version 8.4. This property defines to EMS to exit when there is a disk error reading/writing to a device. It must be enabled. This property will help prevent “Dual Active Server” conditions, sometimes seen in networked storage devices.
always_exit_on_disk_error = enable
FT properties – Normal properties for the defining the peer EMS server instance, heartbeat between instances, and etc. must be set. Ensure the ft_reconnect_timeout is set to the value shown.
ft_reconnect_timeout = 120 Increase the Maximum Message Memory to 1024 or greater, depending on how much RAM is allocated to the AWS EC2 instance.
![Page 21: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/21.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 21
max_msg_memory = 1024
Define a value for destination_backlog_swapout. This will help limit excessive reads to the shared disk. A minimum of 10000 is recommended. If the queues, will persistent a larger number of messages, increase the size.
destination_backlog_swapout = 10000
5.2.4 Starting the EMS Instances Once the configuration files are updated, EMS can be started. The –forceStart parameter is optional. Start both instances, taking note of which instance is the active EMS instance. Note: It will take ~ 5 minutes for the first EMS instance to become active due to the creation of the data stores on the shared storage. Note: Leave the both windows to the EMS server instances open. This will be needed for the testing.
Figure 15 - Starting an EMS Instance Example
![Page 22: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/22.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 22
6 Testing EMS Fault Tolerance on AWS with Zadara Storage
Once EMS has been started on both of the AWS EC2 instances, the failover testing can be performed. This section will outline several test cases, including EMS Server process failure, machine failure, and network failure. Tests are performed using queues with persistence set. This guarantees that the shared file system will be accessed during the tests.
6.1 EMS Client Application Setup
The third AWS EC2 instance is used to run the test application. EMS is shipped with sample Java applications which can be used for the testing. The tibjmsMsgProducerPerf utility should be used for the testing. All sample Java applications are located in /opt/tibco/ems/8.5/samples/java. Use the following to setup the environment:
• Ensure the version Java 1.8 development environment is installed. • Install EMS version 8.5 on the third AWS EC2 instance following the EMS installation
procedures. • After the installation of EMS is completed:
o cd to $TIBCO_HOME/ems/8.5/samples/java o . ./setup.sh o javac *.java – This should compile all Java apps in the directory
• Ensure that at least one of the EMS server instances is running (both should be running) • Use the TIBCO EMS Administration Tool to create the EMS Queue sync utilizing the
$sys.failsafe data store. This is required for testing with a synchronous data store: o cd to $TIBCO_HOME/ems/8.5/bin o ./tibemsadmin tcp://<server>:port – of the active EMS instance
Note: In the following examples, <server> is the IP address is 10.150.0.5 and the <port> is 7224. Substitute with the appropriate values for the environment.
![Page 23: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/23.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 23
Figure 16 - Create the Sync Queue
6.2 Performing the EMS Fault Tolerant Test Cases
Three different tests should be performed: 1. EMS Process failure – Active EMS instance is stopped 2. Network failure – Network failure between the Active EMS Server machine, and the
Zadara VPSA 3. System failure – Accidental restart of the AWS EC2 running the Active EMS server
instance
This section will outline how to run these three tests, and what the expected results should be. Note: All test cases must be run from the third virtual machine where the Java sample app was compiled.
6.2.1 EMS Process Failure Test This test verifies that an EMS client continues to function correctly, with no message loss during an EMS server process failover. Two EMS server instances will be running in a F/T configuration, while messages are being sent. The active EMS instance will be stopped, the stand-by EMS instance should take over, and continue processing messages until the EMS Java completes publishing messages. Note: In the following examples, <server1> the IP address is 10.150.0.5 and the <port1> is 7224, and <server2> the IP address is 10.150.0.3 and the <port2> is 7224. Substitute with the appropriate values for the environment.
![Page 24: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/24.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 24
6.2.1.1 Running the Process Failure Test Three ssh terminal sessions are needed for this test; one for server1, one for server2, and one for the client Start EMS on server1 and server2 in the foreground. EMS on server1 should be the active EMS instance. From the client, start the Java application
o cd to $TIBCO_HOME/ems/8.5/samples/java o . ./setup.sh o java tibjmsMsgProducerPerf –server tcp://10.150.0.5:7224, tcp://10.150.0.3:7224 –
factory FTConnectionFactory –delivery PERSISTENT –connections 10 –threads 8 –count 20000 –size 1024 –queue sync
Figure 17 - Running the tibjmsMsgProducerPerf Application
Immediately kill/stop the EMS instance on server1, with cntrl-C The standby EMS instance on server2 will become active, and recover all messages. It should be possible to stop and start the EMS instances a few times while the Java test application is running. The number of recovered messages will increase.
![Page 25: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/25.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 25
Figure 18 - The Standby EMS Instance becoming active on Server2
After the Java application completes, run tibemsadmin tcp://10.150.0.3:7224 (or from server1, if it is active), to verify that there are 20000 messages in the sync queue. Restart the EMS instance on server1, and stop the EMS instance on server2. EMS on server1 should become active, and recover all 20K messages with no errors. Use tibemsadmin and purge the sync queue in preparation for the next test.
![Page 26: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/26.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 26
Figure 19 - Purge the Sync queue for tibemsadmin
• Stop and restart EMS on server1 and server2 in the foreground. EMS on server1 should be the active EMS instance.
6.2.1.2 Expected Results The Java test application should complete, with a slight pause during failover but should resume sending messages once the failover is complete. No messages should be lost. There should not be more than 20K messages, and there should never be less than 20K. Depending on the number of messages that must be recovered, the fail-over should be very short, possibly less than 1 second. Complete fail-over from one server to the other with the 20K messages should be under 10 seconds.
6.2.2 Network Failure Test This test verifies that an EMS client continues to function correctly, with no message loss during a network failure between the active EMS server instance, and the Zadara VPSA. Two EMS server instances will be running in a F/T configuration, while messages are being sent. The TCP/NFS port will be blocked between the active EMS instance and the Zadara VPSA via iptables. The active EMS instance should get a write error, and exit, allowing the stand-by EMS instance to gain the locks on the EMS data stores, and take over. The EMS Java application should continue processing messages until it completes. Note: In the following examples, <server1> IP address is 10.150.0.5 and the <port1> is 7224, and <server2> IP address 10.150.0.3 and the <port2> is 7224. Substitute with the appropriate values for the environment.
![Page 27: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/27.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 27
6.2.2.1 Running the Network Failure Test Four ssh terminal sessions are needed for this test; two for server1, one for server2, and one for the client A script will be needed to block the NFS port (2049) on server1 while the Java app is publishing messages. The following figure shows the drop_nfs.sh script. Cut and past the following to create the script. The script must be created in the second ssh terminal window on server1.
# # Script to get the current iptables definitions, drop NFS port, then restore the original table defi #nitions. # echo " Saving existing IP table definitions" echo "" sudo iptables-save >iptables_save # # Drop the NFS port 2049 # date echo " Dropping NFS 2049 port" echo "" sudo iptables -A INPUT -p TCP --dport 2049 -j DROP sudo iptables -A OUTPUT -p TCP --dport 2049 -j DROP # # Sleep for 3 minutes # echo " Sleeping for 3 minutes..." echo "" sleep 3m # # Restore original IP table definitions" # echo " Restoring original IP table definitions" echo "" sudo iptables -F sudo iptables-restore <iptables_save # date echo "Done."
Figure 20 - drop_nfs.sh script
From the client, start the Java application o cd to $TIBCO_HOME/ems/8.5/samples/java o . ./setup.sh o java tibjmsMsgProducerPerf –server tcp://10.150.0.5:7224, tcp://10.150.0.3:7224 –
factory FTConnectionFactory –delivery PERSISTENT –connections 10 –threads 8 –count 20000 –size 1024 –queue sync
From the second ssh terminal window on server1, run sudo drop_nfs.sh
![Page 28: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/28.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 28
Figure 21 - Running drop_nfs.sh script
• The active EMS instance on server1 should terminate with a disk write error:
Figure 22 - Disk Write Error on Server1
The standby EMS instance on server2 should determine EMS on server1 is no longer producing a heartbeat, will attempt to become active. This should only take a few seconds for server2 to become active, depending on the amount of data to be recovered. There may be other warnings, depending on how long it takes for server2 to obtain the locks.
![Page 29: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/29.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 29
Figure 23 - Server2 becoming Active after network failure
After the Java application completes, run tibemsadmin tcp://10.150.0.3:7224 (or on server1, if it is active), to verify that there is a minimum of 20000 messages in the sync queue. Restart the EMS instance on server1, and stop the EMS instance on server2. EMS on server1 should become active, and recover all 20K messages with no errors. While still in tibemsadmin, purge the sync queue in preparation for the next test. Stop and restart EMS on server1 and server2 in the foreground. EMS on server1 should be the active EMS instance.
6.2.2.2 Expected Results The Java test application should complete, pausing during the failover, but should resume sending messages once the failover is complete. No messages should be lost. There can be more than 20K messages, depending on the number of connections/threads, but there should never be less than 20K. Depending on the number of messages that must be recovered, the fail-over should only take a few seconds. Note: It was observed that on occasion, the NFS mount on Zadara does not reconnect after the default iptables configuration was restored, and a reboot of the server1 EC2 instance was required. However, server2 still recovers all messages, and continues. No messages were lost. Note: In some circumstances, the following error may occur with the network failure, if a file_minimum for a data store has been set.
![Page 30: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/30.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 30
SEVERE ERROR: Error occurred while processing Message record id=xxxxxx (get record) - Unable to reconstruct message: 74 - IO failed
This can sometimes occur when there is a file_minimum set, and the file is only partially filled. This is not an error, and can be ignored. All messages will be recovered. If this is seen, restart the tibemsd server with the –forceStart option as shown in Figure 15.
6.2.3 System Failure Test This test verifies that an EMS client continues to function correctly, with no message loss during a system failure on the AWS EC2 instance running the active EMS server instance. This is not a normal occurrence. However, it is possible to accidentally restart the virtual machine from the AWS console. Two EMS server instances will be running in a F/T configuration, while messages are being sent. From the AWS console, the EC2 instance where the active EMS instance is running will be restarted. The stand-by EMS instance should be able to gain the locks on the EMS data stores, and take over. The EMS Java application should continue processing messages until it completes. Note: In the following examples, <server1> IP address is 10.150.0.5 and the <port1> is 7224, and <server2> IP address is 10.150.0.3 and the <port2> is 7224. Substitute with the appropriate values for the environment.
6.2.3.1 Running the System Failure Test Three ssh terminal sessions are needed for this test; one from each of the virtual machines. The AWS console must also be available, and be on the AWS/EC2 Instances dashboard page. Note: In the examples below, Test Instance is server1. Click on the instance, Actions
Figure 24 - AWS Compute Engine/EC2 Instances page
• From the client, start the Java application: o cd to $TIBCO_HOME/ems/8.5/samples/java o . ./setup.sh o java tibjmsMsgProducerPerf –server tcp://10.150.0.5:7224, tcp://10.150.0.3:7224 –
factory FTConnectionFactory –delivery PERSISTENT –connections 10 –threads 8 –count 20000 –size 1024 –queue sync
![Page 31: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/31.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 31
• From the AWS dashboard, click on the server1 instance, then click on the Actions tab, then on Instance Status, and finally on Reboot. Verify to reboot the server. This will reboot the server1 AWS EC2 instance.
• The ssh terminal window to the server1 AWS EC2 instance should immediately terminate, and the stand-by EMS instance should recover all messages, and become active within a few seconds, depending on the number of messages to be recovered. There may be other warnings, depending on how long it takes for server2 to obtain the locks.
Figure 25 – Standby EMS instance recovering from system failure of primary
• After the Java application completes, run tibemsadmin tcp://server2:7224 (or to the active EMS instance), to verify that there is a minimum of 20000 messages in the sync queue.
• Restart the EMS instance on the restarted AWS EC2 instance, and stop the currently active EMS instance on the second AWS EC2 instance. EMS should become active on the restarted instance, and recover all 20K messages with no errors.
• Use tibemsadmin to verify, then purge the sync queue. • Stop EMS on both AWS EC2 instances. • This concludes the tests, so all processes, terminal windows, and AWS EC2 instances can
be stopped.
![Page 32: Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system](https://reader034.vdocuments.mx/reader034/viewer/2022042518/5f8715772b7a81115a42415b/html5/thumbnails/32.jpg)
©2019 TIBCO Software Inc. All Rights Reserved. 32
6.2.3.2 Expected Results The Java test application should complete, pausing during failover, but should resume sending messages once the failover is complete. No messages should be lost. There can be more than 20K messages, depending on the number of connections/threads, but there should never be less than 20K messages. Depending on the number of messages that must be recovered, the fail-over should take less than a minute. There may also be additional warning regarding the clients trying to reconnect, depending on how many messages that must be recovered. Note: In some circumstances, the following error may occur with an abrupt system shutdown, if a file_minimum for a data store has been set. SEVERE ERROR: Error occurred while processing Message record id=xxxxxx (get record) - Unable to reconstruct message: 74 - IO failed
This can sometimes occur when there is a file_minimum set, and the file is only partially filled. This is not an error, and can be ignored. All messages will be recovered. If this is seen, restart the tibemsd server with the –forceStart option as shown in Figure 15.