— july 28, 2015 bmc performance manager portal webinar series – webinar 2 connect with bmc...

—July 28, 2015

BMC Performance Manager Portal Webinar Series – Webinar 2

Connect with BMC Performance Manager Portal: Best Practices

Did you attend our last Webinar?

• Thank you to all who attended our first BMC PM Portal Webinar covering new features in 2.11.00, our current release.

• Several users provided great feedback regarding the release and based on that feedback, we have already created several new Ideas which are enhancement requests for the Portal.

• We listened to your comments and concerns and this will help make the product stronger and more user friendly. We appreciate your feedback and we ask that you keep it coming!

• If you have any questions or feedback, please do not hesitate to contact BMC Customer Support or visit our BMC Community and submit an Idea for improvement.

• Let’s take a look at the types of Ideas submitted as a result of our last Webinar.

Copyright © BMC Software 2015

https://communities.bmc.com/welcome

Webinar Feedback is Welcome• BMC Performance Manager Portal - Generate System notification problems email and alert when the RSMs are not

collecting• Allow blackout filtering using element tags in BMC Portal • BMC Performance Manager Portal request to allow users to set end date and end time for blackouts • Request to add a "bulk" blackout option in BMC Performance Manager Portal• Health monitor tab showing total number of parameters and the aggregated ratio of their expected collection times

as compared to their actual collection times to show overall system health broken down by assigned RSM• BMC Performance Manager Portal should have application class blackouts • BMC Performance Manager Portal RSMs should detect when they are being overloaded • Add support for SSH Authentication using BMC Performance Manager Express for Databases • Request to review BMC Performance Manager Express for Log Management to make it more efficient

There are all great requests. We encourage you to access the BMC Community and vote on or submit the Ideas you would like to see in the Portal.

**Please note that these Ideas are not committed to a specific release version. ** We use your feedback to prioritize our release plans.


https://communities.bmc.com/welcome

BMC Performance Manager Portal 2.11.00 Best Practices


Why Best Practices?• Working with a variety of different customers over the years has given

our team of engineers the insight needed to tune the application for use in growing and large scale environments.

• Our Best Practices Guide has evolved using real world data. We have seen the lost data, the errors, the critical alerts, as well as other issues users can encounter when Best Practices are not followed. We want to spare you the same experience.

• BMC wants to help you proactively take care of the BMC Portal environment to ensure continued performance.


BMC Performance Manager Portal 2.11.00 Best Practices

• Our Best Practices Guide is divided into sections and each section should be reviewed and applied to your Portal setup.

• We offer recommendations for Portal administration, self-monitoring, health checks, and tuning. We will explain the various sections in the Guide to reinforce their importance in maintaining a healthy Portal.

• The BMC Performance Manager Portal Best Practices Guide is on our ftp site ftp.bmc.com/pub/BPMPortal/BMC_Performance_Manager_Portal_Best_Practices_Guide.docx

ftp://ftp.bmc.com/pub/BPMPortal/BMC_Performance_Manager_Portal_Best_Practices_Guide.docx



General Administration


Change default passwords (database, superadmin, user, portalmon)


For security purposes, BMC recommends that you change the default passwords after installation. This includes the password for the BMC Portal database.

Change the Portal’s database passwordIf you have not yet changed the default password (“pe”) for the Portal’s Database Login then do the following:

1. Stop the BMC Portal App Server

2. On the Portal Database Host login to the Portal database as the SYSDBA and in the SQL*Plus session change the password like this example:

sqlplus / as sysdba SQL> alter user <Database Login> identified by <New Password>; -- For example: SQL> alter user pe identified by peNewPass123;

3. Launch the BMC Performance Manager Portal Maintenance ToolOn Windows: %BMC_PORTAL_KIT_HOME%\appserver\BMCPortalMaintenanceTool.cmdOn Solaris: $BMC_PORTAL_KIT_HOME/appserver/BMCPortalMaintenanceTool.sh

Change the Portal’s database password


Once the database password has been changed by sysdba, open the BMC Portal Maintenance Tool and change the database credentials via the Maintenance Tool.

Change the Portal’s database password


Click “Next” until you confirm the only thing that has changed is the Database Password as shown above then click “Next” again and “Finish” to restart the Portal Application Server.

Change the Portal’s superadmin password


If you have not yet changed the default password (“superadmin”) for the Portal’s superadmin account please do the following:

1. Logon to the portal as superadmin with password superadmin.

2. Select the Provider tab and then select “Administrators” under the Tasks heading.

3. Under Available Administrators select the Edit button corresponding to the User Name superadmin

4. Enter the Old Password (superadmin) and then enter a new password in the New Password and Confirm Password fields.

5. Click on Save.

Note: If you are doing portal self-monitoring, you will also need to change the password reference for self-monitoring. This would be the superadmin’s password.

Change the Portal’s superadmin password


IMPORTANT NOTE: For each GLOBAL RSM using the superadmin account you will need to run the RSM Maintenance Tool (%RSM_HOME%\RSM211\RSMMaintenanceTool.cmd) and change the superadmin password like this:

Then click “Next” three times and “Finish” to restart RSM with new superadmin password.

Change the Portal’s default “user” and “portalmon” account’s password


Export Portal Configurations Periodically




• It is recommended that on a weekly or monthly basis you export your Portal’s configuration data using the PL/SQL scripts export_data.sql or expdp_conf-11g.sql from BMC Software.

• The export file generated serves as a backup of your portal configurations and can be used to restore your portal in the event of a catastrophic Portal database failure.

• In addition the export file can also be used to replicate your production portal environment in a test and/or development portal environment.

• Portal configuration data exported would include all of your Portal's accounts, elements, parameters, users, passwords, user groups, shared credentials, object groups, element profiles, thresholds, blackout periods and notification rules.

Note: These scripts do not export historical data from elements in the Portal. Please see the BMC Performance Manager Portal Installation Guide – “Backing up the database” for details on exporting historical data.



• The export_data.sql script leverages Oracle’s older export utility “exp” and should be run as the BMC Portal Database user (Also Known As the “SCHEMA_LOGIN” defined in BMCPortalKitInstalledConfiguration.xml – default is “pe”).

• The expdp_conf-11g.sql script requires some prerequisites be setup in the Portal Database as sysdba but has the benefit of using Oracle’s newer export utility “expdp” which is faster and automatically compresses the export data.

• These scripts can be found on BMC’s ftp site here: ftp://ftp.bmc.com/pub/BPMPortal/scripts/

Filenames:expdp_conf-11g.sql expdp_conf-11g.sql_INSTRUCTIONS.htm export_data.sql

• The directions for export_data.sql script are contained in the export_data.sql file at the top of the file. The directions for running the expdp_conf-11g.sql file are in a separate .htm file in the scripts directory.

ftp://ftp.bmc.com/pub/BPMPortal/scripts/





• Take the files generated from either the export_data.sql or the expdp_conf-11g.sql and store these files in a safe place.

• This is very helpful if the Portal database ever needs to be recovered and your regular backup method may have failed.

• If you have any questions or run into issues with exporting the Portal configurations, please contact BMC Customer Support.

For Oracle RAC database instances


Requirement for Oracle RAC portal database instances


• If you are using an Oracle RAC database for Portal with Scan IP then please be sure to perform these changes using the BMC Performance Manager Portal Maintenance Tooljdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hostvip)

(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=dbservice)))

• If you are using RAC without Scan IP, use the original configuration as per the BMC Portal Oracle RAC URL jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=HOST1)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=HOST2)(PORT=1521))(ADDRESS=(PROTOCOL=TCP) (HOST=HOST3)(PORT=1521))(LOAD_BALANCE=OFF)(FAILOVER=ON) (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=BMCPDS_RAC)(FAILOVER_MODE=(TYPE=SELECT) (METHOD=BASIC)(RETRIES=10)(DELAY=10))))

• For example here is what an Oracle RAC JDBC URL looks like for BMC jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=SCAN-ABCDE.bmc.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=BMCPDS_RAC)))

• Please refer to the BMC Portal Installation Guide - “Support for Oracle Real Application Clusters” for more details of the steps to enable the RAC URL in the Portal.

Disable deferred_segment_creation setting in Oracle 11.2




• As per BMC’s Knowledge Article KA410585 disable Oracle 11.2's deferred_segment_creation setting in the Portal database because in some circumstances, having deferred_segment_creation enabled (set to “true”) can prevent the portal from creating partitions. Oracle enables this feature by default although Oracle’s documentation says this:

“Before creating a set of tables, if it is known that a significant number of them will not be populated, then consider setting this parameter to true.”

• In the context of applications that create a large number of empty tables, enabling deferred_segment_creation makes sense however with the portal there is a relatively small number of tables in the portal database (< 120 tables) and most portal database tables have data in them.

• To check if this setting is enabled, login to your Portal database instance as sysdba and run the PL/SQL command "show parameter DEFERRED_SEGMENT_CREATION" like this:

sqlplus / as sysdba SQL> show parameter DEFERRED_SEGMENT_CREATION ;

https://kb.bmc.com/infocenter/index?page=content&id=KA410585



• If DEFERRED_SEGMENT_CREATION is set to TRUE then work with your Database Administrator (DBA) to get this set to FALSE.

• You can change this to FALSE as follows by running the following PL/SQL command as sysdba: ALTER SYSTEM SET deferred_segment_creation=false scope=spfile;

• After changing this setting from TRUE to FALSE your Portal database instance will need to be stopped and restarted for this change to take effect.

• Before stopping the portal database you should always stop your BMC Portal Appserver first and then restart your BMC Portal Appserver after the portal database has been restarted.

• Please contact BMC Customer Support if you require further assistance disabling DEFERRED_SEGMENT_CREATION in your Portal database.

Enable all Portal database datafiles to AUTOEXTEND


How to avoid “unable to extend” Oracle errors in the portal database including:

• ORA-01651: unable to extend save undo segment• ORA-01652: unable to extend temp segment • ORA-01653: unable to extend table • ORA-01654: unable to extend index



• BMC strongly encourages all Portal customers to enable all of the Portal database datafiles to AUTOEXTEND by default to prevent Portal outages and possible data corruption.

• If the Portal database runs out of disk space or tablespace then a significant Portal outage and/or Portal data corruption could result. This could lead to missing data points and gaps in data.

• This becomes very apparent if availability reports are generated as the data will not be available to report availability and you’re likely to see a large time frame of unavailability.

• If autoextend is not enabled and the Portal tablespaces fill up, you will see errors like the following in the portal.log file:

"ORA-01653: unable to extend table PE.JBM_MSG by 128 in tablespace PE01"

Note: This is a sample message and the names of the table and tablespace may change



• There are the 14 tablespaces included in the Portal’s database: DATA01, DATA02, DATA03, DATA04, INDEX01, INDEX02, INDEX03, INDEX04, NSDP01, PE01, PSDP01, PV01, PV02 and TOOLS01

• To determine if the Portal tablespaces are enabled to AUTOEXTEND run the following PL/SQL query against your Portal database as sysdba:

SELECT df.tablespace_name as "Tablespace Name", df.file_name as "File Name", df.AUTOEXTENSIBLE FROM DBA_DATA_FILES DF WHERE df.tablespace_name in ('DATA01','DATA02','DATA03','DATA04','INDEX01','INDEX02','INDEX03',‘

INDEX04','NSDP01','PE01','PSDP01','PV01','PV02','TOOLS01') OR df.tablespace_name LIKE 'UNDO%' ORDER BY df.tablespace_name, df.file_name;

• The query result above will show “YES” in the AUTOEXTENSIBLE column for any datafile set to autoextend as seen in this output example on the next slide.



• If any of the Portal tablespaces are not enabled to autoextend please work with your Database Administrator (DBA) to correct this. Here is an example of how to enable auto extend on a tablespace’s datafile:

sqlplus / as sysdbaalter database datafile 'C:\ORACLE\ORADATA\BMCPDS\UNDOTBS01.DBF' autoextend on;



• In order to proactively detect tablespace issues, BMC recommends using Oracle Self Monitoring explained in upcoming slides.

• If for any reason, you are not able to setup the datafiles to autoextend, BMC also recommends monitoring the portal.log file for the “unable to extend” error.

• Setup an element in the Portal to monitor the application server using the BMC Performance Manager Express for Servers – Log Management solution and monitor the portal.log file

• Configure the element and use the string “unable to extend” to trigger the alert.

• If you see this error in the portal.log file, it is important to address it right away as data can be lost and database corruption can occur.

• For any further questions or issues regarding Portal database tablespaces, please contact BMC Customer Support

Make certain your RSM hosts meet the minimum requirements




• Make certain your BMC Remote Service Monitor (RSM) hosts meet or exceed the minimum requirements specified in the BMC Portal Planning and Implementation Guide for version 2.10 on page 45 where it specifically outlines the requirements for a Windows RSM host as follows:

- 4 CPUs, 3.2 GHz Pentium 4 or better processor, 64-bit- 8 GB or more of memory

• BMC recommends running the RSMs in 64-bit mode. BMC has proven that an RSM running in 64-bit mode can adequately handle much more than an RSM running in 32-bit mode.

• If users install a new RSM, by default it will be installed as 32 bit. Users will need to use the RSM Maintenance Tool to change to 64 bit.

http://documents.bmc.com/supportu/documents/01/59/440159/440159.pdf



• Generally with 8 GB of physical RAM on an 64-bit RSM host an RSM running in 64-bit mode can usually be safely configured with 4 GB (4096 MB) as a “Max Memory” set as a “Custom Size” as seen here:

• It is important to configure an RSM’s “Max Memory” setting to be at most 1.8 GB less than the Available amount of Physical Memory on the RSM host when the RSM Service is not running. So for example on a 64-bit RSM host has 5,500 MB of Available Physical Memory when the RSM Service is not running then the RSM’s “Max Memory” setting should be no more than 3655 MB (5500 – 1845 = 3655).

• The maximum recommended setting for an RSM’s “Max Memory” setting can be expressed with this equation: “Max Memory” < [ (Available Physical Memory when the RSM is not running) – 1.8 ]

Remove unused Performance Managers of type "Solution"




• Unused Performance Managers can add overhead to the amount of processing the Portal Application server does and can slow down the startup time of the Portal Application server.

• Therefore BMC recommends customers remove unused Performance Managers of type "Solution" where both the element count and element profile count are zero.

1. To remove Performance Managers of type "Solution" logon to the Portal as superadmin and select the Portal tab then select “Performance Managers” under Tasks.

2. Next click once on the “Elements” heading in the table listing of the “Performance Managers” loaded on your portal so Performance Managers with zero elements associated with them appear at the top of the list.



3. Select the checkbox next to each PM of type "Solution" where zero (0) appears in both the Elements and Profiles column like this:

4. Click on the Delete button and then click on the Delete button again when prompted to “confirm deletion of the following Performance Managers”.

NotePerformance Managers whose status is "In Use" or whose type is "Core" cannot be deleted. Any Performance Manager removed can be uploaded again later as superadmin if there is a need to use the Performance Manager in the future.

Upgrade to Portal 2.11 and upgrade your Performance Managers




If you are not yet running Portal 2.11.00 then upgrade to Portal 2.11.00. Upgrade by following these steps:

1. Follow the guidelines under the Export Portal configurations periodically section of the BMC Performance Manager Portal Best Practices Guide. In other words, make sure you have at least a backup of the Portal’s configurations if not the entire Portal database before you begin to upgrade your Portal.

2. When running the upgrade (using the Portal 2.11.00 installer) ONLY select the Performance Managers (PMs) during the upgrade of the Portal you plan to upgrade soon and new PMs which you plan to implement soon. After the Portal upgrade is completed you will “Publish” (upgrade) the Performance Managers.



Apply the latest Performance Managers (PMs) obtained from BMC’s EPD site: http://www.bmc.com/support/support-central.html - Select Product Downloads, enter the Performance Manager name, then select the ‘Patches’ tab to obtain the latest solution.

For convenience the PMs listed above are here: ftp://ftp.bmc.com/pub/BPMPortal/Five-PMs.zip Use the table above to determine the version of the commonly used solutions you should upgrade.

*The Oracle 2.8.52 and Unix/Linux 2.7.70 PMs are not in the CD Bundle for Portal 2.11.00Note:First upgrade the Operating System Performance Managers for Unix and Windows above then upgrade the Log Management Performance Manager.

http://www.bmc.com/support/support-central.html

ftp://ftp.bmc.com/pub/BPMPortal/Five-PMs.zip





This is the screen the installer will display where you are prompted to upgrade your Performance Managers. Select only the Performance Managers you want to upload to the Portal to be upgraded manually after the Portal upgrade.



3. After the Portal application is upgraded, make sure you can access the Portal login page.

4. The RSMs will attempt to auto upgrade or you can perform an over the top installation of the RSMs if you have downloaded the RSM executable to each of the RSMs reporting to the Portal. Make sure your RSMs have upgraded and are showing up as online with a green icon ( ) in the Portal UI as seen here:

NOTE: After all your RSMs have automatically upgraded, export the portal configurations (using export_data.sql or expdp_conf-11g.sql) and then take a backup (snapshot VM) of the portal App Server host to have as a milestone checkpoint.



5. Logon to your Portal as a superadmin, upgrade any BMC Performance Manager Solutions which you are using and which may need an upgrade.

Publish each Performance Manager one at a time to ensure there are no issues. Login to the Portal as superadmin, select the Portal tab, select Performance Managers, upload the Performance Manager (PM) *.par file and click Publish. After seeing Published status for the Performance Manager, wait until an “Upgrade thread ### completed” message appears in the portal.log before upgrading the next Performance Manager to ensure all elements have been upgraded to the new performance manager:

On Windows: CD /D %BMC_PORTAL_KIT_HOME%\appserver\websdk\tools\jboss\server\all\log FINDSTR /C:"Upgrade thread" portal.log

On Solaris: cd $BMC_PORTAL_KIT_HOME/appserver/websdk/tools/jboss/server/all/log grep "Upgrade thread" portal.log



6. After the Performance Manager Solutions are upgraded, allow 30 minutes for data collection to resume. Verify data collection is occurring as described in the Check Portal health by running applications-not-collecting.sql daily section of the BMC Performance Manager Portal Best Practices Guide. ftp.bmc.com/pub/BPMPortal/BMC Performance Manager Best Practices Guide.docx

7. As an extra safety measure, we recommend you take a database configuration backup (using export_data.sql) after the solutions have been upgraded. If your database is on a VM, you could also take a snapshot after all the solutions have been upgraded.

If you encounter any difficulties when trying to upgrade to Portal 2.11.00 then please contact BMC Customer Support for assistance.




Check Portal health by running applications-not-collecting.sql daily




“How can I tell if my portal is collecting data?”

• To programmatically check if the Portal is having difficulties collecting monitoring data run the PL/SQL script “applications-not-collecting.sql” at least once per day.

• This script will list any application instances that have not collected data for the last 90 minutes including the account name owning the element, element name, application class, RSM name and the time when the application class most recently collected data.

• Here is a sample output of the applications-not-collecting.sql script:

Account: My Account RSM Name: RSMP02 Element Name: m123.abcxyz.com Application Instance: Windows Service Last Data Point Collected at: 05:43:40 AM SEP 22, 2014



• This script can be automated using the “Portal-Watcher-Bot.zip” to send you an e-mail with the output of the script. The “Portal-Watcher-Bot” can be installed on any host where both Oracle’s sqlplus client and the Java Developer’s Kit (JDK) are installed like your portal database host. The Portal-Watcher-Bot.zip contains the applications-not-collecting.sql script and can be found here:

ftp://ftp.bmc.com/pub/BPMPortal/

• The applications-not-collecting.sql script can be run manually like this:

1. Copy the script over to the Portal database server2. cd <directory where this script applications-not-collecting.sql is saved>3. Login to the Portal database and run the script For example:

sqlplus pe/pe@BMCPDSSQL> @applications-not-collecting.sql

4. The output of the script will be saved to the directory where the script was run






• If any elements appear in the report from the applications-not-collecting.sql script then restarting the RSM Service will usually resolve the data monitoring issue.

• If RSMs and/or elements repeatedly have issues collecting data then please contact BMC Customer Support for guidance on how to best resolve these issues.

Recommended order for stopping and starting Portal components




1. Stop the BMC Portal App Server: If you are not yet running portal version 2.11 then follow the instructions below. Starting in 2.11.00 this will be done automatically in Portal startup scripts and service.

On Windows:a) Locate the BMC Portal Service in Services and stop that service.b) Remove ONLY the following sub-directories by using these commands in a Windows Command prompt:

RMDIR /S /Q "%BMC_PORTAL_KIT_HOME%\appserver\websdk\tools\jboss\server\all\tmp" RMDIR /S /Q "%BMC_PORTAL_KIT_HOME%\appserver\websdk\tools\jboss\server\all\work“ RMDIR /S /Q "%BMC_PORTAL_KIT_HOME%\appserver\websdk\tools\jboss\server\all\data\hypersonic" RMDIR /S /Q "%BMC_PORTAL_KIT_HOME%\appserver\websdk\tools\jboss\server\all\data\wsdl" RMDIR /S /Q "%BMC_PORTAL_KIT_HOME%\appserver\websdk\tools\jboss\server\all\data\xmbean-attrs" RMDIR /S /Q "%BMC_PORTAL_KIT_HOME%\appserver\websdk\tools\jboss\server\all\data\tx-object-store“

NOTES: If you are also running the Impact Portal, DO NOT DELETE everything from the ~/data directory. This cache file clean up is only necessary for Windows installations. Solaris does this automatically during the Appserver start up.

On Solaris:Log on as root and run the following commands:# cd /etc/init.d# ./BMCPortalAppserver stop



2. Stop the BMC Portal Web Server:

On Windows:Locate the BMC Portal Web Service in the Services list and stop that service

On Solaris:Log on as root and run the following commands:# cd /etc/init.d# ./BMCPortalWebserver stop

3. Stop the BMC Portal Database:

On Windows:Locate the instance name in the Services list and stop that service

On Solaris:Log on to the Database system as OS user with Oracle DBA rightsConnect to the instance using SQLPlus as sysdba.# sqlplus / as sysdbaSQL> SHUTDOWN IMMEDIATE

Once all components are stopped, the Portal components can be then be started in the order shown on the next slide.



1. Start the BMC Portal Database.On Windows:

Locate the instance name in the Services list and start that serviceOn Solaris:

Log on to the Database system as OS user with Oracle DBA rightsConnect to the instance using SQLPlus as sysdba# sqlplus / as sysdbaSQL> STARTUPSQL> exit

2. Start the BMC Portal Web Server:On Windows:

Locate the BMC Portal Web Service in the Services list and stop that serviceOn Solaris:

Log on as root and run the following commands:# cd /etc/init.d# ./BMCPortalWebserver stop

3. Start the BMC Portal App Server:On Windows:

Locate the BMC Portal Appserver Service in the Services list and start that serviceOn Solaris:

Log on as root and run the following commands:# cd /etc/init.d# ./BMCPortalAppserver start

Clustered Portal Startup Best Practice




• The App Server cluster is build on a Master => Slave relationship with no ability to designate which is which. Because if this we need to ensure the correct startup sequence for the cluster is in place.

• Prerequisites:Prevent the App Server service from starting on the secondary node automatically when the system is rebooted. We would prefer this change be made on both nodes, but at least one must be done.

On Windows: Set the “Startup Type” of the BMC Portal Application Server & Web Server services to “Manual”.

On Solaris:Check your /etc/rc*.d directories for the SxxBMCPortalAppserver script and remove itExample:

# cd /etc/rc3.d# ls -al *BMC*-rwxrwxr-x 1 root root 1434 Feb 18 2014 S98BMCPortalAppserver-rwxrwxr-x 1 root root 1015 Aug 9 2013 S99BMCPortalWebserver

Check your /etc/init.d directory for the start up scripts as well Example:

# cd /etc/init.d# ls -al BMC*-rwxrwxr-x 1 root root 1434 Feb 18 2014 BMCPortalAppserver-rwxrwxr-x 1 root root 1015 Aug 9 2013 BMCPortalWebserver



• Overview of the steps to shutdown and restart the clustered Portal

1. Shutdown both members of your portal cluster – both Portal App Servers.

2. Start one Portal App server in your cluster.

3. Wait for "Started in" message to appear in portal.log.

4. Wait for the ALL the RSMs to connect to the running portal.

5. Start the second portal App Server.

6. Verify that the portals are communicating with each other.


• You can use the Portal UI to check the status of the RSMsLog in as superadmin, Click on the Portal Tab then Remote Service Monitors under Tasks

OR

• Run the following commands to search the log files for the status.On Windows, to confirm that the "Started in" message appears, and all the RSMs have connected to the running portal, run the following commands: D:\> CD /D %BMC_PORTAL_KIT_HOME%\appserver\websdk\tools\jboss\server\all\log D:\> FINDSTR /C:"Started in" portal.log D:\> FINDSTR /C:" => ONLINE" portal.log

On Solaris, to confirm that the "Started in" message appears, and all the RSMs have connected to the running portal, run the following commands: # cd $BMC_PORTAL_KIT_HOME/appserver/websdk/tools/jboss/server/all/log # grep "Started in" portal.log # grep " => ONLINE " portal.log




• The previous command should find lines in the portal.log similar to the following example: UTC,STDERR> [INFO,AgentServiceImpl,changeStatus] RSM rsm1: UNKNOWN => ONLINE UTC,STDERR> [INFO,AgentServiceImpl,changeStatus] RSM rsm2: UNKNOWN => ONLINE UTC,STDERR> [INFO,AgentServiceImpl,changeStatus] RSM rsm3: UNKNOWN => ONLINE UTC,STDERR> [INFO,AgentServiceImpl,changeStatus] RSM rsm4: UNKNOWN => ONLINE UTC,STDERR> [INFO,AgentServiceImpl,changeStatus] RSM rsm5: UNKNOWN => ONLINE

• To confirm that the portals are communicating, search the portal.log files for "Members : 2" messages.

On Windows, run the following commands on BOTH App Server hosts: D:\> CD /D %BMC_PORTAL_KIT_HOME%\appserver\websdk\tools\jboss\server\all\log D:\> FINDSTR /C:"Members : 2" portal.log

On Solaris, run the following commands on BOTH App Server hosts: # cd $BMC_PORTAL_KIT_HOME/appserver/websdk/tools/jboss/server/all/log # grep "Members : 2" portal.log



• The output of the grep or findstr commands should show that each member of the cluster recognizes the other:

Primary Appserver: <O,11/02/09 08:33:34 UTC,DefaultPartition> All Members : 2 ([123.45.679.47:9379, 123.19.643.49:9379])

Secondary Appserver: <O,11/02/09 08:30:34 UTC,DefaultPartition> All Members : 2 ([123.45.679.47:9379, 123.19.643.49:9379])

Enable “External Summarization”


Enable “External Summarization”


• External summarization for granular data is enabled by default to be executed after every 1 hour in Portal 2.11. The summarized data is grouped into increments of 30 minutes and written to the table.

• The directions given in this section pertain only to portal versions 2.9.10, 2.9.12, 2.10.00 and 2.10.01.

• Enabling “External Summarization”, will externalize the summarization process to the Oracle database job. This will allow Portal Application Server to focus resources on processing data from your RSMs and also on handling user’s browser requests.

To enable “External Summarization” set the following in the drmop.properties file:portal.history.parameter.externalsummarization.enabled=trueportal.history.parameter.externalsummarization.schedulehours=EVERYHOUROn Windows, the drmop.properties file can be found here:

%BMC_PORTAL_KIT_HOME%\appserver\websdk\tools\jboss\server\all\conf\properties\drmop\ On Solaris, the drmop.properties file can be found here:

$BMC_PORTAL_KIT_HOME/appserver/websdk/tools/jboss/server/all/conf/properties/drmop/*****Remember do not save any backup copies of drmop.properties in any sub-directory of

BMC_PORTAL_KIT_HOME as this will cause issues with Portal startup *****

Enable “External CDE”



• The purpose of enabling the Continuous Data Export is to reduce the amount of table space required to retain the desired historical data in the PDS database.

• The purpose of “externalizing” the CDE transfer is to improve the performance of the Portal by moving the processing of the data transfer from the Portal Appserver to Oracle to perform the function using direct data transfer between the PDS and CDE databases.

• The initial CDE configuration be done first. Steps to configure the Continuous Data Export are in the BMC PM Monitoring and Management Guide on pages 212 through 230

• Enabling the External CDE requires additional steps to enable and configure. These steps are also in the BMC PM Monitoring and Management Guide on pages 230 through 244


https://documents.bmc.com/supportu/documents/20/74/462074/462074.pdf






• To determine if “External CDE” is enabled, check the following property in the datafeed.properties file. #Enabling the transfer of the granular data to external Database#Default value is falseportal.datafeed.externalCDE.enabled=true

On Windows, the datafeed.properties file can be found here:%BMC_PORTAL_KIT_HOME%\appserver\websdk\tools\jboss\server\all\conf\properties\drmop\

On Solaris, the datafeed.properties file can be found here: $BMC_PORTAL_KIT_HOME/appserver/websdk/tools/jboss/server/all/conf/properties/drmop/

• Execution of BPMCLI refreshDatafeedMetadata, regularly, is mandatory for both internal and external configurations to push the metadata information for new elements/discovered application instances into CDE database. Example scripts are provided on the next slide.

• Example of a BPMCLI refreshDatafeedMetadata you can schedule to run automatically to accommodate the frequency of changes in your environment.

On Windows:@echo offcd /d "%BMC_PORTAL_KIT_HOME%"\appserver\util\BPM_CLIset PATH="%BMC_PORTAL_KIT_HOME%"\appserver\websdk\tools\jdk\bin;%PATH%set JAVA_HOME="%BMC_PORTAL_KIT_HOME"%\appserver\websdk\tools\jdk@echo offrem Change user name and user password .It should be fullAccess username and password.rem bpmcli -portal PortalWebServerName notAppserver -login adminname -pass adminpassword -c refreshDatafeedMetadata

Example:bpmcli -portal clm-pun-013773.bmc.com -login user -pass user -c refreshDatafeedMetadata



• Example of a BPMCLI refreshDatafeedMetadata you can schedule to run automatically to accommodate the frequency of changes in your environment.

On Solaris:cd $BMC_PORTAL_KIT_HOME/appserver/util/BPM_CLIPATH=BMC_PORTAL_KIT_HOME/appserver/websdk/tools/jdk/bin:$PATHJAVA_HOME=$BMC_PORTAL_KIT_HOME/appserver/websdk/tools/jdkexport JAVA_HOMEexport PATH#Change admin name and admin password .It should be fullAccess username and password.#./bpmcli.sh -portal PortalWebServerName notAppserver -login adminname -pass adminpassword -c refreshDatafeedMetadata

Example:./bpmcli.sh -portal clm-pun-013773.bmc.com -login user -pass user -c refreshDatafeedMetadata



Portal Self Monitoring


Portal Application server and RSM monitoring Solution


The purpose of this section is to highlight the important parameters in the portal and RSM self-monitoring solution. A description of each parameter is explained in the online help as shown in the screen shot below.

Portal Self-monitoring



• It is recommended that you setup the Portal self-monitoring solution in Production as well as in a Development environment.

• Setup the development environment to monitor the Production Application server(s), web server(s), and RSMs as well.

• Then there are two chances for self-monitoring to catch what is going wrong in the Portal. For example, if there is a complete failure of the Production Portal, the notification on the Development Portal which is monitoring the Production Portal will be used.



• Screen shots below highlight the parameters to be used to alert and warning on.

• Change Best settings for thresholds depending on your setup and previous parameter history.

• Choose the defaults to begin with and later tune the values depending on the setup specifics.

• You can also use set the Alarm after N times option to 2.



• Used Memory alerts should be set to 80% and 90% of the “Max Memory” setting as set in the Portal Maintenance tool. In the example shown “Max Memory” is 4096 MB so the Used Memory thresholds are at 3276 MB (80%) and 3686 MB (90%).



• Queue Depth thresholds should be set accordingly and not as the default.

• Queue depth indicates how many data payloads/packets are sent by the RSM to the Portal but Portal has not processed them yet.



• Datafeed monitoring is applicable only if you are using internal CDE.

• The Dropped parameter count is higher if the internal CDE was not able to send data to the CDE database.

• Queue parameters indicate that the RSM has sent the data to the Portal but the Portal has not processed it in the CDE.

RSM Self Monitoring


RSM Self Monitoring


• It is very important to monitor the health of the RSM to avoid any accidental data collection issues.

• For Process Memory and CPU usage, depending on average trend of the memory usage, decide the best value for the threshold. For 4 GB maximum java heap space, 3072 would be an appropriate value.

• The Process Status parameter is the most important parameter which indicates if the RSM is running.

RSM Self Monitoring


• The setting shown in the above snapshot for the “Active Thread Count” parameter is for a 32-bit RSM. For a 64-bit RSM, increase it to 1200 and 2000.

• Depending on your environment, you can change the alert based on the average value of the active thread count.

• Used Memory alerts should be set to 80% and 90% of the “Max Memory” setting as set in the RSM Maintenance tool.

RSM Self Monitoring


• The Portal to RSM clock skew parameter is the second most important parameter for RSM monitoring.

• If the clock is set incorrectly, it will reflect the wrong timestamp for the data points in the Portal This can generate false alarms.

• If the RSM is set to a future date, the Portal will report errors in the portal logs and will cause performance issues.

Notify if an RSM is not collecting data




• BMC recommends if you are using 2.10.00 or below, setup a log file monitoring element for the portal.log file and monitor it for the warning below. Then you can setup an alert to be sent to anyone responsible for administering the RSMs.

• Usually when this message is seen in the log, the best course of action is to restart the RSM(s).

• In Portal version 2.10.00, users can search in the portal log for the following line:

[SingletonScheduler_Worker-8]> [WARNING,PortalDatabaseJobsMonitor,monitorRSMNotCollectingData] BMC-MOP000010S:- Details=[RSM on HOSTNAME.bmc.com has last collection date =06-MAR-15 05.35.:03 UTC. It is not collecting data since last 40615 Minutes ]

Search for “BMC-MOP000010S”



• From Portal version 2.11.00 onward, users can setup an email notification to be sent out if an RSM has not collected data for the past 60 minutes. Setting up this notification will make sure you are notified if an RSM is not working as expected and the RSM can be restarted as needed.

• Change the following properties to enable the email notification in the drmop.properties file. Setting the value to 60 will start notifying after every 60 minutes.

portal.monitor.portalselfMonitorhelper.warn.rsmcollection.enable=true (Default is true)

portal.monitor.portalselfMonitorhelper.warn.rsmnotcollectingsince.minutes=60 (Default is 60) portal.monitor.email.recipients.list=your Admin email ID separated by comma. (Default is empty)

For example: [email protected],[email protected],[email protected]

BMC Portal Oracle Database self-monitoring




• To monitor the Portal database, BMC recommends using the BMC-PM-Express-for-Databases-Oracle Performance Manager solution.

• After successful configuration of the solution, please enable the thresholds defined in the screen shots on the next few slides.

• The rest of the parameters can remain at the default values.

• These parameters which are being featured here are the ones which are most important for monitoring the status of the Portal database.



Oracle Instance Status parameter • Reports the availability of the Oracle database instance.

Recommendation: If you encounter alerts for this parameter, please restart the Oracle instance service and the listener and check for errors in the Oracle alert log. The alert log defaults to: %ORACLE_HOME%\admin\BMCPDS\bdump\alert_bmcpds.log.



Oracle Listener Status parameter• Reports the availability of the Oracle listener

Recommendation: If you encounter alerts with this parameter, please restart the Oracle listener. If this does not solve the issue, please approach your Oracle DBA or BMC Customer Support for further assistance and provide the Oracle listener log found here:

%ORACLE_HOME%\NETWORK\log\listener.log (on Windows) $ORACLE_HOME/NETWORK/log/listener.log (on Solaris)



Space Utilization parameter• Displays the percentage of total database space available that has already been used in the

tablespace.

Recommendation: Ensure that sufficient space is available based on planned tablespace growth. Add an extra data file to the problem tablespace or contact BMC Customer Support for further assistance. This is why we recommend setting the tablespace to autoextend.



Buffer Cache Hit Ratio parameter• Displays the percentage of physical data blocks read compared to requests for data blocks in

the last reporting interval. 100% would mean everything Oracle read was already in memory.

0% would mean that nothing was in memory. • This parameter alarms when too few data requests accessed the buffer cache; too many

requests accessed the disk.

Note: Set these thresholds and setup the Alert after N times option to alert after 10 times.

Recommendation: Depending on the systems memory, provide a higher SGA based on the steps in the BMC Performance Manager Portal Best Practices Guide.


Tuning the Portal for best performance


Portal java heap memory tuning


• It is important to tune the application server’s memory settings to ensure the proper balance of memory for the application to use as well as the OS itself. It is important to allocate enough memory to application server to ensure the best performance from the application server.

• Setting the heap memory is a delicate balance because if you allocate too much to the application, there won’t be enough left for the OS functions and may result in odd system behaviour.

• You can tune the memory parameters by running the BMC Portal Maintenance Tool and under the Configuration tab change the parameters according to your environment.

Access the BMC Portal Maintenance Tool via the following paths: On Windows: %BMC_PORTAL_KIT_HOME%\appserver\BMCPortalMaintenanceTool.cmd On Solaris: $BMC_PORTAL_KIT_HOME/appserver/BMCPortalMaintenanceTool.sh

Portal java heap memory tuning


Configure the application server’s “Maximum appserver memory”.

64 -bit only: “Maximum appserver memory” < [(Available Physical Memory when the Portal app server is not running) – 1.8 GB]

32-bit: “Maximum appserver memory”: 1376 (MB) use “Custom Size”

Portal Oracle database SGA/memory tuning


Portal Oracle database SGA/memory tuning for Oracle 10g• For users running Oracle 10 / BMC Datastore 2.7.00 / 2.7.10 –

• For detailed steps, please refer to Section 2 of the Portal Quick Tune Guide found here: ftp://ftp.bmc.com/pub/BPMPortal/Portal_quick_Tune.txt for detailed steps.

Note: Setting a higher value for these parameters will improve the performance of the Portal Application Server.

alter system set shared_pool_size=0 scope=spfile;alter system set java_pool_size=0 scope=spfile;alter system set LARGE_POOL_SIZE=0 scope=spfile;alter system set sga_max_size=1400M scope=spfile;alter system set sga_target=1400M scope=spfile;

By default on Windows 32-bit you, can set the SGA to 1400MB and PGA to 600MB only. Refer to the Portal Quick Tune Guide for steps to increase this limit.

alter system set sessions=750 scope=spfile;alter system set processes=650 scope=spfile;alter system set pga_aggregate_target=600M scope=spfile;

ftp://ftp.bmc.com/pub/BPMPortal/Portal_quick_Tune.txt




Portal Oracle database SGA/memory tuning for Oracle 111. Please set the value of memory_max_target and memory_target to the optimal memory you

can allocate for your Oracle instance.

• Consider the OS, RAM, and the different applications running on it to decide the best memory value to allocate to Oracle.

• The following chart shows the recommended guideline for the memory target settings "memory_max_target" and "memory_target".

Note: Setting a higher value for these parameters will improve the performance of the Portal Application Server.



2. Get the location of the spfile and make sure you have backed up this show parameter spfile;

3. Create a pfile from the spfile as follows as sysdba: On Windows: create pfile='%ORACLE_HOME%\database\pfile_before_change.ora' from spfile='%ORACLE_HOME%\database\spfileBMCPDS.ORA';

On Solaris: create pfile='$ORACLE_HOME/database/pfile_before_change.ora' from spfile='$ORACLE_HOME/database/spfileBMCPDS.ORA';

4. On the sqlplus command prompt, execute the following statements as sysdba: alter system set memory_max_target=8192M scope=spfile; alter system set memory_target=8192M scope=spfile; alter system set sga_max_size=0 scope=spfile; alter system set sga_target=0 scope=spfile; alter system set pga_aggregate_target=0 scope=spfile;

Note:Setting the sga_target , sga_max_target, and pga_aggregate_target to zero will enable the Automatic Memory Management



5. It will automatically change the SPFILEBMCPDS.ORA.

6. Restart the instance and navigate to a command prompt on the Database Server.

7. The following sql statement will show the changed SGA value. Make sure it is changed to 8 GB.sqlplus / as sysdbashow parameter memory;show sga;

8. In case you want to restore the old SGA value, simply restore the backup of the spfile and restart the Oracle instance.

Set the retention of database tables based on the business need


• Depending on selection for history data retention, it will affect the data being displayed in the history chart and table view, the events displayed in the event tab and the health at a glance report. Overall there will be improvement in Portal performance.

***The value of the portal.history.element.statusHistory.retention property must match the value for the portal.history.element.summarizationDataPoint.retention property.

• The items above are mentioned in detail under the "Properties that control the raw data retention policies" section in the BMC Performance Manager Portal Monitoring and Management Guide. Please set the policies depending on your data retention requirement.

Set the retention of database tables based on the business need


• If you are backing up the files, please backup the files to any directory which does not come under $BMC_PORTAL_KIT_HOME. If you put the backup files in the sub folders under $BMC_PORTAL_KIT_HOME, the Portal will read those files and will result in errors.

• Retention settings can be changed via the following file:%BMC_PORTAL_KIT_HOME%\appserver\websdk\tools\jboss\server\all\conf\properties\drmop\drmop.properties

or$BMC_PORTAL_KIT_HOME/appserver/websdk/tools/jboss/server/all/conf/properties/drmop/drmop.properties

• Once you make any changes to the file, please restart the Portal server services.

• Refer to “Section 4:- Check the retention what you need for your Data” in the Portal Quick Tune Guide


Tune the data collection and reporting based on the business need




Report Update interval

• This is the interval at which the granular data is inserted in the database in regular work flow. • This interval can be modified at the Element and Element Profile level using the configuration screen

under the General Properties for the element. You can check the distribution on your system by using the following SQL.

select (COLLECTIONINTERVAL/1000/60) "Report Update Interval", count(*) "Element counts" from node, websdk_account wa

where istemplate='T' and configstate='ADDED'group by COLLECTIONINTERVAL;



How to change Report Update Interval on a global basis?1) If you are using Element Profiles, please change the Report Update Interval using an element

profile.2) If Element Profiles are not used, then you can change use the following SQL to change the

Report Update Interval globally.

sqlplus pe/pe@bmcpds@update_report_update_interval.sql 'RSM Cluster pattern' 'RSM name Pattern' 'Group Name pattern' 'Element Name Pattern' 'Report update interval'

• For more details and a comprehensive knowledge base for these database scripts and their filters, please refer to ftp://ftp.bmc.com/pub/BPMPortal/bulk-report-update-interval-update-plan-*.zip. * Refers to the latest version.

• To set your preferred report interval update for NEW elements, change the following property in the $BMC_PORTAL_KIT_HOME/appserver/websdk/tools/jboss/server/all/conf/properties/drmop/drmop.properties file.drmop.default.reporting.interval.minutes=10

ftp://ftp.bmc.com/pub/BPMPortal/bulk-report-update-interval-update-plan-*.zip

ftp://ftp.bmc.com/pub/BPMPortal/bulk-report-update-interval-update-plan-*.zip

Database AWR metrics before increasing the report update interval from 5 minutes to 10 minutes.


Database AWR metrics after increasing the report update interval from 5 minutes to 10 minutes. (60% improvement)




Collection Interval

• Collection interval is defined at the application class level. It is the interval at which polling is done by the RSM and it fetches data from the monitored element. It is available in the Application Properties under the Configure tab.



• The higher the value of the "Collection Interval", the lower the rate at which the events are generated.

• The higher the value of the "Collection Interval", the lower the rate at which the granular data is stored in the database.

• You can find the values of the Collection Intervals on your system by using the following SQL statement.

sqlplus pe/pe@Bmcpds

select COLLECTIONINTERVAL "Collection interval in minutes" ,count(*) "Applications"from application_instance where configstate='ADDED'group by COLLECTIONINTERVAL ;

Refer to “Section 9.1:- Reducing the data collection and the size of the granular data table.” in the Portal Quick Tune Guide




How to change the Collection Interval on a global basis?1) If you are using the Element Profiles, please change the Collection Interval at the Element Profile

Level.

2) If Element Profiles are not used, you can use the following SQL to change the Collection Interval globally. Please refer to the Best Practices Guide for more information.

-- The following will set the Collection Interval to 10 minutes.update application_instance set COLLECTIONINTERVAL=10 where node in ( select guid from node where displaytext like '%shr%'and PROFILE is null and configstate='ADDED'and istemplate ='F') and applicationname like '%windows%'; commit; NOTE: Do not set the collection interval greater than the reporting interval

3) Start the portal application server to make the changes take affect.


Tune the UNKNOWN events based on the business need


• We have noticed on scaled systems that there is a problem while sending UNKNOWN events.

• When No Data is displayed in the GUI for a parameter when an UNKNOWN event is received, the state of the parameter is changed to the UNKNOWN state and it generates an UNKNOWN event. Those UNKNOWN events can result in a flooding of the Event table and affects product performance.

• As of BMC Portal 2.9.10, UNKNOWN event generation is disabled by default using the property below in the %RSM_HOME%\RSM<Version>\server\rsm\conf\properties\rsm\rsmcfg.properties file.

com.bmc.patrol.patsdk.services.CollectionJob.ignoreUnknownState=true

Tune parameter collection by deactivating unwanted parameters


• Deactivating parameters helps to reduce the traffic between the portal application server and database server. It reduces the amount of parameter values which are being saved into the database and hence reduces the size of the database.

• BMC Recommends deactivating the parameters which have no thresholds defined unless needed for external reporting.

• Users have two options to deactivate parameters. • Deactivate parameters through the UI on the application/parameter Configure tab.• Deactivate parameters through the use of database scripts (see next slide)

Tune parameter collection by deactivating uneeded parameters


• For deactivation of parameters through the use of database scripts, please refer to ftp://ftp.bmc.com/pub/BPMPortal/de-activate-parameter_procedure_and_steps_*.zip for complete documentation and scripts which can be used to determine which parameters to deactivate. * refers the latest version on the ftp site.

ftp://ftp.bmc.com/pub/BPMPortal/de-activate-parameter_procedure_and_steps_*.zip

Portal AMIGO Program


The BMC Performance Manager Portal Assisting Migration Operations (AMIGO) program is designed specifically to help you upgrade with ease. The purpose of the program is to help you review the upgrade plan you put together and to see where areas of concern may be. The AMIGO program includes:

• A review of the upgrade plan you develop with BMC Customer Support• An upgrade checklist to ensure success• A "Question and Answer" session before you upgrade• Helpful tips and tricks for upgrade success based on previous experience• A follow-up session with BMC Customer Support after you perform the upgrade to discuss the outcome

Refer to our Knowledge Base - KA417814 to view the details and Checklist of details needed for an upgrade. This information will help you to "jump start" your upgrade planning.

If you would like to be a candidate for the AMIGO program, please contact BMC and open a Support issue referencing the AMIGO program and provide the following:Products and Versions involvedDate of planned upgrade(if known)

https://kb.bmc.com/infocenter/index?page=content&id=KA417814

— july 28, 2015 bmc performance manager portal webinar series – webinar 2 connect with bmc...

Documents

bmc community

bmc portal environment

bmc pm portal webinar

bmc customer support

portal administration

portal setup

continued performance

webinar feedback