rmf wto 2006 - file/rmf_wto_2010.pdf · how to generate wto messages from rmf performance...
TRANSCRIPT
How to Generate WTO Messages from RMF
Performance Exceptions
This document can be found on the web, www.ibm.com/support/techdocs
Search for document number WP100402 under the category of “White Papers”.
Version Date: March 2010
IBM Advanced Technical Support
Mike Bonett
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 2 of 15
Trademarks
The following terms are registered trademarks of International Business Machines Corporation in the United States
and/or other countries: AIX, AS/400, DB2, IBM, Micro Channel, MQSeries, Netfinity, NUMA-Q, OS/390, OS/400,
Parallel Sysplex, PartnerLink, POWERparallel, RS/6000, S/390, Scalable POWERparallel Systems, Sequent, SP2,
System/390, ThinkPad, WebSphere.
The following terms are trademarks of International Business Machines Corporation in the United States and/or
other countries: DB2 Universal Database, DEEP BLUE, e-business (logo), ~, GigaProcessor, HACMP/6000,
Intelligent Miner, iSeries, Network Station, NUMACenter, POWER2 Architecture, PowerPC 604,pSeries, Sequent
(logo), SmoothStart, SP, xSeries, zSeries. A full list of U.S. trademarks owned by IBM may be found at
http://iplswww.nas.ibm.com/wpts/trademarks/trademar.htm. NetView, Tivoli and TME are registered trademarks
and TME Enterprise is a trademark of Tivoli Systems, Inc. in the United States and/or other countries.
Oracle, MetaLink are registered trademarks of Oracle Corporation in the USA and/or other countries.
Microsoft, Windows, Windows NT and the Windows logo are registered trademarks of Microsoft Corporation in the
United States and/or other countries.
UNIX is a registered trademark in the United States and other countries licensed exclusively through The Open
Group.
LINUX is a registered trademark of Linus Torvalds.
Intel and Pentium are registered trademarks and MMX, Pentium II Xeon and Pentium III Xeon are trademarks of
Intel Corporation in the United States and/or other countries.
Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States and/or
other countries.
Other company, product and service names may be trademarks or service marks of others.
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 3 of 15
Introduction................................................................................................................................................................. 4 Architecture and Flow Overview .............................................................................................................................. 4 STEP 1: Start an interactive Monitor III session..................................................................................................... 5 STEP 2: Define the desired performance exception(s). ........................................................................................... 5 Step 3: Customize the ERBRMF3B JCL to run Monitor III as a started task or batch job.............................. 11 General Observations and Considerations ............................................................................................................. 14 Summary ................................................................................................................................................................... 14 Further Reference and Detailed Information......................................................................................................... 15
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 4 of 15
Introduction
IBM Resource Measurement Facility (RMF) provides the online RMF Monitor III session. Using
ISPF, a user can invoke an interactive Monitor III session to access reports; one of these is the
Workflow/Exceptions Report. This report presents information about system activity and system
resources. Within this report, system and resource performance threshold exceptions can be
defined. Exceptions (conditions that violate the thresholds) will be displayed on the ISPF screen
during a Monitor III monitoring cycle (for example, a job using a certain amount of CPU
processing, or a DASD volume at a certain I/O activity rate). Details of the
Workflow/Exceptions report options, and the performance metrics available for use in defining
exceptions, are documented in the RMF Report Analysis manual (SC33-7991).
It may be desirable to surface these exceptions as Write to Operator (WTO) messages, where
they can logged in the SYSLOG, observed by human operators, or (best of all) detected and
acted upon by automation products such as Tivoli NetView for z/OS or System Automation for
z/OS. This paper describes one technique to accomplish this, using functions and capabilities
provided by RMF. The steps to perform for this technique are:
1. Starting an interactive RMF Monitor III session.
2. Creating the desired performance exceptions.
3. Customizing JCL to run the RMF Monitor III session as a started task or batch job.
Architecture and Flow Overview
When RMF is installed, a set of CLISTS and REXX procedures are copied into the
SYS1.SERBCLS library (an installation may have changed the library name to meet local
naming standards). Details on the CLISTS and execs are documented in the RMF Users Guide
(SC33-7990), in the “RMF Client/Server Enabling” Chapter. The following diagram shows the
key CLISTS and procedures that are used to generate the WTO messages:
1. JCL (the RMF supplied procedure RMFM3B can be used) invokes the ERBM3B REXX
procedure, which controls the Monitor III background session.
ERBM3B ERB3RPH3
ERBM3BWX
ERBR3WFX
JCL
Started task
or batch job
Monitor III
Batch Control
Table Switch
Reporter Exit
WFEX Exit
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 5 of 15
2. ERBM3BWX allocates the Monitor III table dataset used to hold the report information.
3. ERB3RPH3 determines the report type (which is passed as a parameter in the JCL) and
calls the corresponding report processing exit handler.
4. ERBR3WFX is the WFEX report processing exit handler. It checks for exceptions in the
WFEX report, and issues any found as WTO messages. There are three exit handlers
contained in ERBR3WFX, each performing different WTO processing.
These CLISTs and REXX procedures can be easily modified to suit specific needs. For this
process a specific WFEX report process is used. ERBR3WFX exit handler 2 produces a WTO
message for each exception, and will be used. There are 2 ways accomplish this:
1. Make a copy of the ERBRPH3 REXX procedure, and modify its call to ERBR3WFX to pass
the desired exit handler number. Within the REXX code locate the line containing
SELECT CMD(ERBR3WFX)and change it to SELECT CMD(ERBR3WFX 2).
2. Make a copy of the ERBR3WFX REXX program and set exit handler 2 to always be
invoked. Within the REXX code locate the line Arg handler . and add the line
handler=2 right after it.
Any modified REXX procedure must be placed in the dataset at the top of the SYSPROC DD
concatenation in the JCL used within this process (documented later in this paper).
The rest of this paper will walk through the steps to define a DASD device activity rate threshold
and will show the resulting WTO messages that are generated.
STEP 1: Start an interactive Monitor III session.
The SERBCLS library (or its customized equivalent) must be allocated to the SYSPROC
concatenation of the RMF Monitor III user’s TSO session, to allow a Monitor III session to be
started from ISPF. Most installations have customized ISPF and added RMF as an application
that can be invoked directly via an ISPF command.
When a Monitor III ISPF session is started, a user ISPF output table dataset is allocated (it is
created if it does not exist) by the SERBCLS(ERBRMF3X) CLIST. This output table contains
the defined exceptions. The default name is <TSOUSERID>.RMFOSLEVEL.ISPTABLE (this
can be changed by modifying the ERBRMF3 and ERBRMF3X CLISTs in the SERBCLS
library). In this example, the dataset is named BONETT.RMFZOS12.ISPTABLE.
STEP 2: Define the desired performance exception(s).
From the RMF ISPF Main Panel, select the Monitor III Session (Option 3):
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 6 of 15
From the RMF Monitor III Primary Menu, select OVERVIEW
(Option1):
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 7 of 15
From the OVERVIEW Report Selection Menu, Select the Workflow/Exceptions (WFEX) Report
(Option 1)
The Workflow/Exceptions Report shows the current status of system activity; the bottom portion
of the screen shows current exceptions that have violated existing defined thresholds. These will
be issued as WTO messages when this process is completed.
For this example an exception to report any DASD device with a Device Activity Rate greater
than 5 will be created. Type in RO at the command prompt, to access the Report Options:
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 8 of 15
The Report Options Action Panel displays the defined exceptions. To add an exception
definition, type AD on the first line under the Action column:
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 9 of 15
The Definition and Criteria Screen appears. The information on this screen is defined in detail in the
RMF Report Analysis Guide. The following entries are created to define the I/O rate exception:
- CLASS ==> DEV
- QUALIFIER ==> ALL
- INIDICATOR ==> EX-ANY
- LABEL ===> (optional – left blank here)
- ALERT ===> NONE (relevant to screen display only)
Under Criteria Set 1, on the first line, DAR > 5 is entered. DAR is the metric representing
Device Activity Rate (Details on creating more sophisticated criteria definitions using resource
filtering, multiple metrics and Boolean logic are documented in the RMF Report Analysis
manual):
After completing the screen, press F3. The exception that has just been defined is now listed on
the Report Options Action Panel:
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 10 of 15
Press PF3 to exit the action panel. Any DASD devices that, in the latest cycle, have an activity
rate greater than the threshold are shown in the exceptions section:
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 11 of 15
Press PF3 several times to exit the RMF Monitor III session. The exception has now been saved
in the RMF ISPF user table.
Step 3: Customize the ERBRMF3B JCL to run Monitor III as a started task or batch job.
When RMF is installed, a started task procedure RMFM3B is provided. This runs the
Monitor III session and will invoke the appropriate RMF CLISTS to capture any surfaced
exceptions and produce related WTO messages (RMFM3B can be modified to run as a batch job,
if desired).
The JCL invokes the functions (described in the Overview section of this document) to do the
following:
- Accept parameters for the HLQ of the table dataset with the exceptions, and the type of
report to run.
- Create an ISPF table dataset in which exceptions surfaced by the report will be placed.
- Read the ISPF table dataset that the exception definitions were stored in Step 2 above, to
apply to the performance data.
- Invoke the appropriate report exit (for the workflow/exception report, this is the
ERBR3WFX RMF REXX procedure) to process the report at each cycle and create the WTO
messages for any exceptions.
The RMFM3B JCL will likely have to be modified for specific environments. Here is an
example of a working modification. Note that the JCL shown is for this specific environment and
will have to be modified for any other installation environment. It is based on the RMFM3B
sample provided with RMF (the example changes the task name to RMFWTO, the dataset names
must be adjusted for the target environment):
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 12 of 15
//RMFWTO PROC ISPF='SYS1.ISPF',REPORT=WFEX,HLQ=TEST,CLASS=O //DELETE EXEC PGM=IEFBR14 //ERBPHDS3 DD DSN=&HLQ..ERBPHDS3.ISPTABLE,DISP=(SHR,DELETE,DELETE) //SYSPRINT DD SYSOUT=* //ALLOC EXEC PGM=IEFBR14 //ERBPHDS3 DD DISP=(NEW,CATLG),DSN=&HLQ..ERBPHDS3.ISPTABLE, // UNIT=SYSDA,VOL=SER=DSX304, // SPACE=(TRK,(1,1,1)), // DCB=(LRECL=80,RECFM=FB,BLKSIZE=3120) //SYSPRINT DD SYSOUT=* //ERBM3B EXEC PGM=IKJEFT01,REGION=4M,DYNAMNBR=90, // PARM='ERBM3B &HLQ &REPORT' //ISPPROF DD DSN=&&TEMP,UNIT=SYSDA,SPACE=(TRK,(2,1,2)), // DCB=(DSORG=PO,LRECL=80,BLKSIZE=3120,RECFM=FB) //ISPLOG DD DSN=&&TEMP,UNIT=SYSDA,SPACE=(TRK,(5,5)), // DCB=(DSORG=PS,LRECL=125,BLKSIZE=129,RECFM=VA) //SYSPROC DD DISP=SHR,DSN=BONETT.CLIST // DD DISP=SHR,DSN=SYS1.COMBINED.ISPFCLIB // DD DISP=SHR,DSN=&ISPF..SISPEXEC // DD DISP=SHR,DSN=&ISPF..SISPCLIB //ISPPLIB DD DISP=SHR,DSN=&ISPF..SISPPENU // DD DISP=SHR,DSN=SYS1.COMBINED.ISPFPLIB //ISPSLIB DD DISP=SHR,DSN=&ISPF..SISPSENU //ISPMLIB DD DISP=SHR,DSN=&ISPF..SISPMENU // DD DISP=SHR,DSN=SYS1.COMBINED.ISPFMLIB //ISPTLIB DD DISP=SHR,DSN=&ISPF..SISPTENU // DD DISP=SHR,DSN=&HLQ..ERBPHDS3.ISPTABLE //***THE FOLLOWING IS THE DATASET CONTAINING THE WFEX DEFINITIONS // DD DISP=SHR,DSN=BONETT.RMFZOS12.ISPTABLE //***THE FOLLOWING 2 DATASETS ARE THE REQUIRED RMF ISPF TABLE DATASETS //***THEY MAY HAVE BEEN MOVED ELSEWHERE IN YOUR ENVIRONMENT // DD DISP=SHR,DSN=SYS1.RMF.SERBTENU
// DD DISP=SHR,DSN=SYS1.RMF.SERBT // DD DISP=SHR,DSN=SYS1.COMBINED.ISPFTLIB /ISPTABL DD DISP=SHR,DSN=&HLQ..ERBPHDS3.ISPTABLE //SYSTSPRT DD SYSOUT=&CLASS //SYSTSIN DD DUMMY //SYSPRINT DD SYSOUT=&CLASS
The key aspects of the JCL are highlighted in bold:
- The DELETE and ALLOC steps simply delete and recreate the ISPF table
(&HLQ.ERBPHDS3.ISPTABLE) that will hold active exceptions detected by the task. This
is done to ensure that any exceptions from previous runs are deleted.
- The ERBM3B step invokes the TSO environment. The parameters passed are the initial
CLIST to be run (ERBM3B), the High-Level qualifier of the ISPF table for the active
exceptions (&HLQ), and the Report to be run in batch (&REPORT). Since this example
focuses on the Workload/Exceptions report, the default &REPORT value is set to WFEX.
- BONETT.CLIST is a custom dataset that is the first one allocated to SYSPROC. It contains
copies of the RMF CLISTS and REXX execs from the SERBCLS library, so that they can be
modified as needed. Either ERB3RPH3 or ERBR3WFX will have to be modified as
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 13 of 15
described earlier and placed in a custom dataset to get a WTO messages for each
exception3B.
- &HLQ.ERBPHDS3.ISPTABLE is allocated to both ISPTLIB and ISPTABL DD statements.
- BONETT.RMFZOS12.ISPTABLE was the dataset used in the online RMF Monitor III
session in Step 2, and contains the saved exception definitions; this is allocated to ISPTLIB.
- The SERBT and SERTENU datasets are required in the ISPTLIB allocation.
- The COMBINED datasets for this environment contain the required RMF members for that
DD. For use in another environment ensure that all RMF related ISPF datasets are in the
appropriate ISPFxLIB DD concatenation.
When the started task is submitted, the RMF Monitor III session runs in the address space. At the
Monitor III cycle interval, it will output WTO messages for exceptions occurring during the
interval. The following is an example from the SYSLOG of the output produced:
17:38:27.37 BONETT 00000290 S RMFWTO 17:38:27.74 STC09747 00000090 $HASP100 RMFWTO ON STCINRDR 17:38:27.84 STC09747 00000290 IEF695I START RMFWTO WITH JOBNAME RMFWTO IS ASSIGNED TO USER STCRACF 17:38:27.84 STC09747 00000090 $HASP373 RMFWTO STARTED 17:38:27.88 STC09747 00000090 IEF403I RMFWTO - STARTED - TIME=17.38.27 17:38:36.89 STC09747 00000090 +RMF100I 3B: PROCESSING WFEX REPORT... 17:38:36.89 STC09747 00000090 +RMF100I 3B: NAME REASON CRITICAL VAL. 17:38:36.89 STC09747 00000090 +RMF100I 3B: --------------------------------------- 17:38:36.89 STC09747 00000090 +RMF102I 3B: IAM200 DAR > 5 7.900 /SEC 17:38:36.89 STC09747 00000090 +RMF102I 3B: WEBHFD1 NOT AVAIL 17:38:36.89 STC09747 00000090 +RMF102I 3B: WEBHFD2 NOT AVAIL 17:38:36.90 STC09747 00000090 +RMF102I 3B: ZO2RS1 DAR > 5 8.100 /SEC 17:38:36.90 STC09747 00000090 +RMF102I 3B: HSM NOT AVAIL 17:38:36.90 STC09747 00000090 +RMF103I 3B: END OF WFEX REPORT... 17:39:02.50 STC09747 00000090 +RMF100I 3B: PROCESSING WFEX REPORT... 17:39:02.50 STC09747 00000090 +RMF100I 3B: NAME REASON CRITICAL VAL. 17:39:02.50 STC09747 00000090 +RMF100I 3B: --------------------------------------- 17:39:02.51 STC09747 00000090 +RMF102I 3B: WEBHFD1 NOT AVAIL 17:39:02.51 STC09747 00000090 +RMF102I 3B: WEBHFD2 NOT AVAIL 17:39:02.51 STC09747 00000090 +RMF102I 3B: WRK28C DAR > 5 7.500 /SEC 17:39:02.51 STC09747 00000090 +RMF102I 3B: ZO2RS1 DAR > 5 10.50 /SEC 17:39:02.51 STC09747 00000090 +RMF102I 3B: HSM NOT AVAIL 17:39:02.51 STC09747 00000090 +RMF103I 3B: END OF WFEX REPORT... 17:39:32.92 STC09747 00000090 +RMF100I 3B: PROCESSING WFEX REPORT... 17:39:32.92 STC09747 00000090 +RMF100I 3B: NAME REASON CRITICAL VAL. 17:39:32.92 STC09747 00000090 +RMF100I 3B: --------------------------------------- 17:39:32.93 STC09747 00000090 +RMF102I 3B: WEBHFD1 NOT AVAIL 17:39:32.93 STC09747 00000090 +RMF102I 3B: WEBHFD2 NOT AVAIL 17:39:32.93 STC09747 00000090 +RMF102I 3B: HSM NOT AVAIL 17:39:32.93 STC09747 00000090 +RMF103I 3B: END OF WFEX REPORT... 17:40:02.93 STC09747 00000090 +RMF100I 3B: PROCESSING WFEX REPORT... 17:40:02.93 STC09747 00000090 +RMF100I 3B: NAME REASON CRITICAL VAL. 17:40:02.93 STC09747 00000090 +RMF100I 3B: --------------------------------------- 17:40:02.93 STC09747 00000090 +RMF102I 3B: WEBHFD1 NOT AVAIL 17:40:02.94 STC09747 00000090 +RMF102I 3B: WEBHFD2 NOT AVAIL 17:40:02.94 STC09747 00000090 +RMF102I 3B: HSM NOT AVAIL 17:40:02.94 STC09747 00000090 +RMF103I 3B: END OF WFEX REPORT...
Each Monitor III cycle (on this system the cycle time is 30 seconds) produces the WTO lines that
start with the RMF100I message and end with the RMF103I message. Exceptions are shown as
RMF102I messages. In this example Device Activity Rate exceptions occurred for volumes
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 14 of 15
IAM200 and ZO2RS1 in the first cycle and for WRK28C and ZO2RS1 in the second cycle. The
message also shows the actual device activity rate. In subsequent cycles the rate for all volumes
was below the thresholds, so there were no device activity rate exceptions for those cycles. In all
depicted cycles there were availability exceptions (specific monitored address spaces not found).
For the WFEX report, message IDs and message contents can be modified in the ERBR3WFX
REXX procedure. In this example, modifications were made to output the “Processing WFEX
Report…” and “End of WFEX Report…” messages around the exceptions for each cycle.
The message IDs shown in the output above are specific to the customization in the ERBR3WFX
REXX procedure for the example used in this paper. A specific installation should choose
message IDs that are unique for their environment, if using this procedure.
General Observations and Considerations
- Exceptions will be driven based on the Monitor III cycle time. If an exception occurs during
the cycle, it is not reported until the end of the cycle. The smaller the cycle, the closer the
exception message will be reported to the occurrence time of the exception situation.
- The format of the messages is the same as the exception format on the Monitor III display.
Automation products that process these messages will have to understand the message format
and parse the message contents to extract the desired information.
- There is no “clearing message” for exception conditions that return to normal. The exception
simply does not show up in the next cycle. Automation of these messages must take this into
account, and further customization of the ERBR3WFX procedure may be required. For this
example the RMF100I and RMF103I messages were added in the ERBR3WFX exit handler
2 for use by the automation product to identify the start and end of a monitoring cycle. Each
exception automation found in a cycle was saved, and was analyzed to determine if this was
a new or existing exception, or if an exception saved in a previous cycle was still reported.
This method determined if a previously detected exception did not reoccur – which indicated
the condition causing the exception had returned to normal.
Summary
The RMF Monitor III Workload/Exception report provides the ability to define workload and
performance based thresholds that an installation might want to provide notification. By using
the functions provided by RMF, as described in this paper, these exceptions can be sent as WTO
messages, and allow RMF performance indications to be integrated into operational monitoring
and automation efforts. Information from other Monitor III reports can be accessed in a similar
fashion.
© Copyright 2010, IBM Corporation Version 3/8/2010
http://www.ibm.com/support/techdocs
How to Generate WTO Messages from RMF Performance Exceptions Page 15 of 15
Further Reference and Detailed Information
- RMF Newsletter 14, December 19, 2001 (contains a short article on applying this technique
to capture CPC Capacity Report information). It is available at
http://www.ibm.com/servers/eserver/zseries/zos/rmf/rmfhtmls/rmfnws14.html
- RMF Product Manuals - available online at http://www.ibm.com/servers/eserver/zseries/os
(select the operating system, then the library option to locate them):
- RMF Users Guide – details on the RMF Monitor III batch functions (SC33-7990).
- RMF Report Analysis - details on the Workflow/Exception report, defining, exceptions,
and available metrics (SC33-7991).