rmf overview
DESCRIPTION
TSO RMFTRANSCRIPT
-
RMF Development Edition
March, 6th 2010 2010 IBM Corporation
z/OSResource Measurement Facility
RMF Technical Overview
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation2
TrademarksTrademarksTrademarksTrademarks
The following are trademarks of the International Business Machines Corporation in the United States and/or other countries.
The following are trademarks or registered trademarks of other companies.
* Registered trademarks of IBM Corporation
* All other products may be trademarks or registered trademarks of their respective companies.
Java and all Java-related trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countriesLinux is a registered trademark of Linus Torvalds in the United States, other countries, or both.UNIX is a registered trademark of The Open Group in the United States and other countries.Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation.
APPN*CICS*DB2*DB2 ConnectDirMainte-business logo*ECKDEnterprise Storage Server*ESCON*FICON*GDPS*Geographically Dispersed Parallel Sysplex
HiperSocketsHyperSwapIBM*IBM eServerIBM e(logo)server* IBM logo*IMSLanguage Environment*MQSeries*Multiprise*NetView*On demand business logo
OS/390*Parallel Sysplex*PR/SMProcessor Resource/Systems ManagerRACF*Resource LinkRMFS/390*Sysplex Timer*System zSystem z9TotalStorage*Virtualization Engine
VM/ESA*VSE/ESAVTAM*WebSphere*z/Architecturez/OS*z/VM*z/VSEzSeries*z9
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation3
Anything else?Anything else?Anything else?Anything else?
ZZ SessionPostprocessor
Distributed Data Server
Monitor II
SysplexData ServerMonitor III
Spreadsheet Reporter
CIM Data Provider
Mintime & Range
Interval & Cycle Time
% Workflow % Using / Delay
The RMF product consists of several components for all kinds of performance related disciplines. This session introduces the main RMF components, sheds light on their primary purpose and discusses the most common RMF terms
Even though its impossible to become an RMF expert within one hour: lets get started and lets see how far we get!
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation4
AgendaAgendaAgendaAgenda
Product Structure Address Spaces Controlling the Data Gatherers Historical Reporting
Postprocessor Spreadsheet Reporter
Realtime Reporting Monitor III Monitor II WTO Alerts RMF Performance Monitoring Data Portal
RMF Performance Data APIs
The following topics will be covered in this session: What are the basic functions and what is the primary purpose of these functions? What address spaces have to be started? What are their names, underlying procedures and responsibilities? What are the options which determine how the gatheres work in detail? What are the functions for historical reporting and after the facts analysis? What are the functions for realtime reporting and problem determination? Which API's can be user by components or user written programs to retrieve z/OS performance data from RMF
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation5
RMF Distributed Dataserver (DDS)
RMF Product OverviewRMF Product OverviewRMF Product OverviewRMF Product Overview
Windows Windows, Linux
RMF Sysplex Data Server and APIs
RMF PostprocessorHistorical Reporting,Analysis and Planning
RMF Monitor II and IIIReal-Time Reporting,Problem Determination
SMF
RMFData Gatherer RMFMonitor I
RMFMonitor II
background
RMFMonitor III
SMF
VSAMVSAM
RMF Spreadsheet Reporter Monitor III Data Portal & RMF PM
z/OS Resource Measurement Facility (RMF) is an optional priced feature of z/OS. It supports installations in performance analysis, capacity planning, and problem determination. For these disciplines, different kinds of data collectors are needed:
Monitor I long term data collector for all types of resources and workloads. The SMF data collected by Monitor I is mostly used for capacity planning and performance analysis
Monitor II snap shot data collector for address space states and resource usage. A subset of Monitor II data is also displayed by the IBM SDSF product
Monitor III short-term data collector for problem determination, workflow delay monitoring and goal attainment supervision. This data is also used by the RMF PM Java Client and the RMF Monitor III Data Portal
Data collected by all three gatherers can be saved persistently for later reporting (SMF records or Monitor III VSAM datasets)
While Monitor II and Monitor III are realtime reporters, the RMF Postprocessor is the historical reporting function for Monitor I data
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation6
SYS1
.PA
RMLIB
(ERBR
MF00
)
SYS1
.PA
RMLIB
(ERBR
MF04
)
SYS1
.PA
RMLIB
(GPMS
RV00
)
RMF RMF RMF RMF AddressAddressAddressAddress SpacesSpacesSpacesSpaces / / / / ProceduresProceduresProceduresProcedures
SYS1.PROCLIB(RMF)
RMFControls RMF
Monitor I(zz Session)
f RMF,s ZZ
SYS1.PROCLIB(RMFGAT)
RMFMonitor IIIGatherer
f RMF, s III
SYS1.PROCLIB(GPMSERVE)
RMFDistributed
DataServer
s GPMSERVE
SYS1.PROCLIB(RMFM3B)
RMFMonitor III
Batch Reporting
s RMFM3B
In order to make all RMF functions working, a couple of address spaces have to be activated: RMF is the root address space. It is required for any further activities The Monitor I gatherer is a subtask in the RMF address space. It is started immediately together with RMF (by default) or
later by means of the modify command RMFGAT is the Monitor III data gatherer. It cannot be started as standalone address space. Like Monitor I, it can only be
activated with the modify command. Within a sysplex, one instance per system is needed for RMFGAT The RMF Distributed Data Server is the data source for the RMF Performance Monitoring java client and the Monitor III
Data Portal. Only one instance is needed per sysplex RMFM3B is needed for the generation of console messages in case a certain threshold is exceeded
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation7
WRITECALLINSTALL COPY
keeps Configuration Tables and other Control Information provides the Command Interface to set of modify Options:
f RMF,f ZZ,MEMBER(99) maintains the SMF Data Buffer:
s RMF,,,SMFBUF(SPACE(12M),RECTYPE(70:78))SMF
SMF
Write SMF RecordSysplexData Server
RMF
SMFWTM
ComponentSMFBUF
(DataSpace)
SMF ExitsIEFU83x
all SMF Record Types can be maintained by the RMF Sysplex Data Server !
RMF RMF RMF RMF ControlControlControlControl AddressAddressAddressAddress SpaceSpaceSpaceSpace
The RMF control address space has 3 major responsibilities: Builds up configuration tables, reads and remembers the parmlib options Provides the command interface to set options or to start and stop the gathering functions Maintains the SMF Buffer of the RMF Sysplex Data Server
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation8
exact measurement countspick up consecutive counterscalculating the difference at the end of an interval
sampling counts inspect variable counters continuouslybuilding the average at the end of an interval
eg. queue counts, frame counts...
Interval Gathering
Cycle Gathering
eg. CPU seconds, device connect time...
Data Data Data Data GatheringGatheringGatheringGathering MethodsMethodsMethodsMethods
In order to collect all the information that is displayed in the various reports, RMF uses two different techniques of data gathering
The preferred and cheap method is to retrieve exact measurement counts: this can be timers (e.g. dispatch times) or accumalitve counts like the Start Subchannel count for a device
The costly cycling technique is needed for counters which are changing rapidly and frequently: how long is the dispatcher queue? how many I/O queue elements are chained to a UCB? what is the actual enqueue situation?
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation9
1. Measurements CACHE SMF 74.5 CHANNEL SMF 73 CPU SMF 70.1 CRYPTO SMF 70.2 DEVICE SMF 74.1 ENQ SMF 77 IOQ SMF 78.3 FCD SMF 74.7 ESS SMF 74.8 PAGESP SMF 75 PAGING SMF 71 TRACE SMF 76 VSTOR SMF 78.2 WKLD SMF 72.3
2. Timing CYCLE(1000) NOSTOP SYNC(SMF)
3. Reporting / Recording RECORD REPORT(REALTIME) SYSOUT(A)
4. User Exits NOEXITS
SMFSMF 74.2 XCFSMF 74.3 OMVSSMF 74.4 CFSMF 74.6 HFS
gathered byMonitor III
Monitor I Data Monitor I Data Monitor I Data Monitor I Data GatheringGatheringGatheringGathering
SYS1.PARMLIB
(ERBRMF00)
The directives to setup Monitor I data gathering are kept in SYS1.PARMLIB(ERBRMF00) There are four different categories of control statements:
Measurements: specifies the kind of resources, which are the candidates for data gathering The collected data can be saved to SMF records for later reporting A subset of SMF records is gathered by Monitor III: since the same data is needed for Monitor III reports,
duplicate gathering is avoided Timing: determines the frequency for interval gathering and cycle gathering Reporting/Recording: defines the type and the creation time of the output User Exits: specifies routines which can be executed during session processing to gather or report additional data:
ERBMFIUC (initialization) ERBMFxxx (cycle gatherer, loaded by ERBMFIUC) ERBMFDUC (interval gatherer) ERBMFRUR (report writer) ERBMFTUR (termination)
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation10
1. Measurements IOSUB CFDETAIL CACHE VSAMRLS OPD HFSNAME zFS SGSPACE
2. Timing CYCLE(1000) MINTIME(60) NOSTOP SYNC(00)
3. Recording DATASET(ADD(RMF.M3G.&SYSNAME..DS1)) DATASET(ADD(RMF.M3G.&SYSNAME..DS2)) DATASET(START) DATASET(NOSWITCH) DATASET(WHOLD(7))
4. Buffer WSTOR(32)
ERBVSDEF vsam_ds VSAMVOL(volser)DATASET(ADD(vsam_ds)) DATASET(DEL(vsam_ds))
ERBV2S vsam_ds seq_ds
ERBS2V seq_ds vsam_ds
RMFGAT Address Space
WSTOR(32)DATASET(WHOLD(7))
In Storage Buffer
Monitor III Data Monitor III Data Monitor III Data Monitor III Data GatheringGatheringGatheringGathering
SYS1.PARMLIB
(ERBRMF04)
The directives to setup Monitor III data gathering are kept in SYS1.PARMLIB(ERBRMF04) Only a few measurements can be turned ON or OFF. The data collection cannot be controlled on resource granularity:
almost all resources must be monitored in order to get the complete picture of the address spaces which are using or delayed for certain resources. However, a subset of gathering activities which does not interfere with address space states can be switched on or off
In order to extend the report range for historical data, Monitor III VSAM Datasets can be used. In this case, the buffersizewhich has been specified by the WSTOR option can be reduced to the WHOLD size (since the data which does not fit anymore in the buffer is kept at least in the datasets)
For a smart handling of the VSAM datasets, the following procedures are provided: ERBVSDEF defines a Monitor III VSAM dataset ERBV2S unloads the VSAM data to a sequential dataset ERBS2V reconverts the sequential dataset to VSAM
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation11
has Sysplex-wide scope exploited by RMF PM, Monitor III Data Portal,
CIM Provider talks XML over HTTP no configuration needed! just enter S GPMSERVE or F RMF,DDS on one
image of the Sysplex
RMF M III Data Gatherer
RMF Sysplex Data Server (ERB3XDRS Service)
RMF M III Data Gatherer
RMF M III Data Gatherer
GPMSERVE(RMF Distributed Data Server)
TCP/IP
RMF RMF RMF RMF DistributedDistributedDistributedDistributed Data ServerData ServerData ServerData Server
The RMF Distributed Data Server (DDS) is the component, which serves as data source for the RMF client applications There is just one additional address space (GPMSERVE) needed per Sysplex. The DDS uses the ERB3XDRS service to
retrieve Sysplex-wide Monitor III data. In a mixed release configuration, the DDS should run on system with the highest RMF release
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation12
Activities measured by Monitor ICache SubsystemChannel Path CPU / CryptoDevice EnqueueFicon Director I/O Queuing Page/Swap Data SetPaging Trace Virtual Storage Workload
Activities measured by Monitor IIIXCFOMVSCoupling FacilityHFS
Other ActivitiesLotus Domino ServerHTTP Server
SMF
RM
F Po
stpr
oce
sso
r
RMF Data Buffer
Spre
ads
hee
t App
lica
tion
sLo
tus
1-2-
3 a
nd
Exce
l
RM
F Sp
read
she
et R
epor
ter
HistoricalHistoricalHistoricalHistorical ReportingReportingReportingReporting
The tool of choice for historical reporting and after the facts analysis is the RMF Postprocessor Accordingly to the Monitor I data collection options (there is one SMF record type for each resource), the reporting structure
is resource oriented (exception: the WLMGL record (SMF 72.3) holds performance data and resource consumption on workload granularity)
Some SMF record types (74.x) are written by the Montior III data gatherer address space RMFGAT. However, this address space should be active anyway to guarantee access to the Monitor III reports
With OS/390 R10 the Postprocessor has been enabled to accept foreign SMF records: Type 103 HTTP Server Type 108 Domino Server
While the RMF Postprocessor generates tabular reports, the RMF Spreadsheet Reporter can create graphical charts based on Postprocessor data
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation13
//SMFDUMP EXEC PGM=IFASMFDP//IDD1 DD DISP=SHR,DSN=//IDD2 DD DISP=SHR,DSN=//SMFDATA DD DISP=(NEW,PASS),SPACE=(CYL,(10,10),RLSE),// UNIT=SYSDA,DCB=(RECFM=VBS,LRECL=32760,BLKSIZE=0)//SYSIN DD *INDD(IDD1,OPTIONS(DUMP))INDD(IDD2,OPTIONS(DUMP))OUTDD(SMFDATA,TYPE(70:78))
SMF data is kept in VSAM datasets Postprocessor requires sequential format use SMF dump utility IFASMFDP to unload the data usuallly GDGs are the prefered target:
RMF.SMFDATA.SYSNAME(0) RMF.SMFDATA.SYSNAME(-1)
//RMFSORT EXEC PGM=SORT//SORTIN DD DISP=SHR,DSN=// DD DISP=SHR,DSN=//SYSIN DD *SORT FIELDS=(11,4,CH,A,7,4,CH,A),EQUALSMODS E15=(ERBPPE15,36000,,N),E35=(ERBPPE35,3000,,N)
SMF reords must be sorted by date and time SORT step is required for sysplex-wide reporting
Workload Activity Report Coupling Facility Report Shared DASD Report
PostprocessorPostprocessorPostprocessorPostprocessor: : : : PreparingPreparingPreparingPreparing SMF DataSMF DataSMF DataSMF Data
Before the raw SMF data can be feeded to the Postprocessor, two preparation steps are needed: The data has to be transformed from VSAM to Sequential format
This is achieved with the SMF Dump Utility IFASMFDP In most installations, the SMF data is dumped by default to Generation Data Groups (GDGs)
For Sysplex-wide reporting, the records arriving from the various systems must be sorted by ascending interval time Therefore, a preceeding SORT step must be executed RMF provides the exit routines ERBPPE15 and ERBPPE35 to enable the sorting by RMF interval start time
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation14
DD Names MFPMSGDS Message Output MFPINPUT SMF Input Datasets PPRPTS combined Interval Reports PPSUMnnn Summary Report Output PPXSRPTS Sysplex Report Output PPORPnnn Overview Report Output PPOVWREC Overview Record Output
Control Statements DATE Start / End Date RTOD Start / End Time DINTV Duration Interval Length REPORTS Report Types OVERVIEW Report or Record OVW Overview Control Statement
//RMFPP EXEC PGM=ERBRMFPP//MFPMSGDS DD DISP=SHR,DSN=*.ALLOC.MSG//MFPINPUT DD DISP=(OLD,DELETE),DSN=*.RMFSORT.SORTOUT//PPRPTS DD DISP=SHR,DSN=RMF.INTERVAL.REPORTS//PXSRPTS DD DISP=SHR,DSN=RMF.SYSPLEX.REPORTS//SYSIN DD *DATE(04212003,04232003)RTOD(0000,2400)DINTV(0100)REPORTS(ALL)
JCL can be generated by ISPF Application orSpreadsheet Reporter
PostprocessorPostprocessorPostprocessorPostprocessor JCLJCLJCLJCL
Different DD names and control statements must be specified for the different kinds of report output However, the user doesn't need to take care about any syntax: the JCL to execute the Postprocessor can be generated with
an ISPF dialog or with the RMF Spreadsheet Reporter workstation frontend
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation15
PostprocessorPostprocessorPostprocessorPostprocessor: Standard Reporting: Standard Reporting: Standard Reporting: Standard Reporting
//RMFPP EXEC PGM=ERBRMFPP//SYSIN DD * DATE(10142002,10142002)RTOD(1100,1300) REPORTS(CPU) SYSRPTS(WLMGL(SCPER))SYSOUT(H)
C P U A C T I V I T Y
z/OS V1R8 SYSTEM ID SYSD DATE 02/13/2007 INTERVAL 15.00.048RPT VERSION V1R8 RMF TIME 11.00.00 CYCLE 1.000 SECONDS
CPU 2084 MODEL 314 H/W MODEL B16
---CPU--- ONLINE TIME LPAR BUSY MVS BUSY CPU SERIAL I/O TOTAL % I/O INTERRUPTSNUM TYPE PERCENTAGE TIME PERC TIME PERC NUMBER INTERRUPT RATE HANDLED VIA TPI0 CP 100.00 3.23 3.61 066F7A 14.60 0.691 CP 100.00 2.77 3.18 066F7A 22.53 0.492 CP 100.00 2.58 2.93 066F7A 25.61 0.433 CP 100.00 2.54 2.97 066F7A 29.42 0.46CP TOTAL/AVERAGE 2.78 3.17 92.16 0.50
W O R K L O A D A C T I V I T Y
z/OS V1R8 SYSPLEX SYSDPLEX START 02/13/2007-11.00.00 INTERVAL 000.15.00 MODE = GOAL RPT VERSION V1R8 RMF END 02/13/2007-11.15.00
POLICY ACTIVATION DATE/TIME 02/12/2007 08.43.05
------------------------------------------------------------------------------------------------------------ SERVICE CLASS PERIODS
REPORT BY: POLICY=DEFAULT WORKLOAD=SYSTEM SERVICE CLASS=SYSTEM RESOURCE GROUP=*NONE PERIOD=1 IMPORTANCE=SYSTEM CRITICAL =NONE
-TRANSACTIONS- TRANS-TIME HHH.MM.SS.TTT --DASD I/O-- ---SERVICE---- --SERVICE TIMES-- ---APPL %--- -----STORAGE-----AVG 23.36 ACTUAL 0 SSCHRT 30.3 IOC 15835 CPU 26.498 CP 3.86 AVG 2871.20 MPL 23.36 EXECUTION 0 RESP 1.1 CPU 5183K SRB 7.913 AAPCP 0.00 TOTAL 67066.09 ENDED 0 QUEUED 0 CONN 0.6 MSO 0 RCT 0.017 IIPCP 0.00 SHARED 11.00 END/S 0.00 R/S AFFIN 0 DISC 0.0 SRB 1548K IIT 0.275 #SWAPS 106 INELIGIBLE 0 Q+PEND 0.3 TOT 6747K HST 0.000 AAP 0.00 --PAGE-IN RATES--EXCTD 0 CONVERSION 0 IOSQ 0.1 /SEC 7496 AAP 0.000 IIP N/A SINGLE 0.0 AVG ENC 0.00 STD DEV 0 IIP N/A BLOCK 0.0 REM ENC 0.00 ABSRPTN 321 SHARED 0.0 MS ENC 0.00 TRX SERV 321 PROMOTED 0.000 HSP 0.0
In most cases, the RMF Postprocessor is used to produce tabular reports (one per interval). The chart shows an example for a single system report and for a Sysplex-wide report: The CPU Activity Report shows the processor utilization from the systems perspective (on logical processor level) The Workload Activity Report shows Sysplex-wide transaction and resource consumption statistics (on service class period
level)
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation16
R M F O V E R V I E W R E P O R T
z/OS V1R8 SYSTEM ID SYSF START 08/07/2006-11.00.00 INTERVAL 01.00.00 RPT VERSION V1R8 RMF END 08/07/2006-15.00.00 CYCLE 1.000 SECONDS
NUMBER OF INTERVALS 4 TOTAL LENGTH OF INTERVALS 04.00.00
DATE TIME INT PROCS CPUBSY APPL TRANS RTIME MM/DD HH.MM.SS HH.MM.SS08/07 11.00.00 01.00.00 5.0 9.2 33.9 16.8 40.5608/07 12.00.00 01.00.00 5.0 8.5 33.0 28.1 35.9208/07 11.00.00 01.00.00 5.0 8.3 30.6 38.9 25.6608/07 12.00.00 01.00.00 5.0 8.8 24.5 24.5 44.99
PostprocessorPostprocessorPostprocessorPostprocessor: : : : OverviewOverviewOverviewOverview ReportingReportingReportingReporting
Format of the INTERVAL column changes from MM.SS to HH.MM.SS
OVERVIEW(RECORD,REPORT) OVW(PROCS(NUMPROC)) OVW(CPUBSY(CPUBSY)) OVW(APPL(APPLPER(POLICY)))OVW(TRANS(TRANS(POLICY))) OVW(RTIME(RTIME(POLICY)))DINTV(0100)
(via Spreadsheet Reporter)
When the user wants to focus on specific metrics without analyzing an entire report, the Postprocessor Overview feature provides a smart solution: by means of overview control statements, he can extract specific values and produce his own customized reports
Multiple intervals are visible at a glance: Thus, Overview reporting is well suited for trend analysis over long periods For all SMF 70-78 record types, the user can choose from a rich set of Overview Control statements. For more details, refer
to the RMF Users Guide (SC33-7990), Chapter 17, Overview and Exception Conditions
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation17
converts SMF Data to Spreadsheet Format creates graphical Views for Trend Analysis can be downloaded from the RMF Homepage
RMF Spreadsheet ReporterRMF Spreadsheet ReporterRMF Spreadsheet ReporterRMF Spreadsheet Reporter
In extension to the RMF Postprocessor, RMF offers a graphical workstation frontend for historical reporting: the RMF Spreadsheet Reporter
The visualization of the data is performed by spreadsheet macros, which are shipped as samples together with the Spreadsheet Reporter main application.
The Spreadsheet Reporter is a basic part of the RMF product and doesnt require additional charges. The installable image is shipped with the host file SYS1.SERBPWS(ERB9R2SW) or the most recent version can be downloaded from the RMF homepage: http://www.ibm.com/servers/eserver/zseries/zos/rmf/. You will find the Spreadsheet Reporter in the Toolscategory of the homepage.
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation18
RMF Spreadsheet ReporterRMF Spreadsheet ReporterRMF Spreadsheet ReporterRMF Spreadsheet Reporter
SMF raw Data
Working Sets
Spreadsheet
Overview Records
*.WK1 Files
*.OVW Files
Report Listings Overview Records
Report Listings
*.RPT FilesCPU Contention: System SYS1
RMF Postprocessor
Download
Extract
Convert
Generate
Display
The transition from the SMF raw data to graphical charts is a complex process with multiple steps. The good news - the RMF Spreadsheet Reporter does it all for you by magic: Generates Report Listings or Overview Records from the SMF data with the RMF Postprocessor: on the Resource
notebook page of the Spreadsheet Reporter, specify as input the SMF dump data sets from all the systems of your sysplex. Then check the desired reports on the Reports selection dialog. Now you are ready to exploit the Spreadsheet Reporters remote job execution engine, which allows you to run the Postprocessor from your workstation!
Downloads the Postprocessor output to your workstation Extracts the reports or records Convert the reports or records to spreadsheet format and generates the Working Set
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation19
RMF Spreadsheet ReporterRMF Spreadsheet ReporterRMF Spreadsheet ReporterRMF Spreadsheet Reporter
Macrosfor all Types of
Performance relevantAreas !!
Once a Working Set has been created from SMF data, the user can select from a rich set of macros for all performance relevant areas
Three basic types of macros can be distinguished: Macros based on data contained in one Postprocessor Report (for detailed analysis of one single interval) Macros based on data contained in Overview Records (for trend analysis with multiple intervals) Macros based on data contained in multiple Postprocessor Reports (hybrid macros)
The new XCF Trend Report macro belongs to the family of hybrid macros
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation20
covers all Sysplex related aspects two monitors and a workstation extension
Monitor III, best suited for short-term, real-time and historical reporting online performance analysis goal attainment supervision sysplex-wide and single-system reporting monitoring of exceptional conditions
Monitor II, best suited for snapshot reporting single job and resource monitoring
RMF PM / Data Portal enterprise-wide reporting of z/OS systems based on RMF Monitor III data
CF
WLM
RealtimeRealtimeRealtimeRealtime ReportingReportingReportingReporting
RMF offers three different Monitors for realtime reporting: Monitor III Reporter (ISPF):
Is designed for workflow analysis and goal attainment supervision Can be used in realtime mode, but supports historical reporting as well
Monitor II (ISPF and native TSO): Is a snapshot monitor. The interval length can be determined by the user Consists of a set of address space and resource oriented reports
RMF Performance Monitoring / Monitor III Data Portal: Allows to manage multiple Sysplexes from the workstation The raw data is identical with Monitor III ISPF, but the GUI is totally different
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation21
Monitor III Delay Monitoring Processor Storage Device Enqueue Operator
Message Tape Mount
SubsystemHSM - JES - XCF
Monitor III Activity Monitoring Common Storage Page/Swap Data Sets Storage Frames Device Data Set Level by Job and Volume Cache Coupling Facility Goal Attainment VSAM RLS UNIX System Services Enclaves zFS Diskspace Spin/Suspend Locks
Monitor III Features Cursor-Sensitive Navigation Workflow/Exceptions Monitoring Automatic Customization Support of WTO Messages Continuous Monitoring Hardcopy Reports On-Line Tutorial On-Line Help Adaptive Reports User Reports Sysplex-wide Reports Remote Reporting
Monitor III ReportingMonitor III ReportingMonitor III ReportingMonitor III Reporting
The Monitor III ISPF Reporter provides two basic types of monitoring: WorkloadDelay monitoring
The state samplers are inspecting all hard- and software resources and assign those resources on address space level
Possible states for an address space are: Delayed for Processor, Storage, Device, Enqueue, Operator, Subsystem Using Processor or Device Idle (timer wait or terminal input wait) Unknown (e.g. using an unmonitored waiting mechanism)
ActivityResource Monitoring In addition to the workloaddelay concept, there are lots of resource oriented reports with similar statistics than the
RMF Postprocessor reports Together with hardware or software related innovations new reports have been added with each z/OS release
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation22
States of a JobStates of a JobStates of a JobStates of a Job
delay samplesnumber of samples 100Delay(%) =
using samplesnumber of samples 100Using(%) =
Workflow (%) = using samplesusing samples delay samples+ 100
Unknown
?
Proc DevDevIdle
JESHSM
XCF
Proc
Enq
Stor
Dsn Dsn
Oper
= 50%
= 33%
= 60%
Address spaces are either using resources or they are delayed for resources or they are idle The RMF state samplers are trying to monitor continuously all kinds of hard- and software resources Depending on the result, all address spaces can be associated with the following qualities:
Using% Delay% Workflow%
Since not all address spaces can be catched up every cycle, the remaining states are counted as unknown. Possible reasons for unknown are:
The address space is delayed for unmonitored resource (other than DASD or Tape) The address space is using an unmonitored waiting mechanism (most Started Tasks!) statistical reasons
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation23
Example: Using and DelayExample: Using and DelayExample: Using and DelayExample: Using and Delay
CA B
D
B
C
A
D
A
B
C
D
CA D
BRes Res Res Res
Job I U D Why A * C B * C C * D *
Job I U D Why A * B B * C * D *
Job I U D Why A * B * C * A D *
Job I U D Why A * C B * C * D * C
RMF Monitor III Delay Report
Samples: 4 Time: 06.28.20 Range: 4 Sec
WFL USG DLY IDL Primary Jobname % % % % Reason
A 25 25 75 0 CB 50 25 25 50 CC 66 50 25 25 AD 0 0 25 75 C
RMF Monitor III Resource Delay Report
Samples: 4 Time: 06.28.20 Range: 4 Sec
WFL ADU Jobname USG DLY Reason Resource % % %
Res 40 1.5 A 25 75 CB 25 25 CC 50 25 AD 0 25 C
The example helps to understand what is calculated and reported by RMF Monitor III From the jobs perspective:
Job A 1x Using(25%) 3x Delayed(75%) 0x Idle(0%) Workflow: 1/4 = 25% Job B 1x Using(25%) 1x Delayed(25%) 2x Idle(50%) Workflow: 1/2 = 50% Job C 2x Using(50%) 1x Delayed(25%) 1x Idle(25%) Workflow: 2/3 = 66% Job D 0x Using(0%) 1x Delayed(25%) 3x Idle(75%) Workflow: 0/1 = 0%
From the resource perspective: 4x using, 6x delay Workflow = 4/10 = 40 % Average Delayed User (ADU): 6/4 = 1.5
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation24
RMF V1R7 Delay Report Line 1 of 326
Samples: 100 System: AQTS Date: 07/18/06 Time: 06.28.20 Range: 100 Sec
Service WFL USG DLY IDL UKN ---- % Delayed for ---- Primary Name CX Class Cr % % % % % PRC DEV STR SUB OPR ENQ Reason
SUSANK T TSOPRIME 0 0 100 0 0 0 0 0 100 0 0 HSM CONSOLE S SYSTEM 0 0 15 0 85 0 0 0 0 15 0 Message RRSSERVQ B WLMSHORT 0 0 1 0 1 0 0 0 1 0 0 JES BHBE T TSOPRIME 40 4 6 90 0 6 0 0 0 0 0 JHUGO MORABIT T TSOPRIME 41 37 56 0 7 0 56 0 0 0 0 COMPK5 RONDA2A B COMBUILD 42 29 42 0 29 1 41 0 0 0 0 COMPK5 D24JAP1 T TSOPRIME 49 22 23 56 0 0 23 0 0 0 0 PRIPK5 RRSSERVQ B WLMSHORT 50 2 2 0 0 0 2 0 0 0 0 SPOL1J GRSARTSQ B WLMSHORT 50 1 1 0 0 0 0 0 1 0 0 JES RRSSERVQ B WLMSHORT 50 1 1 0 0 0 0 0 1 0 0 JES CATALOG S SYSTEM 63 57 39 0 12 0 39 0 0 0 0 MCATTS ANFWPROC SO SYSSTC 67 2 1 0 2 0 1 0 0 0 0 SPOL1J GRSARTSQ B WLMSHORT 71 5 2 0 0 0 1 0 1 0 0 SPOL1J SMFDRS S STCMED 71 5 2 0 93 0 0 0 0 0 2 SYSZVVDS JES2 S SYSSTC 73 16 6 0 79 0 6 0 0 0 0 SPOL1J GRSARTSQ B WLMSHORT 80 4 1 0 0 0 0 0 1 0 0 JES ARTXESQ B WLMSHORT 80 4 1 0 0 0 1 0 0 0 0 SPOL1L DFRMM S SYSSTC 83 81 18 0 1 0 18 0 0 0 0 SL3061
Address SpacePerformance at a Glance !
sorted by ascending Workflow Delay Type Breakdown Delay Reason Information
Monitor III: Job Monitor III: Job Monitor III: Job Monitor III: Job DelaysDelaysDelaysDelays
The Address Space Delay report is the basic report that visualizes the overall Monitor III concept The report is sorted by ascending workflow: thus, the critical candidates with low workflow appear on top of the address
space list Beyond the basic statistics (workflow, using, delay, idle, unknown), a delay type breakdown for the following resources is
provided: PRC processor DEV device STR storage SUB software subsystem OPR operator ENQ enqueue
Additionally, the delay reason information is displayed in the rightmost column: For DEV the critical volume For PRC the competing address space with the highest usage For SUB the subsystem name For ENQ the enqueue resource name
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation25
RMF V1R7 Workflow/Exceptions Line 1 of 32
Samples: 100 System: AQTS Date: 07/18/02 Time: 06.28.20 Range: 100 Sec
--------------------------- Speed (Workflow) ---------------------------------Speed of 100 = Maximum, 0 = Stopped Average CPU Util: 81 %
Name Users Active Speed Name Users Active Speed *SYSTEM 560 35 42 *DEV 53 15 46ALL TSO 19 4 17 *MASTER* 1 1 100ALL STC 485 6 52ALL BATCH 39 24 37ALL ASCH Not availALL OMVS 17 1 92*PROC 292 8 43
------------------------------ Exceptions -------------------------------------Name Reason Critical val. Possible cause or action*SLIP* SLIP PER TRAP SLIP ID=U427 is active.ALL TSO DEV -RMFLIB 1.5 users Short block size or excessive seeks.*SYSTEM PROC-SUSANK 12.1 usersBHBE SUBS-JES 100.0 % delay Processing TSO OUTPUT command request.MORABIT PROC-SUSANK 75.0 % delayGANDALF DEV -SMS001 20.0 % delay Device may be overloaded.CONSOLE OPER-Message 13.0 % delay Awaiting reply to operator request 52.HSM Not avail Job HSM is not running.SYSPAG Not avail Volume SYSPAG is not mounted.
Bottleneck Detectionat a Glance !
Workflow for Groups and Resources
Exception Lines for individual Conditions
Monitor III: Monitor III: Monitor III: Monitor III: Workflow/ExceptionsWorkflow/ExceptionsWorkflow/ExceptionsWorkflow/Exceptions
The Monitor III Workflow Exception Report consists of two independent sections: The Workflow section provides an overview about groups of work, individual address spaces or resources The Exceptions section displays alerts when critical thresholds are exceeded
The content of both sections can be customized individually: For the Workflow section, up to 14 candidates for continuous display of workflow can be selected For the Exceptions section, alerts for almost all Monitor III metrics (as well as combination of metrics) can be defined
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation26
RMF 2.10.0 Sysplex Summary - SCLMPLEX Line 1 of 14 Command ===> Scroll ===> CSR
WLM Samples: 240 Systems: 3 Date: 05/15/02 Time: 13.00.00 Range: 60 Sec
Service Definition: SCLM Installed at: 12/06/00, 10.07.24Active Policy: STANDARD Activated at: 12/06/00, 10.07.33
------- Goals versus Actuals -------- Trans --Avg. Resp. Time-Exec Vel --- Response Time --- Perf Ended WAIT EXECUT ACTUAL
Name T I Goal Act ---Goal--- --Actual-- Indx Rate Time Time Time
STC W 88 0.000 0.000 0.000 0.000STCCMD S 3 40 88 0.46 0.000 0.000 0.000 0.000SYSTEM W 69 0.000 0.000 0.000 0.000SYSSTC S N/A 68 N/A 0.000 0.000 0.000 0.000SYSTEM S N/A 70 N/A 0.000 0.000 0.000 0.000TSO W 84 2.100 0.000 0.608 0.608PRDTSO S 84 2.100 0.000 0.608 0.608
1 1 60 1.000 AVG 0.080 AVG 0.08 1.150 0.000 0.080 0.0802 1 0.0 1.500 AVG 0.109 AVG 0.07 0.567 0.000 0.109 0.1093 1 85 2.000 AVG 2.928 AVG 1.46 0.383 0.000 2.928 2.928
MASTER R N/A 47 N/A 0.000 0.000 0.000 0.000
Sysplex Performance at a Glance ! 80 Intervals in GO Mode colored Indication for PI > 1Importance = 1+2Importance > 2
Monitor III: Goal Monitor III: Goal Monitor III: Goal Monitor III: Goal AttainmentAttainmentAttainmentAttainment
The Sysplex Summary Report is the standard report for goal attainment supervision The report provides Goals vs. Actuals on Service Class Period level. In case the Performance Index (PI) for a certain
Service Class Period is higher than 1, coloured lines are generated: A red line for Service Class Periods with high Importance (1, 2) A yellow line for Service Class Periods with low Importance (3, 4, 5)
In GO Mode, a coloured bar shows the state for the last 80 intervals: when at least one line for the current interval becomes red or yellow, a red or yellow square will be added. Otherwise the bar is extended with a green square. Thus, the Sysplex performance is visible at a glance for an 80-minute period (proposed a 60 second reporting range)
For Service Classes with response time goals, the report provides a direct link to Response Time Distribution Report by means of cursor sensitivity
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation27
Monitor III: Session SetupMonitor III: Session SetupMonitor III: Session SetupMonitor III: Session Setup
SYS1.SERBCLS(ERBRMF3X)
TSO Address Space
Monitor III Reporter
In Storage Buffer
RMFGAT
Type text
Type text Type text
VSAM Datasets
ERBTLIB&USER.&REL.ISPTABLESYS1.SERBTENUSYS1.SERBT
&USER.&REL.ADMGDF
alloc f(rmfds00) da(rmf.m3.sys1.data) alloc f(rmfds01) da(rmf.m3.sys2.data)
VSAM Archives
The options for a Monitor III reporter session are stored in tables (e.g exception definitions). The initialization procedure ERBRMF3X performs the table allocations when a reporter session is going to be started
By default, a reporter session accesses directly the actual data collected by the RMFGAT (via Cross Memory) However, it is also possible to establish a standalone reporter session independent from the Monitor III gatherer: in order to
analyze historical data, up to 99 VSAM datasets can be preallocated (DDNAME = RMFDSxx)
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation28
Monitor III Reporter UsageMonitor III Reporter UsageMonitor III Reporter UsageMonitor III Reporter Usage
System InformationSamples: 100 System: AQTS
Group WFL User ...*SYSTEM 30 250 ...*TSO 20 22 ...DMN001 28 16 ...PG002 20 22 ...
HelpReportOptions
SessionOptions
Tutorial
M IIIMain Menu
RMFMain Menu
ROSessionColoretc.
Tutorial
3Report Selections,Short Commands
PostprocessorSelection Menu
Monitor II
1 2
RMF
F3
F1
F3
F3
Enter
Enter
Scroll
BREF/FREF
Find, Color, ICU, Current, GO, and more ...
F10 F11
F7 F8
The Monitor III reporter session can be invoked directly from the RMF Main Menu Various session options (SO command) can be defined and stored persistently, for example:
initial mode (GO or STOP) initial screen range length
By means of report options (RO command), the content of almost all reports can be customized A very detailed tutorial provides lots of information about the Monitor III concepts and features Cursor sensitive HELP (including detailed metric descriptions) is always available via the PF1 key
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation29
Delay Reports
STORCR
STORC
STORF
CHANNEL
IOQUEUE
Resource Utilization Reports
PROC DEV
ENQ
STOR STORS
Job Reports
Job/Workload oriented
JOBDetailed Job Analysis
User Reports
RG
MSI
SYSTREND
DEVN
DEVT
DSD
STORCR
PROC
Modified panelUser-written reportSortable report
DSND DSNV
ENQR
HSM JES
XCF
Resource oriented
DEVR
STORR
SYSRTD
SYSWKM
Sysplex Reports Other
SYSENQ
DSINDEXCACHDET
CACHSUM
Cache Subsystem
CFOVER
CFACT
CFSYS
Coupling Facility
SYSINFO GROUP
System/Workload Reports
CPC
RLSSC
RLSDS
RLSLRU
VSAM RLS
SYSSUM
WFEX DELAY
Monitoring Reports ENCLAVE
OPD
Monitor III Report Monitor III Report Monitor III Report Monitor III Report OverviewOverviewOverviewOverview
Monitor III offers a complete report suite which covers all performance relevant areas. The reports can be grouped by categories:
A family of reports with sysplex wide scope System level reports for various kinds of measurements Job/address space oriented reports Reports from the perspective of the major hard-and software resources User reports which can be adapted or customized
In order to improve the Monitor III capabilities, new reports have been added frequently, e.g. OS/390 R4 DSND DSNV OS/390 R6 CFSYS CFOVER CFACT OS/390 R7 CACHESUM CACHDET ENCLAVE OS/390 R10 RLSSC RLSDS RLSLRU OPD z/OS V1R2 CPC z/OS V1R7 ZFSSUM ZFSACT SPACEG SPACED z/OS V1R8 XCFOVW XCFSYS XCFPATH XCFGROUP (Data Portal only)
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation30
Monitor III: Setup Monitor III: Setup Monitor III: Setup Monitor III: Setup forforforfor WTOWTOWTOWTOssss
Monitor IIIBatch
Reporter
ExitModule
Monitor III Batch Address Space creates Reporting Tables Thresholds can be defined via
Workflow Exception Options Dialog Reporter Phase Exit Module
Console Message is generated by Exit Module (e.g. WLM Capping)
$HASP100 BMAI ON TSOINRDR$HASP373 BMAI STARTEDIEF125I BMAI - LOGGED ON - TIME=12.36.20+RMF300I 3B: Processing CPC Report...+RMF301I 3B: Local Partition Capping State:+RMF303I 3B: Time until Capping (sec): 40 (WTO Limit: 600)+RMF304I 3B: MSU Consumption of critical LPARs:+RMF305I 3B: SYS1 : 64 (WTO Limit: 60)+RMF305I 3B: SYS4 : 48 (WTO Limit: 32)IEF126I BMAI - LOGGED OFF - TIME=12.38.00$HASP395 BMAI ENDED$HASP250 BMAI PURGED+RMF300I 3B: Processing CPC Report...+RMF301I 3B: Local Partition Capping State:+RMF302I 3B: WLM Capping %: 24.2 (WTO Limit: 10.0)+RMF304I 3B: MSU Consumption of critical LPARs:+RMF305I 3B: SYS1 : 82 (WTO Limit: 60)+RMF305I 3B: SYS2 : 12 (WTO Limit: 10)$HASP100 BMGU ON TSOINRDR$HASP373 BMGU STARTEDIEF125I BMGU - LOGGED ON - TIME=12.39.40
WTO
Sample Exits provided for: Workflow Exception Report Sysinfo Report CPC Capacity Report
Monitor IIIGatherer
Set ofSamples
The procedure RMFM3B is supplied to run a Monitor III reporter session in batch mode. This environment is well suited, when a threshold is exceeded and a corresponding message should be displayed on the console
For threshold processing, the user can select from all metrics contained in one of the Monitor III reports: the procedure accepts any report name as input parameter. In other words, the specified report is the one and only report that is produced by the batch reporter session (multiple reports are not supported)
The scanning of the report tables and the comparison against thresholds is performed by exit routines. Depending on the result of this comparison, a WTO can be generated
RMF ships the following REXX exit routines as examples (in SYS1.SERBCLS): ERBR3WFX processes the WFEX report ERBR3SYS processes the SYSINFO report ERBR3CPC processes the CPC report
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation31
Monitor II: Monitor II: Monitor II: Monitor II: OverviewOverviewOverviewOverview
Postprocessor
SMF 79.xf rmf,s aa
SMF 79.x10:12:08 DEVJOBNAME CONN*MASTER* 2618PCAUTH 1264
Monitor II is a Snapshot Reporter collects the status of system resources (CPU, devices, paging activity, ...) collects the status of address spaces (resource usage, state information)
use Monitor II to continuously monitor resource usage determine the state of any address space in the system track CPU usage of problem address spaces collect supplemental information when analyzing performance problems with Monitor III
choose Background Session to collect SMF records for archiving and later postprocessing to automate snapshot reporting
choose Display Session for immediate feedback for online analysis
Monitor II complements the host based RMF reporting suite. It is the RMF snapshot reporting feature Three basic groups of reports can be distinguished:
Address space oriented reports (e.g. ARD, ASD, ASRM) I/O subsystem related reports (e.g. DEV, CHANNEL) Resource reports (e.g. SENQ, SPAG)
Monitor II is the appropriate function, when immediate feedback about address spaces or resources is required However, by means of a background session, Monitor II can also produce SMF Type 79 records. These records can be
feeded to the RMF Postprocessor for after the facts analysis
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation32
Activities measured by Monitor II: Address Space Data
Resource StateSRM
Channel Path Device I/O Queuing Enqueue HFS IRLM Long Locks Paging Page/Swap Data Set SRM Resource Data Sysplex Data Server Library Display
Monitor II ReportingMonitor II ReportingMonitor II ReportingMonitor II Reporting
for most comfortable usage supports sorting and finding started from TSO READY or
from RMF main menu
don't use it anymore!
dropped with z/OS V1R2
ISPF
3270
TSO
Monitor II collects statistics about activities and resources (but no delays like Monitor III) The most heavily used Monitor II reports are the address space oriented reports
Address Space Resource Data(ARD): Resource consumption of address spaces (CPU, I/O Storage) Address Space State Data(ASD): State Information for address spaces (Service Class, Dispatching Priority, Location
e.g IN or OUT) Address Space SRM Data(ASRM): Details from the SRM point of view (Transaction Active Time, Service Units...)
ISPF is the recommended Monitor II reporting environment (although Monitor II reports can be invoked from native TSO as well)
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation33
RMF - ASD
Monitor II Monitor II Monitor II Monitor II CommandsCommandsCommandsCommands
RMF - ARDCommand ===>
MIG=63.1K
11:15:27 DEV FF PRIV LSQAJOBNAME CONN BEL FF CSF
*MASTER* 364.3 0 455 106PCAUTH 0.000 0 2 32TRACE 0.000 0 3 105DUMPSRV 1.113 0 2 62
for manual update
RMFSelection ===>
Enter selection number or command
1 Address Spaces2 I/O Subsystem3 Resource
L Library ListsU User
TSO-Command: RMF MON2
Monitor II Primary Menue.g. ARD
e.g. ASD
Selection
report-name
F6
Sort
Return
F4
e.g. ASD report options
RO
Report Options
F7
F8
Scroll up
Scroll down
Cancel
Reset
GO nn for auto update
H Prints all reportsD Delta-modePrint Prints current screenSys Remote reportingKeys View/Assign PF-keys
Other commands:
Find stringRFind
Monitor II reports can be selected either by menu choice or by report name With the report options command RO, the report contents can be customized (e.g. Workloads TSO, Batch....) Use the PF7/PF8 keys for scrolling (in case not all report lines are visible) Note, that all reports are sortable (place the cursor on the desired coulmn header and press PF6) The GO command provides the automatic refresh of the report data With the SYS command, the reports from all remote systems in the Sysplex can be accessed
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation34
RMF - ARD Address Space Resource Data Line 1 of 48 Command ===> Scroll ===> PAGE
CPU= 45/ 30 UIC=2540 PR= 0 System= SCLM Total
16:04:11 DEV FF PRV LSQA LSQA X SRM TCB CPU EXCP SWAP LPA CSA NVI V&HJOBNAME CONN 16M FF CSF ESF M CR ABS TIME TIME RATE RATE RT RT RT RT
HSM 79920 1 4 136 6 0.0 8508 10575 0.01 0.00 0.0 0.0 0.0 0.0ANTMAIN 28880 2 23 160 12 X 0.0 360.2 427.7 0.00 0.00 0.0 0.0 0.0 0.0*MASTER* 8616 0 731 102 18 0.0 296.6 2652 0.04 0.00 0.0 0.0 0.0 0.0XCFAS 4485 0 2K 1059 459 X 0.0 409.7 796.5 2.41 0.00 0.0 0.0 0.0 0.0IOSAS 3774 0 29 109 10 X 0.0 46.07 46.89 0.02 0.00 0.0 0.0 0.0 0.0RMFGAT 3144 1 20 65 6 X 0.0 5160 5236 0.67 0.00 0.0 0.0 0.0 0.0JES2 773.4 9 66 185 43 0.0 360.8 455.3 2.19 0.00 0.0 0.0 0.0 0.0SMSVSAM 771.2 1 205 764 85 X 0.0 434.8 550.2 1.55 0.00 0.0 0.0 0.0 0.0SMS 697.1 0 2 53 12 X 0.0 166.1 174.2 2.88 0.00 0.0 0.0 0.0 0.0OMVS 548.4 5 131 630 60 X 0.0 85.77 103.0 0.00 0.00 0.0 0.0 0.0 0.0CATALOG 376.5 0 2 234 1 X 0.0 111.7 117.4 0.21 0.00 0.0 0.0 0.0 0.0HSMMON 81.79 0 3 53 9 0.0 99.15 101.8 0.19 0.00 0.0 0.0 0.0 0.0NET 66.93 0 36 79 67 X 0.0 771.0 1556 0.00 0.00 0.0 0.0 0.0 0.0SMF 52.73 0 2 56 8 X 0.0 0.20 3.70 0.00 0.00 0.0 0.0 0.0 0.0LLA 27.73 0 40 67 14 X 0.0 3.65 4.09 0.06 0.00 0.0 0.0 0.0 0.0NETVSCLM 24.02 1 5 148 31 X 0.0 115.1 125.9 0.00 0.00 0.0 0.0 0.0 0.0IXGLOGR 23.65 0 5 87 48 X 0.0 38.28 45.04 0.00 0.00 0.0 0.0 0.0 0.0
Address Space ResourceConsumption at a Glance ! I/O Activity Frame Counts CPU Time
all Table Reportsare sortable !
Monitor II: ARD ReportMonitor II: ARD ReportMonitor II: ARD ReportMonitor II: ARD Report
The ARD report provides a quick check about address spaces and their resource usage (e.g. Device Connect Time or TCB Time)
The report header displays the following information: MVS view of the CPU Utilization LPAR view of the CPU utilization Unreferenced Interval Count Pagein Rate
The tabular part of the report contains similar information than the SDSF DA screen
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation35
H | Print
BHBE.M2.REPORTS
1. ALLOC F(RMFDMTSO) DA('BHBE.M2.REPORTS') disp
DCB=(RECFM=VBA,BLKSIZE=1693,LRECL=137)2. RMF MON2
BHBE.RMFOS260.ISPTABLE
ERBROPT2
Report options
RMF - ARDCommand ===>
MIG=63.1K
11:15:27 DEV FF PRIV LSQAJOBNAME CONN BEL FF CSF
*MASTER* 364.3 0 455 106PCAUTH 0.000 0 2 32TRACE 0.000 0 3 105DUMPSRV 1.113 0 2 62
Monitor II Session SetupMonitor II Session SetupMonitor II Session SetupMonitor II Session Setup
In case the user wants to save the reports for later analysis he can use the Hardcopy and Print commands The reports are written either to SYSOUT or to the preallocated dataset RMFDMTSO For most of the reports you can specify options or filters, e.g.
ARD: address space types BATCH or TSO only DEV: volume names, device numbers or storage groups
For persistence, the report options are written to the dataset &uid.RMF&rel.ISPTABLE
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation36
RMF - ARDJ Address Space ResouCommand ===>
CPU= 19/ 19 UIC=2540
BHOL DEV FF PRIV LSQA LSQA X SRM TCB TIME CONN BEL FF CSF ESF M ABS TIME
13:16:46 16.87 3 3 79 0 65K 7.20 13:16:47 16.88 3 3 79 0 65K 7.20 13:16:48 16.89 3 3 79 0 65K 7.20 13:16:49 16.90 3 3 79 0 65K 7.21 13:16:50 16.91 3 3 79 0 65K 7.21 13:16:51 16.92 3 3 79 0 65K 7.21 13:16:52 16.93 3 3 79 0 65K 7.22 13:16:53 16.94 3 3 79 0 65K 7.22 13:16:54 16.95 3 3 79 0 65K 7.22
RMF - ARD Address Space ResourCommand ===>
CPU= 9/ 8 UIC=2540
13:20:26 DEV FF PRIV LSQA LSQA X SRM TCB JOBNAME CONN BEL FF CSF ESF M ABS TIME
*MASTER* 409.1 0 503 106 0 0.0 34.00 2PCAUTH 0.000 0 2 32 0 X 0.0 0.00 RASP 0.000 0 4 12 0 0.0 0.00 TRACE 0.000 0 3 105 0 X 0.0 0.00 DUMPSRV 1.113 0 2 62 2 0.0 0.05 XCFAS 273.9 0 1420 749 5 X 0.0 28.64 GRS 0.000 0 40 692 0 X 0.0 241.01 3SMXC 0.000 0 2 33 0 0.0 30.28 SYSBMAS 0.000 0 41 73 0 0.0 3.94
Table ReportRow Report
Specific resource or job, e.g. job BHOL
All resources or jobs at a specific point in time
Current status in highlighted line Current status in whole report
Monitor II Display Monitor II Display Monitor II Display Monitor II Display ModesModesModesModes
The address space and volume oriented reports are available in two flavours: Table reports: one sample with all address spaces or all volumes at a glance is displayed Row reports: multiple samples for one address space or one volume are displayed
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation37
Monitor II Report Monitor II Report Monitor II Report Monitor II Report OverviewOverviewOverviewOverview
Central storage, processor, SRMYSRCS
Paging activityYSPAG
Enqueue reserve activitySENQREnqueue contentionSENQSysplex data server statisticsSDS
Page/swap data set activityYPGSP
Library listsLLI
I/O queuing activity dataYIOQUEUEIRLM locking dataILOCK
HFS statisticsYHFS
YYDEVV
Device activity dataYDEV
Channel path activity dataCHANNEL
YASRMJ
Address space SRM dataASRM
YASDJ
Address space state dataASD
YARDJ
Address space resource dataARD
ExplanationISPFRowMon IName
A subset of Monitor II reports is based on Monitor I gatherer data (DEV, IOQUEUE, PGSP). Those reports are not available when the ZZ session is inactive (reason: Supervisor state and Key zero is required for
certain gathering activities) The SORT capability as well as the HFS report is only available in the ISPF environment
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation38
Enterprise-wide performance monitoring of z/OS hosts
Platform independent Java Edition Linux gathering support Graphical user interface Flexible definition of data Persistent definition of views Powerful data reduction Analysis support
RMF Performance RMF Performance RMF Performance RMF Performance MonitoringMonitoringMonitoringMonitoring
RMF Performance Monitoring (RMF PM) is the graphical workstation frontend for Monitor III data. It allows to manage multiple Sysplexes from one single point of control
RMF PM is free of charge. The most recent version can be downloaded from the RMF homepage: http://www.ibm.com/servers/eserver/zseries/zos/rmf/rmfhtmls/pmweb/pmweb.html
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation39
Perfdesk Folder
Perfdesk 2
Dataview 1
Series 1Series n
Dataview 2
Series 1Series n
Perfdesk 1
Dataview 1
Series 1Series n
Dataview 2
Series 1Series n
Dataview 3
Series 1Series n
Dataview 4
Series 1Series n
RMF PM: RMF PM: RMF PM: RMF PM: PerfdeskPerfdeskPerfdeskPerfdesk ConceptConceptConceptConcept
The main window is of the RMF PM application is divided into 2 parts: The left side contains the PerfDesk notebook page, which allows to control the content of the right side of the window The right side contains the Open Perfdesk window: one Perfdesk consists of one or more views with the actual data
(Data Views). The data itself is represented by the so called Series On the host side, the Distributed Data Server (DDS) address space must be started in order to provide the data for the PM
client by means of a TCP/IP connection
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation40
Sysplex
MVS ImageI/O Subsystem
All SSIDsSSID
All LCUsLCU
All ChannelsChannel
All VolumesVolume
ProcessorStorage
Auxiliary StorageCentral Storage
CSA, SQA, ECSA, ESQAExpanded Storage
EnqueueOperatorSubsystems
JES, XCF, HSM
CPCLPAR
Coupling FacilityCF Structure
RMF PM: Resources and RMF PM: Resources and RMF PM: Resources and RMF PM: Resources and MetricsMetricsMetricsMetrics
Resources are components of a computing system. The Monitor III data model maps those resouces in terms of a hierarchical scheme
Each resource is associated with a set of metrics. Individual metrics (or counters) can be selected for data collection and continous display
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation41
data views with click-sensitve bars link to predefined Analysis PerfDesks
RMF PM: Intelligent AnalysisRMF PM: Intelligent AnalysisRMF PM: Intelligent AnalysisRMF PM: Intelligent Analysis
One of the most powerful features of RMF PM is the Intelligent Analysis: once the user has detected an abnormal value, he can navigate to context information or additional views in order to detect the reason of a problem. This is achieved by click-sensitive bars whereby the analysis menu choice offers a selection of various pathes for further drill-down
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation42
RMF PM Java TM Technology Edition - Linux
support available on Internet
Common Viewfor
z/OS and LinuxData
LinuxLinuxLinuxLinux Performance Performance Performance Performance MonitoringMonitoringMonitoringMonitoring
RMF PM has been extended to support the Linux operating system using the existing RMF PM infrastructure. Thus, the PM capabilities (spreadsheet export, filter, historical data collection, graphical trend reports, etc.) are also available for Linux environments
For the RMF PM client, it is transparent whether the host is a z/OS or Linux system On the server side, two additional components are needed:
The Linux compiled version of the Distributed Data Server infrastructure The Linux data gatherer backend (collects data out of the procfile system)
The Linux support can be downloaded free of charge: http://www.ibm.com/servers/eserver/zseries/rmf/pmlin.htm Please note that the Linux support is not an official part of z/OS RMF
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation43
RMF Monitor III Data PortalRMF Monitor III Data PortalRMF Monitor III Data PortalRMF Monitor III Data Portal
direct connection to the RMF Distributed Data Server just specify http://:8803 all resources and metrics supported (same than RMF PM)
Since the RMF Distributed Data Server talks XML over HTTP, the client can be either a fat client like RMF PM or the data can also be accessed directly by means of a web browser. Thats what we call the RMF Monitor III Data Portal
In other words, the RMF DDS server behaves like a webserver: the data can be requested by means of the standard HTTP protocol. In this case, the presentation is managed by associated formatting instructions in terms of Style Sheets (XSL documents)
The chart shows the Overview Performance Desktop (=PerfDesk) with the following metrics: % processor utilization by MVS image % total physical utilization (CP) by partition performance index by important WLM service class period % CSA utilization by MVS image
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation44
RMF Monitor III Data PortalRMF Monitor III Data PortalRMF Monitor III Data PortalRMF Monitor III Data Portal
Add this Metric to My View
click
for Persistence
select any Resourcefrom Configuration View
select favoriteMetric from List
4 Coupling Facility PerfDesk with: Request rates Service times
Beyond the Overview PerfDesk, the user can customize his own private PerfDesk (aka MyView). Thereby he can select from a rich set of metrics for any kind of resource he wants to monitor contionuously. The definitions are stored persistently in terms of browser cookies
The example shows a Coupling Facility PerfDesk with the following metrics: sync request rate by CF structure sync service time by CF structure async request rate by CF structure async service time by CF structure
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation45
RMF Monitor III Data PortalRMF Monitor III Data PortalRMF Monitor III Data PortalRMF Monitor III Data Portal
Report name Scrollable!
Combine strengths of Monitor III with state-of-the-art frontend View full Reports and exploit the SORT capability
All columnsvisible!
Resizeable!
Sortable!
In the past, the Data Portal was limited to display single metrics for selected resources This concept has been extended: with z/OS V1.8 RMF, complete Monitor III reports can be displayed in the browser window Thus, the strengths of the Monitor III ISPF reporting are combined with a state of the art frontend:
The window is resizable and scrollable All report columns can be displayed (including ISPF hidden fields and the content of popup windows) The columns can be sorted by clicking on the column header Support of automatic refresh (GO mode) Multiple reports can be displayed in parallel (tabbed browsing) Print facility and Spreadsheet export
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation46
RMF Performance Data RMF Performance Data RMF Performance Data RMF Performance Data APIAPIAPIAPIssss
ERBSMFI - Monitor II Data (SMF Type 79) RMF Sysplex Data Server (SDS)
SMF Data: ERBDSQRY, ERBDSRECMonitor III Data: ERB3XDRSMonitor II Data: ERB2XDGS
RMF Provider for z/OS CIM Server
RMF offers a couple of powerful APIs to retrieve z/OS performance data. The exploiters can be: Other z/OS components or systems management applications User written applications
All RMF APIs are documented in the RMF Programmers Guide (SC33-7994)
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation47
Sysplex Data Server: Services Sysplex Data Server: Services Sysplex Data Server: Services Sysplex Data Server: Services
all Services are available as High-Level-Language APIs
Sysplex-wide access to SMF data ERBDSQRY requests a directory of available SMF data in the sysplex ERBDSREC requests SMF record data in the sysplex
Sysplex-wide access to Monitor III data ERB3XDRS requests a set_of_samples of Monitor III data
8 does not require an ISPF and Monitor III reporter environment8 provides data reduction features to transfer only the necessary data
Sysplex-wide access to Monitor II data ERB2XDGS requests Monitor II data according to the specified SMF
type 79 subtype 8 returns Monitor II snapshot data8 provides data reduction features like ERB3XDRS
The RMF Sysplex Data Server provides a set of callable services to retrieve performance data from all RMF Monitors Please note, that the SMF data related services are not limited to the SMF 70-78 record types: once the user has specified
any SMF record type for buffering by the Sysplex Data Server (via the SMFBUF parameter), the record images can be obtained with the RMF services ERBDSQRY and ERBDSREQ
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation48
Groovy!Groovy!Groovy!Groovy!
ZZ SessionPostprocessor
Distributed Data Server
SysplexData ServerMonitor III
Spreadsheet Reporter
CIM Data Provider
Mintime & Range
Interval & Cycle Time
Monitor II
% Workflow % Using / Delay
Thats the freshly baked RMF expert! Probably he feels a bit more skilled now. At least he came out of his hiding-place!
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation49
Information and ToolsInformation and ToolsInformation and ToolsInformation and Tools
RMF homepage: www.ibm.com/systems/z/os/zos/features/rmf/n Product information, newsletters, presentations, ...n Downloads
Spreadsheet Reporter RMF PM Java Edition RMF data collector for Linux
RMF email address: [email protected]
Documentation and newsn RMF Performance Management Guide, SC33-7992n RMF Report Analysis, SC33-7991n RMF Users Guide, SC33-7990n Latest version of PDF files can be downloaded from:
http://www.ibm.com/systems/z/os/zos/bkserv/
-
Resource Measurement Facility
RMF Technical Overview 2010 IBM Corporation50
RMF Redbook !!!RMF Redbook !!!RMF Redbook !!!RMF Redbook !!!