rmf overview

50
RMF Development Edition March, 6th 2010 © 2010 IBM Corporation z/OS Resource Measurement Facility RMF Technical Overview

Upload: juanmfloress

Post on 16-Dec-2015

437 views

Category:

Documents


28 download

DESCRIPTION

TSO RMF

TRANSCRIPT

  • RMF Development Edition

    March, 6th 2010 2010 IBM Corporation

    z/OSResource Measurement Facility

    RMF Technical Overview

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation2

    TrademarksTrademarksTrademarksTrademarks

    The following are trademarks of the International Business Machines Corporation in the United States and/or other countries.

    The following are trademarks or registered trademarks of other companies.

    * Registered trademarks of IBM Corporation

    * All other products may be trademarks or registered trademarks of their respective companies.

    Java and all Java-related trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countriesLinux is a registered trademark of Linus Torvalds in the United States, other countries, or both.UNIX is a registered trademark of The Open Group in the United States and other countries.Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation.

    APPN*CICS*DB2*DB2 ConnectDirMainte-business logo*ECKDEnterprise Storage Server*ESCON*FICON*GDPS*Geographically Dispersed Parallel Sysplex

    HiperSocketsHyperSwapIBM*IBM eServerIBM e(logo)server* IBM logo*IMSLanguage Environment*MQSeries*Multiprise*NetView*On demand business logo

    OS/390*Parallel Sysplex*PR/SMProcessor Resource/Systems ManagerRACF*Resource LinkRMFS/390*Sysplex Timer*System zSystem z9TotalStorage*Virtualization Engine

    VM/ESA*VSE/ESAVTAM*WebSphere*z/Architecturez/OS*z/VM*z/VSEzSeries*z9

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation3

    Anything else?Anything else?Anything else?Anything else?

    ZZ SessionPostprocessor

    Distributed Data Server

    Monitor II

    SysplexData ServerMonitor III

    Spreadsheet Reporter

    CIM Data Provider

    Mintime & Range

    Interval & Cycle Time

    % Workflow % Using / Delay

    The RMF product consists of several components for all kinds of performance related disciplines. This session introduces the main RMF components, sheds light on their primary purpose and discusses the most common RMF terms

    Even though its impossible to become an RMF expert within one hour: lets get started and lets see how far we get!

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation4

    AgendaAgendaAgendaAgenda

    Product Structure Address Spaces Controlling the Data Gatherers Historical Reporting

    Postprocessor Spreadsheet Reporter

    Realtime Reporting Monitor III Monitor II WTO Alerts RMF Performance Monitoring Data Portal

    RMF Performance Data APIs

    The following topics will be covered in this session: What are the basic functions and what is the primary purpose of these functions? What address spaces have to be started? What are their names, underlying procedures and responsibilities? What are the options which determine how the gatheres work in detail? What are the functions for historical reporting and after the facts analysis? What are the functions for realtime reporting and problem determination? Which API's can be user by components or user written programs to retrieve z/OS performance data from RMF

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation5

    RMF Distributed Dataserver (DDS)

    RMF Product OverviewRMF Product OverviewRMF Product OverviewRMF Product Overview

    Windows Windows, Linux

    RMF Sysplex Data Server and APIs

    RMF PostprocessorHistorical Reporting,Analysis and Planning

    RMF Monitor II and IIIReal-Time Reporting,Problem Determination

    SMF

    RMFData Gatherer RMFMonitor I

    RMFMonitor II

    background

    RMFMonitor III

    SMF

    VSAMVSAM

    RMF Spreadsheet Reporter Monitor III Data Portal & RMF PM

    z/OS Resource Measurement Facility (RMF) is an optional priced feature of z/OS. It supports installations in performance analysis, capacity planning, and problem determination. For these disciplines, different kinds of data collectors are needed:

    Monitor I long term data collector for all types of resources and workloads. The SMF data collected by Monitor I is mostly used for capacity planning and performance analysis

    Monitor II snap shot data collector for address space states and resource usage. A subset of Monitor II data is also displayed by the IBM SDSF product

    Monitor III short-term data collector for problem determination, workflow delay monitoring and goal attainment supervision. This data is also used by the RMF PM Java Client and the RMF Monitor III Data Portal

    Data collected by all three gatherers can be saved persistently for later reporting (SMF records or Monitor III VSAM datasets)

    While Monitor II and Monitor III are realtime reporters, the RMF Postprocessor is the historical reporting function for Monitor I data

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation6

    SYS1

    .PA

    RMLIB

    (ERBR

    MF00

    )

    SYS1

    .PA

    RMLIB

    (ERBR

    MF04

    )

    SYS1

    .PA

    RMLIB

    (GPMS

    RV00

    )

    RMF RMF RMF RMF AddressAddressAddressAddress SpacesSpacesSpacesSpaces / / / / ProceduresProceduresProceduresProcedures

    SYS1.PROCLIB(RMF)

    RMFControls RMF

    Monitor I(zz Session)

    f RMF,s ZZ

    SYS1.PROCLIB(RMFGAT)

    RMFMonitor IIIGatherer

    f RMF, s III

    SYS1.PROCLIB(GPMSERVE)

    RMFDistributed

    DataServer

    s GPMSERVE

    SYS1.PROCLIB(RMFM3B)

    RMFMonitor III

    Batch Reporting

    s RMFM3B

    In order to make all RMF functions working, a couple of address spaces have to be activated: RMF is the root address space. It is required for any further activities The Monitor I gatherer is a subtask in the RMF address space. It is started immediately together with RMF (by default) or

    later by means of the modify command RMFGAT is the Monitor III data gatherer. It cannot be started as standalone address space. Like Monitor I, it can only be

    activated with the modify command. Within a sysplex, one instance per system is needed for RMFGAT The RMF Distributed Data Server is the data source for the RMF Performance Monitoring java client and the Monitor III

    Data Portal. Only one instance is needed per sysplex RMFM3B is needed for the generation of console messages in case a certain threshold is exceeded

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation7

    WRITECALLINSTALL COPY

    keeps Configuration Tables and other Control Information provides the Command Interface to set of modify Options:

    f RMF,f ZZ,MEMBER(99) maintains the SMF Data Buffer:

    s RMF,,,SMFBUF(SPACE(12M),RECTYPE(70:78))SMF

    SMF

    Write SMF RecordSysplexData Server

    RMF

    SMFWTM

    ComponentSMFBUF

    (DataSpace)

    SMF ExitsIEFU83x

    all SMF Record Types can be maintained by the RMF Sysplex Data Server !

    RMF RMF RMF RMF ControlControlControlControl AddressAddressAddressAddress SpaceSpaceSpaceSpace

    The RMF control address space has 3 major responsibilities: Builds up configuration tables, reads and remembers the parmlib options Provides the command interface to set options or to start and stop the gathering functions Maintains the SMF Buffer of the RMF Sysplex Data Server

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation8

    exact measurement countspick up consecutive counterscalculating the difference at the end of an interval

    sampling counts inspect variable counters continuouslybuilding the average at the end of an interval

    eg. queue counts, frame counts...

    Interval Gathering

    Cycle Gathering

    eg. CPU seconds, device connect time...

    Data Data Data Data GatheringGatheringGatheringGathering MethodsMethodsMethodsMethods

    In order to collect all the information that is displayed in the various reports, RMF uses two different techniques of data gathering

    The preferred and cheap method is to retrieve exact measurement counts: this can be timers (e.g. dispatch times) or accumalitve counts like the Start Subchannel count for a device

    The costly cycling technique is needed for counters which are changing rapidly and frequently: how long is the dispatcher queue? how many I/O queue elements are chained to a UCB? what is the actual enqueue situation?

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation9

    1. Measurements CACHE SMF 74.5 CHANNEL SMF 73 CPU SMF 70.1 CRYPTO SMF 70.2 DEVICE SMF 74.1 ENQ SMF 77 IOQ SMF 78.3 FCD SMF 74.7 ESS SMF 74.8 PAGESP SMF 75 PAGING SMF 71 TRACE SMF 76 VSTOR SMF 78.2 WKLD SMF 72.3

    2. Timing CYCLE(1000) NOSTOP SYNC(SMF)

    3. Reporting / Recording RECORD REPORT(REALTIME) SYSOUT(A)

    4. User Exits NOEXITS

    SMFSMF 74.2 XCFSMF 74.3 OMVSSMF 74.4 CFSMF 74.6 HFS

    gathered byMonitor III

    Monitor I Data Monitor I Data Monitor I Data Monitor I Data GatheringGatheringGatheringGathering

    SYS1.PARMLIB

    (ERBRMF00)

    The directives to setup Monitor I data gathering are kept in SYS1.PARMLIB(ERBRMF00) There are four different categories of control statements:

    Measurements: specifies the kind of resources, which are the candidates for data gathering The collected data can be saved to SMF records for later reporting A subset of SMF records is gathered by Monitor III: since the same data is needed for Monitor III reports,

    duplicate gathering is avoided Timing: determines the frequency for interval gathering and cycle gathering Reporting/Recording: defines the type and the creation time of the output User Exits: specifies routines which can be executed during session processing to gather or report additional data:

    ERBMFIUC (initialization) ERBMFxxx (cycle gatherer, loaded by ERBMFIUC) ERBMFDUC (interval gatherer) ERBMFRUR (report writer) ERBMFTUR (termination)

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation10

    1. Measurements IOSUB CFDETAIL CACHE VSAMRLS OPD HFSNAME zFS SGSPACE

    2. Timing CYCLE(1000) MINTIME(60) NOSTOP SYNC(00)

    3. Recording DATASET(ADD(RMF.M3G.&SYSNAME..DS1)) DATASET(ADD(RMF.M3G.&SYSNAME..DS2)) DATASET(START) DATASET(NOSWITCH) DATASET(WHOLD(7))

    4. Buffer WSTOR(32)

    ERBVSDEF vsam_ds VSAMVOL(volser)DATASET(ADD(vsam_ds)) DATASET(DEL(vsam_ds))

    ERBV2S vsam_ds seq_ds

    ERBS2V seq_ds vsam_ds

    RMFGAT Address Space

    WSTOR(32)DATASET(WHOLD(7))

    In Storage Buffer

    Monitor III Data Monitor III Data Monitor III Data Monitor III Data GatheringGatheringGatheringGathering

    SYS1.PARMLIB

    (ERBRMF04)

    The directives to setup Monitor III data gathering are kept in SYS1.PARMLIB(ERBRMF04) Only a few measurements can be turned ON or OFF. The data collection cannot be controlled on resource granularity:

    almost all resources must be monitored in order to get the complete picture of the address spaces which are using or delayed for certain resources. However, a subset of gathering activities which does not interfere with address space states can be switched on or off

    In order to extend the report range for historical data, Monitor III VSAM Datasets can be used. In this case, the buffersizewhich has been specified by the WSTOR option can be reduced to the WHOLD size (since the data which does not fit anymore in the buffer is kept at least in the datasets)

    For a smart handling of the VSAM datasets, the following procedures are provided: ERBVSDEF defines a Monitor III VSAM dataset ERBV2S unloads the VSAM data to a sequential dataset ERBS2V reconverts the sequential dataset to VSAM

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation11

    has Sysplex-wide scope exploited by RMF PM, Monitor III Data Portal,

    CIM Provider talks XML over HTTP no configuration needed! just enter S GPMSERVE or F RMF,DDS on one

    image of the Sysplex

    RMF M III Data Gatherer

    RMF Sysplex Data Server (ERB3XDRS Service)

    RMF M III Data Gatherer

    RMF M III Data Gatherer

    GPMSERVE(RMF Distributed Data Server)

    TCP/IP

    RMF RMF RMF RMF DistributedDistributedDistributedDistributed Data ServerData ServerData ServerData Server

    The RMF Distributed Data Server (DDS) is the component, which serves as data source for the RMF client applications There is just one additional address space (GPMSERVE) needed per Sysplex. The DDS uses the ERB3XDRS service to

    retrieve Sysplex-wide Monitor III data. In a mixed release configuration, the DDS should run on system with the highest RMF release

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation12

    Activities measured by Monitor ICache SubsystemChannel Path CPU / CryptoDevice EnqueueFicon Director I/O Queuing Page/Swap Data SetPaging Trace Virtual Storage Workload

    Activities measured by Monitor IIIXCFOMVSCoupling FacilityHFS

    Other ActivitiesLotus Domino ServerHTTP Server

    SMF

    RM

    F Po

    stpr

    oce

    sso

    r

    RMF Data Buffer

    Spre

    ads

    hee

    t App

    lica

    tion

    sLo

    tus

    1-2-

    3 a

    nd

    Exce

    l

    RM

    F Sp

    read

    she

    et R

    epor

    ter

    HistoricalHistoricalHistoricalHistorical ReportingReportingReportingReporting

    The tool of choice for historical reporting and after the facts analysis is the RMF Postprocessor Accordingly to the Monitor I data collection options (there is one SMF record type for each resource), the reporting structure

    is resource oriented (exception: the WLMGL record (SMF 72.3) holds performance data and resource consumption on workload granularity)

    Some SMF record types (74.x) are written by the Montior III data gatherer address space RMFGAT. However, this address space should be active anyway to guarantee access to the Monitor III reports

    With OS/390 R10 the Postprocessor has been enabled to accept foreign SMF records: Type 103 HTTP Server Type 108 Domino Server

    While the RMF Postprocessor generates tabular reports, the RMF Spreadsheet Reporter can create graphical charts based on Postprocessor data

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation13

    //SMFDUMP EXEC PGM=IFASMFDP//IDD1 DD DISP=SHR,DSN=//IDD2 DD DISP=SHR,DSN=//SMFDATA DD DISP=(NEW,PASS),SPACE=(CYL,(10,10),RLSE),// UNIT=SYSDA,DCB=(RECFM=VBS,LRECL=32760,BLKSIZE=0)//SYSIN DD *INDD(IDD1,OPTIONS(DUMP))INDD(IDD2,OPTIONS(DUMP))OUTDD(SMFDATA,TYPE(70:78))

    SMF data is kept in VSAM datasets Postprocessor requires sequential format use SMF dump utility IFASMFDP to unload the data usuallly GDGs are the prefered target:

    RMF.SMFDATA.SYSNAME(0) RMF.SMFDATA.SYSNAME(-1)

    //RMFSORT EXEC PGM=SORT//SORTIN DD DISP=SHR,DSN=// DD DISP=SHR,DSN=//SYSIN DD *SORT FIELDS=(11,4,CH,A,7,4,CH,A),EQUALSMODS E15=(ERBPPE15,36000,,N),E35=(ERBPPE35,3000,,N)

    SMF reords must be sorted by date and time SORT step is required for sysplex-wide reporting

    Workload Activity Report Coupling Facility Report Shared DASD Report

    PostprocessorPostprocessorPostprocessorPostprocessor: : : : PreparingPreparingPreparingPreparing SMF DataSMF DataSMF DataSMF Data

    Before the raw SMF data can be feeded to the Postprocessor, two preparation steps are needed: The data has to be transformed from VSAM to Sequential format

    This is achieved with the SMF Dump Utility IFASMFDP In most installations, the SMF data is dumped by default to Generation Data Groups (GDGs)

    For Sysplex-wide reporting, the records arriving from the various systems must be sorted by ascending interval time Therefore, a preceeding SORT step must be executed RMF provides the exit routines ERBPPE15 and ERBPPE35 to enable the sorting by RMF interval start time

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation14

    DD Names MFPMSGDS Message Output MFPINPUT SMF Input Datasets PPRPTS combined Interval Reports PPSUMnnn Summary Report Output PPXSRPTS Sysplex Report Output PPORPnnn Overview Report Output PPOVWREC Overview Record Output

    Control Statements DATE Start / End Date RTOD Start / End Time DINTV Duration Interval Length REPORTS Report Types OVERVIEW Report or Record OVW Overview Control Statement

    //RMFPP EXEC PGM=ERBRMFPP//MFPMSGDS DD DISP=SHR,DSN=*.ALLOC.MSG//MFPINPUT DD DISP=(OLD,DELETE),DSN=*.RMFSORT.SORTOUT//PPRPTS DD DISP=SHR,DSN=RMF.INTERVAL.REPORTS//PXSRPTS DD DISP=SHR,DSN=RMF.SYSPLEX.REPORTS//SYSIN DD *DATE(04212003,04232003)RTOD(0000,2400)DINTV(0100)REPORTS(ALL)

    JCL can be generated by ISPF Application orSpreadsheet Reporter

    PostprocessorPostprocessorPostprocessorPostprocessor JCLJCLJCLJCL

    Different DD names and control statements must be specified for the different kinds of report output However, the user doesn't need to take care about any syntax: the JCL to execute the Postprocessor can be generated with

    an ISPF dialog or with the RMF Spreadsheet Reporter workstation frontend

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation15

    PostprocessorPostprocessorPostprocessorPostprocessor: Standard Reporting: Standard Reporting: Standard Reporting: Standard Reporting

    //RMFPP EXEC PGM=ERBRMFPP//SYSIN DD * DATE(10142002,10142002)RTOD(1100,1300) REPORTS(CPU) SYSRPTS(WLMGL(SCPER))SYSOUT(H)

    C P U A C T I V I T Y

    z/OS V1R8 SYSTEM ID SYSD DATE 02/13/2007 INTERVAL 15.00.048RPT VERSION V1R8 RMF TIME 11.00.00 CYCLE 1.000 SECONDS

    CPU 2084 MODEL 314 H/W MODEL B16

    ---CPU--- ONLINE TIME LPAR BUSY MVS BUSY CPU SERIAL I/O TOTAL % I/O INTERRUPTSNUM TYPE PERCENTAGE TIME PERC TIME PERC NUMBER INTERRUPT RATE HANDLED VIA TPI0 CP 100.00 3.23 3.61 066F7A 14.60 0.691 CP 100.00 2.77 3.18 066F7A 22.53 0.492 CP 100.00 2.58 2.93 066F7A 25.61 0.433 CP 100.00 2.54 2.97 066F7A 29.42 0.46CP TOTAL/AVERAGE 2.78 3.17 92.16 0.50

    W O R K L O A D A C T I V I T Y

    z/OS V1R8 SYSPLEX SYSDPLEX START 02/13/2007-11.00.00 INTERVAL 000.15.00 MODE = GOAL RPT VERSION V1R8 RMF END 02/13/2007-11.15.00

    POLICY ACTIVATION DATE/TIME 02/12/2007 08.43.05

    ------------------------------------------------------------------------------------------------------------ SERVICE CLASS PERIODS

    REPORT BY: POLICY=DEFAULT WORKLOAD=SYSTEM SERVICE CLASS=SYSTEM RESOURCE GROUP=*NONE PERIOD=1 IMPORTANCE=SYSTEM CRITICAL =NONE

    -TRANSACTIONS- TRANS-TIME HHH.MM.SS.TTT --DASD I/O-- ---SERVICE---- --SERVICE TIMES-- ---APPL %--- -----STORAGE-----AVG 23.36 ACTUAL 0 SSCHRT 30.3 IOC 15835 CPU 26.498 CP 3.86 AVG 2871.20 MPL 23.36 EXECUTION 0 RESP 1.1 CPU 5183K SRB 7.913 AAPCP 0.00 TOTAL 67066.09 ENDED 0 QUEUED 0 CONN 0.6 MSO 0 RCT 0.017 IIPCP 0.00 SHARED 11.00 END/S 0.00 R/S AFFIN 0 DISC 0.0 SRB 1548K IIT 0.275 #SWAPS 106 INELIGIBLE 0 Q+PEND 0.3 TOT 6747K HST 0.000 AAP 0.00 --PAGE-IN RATES--EXCTD 0 CONVERSION 0 IOSQ 0.1 /SEC 7496 AAP 0.000 IIP N/A SINGLE 0.0 AVG ENC 0.00 STD DEV 0 IIP N/A BLOCK 0.0 REM ENC 0.00 ABSRPTN 321 SHARED 0.0 MS ENC 0.00 TRX SERV 321 PROMOTED 0.000 HSP 0.0

    In most cases, the RMF Postprocessor is used to produce tabular reports (one per interval). The chart shows an example for a single system report and for a Sysplex-wide report: The CPU Activity Report shows the processor utilization from the systems perspective (on logical processor level) The Workload Activity Report shows Sysplex-wide transaction and resource consumption statistics (on service class period

    level)

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation16

    R M F O V E R V I E W R E P O R T

    z/OS V1R8 SYSTEM ID SYSF START 08/07/2006-11.00.00 INTERVAL 01.00.00 RPT VERSION V1R8 RMF END 08/07/2006-15.00.00 CYCLE 1.000 SECONDS

    NUMBER OF INTERVALS 4 TOTAL LENGTH OF INTERVALS 04.00.00

    DATE TIME INT PROCS CPUBSY APPL TRANS RTIME MM/DD HH.MM.SS HH.MM.SS08/07 11.00.00 01.00.00 5.0 9.2 33.9 16.8 40.5608/07 12.00.00 01.00.00 5.0 8.5 33.0 28.1 35.9208/07 11.00.00 01.00.00 5.0 8.3 30.6 38.9 25.6608/07 12.00.00 01.00.00 5.0 8.8 24.5 24.5 44.99

    PostprocessorPostprocessorPostprocessorPostprocessor: : : : OverviewOverviewOverviewOverview ReportingReportingReportingReporting

    Format of the INTERVAL column changes from MM.SS to HH.MM.SS

    OVERVIEW(RECORD,REPORT) OVW(PROCS(NUMPROC)) OVW(CPUBSY(CPUBSY)) OVW(APPL(APPLPER(POLICY)))OVW(TRANS(TRANS(POLICY))) OVW(RTIME(RTIME(POLICY)))DINTV(0100)

    (via Spreadsheet Reporter)

    When the user wants to focus on specific metrics without analyzing an entire report, the Postprocessor Overview feature provides a smart solution: by means of overview control statements, he can extract specific values and produce his own customized reports

    Multiple intervals are visible at a glance: Thus, Overview reporting is well suited for trend analysis over long periods For all SMF 70-78 record types, the user can choose from a rich set of Overview Control statements. For more details, refer

    to the RMF Users Guide (SC33-7990), Chapter 17, Overview and Exception Conditions

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation17

    converts SMF Data to Spreadsheet Format creates graphical Views for Trend Analysis can be downloaded from the RMF Homepage

    RMF Spreadsheet ReporterRMF Spreadsheet ReporterRMF Spreadsheet ReporterRMF Spreadsheet Reporter

    In extension to the RMF Postprocessor, RMF offers a graphical workstation frontend for historical reporting: the RMF Spreadsheet Reporter

    The visualization of the data is performed by spreadsheet macros, which are shipped as samples together with the Spreadsheet Reporter main application.

    The Spreadsheet Reporter is a basic part of the RMF product and doesnt require additional charges. The installable image is shipped with the host file SYS1.SERBPWS(ERB9R2SW) or the most recent version can be downloaded from the RMF homepage: http://www.ibm.com/servers/eserver/zseries/zos/rmf/. You will find the Spreadsheet Reporter in the Toolscategory of the homepage.

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation18

    RMF Spreadsheet ReporterRMF Spreadsheet ReporterRMF Spreadsheet ReporterRMF Spreadsheet Reporter

    SMF raw Data

    Working Sets

    Spreadsheet

    Overview Records

    *.WK1 Files

    *.OVW Files

    Report Listings Overview Records

    Report Listings

    *.RPT FilesCPU Contention: System SYS1

    RMF Postprocessor

    Download

    Extract

    Convert

    Generate

    Display

    The transition from the SMF raw data to graphical charts is a complex process with multiple steps. The good news - the RMF Spreadsheet Reporter does it all for you by magic: Generates Report Listings or Overview Records from the SMF data with the RMF Postprocessor: on the Resource

    notebook page of the Spreadsheet Reporter, specify as input the SMF dump data sets from all the systems of your sysplex. Then check the desired reports on the Reports selection dialog. Now you are ready to exploit the Spreadsheet Reporters remote job execution engine, which allows you to run the Postprocessor from your workstation!

    Downloads the Postprocessor output to your workstation Extracts the reports or records Convert the reports or records to spreadsheet format and generates the Working Set

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation19

    RMF Spreadsheet ReporterRMF Spreadsheet ReporterRMF Spreadsheet ReporterRMF Spreadsheet Reporter

    Macrosfor all Types of

    Performance relevantAreas !!

    Once a Working Set has been created from SMF data, the user can select from a rich set of macros for all performance relevant areas

    Three basic types of macros can be distinguished: Macros based on data contained in one Postprocessor Report (for detailed analysis of one single interval) Macros based on data contained in Overview Records (for trend analysis with multiple intervals) Macros based on data contained in multiple Postprocessor Reports (hybrid macros)

    The new XCF Trend Report macro belongs to the family of hybrid macros

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation20

    covers all Sysplex related aspects two monitors and a workstation extension

    Monitor III, best suited for short-term, real-time and historical reporting online performance analysis goal attainment supervision sysplex-wide and single-system reporting monitoring of exceptional conditions

    Monitor II, best suited for snapshot reporting single job and resource monitoring

    RMF PM / Data Portal enterprise-wide reporting of z/OS systems based on RMF Monitor III data

    CF

    WLM

    RealtimeRealtimeRealtimeRealtime ReportingReportingReportingReporting

    RMF offers three different Monitors for realtime reporting: Monitor III Reporter (ISPF):

    Is designed for workflow analysis and goal attainment supervision Can be used in realtime mode, but supports historical reporting as well

    Monitor II (ISPF and native TSO): Is a snapshot monitor. The interval length can be determined by the user Consists of a set of address space and resource oriented reports

    RMF Performance Monitoring / Monitor III Data Portal: Allows to manage multiple Sysplexes from the workstation The raw data is identical with Monitor III ISPF, but the GUI is totally different

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation21

    Monitor III Delay Monitoring Processor Storage Device Enqueue Operator

    Message Tape Mount

    SubsystemHSM - JES - XCF

    Monitor III Activity Monitoring Common Storage Page/Swap Data Sets Storage Frames Device Data Set Level by Job and Volume Cache Coupling Facility Goal Attainment VSAM RLS UNIX System Services Enclaves zFS Diskspace Spin/Suspend Locks

    Monitor III Features Cursor-Sensitive Navigation Workflow/Exceptions Monitoring Automatic Customization Support of WTO Messages Continuous Monitoring Hardcopy Reports On-Line Tutorial On-Line Help Adaptive Reports User Reports Sysplex-wide Reports Remote Reporting

    Monitor III ReportingMonitor III ReportingMonitor III ReportingMonitor III Reporting

    The Monitor III ISPF Reporter provides two basic types of monitoring: WorkloadDelay monitoring

    The state samplers are inspecting all hard- and software resources and assign those resources on address space level

    Possible states for an address space are: Delayed for Processor, Storage, Device, Enqueue, Operator, Subsystem Using Processor or Device Idle (timer wait or terminal input wait) Unknown (e.g. using an unmonitored waiting mechanism)

    ActivityResource Monitoring In addition to the workloaddelay concept, there are lots of resource oriented reports with similar statistics than the

    RMF Postprocessor reports Together with hardware or software related innovations new reports have been added with each z/OS release

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation22

    States of a JobStates of a JobStates of a JobStates of a Job

    delay samplesnumber of samples 100Delay(%) =

    using samplesnumber of samples 100Using(%) =

    Workflow (%) = using samplesusing samples delay samples+ 100

    Unknown

    ?

    Proc DevDevIdle

    JESHSM

    XCF

    Proc

    Enq

    Stor

    Dsn Dsn

    Oper

    = 50%

    = 33%

    = 60%

    Address spaces are either using resources or they are delayed for resources or they are idle The RMF state samplers are trying to monitor continuously all kinds of hard- and software resources Depending on the result, all address spaces can be associated with the following qualities:

    Using% Delay% Workflow%

    Since not all address spaces can be catched up every cycle, the remaining states are counted as unknown. Possible reasons for unknown are:

    The address space is delayed for unmonitored resource (other than DASD or Tape) The address space is using an unmonitored waiting mechanism (most Started Tasks!) statistical reasons

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation23

    Example: Using and DelayExample: Using and DelayExample: Using and DelayExample: Using and Delay

    CA B

    D

    B

    C

    A

    D

    A

    B

    C

    D

    CA D

    BRes Res Res Res

    Job I U D Why A * C B * C C * D *

    Job I U D Why A * B B * C * D *

    Job I U D Why A * B * C * A D *

    Job I U D Why A * C B * C * D * C

    RMF Monitor III Delay Report

    Samples: 4 Time: 06.28.20 Range: 4 Sec

    WFL USG DLY IDL Primary Jobname % % % % Reason

    A 25 25 75 0 CB 50 25 25 50 CC 66 50 25 25 AD 0 0 25 75 C

    RMF Monitor III Resource Delay Report

    Samples: 4 Time: 06.28.20 Range: 4 Sec

    WFL ADU Jobname USG DLY Reason Resource % % %

    Res 40 1.5 A 25 75 CB 25 25 CC 50 25 AD 0 25 C

    The example helps to understand what is calculated and reported by RMF Monitor III From the jobs perspective:

    Job A 1x Using(25%) 3x Delayed(75%) 0x Idle(0%) Workflow: 1/4 = 25% Job B 1x Using(25%) 1x Delayed(25%) 2x Idle(50%) Workflow: 1/2 = 50% Job C 2x Using(50%) 1x Delayed(25%) 1x Idle(25%) Workflow: 2/3 = 66% Job D 0x Using(0%) 1x Delayed(25%) 3x Idle(75%) Workflow: 0/1 = 0%

    From the resource perspective: 4x using, 6x delay Workflow = 4/10 = 40 % Average Delayed User (ADU): 6/4 = 1.5

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation24

    RMF V1R7 Delay Report Line 1 of 326

    Samples: 100 System: AQTS Date: 07/18/06 Time: 06.28.20 Range: 100 Sec

    Service WFL USG DLY IDL UKN ---- % Delayed for ---- Primary Name CX Class Cr % % % % % PRC DEV STR SUB OPR ENQ Reason

    SUSANK T TSOPRIME 0 0 100 0 0 0 0 0 100 0 0 HSM CONSOLE S SYSTEM 0 0 15 0 85 0 0 0 0 15 0 Message RRSSERVQ B WLMSHORT 0 0 1 0 1 0 0 0 1 0 0 JES BHBE T TSOPRIME 40 4 6 90 0 6 0 0 0 0 0 JHUGO MORABIT T TSOPRIME 41 37 56 0 7 0 56 0 0 0 0 COMPK5 RONDA2A B COMBUILD 42 29 42 0 29 1 41 0 0 0 0 COMPK5 D24JAP1 T TSOPRIME 49 22 23 56 0 0 23 0 0 0 0 PRIPK5 RRSSERVQ B WLMSHORT 50 2 2 0 0 0 2 0 0 0 0 SPOL1J GRSARTSQ B WLMSHORT 50 1 1 0 0 0 0 0 1 0 0 JES RRSSERVQ B WLMSHORT 50 1 1 0 0 0 0 0 1 0 0 JES CATALOG S SYSTEM 63 57 39 0 12 0 39 0 0 0 0 MCATTS ANFWPROC SO SYSSTC 67 2 1 0 2 0 1 0 0 0 0 SPOL1J GRSARTSQ B WLMSHORT 71 5 2 0 0 0 1 0 1 0 0 SPOL1J SMFDRS S STCMED 71 5 2 0 93 0 0 0 0 0 2 SYSZVVDS JES2 S SYSSTC 73 16 6 0 79 0 6 0 0 0 0 SPOL1J GRSARTSQ B WLMSHORT 80 4 1 0 0 0 0 0 1 0 0 JES ARTXESQ B WLMSHORT 80 4 1 0 0 0 1 0 0 0 0 SPOL1L DFRMM S SYSSTC 83 81 18 0 1 0 18 0 0 0 0 SL3061

    Address SpacePerformance at a Glance !

    sorted by ascending Workflow Delay Type Breakdown Delay Reason Information

    Monitor III: Job Monitor III: Job Monitor III: Job Monitor III: Job DelaysDelaysDelaysDelays

    The Address Space Delay report is the basic report that visualizes the overall Monitor III concept The report is sorted by ascending workflow: thus, the critical candidates with low workflow appear on top of the address

    space list Beyond the basic statistics (workflow, using, delay, idle, unknown), a delay type breakdown for the following resources is

    provided: PRC processor DEV device STR storage SUB software subsystem OPR operator ENQ enqueue

    Additionally, the delay reason information is displayed in the rightmost column: For DEV the critical volume For PRC the competing address space with the highest usage For SUB the subsystem name For ENQ the enqueue resource name

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation25

    RMF V1R7 Workflow/Exceptions Line 1 of 32

    Samples: 100 System: AQTS Date: 07/18/02 Time: 06.28.20 Range: 100 Sec

    --------------------------- Speed (Workflow) ---------------------------------Speed of 100 = Maximum, 0 = Stopped Average CPU Util: 81 %

    Name Users Active Speed Name Users Active Speed *SYSTEM 560 35 42 *DEV 53 15 46ALL TSO 19 4 17 *MASTER* 1 1 100ALL STC 485 6 52ALL BATCH 39 24 37ALL ASCH Not availALL OMVS 17 1 92*PROC 292 8 43

    ------------------------------ Exceptions -------------------------------------Name Reason Critical val. Possible cause or action*SLIP* SLIP PER TRAP SLIP ID=U427 is active.ALL TSO DEV -RMFLIB 1.5 users Short block size or excessive seeks.*SYSTEM PROC-SUSANK 12.1 usersBHBE SUBS-JES 100.0 % delay Processing TSO OUTPUT command request.MORABIT PROC-SUSANK 75.0 % delayGANDALF DEV -SMS001 20.0 % delay Device may be overloaded.CONSOLE OPER-Message 13.0 % delay Awaiting reply to operator request 52.HSM Not avail Job HSM is not running.SYSPAG Not avail Volume SYSPAG is not mounted.

    Bottleneck Detectionat a Glance !

    Workflow for Groups and Resources

    Exception Lines for individual Conditions

    Monitor III: Monitor III: Monitor III: Monitor III: Workflow/ExceptionsWorkflow/ExceptionsWorkflow/ExceptionsWorkflow/Exceptions

    The Monitor III Workflow Exception Report consists of two independent sections: The Workflow section provides an overview about groups of work, individual address spaces or resources The Exceptions section displays alerts when critical thresholds are exceeded

    The content of both sections can be customized individually: For the Workflow section, up to 14 candidates for continuous display of workflow can be selected For the Exceptions section, alerts for almost all Monitor III metrics (as well as combination of metrics) can be defined

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation26

    RMF 2.10.0 Sysplex Summary - SCLMPLEX Line 1 of 14 Command ===> Scroll ===> CSR

    WLM Samples: 240 Systems: 3 Date: 05/15/02 Time: 13.00.00 Range: 60 Sec

    Service Definition: SCLM Installed at: 12/06/00, 10.07.24Active Policy: STANDARD Activated at: 12/06/00, 10.07.33

    ------- Goals versus Actuals -------- Trans --Avg. Resp. Time-Exec Vel --- Response Time --- Perf Ended WAIT EXECUT ACTUAL

    Name T I Goal Act ---Goal--- --Actual-- Indx Rate Time Time Time

    STC W 88 0.000 0.000 0.000 0.000STCCMD S 3 40 88 0.46 0.000 0.000 0.000 0.000SYSTEM W 69 0.000 0.000 0.000 0.000SYSSTC S N/A 68 N/A 0.000 0.000 0.000 0.000SYSTEM S N/A 70 N/A 0.000 0.000 0.000 0.000TSO W 84 2.100 0.000 0.608 0.608PRDTSO S 84 2.100 0.000 0.608 0.608

    1 1 60 1.000 AVG 0.080 AVG 0.08 1.150 0.000 0.080 0.0802 1 0.0 1.500 AVG 0.109 AVG 0.07 0.567 0.000 0.109 0.1093 1 85 2.000 AVG 2.928 AVG 1.46 0.383 0.000 2.928 2.928

    MASTER R N/A 47 N/A 0.000 0.000 0.000 0.000

    Sysplex Performance at a Glance ! 80 Intervals in GO Mode colored Indication for PI > 1Importance = 1+2Importance > 2

    Monitor III: Goal Monitor III: Goal Monitor III: Goal Monitor III: Goal AttainmentAttainmentAttainmentAttainment

    The Sysplex Summary Report is the standard report for goal attainment supervision The report provides Goals vs. Actuals on Service Class Period level. In case the Performance Index (PI) for a certain

    Service Class Period is higher than 1, coloured lines are generated: A red line for Service Class Periods with high Importance (1, 2) A yellow line for Service Class Periods with low Importance (3, 4, 5)

    In GO Mode, a coloured bar shows the state for the last 80 intervals: when at least one line for the current interval becomes red or yellow, a red or yellow square will be added. Otherwise the bar is extended with a green square. Thus, the Sysplex performance is visible at a glance for an 80-minute period (proposed a 60 second reporting range)

    For Service Classes with response time goals, the report provides a direct link to Response Time Distribution Report by means of cursor sensitivity

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation27

    Monitor III: Session SetupMonitor III: Session SetupMonitor III: Session SetupMonitor III: Session Setup

    SYS1.SERBCLS(ERBRMF3X)

    TSO Address Space

    Monitor III Reporter

    In Storage Buffer

    RMFGAT

    Type text

    Type text Type text

    VSAM Datasets

    ERBTLIB&USER.&REL.ISPTABLESYS1.SERBTENUSYS1.SERBT

    &USER.&REL.ADMGDF

    alloc f(rmfds00) da(rmf.m3.sys1.data) alloc f(rmfds01) da(rmf.m3.sys2.data)

    VSAM Archives

    The options for a Monitor III reporter session are stored in tables (e.g exception definitions). The initialization procedure ERBRMF3X performs the table allocations when a reporter session is going to be started

    By default, a reporter session accesses directly the actual data collected by the RMFGAT (via Cross Memory) However, it is also possible to establish a standalone reporter session independent from the Monitor III gatherer: in order to

    analyze historical data, up to 99 VSAM datasets can be preallocated (DDNAME = RMFDSxx)

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation28

    Monitor III Reporter UsageMonitor III Reporter UsageMonitor III Reporter UsageMonitor III Reporter Usage

    System InformationSamples: 100 System: AQTS

    Group WFL User ...*SYSTEM 30 250 ...*TSO 20 22 ...DMN001 28 16 ...PG002 20 22 ...

    HelpReportOptions

    SessionOptions

    Tutorial

    M IIIMain Menu

    RMFMain Menu

    ROSessionColoretc.

    Tutorial

    3Report Selections,Short Commands

    PostprocessorSelection Menu

    Monitor II

    1 2

    RMF

    F3

    F1

    F3

    F3

    Enter

    Enter

    Scroll

    BREF/FREF

    Find, Color, ICU, Current, GO, and more ...

    F10 F11

    F7 F8

    The Monitor III reporter session can be invoked directly from the RMF Main Menu Various session options (SO command) can be defined and stored persistently, for example:

    initial mode (GO or STOP) initial screen range length

    By means of report options (RO command), the content of almost all reports can be customized A very detailed tutorial provides lots of information about the Monitor III concepts and features Cursor sensitive HELP (including detailed metric descriptions) is always available via the PF1 key

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation29

    Delay Reports

    STORCR

    STORC

    STORF

    CHANNEL

    IOQUEUE

    Resource Utilization Reports

    PROC DEV

    ENQ

    STOR STORS

    Job Reports

    Job/Workload oriented

    JOBDetailed Job Analysis

    User Reports

    RG

    MSI

    SYSTREND

    DEVN

    DEVT

    DSD

    STORCR

    PROC

    Modified panelUser-written reportSortable report

    DSND DSNV

    ENQR

    HSM JES

    XCF

    Resource oriented

    DEVR

    STORR

    SYSRTD

    SYSWKM

    Sysplex Reports Other

    SYSENQ

    DSINDEXCACHDET

    CACHSUM

    Cache Subsystem

    CFOVER

    CFACT

    CFSYS

    Coupling Facility

    SYSINFO GROUP

    System/Workload Reports

    CPC

    RLSSC

    RLSDS

    RLSLRU

    VSAM RLS

    SYSSUM

    WFEX DELAY

    Monitoring Reports ENCLAVE

    OPD

    Monitor III Report Monitor III Report Monitor III Report Monitor III Report OverviewOverviewOverviewOverview

    Monitor III offers a complete report suite which covers all performance relevant areas. The reports can be grouped by categories:

    A family of reports with sysplex wide scope System level reports for various kinds of measurements Job/address space oriented reports Reports from the perspective of the major hard-and software resources User reports which can be adapted or customized

    In order to improve the Monitor III capabilities, new reports have been added frequently, e.g. OS/390 R4 DSND DSNV OS/390 R6 CFSYS CFOVER CFACT OS/390 R7 CACHESUM CACHDET ENCLAVE OS/390 R10 RLSSC RLSDS RLSLRU OPD z/OS V1R2 CPC z/OS V1R7 ZFSSUM ZFSACT SPACEG SPACED z/OS V1R8 XCFOVW XCFSYS XCFPATH XCFGROUP (Data Portal only)

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation30

    Monitor III: Setup Monitor III: Setup Monitor III: Setup Monitor III: Setup forforforfor WTOWTOWTOWTOssss

    Monitor IIIBatch

    Reporter

    ExitModule

    Monitor III Batch Address Space creates Reporting Tables Thresholds can be defined via

    Workflow Exception Options Dialog Reporter Phase Exit Module

    Console Message is generated by Exit Module (e.g. WLM Capping)

    $HASP100 BMAI ON TSOINRDR$HASP373 BMAI STARTEDIEF125I BMAI - LOGGED ON - TIME=12.36.20+RMF300I 3B: Processing CPC Report...+RMF301I 3B: Local Partition Capping State:+RMF303I 3B: Time until Capping (sec): 40 (WTO Limit: 600)+RMF304I 3B: MSU Consumption of critical LPARs:+RMF305I 3B: SYS1 : 64 (WTO Limit: 60)+RMF305I 3B: SYS4 : 48 (WTO Limit: 32)IEF126I BMAI - LOGGED OFF - TIME=12.38.00$HASP395 BMAI ENDED$HASP250 BMAI PURGED+RMF300I 3B: Processing CPC Report...+RMF301I 3B: Local Partition Capping State:+RMF302I 3B: WLM Capping %: 24.2 (WTO Limit: 10.0)+RMF304I 3B: MSU Consumption of critical LPARs:+RMF305I 3B: SYS1 : 82 (WTO Limit: 60)+RMF305I 3B: SYS2 : 12 (WTO Limit: 10)$HASP100 BMGU ON TSOINRDR$HASP373 BMGU STARTEDIEF125I BMGU - LOGGED ON - TIME=12.39.40

    WTO

    Sample Exits provided for: Workflow Exception Report Sysinfo Report CPC Capacity Report

    Monitor IIIGatherer

    Set ofSamples

    The procedure RMFM3B is supplied to run a Monitor III reporter session in batch mode. This environment is well suited, when a threshold is exceeded and a corresponding message should be displayed on the console

    For threshold processing, the user can select from all metrics contained in one of the Monitor III reports: the procedure accepts any report name as input parameter. In other words, the specified report is the one and only report that is produced by the batch reporter session (multiple reports are not supported)

    The scanning of the report tables and the comparison against thresholds is performed by exit routines. Depending on the result of this comparison, a WTO can be generated

    RMF ships the following REXX exit routines as examples (in SYS1.SERBCLS): ERBR3WFX processes the WFEX report ERBR3SYS processes the SYSINFO report ERBR3CPC processes the CPC report

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation31

    Monitor II: Monitor II: Monitor II: Monitor II: OverviewOverviewOverviewOverview

    Postprocessor

    SMF 79.xf rmf,s aa

    SMF 79.x10:12:08 DEVJOBNAME CONN*MASTER* 2618PCAUTH 1264

    Monitor II is a Snapshot Reporter collects the status of system resources (CPU, devices, paging activity, ...) collects the status of address spaces (resource usage, state information)

    use Monitor II to continuously monitor resource usage determine the state of any address space in the system track CPU usage of problem address spaces collect supplemental information when analyzing performance problems with Monitor III

    choose Background Session to collect SMF records for archiving and later postprocessing to automate snapshot reporting

    choose Display Session for immediate feedback for online analysis

    Monitor II complements the host based RMF reporting suite. It is the RMF snapshot reporting feature Three basic groups of reports can be distinguished:

    Address space oriented reports (e.g. ARD, ASD, ASRM) I/O subsystem related reports (e.g. DEV, CHANNEL) Resource reports (e.g. SENQ, SPAG)

    Monitor II is the appropriate function, when immediate feedback about address spaces or resources is required However, by means of a background session, Monitor II can also produce SMF Type 79 records. These records can be

    feeded to the RMF Postprocessor for after the facts analysis

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation32

    Activities measured by Monitor II: Address Space Data

    Resource StateSRM

    Channel Path Device I/O Queuing Enqueue HFS IRLM Long Locks Paging Page/Swap Data Set SRM Resource Data Sysplex Data Server Library Display

    Monitor II ReportingMonitor II ReportingMonitor II ReportingMonitor II Reporting

    for most comfortable usage supports sorting and finding started from TSO READY or

    from RMF main menu

    don't use it anymore!

    dropped with z/OS V1R2

    ISPF

    3270

    TSO

    Monitor II collects statistics about activities and resources (but no delays like Monitor III) The most heavily used Monitor II reports are the address space oriented reports

    Address Space Resource Data(ARD): Resource consumption of address spaces (CPU, I/O Storage) Address Space State Data(ASD): State Information for address spaces (Service Class, Dispatching Priority, Location

    e.g IN or OUT) Address Space SRM Data(ASRM): Details from the SRM point of view (Transaction Active Time, Service Units...)

    ISPF is the recommended Monitor II reporting environment (although Monitor II reports can be invoked from native TSO as well)

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation33

    RMF - ASD

    Monitor II Monitor II Monitor II Monitor II CommandsCommandsCommandsCommands

    RMF - ARDCommand ===>

    MIG=63.1K

    11:15:27 DEV FF PRIV LSQAJOBNAME CONN BEL FF CSF

    *MASTER* 364.3 0 455 106PCAUTH 0.000 0 2 32TRACE 0.000 0 3 105DUMPSRV 1.113 0 2 62

    for manual update

    RMFSelection ===>

    Enter selection number or command

    1 Address Spaces2 I/O Subsystem3 Resource

    L Library ListsU User

    TSO-Command: RMF MON2

    Monitor II Primary Menue.g. ARD

    e.g. ASD

    Selection

    report-name

    F6

    Sort

    Return

    F4

    e.g. ASD report options

    RO

    Report Options

    F7

    F8

    Scroll up

    Scroll down

    Cancel

    Reset

    GO nn for auto update

    H Prints all reportsD Delta-modePrint Prints current screenSys Remote reportingKeys View/Assign PF-keys

    Other commands:

    Find stringRFind

    Monitor II reports can be selected either by menu choice or by report name With the report options command RO, the report contents can be customized (e.g. Workloads TSO, Batch....) Use the PF7/PF8 keys for scrolling (in case not all report lines are visible) Note, that all reports are sortable (place the cursor on the desired coulmn header and press PF6) The GO command provides the automatic refresh of the report data With the SYS command, the reports from all remote systems in the Sysplex can be accessed

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation34

    RMF - ARD Address Space Resource Data Line 1 of 48 Command ===> Scroll ===> PAGE

    CPU= 45/ 30 UIC=2540 PR= 0 System= SCLM Total

    16:04:11 DEV FF PRV LSQA LSQA X SRM TCB CPU EXCP SWAP LPA CSA NVI V&HJOBNAME CONN 16M FF CSF ESF M CR ABS TIME TIME RATE RATE RT RT RT RT

    HSM 79920 1 4 136 6 0.0 8508 10575 0.01 0.00 0.0 0.0 0.0 0.0ANTMAIN 28880 2 23 160 12 X 0.0 360.2 427.7 0.00 0.00 0.0 0.0 0.0 0.0*MASTER* 8616 0 731 102 18 0.0 296.6 2652 0.04 0.00 0.0 0.0 0.0 0.0XCFAS 4485 0 2K 1059 459 X 0.0 409.7 796.5 2.41 0.00 0.0 0.0 0.0 0.0IOSAS 3774 0 29 109 10 X 0.0 46.07 46.89 0.02 0.00 0.0 0.0 0.0 0.0RMFGAT 3144 1 20 65 6 X 0.0 5160 5236 0.67 0.00 0.0 0.0 0.0 0.0JES2 773.4 9 66 185 43 0.0 360.8 455.3 2.19 0.00 0.0 0.0 0.0 0.0SMSVSAM 771.2 1 205 764 85 X 0.0 434.8 550.2 1.55 0.00 0.0 0.0 0.0 0.0SMS 697.1 0 2 53 12 X 0.0 166.1 174.2 2.88 0.00 0.0 0.0 0.0 0.0OMVS 548.4 5 131 630 60 X 0.0 85.77 103.0 0.00 0.00 0.0 0.0 0.0 0.0CATALOG 376.5 0 2 234 1 X 0.0 111.7 117.4 0.21 0.00 0.0 0.0 0.0 0.0HSMMON 81.79 0 3 53 9 0.0 99.15 101.8 0.19 0.00 0.0 0.0 0.0 0.0NET 66.93 0 36 79 67 X 0.0 771.0 1556 0.00 0.00 0.0 0.0 0.0 0.0SMF 52.73 0 2 56 8 X 0.0 0.20 3.70 0.00 0.00 0.0 0.0 0.0 0.0LLA 27.73 0 40 67 14 X 0.0 3.65 4.09 0.06 0.00 0.0 0.0 0.0 0.0NETVSCLM 24.02 1 5 148 31 X 0.0 115.1 125.9 0.00 0.00 0.0 0.0 0.0 0.0IXGLOGR 23.65 0 5 87 48 X 0.0 38.28 45.04 0.00 0.00 0.0 0.0 0.0 0.0

    Address Space ResourceConsumption at a Glance ! I/O Activity Frame Counts CPU Time

    all Table Reportsare sortable !

    Monitor II: ARD ReportMonitor II: ARD ReportMonitor II: ARD ReportMonitor II: ARD Report

    The ARD report provides a quick check about address spaces and their resource usage (e.g. Device Connect Time or TCB Time)

    The report header displays the following information: MVS view of the CPU Utilization LPAR view of the CPU utilization Unreferenced Interval Count Pagein Rate

    The tabular part of the report contains similar information than the SDSF DA screen

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation35

    H | Print

    BHBE.M2.REPORTS

    1. ALLOC F(RMFDMTSO) DA('BHBE.M2.REPORTS') disp

    DCB=(RECFM=VBA,BLKSIZE=1693,LRECL=137)2. RMF MON2

    BHBE.RMFOS260.ISPTABLE

    ERBROPT2

    Report options

    RMF - ARDCommand ===>

    MIG=63.1K

    11:15:27 DEV FF PRIV LSQAJOBNAME CONN BEL FF CSF

    *MASTER* 364.3 0 455 106PCAUTH 0.000 0 2 32TRACE 0.000 0 3 105DUMPSRV 1.113 0 2 62

    Monitor II Session SetupMonitor II Session SetupMonitor II Session SetupMonitor II Session Setup

    In case the user wants to save the reports for later analysis he can use the Hardcopy and Print commands The reports are written either to SYSOUT or to the preallocated dataset RMFDMTSO For most of the reports you can specify options or filters, e.g.

    ARD: address space types BATCH or TSO only DEV: volume names, device numbers or storage groups

    For persistence, the report options are written to the dataset &uid.RMF&rel.ISPTABLE

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation36

    RMF - ARDJ Address Space ResouCommand ===>

    CPU= 19/ 19 UIC=2540

    BHOL DEV FF PRIV LSQA LSQA X SRM TCB TIME CONN BEL FF CSF ESF M ABS TIME

    13:16:46 16.87 3 3 79 0 65K 7.20 13:16:47 16.88 3 3 79 0 65K 7.20 13:16:48 16.89 3 3 79 0 65K 7.20 13:16:49 16.90 3 3 79 0 65K 7.21 13:16:50 16.91 3 3 79 0 65K 7.21 13:16:51 16.92 3 3 79 0 65K 7.21 13:16:52 16.93 3 3 79 0 65K 7.22 13:16:53 16.94 3 3 79 0 65K 7.22 13:16:54 16.95 3 3 79 0 65K 7.22

    RMF - ARD Address Space ResourCommand ===>

    CPU= 9/ 8 UIC=2540

    13:20:26 DEV FF PRIV LSQA LSQA X SRM TCB JOBNAME CONN BEL FF CSF ESF M ABS TIME

    *MASTER* 409.1 0 503 106 0 0.0 34.00 2PCAUTH 0.000 0 2 32 0 X 0.0 0.00 RASP 0.000 0 4 12 0 0.0 0.00 TRACE 0.000 0 3 105 0 X 0.0 0.00 DUMPSRV 1.113 0 2 62 2 0.0 0.05 XCFAS 273.9 0 1420 749 5 X 0.0 28.64 GRS 0.000 0 40 692 0 X 0.0 241.01 3SMXC 0.000 0 2 33 0 0.0 30.28 SYSBMAS 0.000 0 41 73 0 0.0 3.94

    Table ReportRow Report

    Specific resource or job, e.g. job BHOL

    All resources or jobs at a specific point in time

    Current status in highlighted line Current status in whole report

    Monitor II Display Monitor II Display Monitor II Display Monitor II Display ModesModesModesModes

    The address space and volume oriented reports are available in two flavours: Table reports: one sample with all address spaces or all volumes at a glance is displayed Row reports: multiple samples for one address space or one volume are displayed

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation37

    Monitor II Report Monitor II Report Monitor II Report Monitor II Report OverviewOverviewOverviewOverview

    Central storage, processor, SRMYSRCS

    Paging activityYSPAG

    Enqueue reserve activitySENQREnqueue contentionSENQSysplex data server statisticsSDS

    Page/swap data set activityYPGSP

    Library listsLLI

    I/O queuing activity dataYIOQUEUEIRLM locking dataILOCK

    HFS statisticsYHFS

    YYDEVV

    Device activity dataYDEV

    Channel path activity dataCHANNEL

    YASRMJ

    Address space SRM dataASRM

    YASDJ

    Address space state dataASD

    YARDJ

    Address space resource dataARD

    ExplanationISPFRowMon IName

    A subset of Monitor II reports is based on Monitor I gatherer data (DEV, IOQUEUE, PGSP). Those reports are not available when the ZZ session is inactive (reason: Supervisor state and Key zero is required for

    certain gathering activities) The SORT capability as well as the HFS report is only available in the ISPF environment

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation38

    Enterprise-wide performance monitoring of z/OS hosts

    Platform independent Java Edition Linux gathering support Graphical user interface Flexible definition of data Persistent definition of views Powerful data reduction Analysis support

    RMF Performance RMF Performance RMF Performance RMF Performance MonitoringMonitoringMonitoringMonitoring

    RMF Performance Monitoring (RMF PM) is the graphical workstation frontend for Monitor III data. It allows to manage multiple Sysplexes from one single point of control

    RMF PM is free of charge. The most recent version can be downloaded from the RMF homepage: http://www.ibm.com/servers/eserver/zseries/zos/rmf/rmfhtmls/pmweb/pmweb.html

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation39

    Perfdesk Folder

    Perfdesk 2

    Dataview 1

    Series 1Series n

    Dataview 2

    Series 1Series n

    Perfdesk 1

    Dataview 1

    Series 1Series n

    Dataview 2

    Series 1Series n

    Dataview 3

    Series 1Series n

    Dataview 4

    Series 1Series n

    RMF PM: RMF PM: RMF PM: RMF PM: PerfdeskPerfdeskPerfdeskPerfdesk ConceptConceptConceptConcept

    The main window is of the RMF PM application is divided into 2 parts: The left side contains the PerfDesk notebook page, which allows to control the content of the right side of the window The right side contains the Open Perfdesk window: one Perfdesk consists of one or more views with the actual data

    (Data Views). The data itself is represented by the so called Series On the host side, the Distributed Data Server (DDS) address space must be started in order to provide the data for the PM

    client by means of a TCP/IP connection

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation40

    Sysplex

    MVS ImageI/O Subsystem

    All SSIDsSSID

    All LCUsLCU

    All ChannelsChannel

    All VolumesVolume

    ProcessorStorage

    Auxiliary StorageCentral Storage

    CSA, SQA, ECSA, ESQAExpanded Storage

    EnqueueOperatorSubsystems

    JES, XCF, HSM

    CPCLPAR

    Coupling FacilityCF Structure

    RMF PM: Resources and RMF PM: Resources and RMF PM: Resources and RMF PM: Resources and MetricsMetricsMetricsMetrics

    Resources are components of a computing system. The Monitor III data model maps those resouces in terms of a hierarchical scheme

    Each resource is associated with a set of metrics. Individual metrics (or counters) can be selected for data collection and continous display

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation41

    data views with click-sensitve bars link to predefined Analysis PerfDesks

    RMF PM: Intelligent AnalysisRMF PM: Intelligent AnalysisRMF PM: Intelligent AnalysisRMF PM: Intelligent Analysis

    One of the most powerful features of RMF PM is the Intelligent Analysis: once the user has detected an abnormal value, he can navigate to context information or additional views in order to detect the reason of a problem. This is achieved by click-sensitive bars whereby the analysis menu choice offers a selection of various pathes for further drill-down

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation42

    RMF PM Java TM Technology Edition - Linux

    support available on Internet

    Common Viewfor

    z/OS and LinuxData

    LinuxLinuxLinuxLinux Performance Performance Performance Performance MonitoringMonitoringMonitoringMonitoring

    RMF PM has been extended to support the Linux operating system using the existing RMF PM infrastructure. Thus, the PM capabilities (spreadsheet export, filter, historical data collection, graphical trend reports, etc.) are also available for Linux environments

    For the RMF PM client, it is transparent whether the host is a z/OS or Linux system On the server side, two additional components are needed:

    The Linux compiled version of the Distributed Data Server infrastructure The Linux data gatherer backend (collects data out of the procfile system)

    The Linux support can be downloaded free of charge: http://www.ibm.com/servers/eserver/zseries/rmf/pmlin.htm Please note that the Linux support is not an official part of z/OS RMF

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation43

    RMF Monitor III Data PortalRMF Monitor III Data PortalRMF Monitor III Data PortalRMF Monitor III Data Portal

    direct connection to the RMF Distributed Data Server just specify http://:8803 all resources and metrics supported (same than RMF PM)

    Since the RMF Distributed Data Server talks XML over HTTP, the client can be either a fat client like RMF PM or the data can also be accessed directly by means of a web browser. Thats what we call the RMF Monitor III Data Portal

    In other words, the RMF DDS server behaves like a webserver: the data can be requested by means of the standard HTTP protocol. In this case, the presentation is managed by associated formatting instructions in terms of Style Sheets (XSL documents)

    The chart shows the Overview Performance Desktop (=PerfDesk) with the following metrics: % processor utilization by MVS image % total physical utilization (CP) by partition performance index by important WLM service class period % CSA utilization by MVS image

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation44

    RMF Monitor III Data PortalRMF Monitor III Data PortalRMF Monitor III Data PortalRMF Monitor III Data Portal

    Add this Metric to My View

    click

    for Persistence

    select any Resourcefrom Configuration View

    select favoriteMetric from List

    4 Coupling Facility PerfDesk with: Request rates Service times

    Beyond the Overview PerfDesk, the user can customize his own private PerfDesk (aka MyView). Thereby he can select from a rich set of metrics for any kind of resource he wants to monitor contionuously. The definitions are stored persistently in terms of browser cookies

    The example shows a Coupling Facility PerfDesk with the following metrics: sync request rate by CF structure sync service time by CF structure async request rate by CF structure async service time by CF structure

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation45

    RMF Monitor III Data PortalRMF Monitor III Data PortalRMF Monitor III Data PortalRMF Monitor III Data Portal

    Report name Scrollable!

    Combine strengths of Monitor III with state-of-the-art frontend View full Reports and exploit the SORT capability

    All columnsvisible!

    Resizeable!

    Sortable!

    In the past, the Data Portal was limited to display single metrics for selected resources This concept has been extended: with z/OS V1.8 RMF, complete Monitor III reports can be displayed in the browser window Thus, the strengths of the Monitor III ISPF reporting are combined with a state of the art frontend:

    The window is resizable and scrollable All report columns can be displayed (including ISPF hidden fields and the content of popup windows) The columns can be sorted by clicking on the column header Support of automatic refresh (GO mode) Multiple reports can be displayed in parallel (tabbed browsing) Print facility and Spreadsheet export

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation46

    RMF Performance Data RMF Performance Data RMF Performance Data RMF Performance Data APIAPIAPIAPIssss

    ERBSMFI - Monitor II Data (SMF Type 79) RMF Sysplex Data Server (SDS)

    SMF Data: ERBDSQRY, ERBDSRECMonitor III Data: ERB3XDRSMonitor II Data: ERB2XDGS

    RMF Provider for z/OS CIM Server

    RMF offers a couple of powerful APIs to retrieve z/OS performance data. The exploiters can be: Other z/OS components or systems management applications User written applications

    All RMF APIs are documented in the RMF Programmers Guide (SC33-7994)

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation47

    Sysplex Data Server: Services Sysplex Data Server: Services Sysplex Data Server: Services Sysplex Data Server: Services

    all Services are available as High-Level-Language APIs

    Sysplex-wide access to SMF data ERBDSQRY requests a directory of available SMF data in the sysplex ERBDSREC requests SMF record data in the sysplex

    Sysplex-wide access to Monitor III data ERB3XDRS requests a set_of_samples of Monitor III data

    8 does not require an ISPF and Monitor III reporter environment8 provides data reduction features to transfer only the necessary data

    Sysplex-wide access to Monitor II data ERB2XDGS requests Monitor II data according to the specified SMF

    type 79 subtype 8 returns Monitor II snapshot data8 provides data reduction features like ERB3XDRS

    The RMF Sysplex Data Server provides a set of callable services to retrieve performance data from all RMF Monitors Please note, that the SMF data related services are not limited to the SMF 70-78 record types: once the user has specified

    any SMF record type for buffering by the Sysplex Data Server (via the SMFBUF parameter), the record images can be obtained with the RMF services ERBDSQRY and ERBDSREQ

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation48

    Groovy!Groovy!Groovy!Groovy!

    ZZ SessionPostprocessor

    Distributed Data Server

    SysplexData ServerMonitor III

    Spreadsheet Reporter

    CIM Data Provider

    Mintime & Range

    Interval & Cycle Time

    Monitor II

    % Workflow % Using / Delay

    Thats the freshly baked RMF expert! Probably he feels a bit more skilled now. At least he came out of his hiding-place!

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation49

    Information and ToolsInformation and ToolsInformation and ToolsInformation and Tools

    RMF homepage: www.ibm.com/systems/z/os/zos/features/rmf/n Product information, newsletters, presentations, ...n Downloads

    Spreadsheet Reporter RMF PM Java Edition RMF data collector for Linux

    RMF email address: [email protected]

    Documentation and newsn RMF Performance Management Guide, SC33-7992n RMF Report Analysis, SC33-7991n RMF Users Guide, SC33-7990n Latest version of PDF files can be downloaded from:

    http://www.ibm.com/systems/z/os/zos/bkserv/

  • Resource Measurement Facility

    RMF Technical Overview 2010 IBM Corporation50

    RMF Redbook !!!RMF Redbook !!!RMF Redbook !!!RMF Redbook !!!