CICS Automation IS Mainframe Automation

CICS Automation

V e r s i o n 1 . 0 0

CICS Automation

Introduction

Copyright Owner© Copyright 2009 Perot Systems Corporation. All Rights Reserved.

DistributionCopies of this document are restricted to the following:

Perot Systems, and its wholly owned companies and/or subsidiaries including but not limited to:

PSC- EuropePSC – AsiaTSI CorporationTSI AmericaPSGSCSRGARSDelphi GroupSolutions ConsultingVision

Document ControlThis document is available in two forms: controlled and uncontrolled.The controlled master copy is maintained electronically and accessed by authorized users of the Perot Systems Corporation network.Uncontrolled versions are all other electronic and printed copies of this document.

Terms and ConditionThis document is strictly “Commercial in Confidence” and is subject to the prevailing terms and conditions under which it is both written and distributed.

CICS Automation

Continuous ImprovementThe process defined in this document is subject to regular review based on input from PSC associates. Suggestions for improvement to the content of this document should be submitted to the IS Mainframe Automation team.

CICS Automation

Version HistoryVersion Date Revision Description (Author)

1.00 04/10/2009 Initial Draft (Laurie Ostrander)1.01 07/10/2009 Updated for upgraded automation (Laurie

Ostrander)

Document Peer ReviewVersion Name Signature/Date

1.00

Final Document ApprovalVersion Name Signature/Date

1.00

CICS Automation

PurposeDocument the standardized, best-practices CICS monitoring automation (developed and installed under BMC Control-O in the CVS LPARs), designed to be ported to any other leveraged LPAR with Control-O in use or to be used as a model for OPS/MVS LPARs.

NotesRelated documents:

o IPL o Task Abend o Task Monitoring

CICS Automation

Standardized Automation

CICS Monitoring Automation

OverviewCICS monitoring is in place to detect region initialization failures and shutdown delays, short-on-storage events, and to provide a scheduled and controlled recycle event.

Components

Non-managed tasks database: Name: NONMANGD Each CICS task occupies a row of the database Each row contains six columns of data Columns defined:

o TASKNAME (name of region)o IDENT (region identification (short name))o GROUPA (abbreviated group name; region’s owner)o VENDOR (vendor of the product)o DESCR (description of the region)o CLASS (region classification. Values are: PROD, UAT, DEV, TEST)

Example of entries:

CICS Automation

Temporary global variable pool: Name: GLVTEMPA Contains frequently used/updated task information Built at system IPL from hard-coded values and values obtained from the

MANAGED and NONMANGD databases and refreshed at regular intervals and by events trapped by the various message rules. (See Task Monitoring process doc)

Variables for CICS tasks:o THROTTLE_throttleID (used to throttle notifications and recovery actions)o regionname_SOS-type_INVOKED (used to throttle notifications and recovery

actions)o regionname_SOS-typeR (switch to track SOS and SOS Relieved conditions)o CURRENT (region’s current state)

Example of temporary task global variables for a CICS task:%%GLVTEMPA\THROTTLE_TSTCCICS_DFHSM0133 = 921923040 03/19/2009 09:44:00%%GLVTEMPA\TSTCCICS_SOSA_INVOKED = 090319

CICS Automation

%%GLVTEMPA\TSTCCICS_SOSAR = 0%%GLVTEMPA\CKP1CICS_CURRENT = ACTIVE

BMESSAGE.DFHSM0131– message rule: Purpose: Sets short-on-storage relieved flag and invokes BRULES.SOSR Message examples:

DFHSM0131 PRD2CICS CICS IS UNDER STRESS (SHORT ON STORAGE BELOW16MB). DFHSM0132 PRD2CICS CICS IS NO LONGER SHORT ON STORAGE BELOW 16MB DFHSM0133 TSTCCICS CICS IS UNDER STRESS (SHORT ON STORAGE ABOVE16MB). DFHSM0134 TSTCCICS CICS IS NO LONGER SHORT ON STORAGE ABOVE 16MB

Triggered by the appearance of any of the above message IDs issued by jobname matching mask ????CICS and jobtype “S” (started task)

Global variables used:o %%GLVTEMPA\regionname_SOS-typeR

Sets regionname to the jobname (issuer of message) Sets short-on-storage type (SOS-type) to “0” (short on storage) or “1” (relieved)

based on messageID (“SOSA” vs. “SOSB”) Invokes BRULES.SOSR if a short-on-storage message (DFHSM0131 or

DFHSM0133). Example global variable:

%%GLVTEMPA\PRD2CICS_SOSBR = 1

BMESSAGE.DFHTM1715– message rule: Purpose: Notify Interactives team if region is not down within ten minutes Message examples:

DFHTM1715 TSTCCICS CICS IS BEING QUIESCED BY USERID PRDSCHD IN TRANSACTION CEMT AT TERMINAL CJCL.

Triggered by the appearance of the message IDs issued by jobname matching mask ????CICS and jobtype “S” (started task)

Global variables used:o %%GLVTEMPA\regionname_CURRENTo %%$GLOBAL\SYS_ID

Database columns used:o TASKNAMEo IDENTo CLASS

Sets regionname to the jobname (issuer of message) Sets CURRENT to the value of “STOPPING” Waits for ten minutes Checks if the value of CURRENT is still “STOPPING”. If so,

CICS Automation

o Sets up default ticketing values as if the CLASS were equal to a value of PROD/UAT: SEV 3 Immediate notification

o Invokes MONITOR.GETNMNGD passing the regionname to obtain the database column values

o Checks if CLASS is undefined If so, no further action (the rule terminates) as this indicates that the

region was not found in the database, therefore no automation is expected to be performed.

o Checks if CLASS is equal to the value of “DEV” If so, changes the ticketing values to:

SEV 4 Delayed notification

Otherwise, checks if CLASS is equal to the value of “TEST” If so, issues a highlighted informational message to the console, clears

it after two seconds, and exits out with no further action.o Builds and issues the OPAS and Alert Manager messages

Example global variables:%%GLVTEMPA\TSTCCICS_CURRENT = STOPPPING%%$GLOBAL\SYS_ID = 3E01

Example informational messageDFHTM1715: TST3CICS is in test mode - no ticketing for shutdown delay.

Example OPAS and Alert Manager messagesQAUT000I TSTCCICSSD,4,3E01,QOL_NOPAGE,03/19/09,09:44:00,TSTCCICS shutdown issued, but region has not endedQAU0CW12 TSTCCICS shutdown issued, but region has not ended - Interactives notified (Delayed)

BMESSAGE.IEF695I– message rule: Purpose: Notify Interactives team if region has not initialized within the

determined wait period (15 minutes for PROD and UAT regions; 60 minutes for DEV and TEST regions)

Message examples:IEF695I START TSTCCICS WITH JOBNAME TSTCCICS IS ASSIGNED TO USER TSTCCICS, GROUP CICSRGNS

Jobname must not be XPEDCICS Triggered by the appearance of the message IDs issued by jobname matching

mask ????CICS and jobtype “S” (started task) and matching the criteria above Global variables used:

o %%GLVTEMPA\regionname_CURRENTo %%$GLOBAL\SYS_ID

Database columns used:o TASKNAME

CICS Automation

o IDENTo CLASS

Sets regionname to the jobname (issuer of message) Sets CURRENT to the value of “STARTING” Sets up default ticketing values as if the CLASS were equal to a value of

PROD/UAT:o SEV 3o Immediate notification

Sets max-loop local variable to a value of “30” Invokes MONITOR.GETNMNGD passing the regionname to obtain the database

column values Checks if CLASS is undefined

o If so, no further action (the rule terminates) as this indicates that the region was not found in the database, therefore no automation is expected to be performed.

Checks if CLASS is equal to the value of “DEV” or “TEST”o If so, changes the ticketing values to:

SEV 4 Delayed notification Sets max-loop local variable to a value of “120”

Loops a maximum of max-loop times, checking CURRENT for a value of “STARTING” every thirty seconds, and exiting the loop when no longer true

Checks if the value of CURRENT is still “STARTING”. If so, o Checks if CLASS is equal to the value of “TEST”. If so,

Issues a highlighted warning message to the console, clears it after two seconds, and exits out with no further action.

o Builds and issues the OPAS and Alert Manager messages Example global variables:

%%GLVTEMPA\TSTCCICS_CURRENT = STARTING%%$GLOBAL\SYS_ID = 3E01

Example informational messageDFHSI1517: TST3CICS is in test mode - no ticketing for initialization delay.

Example OPAS and Alert Manager messagesQAUT000I TSTCCICSID,4,3E01,QOL_NOPAGE,03/19/09,09:44:00,TSTCCICS startup issued, but region has not initializedQAU0CW12 TSTCCICS startup issued, but region has not initialized - Interactives notified (Delayed)

BRULES.SOSR – invoked rule: Purpose: Notify Interactives team not more than once per hour per region of a

short on storage condition and obtain diagnostic information. Not more than once per day, set new EDSALIMIT (if SOS above the line), dump, recycle (if a test region), and create a second ticket to Interactives if SOS is not relieved within five minutes.

CICS Automation

Invoked by BMESSAGE.DFHSM0131 Global variables used:

o %%$GLOBAL\SYS_IDo %%GLVTEMPA\THROTTLE_regioname_messageIDo %%GLVTEMPA\regionname_SOS-type_INVOKEDo %%GLVTEMPA\regionname_SOS-typeRo %%GLVTEMPA\regionname_CURRENT

Database columns used:o TASKNAMEo IDENTo CLASS

Sets regionname to the jobname (issuer of message) Invokes REXX.THROTTLE passing regionname_messageID as the throttle ID and

3600 (seconds) as the time parametero If REXX.THROTTLE returns with a return code of 8, no further action (the rule

terminates) as this indicates that less than one hour has elapsed since the last event.

Invokes MONITOR.GETNMNGD passing the regionname to obtain the database column values

Checks if CLASS is undefinedo If so, no further action (the rule terminates) as this indicates that the region

was not found in the database, therefore no automation is expected to be performed.

Issues commands F ident,CEMT P SNAP and F ident,CEKL I TASK Sets notification text and event ID suffix based on messageID (“above the line”

vs. “below the line”) Sets up ticketing values:

o SEV 4o Delayed notification

Builds and issues the OPAS and Alert Manager messages Checks if INVOKED exists

o If so, checks if the value is equal to the system date If so, exits out with no further action, as this indicates that this logic has

already been performed once today Sets INVOKED to a value of today’s system date If SOS-type is SOSA,

o Waits for three minuteso Otherwise, waits for five minutes

If SOS-type is SOSA and the value of SOS-typeR is “0” (not relieved),o Issues command F ident,CEMT I SYSo Waits ten seconds for a responseo Loops through the message response lines to obtain the value of EDSALIMIT

CICS Automation

If found, sets a new EDSALIMIT value to the retrieved value plus ten million into a local variable

o Checks if the new EDSALIMIT local variable exists (this indicates that the EDSALIMIT value was successfully retrieved. That is, the region responded to the CEMT I SYS command.) If so, issues command F ident,CEMT S SYS EDSALIMIT(new EDSALIMIT)

o Waits for two minutes Checks if the value of SOS-typeR is still “0” (not relieved). If so,

o Issues command DUMP COMM=(regionname SOS NOT RELIEVED)o Waits for response message IEE094D o Replies R nn,JOBNAME=(regionname)o Waits for response message IEA911E

o Checks if CLASS is equal to the value of “PROD” or “UAT” If so, changes the ticketing values to:

SEV 3 Immediate notification

o Builds and issues the OPAS and Alert Manager messages Checks if CLASS is equal to the value of “TEST”. If so,

o Issues command CANCEL idento If after one minute, the region is not INACT, builds and issues OPAS and Alert

Manger messageso Otherwise, issues command OPCICSUP regionname

Example global variables:%%GLVTEMPA\THROTTLE_TSTCCICS_DFHSM0133 = 921923040 03/19/2009 09:44:00%%GLVTEMPA\TSTCCICS_SOSA_INVOKED = 090319%%GLVTEMPA\TSTCCICS_SOSAR = 0%%GLVTEMPA\TSTCCICS_CURRENT = ACTIVE%%$GLOBAL\SYS_ID = 3E01

Example OPAS and Alert Manager messagesQAUT000I TSTCCICSSOSA,4,3E01,QOL_NOPAGE,03/19/09,09:44:00,SOS (above the line) detected for TSTCQAU0CW10 SOS (above the line) detected for TSTC - Interactives notified (Delayed)QAUT000I TSTCCICSSOSANR,4,3E01,QOL_NOPAGE,03/19/09,09:44:00,SOS (above the line) not relieved for TSTC QAU0CW12 SOS (above the line) not relieved for TSTC - Interactives notified (Delayed)QAUT000I TSTCRYCL,4,3E01,QOL_NOPAGE,03/19/09,09:44:00,TSTC not responding to cancel commandQAU0CW12 TSTC not responding to cancel command - Interactives notified (Delayed)

MONITOR.GETNMNGD– invoked rule: Purpose: Obtains the NONMANGD database column values of the passed

regionname Invoked by various rules

CICS Automation

All NONMANGD database columns are made available to the calling rule Valid parm:

regionname (A specific region) Sets $GLOBAL system variable to NONMANGD Sequentially reads each row of the NONMANGD database

o When the retrieved TASKNAME matches the passed regionname, exits the sequential read loop

If the sequential read counter is greater than the value obtained from the DBCOUNT system variable, sets $GLOBAL system variable to $GLOBAL which in effect sets all database columns to the value of UNDEF to the calling rule (i.e., the task was not found in the database)

Returns to the calling rule REXX.THROTTLE – invoked rexx exec:

Purpose: Determines if the specified event has exceeded the specified interval wait time

Invoked by various rules Valid arguments:

eventID interval

Global variables access:%%GLVTEMPA\THROTTLE_eventID

Parses the arguments Sets local variables u_date (current date in the format: mm/dd/yyyy), c_time

(current time in the format: hh:mm:ss) and c_secs (conversion of current date and time to number of elapsed seconds since 1980)

Sets the return code to a value of “0” Obtains the first word of THROTTLE_eventID into local variable eventid_value

o If a not found error is returned, then sets THROTTLE_eventID to a value of the values of c_secs u_date c_time

o Otherwise, checks if the value of c_secs is larger than the value of eventid_value plus interval. If so, updates THROTTLE_eventID to a value of the values of c_secs

u_date c_time If not, sets the return code to a value of “8” (i.e. this event has recurred

before the specified interval) Exits with the value of the return code (“0” or “8”) Example Throttle global variable:

%%GLVTEMPA\THROTTLE_TSTCCICS_DFHSM0133 = 921923040 03/19/2009 09:44:00

Process Improvement Suggestions