pre-con ed: best practices for setting thresholds in ca sysview®

World®’16

TheProcessofSettingThresholdsinCASYSVIEWPerformanceManagementTomQuinn,Sr EngineeringServicesArchitectCAMainframeSolutionCenter

MFX02E

MAINFRAMEANDWORKLOADAUTOMATION

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Abstract

The“bestpractice”forsettingmainframemonitoringthresholdsisaprocess,notasetofvariablesandnumbers.Onlyyoucandecidewhatthemostimportantvariablesareandwhatthresholdsshouldbesetforyourshop.

JointhisinformativesessiontolearnhowyoucansetmeaningfulthresholdvaluesinCASYSVIEW®PerformanceManagementthatarespecifictoyourenvironmentandbusinessgoalsforproactiveperformancemonitoring.

TomQuinn

CATechnologiesSr EngineeringServicesArchitect,MainframeSolutionCenter


Agenda

PROCESSOVERVIEW

HOWCASYSVIEW THRESHOLDSANDSTATESWORK

STEPSINTHEPROCESS

INTEGRATINGWITHMAINFRAMETEAMCENTER:OPERATIONALINTELLIGENCE

1

2

3

4


SettingThresholdsinCASYSVIEW®PerformanceManagement

ProcessOverview


TheProcessofSettingThresholdsinCASYSVIEW

Evaluate

Measure

Analyze

Update

Validate



§ Settingthresholdsisanon-going,cyclicalprocessthatshouldbepartofyourmonitoringstrategy.

§ TheinformationyouneedisavailableonyoursystemthroughCASYSVIEWhistoricaldatalogstreams.

§ Businessrequirementsplayanimportantroleinthisprocessaswell.

BestPractice



§ Youmustfirstunderstandwhatis“normal”andthevariabilityofyourworkload.

§ Whatis“normal”?– TimeofDay– TimeofMonth– DayoftheWeek– Workloadtype

BestPractice



Whydowegothroughthisprocess?

§ PerformanceManagementistheprocessofmonitoringandmanagingend-userexperienceandkeyperformanceindicators(KPIs).– Availability– ResponseTime– Throughput– Utilization

PerformanceManagement


HowCASYSVIEWStatesandThresholdsWork



§ CASYSVIEWprovidestheabilitytomonitorandalertuponboththresholdsandstates.– Thresholdsdefinitionsassignnumericwarningandproblemlimit

valuestoametricorresource.Thresholdsarealwaysnumeric.– Statedefinitionsassignastatustoaspecificconditionofametricor

resource.Statesalwayshavecharactervalues:ACTIVE,INACTIVE,ONLINE,OFFLINE.

§ AllavailablevariablesinCASYSVIEWarelistedintheVARScommanddisplay.

HowCASYSVIEWThresholdandStateDefinitionsWork



§ Thresholdruletypesdefinethealgorithmthatisusedtodeterminewhenacollectedmetrichasexceededtheallowableresourceusage.

§ Multipleruletypesorcheckscanbedefinedforasinglemetric:– UPPER- ifthemetricvalueexceedsthewarningorproblemlimitvalues– LOWER- Ifthemetricvaluefallsbelowthewarningorproblemlimitvalues– CHANGE- definestheamountthatametriccanchange,upordown,interms

ofstandarddeviationsfromthehistoricalaverage.




§ STATEsdonothavemultipleruletypes.– STATEexceptionprocessingassignsastatustothecurrentstateofa

metricorresource.

§ Exceptionprocessing,orthresholdprocessing,ispartoftheCASYSVIEWdatacollectionprocessandcontrolledbytheEventScheduler.



TheProcess



Evaluate

Measure

Analyze

Update

Validate



Evaluateistheprocessofreviewingyourmonitoringstrategy.– Areyoudefiningmonitoringatthesystemorapplicationlevel?– Doyouhavebusinessrequirementsformonitoringthesystemandthe

application?– Doyouhavedifferenttiersofmonitoringinplace?

§ Monitoringtiersallowforthedefinitionofmonitoringlevelsbasedontheimportanceofthesystemorapplication.

§ Moreimportant,orhighertieredapps,mayreceivemoremonitoringthanalessimportant,lowertieredapp.

Evaluate



DeterminingImportance\MonitoringTiers– Disasterrecoveryplan– Businessrequirements

§ Eachtimeyouareintheevaluationphaseofthisprocessyouwillreviewallsystems,applicationsorbusinessrequirementstolookforneworchangedrequirements.

Evaluate



Evaluate

Measure

Analyze

Update

Validate



Measureistheprocessofmonitoringyoursystemsandgatheringthedatarequiredtoanalyzebothyoursystemdataandtheeffectivenessofyourthresholds.

§ ByhavingCASYSVIEWinstalled,configuredandcontinuallyrunningandcollectingdatayouhavetheabilitytomeasureyoursystem’sperformanceandtheeffectivenessofyourthresholds.

Measure



§ InCASYSVIEW,itisrecommendedtocollectexceptiondatabyutilizingtheExceptionLogcapabilities(XLOG),andhistoricaldatathroughthevariousCASYSVIEWlogstreams andCASYSVIEWSMF255records.

§ BesidestheXLOG,theotherlogstreams includeIMSregionandsummaryrecords;MQhistoryrecords;CICSdetail,summaryandsystemrecords;andhistoricalPLOTrecords.

Measure



§ AuditlogandCICSExecInterfaceTraceareavailablelogdatatypesthatarenotnecessarilyusedinthisprocess.

§ Itisrecommendedyougatherthehistoricalrecordtypesthatyouareabletoinordertobestprovideyouwiththenecessarydatatomakemeaningfuldecisionsregardingthresholdlevels.Thisdatawillbereviewedbothon-lineandthroughbatchreporting.

§ https://docops.ca.com/display/CSPM10/Usage+of+SMF+Records

Measure



Evaluate

Measure

Analyze

Update

Validate



Analyze istheprocessofreviewingyourmeasureddatatodeterminewhatmetricsneedthresholds,whatthosethresholdsshouldbe,andhoweffectiveyourthresholdsare.

Theveryfirsttimeyougothroughthisprocess,thiswillbethemosttime-consuming.Analyzemaybethemostimportantstepintheprocesscycle.

Analyze



WheredoIbegin?

§ CASYSVIEWcontainsmanyoutoftheboxthresholdsasexamples

§ AgoodplacetostartiswiththeCASYSVIEWVARS command– VARSlistsallofthemetricsthatcanbeusedforthresholding

Analyze



PLOTLOG

§ Asanexample,PLOTLOGdisplayshourlyintervalrecordsforz/OSandsubsystemdatacollectionmetrics.

§ Alogcanpotentiallycontainalargenumberofrecords,thereforeitisrecommendedtousetheFILTERcommandwithinthePLOTLOGcommanddisplay(CASYSVIEWr14.1andgreater)tolimitthedatayouwishtosearchandreturn.

Analyze



PLOTLOGExample– CPUusage

Analyze


TheProcessofSettingThresholdsinCASYSVIEWAnalyze

CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP CP12:00:00AM

1:00:00AM

2:00:00AM

3:00:00AM

4:00:00AM

5:00:00AM

6:00:00AM

7:00:00AM

8:00:00AM

9:00:00AM

10:00:00AM

11:00:00AM

12:00:00PM

1:00:00PM

2:00:00PM

3:00:00PM

4:00:00PM

5:00:00PM

6:00:00PM

7:00:00PM

8:00:00PM

9:00:00PM

10:00:00PM

11:00:00PM

CA11AverageofAverage 0.3540.3220.3210.3940.4420.4010.4220.3840.4150.4580.556 0.58 0.5760.5890.5550.5130.4710.4590.4840.4850.4770.4370.5380.397

AverageofMinimum 0.173 0.17 0.1640.2170.2180.2230.2150.2140.2260.2780.3630.3930.404 0.42 0.3830.3480.3240.2940.3340.3460.3310.2730.2920.209

AverageofMaximum 0.6910.6070.6290.6990.7210.6990.7240.6330.7390.7620.771 0.79 0.7320.7790.7760.7460.6770.6480.6870.6620.6640.6940.8510.687

00.10.20.30.40.50.60.70.80.9

%

AVGGP%byHour01/17- 02/04/2016



Thisdatashouldstarttogiveinsighttoyoursystem.– Aretherecertaintimesofdaysweseespikes?– Arethereperiodsoftimethatareconsistentlyhigherorat100%?

Basedonthesetrends,thinkagainaboutyourworkload.– Timeofday– batchwindow?– Timeofmonth,year

Onceyoustarttounderstandyourdata,systemandbusinessrequirementsyoucanstarttosetmeaningfulthresholds.

Analyze



CICS

§ MENUCHISTORYisagreatplacetostarttolookatyourCICShistoricaldata.– Remember:youmustbecollectingCICShistoricaldataintheCA

SYSVIEWlogstreams tousethesecommands.– CREVIEW– byregion,thenumberoftransactionsthatraninthepast

24hoursandanhourlybreakdownofthosetransactions.– CSYSDATA– byregion,15-minuteintervalsummary.

Analyze



CICS

§ Historycontinued– CTRANLOG– everyCICStransactionthatranonyoursystem.

§ Usefulforsettingtransactionspecificthresholdsorexclusionrules.– CTRANSUM– summaryofCTRANLOGin15-minuteintervals.

Analyze



ReportWriter

§ Batchreportingagainstthissamehistoricaldatathatisbeingaccessedbytheonlinecommands.

§ Asanexample– allCICStransactionsthatexecutedeachday,theaverage,minimum,maximumlifetimeandCPU.

Analyze



Copyright (C) 2015 CA Technologies, Inc. EXPLORE Report Writer Release: 14.1 07/06/15 20FROM 01/27/16 20.27.00 MONTHLY CICS TRANSACTION DETAIL REPORT 1 PAGE 1

TO 01/30/16 20.27.25 RESPONSE TIME AND CPU ANALYSIS REPORT DATE 02/10/16 11.44

SHIFT NONEEACH MONTHPERIOD NONE

TRAN TRANS AVG TRANS MIN TRANS MAX TRANS AVG MIN MAXDATE ID COUNT LIFETIME LIFETIME LIFETIME CPU CPU CPU

-------- -------- -------- ---------- ---------- ---------- ---------- ---------- ----------01/01/16 CSOL 137 1887.437 1887.424 1887.552 0.000154 0.000125 0.001341

CWBG 72 0.002459 0.000404 0.137902 0.000397 0.000349 0.000653CRSQ 9 0.000449 0.000371 0.000677 0.000360 0.000317 0.000601CRMF 6 0.004195 0.000452 0.022358 0.000598 0.000401 0.001314TKOA 1 3.967686 3.967686 3.967686 0.371662 0.371662 0.371662CSKL 1 1.734323 1.734323 1.734323 0.028685 0.028685 0.028685CSFU 1 0.539743 0.539743 0.539743 0.008386 0.008386 0.008386CATR 1 0.010191 0.010191 0.010191 0.001518 0.001518 0.001518

-------- -------- -------- ---------- ---------- ---------- ---------- ---------- ----------SUMMARY 228 1134.146 0.000371 1887.552 0.002047 0.000125 0.371662

-------- -------- -------- ---------- ---------- ---------- ---------- ---------- ----------

Analyze



§ Onceyouhavethisinformationyoucanstarttobuildyourthresholddefinitions.

§ Askyourselftheseadditionalquestions:– Aretherecompanystandardsforwarningorproblemthresholds?

§ Ex.Awarningthresholdis10%lessthanaproblemthreshold– AretherebusinessrequirementsforCPUor elapsed(response)time?– Aretheretransactionsthatarecriticaltoourbusiness?

Analyze



Evaluate

Measure

Analyze

Update

Validate



Updateistheprocessofaddingneworupdatingexistingthresholdsasneededbasedonyouranalysisofthedata.

§ InCASYSVIEWthisisdoneeitherthroughthecommanddisplaysorthroughupdatingtheCASYSVIEWPARMLIBmembers.– WhichonedependsonhowCASYSVIEWanditsdatacollectorsare

started– WARMvsCOLD.

Update



SubSystem

CASYSVIEWCommand PARMLIBMember

Thresholds States Threshold States

z/OS THRESH STATES MVSTHRSH MVSSTATE

CICS CTHRESH CSTATES CTHRESH CSTATES

IMS IMSTHRSH IMSSTATE IMSTHRSH IMSSTATE

MQ MQTHRESH MQSTATES MQSTHRSH MQSSTATE

TCPIP TCPTHRSH TCPSTATE TCPTHRSH TCPSTATE

Update



Evaluate

Measure

Analyze

Update

Validate



Validate istheprocessofverifyingyourthresholdsareaccuratelybeingtriggeredandexceptionsareraisedasexpected.

ALERTS– TovalidateyourthresholdlevelslookattheALERTSandALERTSUMcommands

withinSYSVIEW.IfyouarewritingexceptionstotheXLOGyoucanviewalltriggeredexceptionsthroughtheXLOGcommandandalsorunbatchreportsusingtheSYSVIEWReportWriteragainsttheXLOGdatasets.

Validate



ExamplereportagainsttheCASYSVIEWExceptionLog(XLOG)START VARIABLE RESOURCE EXC THRSHOLD MAX MIN WARNING PROBLEMDATE TIME NAME 1 STATUS VALUE VALUE VALUE VALUE VALUE

-------- ------------ -------- -------- -------- -------- -------- -------- -------- --------03/17/16 01.17.00.010 IMPLMAX IMSD WARNING 103K 105K 105K 98304 131K

01.17.00.011 IMPLSIZE IMSD PROBLEM 101K 104K 104K 60000 8000001.18.00.014 IMPLMAX IMSD WARNING 104K 106K 106K 98304 131K01.19.00.012 IMSLFULL IMSD PROBLEM 1 1 1 1 101.21.00.024 IMPLSIZE IMSD NORMAL 57344 57344 57344 60000 8000001.23.00.010 IMPLSIZE IMSD WARNING 73728 73728 73728 60000 8000001.24.00.017 IMPLSIZE IMSD NORMAL 32768 32768 32768 60000 8000001.26.00.018 IMPLSIZE IMSD PROBLEM 102K 104K 104K 60000 8000001.27.00.019 IMPLSIZE IMSD PROBLEM 104K 106K 106K 60000 8000001.31.00.014 IMSLARCH IMSD NORMAL 1 101.31.00.016 IMSLFULL IMSD NORMAL 1 101.34.00.006 IMSLARCH IMSD PROBLEM 1 1 1 1 1

Validate



Evaluate

Measure

Analyze

Update

Validate


MainframeTeamCenter:OperationalIntelligence



§ SYSVIEWr14.2introducedMainframeTeamCenter:OperationalIntelligence(MTC-OI)




§ ThecorebenefitsofMTC-OI– MTC-OIisaboutpredictiveanalysis,creatingdynamicthresholdsbased

onhistoricalperformance

§ ThereareotherbenefitsofMTC-OIaswell– YoucanvisualizeandexplorehistoricalCASYSVIEWdatathathasbeen

analyzedtodetermineanormalbaseline– MTC-OImodelsthishistoricaldataprovidedbyCASYSVIEWand

determineswhat’snormalforyourenvironment




§ MTC-OIenablesyoutodothefollowing:– Easilyaccesshistoricaldatafromyoursystem.– Viewahistoricalgraphofametricthatisoverlaidwithactualdataand

coloredbandsthatillustratelikelinessofmetricperformance.A"greenhighway"isproducedfromanalysisusingWesternElectricrules.

– Performanalysistoanswerquestionsaboutaspecificmetric,suchas:§ Whatisnormalforthismetricinmyshop?§ Isthecurrentvalueofthismetricoutsideoftheexpectedvalue?§ Basedonpastobservations,howtightlycouldIsetathresholdwithoutproducingtoomanyfalse-positivealarms



TheProcessofSettingThresholdsinSYSVIEWPM

SohowdoesthisprocesschangewithMainframeTeamCenter-OperationalIntelligence?

§ WithMTC-OItheAnalyzephasewillimproveforsomemetricsbymodelingthehistoricaldataforyouandgraphicallydisplayingtheexpectedbehavior.– Caneliminatetheneedtorunbatchreportsagainstyourhistoricaldata

todeterminetheseboundaries




WhatdataismodeledinMTC-OI?

§ SYSTEMIntervalData– PLOTLOG 255.2records

§ CICSTransactionSummaryData– CTRANSUM 255.25records

§ CICSSystemIntervalData– CSYSDATA 255.28records




§ MTC-OIcanassistyouintheprocessofsettingthresholdsinSYSVIEW

§ MTC-OIgeneratesitsownpredictivealertsafteranalyzingandmodelingyourdataandwillcontinuetoadjustdynamicthresholdsasmoredataisprovidedintothemodelingengine

§ Theprocessofsettingthresholdsrequiresyoutocombinebusinessrequirementsandsysteminformationtousethemostmeaningfulvalues.



RecommendedSessionsSESSION# TITLE DATE/TIMEMFX154E Ideation:IntelligentMainframeManagement– BePartof

ShapingtheFuture! 11/14/2016at4:45pm

MFX05E PredictingtheFirewithOperationalIntelligence11/15/2016at9:00am

MFX06E HowtoGetOperationalIntelligenceUpandRunninginaDay 11/15/2016at10:00am

MFX07E SoYou’reaNewSystemsOperator– WhoYa Gonna Call?11/15/2016at11:00am

MFX144E ImprovingYourResponsetoEvents– PerformanceFirstResponders 11/15/2016at1:00pm

MFX145E OperationalIntelligence– MonitoringthePasttoBetterUnderstandToday 11/15/2016at2:45pm

MFX146E MainframeOperationalIntelligence– PredictingtheFuture 11/15/2016at4:00pm


MustSeeDemosintheMainframeTheater:

IntelligentMainframe:AdvancedAnalytics

CAMainframeOperationsIntelligence:Performance&NetworkMgmtApplicationMonitoringEvents&Automation

IntelligentMainframe:Automation

CAMainframeOperationsIntelligenceCAOPS/MVS®EventManagement&Automation

IntelligentMainframe:End-to-EndPerformance

CACross-EnterpriseAPMCAMainframeApplicationTunerCAUIMforzSystems

IntelligentMainframe:StorageManagement

CAVantageSRMCA1®TapeManagementSystem


Questions?


Thankyou.

Stayconnectedatcommunities.ca.com

pre-con ed: best practices for setting thresholds in ca sysview®

Technology