case study: verizon - ca workload automation esp edition: best practice’s, devops, change...
Post on 10-Jan-2017
72 Views
Preview:
TRANSCRIPT
World®’16
CaseStudy:Verizon- CAWorkloadAutomationESPEdition:BestPractice’s,DevOps,ChangeManagementUtilizingCAEndevor®DaveKovaleski - MTSIVSpecialist- Verizon
MFX138S
MAINFRAMEANDWORKLOADAUTOMATION
2 @CAWORLD#CAWORLD
©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.
Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.
ForInformationalPurposesOnlyTermsofthisPresentation
3 @CAWORLD#CAWORLD
Abstract
VerizonTelecommunicationsmadeCAWorkloadAutomationESPEdition(CAWAESP)thestandardtoolforworkloadmanagementacrosstheWirelineandWirelessorganizationforbothmainframeanddistributedsystemsusingtheCAWAESPagenttechnology.Inthissession,theywillshareinsightsonhowtheysuccessfullyestablishedCAWAESPpresencebyinstillingVerizon'sschedulingbestpractices,DevOps,andchangemanagementutilizingCAEndevor®.
DaveKovaleskiVerizonMTSIVSpecialist
4 @CAWORLD#CAWORLD
VerizonBestPractices,DEV-OPS,ChangeManagementUtilizingCAEndevor®DaveKovaleski
November16,2016
5 @CAWORLD#CAWORLD
Agenda
INTRODUCTION
SCHEDULINGBESTPRACTICES
Q&A
DEV-OPSPROCESSING
CAWAESPANDCAENDEVOR®
SUMMARYANDCLOSINGREMARKS
1
2
3
4
5
6
6 @CAWORLD#CAWORLD
Introduction
VerizonITusesbestpracticesthatsupportandenableaworldclasscustomerexperience.TheProductionControl
departmenthasimplementedbestpracticeswhichincludeusingCAWAESPglobalvariablesandsymlibsasacentral
repositorytostoreapplicationartifactssuchasagent,path,anduserid’s.ThelargeapplicationsprocessedbyVerizon
benefitfromthevaluesthatareusedgloballyacrossapplications.Otherbestpracticesincludeimplementingnaming
standardsrelatedtoeachapplicationt hatsupportDEV-OPSsolutionsforapplicationteams.SomeofVerizon'sapplication
teamsperformtheirownapplicationstopandstarts.Inaddition,VerizondevelopersarenoweffectivelyusingCAWAESP
WebServicesinaDEV-OPStool.VerizonalsohasintegratedtheCAEndevor®SoftwareChangeManager(CAEndevor®)
softwaretotrackapplicationchangesthroughtheCAEndevor®software.
7 @CAWORLD#CAWORLD
ExperiencesVerypositivefeedbackfromtheapplicationteamsandschedulers.CommentsInclude:“Provideonestopupdateformanagingservernamechanges”&“Easytomigratefromtesttoprodbyupdatingvariablesinonelocation!”
BestPracticeDefiningandCreatingGlobalVariablesorSymboliclibrariestostoreagent,scriptname,UserIDinformation.
FindingsTheuseofGlobalVariablesandSymbolicLibrariesprovidesauserwithacentralizedlocationtoupdateandreferencevaluesthatwillbeutilizedgloballythroughoutanapplication.
BestPracticesGlobalVariablesandSymbolicLibraries
8 @CAWORLD#CAWORLD
BestPractices
Mainframe GlobalVariableandSymlib(Member) Variable Name=‘Value’
IPSNESP.CMD.CNTL.SYMLIB.VISN(CMDNS) AGENT='TWASTARDP'FILE_PATH='/starload/DNS'LOG_PATH='/starload/DNS/logs'SCRIPT_PATH='/starload/DNS/process'USERID='5staradm'
GlobalVariableandSymbolicLibraryExample
9 @CAWORLD#CAWORLD
BestPractices
CAWAESPApplicationandJobDefinition EventSimulation
APPLCMD#DBCU
INVOKE'IPSNESP.CMD.CNTL.SYMLIB.VISN(CMDNS)‘
UNIX_JOBDNSCM_BATCH_CBR_UPDATERUNDAILYAGENT%AGENTARGS%SCRIPT_PATH./set_env_dns_callmanager.ksh>>+
%FILE_PATH./callmanager/batchCbr.log2>&1USER%USERIDSCRIPTNAME%SCRIPT_PATH./dnscm_batch_cbr_update.ksh
ENDJOB
INVOKEIPSNESP.CMD.PROCLIB.VISN(CMD#DBCU)APPLCMD#DBCU
INVOKE'IPSNESP.CMD.CNTL.SYMLIB.VISN(CMDNS)'AGENT='TWASTARDP'FILE_PATH='/starload/DNS'LOG_PATH='/starload/DNS/logs'SCRIPT_PATH='/starload/DNS/process'USERID='5staradm'
UNIX_JOBDNSCM_BATCH_CBR_UPDATERUNDAILYAGENTTWASTARDPARGS/starload/DNS/process/set_env_dns_callmanager.ksh>>
/starload/DNS/callmanager/batchCbr.log2>&1USER5staradmSCRIPTNAME/starload/DNS/process/dnscm_batch_cbr_update.kshENDJOB
GlobalVariableandSymbolicLibraryExample
10 @CAWORLD#CAWORLD
ExperiencesVerypositivefeedbackfromtheapplicationteamsandschedulers.CommentsInclude:“Ineedtofocusonmyworkloadandthesetupavoiddistractionsfromoutsideapplications!”
BestPracticeNamingstandardsaredefinedforapplications,eventsandprocedurelibraries.
FindingsTheuseofstandardizednamesallowsouruserstoonlyviewandaccesstheirworkloadwhensubscribingtoworkloadintheCAWAESPWorkstation.Thestandardsalsoalloweaseofmonitoringandsupportforbothouroperationsandapplicationteams.
BestPracticesApplication/Event/ProcLibraryNamingStandards
11 @CAWORLD#CAWORLD
BestPractices
EventNamingStandard
EventnamesaredefinedasEventprefix.Event name.Theeventprefixnamewillbedeterminedbywhatenvironmenttheapplicationisrunningon.
TheEventprefixisdefinedwiththefollowingformat:
$VZWXAAA
Where:
$VZWX-Thefirst4charactersoftheeventprefixwillalwaysstartwith$VZWXAAA- Thelast3characters(5-7)aredefinedasthe3alphaApplicationCharacterCode
TheEventnamewillbetheapplicationnamethatistriggeredbytheevent.
Example:$VZWXCMD.CMD#DBCU
| |Event Prefix ApplicationName
Application/Event/Proclib StandardizedNamesExample
12 @CAWORLD#CAWORLD
BestPractices
Application NamingStandard
Applicationnamesarelimitedto8charactersandaredefinedwiththefollowingformat:
AAAEXXXX
Where:
AAA- Charactersaredefinedasthe3 alph charactercodeE- CharactersaredefinedastheEnvironmentNameXXXX - CharactersaredefinedasFreeForm
Example:
CMD#DBCU
CMD# DBCU|| |
ApplicationEnvironment FreeForm
Application/Event/Proclib StandardizedNamesExample
13 @CAWORLD#CAWORLD
BestPractices
MainframeProclib NamingStandard
MainframeProclibs containApplicationnamesarelimitedto8charactersandaredefinedwiththefollowingformat:
IPSNESP.AAA.PROCLIB.VISL
Where:
IPSNESP– DoesnotchangeAAA- Charactersaredefinedasthe3 alphacharactercodePROCLIB– DoesnotchangeVIS– DoesnotchangeL–MainframeLocationCode
Example:
IPSNESP.CMD.PROCLIB.VISN| |
Application North
Application/Event/Proclib StandardizedNamesExample
14 @CAWORLD#CAWORLD
ExperiencesVerypositivefeedbackfromtheapplicationteamsandschedulers.CommentsInclude:“Controllingourworkloadreducesdowntimethatourcustomersmayhaveincurred!”
Dev-OpsOptionAnoptionwasimplementedbytheSchedulingteamtoallowapplicationteamstostopandstartalloftheirproductionjobs.Theschedulingtechniqueisheavilyusedwhenthereisscheduledmaintenance.
FindingsByimplementinganoptiontostopandstarttheirjobs,theapplicationteamcancontrolandnotrelyonoperationstostopandstarttheirworkload.ThisfacilitatesaDEV-OPSsolutiontoourapplicationteams
Dev-OpsApplicationTeamDev-OpsOptionStop/StartProductionJobs
15 @CAWORLD#CAWORLD
Dev-Ops
Step1:Create ThresholdResource
RESDEFOTA_PRD_DATA_OTA_EASTBATCHADDENTERPRISETHRESHOLD MAX(1)
StopandStartSetup
16 @CAWORLD#CAWORLD
Dev-Ops
Step2:Create StopFileTriggerApplication
APPL OTAECFT/*Thisappl containsallOTAEASTCFTjobs*/
FILE_TRIGGEROTAE_STOP_FT/*TriggersOTAEASTStopEvent$VZWXOTA.OTAESTOPRUNDAILY
AGENTOTAEASTFILENAME'/ota/data/adm_ota/otaStopEastESP'CREATENOCHANGE(1)CONTINUOUS($VZWXOTA.OTAESTOP)DUEOUTEXEC23:59EASTERNNOTIFYOVERDUEALERT(GLB5)
StopandStartSetup
17 @CAWORLD#CAWORLD
Dev-Ops
Step3:CreateStop Application
APPLOTAESTOP/*Thisappl willsetOTA_EASTBATCHto0*/
JOBSTOP.EASTOTA_BATCHTASKSELFCOMPLETING/*ThislinkjobwillstoptheOTAEastbatchjobs.*/RUNDAILYRELEASEADD(Cleanup_Stop_Start_Files.OTAEAST_STOP)ESPRESDEFOTA_PRD_DATA_OTA_EASTBATCHSETAVAIL(0)NOTIFYPNODE(COMPLETE)ALERT(SHTR)
ENDJOB
UNIX_JOBCleanup_Stop_Start_Files.OTAEAST_STOP/*ThisscriptwillcleanuptheESPStop/Startdummyfilescreatedtostop/startOTAbatchRUNDAILYAGENTOTAEASTUSERadm_otaSCRIPTNAME/ota/data/adm_ota/cleanupESP.sh
ENDJOB
StopandStartSetup
18 @CAWORLD#CAWORLD
Dev-Ops
Step4:Create StartContinuousFileTriggerApplication
APPL OTAECFT/*Thisappl containsallOTAEASTCFTjobs*/
FILE_TRIGGEROTAE_STRT_FT/*TriggersOTAEASTStartEvent$VZWXOTA.OTAESTRTRUNDAILY
AGENTOTAEASTFILENAME'/ota/data/adm_ota/otaStartEastESP'CREATENOCHANGE(1)CONTINUOUS($VZWXOTA.OTAESTRT)DUEOUTEXEC23:59EASTERNNOTIFYOVERDUEALERT(GLB5)
ENDJOB
StopandStartSetup
19 @CAWORLD#CAWORLD
Dev-Ops
Step5:CreateStartApplication
APPLOTAESTRT/*Thisappl willsettheOTA_EASTBATCHto1whenholdisreleased
JOBSTART.EASTOTA_BATCHTASKSELFCOMPLETING/*ThisjobswillstartallOTAEastbatchjobs.*/RUNDAILYRELEASEADD(Cleanup_Stop_Start_Files.OTAEAST_START)ESPRESDEFOTA_PRD_DATA_OTA_EASTBATCHSETAVAIL(1)NOTIFYPNODE(COMPLETE)ALERT(SHTR)
ENDJOB
UNIX_JOBCleanup_Stop_Start_Files.OTAEAST_START/*Thisscriptwillcleanupthe ESPStop/Startdummyfilescreatedtostop/startOTAbatchRUNDAILYAGENTOTAEASTUSERadm_otaSCRIPTNAME/ota/data/adm_ota/cleanupESP.sh
ENDJOB
StopandStartSetup
20 @CAWORLD#CAWORLD
ExperiencesAverypositiveexperiencethroughoutourapplication,schedulingandauditingteams.CommentsInclude:“Versioncontroliscriticaltobusinessresiliencyaswellassatisfyingauditrequirements.”
CAEndevor®WithCAWAESPCAEndevor®hasbeenintegratedwithCAWAESPtotrackapplicationupdatesandmanagejobchanges.
FindingsByimplementingCAEndevor®,thesolutionallowsustotrackallapplicationandjobchangesforauditingpurposes.Thetechniquealsoallowsaquickwaytoback-outchangesmadetoapplicationsthatmaybecausingproductionissues.
CAWorkloadAutomationESPEditionandCAEndevor®HowCAEndevor®isIntegratedWithCAWorkloadAutomationESPEdition(CAWAESPEdition)
21 @CAWORLD#CAWORLD
CAWAESPEditionandCAEndevor®
Step1:Creation oftheCAEndevor®Environment
The CAEndevor®AdministratorincoordinationwithourSchedulingteamdefinedtheCAEndevor®environment.TheCAEndevor®environmentcontainsthefollowing:1Environment,1System,27Subsystems,2stagesand2types(APL)forapplicationand(SYM)forGlobal Variables/Symlibmembers.
CAEndevor®Setup
EnvironmentEPSWLS
SystemEPS
SubsystemAAA00(AAA =3Character
Applicationcode)
StageStage 1:PSTGStage 2:PROD
TypeType1:APLType2:SYM
Step2:Creation ofthePackageShipmentRules
Sincethe CAEndevor®softwareisrequiredtobeinstalledonourdevelopmentmainframe,weneededaprocesstosendtheupdatedapplicationstotheproductionmainframeproclibs.TheCAEndevor®AdministratorcreatedPackageShipmentrulestoallowthistooccur.OncetheCAEndevor®packageisexecuted,theCAEndevor®withCAWAESPelementsarethen‘shipped’toourproductionproclibs.
22 @CAWORLD#CAWORLD
CAWAESPEditionandCAEndevor®
Step1:CAEndevor® ADDtoStage1
CAEndevor®WithCAWAESPEditionProcess
23 @CAWORLD#CAWORLD
CAWAESPEditionandCAEndevor®
Step1:CAEndevor®ADDtoStage1
CAEndevor®WithCAWAESPEditionProcess
24 @CAWORLD#CAWORLD
CAWAESPEditionandCAEndevor®
Step1:CAEndevor®ADDtoStage1
CAEndevor®WithCAWAESPEditionProcess
25 @CAWORLD#CAWORLD
Step1:CAEndevor®ADDtoStage1
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
26 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
27 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
28 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
29 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
30 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
31 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
32 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
33 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
34 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
35 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
36 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
37 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
38 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
39 @CAWORLD#CAWORLD
Step2:Package CreationandShipment
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
40 @CAWORLD#CAWORLD
CAWAESPEditionandCAEndevor®
Step2:Package CreationandShipmentReportEmail
CAEndevor®WithCAWAESPEditionProcess
41 @CAWORLD#CAWORLD
JobChangeVerification
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
42 @CAWORLD#CAWORLD
JobChangeVerification
CAWAESPEditionandCAEndevor®CAEndevor®WithCAWAESPEditionProcess
43 @CAWORLD#CAWORLD
CAWAESPEditionandCAEndevor®
Job ChangeVerificationandEventSimulation
CAEndevor®WithCAWAESPEditionProcess
44 @CAWORLD#CAWORLD
Summary
TheVerizonProductionControldepartmenthasimplementedamyriadofoptionsandscheduling
techniquestoenhanceourapplicationusersexperiencewiththeCAWAESPSchedulingtool.Theoptions
thatwereimplementedareBestPracticeswiththeuseofGlobalVariablesorSymlibs andNaming
StandardsaswellasDEV-OPSforenduserstostopandstarttheproductionjobsduringmaintenance
windowsandintegratingCAEndevor®withtheschedulerforversioncontrolmanagement.
VerizoncontinuestorolloutnewBestPracticesintheDEV-OPSarenaplusSelfServiceoptionsthatwill
permitapplicationteamstoholdorreleasebatchjobsandtheuseofCAEndevor®forapplicationteams
totracktheirlowerenvironmentchangeswithinCAWAESP.
top related