best practices with ca workload automation autosys (ae)

Pre-Con Education: Best Practices with CA Workload Automation AutoSys Edition (AE)

Michael Woods

DevOps: Continuous Delivery

CA Technologies

Principal Engineering Services Architect

DO4X110E

2 © 2015 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD

© 2015 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.

The content provided in this CA World 2015 presentation is intended for informational purposes only and does not form any type of

warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.

For Informational Purposes Only

Terms of this Presentation


Abstract

In this session we will cover the CA Technologies recommended configuration and architectures for CA Workload Automation AutoSys (AE) , including the scheduling engine, agents, security and graphical user interface (GUI). These best practices are designed to help maximize performance and minimize maintenance and issues.

Michael Woods

CA Technologies

Principal Engineering Services Architect


Agenda

WHAT ARE BEST PRACTICES?

WORKLOAD AUTOMATION AE BEST PRACTICES

WORKLOAD CONTROL CENTER BEST PRACTICES

1

2

3


What are Best Practices?

Normally reduces issues

Not the same as taking the defaults Has a greater usage in client base Less chance for issues going forward

Not always best for you

You may have unique requirements One size does not always fit all


VIRTUAL MACHINES

BOX JOBS

SUMMARY

START TIMES

PERFORMANCE TUNING

GLOBAL VARIABLES

AgendaCA Workload Automation AutoSys Edition

SCHEDULER

APPLICATION SERVER

SECURITY

SYSTEM AGENTS

1

2

3

4

5

6

7

8

9

10


Test Environment

If at all possible:

Have a sandbox to do any engineering work like testing– New releases– Patches– Different configuration settings– Try out new procedures

ALWAYS have a test system that can be used before applying to production

DR should match production

Create and execute a test plan

Test architecture should mimic production


Scheduler

When the scheduler contacts an agent, it will cache or store the IP address in memory for future use

– Decreases system network calls to increase performance– Gathered again if a communication error occurs to agent

Enabled by default– Good for static environments

Setting EnableIPCaching =0|1– Located in $AUTOUSER/config.$AUTOSERV file

When to turn it off– IP addresses may change frequently– Data Centers may move dynamically– Not using VIP for clusters

Configuration Options


Application Server

Instance Wide Encryption:– The encryption key used to communicate to the application server– This key is used by all AE SDK clients, which includes command-line utilities like

sendevent, autorep,… and WCC– If you write any SDK programs, you must specify the encryption type– Without the encryption key, communication will fail

For example, trying to force start a job from the development instance should not work if you pointed it to the UAT instance

Set to NONE when R11 agents are used– If encryption is required, use the CSAM SSL encryption

Set by UseEncryption setting– We recommend using a unique key for each tier of service

Sandbox should not be using the same key as production

– Use a different one per instance??



Global Variables

Use application name as prefix to prevent using the same name

Use EEM to secure

Limit where changes are made from to reduce race conditions– Day of week, date, on/off switch….

Could Resources be a better fit?– Managed by scheduler– Also securable– Easier to track and maintain

Naming Conventions


Calendars

Use application name as prefix to prevent different groups from modifying each others calendars

Use EEM to secure

Create corporate standard calendars for most commonly used:– First working day of month, last working day of month, third Tuesday…– Use extended calendars when it makes sense, reduce work going forward– Convert existing calendars to extended calendars, when sensible or convenient

Use Extended Calendars when applicable:– Define once and do not need to update again– Generates a year in advance, keeps updating to keep a years worth of data– Limitation: Can not be used as Holiday calendars

Naming Conventions


Boxes

Avoid having different start times inside box, delays the completion of the box

Avoid having different start dates inside the box, introduces further delay

Reduce the depth of nesting to get better performance

Not every job needs to be in a box– Boxes decrease performance when overused, ties up threads until

box is complete

Consider using group and application fields to logically group jobs


Start Times

Try to stagger jobs as much as possible to get a smoother job submission curve:

Will actually submit faster than large number of starts at one time

Use the forecast command or the API to see how many are being started at same time

Typically, 0, 15, 30, 45 are the most commonly used start times

Use diverse start times in your schedule

Try to prevent jobs running every minute when possible


System Agent

Use a consistent agentparm.txt to simplify installs, maintenance and troubleshooting efforts

Keep enough log data for trouble shooting

Some job types like ftp have additional information in spool file, so automatic cleanup at success may not be advantageous

Do not add managers that are not needed to agents, may cause additional overhead– Do not have duplicate manager names specified, can cause

communication errors

Use a common communication protocol (plain recommended)

Reduce agent sleep period for managers that do not respond– 2 minutes default, can delay failovers


Machine Definition

Use opsys field– On Windows – required for domain users– Useful to self document

Description field– Again, self documenting, make it count

Geography, Business line, …

Use fully qualified domain names on node_name attribute

Use different encryption keys per environment– Recommended not to use the DEFAULT key in production


Security

Two security models available:

External: Uses Embedded Entitlements Manager (EEM)– Allows more granular security– Works well with multiple operating systems

Native: “UNIX” style security, owner, group, world– Can be hard to enforce security when owners/end users require

multiple groups

External is best practice

Internal or External


Security

Keep it as simple as possible– Think of maintenance going forward

Use Group membership, not individual user IDs– People move on, groups normally do not

If not using LDAP, use Global groups in EEM to lessen maintenance

EEM Policy


EEM User Store

Use company user store and groups when possible– Cleanup is automatic when user record deleted– Security is owned by security group

Dynamic User Group:– Mapping LDAP group to naming convention– Too many can cause performance issues (1000’s)

Limited Use of Regular expressions in Resource Name:– Backtracking specifically (ABC|XYZ)– Keep to minimum– Impacts performance


EEM Security Administration

Do not use EiamAdmin – makes audit less accurate

Create Security administrators using scoping policy– Check AdministerObjects default policy for sample or add users to it


Client Installations

Only install when required– Reduce maintenance effort– Reduce manual intervention for audit and control– Use the scheduler as a scheduler– Smaller footprint– Run from Scheduler machine?– More secure– Turn on autotrack to have an audit of who did what and when and

from where


Performance Tuning

AE– SCHED_SCALE– DB_CONNECTIONS– Use statistics aggregation to collect performance data– Run stat reports on a regular basis to start spotting trends


Job Lifecycle

Have a strict change control process in place– Sending emails is not a process

Customers should not be moving untested jobs into production– Customer should validate jobs runs to success in Test/UAT

Have a Dev instance for developing code, not production


Data Retention

Keep your DB as small as possible to improve performance

How many days event and job run data to retain?– What are your user requirements?– How many job runs per day?

How many days of archived jobs?– Best practice is 1 day. Cannot be retrieved, so why keep?

How many days of audit information?– Do you action alarms or use a external monitor?– If not using alarms directly, keep only 1 day worth of data


Data Retention Continued

DBMaint:

Should use internal or external method?– Config setting at 3:30 am each day– Create jobs that run commands

For example, should archive events run more often than archive jobs?

How many times a day to run?– Shops that have a large run per day or keep a high number of days of

data may want to run more than once a day– Want to keep processed event table small as possible


Health Monitoring

Automate monitoring of failures:– SNMP traps to your favorite tool– Email– CCS Event Management– iDash for monitoring of SLA

WA AE health:– Alarms self-generated– Monitor processes with your

corporate standard tool

Component Process Name

Scheduler event_demon

Application Server as_server

EEM igateway

CA Directory dxadmin

System Agent cybAgent


Scripts and Programs

Document exit codes:

If not a simple 0|1, they should have meaning to speed debugging or used to decide what job runs next

Think about recovery:– What needs to be rerun?– Document runbook steps.– Should it be broken up into multiple jobs?

Can any checks be made to help jobs succeed?– Job types can check process/services are running– Check CPU/disk space– Check any external machine dependencies


Scripts and Programs Continued

Use fully qualified commands on job definition

Use fully qualified commands and file names in scripts

Use supplied variable when dealing with AE environment– $AUTOSYS/bin/autorep –j test– $AUTOSERV for instance name

Use an env or set command to write out environment variables, ensure you have what is expected


Password Expiration

Service Accounts

If permissible, use un-expiring passwords

When not, use autosys_secure in batch mode to update passwords– autosys_secure –c [email protected] –o oldpass –p newpass


AgendaWorkload Control Center

WCC

CREATE THE ENVIRONMENT

CONFIGURATION

MONITORING VIEWS

ALERTS

PERFORMANCE TUNING

SUMMARY

1

3

4

5

6

7

2


Creating the Environment for Users

Define the roles you need:

Admin

Operations

Scheduler

Application developer

Line or business user


Creating the Environment for Users Continued

Define what tabs (applications) each role should have

What servers should they have access to?– Development– UAT– Production

What tabs per server?– Edit for dev, but not for prod?– Send event / monitoring in prod, but no edit

What views should they have?



One security mode available:

Security is used to determine what will be displayed– Remove unused tabs– Reduce questions/issues on features not used or wanted

Use security to protect what needs protection and also to reduce the noise– Show what they are interested in

Using Security



There are 50+ job types.

How many are required?

Do you have Oracle E Business?

Do you have SAP?

Do you have PeopleSoft?

Do you have z/OS?

Do you allow FTP?




General notes for running commands for job types:– User_name: EEM user, local user with access – Password: Users EEM password– Application_name: qedit or appedit

Command must be run with a userid able to update log files

Use the userid designated at install as the run as user

The default is root

Batch Interfaces



Creating a list of desired job types:

List available job types to see what is currently available and not available:filter_jobtypes -u user_name -p password -t application_name -l

Remove job types from being displayed:filter_jobtypes -u user_name -p password -t application_name –r all|jobtype1[,jobtype2,…,jobtypex]

Add job types to available list:filter_jobtypes -u user_name -p password -t application_name -a all|jobtype1[,jobtype2,…,jobtypex]

Batch Interfaces Continued



Creating views in batch:

Create a list of applications required to have a view

Create a sample view based on your requirements

Export the view– wcc_monitor -u user_name -p password –x export_file –view sampleView

Edit exported view to replace job name to ‘XXXX’

Create script to ‘sed’ template and replace ‘XXXX’ with your application prefix using your naming convention

Import views:– wcc_monitor -u user_name -p password –i input_file

Batch Interfaces Continued


Job Templates CLI

Creates job templates to be used both in Quick Edit and Application Editor

Global template for all users, templates only for specific uses

Syntax:job_template -u,--user <user>

-p,--password <password><-g,--global |-f,--for-user <user> >< -l,--list | -r,--remove <template-name> |-i,--import <template-name> <file-name> |-export <template-name> <file-name> >[-o,--overwrite][-?,--help]


Job Templates – Use Case

1. Use autorep to prepare JIL definitions

2. Modify JIL file to prepare it for import as a template

3. Create a global template for all users

4. Create a template for ejmcommander user only

5. List the global templates available

6. List the templates available for ejmcommander user only

7. Log in to Quick Editor as ejmcommander and use the new templates



Use autorep output to create a JIL file:autorep –q –j JOBNAME > filename.jil

Prepare JIL Definitions

Note 1: After JIL files are ready, copy them to a computer running WCC.

Note 2: It is possible to have only 1 job definition for 1 file to import as template.



Delete insert_job: JOBNAME

In case you want to use job name as a part of template:– rename insert_job: JOBNAME to job_name: JOBNAME– put job_type: TYPE on a new line

Repeat the same for both JIL files

Prepare JIL Definitions Continued



Import global template:job_template.bat -u ejmcommander -p ejmcommander -g -iglobalTemplate c:\temp\global_template.jil

Import as a Global Template



Import user template:job_template.bat -u ejmcommander -p ejmcommander –f ejmcommander -i userTemplate c:\temp\user_template.jil

Import as a User Template



List the global templates available:job_template.bat -u ejmcommander -p ejmcommander –l –g

List the user templates available:job_template.bat -u ejmcommander -p ejmcommander –l -f

List the Templates



Open the Quick Editor Create Object dialog and check for templates:

Use the Templates


Best Practices

Log on again after adding add/modify a template using job_template CLI

Log on again after modifying filter job types filter using filter_jobtypes CLI

Log on again to apply new or updated EEM policies


WCC

Limit who can create views– Duplicate or wildcard for ALL can hurt performance

If 11.3 or higher, go to 11.4 for new features and better performance– Requires EEM 12.51

If 11.3.6 AE and using EEM, use a global ECLI user and restrict access to the Credentials tab


WCC Continued

An external DB when HA is required

– Derby does not support HA

Use an external DB for better performance when one or more apply

– Monitoring and configuration DB: Over 50k defined jobs Over 50k job runs 70 concurrent users Over 200 views Over 500 alerts defined Over 5 AE servers defined

– Reporting DB: Over 1 Million job runs in reporting DB

Can be switched after install


WCC Continued

Let users have self-service in production to relieve Operations from getting overwhelmed and potentially less error prone– Must be auditable– Must be secured

EEM can audit this activity, your policy secures it


Performance Tuning

WCC

Check tuning guide for specific instructions– Java.Maxmemory is major one– Two tomcats– 32 bit – should not exceed 1G– 64 bit – can be set larger, check what is available on your system

Linuxinstall_dir/tomcat/conf/wrapper.conf# Uncomment this line if the entropy of the UNIX machine is low to speed up encryption# The dot between /dev and /urandom is not a mistakewrapper.java.additional.101=-Djava.security.egd=file:/dev/./urandom


Performance Tuning

DB Maintenance

Alarms and Alerts– Turn off if not using AE Alarms– Keep only required days of data

Reporting data– Keep only required amount of data

EEM Event Policy

Disable or update policy for Authenticate With Artifact


Q & A


Recommended Sessions

SESSION # TITLE DATE/TIME

DO4T07S Vision and Roadmap: CA Workload Automation Thursday at 12:15 pm (DevOps

Continuous Delivery Theatre 4)

DO4X177SVision and Roadmap: CA Workload Automation AE

(AutoSys) Wednesday at 4:30 pm (Lagoon D)

DO4X180SCase Study: CenterPoint Energy Achieves .0003 abends

with CA Workload Automation AE (AutoSys®)Thursday at 3:45 pm (Lagoon D)

DO4X177S

Case Study: Large Fuel Company Reduces the Cost and

Complexity of Managing Mission Critical SAP and non-SAP

Jobs

Thursday at 1:00 pm (Lagoon D)


Must See Demos

Maximize SLA Management and Extend Workloads to SAP®

DevOps Continuous Delivery Theatre 4

Reduce the Cost and Complexities

of Managing Hadoop Workloads DevOps Continuous Delivery Theatre 4


Follow Conversations At…

CA Workload Automation

Mainframe Theatre

CA Workload Automation iDash

DevOps Continuous Delivery Theatre

Thursday 1:00PM


For More Information

To learn more, please visit:

http://cainc.to/Nv2VOe

CA World ’15

http://cainc.to/Nv2VOe

http://bit.ly/1wbjjqX

best practices with ca workload automation autosys (ae)

Technology