autosys
TRANSCRIPT
I N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Concept of AutoSys - Apr 2010
Syed Mohani
2
Agenda / Topics to be Covered I N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Topics:-
----------------------------------
--- What is autosys how it works
--- How to check Autosys is running.
--- Jil (Job Information Language)
--- How to setup a new/delete/upgrade Jobs
--- Autosys vs Cron jobs
--- Common commands used (Operate Prospective)
--- Automatic conversion from Cron jobs to Autosys jobs
--- Changes from Solaris to Linux Environment AutoSys jobs
--- Automation on Monitoring
--- Demo
--- Q & A
3
What is AutosysI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
AutoSys is an automated job management system that lets operate schedule, monitor and report.
This presentation is intended for Operate who will be responsible for
defining, monitoring and managing jobs run by AutoSys.
Using Autosys utilities, we can define a job by assigning it a name and specifying the arrributes
that describe its associated behavior.
--- The AutoSys job information Language (JIL)
4
Software ComponentsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
From a software perspective, the AutoSys environment is composed these main components:-
--- Event Server (Autosys Database)
--- Event Processor
--- Remote Agent
5
Jobs TypesI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
--- Command Jobs
--- File Watcher Jobs
--- Box Jobs
6
Jobs TypesI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
--- Box Jobs
A Box is a container of jobs with like starting conditions, either Date/time conditions
or job dependency conditions.
7
AutoSys Commands and MonitorsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
There are several commands unique to AutoSys that can be used to control, configure, and
report on the system behavior.
---- chk_auto_up (to check connectivity with the database)
---- autoping ( -x returns ver info -D check the database info
---- autorep (to check the job status)
---- autostatus (Status of the job)
8
AutoSys Job DefinitionI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
There are four major JIL subcommands.
--- insert_job Allows to add a new job to AutoSys
--- update_job Allows to change an existing job
--- delete_job Allows to remove an existing job
--- delete_box Removes a box job and all of its subordinate jobs
9
AutoSys Job DefinitionI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
A job performs a particular action based on a set of instructions called a job definition.
Starting Conditions:
--- Data and time condition
--- days_of_week
--- start_times
--- run_window
--- Job dependency conditions
--- success(job_name)
--- failure(job_name)
--- notrunning(job_name)
--- done(job_name)
10
Job StatusI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
At every point in processing, a job has what is called its current status or “state”.
--- STARTING ST (The job is in the process of ‘coming up”)
--- RUNNING RU (The job is running)
--- INACTIVE IN (Job has not been processed, has never been run)
--- SUCCESS SU (Exit code equal)
--- FAILURE FA (Job exited with an exit code)
--- TERMINATED TE (Job terminated while in the RUNNING state)
--- RESTARTED RE (Job was unable to start (or failed) and schedule to restart)
--- ACTIVATED AC (Job is itself has not yet started.)
--- ON_ICE OI (Job is removed from all conditions and logic, but is still defined.
--- ON_HOLD OH (Job is on hold and will not be run until it
receives the JOB_OFF_HOLD.
The difference between "on hold" and "on ice" is that when an "on hold" job is taken off hold, if its
starting conditions are already satisfied, it will be scheduled to run, and it will run. On the other
hand, if an "on ice" job is taken "off ice," it will not start, even if its starting conditions are already
satisfied. This job will not run until its starting conditions reoccur.
11
Common CommandsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
# autorep -j %lmc%
Job Name Last Start Last End ST Run Pri/Xit
uat_lmc_live_rate_Dly 04/14/2010 14:26:03 ----- RU 1868/1
Syntax autorep {-J job_name -M machine_name -G global_name} [-s -d -q -o over_num] [-r run_num]
12
Common CommandsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
# autostatus -j <job-name> i.e test-for-autosys-job-creation
SUCCESS
Checking on Jobs in the STARTING state.:-
# chase -E -A
Display all jobs scheduled to run between these two dates: job_depends -t -J ALL -F "04/01/2010 00:00" -T "12/26/2010 00:00"
Get version information:- # autoflags -a
INC2 LINUX ANY 11.0 SP2 46a989d1 snjcqacriba01
To Check AutoSys is up and running:-
# chk_auto_up
Primary Scheduler is RUNNING on machine: <Machine-name>
# ps –ef |grep autosys # autoping -m <Machine-name>
13
Common CommandsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Checking system Status
--- autoflags -a (info about autosys and system configuration)
--- autoping -m ALL -D (Verifies versions AutoSys comm are correctly)
--- autosyslog -e (Display the Event Processor log files)
--- chase –A –E (Verifies autosys database are running)
--- chk_auto_up (Inspects the DB /environment variables)
Definuing Calendars
--- autocal
--- autocal_asc (Adds, delete, and print custom calendar)
Maintaining Databases
--- archive_events (Removes old info from autosys database)
--- clean_files –d days (Removed Remote Agent Log Files)
--- dbstatistics
Managing Security
--- autosys_secure (Maintain Autosys Edit and Exec super user owner)
14
Common CommandsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Reporting Job Dependencies and Conditions
--- job_depends
Reporting Job status
--- autorep
--- autostatus –j <job-name> (Reports current status of the job)
AutoSys job status send
--- sendevent
Backup of All jobs
--- autorep –J ALL –q > dump_file
15
Common CommandsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
To check the jobs schedule between two Dates:-
# job_depends -t -j ALL -F "04/01/2010 00:00" -T "04/30/2010 00:00“
Job Forecast Report
From: 04/01/2010 00:00:00 To: 04/30/2010 00:00:00
Job Name Next Start Atomic Start Conditions
______________________________ _______________
qa_cri_batch02_queue_Health_c_min 04/21/2010 18:00:00 -------
qa_cri_batch03_queue_Health_c_min 04/21/2010 18:00:00 -------
16
Common CommandsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
To specify that the jobs can be run on any day except those days specified in the <name-holidays> calendar.
exclude_calendar : <name-hoilidays>
Sendevent:-
sendevent -J <job-name> -E CHANGE_STATUS -s TERMINATED
sendevent -J <job-name> -E STARTJOB
sendevent -J <job-name> -E FORCE_STARTJOB
sendevent -J <job-name> -E JOB_ON_ICE
sendevent -E DELETEJOB -J <job-name>
17
Common CommandsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Logs:-
# autosyslog -e -J <job-name>
18
SendEvents OptionsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
To start or force start a job manually using sendevent :
sendevent –E FORCE_STARTJOB -J <job-name>
sendevent -E STARTJOB -J <job-name>
To put jobs on OFF ICE or ON ICE :
sendevent -E OFF_ICE -J <job-name>
sendevent -E ON_ICE -J <job-name>
19
SendEvents OptionsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Sendevents to AutoSys for a variety of purposes, including starting or stopping AutoSys jobs,
stopping the Event processor, and putting a job on hold. This command is also used to set AutoSys
global variables or cancel a scheduled event. sendevent is normally used with "-E" & -J option
-J job_name : Specifies the name of the job to which the specified event should be sent.
-E event :Specifies the event to be sent. This option is required. Any one of the following events
may be specified as:
STARTJOB KILLJOB DELETEJOB FORCE_STARTJOB JOB_ON_ICE JOB_OFF_ICE JOB_ON_HOLD JOB_OFF_HOLD CHANGE_STATUS STOP_DEMON CHANGE_PRIORITY COMMENT ALARM SET_GLOBAL SEND_SIGNAL
20
Command AttributesI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
AUTOREP
USAGE: autorep [-J JobName] [-d | -s | -q | -o OverRide # | -a | -w]
[-B GroupName] [-I ApplicationName] [-t]
[-X ExternalInstanceName] [-Y JobType]
[-R Run_num] [-L PrintLevel] [-N Retry] [-x]
[-z GlobalBlobName] [-f BlobDownloadDirectory]
[-G GlobalName] [-M MachineName] [-D DataSource] [-?]
21
Job Physical loactionI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
# pwd
/cri_staging/syed/jil/lmc
*** 2193 smccuatcriwa06 citwl:ugcrmcit /cri_staging/syed/jil/lmc
# ls -altr
total 20
drwxr-xr-x 3 citwl ugcrmcit 4096 Apr 13 17:27 ..
-rwxr-xr-x 1 citwl ugcrmcit 423 Apr 13 18:31 uat_lmc_live_rate_Dly.bak
-rwxr-xr-x 1 citwl ugcrmcit 423 Apr 14 11:42 uat_lmc_live_rate_Dly.jil.20100414
-rwxrwxrwx 1 citwl ugcrmcit 395 Apr 14 12:07 uat_lmc_live_rate_Dly.jil
22
Jobs StreamsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
A job stream (or Flow) refers to the logic of how related AutoSys jobs are run.
---- Date and Time conditions
---- Job dependency conditions
lob_backup Command job has date and time condition (Runs every day of the week at 01:01 am
Job_clean Command job has a job dependency condition specified for its starting parameters.
It will run only on the successful completions of job_backup.
23
Jobs StreamsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
File Watcher Jobs:-
A file watcher job is used to watch for an incoming file. Job has date and time conditions specified
For its starting condition. Example job is to be run every day of the week at 2:00 pm. It watches
For a file arriving. The job_Fwatch job will check every 15 second for the creation of the incoming
File. When condition is met, job_Fwatch will notify the server and complete with a SUCCESS status.
24
Defining and Running jobs using Job Information Language (JIL)
I N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
We can use AutoSys proprietary scripting language --- JIL to create jobs definitions. This job definition must be submitted to the AutoSys database before the job can be run.
JIL Sub-Commands:-
--- insert_job
--- update_job
--- delete_job
Jil Attributes Types:-
--- Job Name (insert_job: <job-Name>)
--- Job Type (c f and b) (job_type: c)
--- Machine Name (machine: <machine-name>)
--- Starting Conditions (days_of_week: mo, we, fr)
(condition: success(my_job)
--- File to Watch For (watch_file: /tmp/<file-name>
25
JIL SyntaxI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Writing a JIL script, follow the syntax rules:-
1 – sub_commnad: job_name (insert_job: <user-define-job-name>
2 – attribute_keyword: value
Example:-
Creating a job:vi <test-for-autosys-job-creation.jil>
# JIL Script
# This script will re-direct a message to the file /tmp/test-output.out
update_job: test-for-autosys-job-creation job_type: c
machine: cri_batch_job_queue
permission: gx,wx,mx
command: /bin/echo "AutoSys KT demo" > /cri_staging/syed/jil/demo/test-output.out
std_out_file: /cri_staging/syed/jil/demo/test-output.out
Saving a job:
Jil < test-for-autosys-job-creation.jil
26
Jobs DetailsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
# autorep -j uat_lmc_live_rate_Dly -q
/* ----------------- uat_lmc_live_rate_Dly ----------------- */
insert_job: uat_lmc_live_rate_Dly job_type: c
command: /apps/ss/userprojects/lmc/live_rate.sh
machine: cri_web_job_queue
owner: citwl@smccuatcriwa06
permission: gx,wx,mx
date_conditions: 1
days_of_week: mo,tu,we,th,fr
start_times: "22:00"
description: "Live Rate Job"
std_out_file: /local/ss/cit/userprojects/log/lmc/lmc_liverate.stdout
std_err_file: /local/ss/cit/userprojects/log/lmc/lmc_liverate.stderr
alarm_if_fail: 1
27
JIL SyntaxI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Delete a Job:
At the unix Command Prompt
# jil <enter-key>
At the jil prompt
jil>>1> delete_job: <job-name>
exit
28
Job Delete thru jil:-I N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Delete the Job:-Step 1 --- Vi the jil file and replace command with delete_job:
/* ----------------- prd_cit_cap_CPPmonthRun ----------------- */delete_job: prd_cit_cap_CPPmonthRun job_type: ccommand: `cd /home/capbg/scripts; ./CPP_monthly.sh`machine: snjprdcria1.jpmchase.com#owner: [email protected]: gx,ge,wxcondition: success(prd_cit_cap_CPPfile_fwatch)std_out_file: /home/capbg/scripts/CPP_month.stdoutstd_err_file: /home/capbg/scripts/CPP_month.stderralarm_if_fail: 1
Step 2 --- Save the jil file into Database.
# jil < [Jil-file-name]
Step 3 --- received the outpuy from save
Attempting to Delete Job: prd_cit_cap_CPPfile_fwatch
Jobs which are dependent on the
DELETED Job: prd_cit_cap_CPPfile_fwatch
prd_cit_cap_CPPmonthRun
Delete was successful!
Exit Code = 0
29
Auto Transfer Cronjobs into Autosys jobsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Commands:-
cron2jil - Translates crontab files into JIL format
cron2jil -f crontab_file [-d output_directory] [-i include_file] [-m machine] [-p prefix]
---- input as crontab file
---- # cron2jil –f file-name
---- convert files names i.e. cp_[123]
run_calendar: <callendar-name>
autocal_asc -s test_demo -E test_export.txt
Export the calendar entries into file
autocal_asc -lc
List the cycle calendar
Note:
cron2jil does not generate JIL files for jobs that are defined in crontab to start every minute
After the *.jil or *.cal files are generated, you must submit them to the database using the
jil and the autocal_asc commands, respectively.
30
Auto Transfer Cronjobs into Autosys jobsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Combing Calendars:-
Calendars can be combined in a number of ways. For expample, we can create a calendar
that includes all the dates that are in either one calendar or another.
We can import calendar text files. Calendars contained in ASCII test files can be imported
Into AutoSys database. These test files may contain multiple calendars, each of which must
be delimited with the calendar: Calendar_name attributes.
calendar: Q1paydays
01/01/2010
01/15/2010
02/01/2010
02/15/2010
03/01/2010
06/15/2010
calendar: Q1holidays
01/01/2010
31
Auto Transfer Cronjobs into Autosys jobsI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Date Range:-
The following resource sets the number of years in the data range of the calendar, as a
default at start up. This can be overridden manually by way of the Date Range option from
the options menu.
! 1 = prior year plus current year.
! 2 = thru next year, 3 = thru third year,
! 4 = thru forth year, 5 = thru fifth year,
! 10 = thru tenth year
Autocal.dateRange:
32
New Setup for Linux EnvironmentI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Reporting Job Dependencies and Conditions
Virtual Machine Physical Machine
crrt_job_queue citcrrtapp[12]-qa.us.jpmchase.net
crm_crws_job_queue citportalapp[123]-qa.us.jpmchase.net
crm_rad_job_queue citportalapp[12]-qa.us.jpmchase.net
cri_batch_job_queue citssbatchapp[1234]-qa.us.jpmchase.net
cri_web_job_queue citsswebapp[1234]-qa.us.jpmchase.net
machine: cri_web_job_queue <------ Virtual Machine Name
permission: gx,wx,mx <------ Job Permissions add “mx”
To convert existing Jobs definitions
Jil out the current job definition from existing environment into a text file Update machine name with virtual machine name (refer next slide )update existing permission to add additional permission “mx”Jil in the job definition to new environment.
33
AutoSys Jobs naming conventionI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
Get the AutoSys jobs status thru web tool.
While apply the naming convention we can write a cgi script and use apache to get
status on web.
Env-App-JobName
XX-XXXXX-XXXXXXXXXXXXXXXXXXX
XX = [DR/ST/QA/UT/PR]
XXXXX = [CMART/CAPLO/BASEL/IBFTP/……]
XXXXXXXXXXXXXXXXXXXX = JOB Name
34
AutoSys vs CronJobI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y
max_run_alarm: <mins>
Specifies the max run time that a job should require to finish normally. This test
can catch an error. Such as the applications stuck in a loop.
n_retrys: 5
To set the job to be automatically restarted up to five times after an application
Failure.
Priority: 1
Set the job priority for run.
run_calender: <calendar-name>
Running job on the last business day of the every month.
Run_window: <time-time>
Before starting the job autosys will check the window time.
watch_file: <path>
Attributes used as Watch file min file size and watch interval
35
Q&AI N
V E
S T
M E
N T
B
A N
K
T E
C H
N O
L O
G Y