5/21/2018 Sequences

1/64

Sequences:

Job Sequencer now supports:

Checkpoint/Restart

Looping

Expression Support

User Variables Stage

Automatically Handling Aborts

Terminator Stage ~ aborting a sequence

Enhanced Variable Support Enhancements to other activity stages:

WaitForFile Stage

Notify Stage

Checkpoint/re-start mechanism for Sequences:

Set/unset via a job property Compiler generates different code.

Continued

5/21/2018 Sequences

2/64

Sequences

If a job fails within a sequence, the sequence can be restarted and will skip tothe pointing of failure before starting execution:

Successfully run jobs are not restarted Failed (aborted) jobs are reset & run Activities occurring beyond the original point of failure are run when sequence

reaches them.

If a sequence is in the "Aborted / Restartable" state: "Run" will restart it from the checkpoint

"Reset" will clear the checkpoint information so the next "Run" will start theentire sequence again Recompiling the sequence clears checkpoint information too

5/21/2018 Sequences

3/64

Checkpoints:

Checkpointing must be enabled at design time:

Project wide setting to enable checkpointing in all new sequences.

Job level property to override the project setting for a given sequence.

Activity (stage) level property to allow disabling of checkpointing for a specific job runor activity (routine, command etc.) within a sequence.

5/21/2018 Sequences

4/64

When SEQUENCE runs:

JOB1fails

JOB1OK, JOB2fails

JOB1OK, JOB2OK

JOB3fails

JOB1OK, JOB2OK

JOB3OK

When SEQUENCE is restarted:

Reset JOB1 and run it again

Skip JOB1, reset JOB2 and run it again

Skip JOB1 & JOB2, reset JOB3 and run it again

Start at JOB1 again.

Method of execution:

5/21/2018 Sequences

5/64

Step1 :This is how a sequence of jobs looks like.

Recovery and Re-Startability :

5/21/2018 Sequences

6/64

Step2 :Go to Job Activity Stage's Properties and check "Add checkpoints so sequence is re-startable on failure" option.

5/21/2018 Sequences

7/64

Step3 :For example, let the "Loan Management" job alone is erroneous. After rectifying,check the "Do not checkpoint" option so as to skip the "Customer Management" from re-execution.

5/21/2018 Sequences

8/64

Step4 :The "Customer Management" is skipped, the "Loan Management" alonere-executed.

"CustomerManagement"

is skipped.

"Loan

Management"is re-executed.

5/21/2018 Sequences

9/64

Job Sequencer Loop Constructs:

Loop Variable available to activities within the loop

startloopname.$Counter

Contains loop iteration number or current list item

Loops may be nested

each End must point to one Start

2 counter variables are available in the inner loop

5/21/2018 Sequences

10/64

Loop Activities :

5/21/2018 Sequences

11/64

Loop Property :

5/21/2018 Sequences

12/64

Loop Activity Log :

5/21/2018 Sequences

13/64

Job after a successful run :

5/21/2018 Sequences

14/64

Expression Support:

Transformer-like expression syntax added to:

Job activity ~ setting job parameter values

Routine activity ~ setting routine argument values

Triggers ~ defining custom conditions

Context-menu help available when editing expressions, similar to Server jobtransformer + activity variables

Call Routines/Transforms/DSMacros

Do Arithmetic/Comparisons/If...Then...Else etc.

All other single-line fields:

Accept either straight text, or job parameters/activity variables surrounded by "#", orcombination

Have button to help select from variables available at that point

E.g. Command Activity, Notification (SendMail), WaitForFile

5/21/2018 Sequences

15/64

5/21/2018 Sequences

16/64

Job Sequencer User Variables:

5/21/2018 Sequences

17/64

Continued

User Variables

5/21/2018 Sequences

18/64

User Variables

5/21/2018 Sequences

19/64

Job Sequencer Terminator Stage:

Easy way to abort a sequence cleanly:

Optionally send STOPs to all running jobs

Optionally wait for all jobs to finish

Log final abort message

Can have any number of these in different branches of a sequence:

Only one will get executed of course

5/21/2018 Sequences

20/64

Terminator Activity :

A terminator stage can be placed in a job sequence to ensure that the sequence is stoppedcleanly if certain situations arise. You can have multiple terminator activities and can placethem anywhere in the sequence. They are connected to other stages by triggers, which specifywhen a terminator will be invoked.

5/21/2018 Sequences

21/64

Terminator Property :

5/21/2018 Sequences

22/64

Example job :

5/21/2018 Sequences

23/64

Handling Exceptions (Automatic):

Whathappen toWarnings?

Automaticallyaborts when an

error is detected.

Triggers"Executed OK"

5/21/2018 Sequences

24/64

Handling Exceptions with Triggers:

5/21/2018 Sequences

25/64

Handling Exceptions (Best Practice):

Warnings are Explicitly HandledOr

Set Warning Threshold to 1

AutomaticException

Handler

Uses variabels:$ErrSource

$ErrNumber$ErrMessage

ControlParallel

Streams

5/21/2018 Sequences

26/64

Parameter Management Strategies:

Standard parameter management:

Works fine if you are organized.

Naming standards a must. Establish early. Parent job propagates to child job.

Set default values in director.

Utilize templates to manage frequently used parameter sets.

Utilize environment variables for project wide values.

Password management: Encrypt

Must use Standard parameter management for security.

Set default values in director.

Or use encrypted environment variables per project.

5/21/2018 Sequences

27/64

Standard Parameter Management:

Job Parameters.

Sequence.

Environment variables.

Encrypted environment variables.

St d d P t M t

5/21/2018 Sequences

28/64

A sequence of two jobs FileSet and DataSet.

Continued

Standard Parameter Management

Standard Parameter Management

5/21/2018 Sequences

29/64

Continued

Standard Parameter Management

Standard Parameter Management

5/21/2018 Sequences

30/64

Enter the value expression using Insert Parameter Value:

Continued

Standard Parameter Management

Standard Parameter Management

5/21/2018 Sequences

31/64

Insert Parameter value with the same procedure to the next job activity also

Compile and run the job sequence

Insert Parameter val

Continued

Standard Parameter Management

Standard Parameter Management

5/21/2018 Sequences

32/64

View Log File:

Continued

Standard Parameter Management

A f I d F db k & A di

5/21/2018 Sequences

33/64

Aspects of Integrated Feedback & Audit:

Tools and methods:

Getting row counts: numRowsModified = DSGetLinkInfo(JobHandle, ActiveStageName,

LinkName, DSJ.LINKROWCOUNT)

Automatic Sequence Features Log Warnings for Activities that finish with status other than okay. Log Report Messages after each job run

Escalate with DSJobController ControllerHandle = DSAttachJob (DSJobController, DSJ.ERRFATAL ) EventError =DSLogEvent (ControllerHandle, DSJ.LOGWARNING,

MessgeTxt)

Stats from the command line : dsjobrepot

DSMakeJobReport

Other tools (for reading files, invoking OS commands and parsing output) Call DSExecute(ShellType, Command, Output, SystemReturnCode)

R Ti M H dli

5/21/2018 Sequences

34/64

Run Time Message Handling:

Runtime Message Customization:

Allows the user to customize how runtime messages are handled in DataStage

Based on definition of "Message Handlers"

Specify the action to be taken if a particular message is generated

Customization is based on the message ID

Three customization actions are supported:

Suppress message [Informational & Warning messages only] Demote message [from Warning to Informational]

Promote message [from Informational to Warning]

Message Handlers can be applied to whole projects or to individual jobs.

A Message Handler is a named file containing the customization rules.

Continued

Runtime Message Handling

5/21/2018 Sequences

35/64

To define a Message Handler: From Director log view, select the message to handle and then select "Job->Add rule

to message handler" menu option or right-click & select "Add rule to messagehandler" on the context menu.

Message can be added to a specific message handler file, a project handler or localhandler for this job.

Action is dependent on message type.

Types of Message Handlers: Local runtime message handler

Applies to the current job, and takes effect next time the job is run. Set in Job Properties->Defaults tab Gets compiled into the job and is exported with the executable Applied before any project level handler

Project message handler A Message Handler file which is associated with the current project Set in the Administrator client on the Project Properties->Parallel tab Applied after the local runtime handler (if enabled)

The combination of the two levels (both optional) defines the message handling for agiven job.

g g

Continued

Runtime Message Handling

5/21/2018 Sequences

36/64

Managing Message Handlers: A management tools enables the user to Create, View, Edit and Delete message handler

files. Available from a "Message Handlers" option on the Manager and Director Tools menus for

Parallel enabled projects only.

Managing Handler Manager: Displays handler rules in either the local runtime handler, the project handler or the

selected handler.

Rules can be edited * the handler saved (overwriting or "as new").

g g

Continued

M H dl

5/21/2018 Sequences

37/64

Message Handlers:

Continued

Message Handlers

5/21/2018 Sequences

38/64

Director > Job > Add rule to message handler

Message Handlers

Continued

Message Handlers

5/21/2018 Sequences

39/64

Click previous till message text appears, select the action to be done:

Message Handlers

Continued

Message Handlers

5/21/2018 Sequences

40/64

Click Add Rule and click OK:

Message Handlers

Continued

Message Handlers

5/21/2018 Sequences

41/64

Close and run the job View the log file:

Message Handlers

Continued

Message Handlers

5/21/2018 Sequences

42/64

Creating new message handler:

Director > Tools > Message Handler Management

Message Handlers

Continued

Message Handlers

5/21/2018 Sequences

43/64

Click New and click Save Enter a name:

Message Handlers

Continued

Message Handlers

5/21/2018 Sequences

44/64

Designer > View > JobProperties > Defaults > select the name given:

g

Continued

Message Handlers

5/21/2018 Sequences

45/64

Open Add rule to message handler:

Select Add rule to message handler

Select the given name from message handler

Click Add rule

Click OK, Close

Compile and run the job

g

Continued

Message Handlers

5/21/2018 Sequences

46/64

View director log:

g

Continued

Nested Condition:

5/21/2018 Sequences

47/64

Nested Condition:

Continued

Nested Condition

5/21/2018 Sequences

48/64

Continued

Nested Condition

5/21/2018 Sequences

49/64

Continued

Log file:

Debugging Changes:

5/21/2018 Sequences

50/64

Debugging Changes:

Debugging helpers:

Option to append row number and file name to row being processed, option to read only

(n) number of rows and first line is columns names

Routine Activities for Unlimited Flexibility:

5/21/2018 Sequences

51/64

Routine Activities for Unlimited Flexibility:

Combine repetitive operations.

Reuse.

More parameter control.

Take advantage of graph walking:

Checkpoint

Sub flow selection

Return status of flow

Routine Activity Stage :

5/21/2018 Sequences

52/64

Routine - DSJobReport :

Routine Activity Stage :

5/21/2018 Sequences

53/64

RoutineDSSendMail :

5/21/2018 Sequences

54/64

E-Mail Notification Property :

5/21/2018 Sequences

55/64

5/21/2018 Sequences

56/64

ExecDOS Routine :

Step1:

5/21/2018 Sequences

57/64

Step2:

5/21/2018 Sequences

58/64

Log Display:

5/21/2018 Sequences

59/64

Step1:

ExecTCL Routine :

5/21/2018 Sequences

60/64

Step2:

5/21/2018 Sequences

61/64

Log Display:

5/21/2018 Sequences

62/64

ExecSH : ExecSHSilent :

5/21/2018 Sequences

63/64

ExecSH Property : ExecSHSilent Property:

5/21/2018 Sequences

64/64

ExecSH Log : ExecSHSilent Log: