02 principles of parallel execution and partitioning

7/25/2019 02 Principles of Parallel Execution and Partitioning

1/23

DataStage Fundamentals (version 8 parallel jobs) Staffordshire, December 2010

Module 2 Principles of Parallel Execution

DataStage parallel jobs, as the name suggests, eecute a number ofprocesses in parallel! "here are t#o aspects to this parallelism$ that of

having more than one stage eecuting at the same time, passing ro#s ofdata along the stream of stages, and that of dividing (%partitioning%) thedata over one or more logical processing nodes!

&e also introduce ho# 'ou can specif' ho# data are to be partitioned intomultiple nodes, or collected from multiple nodes into one!

Objectives

aving completed this module 'ou #ill be able

to define pipeline parallelism and partition parallelism

to name five partitioning algorithms used b' DataStage

to name t#o situations in #hich *e'+based partitioning is

necessar' to 'ield correct results

to state ho# pipeline parallelism is implemented in DataStage

parallel jobs

to name t#o DataStage structures in #hich partitioned data can be

stored

age 2+1


2/23


Parallel Execution

arallel eecution is best understood b' contemplating #hat happens#hen parallel eecution is not occurring!

-n the #orst case, one ro# is processed right through the ."/ process, anddisposed of in accordance #ith the design! "hen the net ro# is processedright through the ."/ process, and so on! -n short, processing occurs onero# at a time!

"here are t#o #a's that parallelism that is, processing more than onero# at the same time can occur! "here are names for these #a's$ the'are called pipeline parallelism and partition parallelism! DataStageparallel jobs emplo' both, largel' automaticall'! -t is possible to designboth into DataStage server jobs too, in a non+automatic implementation,but that discussion is outside the scope of these notes!

Pipeline Parallelism

-f a DataStage job consists of stages connected b' lin*s, then the data flo#through that job is indicated b' the arro#heads on the lin*s!

3' associating some *ind of memor' buffer #ith each of the lin*s, then itbecomes possible to transmit more than one ro# at a time, and forupstream stages to begin processing ro#s #hile other, earlier ro#s, arestill being processed b' do#nstream stages!

"his is *no#n as pipeline parallelism! -t establishes a pipeline from one

stage to the net! .ach stage is a producer of ro#s, a consumer of ro#s, orboth!

-n a DataStage parallel job a specific structure, called a Data Set (moreeactl' a virtual Data Set) is used to implement these buffers! 3' defaultthere is a pair of buffers associated #ith each lin*, #ith the upstream stage

age 2+2


3/23


producing ro#s into one and the do#nstream stage consuming ro#s fromthe other! "hese roles s#itch #hen certain tuneable thresholds1occur!

"his mechanism tends to smooth out speed variations in stage eecutionand allo#s processing to continue as fast as possible!

4nother #a' to thin* of pipeline parallelism is as a conve'or belt movingbuffers (boes5) of ro#s from one stage to the net!

6onceptuall' the operating s'stem idea of a pipeline ma' be used to aidunderstanding of #hat is happening! 7perating s'stems allo# pipelines ofcommands to be created, #ith operators such as the pipe (), re+directionoperators such as 9 and conditional pipe operators such as :: joiningthem and controlling #hat happens!

-f 'ou imagine for a moment that each stage in a DataStage parallel jobdesign generates an operator #hen the job is compiled then 'ou mightcontemplate the eecution path of a parallel job as being able to bedescribed as something li*e

op0 op1 op2 op;

"o provide for the buffering the virtual Data Sets this model needs tobe slightl' refined!

op0 9 ds0 op1 < ds0 9 ds1 op2 < ds1 9 ds2 op; < ds;

-n this model, op0 through op; refer genericall' to operators and ds0through ds2 refer genericall' to Data Sets2!

-ndeed, #hen 'ou eamine the 7rchestrate shell script that is generated b'compiling a job, or #hen 'ou eamine the score that is actuall' eecuted,'ou see precisel' this relationship bet#een operators and Data Sets!

-n the 7rchestrate parallel eecution engine, the onl' structure #ith #hichan operator can deal is an 7rchestrate Data Set! "his has implications fordealing #ith real #orld data, #hich #ill be discussed in a later module!

1 Details on ho# these mechanisms are tuned are be'ond the scope of these notes! "he'properl' belong in an advanced s'llabus!

2 -n the 7rchestrate parallel eecution environment, since ever'thing is #ritten in 6 or 6==, counts al#a's begin from >ero! "hat is, counts are >ero based!

age 2+;


4/23


Partition Parallelism

artition parallelism involves dividing the data into subsets of ro#s andprocessing each subset identicall' and in parallel! rovided that there aresufficient resources to handle this parallel processing, overall eecution

time should be reduced linearl'!

"hat is, if 'ou divide 'our data into ? e@ual subsets of ro#s, then it shouldta*e ver' close to 1A? the amount of time to eecute as it #ould to processthe entire set of ro#s in a single stream!

-n practice there #ill be factors, such as the need to *eep common *e'values together and the overheads of starting up and shutting do#n theadditional parallel processes, that #ill mitigate against achieving thisoptimal result, but as a general rule partition parallelism #ill 'ield reducedthroughput time!

DataStage parallel jobs achieve partition parallelism automaticall', using

the 7rchestrate parallel eecution mechanism described belo#!

"his means that 'ou onl' have to design as if there #ill be a single streamof processing$ the DataStage parallel engine loo*s after the rest (though'ou do need to remain alert to implications for partitioning data andgenerating se@uences of numbers in a parallel eecution environment)!

6onsider the follo#ing job design (donBt #orr' too much 'et about #hat itactuall' does)!

age 2+C


5/23


&hen 'ou eecute this parallel job in an eecution environment thatspecifies four processing nodes, #hat actuall' gets eecuted might berepresented as follo#s! 4gain, this simplifies the actual situation, but isuseful as an eplanation!

"he number of processing nodes is specified b' the parallel eecutionconfiguration file (of #hich more shortl')! 4 DataStage server #illt'picall' have more than one configuration file available! "he one that aparticular job #ill use is specified b' an environment variable called4"67?F-EF-/.!

"he value of 4"67?F-EF-/. is a pathname of a DataStage paralleleecution configuration file! 3' default these are stored in a director'called 6onfigurations that is a sibling to the DataStage engine directories,but ma' be stored in an' director'! &e strongl' recommend conformingto the conventional storage location, so as to cause least surprise to an'onesubse@uentl' maintaining the s'stem!

age 2+


6/23


4l#a's include the environment variable G4"67?F-EF-/. as a jobparameter in DataStage parallel jobs and in job se@uences that invo*eDataStage parallel jobs! "his allo#s the configuration file for paralleleecution to be selected on a job b' job basis #hen the job is eecuted!

"he default value for this job parameter should be the special valueGH7ID.F, #hich means that the project+level default #ill be used unlessit is overridden #hen the job run is re@uested!

age 2+J


7/23


Execution Environment

-t #ould be #ell here to ta*e a short sidestep to discuss some terminolog'!

"he term s'mmetric multi+processing (SK) is often used to describe asingle machine #ith more than one 6L!Kulti+core 6L chips count as multipleprocessors for this discussion! 4lthoughtechnicall' it is not @uite the same thing,non+uniform memor' architecture (?LK4)is treated as SK for DataStage !

-n an SK environment, ever' processor hasaccess to all of the memor' and all of the dis*resources available to that machine! For thatreason, it is sometimes usefull' referred to as

a shared ever'thing environment!

&hen processing occurson multiple machines,#hich ma' beconnected together in acluster or gridconfiguration, then eachmachineBs 6Ls haveaccess onl' to that

machineBs memor' anddis* resources! For thatreason, such aconfiguration issometimes referred to as

a shared nothing environment! "he term massivel' parallelprocessing (K) is also encountered #hen describing this *ind ofenvironment!

.ecution on a grid configuration, #here the number of available

machines ma' var' over time, re@uires a more d'namic configuration file!DataStage has the capabilit' to manage or, more accuratel', to bemanaged b' a grid environment! "his re@uires installation of the Erid.nablement "ool*it!

age 2+M


8/23


Configuration File

4 DataStage parallel eecution configuration file contains a list of one ormore node names, the location of each node and the resources available oneach node! 4 node, in this contet, is a purel' logical construct called a

processing node there is no fied relationship bet#een the number ofnodes and, for eample, the number of 6Ls available!

7n a four 6L machine (or cluster or grid), for eample, it is li*el' thatone+node, t#o+node, four+node and even eight+node configurations mightbe defined! 4nd #here the nodes are on separate machines, the sameconsiderations appl' (consider the total number of 6Ls available), butthere are some small overheads in communication bet#een the machinesthat needs to be ta*en into consideration!

-n an SK environment, ever'thing is on the same machine in thenet#or*! "herefore the machine location of ever' processing node in the

parallel eecution file #ill be the same machine name! -n an Kenvironment, ho#ever, there #ill be different machine names associated#ith different processing nodes! -t still ma' be the case that there is morethan one node on each of these machines, but if there is ever more thanone machine named in a configuration file, then it describes an Kconfiguration of some *ind!

/etBs eamine the default configuration file that ships #ith DataStageserver to be installed on a &indo#s operating s'stem!

{

node "node1"{fastname "PCXX"pools ""resource disk "C:/IBM/InformationServer/Server/Datasets"

{pools ""resource scratc!disk "C:/IBM/InformationServer/Server/Scratc!"

{pools ""

"he outer set of curl' braces contains the entire configuration definition!"he onl' thing that can appear outside of these curl' braces are 6+st'le

comments (that is, tet introduced b' AN and terminated b' NA)!-n the default configuration file sho#n there is onl' one node, and it isnamed node1! ?ode names are simpl' names$ the' are used onl' inerror, #arning and informational messages to be read b' humans!DataStage internall' uses the node number$ the ordinal number, beginningfrom >ero, of the node #ithin the current configuration file! "herefore, inthe default configuration file sho#n here, 'ou #ould epect to see node O0referred to in areas li*e monitoring!

age 2+8


9/23


4fter each node name there is another set of curl' braces, #hich containsthe definition of that particular node! 4 node definition contains

the fastname of that node that is, the net#or* name of the

machine on #hich this nodeBs processing is to occur it is calledfastname because 'ou al#a's specif' the name b' #hich that

machine is *no#n on the fastest possible net#or* connection

a list of the node pools in #hich this node #ill participate this is

a space separated list introduced b' the #ord pools #ith one ormore node pool names each surrounded b' double @uote characters

a list of resources associated #ith the node resources called dis*

and scratchdis* are al#a's defined and there ma' be others dis*and scratchdis* resources ma' be further @ualified b' a list of dis*pools in #hich the resource #ill participate this is a spaceseparated list enclosed in curl' braces and introduced b' the #ordpools #ith one or more dis* pool names each surrounded b'

double @uote characters

4 node pool or dis* pool name that consists of a >ero length string, assho#n in the default configuration file displa'ed above, is the defaultpool! 4t least one node must belong to the default node pool unless ever'stage in the job design has an eplicit node pool specified!

Figure 2-1 DataStage Configurations Editor

age 2+P


10/23


From DataStage Designer client (under the "ools menu) 'ou can open atool called the 6onfigurations editor, #hich is illustrated in Figure 2+1! 4s#ell as being able to vie# the configuration files available on the s'stem,this editor allo#s ne# configuration files to be created from old, eistingconfiguration files to be edited;, and the validit' of a configuration file to

be chec*ed!/etBs eamine a configuration file from a L?-Q s'stem! ("he onl'difference on a &indo#s s'stem #ould be &indo#s pathnames!)C

/ #our$node confi%uration file four&node'apt // Created ())*$)($((+ ,a- .urlod /{

node "node)"{ fastname "dsdevsvr" pools "" "node)" "lo(" resource disk "/usr/data1/DataSta%e/disk))"

resource disk "/usr/data(/DataSta%e/disk))"

resource scratc!disk "/usr/tmp/DataSta%e/disk))" { resource scratc!disk "/var/tmp/DataSta%e/disk))" {pools "0uffer"node "node1"{ fastname "dsdevsvr" pools "" "node1" "lo(" resource disk "/usr/data1/DataSta%e/disk)1"

resource disk "/usr/data(/DataSta%e/disk)1" resource scratc!disk "/usr/tmp/DataSta%e/disk)1" { resource scratc!disk "/var/tmp/DataSta%e/disk)1" {pools "0uffer"node "node("{ fastname "dsdevsvr"

pools "" "node(" "!i%!(" resource disk "/usr/data1/DataSta%e/disk)("resource disk "/usr/data(/DataSta%e/disk)("

resource scratc!disk "/usr/tmp/DataSta%e/disk)(" { resource scratc!disk "/var/tmp/DataSta%e/disk)(" {pools "0uffer"node "node"{ fastname "dsdevsvr" pools "" "node" "!i%!(" resource disk "/usr/data1/DataSta%e/disk)"

resource disk "/usr/data(/DataSta%e/disk)" resource scratc!disk "/usr/tmp/DataSta%e/disk)" { resource scratc!disk "/var/tmp/DataSta%e/disk)" {pools "0uffer"

; Kost sites protect the 6onfigurations director' from being able to allo# edits other thanb' administrative staff! -f this is the case at 'our site, 'ou #ill receive an access deniedmessage #hen 'ou attempt to save changes!

C 6onfiguration files for grid environments have other re@uirements due to the d'namicnature of nodes resulting from grid management soft#are! "hese are be'ond the scope of thesenotes!

age 2+10


11/23


"his is clearl' an SK s'stem (the fastname is the same for all nodes)!"here are four single+node node pools, and t#o t#o+node node pools! "het#o+node node pools are named lo#2 and high2! "he four one+nodenode pools are epon'mousl' named for the node to #hich each relates!

?ode pool names can be dedicated! "here are several reserved node pool

names in DataStage, among them D32, -?F7HK-Q, 7H46/., S4S andsort! "he first four tie in #ith resources available on that node! "he sortnode pool can be used to specif' a separate scratch dis* resource that #illbe used b' sort rather than using the default scratch dis* dis* pool!

Directories listed under resource diskare used to store data in Data Sets,File Sets and /oo*up File Sets! &e shall see later that files stored in thesedirectories are recorded in a control file for each of these particularobjects!

Directories listed under resource scratchdiskare, as the name suggests,used for scratch space b' various *inds of operator! For eample, the

Sort stage allo#s a certain maimum amount of ph'sical memor' pernode to be allocated to the sorting process! 7nce all this memor' is in use,the sorting operation overflo#s into files in the scratch space! "hese filesshould automaticall' be deleted once the job completes successfull'!

Scratch space ma' also be used b' other facilities! For eample, if the7racle bul* loader (s@lldr) eperiences a problem and generates a badfile and a log file, and the designer has not specified locations #herethese should be #ritten, then DataStage #ill direct them to the scratchspace!

"he parallel engine uses the default scratch dis* for temporar' storage

other than buffering! -f 'ou define a buffer scratch dis* pool for a node inthe configuration file, the parallel engine uses that scratch dis* pool ratherthan the default scratch dis* for buffering, and all other scratch dis* poolsdefined are used for temporar' storage other than buffering! .achprocessing node in the above eample has a single scratch dis* resource inthe bufferpool, so buffering #ill use AvarAtmpADataStageAdis*nnbut #illnot use AusrAtmpADataStageAdis*nn (other operators #ill use this scratchspace)!

o#ever, if AvarAtmpADataStageAdis*nn#ere not in the bufferpool,both AvarAtmpADataStageAdis*nnand AusrAtmpADataStageAdis*nn#ould beused because both #ould then be in the default pool!

7ther resource t'pes ma' (indeed ma' need to) be defined, for eampleresource DB2, resource INFORI!, resource OR"C#Eand resourceS"S!

For more information on configuration files and the configuration fileeditor tool refer to the chapter on the parallel engine configuration file inthe DataStageParallel Job Developer Guide!

age 2+11


12/23


Configuration File in Grid Environment

-n a grid environment the available resources ma' var' over time, so that adifferent mechanism than a static configuration file is re@uired!

"o enable a d'namic configuration to be used, the Erid .nablement

"ool*it introduces a number of additional environment variables andscripts! "he Erid .nablement "ool*it environment variables are installedin all eisting DataStageARualit'Stage projects, and in the template from#hich ne# projects are created!

4ll but one of the environment variables have names beginning #ith%4"EH-D%! "he single eception is an environment variable called4"D?4K-667?F-ES6H-", #hich allo#s a name to be given tothe generated d'namic configuration file!

Erid+enabling a single job is triggered b' setting the environment variable4"EH-D.?43/. to .S and providing values to t#o othervariables

4"EH-D67KL".?7D.S T m#here mis the number of

machines from the grid on #hich processes #ill be run

4"EH-D4H"-"-7?S T n#here nis the number of

%processing nodes% (in the DataStage sense) to be created on eachof those machines

"he d'namic configuration file is dumped into the job log! -ts %nodes% arenamed according to the convention nodemn #here m is the machinenumber and n is the partition number on that machine! For eample, on a22 configuration (t#o compute nodes and t#o partitions per compute

node), the configuration file #ould refer to node11, node12,node21 and node22! ("his numbering is an eception to the >ero+based numbering encountered in most aspects of parallel job eecution!)

-f 4"EH-D.?43/. is not set to .S then DataStage #ill use thestatic configuration file #hose pathname is given in 4"67?F-EF-/.!

Se@uences use a different mechanism! Se@uences al#a's eecute on asingle machine! -n a se@uence 'ou begin #ith an .ecute 6ommandactivit' that eecutessequencer.shthat ta*es the name of the ne#configuration file and the number of compute nodes and partitions asarguments, and returns a space+separated list of t#o se@uential file host

names (discussed in a later module) and 4"67?F-EF-/. pathname!

Erid management soft#are uses the term %node% to mean a machine in the grid!DataStage uses the term %node% to mean a logical subset of available processing resources, asdescribed in the earlier section on configuration files!

age 2+12


13/23


Partitioning

4 partitioning algorithm describes to DataStage ho# to distribute ro#s ofdata among the processing nodes! "he partitioning algorithm to be used is

specified on the input lin* to a stage!-f 'ou do not eplicitl' specif' a partitioning algorithm, then (4uto) isfilled in, #hich instructs DataStage to choose the appropriate partitioningalgorithm #hen the job design is converted for parallel eecutionJ!

$ab%e 2-1 DataStage &artitioning "%gorith's

"%gorith'

(e)-

Based* Descri+tion

Hound

Hobin

? first ro# to first node (number 0), second ro# to

second node (number 1), and so onHandom ? ro#s randoml' allocated to nodes

Kodulus (integer) *e' value is divided b' the number ofnodes and the remainder used as the nodenumber to #hich to allocate the ro#

ash a hashing algorithm is allocated to the (an' datat'pe) *e' value to determine the node number to#hich to allocate the ro#

D32 ? matches the D32 partitioning algorithm

.ntire ? ever' ro# is placed on ever' node

Hange ? ro#s are allocated to nodes based on range ofvalues in a field, via a pre+created range map

4lso on the list of available partitioning algorithms are t#o pseudoalgorithms called (4uto) and Same!

(4uto) specifies that DataStage is to determine the partitioningalgorithm according to the follo#ing rules!

Lse Same on a parallel to parallel lin*!

Lse Hound Hobin on a se@uential to parallel lin*

Lse .ntire on the reference input lin* to a /oo*up stage!

Lse D32 on a lin* that connects to a D32 .nterprise stage or

D32 6onnector!

J For reasons that #ill become clear in a later module, this process is called composingthe score!

age 2+1;


14/23


"hese are not al#a's appropriate!

articular circumstances #here a better partitioning algorithm can befound essentiall' group into three categories #here *e' values need to becompared, #here *e' values need to be combined, and one special case,that of a loo*up in an K environment!

&here *e' values need to be compared, such as in a /oo*up, a Ioin, aKerge (master and updates) or #hen removing duplicates, then it is vitalthat all occurrences of each *e' value occur on the same node as eachother! "he abilit' to search other nodes for matching values is not part of#hat DataStage does$ eecution on each node is isolated from eecutionon other nodes!

&here *e' values need to be combined, such as in an 4ggregator, a Sortor some calculations in a "ransformer stage, again it is vital that alloccurrences of each *e' value occur on the same node as each other! Foreample consider a summar' b' state of LS data! -n a four+node parallel

configuration, #ith *e' adjacenc', the correct result (at most 0 ro#s) #illbe the result! &ithout *e' adjacenc', as man' as 200 ro#s (0 from eachof four nodes) ma' occur, #hich #ould not be the desired summar'!

"o get around the problem of *e' adjacenc' #ith a /oo*up stage, thedefault is to .ntire partition on the reference input! "hat #a', no matterho# the stream input is partitioned, ever' ro# of the reference input isfindable, because ever' *e' occurs on ever' partition! -n an SKenvironment this is a good #a' to do things due to the shared ever'thingnature of the environment in fact, the reference input is placed into sharedmemor' so that there is onl' one cop' of the Data Set! o#ever, in anK environment, .ntire partitioning #ill involve moving all the ro#s of

the reference Data Set to ever' processing node, via "6A-! "his ma' betoo great an overhead compared to using a *e'+based partitioningalgorithm on the reference input!

age 2+1C


15/23


Configuring Partitioning

.ver' stage t'pe that has an input lin* and #hich eecutes in parallelmode can have the artitioning properties of that input lin* configured!7n the input lin* properties page there is a tab captioned artitioning!

Figure 2-2 Configuring &artitioning

-n the artitioningA6ollecting frame there is a drop+do#n list #here thepartitioning algorithm (partition t'pe) ma' be selected! -f a *e'+basedpartition t'pe is selected, then one or more columns must be selected fromthe 4vailable columns field immediatel' belo# the drop+do#n list!

"o the right of the drop+do#n list is a tool that is enabled if certain of thealgorithms are chosen! For eample, if Hange is selected the name of theassociated range map must be specified$ if D32 is selected the name of theD32 database name, instance name and partition table must be specified!-f ?/S is enabled, this tool ma' also be used to select the appropriate

locale to use #hen performing calculations! /ocale can specif' sorting andcomparison order!

age 2+1


16/23


Configuring Collecting

.ver' stage t'pe that has an input lin* and #hich eecutes in se@uentialmode can have the 6ollecting properties of that input lin* configured! 7nthe input lin* properties page there is a tab captioned artitioning!

Figure 2-, Configuring Co%%ecting

-n the artitioningA6ollecting frame there is a drop+do#n list #here thecollecting algorithm (collector t'pe) ma' be selected! -f a *e'+basedcollector t'pe is selected, then one or more columns must be selected fromthe 4vailable columns field immediatel' belo# the drop+do#n list!

"o the right of the drop+do#n list is a tool that is enabled if Sort Kergealgorithms is chosen! "his tool ma' also be used to select the appropriatelocale to use #hen performing sorting!

age 2+1J


17/23


Parallel Data Storage

Ruic*est throughput in a DataStage job is achieved b' *eeping the ro#s inmemor'$ that is, not letting them touch do#n on dis*! "he logic behind

this statement is attractive$ memor' access times are faster than dis*access times b' several orders of magnitude!

o#ever there #ill be times #hen 'ou do not #ant etraction,transformation and load to occur all in one fell s#oop! For eample,there ma' be limited, and different, time #indo#s #hen the source andtarget databases are available! 4lso, if data are in memor' #hen somecatastrophic failure occurs that shuts do#n the machine, those data arelost! 6reating staging points for data affords an aid to restartabilit'!

Hecall, from earlier in this module, that the onl' input or output that canbe used #ith a DataStage operator is a Data Set! &hen resident inmemor' (that is, #hen directl' associated #ith a lin*), #e refer to avirtual Data Set!

6onceptuall' a Data Set spans all partitions, though ph'sicall' each nodehas its o#n subset of ro#s! -n addition, the use of node pools or dis*pools can limit the nodes on #hich a Data Set has ro#s!

4nother characteristic of a Data Set is that the data in it are in internalformat$ that is, for eample, numeric data are stored in binar' format#hile variable length character strings are stored in a fied si>e buffer andhave a numeric length indicator associated #ith them!

-t turns out that the data in a virtual Data Set can be stored on dis* in a

structure called a persistent Data Set!3oth *inds of Data Set have, #hen created, a control file that containsinformation about the record schema (the la'out of each record in the DataSet)!

For a virtual Data Set, this file has a name created from the stage

and lin* to #hich it relates, and a suffi of !v! Uirtual Data Setcontrol files are deleted automaticall' #hen the job finishes! "hename of a virtual Data Set control file can be seen in the osh scriptgenerated b' compiling a DataStage parallel job!

For a persistent Data Set, the control file has a name given b' the

developer, #hich conventionall' has a suffi of !ds! ersistentData Set control files contain not onl' the record schema but alsocontain the location of the ph'sical files on each processing nodein #hich the data are stored! "hese files al#a's reside indirectories mentioned in the resource diskportion of theconfiguration file!

age 2+1M


18/23


/et us contemplate, therefore, #hat a persistent (or virtual) Data Set loo*sli*e!

Figure 2- Contents of a Data Set

"here are ro#s of data on each of the nodes, and a control file that outlinesthe structure of each record!

&ithin that structure the final four fields are used to control such things as

preservation of sorted order in the Data Set! "hese are used onl' #ithinDataStage the' are no accessible to developers! o#ever, the recordschema can be vie#ed!

?ote that there is no mention in the control file, #hen #e are considering avirtual Data Set, of #here the data files actuall' are! "his is because the'are onl' in memor'! For a persistent Data Set the control file #ill alsocontain the pathnames of the data files for each segment of the Data Set!"hese ma' be vie#ed in the Data Set Kanagement tool!

Data are moved to and from persistent Data Sets using a Data Set stage,#hich in turn compiles to a cop' operator! "hat is, an' sorting and

partitioning in the Data Set is preserved! "his ma*es the Data Set the idealstructure for intermediate storage of data bet#een jobs! "hat is, if 'ouneed one job to store data on dis* for another job to pic* up later, then thepersistent Data Set should be the structure in #hich to store them!

-n DataStage, persistent Data Sets are read from or #ritten to using theData Set stage! "he' can be manipulated (inspected, copied or deleted)

age 2+18


19/23


using the Data Set Kanagement tool (from the "ools menu in Designer) orusing the orchadmincommand!

age 2+1P


20/23


File Set

4 File Set is a similar structure to a Data Set!

4 File Set has a control file (this time #ith a !fs suffi) and one or moredata files on each processing node! Data in a File Set preserve both sortedorder and partitioning!

"he difference is that the data stored in a File Set are in human+readable,or eternal format, and therefore must go through conversion from or tothe internal format used #ithin the DataStage job! 7n the other hand, datain a File Set are readil' accessible to other applications!

Figure 2-. Structure of a Fi%e Set

"he /file entries in the control file (on the right) are the ph'sical locations#here data are stored on each node! "he director' part of each of thesepathnames is the director' recorded as resource diskin the configurationfile for parallel eecution (the one referred to b' the environment variable4"67?F-EF-/.)!

Data in each of these files is human readable$ in this cases the format iscomma+separated values #ith strings double+@uoted, #hich can be seen in

the format portion of the record schema! "hese fields are not visible!

4s #ith Data Sets, the final four fields are used to control such things aspreservation of sorted order in the File Set! "he control file for a File Setis clear tet, so it is straightfor#ard for other applications to read it (thecontrol file) to determine the locations of data!

-n DataStage, File Sets are read from and #ritten to using the File Setstage!

age 2+20


21/23


oo!up File Set

"he /oo*up File Set is neither a File Set nor a Data Set, though it hascharacteristics of both!

-t has a control file that has a !fs suffi! "hat is its onl' resemblance to aFile Set!

-ts data are stored in internal format, but not in ro#s li*e in a Data Set!-nstead, ro#s are stored in bloc*s #ithin the /oo*up File Set, and an indeto the location of each ro# is built and incorporated into the /oo*up FileSet!

3ecause of this indeed structure, there is no capacit' for vie#ing the datain a /oo*up File Set there is no utilit' and the /oo*up File Set stagedoes not support a Uie# Data capabilit'!

"his structure can onl' be used to serve ro#s into the reference input of a/oo*up stage, #ith #hich #e #ill deal in a later module!

age 2+21


22/23


Review

ipeline arallelism is li*e a conve'or belt in #hich buffers are usedbet#een operators so that more than one ro# at a time ma' be passed!

artition arallelism involves splitting the data into ro#+based subsets andtreating each subset identicall' to the original design!

"here are seven artitioning 4lgorithms available in DataStage paralleljobs$ round robin, random, entire, hash, modulus, range and D32! ashand modulus are *e'+based algorithms! 4uto and same cause analgorithm to be selected #hen the job score is composed!

Situations in #hich *e'+based partitioning is necessar' to achieve correctresults include an' situation in #hich *e's must be compared (such as inthe /oo*up, Ioin, Kerge and Hemove Duplicates stage t'pes) or combined(such as in the Sort and 4ggregator stage t'pes)!

DataStage structures in #hich partitioned data can be stored include DataSets (in #hich data are stored in internal format), File Sets (in #hich dataare stored in eternal, or human+readable, format) and /oo*up File Sets(in #hich data are stored in an indeed internal format)! &e eaminethese structures in more detail in a later module!

Further Reading

"'pes of arallelism Parallel Job Developer Guide6hapter 2

6onfiguration Files Parallel Job Developer Guide6hapter 2

artitioning Parallel Job Developer Guide6hapter 2

Data Sets Parallel Job Developer Guide6hapter 2 and C

File Sets Parallel Job Developer Guide6hapter J

/oo*up File Sets Parallel Job Developer Guide6hapter M

age 2+22


23/23


02 principles of parallel execution and partitioning

Documents