egee-ii infso-ri-031688 enabling grids for e-science p-grade overview and introduction: workflows ...
DESCRIPTION
Enabling Grids for E-sciencE EGEE-II INFSO-RI Scaling up a workflow to a parameter study with P-GRADE Complete workflow Output file set in an LFC directory (e.g. lfc:/grid/gilda/kozlovszky/myoutputs) Input file set in an LFC directory (e.g. lfc:/grid/gilda/kozlovszky/myinputs)TRANSCRIPT
![Page 1: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/1.jpg)
EGEE-II INFSO-RI-031688
Enabling Grids for E-sciencE
www.eu-egee.org
P-GRADE overview and introduction: workflows & parameter sweeps(Advanced features)
Miklos KozlovszkyMTA [email protected]
![Page 2: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/2.jpg)
2
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Parameter Sweep applications
• Typical usage of the Grid is that the same application is executed with many different parameters
• P-GRADE portal from release 2.5 supports this functionality
• The next part of the lecture shows how parameter sweep (PS) applications are supported by P-GRADE portal
![Page 3: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/3.jpg)
3
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Scaling up a workflow to a parameter study with P-GRADE
Complete workflow
Output file set in an LFC directory (e.g. lfc:/grid/gilda/kozlovszky/myoutputs)
Input file set in an LFC directory (e.g. lfc:/grid/gilda/kozlovszky/myinputs)
![Page 4: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/4.jpg)
4
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Turning a WF into a parameter study
By turning at least one of the open input ports into a “PS Input port”
the WF is turned into a Parameter Study WF
![Page 5: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/5.jpg)
6
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
PS Input Port of Simple PS
Remote file Directory
instead of FILE reference
Do not use the prefix lfn:
if the directory isEGEE Grid file
catalogue
![Page 6: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/6.jpg)
7
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Simple PS Activity 2: placement of result
Properties of the VO File Catalog
Menu item PS Properties is available in the Workflow
menu
The Output directory will contain the ZIP-ped
result files of the Workflow executions One SE of the VO
– Choose a reliable one!
![Page 7: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/7.jpg)
8
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Executing PS workflows
PS Details for parameter sweep
workflow applications
![Page 8: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/8.jpg)
9
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Workflow Manager List PS Details view showing eWF-s
New, middle level list to render the details of a PS Workflow.(Status of complete application)
Statistics shows the progress of the elaboration of the whole PS
The eWorkflow buffer list shows the state of the Workflows being processed.
![Page 9: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/9.jpg)
10
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Details view of the eWF Ax_EQU_B_voce_PS.6
Job level details of an eWorkflow
See, that the button Attach is missing as there is not to
much importance to access the WE until the eWorkflow list
is exhausted
![Page 10: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/10.jpg)
EGEE-II INFSO-RI-031688
Enabling Grids for E-sciencE
www.eu-egee.org
Advanced PS WFs in P-GRADE Portal
![Page 11: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/11.jpg)
12
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Advanced parameter studies
Generator component(s)
Initial input data
Generate orcut input into smaller pieces
Collector component(s)
Aggregate result
Files in an LFC directory (e.g. /grid/gilda/kozlovszky/myinputs)
Files in an LFC directory (e.g. /grid/gilda/kozlovszky/myoutputs)
Complete workflow
![Page 12: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/12.jpg)
13
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Concept of advanced parameter study workflows
GEN
SEQ
COLL
SEQSEQSEQ
Parameter study part
Collector part evaluates and
integrates the results
Generator part generates the
input parameter space
![Page 13: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/13.jpg)
14
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
The generic PS workflow application
Third phase:
executing the
collectors
Second phase:
executing the “real”
processor jobs for each
input file combination
First phase: executing
the generators
![Page 14: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/14.jpg)
16
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Parameter generator
Generator can be attached to any parameter input port
Generator can be• Auto generator: to generate text files• Custom generator: to generate any content
Generated files are moved into SE by the portal
![Page 15: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/15.jpg)
17
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Definition Window of Auto Generator Job
User defines the template of the text file
User puts keys into the template
User defines values for the keys• Integer number• Real number• Custom set• …By clicking on
a key the definition
window for this key is opened.
![Page 16: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/16.jpg)
18
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
(Auto) Generator Attribute Editor for SE definition
Attribute Editor defines the properties of remote files created by the Generator:
1. Storage Element must be defined if an LCG like (EGEE ) file access has been defined in the PS Output Port belonging to the Generator
![Page 17: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/17.jpg)
19
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Detailed view of a PS workflow
Workflow instances
Overall statistics of workflow instances
Collector job(s)
Generator job(s)
![Page 18: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/18.jpg)
EGEE-II INFSO-RI-031688
Enabling Grids for E-sciencE
www.eu-egee.org
PS practical
![Page 19: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/19.jpg)
21
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
Matrix multiplication PS10 PS workflows
Multiplication job
1 2 34 5 6
7 8 12
2 1 31 1 13 3 3
Matrix2
1 2 34 5 67 8 9
1 2 34 5 6
7 8 15 . . .
. . .
. . .
![Page 20: EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows parameter sweeps (Advanced features)](https://reader035.vdocuments.mx/reader035/viewer/2022062401/5a4d1b7b7f8b9ab0599b918c/html5/thumbnails/20.jpg)
22
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
ExerciseMatrix multiplication PS
Multiplication job
Auto generator
Auto generator
Input files stored on SEs and registered in LFC catalog
1 2 34 5 67 8 9
1 2 34 5 6
7 8 12
1 2 34 5 6
7 8 15
1 2 34 5 67 8 Y
9 <= Y <=36, step 3
2 1 31 1 13 3 3
Matrix2 Output files stored on SEs and registered in LFC catalog
. . .
. . .
. . .