NGS UEEUniform Execution Environment

Jonathan ChurchillSTFC RAL 6 May 09

• A wrapper script to run an application….• in a standard location …– /usr/ngs

• with a standard ‘API’.

What is it ?

#!/bin/bash## Run script for PFT2## Jonathan Churchill# Version Date Description#-----------+---------------+-------------------+# 1.0 | 5-Sep-08 |Initial release |#-----------+---------------+-------------------+

# Load the required module.. /etc/profile.d/modules.shmodule load pft3dr/32/2.0

# and run the program/apps/pft3dr/2.0/bin/.exe/pft2 $*exit $?

How’s that clever ?• Under NGS1 there was NGSMODULES:

• Under UEE we instead have# globusrun -b -r ngs.rl.ac.uk/jobmanager-lsf \ '&(executable=/usr/ngs/PFT2) \ (jobType=single)(stdout=pft2.so)(stderr=pft2.e'

# globusrun -b -r ngs.rl.ac.uk/jobmanager-lsf \ '&(environment (NGSMODULES pft3dr/32/2.0)) (executable=/usr/local/bin/pft2)(jobType=single)

(stdout=pft2.so)(stderr=pft2.e)'

How’s that clever (2)?

• Benefits for users:– Same “executable” at all supporting sites.– Commandline same/similar to launching on their

desktop.– Only need the site + UEE name to run a globus job. – Common app launch page for an application on the

NGS web site regardless of installer site (see later)

• Benefits for Sites – Environment modules installation not required.– jobmanager NGSMODULES hack not required.– I get to write your app launch documentaton ;-)

What about MPI ?

• UEE Script wraps the MPI linked executable.

• mpirun added (as normal) by jobmanager

#!/bin/bash## Run script for DLPOLY2 2.1.6## Load the required module.. # Includes LD_LIBRARY_PATH entries for libmpi.so etc. /etc/profile.d/modules.shmodule load dl_poly/2.16

# and run the programDLPOLY.X $*exit $?

What about MPI ?

• So a normal globus mpi job submission :

• Gets translated by the job manager to :(Apologies for LSF centric batch script ! And missing globus detail)

# globusrun -b -r ngs.rl.ac.uk/jobmanager-lsf '&(executable=/usr/ngs/DLPOLY2_2_1_6) (jobType=mpi)(count=8) (stdout=out.o)(stderr=out.e)'

#!/bin/bash#BSUB –n 8#BSUB –o out.o#BSUB –e out.empirun /usr/ngs/DLPOLY2_2_1_6

API and SW Versions• All sites must follow the same “API” for a UEE.• Lead site for an app sets the API.

Captures it: https://www.ngs.ac.uk/ops/privops/ngs2_config/UEEdocs.html

• UEE Name must follow the format:APPNAME_1_2_3

– No ‘_’ in APPNAME– APPNAME must all by uppercase (for BDII see later)– First ‘_’ Denotes start of primary version name/number/dd,ww,mm combination– Version numbers don’t have to be digits or have major and minor release numbers.

• Format assumed by consumers– Eg Applications list table at : http://www.ngs.ac.uk/apps

• Sym link from /usr/ngs/APPNAME to /usr/ngs/APPNAME_1_2_3– Users who don’t care get a bigger selection of sites to run at using /usr/ngs/APPNAME– Users who care can picked the specific version they need using /usr/ngs/APPNAME_1-2-3– Indicates the ‘default’ not the latest.

UEE API

• Note: Eg Emboss and BLAST TOOLBOX use 1 UEE script for different binaries. Stops generating 100’s of UEEs

Effects of other formats

Other sites have _7_1_12Digit not letter

Has no underscore

Mix and match digit and letter in same version field : 8a

One place to publish them all

• BDII• GlueHostApplicationSoftwareRunTimeEnvironment

• Values are UEE script names

• vdt/1.10.1/lcg/var/gip/plugin/ngs-uee-gip-plugin

And one place to use them.

• Resource Broker JDL.

• Users also don’t need local home dir names

Type = "Job";JobType = "Normal";Executable = "/usr/ngs/BLAST-NCBI";Arguments = "blastall -p blastn -d est_hum -i aadnafra.fsa -a 1";StdOutput = "blast_ncbi.out";StdError = "blast_ncbi.err";InputSandbox = {"aadnafra.fsa"};OutputSandbox = {"blast_ncbi.out","blast_ncbi.err"};Requirements = Member("BLAST-NCBI", other.GlueHostApplicationSoftwareRunTimeEnvironment);ShallowRetryCount = -1;

Where can I run this job ?

• Cool ! Lots ofplaces to run ?

• Yes but how do Glasgow supportgridPP and NGShere ?

# glite-job-list-match blast1.jdl

Selected Virtual Organisation name (from UI conf file): dteamConnecting to host ngsrb01.ngs.rl.ac.uk, port 7772

*********************************************************** COMPUTING ELEMENT IDs LIST The following CE(s) matching your job requirements have been found:

*CEId* ngs.oerc.ox.ac.uk:2119/jobmanager-pbs-workq ngs.rl.ac.uk:2119/jobmanager-lsf-ngs svr021.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q1d svr021.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q2d svr021.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q30m svr021.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q3d svr021.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q6h svr026.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q1d svr026.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q2d svr026.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q30m svr026.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q3d svr026.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q6h ngs.leeds.ac.uk:2119/jobmanager-pbs-mpi*********************************************

GridPP/NGS sw Publishing

VO-NGS.AC.UK

• GridPP prefix with VO-<voname> to APPNAME• Eg: Brunel publish

VO-biomed-GATE-4_0_0 and VO-biomed-GATE-3_1_2– Same app as running at RAL (GATE_4_0_0, GATE_3_1_2)

• Which one do you use? How do you request it?• We need to republish as:

VO-NGS.AC.UK-APPNAME_1_2_3• But publication transform (in ngs-uee-gip-plugin).• Not by UEE name change

• Loose approximation to user familiar command names.

Documention.

• Application Docs:• http://www.ngs.ac.uk/sites/ral/applications• Linked from the http://ngs.ac.uk/apps table• Command examples written to work at all NGS sites:• Highlights the jobmanager string to change

• NGS3 +/- WMS work packages: • Opportunity for site agnostic application docs set• RB/WMS :

• No home directory specifics• No Job manager specifics

References

• Original UEE Definition– http://www.grid-support.ac.uk/files/UniformExecutionEnvironment.pdf

• UEE API– https://www.ngs.ac.uk/ops/privops/ngs2_config/UEEdocs.html

• How to install an App on the NGS– https://www.ngs.ac.uk/ops/privops/ngs2_config/ngs2-apps-install-flow.html

• NGS Apps tables– http://www.ngs.ac.uk/apps