catania grid & cloud engine mario torrisi (mario.torrisi@ct.infn.it) istituto nazionale di...

Catania Grid & Cloud EngineMario Torrisi (mario.torrisi@ct.infn.it)

Istituto Nazionale di Fisica Nucleare – Sezione di Catania

Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA

Outline The Catania Grid & Cloud Engine

Job Engine Architecture Features

Code Samples Portlet for job management

MyJobs mi-hostname-portlet mi-parallel-app-portlet

References Summary and conclusion

Catania - 12 Marzo 2014

The Catania Science Gateway model

AdministratorPower UserBasic User

Grid/Cloud/Local middleware supported so far

....... Science

App. 1 App. 2App.

Embedded Applications

Users from different

organisations having different

roles and privileges

Standard-based (SAGA) middleware-independent

Grid Engine

The Catania Grid & Cloud Engine

Grid/Cloud Engine

UsersTracking

Science GW Interface

SAGA/JSAGA API

Job EngineData Engine UsersTrack &Monit.

ScienceGW 1

ScienceGW 2

ScienceGW 3

Grid/Cloud/Local MWs

Liferay Portlets

eTokenServer

NewModified New Modified NewModified

Job Engine

The Job Engine is made of a set of libraries to develop applications able to submit and manage jobs on a grid infrastructure;

It is compliant with the OGF SAGA standard; It is optimized to be used in a Web Portal running

an application server (e.g. Glassfish, Tomcat,…) based on J2EE;

It could be used also in stand-alone mode; JSAGA is the SAGA implementation adopted.

A Simple API for Grid Applications (SAGA)

SAGA is a family of related standards to define application programming interface (API) for common distributed computing functionality

It is independent of the details of the underlying infrastructure (e.g., the middleware);

SAGA is an OGF specification: http://www.gridforum.org/documents/GFD.90.pdf

SAGA is composed by:

SAGA Core Libraries: containing the SAGA base system, the runtime and the API packages (file management, job management, etc.);

SAGA Adaptors: libraries providing access to the underlying grid infrastructure (adaptors are available for Globus, gLite, etc.);

N.B. SAGA defines a standard Catania - 12 Marzo 2014

We then need animplementation!

JSAGA is a Java implementation of SAGA developed at CCIN2P3;

JSAGA: Enables uniform data and job management

across different grid infrastructures/middleware;

Makes extensions easy: adaptor interfaces are designed to minimize coding effort for integrating support of new technologies/middleware;

Is OS indenpendent: most of the provided adaptors are written in full Java.

Job Engine Architecture

USERSTRACKING

JobsSubmission

JobsCheck status/

Get output

Worker Threads for Job Submission

Worker Threads forJob Check Status

Jobs Queue

WT WT WT

USER TRACK & MONITOR

Job Engine - Features

Job Engine Middleware Independent

JSAGA supports gLite, Globus,ARC, UNICORE, etc.

Adding new adaptors in JSAGAis a easy job

Job EngineEasiness

Allow to develop application able to submit jobs on the grid in a very short time;

A very intuitive API is exposed to the developers;

Support MPI applications; The developer has only to submit the job:

The Job Engine periodically check the job status;

When the Job is done, the job output is automatically downloaded by the Job Engine in the local machine.

Job EngineScalability (1/2)

The Job Engine is able to manage a huge number of parallel job submissions fully exploiting the HW of the machine where it is installed;

It enqueues all the parallel requests received serving it according to the HW capabilities;

The Job Engine thread pools can be configured to optimally exploit the HW capabilities;

A burst of parallel job submissions cannot damage the Job Engine responsiveness thanks to the protection provided by the thread pool mechanism.

Job EngineScalability (2/2)

The answer time is linear;

Response time depends on the HW capabilities and thread pools configuration.

Job EnginePerformance

All the delays due to grid interactions are hidden to the final users: The Job Engine provide asynchronous functions for each

job management actions (submit, check status, download output, cancel);

Final users “feel” a response time equals to 0. The Job Engine is able to submit thousands of jobs in

a short time: The Job Engine submit jobs using a configurable thread

pool; The Job Engine is able to submit 10000 jobs in less than 1

hour with 50 threads in the thread pool; Increase the number of threads in the thread pool, can

improve the Job Engine performance.Catania - 12 Marzo 2014

Job EngineAccounting

A very powerful accounting system is included in the job engine;

It is fully compliant with EGI VO Portal Policy and EGI Grid Security Traceability and Logging Policy;

The following values are stored in the DB for each job submitted: Users; Job Submission timestamp; Job Done timestamp; Application submitted; Job ID; Proxy used; VO; Site where the job is running (e.g., CE for EMI-gLite).

Job EngineFault tollerance

Job Engine implements an advanced mechanism to guarantee job submission: Developers can set an appropriate value of

“shallow retry”;

Automatic re-submission mechanism when a job is aborted: Hide every failure to the final users

Job EngineCollection, Parametric & Workflow

Currently the Job Engine natively support: Collection jobs; Parametric jobs; N-1 workflows.

In the next releases we’ll add the support for DAG (Direct Acyclic Graph)

Code ExampleJob submission

String wmsList[] = {"wms://wms-4.dir.garr.it:7443/glite_wms_wmproxy_server“,"wms://wms005.cnaf.infn.it:7443/glite_wms_wmproxy_server"/};

Resource Manager List

GEJobDescription description = new GEJobDescription();description.setExecutable("/bin/sh");description.setArguments("hostname.sh");description.setInputFiles("/home/mario/Documenti/hostname.sh");description.setOutput(“Output.txt");description.setError(“Error.txt");

Job Description

JSagaJobSubmission tmpJSaga = new JSagaJobSubmission(description);tmpJSaga.useRobotProxy("etokenserver.ct.infn.it", "8082", "332576f78a4fe70a52048043e90cd11f", "gridit", "gridit", true);tmpJSaga.setUserEmail("mario.torrisi@ct.infn.it");tmpJSaga.setResourceManagerList(wmsList);

tmpJSaga.submitJobAsync("mtorrisi", "193.206.208.183:8162", 1, "gLite Test job“);

Submit the Job

Code ExampleWorkflow N-1(1/4)

JOB 1 JOB 2 JOB 4JOB 3 …

Final Job

OUTPUT

InfrastructureInfo infrastructures[] = new InfrastructureInfo[2];

String wmsList[] = { "wms://wms005.cnaf.infn.it:7443/glite_wms_wmproxy_server" };infrastructures[0] = new InfrastructureInfo("gridit", "ldap://gridit-bdii-01.cnaf.infn.it:2170", wmsList, "etokenserver.ct.infn.it", "8082“, "332576f78a4fe70a52048043e90cd11f", "gridit", "gridit");

String globusList[] = {"wsgram://xn03.ctsf.cdacb.in:8443/GW"};Infrastructures[1] = new InfrastructureInfo("GARUDA","wsgram","", globusList, "etokenserver.ct.infn.it","8082","332576f78a4fe70a52048043e90cd11f","gridit","gridit");

Define Infrastructures

for (int i = 0; i < 3; i++) {GEJobDescription description = new GEJobDescription();description.setExecutable("/bin/sh");switch (i) {case 0:

description.setArguments("hostname.sh");description.setInputFiles("/home/diego/test_wf/hostname.sh");break;

case 1:description.setArguments("ls.sh");description.setInputFiles("/home/diego/test_wf/ls.sh");break;

case 2:description.setArguments("pwd.sh");

description.setInputFiles("/home/diego/test_wf/pwd.sh");break;

}description.set…descriptions.add(description);

Define First Level Jobs

GEJobDescription finalJobDescription = new GEJobDescription();finalJobDescription.setExecutable("/bin/sh");finalJobDescription.setArguments("ls.sh");String tmp = "";for(int i = 0; i < descriptions.size(); i++){

if(tmp.equals("")) tmp=descriptions.get(i).getOutput();else tmp+=","+descriptions.get(i).getOutput();

}finalJobDescription.setInputFiles(tmp + ",/home/diego/test_wf/ls.sh,/home/diego/test_wf/ifconfig.sh");finalJobDescription.setOutput("myOutput-FinalJob.txt");finalJobDescription.setError("myError-FinalJob.txt");

Define Final Job

JobCollection wf = new WorkflowN1("scardaci", "Workflow N-1", "diego.scardaci@ct.infn.it", "/tmp", descriptions, finalJobDescription);JobCollectionSubmission tmpJobCollectionSubmission = new JobCollectionSubmission(wf);

tmpJobCollectionSubmission.submitJobCollection(infrastructures, "193.206.208.18:8162", 1);

Submit the Workflow

Grid & Cloud Engine javadoc

http://grid.ct.infn.it/webinar-liferay/

Job Management portletMyJobs

Active Jobs

Job Status and Output

Search on job description

Job description

Job Management portletmi-hostname-portlet

Job Management portletmi-parallel-app-portlet

References

The Catania Science Gateway Framework: http://www.catania-science-gateways.it/

Grid & Cloud Engine Javadoc: http://www.catania-science-gateways.it/training-material

A Simple API for Grid Applications (SAGA): http://www.gridforum.org/documents/GFD.90.pdf;

JSAGA: http://grid.in2p3.fr/jsaga/;

Webinar on the DCH-RP e-Culture Science Gateway28

Thank you !

Catania - 24 June 2013

For more information on Catania Science Gateway framework please visit:

http://www.catania-science-gateways.it/

catania grid & cloud engine mario torrisi (mario.torrisi@ct.infn.it) istituto nazionale di...

Documents

nucleo poro nucleare involucro nucleare nucleolo sintesi...

fisica nucleare radioattività, fusione e fissione...

lucio torrisi wg2-cosmo gm, bucharest 2006 recents...

proliferazione nucleare

energia nucleare

the glite support system giuseppe la rocca infn catania...

epap ente di previdenza e assistenza pluricategoriale...epap...

centrale nucleare

introduzione al marketing docente gaetano torrisi

fusione nucleare - · pdf fileavvicinarsi abbastanza da...

#bmo15 | per una navigazione sicura | carmen torrisi

nucleare introduzione - hub - eniscuola · home / energia /...

torrisi casa volantino natale 2010

live tweeting highlights - salvatore torrisi e andrea scotti

lenergia nucleare. energia nucleare cose generalita reattore...

100 anni di caffè torrisi

the genius grid portal giuseppe la rocca infn catania...

infn-grid testbed monitoring system roberto barbera...

noidellalampedusagullotto dome-nica torrisi francesca grazia...

unione europea jorge sevilla cedillo...