www.eu-eela.org e-science grid facility for europe and latin america eela-2 jra1 services diego...

www.eu-eela.org

E-science grid facility forEurope and Latin America

EELA-2 JRA1 Services

Diego Scardaci

INFN (Italy)

Joint EELA-2/EGEE-III Tutorial for Trainers

Catania (Italy), 01.07.2008

Catania, Joint EELA-2/EGEE-III Tutorial for Trainers, 1° July 2008 2www.eu-eela.org

Outline

• NA3/JRA1 Synergies

• EELA-2 Services already available:

– GENIUS

– GFAL Java API

– gLibrary/DRI

– Grid2Win

– Secure Storage

– Storage Accounting (SAGE)

– Transactional Grid Storage Access Framework (T-GSAF)

– Watchdog

All information available on the JRA1 section of the EELA-2 wiki:

http://wiki.eu-eela.eu → JRA1


Grid Services Developing

Develop Services that will decisively contribute to a

durable, versatile and cost-effective use of the e-

Infrastructure

Applications Support

Provides support to EELA-2 Applications

Applications RequirementsIdentify Services & Tools useful for EELA-2 Applications

REQUIREMENTSIDENTIFICATION

SUPPORT

GRID SERVICES DEVELOPMENT

NA3 & JRA1 Synergies


EELA-2 Services

In the next slides we will see for each EELA-2 Service:• A brief introduction• Main features• Links to installation packages, web/wiki site,

documentations, etc.• RoadMap

All EELA-2 Services will soon be submitted under the evaluation of the

EGEE RESPECT programme


GENIUS

Gilda Team & Nice S.R.L.


GENIUS GRID portal

• It can be accessed from everywhere and by “everything” (desktop, laptop, PDA, cell phone).

• The same user interface to several back-ends.

• All available grid services incorporated in a logic way, just “one mouse click away”.

• Layout easily understandable and user friendly.

• Secure at all levels: – secure for web transactions– secure for user credentials– secure for user authentication– secure at VO level


Main advantages

Thanks to the GENIUS and EnginFrame (developed by the Italian Company NICE srl) user can

• Interact with files on the UI • Send jobs to the Grid • Manage the data belonging to the given Virtual Organization.

Moreover:• No needs of a particular Operating System and middle-ware

running on the client side


The Genius Home Page(https://glite-tutor.ct.infn.it)


GFAL Java API

Diego Scardaci

INFN – Catania


GFAL Java API

Grid File Acces Library (GFAL):• Client POSIX (like) I/O library for directly data access.

Aim:• Provide GFAL API to Java Developers.

Why Java?• Reduce developing time;• More Java developers are “available”;• More applications are “pure Java” applications.


GFAL Java API Classes

• GFalFile: implements methods to manage remote files (stored on a SE).

• GFalDirectory: implements methods to manage SE remote directory.

• GFalUtilities: provides some important utility methods.


References

• GFAL C API:http://grid-deployment.web.cern.ch/grid-deployment/gis/GFAL/GFALindex.html

• GFAL Java API wiki page:https://grid.ct.infn.it/twiki/bin/view/GILDA/APIGFAL

• GFal Java API Javadoc:https://grid.ct.infn.it/twiki/GFAL/


A GRID based platform to host multiple repositories for digital content

A. Calanducci, J.M. González, R. Ramos, M. Rubio, D.TcaciEELA-II Kick-Off Meeting

20th-23rd April 2008 – Trujillo (Spain)


• A platform to easily build digital repositories on a grid infrastructure– Digital Repository: annotated digitalized data offered in a

structured manner to users

• Features:– Host multiple repository of arbitrary structure (defined by

repository providers)– Totally grid (gLite) based:

Authentication/Authorization based on GSI/VOMS Digitalized data files saved on any SRM Storage Element Annotations stored on Metadata Service (AMGA) or RDBMS Specific algorithm to process data run through the grid WMS

– Easy-to-use: AJAX web front-end offering a-la-iTunes browsing– a DRI APIs (Java based) available to easily deploy new

repositories

What is gLibrary/DRI ?


gLibrary/DRI web interface


• One repository must provide• A description of its navigational structures (trees, filters) and viewers• A description of its data model• A storage engine (for data model persistence)

• The DRI API specification describes HOW this is provided

• In practise, three Java modules should implement the following interfaces:

• DRIUIInterface for describing trees, filters and viewers

• DRIStorageInterface for storing and retrieving data

• DRINodeInterface for defining repository data model• One repository can

• Make its own implementation of the specification• Use the default one provided

gLibrary/DRI API


Mammograms rep with viewer


Web interface Servlets, php, java bridge Applets

• For user authentication with their VO certificate

• For viewers implementation Java Introspection XML AMGA, gLite Java APIs

18

Technologies involved


• Engine deployed and working, API and default implementation (save metadata on AMGA) working

– MGPlus repository implemented on DRI– Generic Uploader ready

Current work: Interaction with LFC/SE improvements Multi-node selection for loading and viewing

• To Do:– Deployment system for new repositories– Generic viewer for repository that doesn’t provide a specific one– Run grid jobs with selected algorithms with data coming from

repositories– Secure Storage/GSAF Integration– Service Orientation for the DRI Engine (Web Service)– Documentation, packaging, installation systems to be RESPECT

compliant

Current status


• gLibrary/DRI presentation at the EGEE User Forum III in Clermont-Ferrand– https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-UF3.pdf

• Paper submitted to the 21st International Symposium on Computer Based Medical Systems (CBMS 2008)– https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-CBMS08.pdf

• Chapter submitted for the “Handbook of Research on Computational Grid Technologies for Life Sciences, Biomedicine and Healthcare”– https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-chapter.doc

• Poster presented at the User Forum III– https://glibrary.ct.infn.it/glibrary/downloads/DRI/poster_glibraryDRI%20final.pdf

• Developer’s web server:– https://dri-dev.ceta-ciemat.es/index.php

• Wiki (internals only)– http://www.ceta-ciemat.es/projects/CETA-DRI/

References


Grid2Win Porting of gLite middleware to Windows

Platform

Dario Russo, Fabio Scibilia, Elisa IngràINFN –Catania


• The Grid2Win project aims to create a gLite User Interface (UI) and a gLite Computing Element (CE + WNs) running on Microsoft Windows.

Gatekeeper + LRMS

User Interface

Linux WNs

WMProxy

User Interface

WindowsXP Wns

new users come in to the grid !

windows applications can run on the grid !

Grid2Win

• Cygwin emulates a POSIX environment on Windows;

• CE based on PBS and Microsoft CCS


We have ported:

• Grid Security Infrastructure: Credentials management with VOMS extensions commands (voms-proxy-*)

• Workload Management: Job submission and output retrieving commands (edg-job-*)

• File Transfer: Secure transfer with credential management GSIFTP (globus-url-copy) RFIO commands (rf*)

• Data Management: File Catalogue Browsing commands (lfc-*) and data movement & replication (lcg-*)

• Information System: Command lcg-infosites and existing LDAP Explorer integrated in our GUI.

Grid2Win – Main Features


Links

You can download the last versione of Grid2Win from the project homepage :

http://grid2win.gilda-forge.ct.infn.it/new/index.php

Installation and configuration instructions (for VO different than GILDA) can be found here:

https://grid.ct.infn.it/twiki/bin/view/GILDA/Grid2WinGUI

The online Documentation:

http://grid2win.gilda-forge.ct.infn.it/onlineDocumentation/index.php

You can read more about the project here:

http://grid2win.gilda-forge.ct.infn.it/Grid2Win_ETNGrid2007_final.pdf


T-GSAF

Transactional Grid Storage Access Framework

Salvatore ScifoConsorzio Cometa – Catania

Salvo ParisiIR&T Engineering s.r.l. - Catania


gLite C/C++ API weakness

• LGC Utils is the only super interface– Doesn’t include GsiFTP– Doesn’t include GFAL– Misses Metadata integration (AMGA)– messy overlap among different semantic APIs

GFAL is only a POSIX-like file manage interface, it should not wrap LFC API than just RFIO would be enough

• LGC Utils is not a framework

LCG Utils APIGFAL API

LFC API

CNS API RFIO API

Globus Security API

GridFTP API SRM API

Security

POSIX

File Management

File Catalogue

DMS API

• LGC Utils is not GOOD for Data Grid Application


GSAF -Data Grid Application solution

• GSAF is an Object Oriented Framework– Java 100% - Design Pattern based– Clear design : object modeling / functional modeling– Solves the fragmentation of DMS APIs– Solves the natural (due to OGSA) inconsistence of services– Provides ACID Transactional Model (coming soon)

• Road Map– RFIO java native implementation– SRM encapsulation– Grelc integration– Secure Storage integration

GSAF

LFC API

CNS API RFIO API

Globus Security API

GridFTP API SRM API

Security

POSIX

File Management

File Catalogue

DMS API

AMGA API

Road Map• New– LFC API– CNS API

• AMGA is supported

• only one super interface for Data Grid Application

• modular and extensible


GSAF as Design Help• We have common features, we have common problems we need a

Design Pattern

GRID FARM(Redundancy, High Availability, Data Backup&Recovery, High Storage Capability, Net Access Security)

GRID Metadata Service GRID Data Service

Grid Storage Access Framework(Data access, API Fragmentation, Vertical architecture, Knowledgment gap)

Healt Science Application Earth Science Application Cultural Heritage Application

• Built on top of the Grid Metadata Service and Grid Data Service– collects and implements functionalities shared among applications

according to “write once use anywhere” principle

– reduces the knowledge gap hiding the complexity and the fragmentation of the several underlying APIs exposing a unified interface more near to the developer mind (design patterns)

rather than the Grid stuff details (API syntaxes)

– acts as a black box providing classes and related methods for applications located above interfaces to extend the implemented capabilities


1° Use Case: DMS Web Interface


Transactional GSAF

• Transaction Manager– Provides developers a Transaction Pattern

autocommit(), execute(), commit(), rollback() performs several data manipulation in Atomic Mode

– Hold Runtime Exceptions to limit Data Inconsistence– System Crash Recovery to avoid Data Loss

• Challenge– SOA of Grid middleware makes transactions impossible at

server side– GSAF works at client side

Not full access to resources and services (locks) Not cache mechanism to save temporary data (memory) Not backup system to store previous data status (restore) Not full access control to distributed resources (centralization) …..


State of the Art

• GSAF – Framework (v1.01) released– Candidate at RESPECT program– Adopters

ADAT Project (Archdiocese of Catania): Cultural Heritage Digital Archive (antique manuscripts)

BM Portal project (Bio-Lab, DIST University of Genoa ) :Service platform to deploy and delivery bio medical and bio informatic applications

Aiuri project (COPPE/UFRJ - BRAZIL):aims to implement a Grid Oriented platform to support data and text mining applications uses GS.

– References http://gilda-forge.ct.infn.it/projects/gsaf/ https://grid.ct.infn.it/twiki/bin/view/PI2S2/GSAF http://www.ieeexplore.ieee.org/xpl/freeabs_all.jsp?isnumber=4407105&arnu

mber=4407174&count=98&index=68

• T-GSAF– Analysis and Design (progress)– Development, testing, production (roadmap)


A Secure Storage Service for the gLite Middleware

Diego Scardaci – INFN Catania

Giordano Scuderi – UNICO S.R.L.


Insider Abuse: the problem

• A grid user could store sensitive data in a Storage Elements managed by external organizations.

• Storage Elements Administrators could access data (but the data are sensitive!). For this reason data MUST be stored in an encrypted format.

• Data Encryption/Decryption MUST be performed inside user secure environment (for example inside the user’s organization).


SE

Key Repository

SE

USER (VIRTUAL) ORGANIZATION

Key

File Encryption /Decryption

Encrypted File

Encrypted File

Insider Abuse: the solution

SECURE ENVIRONMENTSECURE ENVIRONMENT


A Secure Storage service for the gLite Middleware

• Provides gLite users with suitable and simple tools to store confidential data in storage elements in a transparent and secure way.

The service is composed by the following components:• Command Line Applications: commands integrated in

the gLite User Interface to encrypt/upload and decrypt/ download files.

• Application Program Interface: allows the developer to write programs able to manage confidential data .

• Keystore: a new grid element used to store and retrieve the users’ keys.


lcg-scr: Encryption and Storage


lcg-scp: Retrieval and Decryption


The Keystore

• The Keystore is a new grid element used to store and retrieve the users’ key in a secure way.– is identified by an host X.509 digital certificate;– all its Grid transactions are mutually authenticated and encrypted

as required by the GSI model;– should be placed in a trusted domain and should be

appropriately protected by undesired connections;– is a black box with a single interface towards the external world.

This interface accepts only GSI authenticated connections;– the client request is processed only if the client is a member of a

enabled users list and/or it belongs to an enabled Virtual Organization;

– if the client want to retrieve a key, the keystore checks if the request is coming from an authorized user inserted on the ACL associated to the request key.


Storage Accounting for Grid Environments

Fabio Scibilia

Consorzio COMETA

Catania, Italy


SAGE: Accounting of Storage Resources

• Accounting in gLite:– CPU: DGAS makes accounting of CPU

cycles consumed by jobs.– Storage: Currently no system exists.

• SAGE: Storage Accounting for Grid Environments– Measures disk space used by files

through a scandisk and by intercepting accesses to files

– Will be integrated with the database of DGAS (HLR).

– Allow users to create reports on the usage of disk space

• Measure of unit:– Considers both file life time and its space

in bytes– Consumption is defined as the integral of

the file size (bytes) in time (seconds).

• http://sage.gilda-forge.ct.infn.it/

logsData

Collecting

User

pullSAGE-

Database

push

Data Accounting

Data Monitoring

HLR

DPM

DGAS


Features

• Interacts with the Mass Storage System (MSS) to resolve local files into grid files.

• The local MSS (DPM by now) is unaware of being monitored

• Can be extended to more with more MSS by adding some simple C++ classes to the library

• It can scale to many disk servers in the same pool

• It uses MySql as DBMS to store accounting data

• Accounting is made per file, per user, per VO, per day.


Example of web reporting in SAGE

VO admin report generation

Consider this storage

Consider these VO (P.S.: you must be a VO-admin to ask for this)

In this time range

Tell me user subjects

All the whole site

Detailing on these VO

Billing of users (one by one)

Disk usage in

MB*hours


Links

SAGE on GILDA forge

- http://sage.gilda-forge.ct.infn.it/

SAGE on EELA-2 Wiki:

- https://grid.ct.infn.it/twiki/bin/view/EELA2/StorageAccounting


WatchDog Monitoring and controlling job execution

on the WorkerNode

Riccardo Bruno

INFN – Catania


Why do we need a WatchDog ?

• Why– Expecially long term jobs require monitored and controlled during their execution.

• How– Perform job control and monitoring using grid services in the less invasive way.

• Observations– Almost all jobs submitted on the grid are piloted by shell scripts

Shell scripting allow to get precious info in case of faults Shell scripting can pilot more complex batch execution

– Both AMGA and SE+LFC can be used as the simplest IS on the grid. lfc-* and lcg-* tools already available for file creation and retrieve The latency of CLI tools for the storage is very low compared to long term jobs

• Requirements:– Monitor job execution watching snapshot of files produced by the job execution

File snapshot will be reported on LFC+SE or AMGA servers– It would be useful to configure the monitoring tool accordingly to the user needs

Few shell environment variables can be used to configure the watchdog tool– Control the job execution accessing directly on the WN

It is possible to send commands on the WN


How does the WatchDog work ?

• The Watchdog is a shell script to be included in the JDL main script.

– Some watchdog features: It starts in background before to run the

long term job The watchdog runs as long as the main job The main script can control, stop and wait

until the watchdog has finished Easily and highly configurable and

customizable The watchdog does not compromise the

CPU power of the WN Amga LFC+SE

Commandsor Scripts

JDL

WN

File snapshots

WN

WN

UI

UI UI


Links

• WatchDog wiki page– https://grid.ct.infn.it/twiki/bin/view/EELA2/WatchDog


EELA-2 Apps. & JRA1 services(a proposal) (1/2)


EELA-2 Apps. & JRA1 services(a proposal) (2/2)

www.eu-eela.org e-science grid facility for europe and latin america eela-2 jra1 services diego...

Documents