EGrid Software Packages Overview

EGrid Introduction

Egrid Introduction : A description of the main software packages

EGrid Inside : A detailed description of the software packages

Data Management

Primitive data management operations (upload, download, withdrawal, etc.).

Operations common to all end users.

User requirements

Data management Transformations Filters

Transformations

Advanced data management operations.

Operations common to restricted group of persons or to individual scientists.

The same computational infrastructure is used by all project participants.

User requirements

Data management Transformations Filters

Filters

Advanced data management operations.

Operations particular to each researcher.

User requirements

Data management Transformations Filters

EGrid Inside

Choose your directions:

Data Management

Transformations

Filters

Data Management

OperationalClassification

Data upload Data download Data removal Data movements Data browsing

FunctionalClassification

Cache management. File management.

Data Upload

Uploads data in the localcache :

egrid-cache-upload <file> <lfn>Example:

egrid-cache-upload ./nyse-2002-01.tar.gz lfn:/00010/cd/nyse-2002-01.tar.gz

Uploads data in the main storage :

egrid-upload <file> <lfn>Example:

egrid-upload ./nyse-2002-01.tar.gz lfn:/00010/cd/nyse-2002-01.tar.gz

cache

Main storage

Data Download

Downloads a file from the Grid to the local machine :

egrid-download <lfn> …Example:

egrid-download lfn:/00010/cd/nyse-2002-01.tar.gz

cache

Main storage

storage

Data Withdrawal

Removes a file from the local cache :

egrid-cache-remove <lfn>Example:

egrid-cache-remove lfn:/00010/cd/nyse-2002-01.tar.gz

Removes a file from theGrid :

egrid-remove-file <lfn>Example:

egrid-remove-file lfn:/00010/cd/nyse-2002-01.tar.gz

cache

Main storage

storage

Data Movement - Replication

Replicates a file from the local cache to the main storage :

egrid-replicate-to-main <lfn>|<lfnpattern> …Example:egrid-replicate-to-main lfn:/00010/cd/nyse*

Replicates a file from the main storage to the local cache :

egrid-replicate-to-cache <lfn>|<lfnpattern> …Example:egrid-replicate-to-cache lfn:/00010/cd/nyse*

cache

Main storage

Data Movement - Copying

Copy a file from the local cache to the main storage with a new logical filename :

egrid-copy-to-main <lfn> <newlfn>Example:

egrid-copy-to-main lfn:/00010/cd/nyse-2001.tar.gz lfn:/brownlees/nyse-2001.tar.gz

Copy a file from the main storage to the local cache with a new logical filename :

egrid-copy-to-cache <lfn> <newlfn>Example:

egrid-copy-to-main lfn:/brownlees/nyse-2001.tar.gz lfn:/brownlees/nyse-test.tar.gz

cache

Main storage

Data Browsing

Shows files in the local cache:

egrid-cache-list-files <lfnpattern>Example:

egrid-cache-list-files lfn:/*brownlees*

Shows files in the Grid:

egrid-list-files <lfnpattern>Example:

egrid-list-files lfn:/*brownlees*

Shows where a file is located:

egrid-where <lfnpattern>Example:

egrid-where lfn:/*brownlees*

cache

Main storage

storage

Transformations

Operations regarding two classes of users

End users – users performing transformations.

Software developers – users developing transformation programs.

Transformations – End Users 1

Are mainly used to transform data CDs from a proprietary format to ASCII.

Are performed using the job submission mechanisms. Are independent on the data format offering the

possibility to reuse the same infrastructure in other contexts.

The module can be used as a starting point for custom transformations.

Transformations – End Users 2

Transforms a file applying a certain transformation :

egrid-transform <transformation> <lfn> <lfnprefix> Example:egrid-transform taq-trades lfn:/brownlees/nyse-test.tar.gz lfn:/brownlees/test-data/

egrid-transform <transformation> <lfn> <lfnprefix> [<config>]Example:egrid-transform taq-trades lfn:/brownlees/nyse-test.tar.gz lfn:/brownlees/test-data/ tickers.conf

Generates to standard output a .JDL file which can be further customized :

egrid-transform –n <transformation> <lfn> <lfnprefix> Example:egrid-transform -n taq-trades lfn:/brownlees/nyse-test.tar.gz lfn:/brownlees/test-data/

egrid-transform –n <transformation> <lfn> <lfnprefix> [<config>]Example:egrid-transform -n taq-trades lfn:/brownlees/nyse-test.tar.gz lfn:/brownlees/test-data/ tickers.conf

Transformations – Software developers

Guidelines are provided to software developers on how to develop transformation programs which can be interconnected with the Grid technologies.

What interfaces the programs must implement.

What the runtime environment is.

Where the programs are installed.

Filters

Are custom operations which are dependent on the researcher needs.

Software modules will be implemented on demand.

Example : Requirement : Get all the files containing data in a time interval. Solution : A program which analyses the LFNs in RLS and decides if the file

contains data in the requested interval.