the interactive european grid project

The Interactive European Grid Project

introduced by Dr. Jesús Marco de LucasCSIC, Instituto de Física de Cantabria, Santander, SPAIN

EELA Conference, 4 September 2006, Santiago de Chile

J. Marco , EELA Conference, 4 September, Santiago de Chile2

Example: Ultrasound Computer TomographyA new method of medical imaging based on the reconstruction by numerical techniques of an image, using as input the data measured by a scanner of ultrasounds which surrounds the object of interest. The application requires analyzing about 20 Gb of data, which would take order of one month in a workstation…

Researchers demand resources

Data Acquisition System

Site A

Site B

Site C

~ 20 GBytes

========

Total < 30 minutes

~ 4 GBytes

+

+

Bandwidths ~100 Mb/s

MatlabOctave Java

Data Acquisition System

Site A

Site B

Site C

~ 20 GBytes

========

Total < 30 minutes

~ 4 GBytes

+

+

Bandwidths ~100 Mb/s

MatlabOctave Java

“Interoperable production-level e-Infrastructure for

demanding interactive applications to impact the daily work of researchers”

• Distributed Parallel (MPI) Interactive Computing & Storage at the Tera level • User Friendly Access• Grid Interactive Desktop

int.eu.grid VISION

int.eu.grid in three slides


A Real Challenge!

the int.eu.grid project aims to change the way researchers can use the available e-Infrastructure, exploiting the interactivity and collaboration possibilities

Researchers need to be convinced that they can: Transfer and process gigabytes of information in minutesForesee more complex algorithms on larger statistics, test and tune them, use more powerful visualization techniquesCollaborate across the network in a rewarding mode, from sharing information to discussing and presenting remotely through enhanced videoconference environments.


Interactive European Grid(http://www.interactive-grid.eu)

Project acronym int.eu.gri

d

Contract number 031857

Instrumen

t I3

Duration 2 years

may ´06-april ´08

“providing transparently the researcher’s desktop with the power of a supercomputer, using distributed resources”

http://www.interactive-grid.euCoordinator: CSIC, Jesús Marco, IFCA, Santander, SPAIN

[[email protected]]


My summary of the GRID(testbed/integrator/developer/user view)

The Grid “idea” is as good as difficult to put in practiceUsing the network, and open standards, join resources from different sites and make them available to user communities in a transparent way:

Sit users in front of a 1000 GHz/ 100 Tbytes computer Theoretically, it makes sense:

The network is there now (and evolves even faster than CPUs!)Basic middleware exists (for a “secure” communication sharing resources)

KEY point: “SHARING” & “COLLABORATION”This is EXTREMELY difficult, why should we just bother?????Why not use just resources at one site (clusters, SC centers)?The answer is two-fold:

e-Science scenarioMeta-computing possibilities


About a 1000GHz/100Tbytes shared computer…

We have a clear idea of how to setup and also of how to use it!

LARGE CLUSTER

• Get an account/password from administrator• Develop your code, parallelize/distribute• Use a repository• Compile using MPI, make executable, store

it • Set links to data• Submit script to PBS or LSF queue• Monitor job status• Retrieve output & logs from queue• Display output

GRID FRAMEWORK

• Get certificate, register in Virtual Organization• Develop your code, use Grid API• Use a CVS repository to share with others• Compile using MPI, make executable, store

in Virtual directory (SE) • Define data lfn (will be stored in SE)• Submit to WMS/RB from a Grid Desktop• Check jobstatus• Get output, store in Virtual directory• Display output in Grid Desktop


Executing Applications: HTC vs.

HPC High Throughput Computing :

Maximize the number of program executionsExample: 1.000.000 independent simulations

Use 1000 processors, do 1000 simulations in each oneAggregate for final result

Works perfectly in a clusterExamples: Condor, batch queues: LSF, PBS

Works perfectly in a GRID with a Workload Management System

High Performance Computing:Minimize the answer timeExample: Train a complex Neural Network

Use 1000 processors in parallelThe information of all of them is required along the jobNeeds a “parallelized description” (e.g. MPI) + and a topology (master-workers)

How to handle network latency in a GRID framework? (or in general QoS)Metacomputing “issues”


Benefit for applications

y = 17180x-0.971

R2 = 0.9994

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

0 10 20 30 40 50 60 70

number of computing nodes

time

(seg

)

Time reduction: from 5 hours down to 5 min

using 60 nodes!

Scale as 1/N

412980 events,16 variables16-10-10-1 architecture1000 epochs for training

Modeling including latency <300 ms needed !

Parallelization techniques reduce waiting time in Neural Network Application


Innovation

What innovation do we need?

The CrossGrid project demonstrated the feasibility of an interactive Grid framework showing applications running in a testbed using middleware developed on top of EDG/LCG services. int.eu.grid will provide a production quality service supporting MPI and interactivity on top of EGEE services

Web Services approach will be followed (gLite as underlying middleware)

MPI support will be based on PACX-MPI + OPEN-MPIEuropean MPI software (HLRS) + International effortMPI across sites without the need for public IP on WNs

Interactivity support will integrate: User friendly interface to the infrastructure Powerful visualization capabilities

Consideration of QoS4


Single sign-on / authorisation

Platform independent

Batch jobs MPI jobs Running

interactive applications using java plugins or VNC

Monitoring grid applications

Flexible Application framework

User profile management

Easy application add on

Local and grid file management

Desktop tools: Job Wizard Job Monitor Application Container and Plugin

Grid Desktop

GridFTP Commander User Profile Manager Private Storage Management VNC/SSH console


Project Resources

Grid-empowered Infrastructure Initial core: 13 sites in Europe

linked through GEANT-2

ComputersClusters with 10-100 Pentium/Xeon nodesbut also Itanium, Opteron, Power5

2-4 processors nodes, multi core, large RAM

InteroperableEGEE batch jobs will be able to execute in background mode

3

Human resources Excellent subteams already “in place”, examples:

CrossGrid WP4-sites & Integration Team, driven by FZK & LIPPSNC-UAB-CSIC on interactivity, MPI use, and applications

Operation of infrastructureContinuous effort along the projectBenefit from support expertise for middleware Integration (experience from CrossGrid)

e-Infrastructure


Summary: the int.eu.grid Mission

“To deploy and operate a production-quality Grid-empowered eInfrastructure oriented to service research communities

supporting demanding interactive applications.”

Deployment of e-InfrastructureOriented to interactive useSite integration supportGrid operations service

Middleware for interactivity and MPIAdapt/integrate existing middlewareguarantee interoperability with EGEE

Provide a complete interactivity suite

Desktoproaming accessscheduler with prioritization servicescomplex visualization.

Sub-objectives through the I3 instrument:

Support for interactive applications:setup of collaborative environment and VOconsideration of performanceinteractivity and visualization requirementsidentification and selection of research

oriented interactive applications

Support remote collaboration activities:research, management, integration, training

Approach target research communities Provide security measures for interactivity

2


Potential Impact

INPUT TO/FROM OTHER PROJECTS

On experience, middleware, application support, training, test of infrastructure

Input to National Grid initiativesDGRID, AustriaGrid, GridIreland

Collaboration with EGEEAim for interoperabilityObjective: two way synergy: transfer experience and middleware to/from EGEETraining and DisseminationOffer complementary services to a large user base

Dissemination effortPresence in EGEE/GGF meetingsAddressing different research communities at their workshops (national and international wide contacts)

Ex. FZK, PSNC, LIP, TCDProfit from multidisciplinary framework

Ex. CSIC: horizontal program

11

APPLICATIONS:Medicine:

Ultrasound Computer Tomography Simulation, Visualization, Remote

AccessEnvironmental studies:

Landscape modelling , PollutionPhysics:

Astrophysics: Planck missionFusion: visualization and 3D transport codes HEP: filtering of data produced by ATLAS

Attracting others:Through dissemination activitiesIdentification of Applications

Outreach: workshops, seminars, coursesDefinition of requirements (and use cases) for a given application/communityAnalyze the possibilities for use of the eInfrastructureTest and train on real infrastructure

Profiting from links with National Grid InitiativesEGEEOther international projects (like EELA)


Collaboration with EELA

The Interactive European Grid open to collaboration

Listen/Support researchers needs for interactive demanding applications

Offer MPI expertise, in particular for distributed applicationsRun on resources adapted for MPI (multi cpu, large RAM, etc)

Interactive oriented setup

Test collaborative and user friendly frameworkTry Grid DesktopAdd Powerful visualizationJoin Access Grid collaborative experience

Collaborate on development/deployment issuesSimplified installation and virtualizationApplication Development framework

Common partners: LIP, CSIC

Hope this collaboration will be fruitful! Thanks! 4

the interactive european grid project

Documents

grid apiuse

grid contract number

interactive applications

researchers desktop

interactive european

way researchers

sharing information

available einfrastructure