m.kunze, nec2003, varna the european crossgrid project marcel kunze institute for scientific...

M.Kunze, NEC2003, Varna

The European CrossGrid Project

Marcel Kunze

Institute for Scientific Computing (IWR)

Forschungszentrum Karlsruhe GmbH

www.eu-crossgrid.org


Outline

Project Overview

Applications

Toolbox

Experience (Testbeds)


CrossGrid Collaboration

Poland:Cyfronet & INP CracowPSNC PoznanICM & IPJ Warsaw

Portugal:LIP Lisbon

Spain:CSIC SantanderValencia & RedIrisUAB BarcelonaUSC Santiago & CESGA

Ireland:TCD Dublin

Italy:DATAMAT

Netherlands:UvA Amsterdam

Germany:FZK KarlsruheTUM MunichUSTU Stuttgart

Slovakia:II SAS Bratislava

Greece:AlgosystemsDemo AthensAuTh Thessaloniki

Cyprus:UCY Nikosia

Austria:U.Linz

21 21 institutinstitutees s

11 11 countriescountries


GRIDLAB

GRIA

EGSO

DATATAG

DATAGRIDGRIP EUROGRID

DAMIEN

Science

IST Grid Project Space

Industry/Business Science

Applications

Middleware

Fabric

CROSSGRID


Mission Statement

Development of a Grid Environment

for

Interactive Applications

Make the Grid more user friendly, secure and efficient

Timeline: 2002-2005


Workpackages

WP1 – Application Development

WP2 – Grid Application Programming Environment

WP3 – New Grid Services and Tools

WP4 – International Testbed Organisation

WP5 – Project Management


Applications

Middleware & Tools

Fabric (Infrastructure)

Architecture: Layered Structure


WP1 –Application Development1. Interactive simulation and visualization

of a biomedical system2. Flooding crisis team support3. Distributed data analysis in HEP4. Weather forecast and air pollution

modeling

WP2 – Programming Environment

1. MPI code debugging and verification

2. Metrics and benchmarks 3. Interactive performance

evaluation toolsVisualization Kernel

Data Mining

WP3 – New Services

Globus Middleware

WP4 – Testbeds - Fabric Infrastructure

Grid Services:

DataGridGriPhyN

...

1. Portals and roaming access2. Grid resource management3. Grid monitoring4. Optimization of data access

High Level Architecture

Technical Components


Detailed Architecture

Supporting Tools

1.4Meteo

Pollution

1.4Meteo

Pollution

3.1 Portal & Migrating Desktop

3.1 Portal & Migrating Desktop

ApplicationsDevelopment

Support

2.4Performance

Analysis

2.4Performance

Analysis

2.2 MPI Verification

2.2 MPI Verification

2.3 Metrics and Benchmarks

2.3 Metrics and Benchmarks

App. Spec Services

1.1 Grid Visualisation

Kernel

1.1 Grid Visualisation

Kernel

1.3 DataMining on Grid (NN)

1.3 DataMining on Grid (NN)

1.3 Interactive Distributed

Data Access

1.3 Interactive Distributed

Data Access

3.1Roaming Access

3.1Roaming Access

3.2Scheduling

Agents

3.2Scheduling

Agents

3.3Grid

Monitoring

3.3Grid

Monitoring

MPICH-GMPICH-G

Fabric

1.1, 1.2 HLA and others

1.1, 1.2 HLA and others

3.4Optimization of

Grid Data Access

3.4Optimization of

Grid Data Access

1.2Flooding

1.2Flooding

1.1BioMed

1.1BioMed

Applications

Generic Services

GRAMGRAM GSIGSIReplica CatalogReplica CatalogGIS / MDSGIS / MDSGridFTPGridFTP Globus-IOGlobus-IO

DataGridReplica

Manager

DataGridReplica

Manager

DataGrid Job Submission

Service

DataGrid Job Submission

Service

Resource Manager

(CE)

Resource Manager

(CE)

CPUCPU

ResourceManagerResourceManager

Resource Manager

(SE)

Resource Manager

(SE)Secondary

StorageSecondary

Storage

ResourceManagerResourceManager

Instruments ( Satelites,

Radars)

Instruments ( Satelites,

Radars)

3.4Optimization of

Local Data Access

3.4Optimization of

Local Data Access

Tertiary StorageTertiary Storage

Replica CatalogReplica Catalog

GlobusReplica

Manager

GlobusReplica

Manager

1.1User Interaction

Services

1.1User Interaction

Services

CrossGridDataGridGlobus


CrossGrid Applications


Key Features of CrossGrid Applications

Data Data sources and data bases geographically

distributed

To be selected on demand

Processing Large processing capacity required; both HPC & HTC

Interactive

Presentation Complex data requires versatile 3D visualisation

Support for interaction and feedback to other components


Biomedical Application

CT / MRI scan

MedicalDB

Segmentation

MedicalDB

LB flowsimulation

VEWDPC

PDA

Visualization

Interaction

HDB

10 simulations/day60 GB/simulation> 20 MB/s


Bypass Surgery: Simulated Treatment Planning


Interactive Treatment Planning

The vascular geometry can be modified using a library of models

Draw interactively Computational geometry


Sample pulsatile Flow Simulation


Flood Simulation

Data sources

Meteorological simulations

Hydraulic simulations

Hydrological simulations

Users

Output visualization


3D Visualization


Flood Simulation: Flow and Water Depth


Distributed Data Analysis in HEP

Objectives Distributed data analysis

Distributed data mining techniques with neural networks

Issues Typical interactive requests will run on o(TB) distributed data

Transfer/replication times for the whole data about one hour

Data transfers once and in advance of the interactive session

Allocation, installation and set-up of corresponding database servers before the interactive session

Integration of user-friendly interactive access (based on PROOF)


Parallel ROOT Facility: PROOF

Local

Remote

Selection

Parameters

Procedure

Proc.C

Proc.C

Proc.C

Proc.C

Proc.C

PROOF

CPU

CPU

CPU

CPU

CPU

CPU

TagDB

RDB

DB1

DB4

DB5

DB6

DB3

DB2

http://root.cern.ch


Distributed/parallel codes on the Grid Coupled Ocean/Atmosphere Mesoscale Prediction System

STEM-II Air Pollution Code

Integration of distributed databases

Data mining applied to downscaling weather forecast

Weather Forecast and Air Pollution Modeling


Weather Forecast and Air Pollution Modeling


CrossGrid Toolbox


Migrating Desktop

Idea Save and resume a user grid session

Look and feel of a windows desktop

Implementation Roaming Access Server and Clients

Java Web Services (Portability)

Integration of Tools Job submission wizard

Job monitoring dialog

GridExplorer dialog

GridCommander dialog


User Login Dialog


Grid Commander Tool


Grid Explorer Tool


Job Submission Dialog


Job Monitoring/ Visualisation Plugin


CrossGrid Testbeds

Example sites: Valencia GoG farm, Santander (GridWall), FZK

Various instances: Production, Validation, Test


Testbed Monitoring

Mapcentergrid monitoring framework.

Mapcenter was developed by DataGrid and adapted to CrossGrid.

http://mapcenter.lip.pt


Production Resource Broker Statistics

Total users 61

Submitted 2903

Accepted 2716

Matching 2554

Sent by JSS 2514

Jobs run 2355

Jobs done 2299

Graphics withRB statisticsavailable fromMapcenter

EDG 1.4


Production RB Statistics (continued)

Most of the failures are related with: Authorization Matchmaking Testbed sites

Submitted 2903

Accepted 2716

Matching 2554

Sent by JSS 2514

Jobs run 2355

Jobs done 2299

187 Not accepted

162 Matching failures

40 Not submitted

159 Didn’t run

56 Didn’t reach the end

604 Jobs failedEDG 1.4


The Future


1980s: Internet 1990s: Web 2000s: Grid

Where do we need to get to ?

Applications to support an “e-society” (“Cyber-Infrastructure”)

A Grid infrastructure which hides the complexities from the users (“Invisible Computing”)

A powerful and flexible network infrastructure (GEANT 2)

Where do we need to invest ?Applications targeted at realistic problems in “e-science”

Prototypes of Grid infrastructures

Maintain and improve the GEANT network

EU FP6 proposal: EGEEVision: Create European e-Infrastructure

70 Partners, 31.5 MEuro

Start: April 2004

m.kunze, nec2003, varna the european crossgrid project marcel kunze institute for scientific...

Documents