EGI: European Grid Infrastructure

Steven Newhouse

Interim EGI.eu Director

EGI-InSPIRE Project Director

Technical Director EGEE-III1

European e-Infrastructure

• European Data Grid (EDG)– Explore concepts in a testbed

• Enabling Grid for E-sciencE (EGEE)– Moving from prototype to production

• European Grid Infrastructure (EGI)– Routine usage of a sustainable e-infrastructure

2

What is e-Infrastructure?

• Resources linked by high speed networks– Compute, Storage, Instruments, ...

• Controlled access to shared resources– Authentication, Authorisation, Accounting, ...

• Dependable services for others to use– Driven by availability and reliability metrics

• Services that are there for the long-term– Supporting experiments lasting decades

Swiss Grid Day 3

The EGEE-III Project17,000 users

139,000 LCPUs (cores)

25Pb disk

39Pb tape

12 million jobs/month

+45% in a year

268 sites

+5% in a year

48 countries

+10% in a year

162 Virtual Organisations

+29% in a year

Over 20 active communities in 112 VOs

EGICollaboration

NGI

NGI

NGI

NGI

ResearchCommunity

ResearchCommunity

ResearchCommunity

ResearchCommunity

EIRO

EIRO

EGI.eu

ResearchCommunity

5

EGI.eu

• Coordination for European Grid resources– Roadmap to integrate HTC, HPC, Data, Instruments, ...– Policy & services needed to run a grid

• Governance & ownership by its stakeholders– EGI Council votes proportional to GDP– EGI Council fees proportional to votes– Sustainable small coordinating organisation (EGI.eu)– Builds on resources from within its stakeholders

• Location selected to be Amsterdam– Distributed staff with a core (~50%) in Amsterdam– Approximately 40 staff, €3.5M/year

Swiss Grid Day 6

EC Funding

National Funding

Relationship betweenNGIs & EGI

Swiss Grid Day 7

NGI National Tasks (NGI)

NGIInternational

Tasks (EC & NGI)

NGI National Tasks (NGI)

NGIInternational

Tasks (EC & NGI)

NGI National Tasks (NGI)

NGIInternational

Tasks (EC & NGI)

EGI.euEGI Global Tasks (EC, NGI, EGI.eu)

EGI means Innovation

• Deploy Technology Innovation– Distributed Computing continues to evolve

• Grids Desktops Virtualisation Clouds ?

• Enable Software Innovation– Provide reliable persistent technology platform

• Today: Tools built on gLite/UNICORE/ARC

• Support Research Innovation– Infrastructure for data driven research

• Support for international research (e.g. ESFRI)

Swiss Grid Day 8

Technology Innovation

• Will come from outside EGI– EGI runs DCI technologies in production

• Partnership with technology projects– EMI (European Middleware Infrastructure)

• Continued evolution of gLite, ARC, UNICORE

– RESERVOIR• Public cloud functionality to ERA

– StratusLab (Proposed)• Use RESERVOIR to support gLite environments

Swiss Grid Day 9

Swiss Grid Day 10

Prototype

Requirements

Implementation

Usage

Release

Evaluate

Deployment

Feedback

Collaboration

Technology innovations taking place outside the production infrastructure

Innovating technology being deployed within the production infrastructure

NGIs & EIROs

EC &National

ResearchProjects

Release

Software Innovation

• Will come from outside EGI– EGI is a neutral platform for applications

• EGI cannot support all services in its core– Every community needs something different

• Foster innovation within ‘sectors’– High Throughput Computing– Digital Libraries– ...

Swiss Grid Day 11

EGEE Maintained Components External Components

Example of Deployed Services

12Physical Resources

General Services

LHC FileCatalogue

HydraWorkload

Management Service

File TransferService

Logging &Book keeping

Service

AMGA

Storage Element

Disk Pool Manager

dCache

Information S

ervices

BDII

MON

User InterfaceUser Access

SecurityServices

Virtual Organisation Membership

Service

Authz. Service

SCAS

Proxy Server

LCAS & LCMAPS

Compute Element

CREAM LCG-CE

gLExec

BLAH

Worker Node

User Interface

BDII

CORE UMD SITE SERVICES (TBC)

COMMUNITY SERVICES

RESPECT USER

Deployed by EGIMaintained & Supported

through EGI

Deployed by EGIMaintained &

Supported by the community

Digital Library Community

• Community specific software stack– gCube from D4Science project

• VREs: Manage access to libraries– Big focus on usability

Swiss Grid Day 13Physical Resources

Core UMD Services

Information Service

Virtual Research

Environment (VRE)

Manager Service Container

VREServices

VREServices

High Throughput Computing

• Managing large collections of jobs & files– HEP, LS, ...

• Problem specific Applications and Tools– Maintained & supported by that community

Swiss Grid Day 14Physical Resources

Core UMD Services

GANGA DIANE

HTC Community Services (e.g. WMS, FTS, LFC , AMGA, ...)

User ApplicationsWISDOM

Research Innovation

• An infrastructure to support the ERA– Within the EU27– Geographical Europe– Interoperability worldwide for collaboration

• Work with Virtual Research Communities– Groupings of aligned Virtual Organisations– Provide community specific:

• Support, training, consultancy, requirements etc.

Swiss Grid Day 15

Swiss Grid Day 16

The ESFRI Projects

• First roadmap 2006 and updated in 2008 with 44 projects

• Preparatory phase funding for most with second round soon

• Big push in FP8?

ESFRI: European Strategy Forum on Research Infrastructures

ESFRI Environmental Sciences

LIFEWATCH

EMSO

IAGOS-ERI

AURORA BOREALIS

EUFAR-COPAL

ICOS

EURO-ARGO

EPOS

EISCAT-3D

SIOS

Status 2009

17Swiss Grid Day

EGI User SupportU

SE

RS VOs

VOs

Virtual Research

Community

UserForum

SteeringCommitteeU

SE

RS VOs

VOs

Virtual Research

Community

US

ER

S

NGI

NGIHelpdesk

EGI.eu

Training Events

Trainers

Apps.DB

EGIHelpdesk

VRCHelpdesk

OtherHelpdesk

ESFRI Project

VOs

Virtual Research

Community

18

Swiss Grid Day 19

Moving from EGEE to EGI

• What is different?– EGEE did ‘everything’– EGI focuses just on infrastructure operations

• What is the same?– Running a 24/7 production quality infrastructure– Providing a support framework for the users– Collaboration to drive European DCI forward

The EGI-InSPIRE Project

Integrated Sustainable Pan-European Infrastructure for Researchers in Europe

• Submitted to INFRA-2010 1.2.1• Targeting call objectives:

– 1.2.1.1: European Grid Initiative– 1.2.1.2: Service deployment for heavy users

• A 4 year project• Seeking a total €25M EC contribution

– Project cost €70M

How much does it cost?

24

25

Annual Running Costs EGEE-III (M€)Staff (EC & National)

Hardware (National)

5127

14

8

% Staff Effort in EGEE-III

Operations User CommunityMiddleware Coordination

Annual CostsEGEE EC Contribution: €16M

(EGI Equivalent: ~€11M)Infrastructure Cost: ~ €80M

EGI Project Ecosystem

EGI

SoftwareProviders

SoftwareProviders

SoftwareProviders

Com

munity

Com

munity

VRC

VRC

SSC

SSC

EC Projects

VRC

SpecialisedSupportCentres

SupportingProjects

Virtual ResearchCommunities

22

Project Structure

23Project Management

External Relations: Policy, Dissemination and Standards

SA3SoftwareSupport

SA2Middleware

Unit

SA1Operations

JRA1Operational

Tools

SA4Services for

HUCs

NA3User

SupportSoftwareProvisioning

andSupport

Operations and Tools

User Support and

Services

Project Structure

EGI-InSPIRE

Managerial Collaborations (e.g. EC, DEISA, OSG, ...)

Technical Collaborations(e.g. CHAIN, GISELR, SIENA, ...)

VRC & SSCProjects

(e.g. ROSCOE, SAFE, CUE, TAPAS, ...)

SoftwareProjects

(e.g. EMI, IGE, SGI...)

User Support & Services

• Support User Communities– Researchers in International Collaborations– Scale up from the single VO to a community

• Provide a federated Helpdesk linking:– Discipline specific support (e.g. Bio Apps)– National infrastructure support (e.g. NGS)– Generic services (e.g. Training)

• Provide core services to support users– Manage VOs, Application DB, Training DB

Swiss Grid Day 25

Possible Virtual Research Communities (FP7 funding)

• High Energy Physics (ROSCOE)• Life Sciences (ROSCOE)• Computational Chemistry & Material Science (ROSCOE)• Grid Observatory [Computer Science] (ROSCOE)• Complexity Science (ROSCOE)• Photon Science (ROSCOE)• Humanities (ROSCOE)• Astronomy & Astrophysics (SAFE)• Fusion (SAFE)• Earth Science (SAFE)• Molecular Biodiversity (SIMBIOME)• NMR in Structural Biology (WeNMR)• Hydro-Meteorology (DRIHME)

Swiss Grid Day 26

Training Events

Registry of Trainers

Applications Database

29

http://appdb.eu-egee.org

Alt. link: http://grid.ct.infn.it/egee_applications/

EGEE RESPECT EGI

• Identify third-party software that works well with Grid– gLite: http://technical.eu-egee.org/index.php?id=290 – Expand to include UNICORE, ARC, Globus, …

• Simplified Access– P-GRADE, Ganga, Migrating Desktop,

g-Eclipse, i2glogin, Virtual Control Room• Workload Management

– GridWay Metascheduler, DIANE• New Resources

– GRelC, Instrument Element• Infrastructure Services

– StoRMSwiss Grid Day 30

Other Activities

• Dissemination– With NGIs, VRCs, SSCs and other projects

• Support for Heavy User Communities– General & community specific services

• Events– Two Annual meetings: Users & Technology

• Technology– Liaison with software providers– Definition and verification of requirements

Swiss Grid Day 31

Resources

• EGI.eu does not own or control resources• Resources owned by individual organisations

– They manage access for their user communities• Middleware enables secure authorised access

for external collaborators– EGI provided Unified Middleware Distribution (UMD)– EGI defined software & interfaces– UMD compatible software must be deployed– Interoperation within your country and internationally

Swiss Grid Day 32

Transition to EGI Model

• Has already started within EGEE-III• Will continue in EGI-InSPIRE

– Integrated Sustainable Pan-European Infrastructure for Researchers in Europe

– 4 year project, seeking €25M EC contribution• User facing changes will be gradual

– Focused communication from EGI & EGEE• Main public points of contact have few changes

– Helpdesk & software

Swiss Grid Day 33

Summary

• Move away from a project based model– Sustainable e-Infrastructure

• User services will evolve to this new model– Coordination of services provided through NGIs– Expect little change in the delivered services

• Start coordinating European e-Infrastructure– 1st May 2010

• Contact: director@egi.eu

Swiss Grid Day 34