EGEE – EGI TransitionJürgen Knobloch, CERN

EGI_DS Technical Director

WLCG OB

CERN

13 November 2009

J. Knobloch/CERN: European Grid Initiative – EGI

EGI_DS – European Grid Initiative – Design Study

• EGI Design Study started Sept 2007 to establish a sustainable pan-European grid infrastructure after the end of EGEE-3 in 2010

• The main foundations of EGI are 37 National Grid Initiatives (NGIs)

• Project is funded by the European Commission's 7th Framework Program

www.eu-egi.eu

Work packages Lead by

Project managementLudek MatyskaCESNET, Czech

RepublicEGI Requirements Consolidation

Fotis Karayannis, GRNET, Greece

EGI Functions DefinitionLaura PeriniINFN, Italy

Design Study of EGI Legal and Organisational Options

Beatrice MerlinCNRS, France

Establishment of EGIJürgen Knobloch

CERNEGI Promotion and Links with Other Initiatives

Per ÖsterCSC, Finland

Other EGI_DS members:• DFN-Verein (DFN), Germany• LMU, Germany• Science & Technology Facilities

Council (STFC), UK

22-Oct-09 2

The Road to the EGI Blueprint

22-Oct-09 3J. Knobloch/CERN: European Grid Initiative – EGI

WP3 consolidation WS

29-31-Jan-08

• MW: Coordination – no development

• Provision of a large-scale production Grid infrastructure

Rome workshop13-14-March-08

D5.1: Draft Definition of EGI OrgD5.2: Draft Convention

Geneva workshop30-June-08

Istanbul workshop22-Sep-08

D4.4: Early Draft of Blueprint

Second Draft of Blueprint

Feedback

Requirementsfrom NGIs, EGEE

Final Blueprint

Prague Policy Board20-Jan-09

Feedbac

k

Feedback

Feedback from:NGIs,EGEE,

other GridsPRACE,WLCG,

etc.

Blueprint approved!

The EGI Actors

J. Knobloch/CERN: European Grid Initiative – EGI

NGI2NGI1

NGIn

…

EGI

National Grid Initiatives (NGIs)Resource Centres

Research TeamsResearch Institutes

EGI.euEGI.eu

22-Oct-09 4

EGI Management/Governance

J. Knobloch/CERN: European Grid Initiative – EGI

EGI.euEGI Director

Non-voting Representativesextra-EU NGIs, Chair of UFSC, …

Associate Memberse.g. EIROforum member, …

MembersNGI1, NGI2, NGI3, … NGIn

EGI Council

EGI

AdvisoryCommittees

e.g.Middleware

CoordinationBoard(MCB)

User Forum Steering

Committee(UFSC)

UserForum (UF)

MiddlewareUnit

Administration& PR Unit

OperationsUnit

User Community Services

CTOMiddleware

Maintenance

CAOAdmin & PR

UCOUser Coordination

COOOperations

22-Oct-09 5

EGI Financial Reference Model

J. Knobloch/CERN: European Grid Initiative – EGI

NGI International tasks

Middleware maintenance EGI.eu Total

Operations User services

C o s t sEGI Effort (FTEs) 225 110 70 51

EGI costs (M€/year) 20.25 9.90 6.30 4.59 41

I n c o m eNGI membership fees (M€/year) 1,0 1

EC Contribution (M€/year) 10.125 4.95 3.15 1.80 20

NGI, Middleware consortia and EIRO/ESFRI co-funding Contribution (M€/year)

10.125 4.95 3.15 1.80 20

22-Oct-09 6

7

EGI Operations Model

NGI’s are the main actors• Aims

– Autonomy for NGIs using a common infrastructure for local and international work for greater efficiency

– Sustainability– Subsidiarity: do things at as local a level as possible

(EGI.eu pulls things together) – Increased reliability through pushing responsibility

down to sites– Preserving current scalability in presence of more

middleware stacks to be supported, more non-EGEE Grids integrated, ...

J. Knobloch/CERN: European Grid Initiative – EGI 22-Oct-09

8J. Knobloch/CERN: European Grid Initiative – EGI

EGI Operations Tasks (1/2)

Performed by the NGI’s with the support of EGI.eu

• Operation of tools and services– Grid configuration repositories– Grid accounting repositories– Grid repositories for SLA compliance and performance

monitoring– Grid operations portal– Monitoring of status of services operated by sites

• User support– Central ticket handling system– Gathering requirements for user support tools and

process

22-Oct-09

9J. Knobloch/CERN: European Grid Initiative – EGI

EGI Operations Tasks (2/2)

• Security– Security policy development and maintenance– Coordination of security and incident response– Expert team for security vulnerabilities in MW

• Other international tasks– Middleware deployment/roll-out and support– Brokering availability of NGI’s resources for VO’s– Interoperations between NGI’s and with other grids– Network support– Definition of best practises, procedures, requirements– Catch-all production grid core services

22-Oct-09

J. Knobloch/CERN: European Grid Initiative – EGI

Characteristics of NGIs

Each NGI … should be a recognized national body with a single

point-of-contact … should mobilise national funding and resources… should ensure an operational national grid

infrastructure… should support user communities (application

independent, and open to new user communities and resource providers)

… should contribute and adhere to international standards and policies

Responsibilities between NGIs and EGI are split to be federated and complementary

22-Oct-09 10

The NGI Tasks

• NGI International Tasks– Necessary for the functioning of any NGI

• EGI Global Tasks– Central tasks part of EGI.eu not necessarily centrally

located– Bidding by NGIs – selection procedure – review by

Council

• Regional Operations Centres (ROC) NGIs– ROCs organized by geographical region– Migration will proceed in steps– Decentralise key tasks– Devolution to NGIs will continue after EGEE III

22-Oct-09 11J. Knobloch/CERN: European Grid Initiative – EGI

2. EGI_DS Review

www.eu-egi.eu12

NGIs in Europewww.eu-egi.eu

Grid Middleware

22-Oct-09 13J. Knobloch/CERN: European Grid Initiative – EGI

European Grid Initiative

EGI

NGI2

NGI1

NGIn…

…

Institute i Institute Z

Discipline A Application X

VOA VOX1VOX2

VOXn

Authentication of users

Authorization of VOs to run jobs

Distribution and scheduling of jobs

Monitoring of submitted jobs

Accounting of users and VOs

Management of data

14J. Knobloch/CERN: European Grid Initiative – EGI

Role of EGI.eu in Middleware• Central technical coordination by EGI.eu middleware unit

(led by CTO)• Objectives for EGI in middleware field

– Maintenance– Support– Development for standard compliance and stack convergence – Developments/Procurement as required by Applications and

Operations– Certification– Establishment of common requirements for interoperability

and new developments• Only last 2 points by EGI.eu personnel (8 FTE’s) • The other ones by Consortia seeking EU co-funding in

agreement with EGI, but outside its budget– A first estimation for support, maintenance and devel of the

components in the list in the next slide is about 100 FTE’s

22-Oct-09

J. Knobloch/CERN: European Grid Initiative – EGI

EGI.eu Functionality Overview

• Management, Outreach & Dissemination - Representation of EU Grid Efforts

• Operations & Resource Provisioning & Security

• Application Support & Training• Middleware (Build&Test, Component

Selection/Validation/Deployment)• Standardisation & Policies• Industry take-up

22-Oct-09 15

16J. Knobloch/CERN: European Grid Initiative – EGI

Application Support and Training

• The goal of EGI is to offer the European Research Teams the means for an easier and more efficient use of IT resources, for their Applications

• The primary actors of grid Application Support and Training are the NGI’s

• International Applications will require coordinated NGI’s effort– EU co-funding expected for this critical NGI

international task– Coordination implemented via Specialised Support

Centres, manned and controlled by the NGI’s• For EGI.eu only some specific “gluing” tasks

22-Oct-09

Transition in progress

• The transition is in progress – already achieved these steps:– Acceptance of the strategy defined in the Blueprint by the EGI_DS

Policy Board;– Choice of the location of the central EGI location, EGI.eu (previously

called EGI.org) in Amsterdam;– Acceptance of the EGI Memorandum of Understanding (MoU) and

subsequent establishment of the EGI Council;– Appointment of the interim EGI Director;– Publication of calls for proposals in the 7th Framework of the European

Commission targeted to EGI and related e-infrastructures;– Establishment of editorial structure for the preparation of proposals for

EGI-InSPIRE, SSCs, Middleware, and other topics related to EGI;– gLite Consortium – Collaboration Agreement signature progressing;– Drafts of proposals being finalized 11 days to go!– S. Newhouse elected director of InSPIRE

22-Oct-09 18J. Knobloch/CERN: European Grid Initiative – EGI

Memorandum of Understanding

• Goals– Identify parties ready to commit resources– Establish EGI (interim) collaboration

• First MoU draft at Policy Board March 2nd – Followed by work of legal experts

• Second MoU draft May 7th – Accompanied by Letter of Intent (LoI) to sign

• MoU distributed for signature June 13th • 24 signatures Council meeting July 9th

22-Oct-09 19J. Knobloch/CERN: European Grid Initiative – EGI

EGI Council

22-Oct-09 20J. Knobloch/CERN: European Grid Initiative – EGI

22 Oct 2009:33 NGIs2 EIRO (CERN, EMBL)+ Observers (Armenia, Belarus, Georgia,Kazakhstan, Moldova, Russia, Ukraine)

EGI Questionnaire to NGIs

• March 6th submitted questionnaire to Policy Board members – 24 countries, institutions filled questionnaire:

• Austria, Belgium, Belarus, Bulgaria, CERN, Croatia, Czech Rep, Finland, France, Germany, Greece, Ireland, Italy, Lithuania, Moldova, Netherlands, Poland, Portugal, Slovakia, Spain, Sweden, Switzerland, Turkey, UK

– 10 countries provided only contact names:• Albania, Belarus, Denmark, NDGF, Georgia, Latvia, Norway, Russia, Slovenia,

Ukraine

• Outcome:– Established 7 mailing lists on specific topics– The required functions are well covered by NGIs– The person power matches approximately the

blueprint needs22-Oct-09 21J. Knobloch/CERN: European Grid Initiative – EGI

Responses

22-Oct-09 22J. Knobloch/CERN: European Grid Initiative – EGI

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Active relationship to non-EUGrid Dissemination team?

WLCG relationship?International Application Support team?

International Application Support?Documentation training team?

MW rollout responsibility in future?MW rollouut team

Preproduction/pilot service?Responsility for Op Tools

OpTool development & deployment team?Will take responsibility for net op

team taking care of network operation for GridTake responsibility for security

Have team for devt and enforcement of security policy?EGEE security policy unchanged?Monitor and implement security

Monitor and escalate support tickets?Run helpdesk?

Take responsibility for monitoring?Team for monitoring fct?

Monitor and enforce SLAs?SLAs with sites?

First line support for ops?Integrate w central msg system?

Use SAMAP?Operate accounting DB and portal?

Operate national Dashboard?Operate repositories storing results from probes?

National GOCDB?Account repository about usage?

Plan CA responsibility?Creating a VO?

Use a certification authority?

Yes

No

Country 1 4 6 7 12 13 14 15 16 17 18 19 20 21 23 25 26 30 31 32 34 36 37 39 41 43 44 46 48 53 54 57 59 63 SumY

Bulgaria 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 33Lithuana 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 33Italy 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 32Greece 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 32

CERN 1 1 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 32

Spain 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 1 1 1 1 1 0 1 1 1 1 1 31UK 1 1 1 1 1 1 1 1 0 1 1 0 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 31France 1 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 31Netherlands 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 1 1 1 1 30Turkey 1 1 1 1 1 1 1 1 0 1 1 0 0 0 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 28Poland 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 0 1 1 1 0 28Sweden 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 1 0 0 28

Moldova 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 0 1 1 1 1 1 1 1 0 1 0 27Czech Republic 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 1 0 1 1 26Ireland 1 1 0 1 1 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 1 0 1 1 26Finland 1 1 1 1 1 0 1 1 0 0 1 1 1 1 1 1 1 1 0 1 1 1 1 0 0 1 0 1 1 0 1 1 1 0 25Germany 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 0 1 0 0 0 1 1 0 0 1 0 1 1 1 25Portugal 1 1 1 0 1 1 1 0 1 1 1 1 0 1 1 0 1 1 1 1 0 1 0 0 0 1 1 1 1 0 0 1 1 1 24Croatia 1 1 1 1 0 0 0 0 0 0 1 1 1 1 1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 0 0 0 1 0 20

Belgium 1 1 1 0 0 0 0 1 0 0 1 0 0 1 1 1 0 1 1 1 1 0 1 0 0 0 1 1 1 0 1 0 0 0 17Switzerland 1 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 0 1 1 1 1 1 1 1 1 1 0 0 0 1 0 15Austria 1 1 0 1 0 0 0 0 0 0 1 0 0 1 0 1 1 1 1 0 0 0 0 0 0 1 0 1 1 0 0 1 1 0 14Belarus 0 1 1 1 0 0 0 0 0 0 0 1 0 0 0 1 1 1 1 0 0 1 1 0 0 0 0 0 1 0 0 0 1 0 12Slovakia 1 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 1 1 0 0 1 0 0 0 0 0 0 1 1 1 0 1 0 10Sum 23 22 20 19 16 16 19 17 14 16 22 16 13 18 19 19 20 22 18 19 18 17 16 15 16 20 20 21 21 11 18 14 22 13

NGI vs Task

22-Oct-09 23J. Knobloch/CERN: European Grid Initiative – EGI

SLAsGOCDBAccountingDashboard

Ope

ratio

ns T

ools

Dvt

.

Appl

icati

on S

uppo

rt T

eam

WLC

G R

elati

onsh

ip

EGEE Communities

22-Oct-09 24

>270 VOs from several scientific domains:• Astronomy & Astrophysics• Civil Protection• Computational Chemistry• Comp. Fluid Dynamics• Computer Science/Tools• Condensed Matter Physics• Earth Sciences• Fusion• High Energy Physics• Life Sciences

0

20

40

60

80

100

120

140

160

Time

other VOs

LHC

EGEE CPU usage[units 1 G.SI2K.Hours]

J. Knobloch/CERN: European Grid Initiative – EGI

CPU usage increased *10

The second year of EGEE III

• Close collaboration EGEE III ↔ EGI_DS• EGEE III second year:

– Management – strong links with project proposal preparation

– Dissemination and training – prepare EGEE community for EGI and move activities to NGIs

– Application community – move towards structures defined in EGI blueprint; prepare the transition from scientific discipline clusters (SDCs) to Specific Support Centres (SSCs)

– Operations – prepare move from central to national operations

– Software certification – forming integrated product teams

22-Oct-09 25J. Knobloch/CERN: European Grid Initiative – EGI

EGI proposals being developed

• EGI-InSPIRE (Integrated Sustainable Pan-European Infrastructure for REsearchers)– Transition to a sustainable federated European e-infrastructure built from

national resources– Deliver production quality services to all its user communities and users– Provide a scalable user support model for international research communities

within Europe and their worldwide collaborators– Interoperation and interoperability with other e-infrastructures

• ROSCOE (Robust Scientific Communities for EGI)– Provide SSC service for heavy grid users (HEP, Life Science, Computational

Chemistry & Materials Science Technology, Grid Observatory, Photon Science, Complexity Science, and Humanities)

• SAFE (SSCs for Astronomy & Astrophysics, Fusion & Earth Science)– Astronomy & Astrophysics, Fusion & Earth Science

• EMI (European Middleware Initiative)– Expected to be the major middleware source for the Unified Middleware

Distribution (UMD) serving EGI (and also PRACE)

22-Oct-09 26J. Knobloch/CERN: European Grid Initiative – EGI

Other related proposals

• Extra-European Collaboration –CHAIN– CHAIN (Co-ordination and Harmonisation of Advanced e-

Infrastructures) to foster collaboration with the HPC community as well as with extra-European Grids e.g. in Africa, Asia and Latin America.

• Training, Dissemination, Business Outreach– CUE (Creating Users of E-infrastructures) enable users and

research to access and use e-infrastructures

• Dissemination– GridTalk-II – bringing the success stories of Europe’s e-infrastructure to

policy makers in government and business, to the broader scientific community and to the general public

• .... and there are more– Additional SSC proposals, OGF-Europe II, etc.

22-Oct-09 27J. Knobloch/CERN: European Grid Initiative – EGI

Relationships between projects

22-Oct-09 28J. Knobloch/CERN: European Grid Initiative – EGI

Project Relationships

EGI.eu

Operations Main SSC Middleware

CHAIN Other ‘SSC’OGF-Europe II Other EU Projects

Projects need to define their relationships by e.g. • Service Level Agreement• Service Level Description• Memorandum of Understanding

Conclusion

EGI_DS has defined the vision of a sustainable grid infrastructure in Europe,

it developed an implementation plan for EGI, and

it secured approval of the vision and plan by the European NGIs.

The plan is now being implemented

22-Oct-09 29J. Knobloch/CERN: European Grid Initiative – EGI