a virtualised european grid infrastructure?

www.egi.euEGI-InSPIRE RI-261323 www.egi.euEGI-InSPIRE RI-261323

EGI-InSPIRE

www.egi.euEGI-InSPIRE RI-261323

EGI-InSPIRE

www.egi.euEGI-InSPIRE RI-261323

A Virtualised European Grid Infrastructure?

Steven Newhouse

Project Director, EGI.eu

22/04/23 UK eScience Forum 2010 1


What is EGI?

• European:– Focus on meeting the needs of European Researchers

– Their key need is to collaborate internationally

• Grid:– Federation of shared resources, not a single technology

– Supports a collaborative social network

• Infrastructure:– Established physical and organisational structures

– Services and facilities for an economy to function



European Grid Infrastructure

• European Data Grid (EDG)– Explore concepts in a testbed

• Enabling Grid for E-sciencE (EGEE)– Moving from prototype to production

• European Grid Infrastructure (EGI)– Routine usage of a sustainable e-

infrastructure



European Grid InfrastructureStatus April 2010 (yearly increase)

•10000 users: +5%

•243020 LCPUs (cores): +75%

•40PB disk: +60%

•61PB tape: +56%

•15 million jobs/month: +10%

•317 sites: +18%

•52 countries: +8%

•175 VOs: +8%

•29 active VOs: +32%

422/04/23 UK eScience Forum 2010


Are users willing to follow?


Sustainability

Relevance

Cost

Technology


Sustainability

My project will last for 5 years, and I will need to read and analyse my data for 10 years after that….

Will you be around that long?



EGICollaboration

NGI

NGI

NGI

NGI

ResearchCommunity

ResearchCommunity

ResearchCommunity

ResearchCommunity

EIRO

EIRO

EGI.eu

ResearchCommunity



EGI.eu Governance

• EGI.eu established as non-profit foundation

• Governance & ownership by its participants– Participants:

• European NGIs

– Associated participants:• Other groups willing to agree with EGI.eu objectives

• EGI Council: All participants represented– Votes linked to fees



EGI.eu

• Coordination for European Grid resources– Established February 8th 2010– Central policy & services needed to run a grid– Sustainable small coordinating organisation

• Based in Amsterdam– Coordinating core (~20 people) in Amsterdam– Technical services from partners (~20 people)


EGI and EGI.eu supported initially by EGI-InSPIRE


The EGI-InSPIRE Project

Integrated Sustainable Pan-European Infrastructure for Researchers in Europe

• A 4 year project with €25M EC contribution– Project cost €72M– Total Effort ~€330M– Effort: 9261PMs


Project Partners (50)

• EGI.eu, 40 NGIs, 2 EIROs

• Asia Pacific (8 partners)

Funded

Un-Funded


A Virtuous Service Cycle


Users represented by Virtual Research

Communities in the User Community Board

Resource providers represented by NGIs/EIROs in the Operations Management

Board

USAG: User Services Advisory Group

OTAG: Operational Tools Advisory Group

Technology Coordination Board prioritised for

Technology Providers

New technology releases assessed against defined

criteria before being deployed

Production quality infrastructure built from technology deployed

across Europe

11


EGI Supports Innovation

• Deploy Technology Innovation– Distributed Computing continues to evolve

• To include: Grids, Desktops, Virtualisation, Clouds, …

• Enable Software Innovation– Provide reliable persistent technology platform

• Tools built on gLite/UNICORE/ARC/Globus

• Support Research Innovation– Infrastructure for data driven research

• Support for international research (e.g. ESFRI)



Relevance

OK so you have processes….

How are you going to help me do what’s relevant to me and my community?


3) COMMUNITY

Human Network

Representation in the e-Infrastructure


Scaling the Human Network

• Across an NGI Research Community

• Across an International Research Community

• Across different disciplines & skill levels

• From an Individual to a Community

• Services that enable the human network

Services that support building your

Virtual Research Community



An Evolving Relationship


EGI

Virtual Research

Community

User support teams and helpdesks

User Community

Board

User Support Advisory

Group

National Grid

Initiatives

EGI.eu

Virtual Organisation


. . .

Virtual Research

Community



. . .

.

.

.

Defined by MoU

User Community

Support Team

(EGI.eu)

User Services Advisory

Group


Our Interaction with VRCs

• VRCs link EGI to a community structure– Defined by MoU, conference, organisation, …

• A focus for collecting requirements for:– Human and Technical services (e.g. Training)– Discipline specific support (e.g. Bio Apps)– Operation teams in the resource centres

It is the human collaborative side of the grid



A Virtuous User Cycle


Integrated Services•Human•Technical•Infrastructure

Feedback through VRCs in the User

Community Board

User Services Advisory Group to

drive detailed design


Current Technical Services

• Training Events Calendar

• Training Material Repository

• Applications Database

• Services for your community– Membership: VOMS, VO registration database– Access: Basic portals– Status: Dashboards



Current Human Services

• Building a community for all users– Two Annual meetings: Users & Technology

• For Heavy User Communities– Expand community specific services

• Evolving the technology– Definition and verification of requirements– Liaison with software providers

• Communication– With NGIs, VRCs, SSCs and other projects



Why become a VRC?

• User Community Board– Policy, procedures and technical management

• User Services Advisory Group– Technical evolution of the provided services

• Share your experiences with others

• Integrate your resources into production

• Access a European support infrastructure



Technology

I’ve told you what I want, how are you going to support my exact needs, now and in the future, instead of my competitors?



Be a Neutral Infrastructure

• Consider IP network providers– Supports traffic from different communities– Customised solutions within a generic framework– Standards drive integrated deployment

• And for sustainable e-Infrastructures?– Any application, any domain, any technology– A platform for domain specific innovation & use– Integration of any compliant resource



EGIEGEE

Networking

European e-Infrastructure


Applications

ResourcesDesktops HTC Grids

23

PRACE

PetascaleHPC

HPC GridsVirtualised Resources


The DCI Projects


Project EDGI EGI-InSPIRE EMI IGE StratusLab VENUS-C

Start Date 01/06/2010 01/05/2010 01/05/2010 01/10/2010 01/06/2010 01/06/2010

Duration (months) 24 48 36 30 24 24

Total Budget 2,436,000 € 72,000,000 € 23,000,000 € 3,693,000 € 3,137,221 € 8,803,046 €

Funding from the

EC2,150,000 € 25,000,000 € 12,000,000 € 2,350,000 € 2,300,000 € 4,500,000 €

Total effort in person-month

281 9241 1115 277 340 639


Venus-CEGI-InSPIRE

OperationalInfrastructure

EDGI StratusLab

DCI Taxonomy


EMIIGE

Resources

Technology

Users


Can we learn from others?

• Grids have benefited from commoditisation– Hardware: HTC & HPC affordable to all– Networking: GBs can be moved over WAN– Software: Open source software comes of age

• Impacts of commodity virtualisation…– For transactional models

• The ‘Cloud’: A model based on compute not data

– For large distributed data-oriented models • The emergence of true ‘function shipping’?



Dealing with Virtualisation

• Who makes your Virtual Machine Images?

• Who checks that they will run anywhere?

• Who checks that they are running?

• Who checks that they are secure?

New technology requires new expertise



A Virtualised Future?


VO Specific Operations Staff

InfrastructureProviders

Infrastructure Providers(Research & Commerical – National, European & Global)

End-Users(National, European & Global

Collaborations)

Experts(Communicating between

users & providers)

Virtual Research Community

End-User Technology Experts(National, European & Global Collaborations)


New Research Challenges

• Virtualised Resources across Europe– Federated Cloud: partnership agreements– Multi-clouds: co-ordinated access– Consolidation in data centres & commercial integration

• Provisioning of experimental infrastructure– Deploy on demand for 1 day 18 months– Record provenance so it can be replayed– Self-healing to lower management burden

• Security issues still remain– Site: Trusting the software in the VM– Network: Link VMs across WAN by dedicated networks– Data: Where is my data going & who has access?

22/04/23 SSAI Collaboration Meeting 2010 29


Cost

This sounds wonderful! How much is it going to cost me?



Current Costs

31UK eScience Forum 201022/04/23


Running EGI.eu (4 yrs)


European CommissionEGI-InSPIRE

(25000K) EGI.eu Activity(8010K)

Partner Tasks(International &

Global Tasks)(65848K)

NGIs(49455K)

EGI.eu

NGI Activity

EGI-InSPIRE Central Budget

(597K)

4653K

2643K

43455K

6000K19750K

597K

This moves EGI.eu towards

sustainability


Summary

• EGEE:– Demonstrated a production e-infrastructure

• EGI:– Provide a sustainable production e-infrastructure

• EGI.eu established in Amsterdam– Supported transition through EGI-InSPIRE– Moving towards sustainable coordination



Questions?


• Call for papers opens October 2010

• See www.egi.eu/EGIUF2011 for more details

• Contact:– Steven Newhouse

• [email protected]

a virtualised european grid infrastructure?

Documents