egi federated cloud, developing a production quality federated cloud infrastructure

33
1 EGI federated Cloud, developing a Production Quality Federated Cloud Infrastructure Dr David Wallom Chair EGI Federated Cloud Task Force Associate Director, Oxford eResearch Centre, University of Oxford [email protected]

Upload: justin

Post on 23-Feb-2016

52 views

Category:

Documents


0 download

DESCRIPTION

EGI federated Cloud, developing a Production Quality Federated Cloud Infrastructure. Dr David Wallom Chair EGI Federated Cloud Task Force Associate Director, Oxford eResearch Centre, University of Oxford [email protected]. Outline. EGI, new c hallenges and c loud c omputing. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

1

EGI federated Cloud, developing a Production Quality Federated

Cloud InfrastructureDr David WallomChair EGI Federated Cloud Task ForceAssociate Director, Oxford eResearch Centre, University of [email protected]

Page 2: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

2

Outline

• EGI, new challenges and cloud computing.

• Federation model objectives, deliverables, mandate and membership.

• Standards and core infrastructure services.

• Federation test bed.

• Use cases.

• Conclusions.

Page 3: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

www.egi.euEGI-InSPIRE RI-261323

Resource Centres

EGI-InSPIRE & EGI Council members 319

Including integrated RPs 351

Countries EGI-InSPIRE & EGI Council members 42

Including integrated RPs 54Integrated EGI-InSPIRE Partners and EGI Council Members

Internal/External Resource Providers (being integrated)

External Resource Providers (integrated)

Peer Resource Providers

Registered Users: 21714 VOs: 233LCPUs: 470,000 Disk: 143PB Tape: 138PBJobs: 1.62 million/day

Operational Infrastructure

3Thanks to Steven Newhouse

Page 4: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

4

EGI New Challenges and Cloud Computing• New communities and application design models

• Need for long running services (not only jobs).

• Workflows that integrate local and remote systems.

• Integrating community-specific resources (sensors, antennas, repositories, ...).

• Since ~2010 the trend has been for resource providers to move Grid middleware in virtualised environments.

hardware

grid middleware

portals

hypervisor

dev env

OS OS OS

hardware

grid middleware

dev env custom services

OS OS OS OS OS

portals

Page 5: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

5

EG

I.eu

Coo

rdin

atio

nC

ore

softw

are

and

supp

ort

gLite UNICOREdCache ARCCommunity

Platform

IaaS NGI

VM Mgmt Data Image Sharing

Monitoring Accounting Notification

NGI

Monitoring Accounting Notification

EGI-wide message bus

NGI

VM Mgmt Data Image Sharing

Monitoring Accounting Notification

Commercial

VM Mgmt DataImage

Sharing

Monitoring Accounting Notification

Personalised environments for individual research communities in the European Research Area.

Community Services

Community Services

Globus

Globus

EGI New Challenges and Cloud Computing

Page 6: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

6

EGI Federation Model

HardwareHardware

HardwareHardware

Hardware

Cloud ManagementCloud Management

Cloud ManagementCloud Management

Cloud Management

User CommunitiesUser Communities

User Communities

Federated interfaces

Federated services

• Standards and validation: emerging standards for the interfaces and images – OCCI, CDMI, OVF.

• Resource integration: Cloud Computing to be integrated into the existing production infrastructure.

• Heterogeneous implementation: no mandate on the cloud technology.

• Provider agnosticism: the only condition to federate resources is to expose the chosen interfaces and services.

Page 7: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

7

EGI Task Force Federated Clouds

• Stakeholders1. Resource Centres

offer Cloud services2. Technology Providers

tender solutions3. User communities

Consume cloud services

• 6 scenarios = 6 key federated cloud capabilities

1. Manage VM instances2. Data access/transfer interface3. Cloud service information federation4. Resource consumption management5. Cloud service availability6. Notification & Automation

Resource Centre

Cloud Management Technology

Provider

User Communitie

s

offerconsume

tender

appoint

7. Federated AAI8. VM Image Management9. Brokering10.Contextualisation

10 scenarios = 10 key federated cloud capabilities

Page 8: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

8

Objectives and Deliverables

• Engagement: identify and work with resources providers, technology providers, and user communities.

• Integration: integration of cloud resources within EGI’s production infrastructure – e.g. monitoring, accounting and information publishing.

• Recommendations: identify issues that need to be addressed by other areas of EGI – e.g. policies, operations, support and dissemination.

• Blueprint document: advice/full documentation to resource providers/users on how to engage with the federated virtualised environment. A living document on the EGI Wiki.

• Test bed: implement interfaces and services for a federated IaaS cloud on the basis of the Task Force blueprint and the available standards and technologies.

Page 9: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

9

Task Force – Apr 2013

Members• 70 individuals (+17%).• 40 institutions (+34%).• 13 countries (=).

Stakeholders• 22 Resource Providers (+46%).• 10 Technology Providers (+42%).• 7 User Communities (+40%).• 4 Liaisons (+33%).

Technologies• OpenNebula.• StratusLab.• OpenStack.• Synnefo.• WNoDeS.

BSC

CNRS

LMU

OeRC

Masaryk

TUD

IFAE

Cyfronet

100%IT

CESNET

RADICAL

SRCE

DANTE

FZJ

GRNET

GWDG

STFC

SARA

KTH

INFNFCTSG

EGI.eu

Imperial

CESGACETA

IFCA

IGI

IPHC

IN2P3

SZTAKI

IISAS SixSq

Page 10: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

www.egi.euEGI-InSPIRE RI-261323

EGI’s Cloud Infrastructure

EGI Core Infrastructure Platform

Federated AAI

Central ServiceRegistry

Monitoring Accounting

EGI Cloud Infrastructure Platform

Cloud Management Stacks(OpenStack, OpenNebula, …)

VMMgmt

StorageMgmt Information

EGI C

olla

bora

tion

Platf

orm

Imag

e M

etad

ata

Mar

ketp

lace

Imag

e Re

posit

ory

19Thanks to Steven Newhouse

Page 11: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

20

Federation Test bed: Set Up – March 2012

Resource ProvidersOpenNebula

VM MngmtOCCI 1.1

InformationGLUE 2.0+LDAP/BDII

MonitoringNagios

Message Bus

AccountingOGF UR+

AAILogin/Passwdcurl

DataCDMI 1.0

Page 12: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

21

Federation Test bed: Consolidation – September 2012

Resource ProvidersOpenNebula, OpenStack

VM MngmtOCCI 1.1

InformationGLUE 2.0+LDAP/BDII

VM sharingMarketplace

Message Bus

AAIX509

DataCDMI 1.0

MonitoringNagios

AccountingOGF UR+

curl

Page 13: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

22

Federation Test bed: Integration – April 2013

Resource ProvidersOpenNebula, OpenStack, WNoDeS

VM MngmtOCCI 1.1

InformationGLUE 2.0+LDAP/BDII

VM sharingMarketplace

Message Bus

AAIX509

DataCDMI 1.0

MonitoringNagios

AccountingOGF UR+

curl

Page 14: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

23

Cloud Services

Marketplace. A repository were Resource Providers and EGI can publish metadata about images from which virtual machines can be instantiated. When needed, a single image can be signed and then endorsed by multiple providers.

http://marketplace.egi.eu/metadata/

Page 15: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

24

Federation Test bed – April 2013FedCloud Core

Services

Information SystemTop-DBII

MonitoringNagios

Image Metadata

Marketplace

Appliance Repository

VOVOMS server

User InterfacesOCCI Clients

rOCCI; WNoDeS-CLI

CDMI ClientsLibcdmi-java

Resource ProvidersVOMS proxy

Vmcatcher

VmcasterMarketplac

e client

SSM

CDMI server LDAP

OpenNebula

OpenStackStratusLabWNoDeS

OCCI server

AAI

VOMS ProxyEGI Production Infrastructure

Service Availability

SAM

AccountingAPEL

Configuration DB

GOCDB

External ServicesCertificatio

n Authorities

Page 16: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

25

Cloud Federation Test bed – Future

FedCloud Services

Image Metadata

Marketplace

Appliance Repository

User InterfacesOCCI Clients

rOCCI; WNoDeS-CLI

CDMI ClientsLibcdmi-java

Resource ProvidersVOMS proxy

Vmcatcher

VmcasterMarketplac

e client

SSM

CDMI server LDAP

OpenNebula

OpenStackStratusLabWNoDeS

OCCI server

AAI

VOMS Proxy

EGI Production Infrastructure

Service Availability

SAM

AccountingAPEL

Configuration DB

GOCDB

External ServicesCertificatio

n Authorities

Information SystemTop-DBII

MonitoringNagios

VOVOMS server

Page 17: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

26

Services for the End User

Page 18: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

27

Services for the End User

Page 19: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

28

VOMS server – PERUNSystem for managing VOs, users, resources and services. Currently, 32 VOs, 1900+ users, 1800+ machines, 20+ services. http://perun.metacentrum.cz

OCCI client/server implementations• rOCCI: Developed in Ruby by CESNET and GWDG, offers a server

implementation of OCCI 1.1 for OpenNebula and StratusLab and a client compatible also with the OpenStack OCCI server.

• WNoDeS-CLI: developed in Python at INFN/CNAF, it leveraging the EGI core services and offers interoperability across WNoDeS, OpenNebula and OpenStack. Designed as a back-end for a web-based portal.

Image management backend – Vmcaster/catcherSystem to distribute, download, endorse, update OS images across multiple resource providers. Used as backend image management system.

Integration of new technologies – April 2013

Page 20: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

29

NB: Users = researchers

IaaS for power users• Comparable experience to ec2-api-tools, euca2ools.• Command line interface for easy scripting and integration with workflow.• Multiple language bindings to create a developer-oriented ecosystem.

IaaS as a backend • Promote uniform OCCI server implementations.• Facilitate the development of connectors and shims.• Platforms.

IaaS for end users• Portals. Upcoming IGI portal + SlipStream EGI mini project.• Community services.

FedCloud Test Bed Use Models

Page 21: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

30

BNCweb (CLARIN)

• Interface to the British National Corpus (100ml words).

• Used in a wide variety of computational linguistic applications.

• Searching the text and exploiting the detailed textual metadata.

• Open source project, freely available for educational and research purposes.

Use cases

1. Teaching a masters course in 'Exploring English Usage' in October-November 2012, and 'Corpus Linguistics' in February-March 2013.

2. Federated search in the CLARIN European e-Infrastructure.

3. Build additional web services on top of BNCWeb, e.g. adding improved visualizations of the search results.

Demo Use Cases – September 2013

Page 22: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

31

EU Federated IaaS Cloud for CLARIN and BNCWeb

Linguistic Datasets and

corpa

Federated Cloud

Community Tools as Virtual

Machines in Marketplace

Resource Discovery Service

1st line infrastructure

2nd line Community

support

SSO

Page 23: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

32

WeNMR – Structural Biology

• Validating biomolecular NMR structures. VM images pre-packaged with VirtualCing and ToPoS.

• Training. VM images pre-packaged with Gromacs.

BioVel/OpenBio – Ecology

• remote hosting of OpenModeller service.

• OpenModeller + Venus-C COMPS + FedCloud test bed.

WS-PGRADE/gUSE – Software Engineering

• User support environment for grid and clouds + GUI.

• Flexible deployment of virtualised DCI to run predefined, repeatable tests.

Use Cases – April 2013

Page 24: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

33

Power User

OCCI 1.1

CDMI 1.0

InformationLDAP

MonitoringNagios

AccountingAPEL

VM metadataMarketplace

OCCI 1.1

CDMI 1.0

OCCI 1.1

OCCI 1.1

CDMI 1.0

OCCI 1.1

OCCI 1.1

OCCI 1.1

VOMSPERUN

Certification Authority

Appliance Repository

Resource ProviderINFN/CNAF

OCCI 1.1

User(s)

1. VOMS

2. SSH key

3. [OS image]

4. OCCI client

Resource ProviderGWDG

Resource ProviderCESNET

Resource ProviderIISAS

Resource ProviderKTH

Resource ProviderCESGA

Resource ProviderBSC

Resource ProviderFZJ

Resource ProviderLAL

Page 25: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

34

Federation View – Image Management

Appliance Repository

VM metadataMarketplace

Resource ProviderBSC

Resource ProviderGWDG (ON/OS)

Resource ProviderCESNET (ON)

Resource ProviderCYFRONET (ON)

Resource ProviderKTH (ON)

MP client

Vmcstr/ctchr

Resource ProviderCESGA (ON)

Resource ProviderFZJ (OS)

ON = OpenNebula.

OS = OpenStack.

WN = WNoDeS.

MP = Marketplace.Resource Provider

LAL (SL)

Resource ProviderINFN/CNAF (WN)

MP client

Vmcstr/ctchr

MP client

Vmcstr/ctchr

MP client

Vmcstr/ctchr

MP client

Vmcstr/ctchr

MP client

Vmcstr/ctchr

MP client

Vmcstr/ctchr

MP client

Vmcstr/ctchr

Page 26: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

35

Federation View – Monitoring

Configuration DB GOGDB

MonitoringNagios

Resource ProviderBSC

Resource ProviderGWDG (ON/OS)

Resource ProviderCESNET (ON)

Resource ProviderCYFRONET (ON)

Resource ProviderKTH (ON)

OCCI

Resource ProviderCESGA (ON)

Resource ProviderFZJ (OS)

ON = OpenNebula.

OS = OpenStack.

WN = WNoDeS.

Resource ProviderLAL (SL)

Resource ProviderINFN/CNAF (WN)

OCCI

CDMI

OCCI

CDMI

OCCI

OCCI

CDMI

OCCI

OCCI

OCCI

SSM

BDII

SSM

BDII

SSM

BDII

SSM

BDII

SSM

BDII

SSM

BDII

SSM

BDII

SSM

BDII

Page 27: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

36

Federation View – Accounting

AccountingAPEL

Resource ProviderVenus-C

Resource ProviderGWDG (ON/OS)

Resource ProviderCESNET (ON)

Resource ProviderCYFRONET (ON)

Resource ProviderKTH (ON)

SSM

Resource ProviderCESGA (ON)

Resource ProviderFZJ (OS)

ON = OpenNebula.

OS = OpenStack.

WN = WNoDeS.

SSM = Secure Stomp Messenger.Resource Provider

LAL (SL)

Resource ProviderINFN/CNAF (WN)

SSM

SSM

SSM

SSM

SSM

SSM

SSM

SSM

Configuration DB GOGDB

Page 28: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

37

Federation View – Information Publishing

Resource ProviderVenus-C

Resource ProviderGWDG (ON/OS)

Resource ProviderCESNET (ON)

Resource ProviderCYFRONET (ON)

Resource ProviderKTH (ON)

LDAP

Resource ProviderCESGA (ON)

Resource ProviderFZJ (OS)

ON = OpenNebula.

OS = OpenStack.

WN = WNoDeS.Resource Provider

LAL (SL)

Resource ProviderINFN/CNAF (WN)

LDAP

LDAP

LDAP

LDAP

LDAP

LDAP

LDAP

LDAP

Configuration DB GOGDB

Information SystemTOP-BDII

Page 29: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

38

Conclusions

• Towards production-grade cloud resources within the EGI infrastructure:– EGI mini-projects and further development of deployed technologies;– transition into EGI operations.

• Biggest challenges:– contextualisation;– consistent OCCI client/server implementations –> cloud profile– data.

• Focusing on use cases and early adopters:– dedicated user support;– policies for tailored image management and support;– SLA.

• Next Phase – September 2013 – Pre-production

Page 30: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

www.egi.euEGI-InSPIRE RI-261323

EGI’s Grand Vision

Establish by 2020 a distributed open compute and data infrastructure comprising

a 10M Core Federated Cloud and 10 Exabyte of Federated Cloud Storage

across Europe that is able to support the data analysis activities of all researchers

within the European Research Area

EGI Council Discussion – June 2013

39

Page 31: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

40

With thanks to…

Mattieu Puel

Boris Parak

Jan Meizner

Björn Hagemeier

Florian Feldhaus

Piotr Kasprzak

Mohammed Airaj

Zeeshan Ali Shah

Roberto Rosende Dopazo

Emir Imamagic

Alvaro Simon

Michel Drescher

Peter Solagna

Alison Packer

John Gordon Kostas Koumantaros

Daniele Lezzi

Elisabetta Ronchieri

Ilja Livenson

Viet TranSandor Acs

Michel Jouvin

Ivan Diaz AlvarezOwen Synge

Andy Edmonds

Matteo Turilli

Page 32: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

www.egi.euEGI-InSPIRE RI-26132341

Page 33: EGI federated Cloud, developing a  Production Quality Federated Cloud  Infrastructure

42

Thank you

Task Force resources• Mailing List: [email protected]• Wiki site: http://go.egi.eu/tf-fedclouds• GitHub: https://github.com/EGI-FCTF• Indico site: https://www.egi.eu/indico/categoryDisplay.py?categId=56• Contact the chair: [email protected]