egi federated cloud update

www.egi.euEGI-InSPIRE RI-261323

EGI-InSPIRE


EGI Federated Cloud Update

Peter Solagna – EGI.eu

European Grid Infrastructure

1


EGI Fedcloud task force

• Activities started two years ago• Two main goals:

– Enable cloud services technical integration in the EGI Infrastructure

– Support user communities in porting their applications to a federated cloud environment


EGI’s Cloud Infrastructure

EGI Core Infrastructure

Federated AAI

ServiceDiscovery &

StatusMonitoring Accounting

EGI Cloud Infrastructure Platform

Providers Cloud Management Framework(OpenStack, OpenNebula, Stratuslab …)

VMMgmt

StorageMgmt

EGI C

olla

bora

tion

Tool

s

EGI A

pplic

ation

D

BIm

age

Repo

sito

ry

3

EGI C

loud

Ser

vice

Mar

ketp

lace

Providers Cloud Management Framework(new one we don’t now about and don’t care about)

VMMgmt

OCCI CDMI

URGLUE2

X509SAML


Core infrastructure integration - 1

• GOCDB– Cloud services registered in GOCDB

• Information system– Cloud sites are publishing partial information

in the Top-BDII– Extension for GLUE2 submitted to OGF

• Federated AA– Integration of X509 in Keystone

Michel Drescher

Not only Keystone, which is OpenStack's AA service.It needs to and is enabled in *all* CMFs, i.e. OponNebula (via rOCCI-server) and ~okeanos


Core infrastructure integration - 2

• Accounting– Usage of cloud resources is sent using an extended

Usage Record to the APEL repositories. – Now using the production accounting infrastructure– New view in the accounting portal (beta)

• Monitoring– The following probes are available:

• OCCI, test of the functionalities • CDMI, basic connection test• Accounting publishing• BDII publishing

http://accounting-devel.egi.eu/cloud.php

Michel Drescher

It is - AFAIK - using the approved OGF UR2 which is based on the FedCloud input

Michel Drescher

It tests the accounting data freshness, not that accounting data can be cut and sent to the repo!


Cloud specific services

• Cloud Marketplace in AppDB– Metadata repository for virtual appliances– Distribution mechanism of images to the sites

• Integrating VMCatcher & VMCaster technology• Sites subscribe to individual’s or VO image lists• Image list updates trigger sites to pull updates automatically• VO Managers to distribute VMI in the sites supporting their VO

– Support for VMCatcher/VMCaster– Marketplace for VMI discovery/search– Future work: full integration with the info system

• Brokering– SlipStream is being extended within an EGI mini-project to

support the OCCI interface

https://wiki.appdb.egi.eu/

Michel Drescher

"Cloud Marketplace in" is added.It is part of AppB, but there are no hard links or dependencies required between applications registered in AppDB and VM images registered in AppDB's Cloud marketplace


From testbed to production

• Now EGI Cloud resources are moving to full production– May 2014– Sites who successfully completed the

certification

• Cloud resources available for the users– May 2014: 5000 cores, 225 TB storage– End of 2014: 18.000 cores, 6000 TB storage


Certification process

The certification procedure for cloud sites is very similar to the certification of Grid sites:• Registration in GOCDB, with all the required information• Check for:

– Published in the information system– Publishing accounting data– Resources are monitored (using ops VO)

+ Security assessment with the EGI CSIRT– To be replaced with the security survey

• Site is passing successfully all the SAM tests for three days

Certified

Michel Drescher

Currently, security assessment is skipped according to PROC18, but is on the roadmap for PY5 - see workshop at CF14


Security survey

• EGI SVG team and CSIRT prepared a security survey for Cloud technology poviders and service provider– Assess the compliance with the EGI security policies

• Survey for the Cloud service providers:– Cloud Infrastructure (6 questions)– About Cloud services (3 questions) – About VMs (5 questions) – About EGI and Non-EGI co-tenancy of services (6

questions)

• Survey has been beta-tested by two Cloud sites, and will be part of the future certification processes


Certified sites

The following 14 sites/Clouds are already certified to be production cloud sites:• 100IT, Commercial provider (UK)• BIFI, Ibergrid• Cyfronet-CLOUD, NGI_PL• CESGA, Ibergrid• CESNET-MetaCloud, NGI_CZ• GoeGrid, NGI_DE• IFCA-LCG2, Ibergrid• HG-09-Okeanos-Cloud, NGI_GR• IISAS-FedCloud, NGI_SK• INFN-CATANIA-NEBULA, NGI_IT• INFN-CATANIA-STACK, NGI_IT• KTH-CLOUD, NGI_SE• PRISMA-INFN-BARI, NGI_IT• SZTAKI, INFN_HU

Sites under certification• TR-FC1-ULAKBIM, NGI_TR• UKIM, NGI_MK


What ‘production’ means for a site?

• A (cloud) site has successfully done passed the certification procedure

• The (cloud) site is endorsing the EGI policies and following the procedures

• The cloud services are monitored using SAM as the grid services– Alarms generate tickets to the site administrators– Site must guarantee the minimum

availability/reliability


PoC deployed in the testbed

Use cases ready to use cloud for production:• Peachnote, BioVel (OpenModeller,

BioSTIF, Openrefine), CHAIN-REDS science gateways (Octave, R, WRF)

PoC deployed or under deployment:• WeNMR, EISCAT-3D, DRHIM, ESA*

– (*) through the collaboration with HN


Thanks

• Questions?

egi federated cloud update

Documents

certification of grid

egi miniproject

accounting data freshness

certification procedure

gocdbcloud services

vm images

eueuropean grid

fedcloud inputit