page 3

news from the EGI communityISSUE 31JUNE 2018

TOP STORIES

page 5

MORE

www.egi.eu

IBERGRID looking towards the EOSCpage 1

Inspired

In the spotlight: EGI Operations Team

EGI FedCloud use case: PhenoMeNal

07 GDPR and research collaborations

page 4

EGI & DIRAC join forces for a new service

08 New EGI service: ISO 27001 Training

08 DI4R – Submit an abstract and register!

Virgo is using the EGI Workload Managerpage 6

This edition of the newsletter is focused on updates fromthe EGI community, new services and use cases.

Your feedback and suggestions are always welcome!

Send an email to Sara & Iulia at:

press@egi.eu

Welcome to issue 31!

IBERGRID looking towards the EOSC

Isabel Campos and Jorge Gomes highlight the IBERGRID developments

1EGI Inspired Newsletter // June 2018 // www.egi.eu

The signature on November 8thin 2003 of the Scientific & Tech-nology Cooperation Agreementbetween Spain and Portugal,marked the start of a strategicscientific cooperation in theIberian Peninsula, which includeddistributed computing as one ofthe key topics.

IBERGRID was born out of theIberian Common Plan for distri-buted infrastructures released in2007. Since then, IBERGRID hasbeen federating infrastructuresfrom Iberian research & academicorganisations mainly focused ongrid, cloud computing and dataprocessing. The IBERGRIDinfrastructure comprises 12computing and data centers inSpain and Portugal.

A number of replicated servicesguarantees data integrity andresilience in the operations ofthe critical services.

The infrastructure has provided984 million processing hourssince 2006 to support the HEPexperiments and several usercommunities. This includes 19million hours on biomedicalapplications and ~6 million hourson computational chemistry.

Strictly on cloud support, morethan 216,000 Virtual Machineshave been instantiated providingmore than 2 million cloudprocessing hours to LifewatchERIC in the last year.

On the R&D side, serviceintegration activities are takingplace in numerous areas. Anexample is OPENCoastS, aservice to provide on-demandcirculation forecast systems as aservice for the Atlantic coasts.The service is deployed at thecomputing site NCG-INGRID-PT,part of the EGI Federation, but itis being integrated into EOSC-hub as a Thematic Service incollaboration with LIP, LNEC,INCD, UNICAN, CNRS, and CSIC.

2EGI Inspired Newsletter // June 2018 // www.egi.eu

On the software developmentside, IBERGRID is contributing inmany areas. CSIC has developedOpenStack support for VOMSauthorization & authentication,cloud pre-emptible instances(OPIE) as well as CPU Cloudaccounting.

The Technical University ofValencia developed & maintainsthe Infrastructure Manager (IM),a key service to support theinstantiation of tailored clustersnow part of the EOSC-hubservice catalogue.

Support to user-level containerexecution has been developed &is maintained by the IBERGRIDsoftware teams at LIP. Udockeris an extremely successful soft-ware product - more than 310stars in GitHub: which is beingexplicitly recommended in manycomputing centers around theworld as the best solution forusers to execute containers,without requiring the interventionof the system administrator layer.

Software Quality Assurance hasgenerated an enormous amountof activity in the Iberian area.LIP, CSIC, CESGA and UPVLC arein charge of ensuring the qualityof the UMD software deployedby EGI. The Accounting Portal ofEGI is maintained & developedby CESGA for the EGI community.

The IBERGRID community gathersaround conferences in the Iberianarea, and a dedicated event, theIBERGRID conference series, thisyear takes place in Lisbon from11th to 12th of October.

IBERGRID looks into the futureEOSC with optimism. From theuser support side our mainassets are a very consolidateduser-base, and well-reputed userengineering and support teams.

From the technical point of view,we count on worldwide-recognisedteams, with expertise & technicalbackground to address the spe-cific requirements from scientificcommunities in the EOSC era.

IBERGRID in EGIIBERGRID is a key OperationsCentre of the EGI Federation.The resources made available byIBERGRID sites have been instru-mental in supporting the 4 largestscientific collaborations based atthe Large Hadron Collider (ALICE,ATLAS, CMS, LHCb).

Other examples of researcherssupported by IBERGRID are:

CLIC and ILC: the next adventurein High Energy Physics

The Compact Linear ColliderStudy (CLIC) & the InternationalLiner Collider (ILC) are two colla-borations set up to explore whathappens when electrons andpositrons (which are antielectrons)collide at high-energy.

CLIC and ILC use electrons &their anti-particles (instead ofprotons as the LHC does) to co-llect a new range of high-precisionmeasurements & get a differentview on high-energy physics.

(+ 760,000 CPU hours in IBERGRID)

Genetics

Biologists found that some oldindustrial landscapes can havesurprising benefits for biodiversity.

(+18 million CPU hours in IBERGRIDused by the biomed VO since 2006)

Computational Chemistry

A team is pioneering a way toproduce high-quality, carbon-neutral methane gas usingindustrial waste carbon dioxide.

(+5 million CPU hours in IBERGRIDused by the compchem VO since 2007)

The WeNMR suite of portals

WeNMR supports +9,000structural biologists worldwidewith a suite on online, user-friendly portals. The portals arepowered by High-ThroughputCompute resources provided viaan SLA by IBERGRID and othersites of the EGI Federation.

(+330,000 CPU hours in IBERGRIDsince 2013)

LifeWatch ERIC

LifeWatch is a distributed Researche-Infrastructure to advance bio-diversity research and to addressthe big environmental challengesand support strategic solutionsto environmental preservation.

(Supported with IBERGRID cloudresources of +2 million hours in thelast 12 months)

OPENCoastS

Operational Coastal CirculationForecast Services provides an on-demand circulation forecastsystem-as-a-service for theEuropean Atlantic coasts.

(Currently being integrated intoIBERGRID and EGI as an EOSC-hubthematic service).

More information

Isabel Campos is IBERGRIDCoordinator in Spain.

Jorge Gomes is IBERGRIDCoordinator in Portugal.

IBERGRID

wibergrid.lip.pt/site/

3

The EGI Operations Team leadsthe work of the distributed teamof National Operations Centresand regional staff in addition toteams dedicated to securityresponse and technical experts.

The goals are:

- Service delivery in a reliableand trusted way

- Gathering requirements fromend users and understandingwhen a new functionality is needed

- Defining and maintainingOperational Level Agreements(OLAs) from service providers

- Making sure service monitoringis working to meet obligationsdefined in the OLAs

- Incorporating OLAs withService Level Agreements (SLAs)for end user communities

- Tracking usage with the EGIAccounting service, which enablesthe future capacity planning &knowing who used what.

Security is a paramount concernin EGI. The Operations Team isinvolved in the ongoing evaluationand response of security threatsand the monitoring of securityvulnerabilities across the fede-ration, ensuring that patching isdone sufficiently quickly.

The team supports thedevelopment and running ofinfrastructure services dealingwith user Authentication andAuthorization (AAI).

In this way, access can begranted to resources in an easyway, across the federation andglobally, with other infrastructures.

In the spotlight: EGI Operations Team

Matthew Viljoen highlights the team’s activities and plans

This allows the exchange of bestpractices and paves the way fornew collaborations.

In addition to these efforts, theOperations Team runs & servesa distributed helpdesk which letsusers submit problem tickets orrequests for new functionalities.

Software distribution is anotherimportant aspect of the EGIOperations work. The teammaintains trusted channels forobtaining software and engageswith early adopters to ensurethat the software is ready forgeneral usage.

Last but not least, EGI Operationsensures that the knowledgerequired to use, operate andmaintain operations is accessibleand retained in the form ofadequate documentation.

Operations Plans for 2018Over the course of 2018, the EGIOperations Team is engaging ina number of different activitiesto continue to improve thequality of service delivery.

The central EGI Operations teamis currently leading the ServiceManagement development ofthe EOSC-hub project to ensurethat best practices formedduring the achievement of ISO9001 and ISO/EIC 20000-1certification in 2017 are followedduring the establishment of theEuropean Open Science Cloud.

The team is also working toimprove the security, generalquality and usability of the EGIFederated Cloud service.

EGI Inspired Newsletter // June 2018 // www.egi.eu

More information

Matthew Viljoen isOperations Manager of thecentral operations team atthe EGI Foundation.

Core EGI Operations Team:(in the photos)

Bruce BeckerBaptiste GrenierAlessandro PaoliniVincenzo SpinosoMatthew Viljoen

What is PhenoMeNal?The PhenoMeNal e-infrastructuresupports data processing andanalysis workflows for largedatasets generated in metabolomicsresearch. The computationalservices provided within Pheno-MeNal enable researchers tobuild and run reproducible dataprocessing workflows in cloudenvironments. They aim toimprove the understanding ofthe causes and mechanismsunderlying health and disease,and to render the generateddata & metadata more traceableas demanded by reproducibleresearch. PhenoMeNal’s servicescan enhance a wide range ofmetabolomics applications inbiological, agricultural andenvironmental sciences.

PhenoMeNal servicesThe PhenoMeNal VirtualResearch Environment (VRE) isimplemented as a microservicearchitecture with Docker containersorchestrated in a Kubernetesenvironment. The containers canbe connected into analysis pipe-lines using scientific workflowplatforms, such as Galaxy. Fordeployment, PhenoMeNal hasdeveloped the KubeNow clouddeployment software thatsimplifies instantiation of completevirtual infrastructures withKubernetes, including storage,network and other requiredservices readily available.

The PhenoMeNal Gatewayprovides a web-based GraphicalUser Interface (GUI) on top ofKubeNow. The use of Galaxy asthe main workflow systemprovides a user friendly GUI forworkflow authoring, and exe-cution of tools and workflows.

The PhenoMeNal App library isa service catalogue of all opensource metabolomics toolsavailable through the VRE. Thelibrary also provides an onlinetraining featuring tutorials andtechnical documentation on thedeveloped tools and workflows.

Deployment on the EGIFederated CloudPhenoMeNal has worked closelywith the EGI Federated Cloud toharness EGI’s strategy for pro-vision of a secure infrastructure.PhenoMeNal VREs are nowsuccessfully deployed on EGIFederated Cloud resources(Cloud provider IN2P3-IRES).

This achievement was a jointeffort between the two consortiaand covered the extension ofthe PhenoMeNal deployment

engine (KubeNow) to includeOpenStack Virtual OrganizationManagement System (VOMS)authentication methods.

The work was carried out on EGIcloud resources provided byIN2P3-IRES within the ELIXIR-FRvirtual organisation.

4EGI Inspired Newsletter // June 2018 // www.egi.eu

Deploying PhenoMeNal virtual research environmentson the EGI Federated Cloud

Ola Spjuth and Anders Larsson on the successes of the PhenoMeNal project

More information

Ola Spjuth and AndersLarsson are members of thePhenoMeNal consortiumbased at Uppsala University.

PhenoMeNal

phenomenal-h2020.eu/home

5

DIRAC was created to supportthe data management andprocessing of the LHCb experimentat CERN. Today it acts as acomplete solution to supportgrid, cloud and HPC resources,targeting various large scientificcommunities including LHCb,Belle II, EGI, CTA, GridPP, WeNMR,VIP, FranceGrilles, SKA. DIRACprovides complete solutions forproduction managements, forhandling distributed largevolumes of scientific data andoptimising job executions.

The software package includes:

- A data management systemthat comes up with accessprotocols to various storages,reliable data transfer (based onFTS), file fatalogues allowingmetadata-based data-discovery,and optimising routine datadistribution tasks.

- A workload management thatallows to aggregate resources ofdifferent source & nature in asingle system computing, andoptimise the job submission.

- A flexible transformationsystem that provides automateddata-driven submission ofprocessing jobs with workflowsof arbitrary complexity

- An accounting system tocollect & store data regarding tothe activities, and a monitoringsystem to monitor components.

- A webportal with an easyaccess user interface.

The modular organisation of theDIRAC components allows selectinga subset of the functionalitysuitable for particular applicationsor easily adding the missingfunctionality. These are veryuseful for communities to havecustomised environments forhandling their own data. DIRACevolves over time and in recentdevelopment, it includes newfeatures such as supports forCloud, HPC, muti-VO etc.

In order to better supportresearch communities, the EGIteams collaborated with DIRACto co-develop a new service: theEGI Workload Manager, alsoknown as DIRAC4EGI.

The service is led by the EGI Foun-dation & operated by IN2P3 onresources provided by CYFRONET.

The EGI Workload Manager isalready used in production byearly adopters like WeNMR, thatwere able to easily switch theirScience Gateways from gLiteWMS to DIRAC. The work iscurrently being supported by theWeNMR Thematic Service underthe EOSC-hub umbrella.

The accessibility of this servicewill also be extended soon:currently, authentication isbased on the old X509-basedmodel that is well known in thedistributed computing for re-search world, but in the contextof the EOSC-hub project, thisservice is being integrated withthe EGI Check-in service.

EGI Inspired Newsletter // June 2018 // www.egi.eu

This will allow users to connectusing their institute’s IdentityProviders without having to owna personal grid certificate.

EGI has a formal procedure toinclude production-level servicesinto the EGI service catalogue,including a complete businessplan for service provisioning,assurance of resource capacity,full documentation, serviceorder and help desk.

EGI is now in the final stage tosign an OLA with the technologyand resource providers of theDIRAC Workload Manager. Withan OLA in place, this service willbe officially included into the EGIservice catalogue and madeaccessible to the EGI community.

More information

Yin Chen is SeniorTechnical Outreach Expertat the EGI Foundation.

Baptiste Grenier is SeniorOperations Officer at theEGI Foundation.

DIRAC

http://diracgrid.org/

EGI and DIRAC join forces to develop a new service

Yin Chen and Baptiste Grenier on the achievements of the collaboration

The 8th DIRAC user workshop was held 22-25 May inLyon and hosted at CC-IN2P3 by CNRS.

6EGI Inspired Newsletter // June 2018 // www.egi.eu

Virgo is a giant laser interferometerdesigned to detect gravitationalwaves & located at the EuropeanGravitational Observatory (EGO)site in Italy. Virgo is a collaborationof the French National Center forScientific Research (CNRS) andthe National Institute for NuclearPhysics (INFN) and is operated byan international consortium.

The Virgo collaboration hasdemanding needs to store andanalyse a large amount of datarecorded by the interferometer.A typical analysis run can last forabout 6 months, and dependingon the amount of data, hundredsof thousands of jobs can besubmitted to the computingresources, with each job lastinganywhere between a few hoursto a full day.

That is why Virgo chose to usethe EGI Workload Manager – toeasily dispatch jobs to compu-ting resources and to managethe data necessary as input forthe jobs or produced as output.

The EGI Workload Manager (alsoknown as DIRAC4EGI) is an EGIservice based on the DIRAC tech-nology and suitable for usersthat need to exploit distributedresources in an optimised andtransparent way.

The type of resources that DIRACcan support include computingresources (grid, cloud, and batchsystems), storage and catalogresources. Access to DIRAC4EGI,a multi-VO DIRAC server, isprovided by EGI to communities

that lack resources for installingand managing an own dedicatedserver or are simply looking to trythe functionalities of the tool.

Virgo is now performing testsusing this instance. The fact thatDIRAC is already used by manycommunities as a mature toolwas a factor in the decision.

In addition to the EGI WorkloadManager, the Virgo collaborationalso decided to test distributeddata management solution tobetter understand its potential.

For that, it was agreed to set-upa dedicated DIRAC file catalogcomponent as well, hosted atthe INFN data centre in Italy.

The tests conducted so far showgood performance results. Forexample, the catalog was popu-lated with millions of records,and the performances were goodeven with a number of recordssimilar to the real numbers thatare expected to be in production.

The tests also allowed to findand fix some misconfigurationson the resource centres availablein France, Italy & the Netherlands.In the following months, evenmore sites will be involved andthere are plans to move andregister the production databetween the sites, using theDIRAC data transfer feature.

To help the VIRGO communityunderstand this technology, awinter school will be jointlyorganised by EGI, DIRAC andVIRGO in late 2018.

The school targets new VIRGOmembers, postdocs, and seniorresearchers and will providecourses on the following topics:

- grid and cloud concepts andthe EGI e-infrastructure

- high-level solutions developedby INFN for provisioning, creating,managing & accessing pool ofheterogeneous computingresources (DODAS – Dynamic OnDemand Analysis Service)

- use of the DIRAC system tohandle user payloads runningon any EGI grid- and cloud-based resources and on othercomputing resources.

Virgo is using the EGI Workload Manager

Alessandro Paolini reports on the new collaboration

More information

Alessandro Paolini isOperations Officer at theEGI Foundation.

Virgo

www.virgo-gw.eu/

EGI Workload Manager

wiki.egi.eu/wiki/Workload_Manager

The observation of gravitational waves by the Virgo & LIGO Scientific Collaborations paved theway to the Physics Nobel Prize awarded in 2017 to Rainer Weiss, Kip Thorne & Barry Barish.

7

We want researchers to be ableto use resources from multiplee- and research-infrastructures.If the infrastructures have policiesin common, their services cantrust each other so they canmore easily exchange data.

This makes it easier for theirusers to accept the policies nomore than once.

However, policies are presentedwhen a researcher joins a commu-nity and they must match therequirements of e-infrastructures(such as EGI) and comply withthe new GDPR. For example, ifyou are a Community Managerand you organise users intogroups to determine who getsaccess to what service – oursuite of guidelines can come invery handy. Many of them areco-developed by AARC and EGI.

For those of you worrying aboutGDPR compliance: remember itis all about striking the rightbalance between your legitimateneed to manage your communityand its resources, versus theimpact on researchers.

That is what our new guidancehelps you do: with federatedidentity management (FIM), thedata about your members isminimal by design.

We have brought together thebest of the guidance from eachof the privacy regulators acrossEurope, with a focus on ourresearch communities.

Making GDPR easier for research collaborations

David Groep introduces the guidelines published by the AARC project

So when you set up a researchcommunity, broaden your usecases, or extend your services,you should see what the impacton privacy will be.

Is it truly “risky”? If your researchitself is about people, you willlikely need to do a risk assess-ment. If you are just using yourusers’ personal data to accessservices in the infrastructure,then the AARC guidelines willmake it a lot easier.

We have 2 documents in the area:The Data Protection ImpactAssessment Guide for Communitiesgives you a set of handles todetermine whether or not you fitin the most common scenario.And if you need a specific imple-mentation guide: why not look atthe guidance we gave the LifeScience community, which isscoped to community needs.

If you are in doubt, contact theAARC team and we can workjointly to analyse your needs.

The EGI structure for organisingpolicies has been the basis ofthe AARC policy starter kit, a‘handbook’ and set of templatesyou can use to ensure that yourcommunity’s or infrastructure’spolicy meets global expectations.

We will expand it to ensure allpolicy aspects needed forcommunities & infrastructuresare part of that kit. But we needpeople to work with us so thatour policies reflect their needs.

EGI Inspired Newsletter // June 2018 // www.egi.eu

More information

David Groep is the TeamLeader of AARC’s PolicyTeam & member of the EGISecurity Coordination Group.

AARC Project Policies

aarc-project.eu/policies/

So please feel free to join the EGIsecurity policy group, the WISEcommunity, IGTF or REFEDS towork with us. There’s a limitednumber of policy experts in theworld and we really need yourengagement & support to makethis activity a success!

DI4R – Submit an abstract and register!

The DI4R 2018 (9-11 October) isjointly organised by EOSC-hub,GÉANT, OpenAIRE and PRACEand this year is hosted in Lisbonby LIP – the Portuguese ParticlePhysics Laboratory.

Keeping with the tradition ofinviting researchers to take thelead of the programme, DI4R2018 will be chaired by two topmembers of the High-EnergyPhysics community: Sinéad Ryan,from Trinity College Dublin, &Volker Gülzow, from DESY.

“Digital infrastructures underpinresearch – enabling simulation,visualisation, storage & analysis& helping collaboration andinnovation in an era of big dataand large-scale computation,”says Professor Ryan.

“No Computing – No Science!DI4R discusses new technologiesbut more important is gettingpeople together across disciplinesand talk to each other duringworkshops or over coffee,” saysProfessor Gülzow.

“So take the opportunity and joinus for a great event at DI4R 2018in Lisbon!”

Registration is now open, and aCall for Abstracts will be onlineuntil 29 June.

More information

ISO 27001 Training

egi.eu/services/iso-27001-training/

8

EGI Foundation welcomes a newaddition to its training services:Information Security Managementaccording to ISO/IEC 27001.ISO/IEC 27001 is part of theISO/IEC 27000 family of standardsdesigned to help organisationskeep information assets secure.It is complementary to FitSM,with similar principles towardsimplementing managementsystems and diving deeper intoareas around informationsecurity e.g. risk managementand implementing technical,physical and organisationalinformation security controls.

Value of ISO/IEC 27001:- Ensures that security risks areappropriately managed andprioritised

- Guards organisations againstinformation security risks

- Protects the data entrusted toorganisations

- Supports fulfillment of legalresponsibilities (e.g. GDPR)

New EGI Service: ISO 27001 Information Security Training

The courses are split into:- Foundation level (2 days) focusedon providing the fundamentalsof Information Security and howto implement a managementsystem (ISMS) in organisationsthrough a combination ofpeople, processes & IT systems.

- Professional level (3-4 days) formanagers & personnel workingto implement, maintain andoperate information securitywithin an organisation. Achievingthe ISO/IEC 27001 Professionallevel provides an additional role-based certification “InformationSecurity Officer”.

EGI delivers trainings in 2 ways:- Open Registration: for individualregistrations; the training will beorganised at a pre-determineddate and location;

- In-House: for organisationsneeding several members ofstaff to be trained; the date andlocation of the training will bemutually agreed.

EGI Inspired Newsletter // June 2018 // www.egi.eu

The first ISO 27001 Foundationtraining is set to take place inAmsterdam, 19 to 21 September.A formal certification will beoffered to all participants of thecourse that successfully pass thefinal exam.

The training topics include:- Introduction to ISO/IEC 27001

- Foundations of InformationSecurity Management Systems(ISMS)

- ISO/IEC 27001 – Minimumrequirements & security controls

- Selected practical recommen-dations & guidelines

- Related standards & frameworks

- Certification options accordingto ISO/IEC 27001