workflow sharing and integration services by the er-flow project
DESCRIPTION
Workflow sharing and integration services by the ER-flow project. k itti . v arga@sztak i.mta . hu on behalf of the ER -flow consortium. EGI Community Forum, Manchester, 8-12 April, 2013. Outline. Motivations SHIWA ER-flow Technology services SHIWA Simulation Platform - PowerPoint PPT PresentationTRANSCRIPT
Workflow sharing and integration services by the ER-flow project
[email protected] behalf of the ER-flow consortium
EGI Community Forum, Manchester, 8-12 April, 2013
Outline• Motivations
• SHIWA ER-flow
• Technology services• SHIWA Simulation Platform• Integrated workflow engines
• Example• Integrating Moteur & WS-PGRADE workflows
• User support services
2
MotivationIsolation by technology
– Many different workflow systems exists We expect no change– Technological choice isolates users and user communities– Technological choice selects DCI/middleware (WS, gLite,
Globus, …)
Benefits of the SHIWA Simulation Platform– Share your own workflow, re-use workflows of others– Create and execute ‘meta-workflows’: built from smaller
workflows that use different workflow languages/technologies– Combine workflows and DCIs
SHIWA ER-flow– SHIWA: FP7 R&D project. Created the SHIWA Simulation Platform (2010-2012)– ER-flow: FP7 support action project. Disseminates the SHIWA technology (2012-
2014).3
Generic use-cases
DCI #1DCI #2
CE SECE SE
DCI – distributed computing infrastructure; CE/SE – computing/storage element
Generic use-cases
DCI #1DCI #2
CE SECE SE
combine WFs
find and re-use
with own data
on own CE?
DCI – distributed computing infrastructure; CE/SE – computing/storage element
ER-flow project
6
University of Westminster UoW United KingdomMagyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezete
MTA-SZTAKI Hungary
Centre National de la Recherche Scientifique CNRS FranceStichting European Grid Initiative EGI.eu The NetherlandsAcademic Medical Center of the University of Amsterdam AMC The Netherlands
Technische Universität Dresden TUD GermanyLudwig-Maximilians-Universität München LMU GermanyUniversity College London UCL United KIngdomTrinity College Daudublin TCD IrelandIstituto Nazionale di Astrofisica INAF Italy
Technology providers + User support:MTA-SZTAKI, UoW, CNRS, EGI.eu (through the NGIs)Research Communities:Astro-physics INAFComputational Chemistry LMU + TUDHeliophysics TCD + UCLLife Science AMC+ WE ARE LOOKING FOR NEW COMMUNITIESDuration:September 2012 – August 2014
Aims, services, actors
7
Aims:• Provide a platform for research communities that enables
seamless execution of workflows of different systems• Investigate data interoperability issues in the workflow domain
and propose solutions
Services:• SHIWA Simulation Platform• Dissemination and outreach• Training for workflow developers, workflow users
Key actors:• Workflow engine developers• Workflow developers• Researchers
ER-flow technology:SHIWA Simulation Platform
8
Access to various workflow engines and DCIs where these WF applications can be run
CloudsLocal clusters
Supercomputers
Desktop grids(BOINC, Condor, etc.)
Cluster grids(EGI, OSG, …)
Supercomputer grids
(PRACE, XSEDE)
Grid systemsDistributed Computing Infrastructures (DCIs)
What does the developer of a (meta-)workflow need?
SHIWA Repository
Access to a large set of ready-to-run scientific workflow applications
SHIWA Portal
A portal/desktop to integrate, parameterize and run these applications
Web services
+ Workflow engines
SHIWA Repository
Facilitates publishing and sharing workflows Supports:• Abstract workflows with
multiple implementations of over 10 workflow systems
• Concrete workflows with execution specific data
Available:• From the SHIWA Portalhttp://ssp.shiwa-workflow.eu• Standalone interface:http://repo.shiwa-workflow.eu
Example: Create meta-workflow from Moteur and WS-PGRADE workflows
SHIWA Repository
SHIWA portal
DCI 1gLite
DCI 2Cloud
DCI nARC
12
Access to Moteur WF
Connected workflow engines
Moteur WF engine
An example: SHIWA solution for LINGA
Sub-WorkflowsManagement
13
Multi-Workflow
Computational Chemistry Applications in ER-flow
14
workflow system middleware VOGROMACS Molecular Dynamic application Energy Minimisation WS-PGRADE gLiTE/UniCore MoSGrid VO Equilibration WS-PGRADE gLiTE/UniCore MoSGrid VO Single TPR WS-PGRADE gLiTE/UniCore MoSGrid VOCADDSuite docking application Docking with ligand generation WS-PGRADE gLiTE/UniCore MoSGrid VO Docking without ligand generation WS-PGRADE gLiTE/UniCore MoSGrid VONWChem Quantum Chemistry/Molecular Dynamics application Geometry optimisation WS-PGRADE gLiTE/UniCore MoSGrid VO Spectroscopic analysis WS-PGRADE gLiTE/UniCore MoSGrid VO Parameter sweep WS-PGRADE gLiTE/UniCore MoSGrid VO Transition state analysis WS-PGRADE gLiTE/UniCore MoSGrid VO High throughput WS-PGRADE gLiTE/UniCore MoSGrid VO
Separate talks later today about Astro & Heliophysics workflows!
ER-flow user support• SHIWA User Forum
– www.erflow.eu/shiwa-user-forum • Workshops:
– Workflow Application Porting Workshop, London 19-22 March 2013 http://www.erflow.eu/19-22-march-2013-application-porting-workshop
15
Conclusions• Scientific workflow landscape is fragmented
– Many workflow system with specialised & small user base
– Many workflows with specialised & small user base• SHIWA Simulation Platform enables integration
– Within disciplines– Across disciplines
• ER-flow project provides user support– Consultancy & Workshops– User forum– Open for new communities
16
Questions?
17
www.erflow.eu/shiwa-user-forum
www.erflow.eu