running workflows on clouds and grids gabor terstyanszky, university of westminster t. fahringer, p....
TRANSCRIPT
Running Workflowson Clouds and Grids
Gabor Terstyanszky, University of Westminster T. Fahringer, P. Kacsuk, J. Montagnat, I. Taylor
e-Science Workshop, Oxford, 15 March 2012
SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585
2
Workflow Interoperability Challenge
33
Computer and Automation Research Institute, Hungarian Academy of Sciences
MTA SZTAKI Hungary
University of Innsbruck UIBK Austria
Charité - Universitätsmedizin Berlin C-UB Germany
French National Centre for Scientific Research CNRS France
University of Westminster UOW United Kingdom
Cardiff University CU United Kingdom
Academic Medical Centre of the University of Amsterdam AMC Netherlands
University of Southern California USC USA
SHIWA ProjectPartners:
Duration:July 2010 – June 2012 (extended to September 2012)
Project Objectives and Services
Objectives:• To create an environment which enables seamless execution of
workflows of different workflow systems through workflow interoperability
Services:• To support the whole workflow lifecycle: editing, uploading, browsing
downloading and executing workflows
• To provide coarse- and fine-grained workflow interoperability solutions
• To offer Distributed Computing Infrastructure interoperability solution
• To provide desktop computer and portal interfaces to manage workflows
Key actors:• researchers workflow engine developers
workflow developers
4
5
Workflow description abstract workflow•plain text which describes the inputs and outputs and explains what the workflow doesWorkflow implementations concrete workflow
• It contains all implementation files or references to these files (via eg. URLs) and also holds other data/metadata necessary to run the workflow on its associated workflow engine.
Workflow configurations• It contains pre-defined input files and parameters or references to these files and parameters and other
data/metadata of a workflow.Workflow engines• It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references
to them
SHIWA Data Structure
6
J2 WF4
J1
J3
GEMLCA client
GEMLCA Service
Workflow Engine B
Workflow of Workflow Engine B
Workflows of Workflow Engine A
Coarse-Grained Interoperability
DCI
• native jobs (or workflows): J1, J2, J3• non-native workflows: WF4
- black boxes which are managed as legacy code applications
Workflow Engine A
7
SHIWA Science Gateway Resourcesnative WE WS-PGRADE local resources: invocation of locally deployed WEs portal WS-PGRADE v3.3.2WE submission to local cluster repository GEMLCA + SHIWA repo remote resources: through remotely pre-deployedsubmitter GEMLCA with GIB WEs to gLite and Globus DCIsproxy management SHIWA Proxy Server
SHIWA Portal
Legacy Code GUI
SHIWA Science Gateway
WE1 WEp
GEMLCA Repository
WF1 WFm
SHIWA Repository
WF1 WFn
WS-PGRADE Workflow
engine
WS-PGRADE Workflow
editor
gLite DCI Globus DCI
pre-deployed-WEs
MOTEUR WE
Kepler WE
Taverna WE
Triana WE
local cluster
MOTEUR WE
GWES WE
ASKALON WE
SHIWA VO
ASKALON WE
GEMLCA Service
GEMLCA with GIB
SHIWA Proxy Server
Proxy Server
CGI Infrastructure
ProActive WE
8
SHIWA Portal: Editing Workflows
9
SHIWA Portal: Configuring Workflows
10
SHIWA Portal: Executing Workflows
11
SHIWA Workflow Repository
Browse Workflows
Browse Implementations
Search Sort
12
SHIWA Workflow Repository
13
SHIWA RepositorySHIWA Portal
WF1
SHIWA Science Gateway
GEMLCA Service
WFn
WE1 WEp
GEMLCA Repository
WE + WF
WF1 WFm
GEMLCA with GIB
WF list
WS-PGRADE Workflow
engine
WS-PGRADE Workflow
editor
edit WF
s2
search WF
s1
s5
s4
gLite DCI
MOTEUR WE
GWES WE
Globus DCI
pre-deployed-WEs
MOTEUR WE
Kepler WE
Taverna WE
Triana WE
local cluster
ASKALON WE
SHIWA VO
ASKALON WE
user
invoke WEs7
CGI User Scenario: Native WE
Legacy Code GUI
SHIWA Proxy Server
Proxy Server
s3
s7
submit WE
s6
ProActive WE
141414
SHIWA RepositorySHIWA Portal
WF1
SHIWA Science Gateway
GEMLCA Service
WFn
WE1 WEp
GEMLCA Repository
WF1 WFm
GEMLCA with GIB
WS-PGRADE Workflow
Engine
WS-PGRADE Workflow
Editor
search WF
s1
WE + WF s5
gLite DCI
MOTEUR WE
GWES WE
Globus DCI
MOTEUR WE
Kepler WE
Taverna WE
Triana WE
local cluster
ASKALON WE
SHIWA VO
ASKALON WE
user
invoke WE s7
CGI User Scenario: Non-native WE
Legacy Code GUI
GEMLCA Client
MOTEURWorkflow
Engine
MOTEUR Workflow
editor
s2
s3
GEMLCA UI
SHIWA Proxy Server
Proxy Server
s6
s4
ProActive WE
15
DCI Interoperability
JSDL Translator
Workflow Engine
DCI Bridge
Metabroker
Proxy Server
DCI
J2
J1
J4
J3
jobs in JSDL
J2
J1
J4
J3
jobs in non-JSDL
16
DCI BridgeFeatures:• web service based app which provides standard access to different
DCIs: clouds - EC2, OpenNebula clusters - LSF, PSB desktop grids - BOINC, OurGrid, XtremWeb service grids - ARC, gLite, Globus, Unicore
• workflow jobs are submitted to the BES Factory Service of the DCI Bridge where jobs are described in JSDL
Components:• resource registry - configuration interface of DCIs
• application manager - BES management
• runtime system - job execution management via BES Factory
• monitoring - logs and messages handling and visualisation
17
DCI Bridge
WS-PGRADE Workflow
Engine
WS-PGRADE Workflow
Editor
Portal
BES Factory Service
Job Registry
Input Queue
Upload Marnager
Plug-in Manager
PluginA
PluginB
PluginC
inputs & executables
outputs
clouds
desktop grids
service grids
DCI Bridge
proxy server
s1
s2 s3
s4s5
s6
s7
s8s10
s9
s11
181818
Fine-Grained Interoperability
• front-end plug-in: converts workflow of workflow engine A into IWIR (Intermediate Workflow Interpreter Representation)
• back-end plug-in: converts from IWIR into workflow of workflow engine B
IWIR workflow
J2
J1
J4
J3
WFA ->IWIR frontend plug-in
J2
J1
J4
Workflow of Workflow Engine A
J3
WFA ->IWIR backend plug-in
Workflow Engine B
DCI
J2
J1
J4
J3
Workflow of Workflow Engine B
ASKALON
MOTEUR
PGRADE
Triana
SHIWA RepositorySHIWA Portal
WF1
SHIWA Science Gateway
WFn
WE1 WFp
GEMLCA Repository
WF1 WFm
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
GEMLCA submitter
Kepler WEGWES WE
Taverna WE
Triana WE
MOTEUR WE
ASKALON WE
WS-PGRADE Workflow
engine
WS-PGRADE Workflow
editor PGRADE WE
pre-deployed-WEs
SSP DCI & FGI Infrastructure
SHIWA Science Gateway Resourcesnative WE ASKALON, MOTEUR, WS-PGRADE
local WEs: Kepler, MOTEUR , Taverna & Triana WEsportal WS-PGRADE v3.4.1 submit to the local cluster repository GEMLCA + SHIWA remote WEs: ASKALON, GWES & MOTEUR WEsGEMLCA
ARC, gLite, Globus and Unicore DCIsproxy server
DCI Bridge
Meta Broker
Proxy Server
JSDL Translator
ProActive WE
2020
SHIWA RepositorySHIWA Portal
WF1
SHIWA Science Gateway
WFn
WE1 WFp
GEMLCA Repository
WF1 WFm
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCIWS-PGRADE
Workflow engine
DCI Interoperability Scenario
DCI Bridge
Meta Broker
Proxy Server
JSDL Translator
search WF
s1
s6
s4s2
edit workflow
s7
submit workflow
s5
s3
s3
WS-PGRADE Workflow
editor
2121
SHIWA RepositorySHIWA Portal
WF1
SHIWA Science Gateway
WFn
WE1 WFp
GEMLCA Repository
WF1 WFm
DCIs
WS-PGRADE Workflow
Engine
WS-PGRADE Workflow
Editor
FGI Interoperability Scenario
ccDCI
Bridge
Meta Broker
Proxy Server
JSDL Translator
search WF
s1
Workflow Engine
Workflow Editor
s4FP1
FP2
FPn
cc
BP1
BP2
BPm
s3s2 edit WF
submit WFs7
gLite DCI Globus DCI
ARC DCI Unicore DCI
s5
s6
2222
SHIWA Simulation Platform:Access
SHIWA Science Gateway (SHIWA Portal + SHIWA Repository)
• joint portal and repository account given by the science gateway administrator
DCI resources
• certificate given by the relevant DCI administrator
Access to the SHIWA Portal
shiwa-portal.cpc.wmin.ac.uk/liferay-portal-6.05
Access to the SHIWA Repository
shiwa-repo.cpc.wmin.ac.uk
23
Conclusions
• the simulation platform supports the whole lifecycle of workflows, i.e. creating, testing, uploading, browsing, downloading and running workflows
• users can use workflows of their own and of other workflow systems through the same user interface, i.e. they can re-cycle or share workflows created by other research teams or even other research communities
• workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform
• researchers (astro physicists, earth scientists, bio scientists, etc.) can browse the workflow repository, select and run workflows through the simulation platform
• the simulation platform provides access to major European e-infrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources