the knowledge-based workflow system for grid applications ladislav hluchý, viet tran, ondrej habala...
TRANSCRIPT
The Knowledge-based The Knowledge-based Workflow System for Grid Workflow System for Grid
ApplicationsApplications
Ladislav HluchLadislav Hluchýý, , Viet Tran, Ondrej HabalaViet Tran, Ondrej Habala
II SAS, SlovakiaII SAS, Slovakia
[email protected]@savba.sk
http://www.ui.sav.sk/http://www.ui.sav.sk/
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 2
KK-Wf Grid: -Wf Grid: PartnersPartners
Fraunhofer FIRST (Berlin, Germany) Fraunhofer FIRST (Berlin, Germany)
UIBK (Innsbruck, Austria) UIBK (Innsbruck, Austria)
IISAS (Bratislava, Slovak Republic) IISAS (Bratislava, Slovak Republic)
CYFRONET (Cracow, Poland) CYFRONET (Cracow, Poland) LogicDIS S.A. (Athens, Greece)LogicDIS S.A. (Athens, Greece)
Softeco Sismat SpA (Genoa, Italy)Softeco Sismat SpA (Genoa, Italy)
http://www.kwfgrid.euhttp://www.kwfgrid.eu
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 3
K-WfK-Wf GridGrid: Consortium: Consortium
www.kwfgrid.netwww.kwfgrid.net
Bratislava
Athens
Genova
Kraków
Innsbruck
Berlin
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 4
K-Wf Grid: ObjectivesK-Wf Grid: Objectives
Objectives of K-Wf gridObjectives of K-Wf grid To enable users to To enable users to create complex workflowscreate complex workflows and and
use grid resources without detailed knowledge of griduse grid resources without detailed knowledge of grid To construct workflows optimized for underlying To construct workflows optimized for underlying
infrastructure, infrastructure, using its advantages and avoiding using its advantages and avoiding its bottlenecksits bottlenecks
To (semi-)automatically To (semi-)automatically construct workflows based construct workflows based on user’s requirementson user’s requirements, using semantic annotation , using semantic annotation of services, data, applications and resourcesof services, data, applications and resources
To constantly renew information about the grid by To constantly renew information about the grid by using complex monitoring network – using complex monitoring network – to learn from to learn from experienceexperience
To provide To provide simple, easy-to-use interfacesimple, easy-to-use interface to K-Wf to K-Wf Grid servicesGrid services
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 5
Knowledge-based Workflow System for Knowledge-based Workflow System for Grid Applications – K-Wf GridGrid Applications – K-Wf Grid
Execute workflowExecute workflow
Capture knowledgeCapture knowledgeReuse knowledgeReuse knowledge
Monitor environmentMonitor environment
Analyze informationAnalyze information
Construct workflowConstruct workflow
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 6
K-Wf Grid: AdvantagesK-Wf Grid: Advantages
Composition of workflow from a set of servicesComposition of workflow from a set of services System composes the workflow for you – just tell him System composes the workflow for you – just tell him
what you want to get at the endwhat you want to get at the end System uses services which are System uses services which are available at the available at the
timetime and which are expected (based on past and which are expected (based on past experience) to provide good results (good = what you experience) to provide good results (good = what you want)want)
UsabilityUsability Less grid language, more application domain Less grid language, more application domain
languagelanguage Integrated collaboration interfacesIntegrated collaboration interfaces
Reuse of componentsReuse of components K-Wf Grid is based on respected standardsK-Wf Grid is based on respected standards
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 7
K-Wf Grid: Workflow RefinementK-Wf Grid: Workflow Refinement
Web Service ClassesWeb Service Classes
Web Service OperationsWeb Service Operations
Abstract OperationAbstract Operation
Grid ResourcesGrid Resources
User RequestUser Request““I want this data””
““There is my data!””
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 8
K-Wf Grid: Pilot applicationsK-Wf Grid: Pilot applications
PAs as domain-related instances of a typical K-Wf PAs as domain-related instances of a typical K-Wf Grid-based application systemGrid-based application system Common horizontal features / Specific vertical Common horizontal features / Specific vertical
customizationcustomization Flood Forecasting Simulation Cascade (FFSC, PA1)Flood Forecasting Simulation Cascade (FFSC, PA1)
Chain of simulations, targeting hydraulic simulation of Chain of simulations, targeting hydraulic simulation of a flooded areaa flooded area
Stateful WSRF services implementation in Java, Stateful WSRF services implementation in Java, using Globus Toolkit 4using Globus Toolkit 4
Enterprise Resource Planning (ERP, PA2)Enterprise Resource Planning (ERP, PA2) Stateless web services in Java (Tomcat/Axis)Stateless web services in Java (Tomcat/Axis)
Coordinated Traffic Management (CTM, PA3)Coordinated Traffic Management (CTM, PA3) Traffic simulation in Genoa, ItalyTraffic simulation in Genoa, Italy Developed as stateless web services in PerlDeveloped as stateless web services in Perl
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 9
K-Wf Grid: PA1 – FFSCK-Wf Grid: PA1 – FFSC
Flood Forecasting Simulation CascadeFlood Forecasting Simulation Cascade a cascade of simulation models for weather a cascade of simulation models for weather
forecasting, hydrological and hydraulic simulations, forecasting, hydrological and hydraulic simulations, able to produce predictions of possible floods in able to produce predictions of possible floods in target areastarget areas
InputInput– map of area, precipitation, river water level etc.map of area, precipitation, river water level etc.
OutputOutput– water flow in target areawater flow in target area
Suggestions / user supportSuggestions / user support– applicability of different models to specific scenarios (QoS)applicability of different models to specific scenarios (QoS)– comparison between stored predictions and real datacomparison between stored predictions and real data– future dispersion forecastingfuture dispersion forecasting
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 10
Flood Forecasting ApplicationFlood Forecasting Application(PA1-FFSC)(PA1-FFSC)
Flood forecasting based on a Flood forecasting based on a series of simulationsseries of simulations
Several simulation models forSeveral simulation models for MeteorologyMeteorology HydrologyHydrology HydraulicsHydraulics
2D/3D visualization2D/3D visualization Implementation using GT4 Implementation using GT4
WSRFWSRF Being extendedBeing extended
More modelsMore models More services – towards risk More services – towards risk
managementmanagement
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 11
Replica location
Metadata catalog
Ce
ntr
al s
erv
ice
s
Informationindex
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 12
Replica location
Metadata catalog
Ce
ntr
al s
erv
ice
s
Informationindex
Monitoringservice
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 13
Replica location
Metadata catalog
Ce
ntr
al s
erv
ice
s
Informationindex
Monitoringservice
Knowledgestore
Informationprocessing
Workflowmanagement
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 14
Replica location
Metadata catalog
Ce
ntr
al s
erv
ice
s
Informationindex
Monitoringservice
Knowledgestore
Informationprocessing
Workflowmanagement
Portal
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 15
Replica location
Metadata catalog
Ce
ntr
al s
erv
ice
s
Informationindex
Monitoringservice
Knowledgestore
Informationprocessing
Workflowmanagement
Portal
Execute workflow
Monitor environment
Analyze information
Capture knowledge
Reuse knowledge
Construct workflow
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 16
Replica location
Metadata catalog
Ce
ntr
al s
erv
ice
s
Informationindex
Monitoringservice
Knowledgestore
Informationprocessing
Workflowmanagement
Portal
Execute workflow
Monitor environment
Analyze information
Capture knowledge
Reuse knowledge
Construct workflow
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 17
Replica location
Metadata catalog
Ce
ntr
al s
erv
ice
s
Informationindex
Monitoringservice
Knowledgestore
Informationprocessing
Workflowmanagement
Portal
Execute workflow
Monitor environment
Analyze information
Capture knowledge
Reuse knowledge
Construct workflow
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 18
Replica location
Metadata catalog
Ce
ntr
al s
erv
ice
s
Informationindex
Monitoringservice
Knowledgestore
Informationprocessing
Workflowmanagement
Portal
Execute workflow
Monitor environment
Analyze information
Capture knowledge
Reuse knowledge
Construct workflow
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 19
Replica location
Metadata catalog
Ce
ntr
al s
erv
ice
s
Informationindex
Monitoringservice
Knowledgestore
Informationprocessing
Workflowmanagement
Portal
Execute workflow
Monitor environment
Analyze information
Capture knowledge
Reuse knowledge
Construct workflow
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 20
Replica location
Metadata catalog
Ce
ntr
al s
erv
ice
s
Informationindex
Monitoringservice
Knowledgestore
Informationprocessing
Workflowmanagement
Portal
Execute workflow
Monitor environment
Analyze information
Capture knowledge
Reuse knowledge
Construct workflow
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 21
Porting to gLitePorting to gLite
In K-Wf grid, applications are implemented as WSRF In K-Wf grid, applications are implemented as WSRF servicesservices
In gLite, the services are replaced by a set of worker In gLite, the services are replaced by a set of worker jobs that are submited by workflow manager via jobs that are submited by workflow manager via gLite WMSgLite WMS When started, the worker jobs will contact to When started, the worker jobs will contact to
workflow manager for executing tasksworkflow manager for executing tasks
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 22
Replica location
Metadata catalog
Ce
ntr
al s
erv
ice
s
Informationindex
Monitoringservice
Knowledgestore
Informationprocessing
Workflowmanagement
Portal
Execute workflow
Monitor environment
Capture knowledge
Reuse knowledge
Construct workflow
gLite WMS
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 23
Replica location
Metadata catalog
Ce
ntr
al s
erv
ice
s
Informationindex
Monitoringservice
Knowledgestore
Informationprocessing
Workflowmanagement
Portal
Execute workflow
Monitor environment
Capture knowledge
Reuse knowledge
Construct workflow
gLite WMS
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 24
Int.eu.gridInt.eu.grid
GoalsGoals Deploy and operate an interoperable production-level e-Infrastructure for Deploy and operate an interoperable production-level e-Infrastructure for
demanding interactive applicationsdemanding interactive applications Distributed (MPI-) parallel interactive computing and storage at Tera levelDistributed (MPI-) parallel interactive computing and storage at Tera level User friendly access through a interactive Grid desktopUser friendly access through a interactive Grid desktop Support for virtual organizations at all levels:Support for virtual organizations at all levels:
Setup, Collaborative environment, Grid enhancement of applications, Execution and Setup, Collaborative environment, Grid enhancement of applications, Execution and monitoring toolsmonitoring tools
Project detailsProject details Project start: May 1, 2006Project start: May 1, 2006 Project duration: 24 monthsProject duration: 24 months
Contribution of UI SAVContribution of UI SAV Operation of computer cluster with 32 CPUs and 32GB RAM as part of the project Operation of computer cluster with 32 CPUs and 32GB RAM as part of the project
production testbedproduction testbed User support, organization of tutorials and users’ trainingsUser support, organization of tutorials and users’ trainings Support for environmental applications and their usersSupport for environmental applications and their users Implementation of interactivity to flood simulation application Implementation of interactivity to flood simulation application Support for gridification and interactivity of radiocative pollution simulation aplication Support for gridification and interactivity of radiocative pollution simulation aplication
(owned by MicroStep-MIS)(owned by MicroStep-MIS)
http://www.interactive-grid.eu/
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 25
Int.eu.grid - PartnersInt.eu.grid - Partners
BIFI, Zaragoza/SpainBIFI, Zaragoza/Spain CESGA, Santiago de Compostela/SpainCESGA, Santiago de Compostela/Spain CSIC-IFCA, Santander/SpainCSIC-IFCA, Santander/Spain CYFRONET, Cracow/PolandCYFRONET, Cracow/Poland FZK, Karlsruhe/GermanyFZK, Karlsruhe/Germany GUP, Linz/AustriaGUP, Linz/Austria HLRS, Stuttgart/GermanyHLRS, Stuttgart/Germany ICM, Warsaw/PolandICM, Warsaw/Poland LIP, Lisbon/PortugalLIP, Lisbon/Portugal PSNC, Poznan/PolandPSNC, Poznan/Poland TCD, Dublin/IrelandTCD, Dublin/Ireland UAB, Barcelona/SpainUAB, Barcelona/Spain UI SAV, Bratislava/SlovakiaUI SAV, Bratislava/Slovakia
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 26
Int.eu.grid - InteractivityInt.eu.grid - Interactivity
Migrating Desktop
Remote Access Server
Job SubmissionServices
gLogin
CrossBroker
Computing Element
Gatekeeper
LRMS
Worker node 0
Worker node 1
Worker node N
…
JDL
Job control
Interactive channels
Interactive input
Interactive output
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 27
Control Output/visualizations/state
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 28
Control Output/visualizations
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 29
Flood Interactivity in Int.eu.gridFlood Interactivity in Int.eu.grid
Job is internally controlled by a workflow managerJob is internally controlled by a workflow manager Job steps can be added/removed during runtimeJob steps can be added/removed during runtime Can be paused/restarted/modifiedCan be paused/restarted/modified
Job outputs/visualized data are available through Job outputs/visualized data are available through the MD interfacethe MD interface
Job may be clonedJob may be cloned State = workflow state + intermediate data, may be State = workflow state + intermediate data, may be
easily transferredeasily transferred Good for parameter studiesGood for parameter studies
May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 30
ConclusionConclusion
Current research focuses on SOKU (Service Current research focuses on SOKU (Service Oriented Knowledge Utilities)Oriented Knowledge Utilities) K-Wf Grid is one of the early implementations of K-Wf Grid is one of the early implementations of
SOKU conceptsSOKU concepts How to adapt infrastructure research to this shift in How to adapt infrastructure research to this shift in
paradigm?paradigm? Application developers & end users need easy Application developers & end users need easy
access to grid infrastructureaccess to grid infrastructure SOKU is the way to achieve thisSOKU is the way to achieve this How to extend gLite towards SOKU?How to extend gLite towards SOKU?