workflow optimisation services for e-science applications

Workflow Optimisation Services for e-Science Applications

David W. Walker

Cardiff University

WOSE Overview

• Draws together JISGA and Triana work at CU, with ICENI at IC, and portal expertise at DL.

• Topics addressed– Service aggregation and deployment– Runtime discovery and late binding of services– Service discovery and selection from multiple

semantically equivalent services

Workflow Optimisation

• Types of workflow optimisation– Through service selection– Through workflow re-ordering– Through exploitation of parallelism

• When is optimisation performed?– At design time (early binding)– Upon submission (intermediate binding)– At runtime (late binding)

Service Binding Models

• Late binding of abstract service to concrete service instance means:– We use up-to-date information to decide

which service to use when there are. multiple semantically equivalent services

– We are less likely to try to use a service that is unavailable.

Late Binding Case

• Search registry for all services that are consistent with abstract service description.

• Select optimal service based on current information, e.g, host load, etc.

• Execute this service. • Doesn’t take into account time to transfer

inputs to the service.• In early and late binding cases we can

optimise overall workflow.

WOSE Architecture

Work at Cardiff has focused on implementing a late binding model for dynamic service discovery, based on a generic service proxy, and service discovery and optimisation services.

History database

Proxy

Configuration script

Workflow script

User

Converter ActiveBPEL workflow engine

Web service instance

Discovery Service

Optimization Service

Registry services (such as UDDI)

Performance Monitor Service

Service Discovery Issues

• Service discovery and optimisation is based on service metadata.

• Could store in a database.

• Could obtain by interrogating service.

Optimisation by Re-Ordering

• Work at Imperial has looked at static optimisation– Optimise the runtime execution of workflow before it is

executed

– Achieves the goal through:• Re-ordering of components

• Addition of components

• Substitution of components

• Pruning of the workflow

• Performance and workflow aware Scheduling• Runtime Optimisation

– through monitoring, check-pointing and migration

Component Manipulation

• Re-ordering: Workflows (often composed from composite workflows) may contain non-optimal ordering of components– Use re-ordering to improve performance

multiplyMatrixGen

VectorGen

MatrixGen

multiply

multiply

MatrixGen

VectorGen

MatrixGen

multiply

Component Addition

• Addition: For a component requiring a specific format of data as input, a transformer component could be added to achieve the desired format.– Allows more optimal components to be

used togetherInput required in MPS formatOutput in LP

formatC 1 LP to MPS C 2

Component Substitution

• Substitution:– A Jacobi Iteration linear solver replaced by

Conjugate Gradient linear solver according to the output of the Discretizer (FEM)

• Based on observing the meta-data associated with previous components

b

A (sparse and diagonally dominant)

FEM JI linear solver

Pruning

• Workflow Pruning:– Workflows may contain unused

components. Especially when composed from other sub-workflows

• Remove redundant components

Not needed

a b

fc

eg

d

hNot needed

Work Done at Daresbury

• Daresbury started working with evaluating different BPEL engines to have proper understanding of BPEL specification i.e. ActiveBPEL, Oracle BPEL engine initially Collaxa BPEL engine.

• Most projects at CCLRC are portal based, so we studied the relevance of Portal based workflow monitoring and controlling solution.

• Web Services for Remote Portlets (WSRP) is an emerging specification, with limited support - seems to bridge Portal and Web Services and a lot of work has been done, with demo prototype.

• WSRF is extension of existing Web Services and provides extra layer of functionality without changing the Web Service implementation.

• WSRF extension for Web Service starts from modification of existing WSDL file of specific Web Service.

• WSRF extension of WSDL is not straight forward and needs a thorough understanding of WSDL and WSRF.

• Daresbury has built working prototype for testing Web Services based on WSDL.

Future Work

• Complete WOSE prototype with dynamic scheduling.

• Compare dynamic and static approaches at Cardiff and Imperial.

• Improve Discovery Service mechanisms.

• Investigate different approaches to optimisation.