workflow adaptation as an autonomic computing problem

Combining the strengths of UMIST andThe Victoria University of Manchester

Workflow Adaptation as an Autonomic Computing Problem

Kevin Lee, Rizos Sakellariou, Norman W. Paton and Alvaro A. A. Fernandes

School of Computer Science, University of Manchester

{klee, rizos, norm, alvaro}@cs.man.ac.uk

Kevin Lee 25th June 2007


Talk Overview

1)Reasons for Adaptation in Workflows

2)Autonomic Computing

3)Our View of Workflows

4)Potential Workflow Adaptations

5)Categorising Workflow Adaptations

6)Future work


1. Reasons for Adaptation in Workflows

•Very long running•Small delays can have large effects due to dependencies•often involve highly distributed resources•Limited control over resources•Uncertain execution times•Uncertain queue waiting times

Execution Characteristics of Scientific Workflows


1. Reasons for Adaptation in Workflows•scheduling of a workflow is decided before it starts executing

•Using current information about the execution environment

•What happens if the environment changes?•Resources disappear•Loads change•New resources appear

•What if a execution resource becomes unavailable•Only real option is to Re-Submit to different resources•Reduced performance

•New resources are not taken advantage of

•The Execution characteristics of workflows combined with these lead to lower than potential execution performance

Adaptation would be desirable.But what would workflow adaptation look like?


Many systems nowadays face these issuesBuilding adaptive systems is hard always done in ad-hoc waysleads to brittle and non-reusable adaptationTo this end adaptive systems are often seen in the Autonomic

Systems Community as functionally decomposable into the components:

2. Autonomic Computing

Monitor: Events from a source:

log filesin-memory processsensors

Analyze: When an event occurs, what to do about it...

Plan: After the event is detected and analysed, the system needs to determine what to do about it.

Execute: Perform the necessary changes

But, how do we think about workflows in these terms?


3. Our View of WorkflowsTo look at workflows in a generic way, we’ve adopted a view of the use of workflows as follows:

An abstract workflow: describes the workflow at the level of tasks that perform transformations on data.

A concrete workflow: describes the workflow at the level of actual services•file-based inputs and outputs.

Mapping an abstract workflow•choosing appropriate services for tasks•finding data sources and output files

Scheduling a concrete workflow•assigning each of the services to execution nodes.

We have a more formal notation in the paper


4. Potential Workflow Adaptations

In general, adaptations can usefully be thought of as a revision of decisions made previously

Thus, based on the previous slide, workflow adaptations can be classified as either mapping or scheduling adaptations.

Adaptations can be performed for different reasons:•prospective (to improve future performance)•reactive (to react to previous results)•altruistic (to aid other areas of the system).

Adaptations can also affect the workflow at differentlevels of granularity:•single node•some nodes•all of the workflow.


4. Potential Workflow Adaptations: Mapping Adaptations

Mapping adaptations are adaptations where the mapping from the abstract workflow to the concrete workflow changes depending on the environment.

Examples:

•Change abstract node to concrete node mapping:•Reduce the number of concrete nodes for an abstract task•Increase the number of concrete nodes for an abstract task (task-splitting).•Remove an abstract node•Change data source/sink for a service

See paper for further detail


4. Potential Workflow Adaptations: Scheduling Adaptations

Adaptive scheduling involves the alteration of the scheduling policy in response to changes in the environment.

Examples:

•Increase the level of parallelism of a service. •Decrease the level of parallelism of a service. •Restart service. •Pause service. •Move service between execution nodes.

See paper for further detail


5. Categorising Workflow Adaptations

Monitoring Analysis Planning ExecutionProgress of a service

Completion of a service

Data consumption rate of a service

Data production rate of a service

Available execution nodes

Load on an execution node

Load on a network link

Memory usage on an execution node

Available services

Available data resources

Load Imbalance

Bottleneck

Potential Workflow QoS miss

Execution node failure

Free capacity

New service available

New data available

Underutilised execution node

Increase service parallelism

Reschedule a service

Replace a service

Use free execution nodes

Move services

Change data sources

Execute changes

This level of understanding, combined with a adaptivity infrastructure

provides a solid basis for providing adaptivity functionality

We used the MAPE functional decomposition to look at workflow adaptation

The Monitoring, Analysis, Planning and Execution functional phases can be used to investigate the adaptive opportunities

They provide a consistent, abstract viewpoint with which to expressadaptation strategies

The Various options in M,A,P,E can be arranged as follows:


6. Current activities and Future work

Creating an infrastructure to support the Systematic Development of Adaptive Systems based on the ideas presented today Ease the development of adaptive systems. Support the development of better adaptive systems Investigate the use of the infrastructure in a number of

different domains Use the infrastructure to improve the general

understanding of adaptive systems Applying the infrastructure to related domains

Simulated DAG Scheduling Workflow processing with the Pegasus team Concurrent business workflows Distributed Query Processing

case studies...


Project Organisation

Kevin Lee, Norman W. Paton, Alvaro A. A. Fernandes, Rizos SakellariouSchool of Computer Science, University of Manchester

Oxford Road, Manchester, M13 9PL, U.K.{klee, rizos, norm, alvaro}@cs.man.ac.uk

Jim Smith, Paul WatsonSchool of Computing Science, Newcastle University

Claremont Road, Newcastle upon Tyne, NE1 7RU, U.K.{Jim.Smith, Paul.Watson}@ncl.ac.uk

EPSRC e-Science project entitled:

“An Infrastructure for Adaptive Systems Development”


Questions?/Comments?

workflow adaptation as an autonomic computing problem

Documents