unibasel christoph langguth, paola ranaldi, heiko schuldt university of basel, database and...

Christoph Langguth - Towards QoS for Scientific Workflows - SWF 2009 - July 10 2009, Los Angelesunibasel

Christoph Langguth, Paola Ranaldi, Heiko Schuldt

University of Basel, Database and Information Systems Group Bernoullistr 16, CH-4056, Basel, Switzerland

{firstname.lastname}@unibas.ch

This work has been partly supported by the Hasler Foundation within the project COSA (Compiling Optimized Service Architectures)

Towards Quality of ServiceFor Scientific Workflows by usingAdvance Resource Reservations

Christoph Langguth - Towards QoS for Scientific Workflows - SWF 2009 - July 10 2009, Los Angeles 2unibasel

A motivating example: weather forecast

• Long-running and data-intensive• And time-critical: must be finished by the time of evening news

ADAS-ARPSData Analysis

Satellite DataRemapper

L3 RadarRemapper

L2 RadarRemapper

TerrainPreprocessor

Gridded DataInterpolator

SurfacePreprocessor

ARPS to WRFConverter

WRFModel

WRF to ARPSConverter

ARPSPlotting

Preprocessing

Misc.Transformation

Analysis

Legend

(*) Simplified version of workflow from Droegemeier, Gannon, Reed, et al: Service-Oriented Environments for Dynamically Interacting with Mesoscale Weather


QoS, and how to provide it – in a nutshell

• Individual services give guarantees about their execution time– Which are combined to QoS guarantees for the entire WF

– users could specify: this WF must be executed as [fast, cheap, energy-efficient,...] as possible

• Contracts negotiated using WS-Agreement

• Needs predictable resource utilization on provider's side– Resources needed must be announced, and reserved for execution

– Advance Reservations for every service call

• A bunch of metadata is required for planning the execution and setting up the reservations


WF execution in DWARFS: the big picture

A

B

D E

F G H

CEng1Eng3

Eng2

A

B

D E

F G H

CX:50GB

T:40GB

V:35GB

W:23KBZ:10KB

Y:15KB

small amount of data

Data dependencies

large amount of data

DWARFS: Distributed Workflow execution engine with Advance Reservation Functionality Support

• Partition the process for distributed execution:

– Keeping large data transfers “local“, as far as possible

– Inter-partition data transfers are handled by special DWARFS storage subsystem

Start: 11:30End: 13:07CPU: 80%Storage: 10%

Start: 17:04End: 17:53CPU: 25%

• Scheduling, Partitioning, Reservations are all interrelated

• And depend on the metadata that service providers make available


Required Metadata for Reservations...

CPU:

storage:

HW:

Provider A Provider B

I'll need to call Operation A w/INPUT: size=50GB, Class X

OUTPUT: 25GB, Class Y

timing:

50 % max.duration inv. prop.

30 % min.

10 % exactly

2:30 h

100 % max.duration inv. prop.

5 %min.

100 % exactly

3:50 h


...Required Metadata for Reservations

• Call of operation X:– Input characterization (size, class)– Resource requirements– Timing prediction, and resource ↔ timing relationship– Output characterization

• Also need providers' current resource allocation schedules, and cost functions

• Unified notion of resource „share of capacity“• Data characteristics metadata and duration prediction

highly domain-specific– Only requirement for us: determine size of data


Past, present and future

• Prototype of CPU share enforcement/timing prediction• Formal model of DWARFS Workflow

• WS-Agreement with renegotiation support• Planner implementation

• Storage Subsystem• Bringing it all together :-)


Thank you for your attention!


...CPU enforcement in action


Required Metadata for reservations• Call of service operation X

•with input characterized so and so,

– How long will this take?

– Which resources are needed?t, and how does modifyingthe share influence the timing?

– Which output will this produce?

• Providers need to also make available theircurrent resource allocation schedules, and therespective cost functions

• Any kind of resource can be represented using the unified notion of „share of local capacity“

• Data characteristics metadata will be domain-specific.

– Only requirement for DWARFS: be able to determine the size of it

– (prediction of duration is also highly domain- and implementation-specific)

unibasel christoph langguth, paola ranaldi, heiko schuldt university of basel, database and...

Documents