Streamflow - Programming Model for Data Streaming inScientific Workflows

Chathura Herath

OutlineBackgroundMotivationApproachArchitectureProgramming ModelDomain application

BackgroundScientific workflow are a

good programming model for scientific computing

Scientific domains have high volumes of data

Most of the data are coming from sensors, catalogs and other experiments.

Most data sources are data streams or can be modeled as streams.

Motivation Huge data sources require preprocessing and mining

and scaling down of data volumes. Compute resources are limited when taking the scale of

date. Currently experts determine which data sets contain

the interesting data Preserve the workflow programming model for the user. Users are familiar with DAG execution Define workflow patterns for use as new workflow

semantics that can capture data streams Goal

◦ Real-time data mining, filtering and preprocessing◦ Data-driven reactive workflow systems◦ Feedback systems

Data to Information Data Storage

Supercomputing

Information RateData Rate

Data to Information Data Storage

Supercomputing

Information RateData Rate

Scientific workflow

Stream Mining

Streamflow Data Storage

Supercomputing

Information RateData Rate

Streamflow

Why Workflow Streaming?Most scientific workflows

are staticConsiderable segment of

scientific data for scientific workflows are produced by scientific sensors

Sensor data tend to behave as repeating data streams

It is possible to provide a programming abstraction to capture data search and filtration?

Possible approachesComplete decoupled systems where

workflows and the data mining is separate.◦ Data mining rules or queries would produce

outputs which would may get refined again and again.

◦ Some interesting event would launch the workflow.◦ It may loose the insight and abstraction provided

by the workflows◦ The Data mining itself may have complex data and

control dependencies Pure workflow approach

◦ Workflow languages are not designed for streaming

Stream Integration Approach Complex Event Processing system

◦ Interact with the streams◦ Filter and bundle data◦ Publish input datasets to workflows

Workflow system◦ Handles the scientific computations◦ Gets invoked when dataset of specified

nature gets published to the CEP system

Resources

Streamflow Semantics

StreamBase Workflow

Streamflow Composer

Esper

STREAMing workFLOWS -Streamflows

Streamflows are enhancement of workflows to handle data streams

Allows the complex experimental logic to be encapsulated using scientific workflows

Allows the management of large streams of data with stream mining

Provide a programming model similar to workflow composition to handle streams Workflo

w Streamflow

Stream Integration

Select * from DataminedRUCDATA(reflectivity> 3.5).win:time_batch(1h)

Workflow Semantics Conventional SOA components

can be used as it is. Workflow components may

change behavior based on input data or stream.

Filter nodes will change the “cardinality” of the output stream

Aggregator will aggregate data over a window.

Generator node interface external stream to the Streamflow

Programming model

Join semantics◦ Constant inputs need to be matched to

streams.Inputs Streamed into the workflow

from Stream EngineOutputs are published back by stream

sinks and may be used for feedback.

Evaluation Deployment Overhead

◦ Extra overhead as the workflow is flat. Θ(1)

◦ Extra overhead are comparable to the normal workflow deployment because it may need to deploy new workflows

Runtime Latency◦ Latency of

event arriving at the framework to be delivered the workflow.

Evaluation

DomainsMeteorologyAstronomy

On-DemandGrid Computing

StreamingObservations

Storms Forming

Forecast Model

Data Mining

Astronomy

Meteorology

Related work B. Biornstad. A workflow approach to stream processing,

PhD Thesis, Computer Science Department, ETH Zurich. Y. Liu, N. Vijayakumar, and B. Plale. Stream processing in

data-driven computational science. In Proceedings of the 7th IEEE/ACM International Conference on Grid Computing, pages 160–167. IEEE Computer Society Washington, DC, USA, 2006.

J. Buck, S. Ha, E. Lee, and D. Messerschmitt. Ptolemy: A framework for simulating and prototyping heterogeneous systems. International Journal of Computer Simulation, 4(2):155–182, 1994. – DataTurbine

Y. Cai et al. MAIDS: Mining Alarming Incidents from Data Streams Automated Learning Group, NCSA, University of Illinois at Urbana-Champaign, U.S.A.

Future workDevelop a formal model for the

workflow semanticsEvent order guarantees How to handle missing streamsProvenance for data streams.

Questions ?