dataflow: the concurrency/parallelism architecture you need

@russel_winder#devoxxuk #dataflowrules Copyright © 2014 Russel Winder

Dataflow:

Russel Winder@russel_winder http://[email protected]

The Concurrency/ParallelismArchitecture You Need


What is Dataflow?


What are (in computing†):

Concurrency:

Structuring solution and code such that multiple parts may execute independently and possibly even at the same time.

Parallelism:

Execute multiple parts of a system at the same time on different processors so as to get things working faster.

†In natural language these words have very different meanings.


What is Dataflow?

An architecture comprising channels allowing data to flow from one operator to another, where each operator has multiple input channels and multiple output channels, and executes code only in response to the arrival of data on the inputs.


Historically

Dataflow computers:– Values flowing between…–…operators that calculate…–…new values to pass to…–…other operators.

Dataflow hardware didn't take off, but the architecture works at various scales.

The Manchester Prototype Dataflow Computer J R Gurd, C C Kirkham, I WatsonCACM 28(1), 1985-01.


Dataflow diagrams have been anintegral part of analysis and design ofinformation systems since the 1970s

T de Marco, Structured Analysis and Systems Specification,Yourdon Press, NY, 1978.


Dataflow and Functional

Operators seem like they might be pure functions, but…

…they are not necessarily, operators may have internal state.

Operators may be referentially transparent, but they may be not.

Operators may even have side effects.


Dataflow is anevent-basedarchitecture


Dataflow systems are(possibly)

reactive systems.

Which would make them exceedinglytrendy even if the idea is very old.


Dataflow systems have

no†

shared memory.

† or at least should have no.


operatorchannel


Dataflow systems aremessage passing systems.


Each operator must†

be single threaded.

† or at least should.


Dataflow Frameworks

Scala:–Future

Akka:–Dataflow variables, aka

Promise–Deprecated in favour of Async

Java:–Pre-8, Future–8+, CompletableFuture, aka

Promise


Architectural Issue

Each of the aforementioned frameworks assumes that each operator creates a single value. Communication is by dataflow variables: each dataflow variable is a thread-safe single assignment variable.


GPars…

Has dataflow variables (promises) and tasks and so can do everything Akka and Java can offer.

Has DataflowQueue, and so can create real dataflow networks.


One does like to code…

…doesn't one.


We need a problem…


A Problem

Calculate mean and standard deviation of a data sample.

x̄ =1n∑i=0

nxi

s = √ 1n−1∑i=0

n(x i− x̄)2


Amend the Problem

s = √ 1n−1 ( (∑i=0

nx i

2 )−n x̄ x̄ )

x̄ =1n∑i=0

nxi


Switch to using an IDE for this.Switch to using an IDE for this.

Code Example


Summary

Dataflow is an architecture:

Event-driven, single-threaded operators communicating by message passing using channels.

Dataflow is an easement:

Synchronization is inherent in the model, and there is no shared memory, so all deadlocks are trivial.


Dataflow is a way of harnessingconcurrency and parallelism

in easy to program ways.


GPars is usable from Javaas well as Groovy.


Testing is really Groovy with Spock.


Dataflow is an architecture ofcode you need to know.


Dataflow:

Russel Winder@russel_winder http://[email protected]

The Concurrency/ParallelismArchitecture You Need

dataflow: the concurrency/parallelism architecture you need

Technology

russel winder c

russel winder switch

russel winder gpars

russel winder dataflow

russel winder summary

russel winder operator

russel winder s um

winder http