Multi-language FBP with Flowex

#pivorak 21, Elixir edition, March 24, 2017

Hello!I am Anton Mishchuk

▷ Ruby developer at Matic Insurance Services(we are hiring!)

▷ A big fan of Elixir programming language▷ Author and maintainer of:

ESpec and Flowex libraries

github: antonmi

2

Preface

Flow-based programming with Elixir

3

Preface

Flowex - FBP with Elixir GenStage

4

What’s next?

▷ Let Elixir do what it does best

▷ The rest can be implemented in any other programming language

5

Agenda

▷ Quick intro to FBP

▷ Flowex library overview

▷ Tools for running external programs

▷ Multi-language components example:Ruby, Python, shell

6

1.Flow-Based Programming

Basic concepts

7

Flow-Based Programming is

a programming paradigm which models application as a graph of independent processes exchanging data across predefined connections

8

FBP diagram

A

B

D

C

IN 1

IN 1

IN 1

IN 2

IN 1

IN 2

OUT 1

OUT 2

OUT 1

OUT 1

OUT 1

9

FBP advantages

▷ Parallelism are natural

▷ No problems associated with shared memory and locks

▷ Visualisation of data flow networks is simple

▷ Components are independent and reusable

10

Two layers in FBP application

▷ “Bottom layer” - set of components implementing parts of business logic

▷ “Top layer” - “communication logic”  -  an organization of data flows from one component to another

11

“Bottom layer” - components

▷ Component is independent unit with input and output

▷ The only thing it does - transform input IP to output IP

▷ In general, can be implemented using any programming language

12

“Top layer” - communication

▷ Need “coordination language”

▷ Actor model of Elixir/Erlang

▷ GenStage

13

GenStage

▷ behavior for exchanging events with back-pressure between Elixir processes

▷ José Valim defined GenStage as “better abstractions for working with collections”

▷ but we can use it for “better interprocess communication”

14

2.Flowex

Railway Flow-Based Programming

15

Flowex

▷ a set of abstractions built on top Elixir GenStage which allows writing program with FBP paradigm

▷ a mix of FBP and Railway Oriented Programming (ROP) approach

16

Railway oriented programming

17

Railway FBP

Let’s place each function inside separate process

18

use Flowex.Pipeline

19

Functions’ definitions

20

Control and evaluation

21

Some new functions appear in the FunPipeline module:

▷ “start” and “stop”

▷ “call” and “cast”

FunPipeline.start

22

“call” and “cast” (sync and async)

23

FunPipeline.call

24

add_one

minus_three

mult_by_two

producer

consumer

number: 2

number: 2

number: 3

number: 6number: 3number: 3

self()

“error_pipe”

If an error happens in some pipe, functions in next pipes will not be called and IP will be bypassed to the “error_pipe”

25

Run pipeline via Flowex.Client

▷ Use Clients for parallel processing of many IPs

▷ Allows effective using of pipeline even with synchronous “call”

26

Fight bottlenecks

27

“Controlled parallelism”

28

Reusable components with module pipes

29

One should implement “init” and “call” functions(like in Elixir’s Plug library)

Advantages of Flowex approach

30

▷ Explicit definition of:- structure of data will be processed- way the data will come

▷ Easy to maintain and reuse:- predefined set of working processes- components are isolated and reusable

▷ Controlled parallelism:- one can control number of clients supplying data - one can control the number of processes available for each component

3.Running external programs

Tools

31

Why do we need this?

▷ Elixir/Erlang ecosystem is not so big as Java, Ruby, Python, etc.

▷ Elixir/Erlang is not the fastest platform

▷ There are not so many Elixir/Erlang developers

32

Erlang ports

▷ provide a byte-oriented interface to an external program over STDIN and STDOUT

▷ are process-specific resources - only owner process can talk to external process

▷ are flexible, safe and allows error trapping

33

ErlPort project

▷ helps connect Erlang to other programming languages

▷ currently supported external languages are Python and Ruby

▷ has Elixir wrapper - export

Read more on erlport.org

34

Porcelain project

▷ helps launching and communicating with external OS processes from Elixir

▷ launches external programs in a synchronous or asynchronous manner

▷ implements multiple ways of passing input to the program and getting back its output

Read more on alco/porcelain35

4.Multi-language components

Example project

36

https://github.com/antonmi/multi_flowex

MultiFlowex pipeline

▷ There will be Ruby, Python and shell “pipes”

▷ Each will say “Hello”

▷ The result will be:“Hello from Ruby, Hello from Python, Hello from shell”

37

multi_flowex dependencies

38

Ruby component

Demonstrates how one can create components using Ruby, Python or Shell

Read more on ….

39

Ruby code in main.rb file

40

Ruby component specs

41

Python component

Demonstrates how one can create components using Ruby, Python or Shell

Read more on ….

42

Python code in main.py file

43

Shell component

44

The last, Elixir component

45

MultiPipeline

46

Pipeline starts external processes

47

▷ 3 Ruby processes

▷ 4 Python processes

MultiPipeline specs

48

Conclusion

49

▷ FBP paradigm is an advantageous alternative to “conventional programming”

▷ Flowex makes it easy to convert sequential code into a pipeline of independent components

▷ Erlang ports open new horizons, so you can leverage the advantages of other programming languages

Thanks!Any questions?

50