akka streams in action @ scaladays berlin 2016

Konrad `@ktosopl` Malawski // ScalaDays Berlin pic: Armored Core for Answer, White Gilt

streamsand Reactive Streams in action

Konrad `ktoso` Malawski

Akka Team, Reactive Streams TCK,

Persistence, HTTP

Konrad `@ktosopl` Malawski

akka.iotypesafe.comgeecon.org

Java.pl / KrakowScala.plsckrk.com

GDGKrakow.pl lambdakrk.pl

http://akka.io

http://typesafe.com

http://geecon.org

http://java.pl

http://krakowscala.pl

http://sckrk.com

http://gdgkrakow.pl

http://www.lambdakrk.pl

… in Action

Yes, we’ll build stuff.

… in Action

Yes, we’ll build stuff.

With pre-release stuff, because ScalaDays.

“Stream”

“Stream” What does it mean?!

Akka Streams

Akka Streams && Reactive Streams

Reactive Streams - story: early FRP

http://blogs.msdn.com/b/rxteam/archive/2009/11/17/announcing-reactive-extensions-rx-for-net-silverlight.aspxhttp://infoscience.epfl.ch/record/176887/files/DeprecatingObservers2012.pdf - Ingo Maier, Martin Odersky

https://github.com/ReactiveX/RxJava/graphs/contributorshttps://github.com/reactor/reactor/graphs/contributors

https://medium.com/@viktorklang/reactive-streams-1-0-0-interview-faaca2c00bec#.69st3rndy

- .NETs’ Reactive Extensions

.NET 3.5

http://blogs.msdn.com/b/rxteam/archive/2009/11/17/announcing-reactive-extensions-rx-for-net-silverlight.aspx

http://infoscience.epfl.ch/record/176887/files/DeprecatingObservers2012.pdf

https://github.com/ReactiveX/RxJava/graphs/contributors

https://github.com/reactor/reactor/graphs/contributors


Reactive Streams - story: 2013’s impls

~2013:

Reactive Programming becoming widely adopted on JVM.

- Play introduced “Iteratees”- Akka (2009) had Akka-IO (TCP etc.)- Ben starts work on RxJava




Teams discuss need for back-pressure in simple user API.Play’s Iteratee / Akka’s NACK in IO.

}







Play Iteratees – pull back-pressure, difficult API




Akka-IO – NACK back-pressure; low-level IO (Bytes); messaging API

RxJava – no back-pressure, nice API






Reactive Streams - Play’s Iteratees

def fold[B]( done: (A, Input[E]) => Promise[B], cont: (Input[E] => Iteratee[E, A]) => Promise[B], error: (String, Input[E]) => Promise[B]): Promise[B]

// an iteratee that consumes chunkes of String and produces an IntIteratee[String,Int]

https://www.playframework.com/documentation/2.0/Iteratees

Feb 2013 Iteratees solved the back-pressure problem,but were hard to use.

Iteratee & Enumeratee – Haskell inspired.

Play / Akka teams looking for common concept.

https://www.playframework.com/documentation/2.0/Iteratees

Reactive Streams - expert group founded

October 2013

Roland Kuhn (Akka) and Erik Meijer (Rx .NET) meet in Lausanne,while recording “Principles of Reactive Programming” Coursera Course.

Viktor Klang (Akka), Erik Meijer, Ben Christensen (RxJava) and Marius Eriksen (Twitter) meet at Twitter HQ.

The term “reactive non-blocking asynchronous back-pressure” gets coined.

https://www.coursera.org/course/reactive


October 2013




Goals:- asynchronous- never block (waste)- safe (back-threads pressured)- purely local abstraction- allow synchronous impls.

Also, for our examples today:- compatible with TCP



October 2013




December 2013Stephane Maldini & Jon Brisbin (Pivotal Reactor) contacted by Viktor.



October 2013





Soon after, the “Reactive Streams” expert group is formed.

Also joining the efforts: Doug Lea (Oracle), Endre Varga (Akka), Johannes Rudolph & Mathias Doenitz (Spray), and many others, including myself join the effort soon after.


October 2013





Soon after, the “Reactive Streams” expert group is formed.

Also joining the efforts: Doug Lea (Oracle), Endre Varga (Akka), Johannes Rudolph & Mathias Doenitz (Spray), and many others, including myself join the effort soon after.


I ended up implementing much of the TCK.Please use it, let me know if it needs improvements :-)



2014–2015:

Reactive Streams Spec & TCK development, and implementations.

1.0 released on April 28th 2015, with 5+ accompanying implementations.

2015Proposed to be included with JDK9 by Doug Leavia JEP-266 “More Concurrency Updates”

http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/6e50b992bef4/src/java.base/share/classes/java/util/concurrent/Flow.java

http://openjdk.java.net/jeps/266


2014–2015:

Reactive Streams Spec & TCK development, and implementations.

1.0 released on April 28th 2015, with 5+ accompanying implementations.

2015Proposed to be included with JDK9 by Doug Leavia JEP-266 “More Concurrency Updates”



http://openjdk.java.net/jeps/266


JEP-266 – soon…!public final class Flow { private Flow() {} // uninstantiable

@FunctionalInterface public static interface Publisher<T> { public void subscribe(Subscriber<? super T> subscriber); }

public static interface Subscriber<T> { public void onSubscribe(Subscription subscription); public void onNext(T item); public void onError(Throwable throwable); public void onComplete(); }

public static interface Subscription { public void request(long n); public void cancel(); }

public static interface Processor<T,R> extends Subscriber<T>, Publisher<R> { }}

Reactive Streams: goals1) Avoiding unbounded buffering across async boundaries2) Inter-op interfaces between various libraries

Reactive Streams: goals1) Avoiding unbounded buffering across async boundaries

2) Inter-op interfaces between various libraries



Argh, implementing a correct RS Publisher or Subscriber is so hard!



Argh, implementing a correct RS Publisher or Subscriber is so hard!

Don’t implement RS interfaces directly, use GraphStage instead!

Akka Streams

Streams complement Actors, they do not replace them.

Actors – distribution (location transparency) Streams – back-pressured + more rigid-blueprint

13番: Akka is a Toolkit, pick the right tools for the job.

Runar’s excellent talk @ Scala.World 2015

Asynchronous processing toolbox:

Power

Constraints

https://www.youtube.com/watch?v=GqmsQeSzMdw


Asynchronous processing toolbox:

Constraints

Power


Single value, no streaming by definition.Local abstraction. Execution contexts.

Power

Constraints


Mostly static processing layouts.Well typed and Back-pressured!

Constraints

Power


Plain Actor’s younger brother, experimental.Location transparent, well typed.Technically unconstrained in actions performed

Constraints

Power


Runar’s excellent talk @ Scala.World 2015

Location transparent. Various resilience mechanisms.

(watching, persistent recovering, migration, pools)

Untyped and unconstrained in actions performed.

Constraints

Power

https://www.youtube.com/watch?v=GqmsQeSzMdw

Lightbend monitoring

Measure. Don’t guess.

http://monitoring.lightbend.com/docs/latest/home.html


Lightbend monitoring

Measure. Don’t guess.

Let’s find if & where we have a bottleneck…


DEMO


Materialization

Gears from GeeCON.org, did I mention it’s an awesome conf?

http://geecon.org

What is “materialization” really?

Custom Stages &

Understanding the Types

Custom stages, made simple (!)

Understanding the Types

Source.fromGraph(Graph[SourceShape[Status], M]) .via(Graph[FlowShape[Status, T], M]) .to(Graph[SinkShape[Status], _])

Custom != hard; custom == extensible

Your Stages “feel” native.

DEMOLet’s try “Hundred-ify”.

http://doc.akka.io/docs/akka/2.4.7/scala/stream/stream-customize.html#graphstage-scala

See also mapConcat / statefulMapConcat.


Akka HTTP ~ streaming all the way

Streaming all the way with Akka HTTP

Streaming in Akka HTTPDEMO

http://doc.akka.io/docs/akka/2.4.7/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” https://github.com/akka/akka/pull/20778

HttpServer as a: Flow[HttpRequest, HttpResponse]


https://github.com/akka/akka/pull/20778




HTTP Entity as a: Source[ByteString, _]






HTTP Entity as a: Source[ByteString, _]

Websocket connection as a: Flow[ws.Message, ws.Message]



Persistence Query (experimental)

“The Query Side” of Akka Persistence

Persistence Query (experimental)

Persistence Query Journals

akka/akka-persistence-cassandra 0.16 akka/akka @ leveldb-journal 2.4.8

dnvriend/akka-persistence-jdbc 2.3.3 scullxbones/akka-persistence-mongo 1.2.5

…and more, that I likely forgot about.

DEMO

Implementation of “data-pump” pattern.

Akka + Kafka = BFF

Reactive Kafka

+

Started by Krzysiek Ciesielski & Adam Warski @ SofwareMill.com

http://sofwaremill.com

“ACKnowladged streams”

happy ACKing!

Kafka + Akka = BFF

Akka is Arbitrary processing.

Kafka is somewhat more than a message queue, but very focused on “the log”.

Spark shines with it’s data-science focus.

Kafka + Akka = BFF

Streams talking to Actors &&

Actors talking to Streams

Streams <=> Actors inter-op

Source.actorRef (no back-pressure) Source.queue (safe)

Sink.actorRef (no back-pressure) Sink.actorRefWithAck (safe)

Exciting times ahead!

Next steps for Akka

Completely new Akka Remoting (goal: 1M+ msg/s (!)),(it is built using Akka Streams).

More integrations for Akka Streams stages, also dynamic fan-in/out A.K.A. “the Hub”.

Reactive Kafka polishing and stable release with SoftwareMill.

“Confirmed Streams” work from Reactive Kafka generalised.

Akka Typed likely to progress again.

Of course, continued maintenance of Cluster and others.

Happy hAkking!

Happy hAkking!

Announcing that there will be an announcement.

hAkknowledge (pilot name?)

“initiative” / “competition”

Akka <3 contributions• Easy to contribute:

• https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute • https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22

• Akka Stream Contrib • https://github.com/akka/akka-stream-contrib

• Mailing list:• https://groups.google.com/group/akka-user

• Public chat rooms: • http://gitter.im/akka/dev developing Akka • http://gitter.im/akka/akka using Akka

https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute

https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22

https://github.com/akka/akka-stream-contrib

https://groups.google.com/group/akka-user

http://gitter.im/akka/dev

http://gitter.im/akka/akka

Thanks!

ktoso @ typesafe.com twitter: ktosopl

github: ktosoteam blog: letitcrash.com

home: akka.io

Thus spake the Master Programmer:

“After three days without programming,life becomes meaningless.”

http://twitter.com/ktosopl

http://github.com/ktoso

http://letitcrash.com

http://akka.io

Q/A (Now’s the time to ask things!)

ktoso @ typesafe.com twitter: ktosopl

github: ktosoteam blog: letitcrash.com

home: akka.io

http://twitter.com/ktosopl

http://github.com/ktoso

http://letitcrash.com

http://akka.io

akka streams in action @ scaladays berlin 2016

Technology