ruslan.shevchenko: most functional-day-kiev 2014
Post on 10-May-2015
298 Views
Preview:
DESCRIPTION
TRANSCRIPT
SCALA - organizing control flow (between imperative and declarative approaches)Ruslan Shevchenko GoSave !ruslan@shevchenko.kiev.ua @rssh1 https://github.com/rssh
Styles of control flow organizations:
!• Future-s … • async/await • Actors • Channels • Reactive Collections • DSLs for Computation Plans
Control Flow
val x = readX val y = readY val z = x*y
scala
X
Y
Z
Imperative = we explicitly set one
directed by evaluation strategy
z = x*y where x = readX y = readX
haskellX Y
Z
let x = readX y = readX in x*y
Control Flow:
!• Imperative/Declarative (?) !• Declarative + Understanding =
Imperative
Control flow: what we need ?
!• Manage multi{core,machine} control-
flows. !• Optimize resource utilization
( see reactive manifesto http://www.reactivemanifesto.org/ )
Reactivity
!• Ugly situation in industry !• In ideal world - {operation system/
language VM} must care about resource utilization, human - about logic !
• Our world is far from ideal
Control Flow
val x = readX val y = readY val z = x*y
Can we readX and Y in parallel
X Y
Z
Low level@volatile var x = _ val xThread = new Thread( public void run() { x = readX } ).start(); !@volatile var y = _ val yThread = new Thread( public void run() { y = readY } ).start(); yThread.join() xThread.join() z = x+y
As GOTO from 60-s
Low level with thread poolval xTask = new Task( public X run() { readX } ); pool.submit(xTask) !val yTask = new Task( public Y run() { readY } ) pool.submit(yTask) !z = xTask.get()+yTask.get()
X Y
Z
Scala Future
X Y
Z
val x = Future{ readX } val y = Future{ readY } val z = Await.result(x, 1 minute) + Await.result(y, 1 minute)
Future
!• Future[T] = pointer to the value in the
future !• isComplete: Boolean
!• Await.result(feature, duration): T !• onComplete(Try[T] => U) : Unit
Future { readX }
object Future { ! def apply[T](body: =>T) : Future[T] = ………… !}
Call by name syntax = [call by function .. ] by name - term from Algol 68
Own functions like control flow constructions
Future-s are composable.
!• map: X=>Y Future[X] => Future[Y] !• Future[X].map[Y](f:X=>Y): Future[Y]
!• flatMap: • X => Future[Y] Future[X]=>Future[Y] !
• Future[X].flatMap[Y](f: x => Future[Y]):Future[Y]
Future-s are composable: map
!• Future[X].map[Y](f:X=>Y): Future[Y] ! Future{ calculatePi() } map ( _ + 1) = ! Future{ calculatePi() } map (x => x+1) = ! Future{ => 4.1415926 }
Future-s are composable: flatMap
!• Future[X]:
• flatMap[Y](f:X=>Future[Y]): Future[Y] ! Future{ calculatePi() } flatMap ( calculateExp(_) ) = ! Future{ => e^pi … }
Scala Futureval x = Future{ readX } val y = Future{ readY } val z = Future{ … after X and Y }
val z = Future{ readX } flatMap { x => readY map ( y=> x+y ) } !
Scala Futureval x = Future{ readX } val y = Future{ readY } val z = Future{ … after X and Y }
val xFuture = Future{ readX } val yFuture = Future{ readY } val z = xFuture flatMap { x => yFuture map ( y=> x+y ) }
for{ x <- Future{ readX }, y <- Future{ readY }) yield x+y !
Using monadic syntax:
Scala: monadic syntaxfor{ x <- Future{ readX }, y <- Future{ readY }) yield x+y !
Future{ readX } flatMap{ x => for (y <- Future{ readY }) yield x+y }
Future{ readX } flatMap{ x => Future{ readY } map ( y => x+y ) }
Future
!• Good for simple cases !• Hard when we want to implement
complex logic !
SIP22 (Async/Await)
!• when we want to implement complex
logic !• Ozz style -> (limited) F# ->(limited) C# !• in scala as library: https://github.com/
scala/async !
SIP22 (Async/Await)
!• async(body: =>T):Future[T] !• await(future: Future[T]): T
• can be used only inside async !• async macro rewrite body as state
machine. !
Async/Awaitval x = Future{ readX } val y = Future{ readY } val z = Future{ … after X and Y }
val z = async{ val x = future{ readX } val y = future{ readY } await(x) + await(y) }
Async/Awaitval z = async{ val x = Future{ readX } val y = Future{ readY } await(x) + await(y) }
val z = { var state=0, awaitX=false, awaitY=false def f = state match { case 0 => state = 1 Future{ x=readX(); awaitX=true } onComplete f Future{ x=readY(); awaitY=true} onComplete f case 1
// Just show the idea, not actual
Async/Await
val z = { var state=0, awaitX=false, awaitY=false val res = Promise[X]() def f = state match { case 0 => state = 1 Future{ x=readX(); awaitX=true } onComplete f Future{ x=readY(); awaitY=true} onComplete f case 1 => if (awaitX && awaitY) { res.successful(x + y)} } f() res.future() }
// Just show the idea, not actual
Async/Await
!• Good for relative complex logic !• No support for awaits inside closures
inside async block. !• Still low-level, can’t be used for
organizing program structure !
Akka http://www.akka.io
!• Erlang-style concurrency !• Actor - active object
• available by name in akka cluster. • send message (opt. receive answer) • have local state
!!
!!
!
Actor
Mailbox
Processor (with state) 1
Akka
!• tell(x:Any)=>Unit — send and forget
actor ! x !!• ack(x:Any)=>Future[Any] — send and
receive future to answer actor ? x !!
Actor!class EventsProcessor extends Actor { var nMessages=0 ! def receive = { case Event(msg) => println(msg) nMessages+=1 case Echo(msg) => sender ! Echo(msg) case Ping => sender ! Pong case Stop => context.stop(self) } !}
Akka
!• Actor Supervising (restart if fail) !• Utils (Scheduler, EventBus, … )
!• Common patterns
• Load balancing • Throttling messages • …..
!!
Akka
!• Scale on cluster
!• Persistent Queue/ Actor State !• Optional monitoring console
(commercial) !!
Akka : Differences from Erlang Model
!• No blocking inside actors. [Use
additional tread-pool] !
• Scheduler switch to other actor after processing <N> messages (<N> instructions in Erlang)
!• Supervising is more robust. !!
Go-like channels
!• Bounded Queue !• coroutines (different threads) can
• write to queue (wait if full) • read from queue (wait if full) • select/wait one from possible
operations
Go-like channels
!• Implemented as library on top of Akka
and SIP22 !
• Fully async (blocked reads must be in async block) !
• (yet not ready) ;))) https://github.com/rssh/scala-gopher branch “async/unsugared.”
Rx streams
!• Reactive extensions
• http://rxscala.github.io/ !• Rx collection call you.
!
Rx scala
!trait Observer[E] { def onNext(e: E) def onError(e: Throwable) def onCompleted() !} !!
!trait Observable[E] { ! def subscribe(o:Observer) ! ………. ! map, flatMap, zip, filter … } !
Rx scala: Observer/Iterator duality
!trait Observer[-E] { def onNext(e: E) def onError(e: Throwable) def onCompleted() !} !!
!trait Iterator[+E] { ! def next: E // throw …. def hasNext() ! } !
Rx streams
!• Use - if event-source provide this format. !• Typical pattern - collect event-sources
into you collection via hight-level operations, than process. !
Computation plan DSL !• Collections operations can be distributed !• Simple form: .par
for( x <- collection.par) yield x+1 !
!• Same idea for hadoop map/reduce !
Computation plan DSL for Hadoop
!• Scalding
• https://github.com/twitter/scalding !• Scoobi
• http://nicta.github.io/scoobi/ !
• Scrunch • http://crunch.apache.org/scrunch.html
Scoobi!val lines = fromTextFile("hdfs://in/...") !val counts = lines.mapFlatten(_.split(" ")) .map(word => (word, 1)) .groupByKey .combine(Sum.int) !counts.toTextFile(\“hdfs://out/…", overwrite=true).persist(ScoobiConfiguration())
map, groupByKey, combine => Map/Reduce tasks
Mahout: computation plan DSL for Spark
!• https://mahout.apache.org/
!• Scalable machine learning library. • R-like matrix operations • Optimizer for algebraic expression
Mahout!// R-like operations (linear algebra) !val g = bt.t %*% bt - c - c.t + (s_q cross s_q) * (xi dot xi) !drmA.mapBlock(ncol = r) { case (keys, blockA) => val blockY = blockA %*% Matrices.symmetricUniformView(n, r, omegaSeed) keys -> blockY }
Match operations => computations plans on storm claster
Mahout!val inCoreA = dense(! (1, 2, 3, 4),! (2, 3, 4, 5),! (3, -4, 5, 6),! (4, 5, 6, 7),! (8, 6, 7, 8)! ) val A = drmParallelize(inCoreA, numPartitions = 2) !val inCoreB = drmB.collect
In core => out core transformation
Scala: organization of control flow
!• Many styles. No one is better. !• Low-level: futures & callbacks !• Middle-level: actors, channels, streams !• Hight-level: declarative DSL
Scala: organization of control flow
• Possibilities: • Flexible syntax • Call-by-name • Macroses
!• Limitations:
• JVM • Complex language constructions (hard
to change structure.)
Scala: organization of control flow
!• Thanks for attention. !• Questions (?) !• //Ruslan Shevchenko @rssh1
<ruslan@shevchenko.kiev.ua> !
• GoSave, inc. !
top related