The dark side of Akkaand the remedy

Ákos Kriváchy

2

Introduction - Ákos Kriváchy

• Scala disciple/convert/fanatic since 2013 February• First FP language

• Akka since 2014 February• Things I love about Scala:• Static typing• Partial Functions• foldLeft, tail recursion

3

Recap on Akka

• Akka in key points:• Messaging• Actors• Mailboxes (ActorRefs)• Hierarchy• Supervision• Location transparency

Source: http://www.scottlogic.com/blog/2014/08/15/using-akka-and-scala-to-render-a-mandelbrot-set.html

4

Example from previous Meetup: URL scraping

5

To Akka or not to Akka?

•Akka solves:• Concurrency• Scalability and distributability• Resilience

•Akka in our team:•We had to rewrite a legacy Java component• Chose Akka to try it out (after 4 months of research)

6

Problem #1: Any and Actor Ref• All messages are Any-s• Anything that’s not

handled ends up as a “dead letter”• Essentially a loss of all

typesafety• Requires extensive testing

to “feel safe”

class MyActor(databasePersistence: ActorRef, emailSender: ActorRef, MQSender: ActorRef, widgetDao: ActorRef, twitterService: ActorRef) extends Actor { def receive: Receive = { case Message(data) => twitterService ! Tweet(data) // ... case OtherMessage(_) => // ... }}

}

7

Solution: Typed Channels

val channelA: ChannelRef[(MsgA, MsgB) :+: TNil] = ???

val a = new MsgA

channelA <-!- a // send a to channelA

a -!-> channelA // same thing as above

val fB: Future[MsgB] = channelA <-?- a // ask the actor

Instead of the current solution:val fB: Future[Any] = actorRef ? a

http://doc.akka.io/docs/akka/2.2.0/scala/typed-channels.html

8

NOPE!

http://doc.akka.io/docs/akka/2.3.0-RC1/project/migration-guide-2.2.x-2.3.x.html

9

It’s on the roadmap

Lastly, we dare a look beyond the next major release: after Akka 2.4.0 [early 2015] we will tackle the number one feature request, namely more type-safety for inter-Actor messaging. This will be a major research effort and likely also a disruptive API change. For this reason Akka 2.4 is planned to be a long-term support release, because even though we will try to find ways for compatibility or at least an easy migration path we cannot currently foresee the exact nature of all related changes.

August 28, 2014: http://typesafe.com/blog/akka-roadmap-update-2014

10

In the meantime: Don’t use Actors for everything• What do we use instead?• Scala Futures are extremely powerful• Futures compose more nicely than Actors

• When should we use Actors?• State in the application

• A cache is not state• Extreme scaling is needed (location transparency)

• Need the flexibility to place any operation on any JVM -> use Actors for everything• Realtime data flow applications (not request/response)

• What about supervision and „let it crash”?• Future’s handle failure also

11

Code: https://github.com/krivachy/AkkaWithFsm

12

class Getter(url: String, depth: Int) extends Actor with ActorLogging with LinkParser {  lazy val client = new NingWSClient(new Builder().build()).url(url)  override def preStart() = { client.get().map(_.body).map(Response).pipeTo(self) } 

def receive = { case Response(body) => val links = parseLinks(body).toList log.info(s"URL $url at depth $depth had ${links.size} links.") links.foreach { link => log.info(s"Sending link '$link'") context.parent ! Controller.Check(link, depth) } context.stop(self) case Status.Failure(cause) => log.error(s"Failed to GET $url", cause) context.stop(self) } }

13

Simplified with Futures:object Getter extends LinkParser { case class Urls(urls: Set[String], depth: Int)  lazy val client = new NingWSClient(new Builder().build())  def get(url: String, depth: Int)(implicit ec: ExecutionContext): Future[Urls] = { client.url(url).get().map { response => Urls(parseLinks(response.body).toSet, depth) } }}

14

Problem #2: Hellish code complexity

15

var hell

• Actors have too much mutable state• Our worst scenario: Actor with 300 lines and 20 vars

• Hard to reason about state when everything is “global” inside the Actor• How do you initialize state that is only used in some cases?• var something: SomeType = _

• NPE• var something: Option[SomeType] = None

• Always need to “getOrElse”

16

become/unbecome hell

• Pushing and popping state on a stack• context.become(behavior: Receive, discardOld: Boolean = true)• context.unbecome()

• “context.become” isn’t enforced to be called last• You use methods to keep things short, but there will be multiple methods

trying to modify the behaviour• i.e. you could end up inadvertently overwriting behavior

• One place: context.become(handleCoolMessages orElse waitForNewRequests)• Somewhere else: context.become(waitForNewRequests, discardOld = true)

• When things blow up you have no idea how you got there

17

class Receptionist extends Actor { def receive = waiting  def waiting: Receive = { case Api.Scrape(url, depth) => context.become(next(Vector(Job(sender, url, depth)))) }  def running(queue: Vector[Job]): Receive = LoggingReceive { case Result(links) => // … context.become(next(queue.tail)) case Api.Scrape(url, depth) => context.become(enqueue(queue, Job(sender, url, depth))) case Terminated(_) => // … context.become(next(queue.tail)) }  def enqueue(queue: Vector[Job], job: Job): Receive = LoggingReceive { // … running(queue) }  def next(queue: Vector[Job]): Receive = LoggingReceive { // … running(queue) }}

18

Debug hell

• Debugging Actors is hard• Stacktraces lack meaning• Need to have a lot of boilerplate utility code:

• What was the message?• Who sent the message?• What’s my internal state?

• Possible solution:• always override def preRestart(reason: Throwable, message: Option[Any])• Still don’t know how we ended up in that become/unbecome state.

19

[ERROR] [akka://simple-actor/user/receptionist/controller-1/$a] Exception happenedjava.lang.Exception: exception happened here

at meetup.akka.simple.Getter$$anonfun$receive$1$$anonfun$1.apply(Getter.scala:37)at meetup.akka.simple.Getter$$anonfun$receive$1$$anonfun$1.apply(Getter.scala:37)at scala.util.Try$.apply(Try.scala:191)at meetup.akka.simple.Getter$$anonfun$receive$1.applyOrElse(Getter.scala:37)at akka.actor.Actor$class.aroundReceive(Actor.scala:465)at meetup.akka.simple.Getter.aroundReceive(Getter.scala:16)at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)at akka.actor.ActorCell.invoke(ActorCell.scala:487)at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)at akka.dispatch.Mailbox.run(Mailbox.scala:220)at

akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

20

We need Logging for Debugging

• Akka provides:• with ActorLogging

• log.info(…)• LoggingRecieve: def receive = LoggingReceive { … }

• turn on: akka.actor.debug.receive=true• Lifecycle logging: akka.actor.debug.lifecycle=true

• For all Actors• Start, restart, stop, supervise, watch events

• Autorecieve logging: akka.actor.debug.autoreceive=true• For all Actors• Automatically handled messages: Stop, Kill, PoisionPill, Terminated, etc.

• Issue:• Akka only provides logging of messages per Receive block (has pros and cons)• If you missed one => good luck debugging issues around it in production

21

[DEBUG] [akka://simple-actor/user/receptionist] started (meetup.akka.simple.Receptionist@14ba772)

[DEBUG] [akka://simple-actor/user/receptionist/controller-1] started (meetup.akka.simple.Controller@38a5d7)

[DEBUG] [akka://simple-actor/user/receptionist] now supervising Actor[akka://simple-actor/user/receptionist/controller-1#4232237]

[DEBUG] [akka://simple-actor/user/receptionist/controller-1] now watched by Actor[akka://simple-actor/user/receptionist#1565954732]

[DEBUG] [akka://simple-actor/user/receptionist/controller-1] received handled message Check(http://doc.akka.io/docs/akka/2.3.5/intro/what-is-akka.html,1)

Logging example

22

FSM

23

“Everything* is a Finite State Machine.”**-me

*not everything** do not quote me

24

FSM conceptsobject Receptionist { object Internal {  sealed trait State case object Sleeping extends State case object Processing extends State  sealed trait Data case class NoQueue(requestId: Int = 0) extends Data case class Queue(currentRequestId: Int, items: Vector[Job]) extends Data  }} class Receptionist extends FSM[Internal.State, Internal.Data] { … }

25

Define handlers for all states

// Initialize with datastartWith(Sleeping, NoQueue())// Handlers for stateswhen(Sleeping)(enqueueNewRequest)when(Processing) (processResult orElse enqueueNewRequest orElse reportError)

26

Enqueue New Request State Functiondef enqueueNewRequest: StateFunction = { case Event(Api.Scrape(url, depth), NoQueue(requestId)) => startControllerFor(requestId + 1, Vector(Job(sender(), url, depth))) case Event(Api.Scrape(url, depth), queue: Queue) => if (queue.items.size > 3) { stay replying Api.Failed(url) } else { goto(Processing) using Queue(queue.currentRequestId, queue.items :+ Job(sender(), url, depth)) }}

• Important:• StateFunction: Event => State• Event(incomingMessage, data) =>• State transition: goto/stay (nextState) using (data) forMax(timeout) replying (message)

27

Monitoring state transitions

onTransition { case Idle -> Active => setTimer("timeout", Tick, 1 second, true) case Active -> _ => cancelTimer("timeout") case x -> Idle => log.info("entering Idle from " + x)}

monitoredActor ! SubscribeTransitionCallBack(self)

def recieve = {

case Transition(monitoredActor, oldState, newState) =>

if (newState == Errored) alert.raiseAlert(...)

}

override def postStop() = {

monitoredActor ! UnsubscribeTransitionCallBack(self)

}

Internal:

External:

28

Handling failurewhenUnhandled { case Event(any, data) => val logUpToHere = prettyPrint(getLog) log.error(s"Unhandled event: ${any}\n$logUpToHere") stay()}

def failure: StateFunction = {

case Event(Status.Failure(cause), _) =>

log.error(s"Failed to GET $url", cause)

stop(FSM.Failure(cause))

}

onTermination {

case StopEvent(FSM.Normal, state, data) => ???

case StopEvent(FSM.Shutdown, state, data) => ???

case StopEvent(FSM.Failure(cause), state, data) => ???

}

29

Result is simplicityclass Receptionist extends FSM[Internal.State, Internal.Data] {

startWith(Sleeping, NoQueue())

when(Sleeping)(enqueueNewRequest)

when(Processing) (processResult orElse enqueueNewRequest orElse reportError)

def enqueueNewRequest: StateFunction = ???

def processResult: StateFunction = ???

def reportError: StateFunction = ???

private def nextQueueItem(queue: Queue): State = ???

private def startControllerFor(requestId: Int, queue: Vector[Job]): State

= ???

whenUnhandled { ??? }

initialize()

}

30

Hell status

var ✔

No more mutable global state inside Actors. Everything is typed to the specific State.

become/unbecome ✔

All methods have to end in a state transition. States are clearly defined what they do.

debug/logging ?

31

LoggingFSM

• Remembers state transitions:• override def logDepth = 8• def getLog: Seq[LogEntry[Internal.State, Internal.Data]]• Use with: onTermination

• Debug logging: akka.actor.debug.fsm=true• Automatically logs important Events: message + internal data• Logs state transitions

• Use with: akka.actor.debug.lifecycle=true

32

Debug log example

[DEBUG] [akka://fsm/user/receptionist] processing Event(Result(Set([…])),Queue(1,Vector(Job(Actor[akka://fsm/system/testActor1#758674372],http://doc.akka.io/docs/akka/2.3.5/intro/what-is-akka.html,1)))) from Actor[akka://fsm/user/receptionist/controller-1#4232237]

[DEBUG] [akka://fsm/user/receptionist/controller-1] transition CollectingResults -> Completed

[DEBUG] [akka://fsm/user/receptionist] transition Processing -> Sleeping

[DEBUG] [akka://fsm/user/receptionist/controller-1] stopped

33

[ERROR] [akka://fsm/user/receptionist] Unhandled event: some stringLast 8 entries leading up to this point: in state: Sleeping with data: NoQueue(2) received: Scrape(http://non-existent.link,5) in state: Processing with data: Queue(3,Vector(Job(Actor[akka://fsm/system/testActor1#758674372],http://non-existent.link,5))) received: Result(Set(http://non-existent.link)) […] in state: Processing with data: Queue(4,Vector(Job(Actor[akka://fsm/system/testActor1#758674372],http://non.existent1,0), Job(Actor[akka://fsm/system/testActor1#758674372],http://non.existent2,0), Job(Actor[akka://fsm/system/testActor1#758674372],http://non.existent3,0), Job(Actor[akka://fsm/system/testActor1#758674372],http://non.existent4,0))) received: some string

34

Hell status

var ✔

No more mutable global state inside Actors. Everything is typed to the specific State.

become/unbecome ✔

All methods have to end in a state transition. States are clearly defined what they do.

debug/logging ✔

FSM does all the debug logging we would ever need.

Conclusion: My experiences

• Akka is awesome, but:• Needs a new mindset• Unexperienced developers + learning curve + easy to make

mistakes = headaches• In large teams (30+) where everyone is expected to be able to

change all code it becomes an issue.• Use Actors only when really needed• All Actors should probably be FSM

Some people, when confronted with a scalability problem, think “I know, I'll use Akka.” Now they have two problems.

@krivachy https://github.com/krivachy/AkkaWithFsm