from polling to real time: scala, akka, and websockets from scratch

Letgo chat From polling to real timeScala, Akka, and WebSockets from scratch

@SergiGP @GVico46

@JavierCane#scbcn16 - Software Craftsmanship Barcelona 2016

@JavierCane@SergiGP

Welcome!

@GVico46

Contents

Context(not Bounded)

Legacy

Getting started

Pain Points

From PHP to Scala

1. Context (not Bounded)

App downloads

Messages sent monthly growth

Messages sent every day

30M

20 - 40%

~4M

Context (not Bounded) Where do we come

● Mobile first

◕ Internal REST API ● Startup with less than 2 years

◕ Externalize services (Parse, Kahuna…) ● Funding: $200M

◕ Ads in TV in USA and Turkey

2. Legacy

Legacy

● PHP ● No test ● New Features

Legacy REST API in PHP

Do I have new messages? No

And now?

And now?

And now?

And now?

No

No!!

NO!!

😑 🔫 💣

Legacy No test

● Rebuild a system without tests => 🦄💩💣💀

● Coupled system => Acceptance tests

◕ Learning what the system does

◕ Find existing weird behaviors

Background: Given there are test users: | user_object_id | user_name | user_token | | 19fd3160-8643-11e6-ae22-56b6b6499611 | seller | sellerToken | | 120291b2-8643-11e6-ae22-56b6b6499611 | buyer | buyerToken | And user "seller" has a product with: | id | objectId | | 120291b2-8643-11e6-ae22-56b6b6499611 | SuperProductId | Scenario: A user can get messages from another user associated to product Given user "seller" has a conversation related to product "SuperProductId" with user "buyer" When user "seller" asks for messages related to product "SuperProductId" from user "buyer" Then the response status code should be 200 And the response should be in JSON And the JSON should be valid according to the schema "messages.schema"

Acceptance test with Behat

Legacy Taking advantage of backwards compatibility

Leaving The Monolith thanks to #EventSourcing @ #scpna

Legacy New Features

● Product always want more features ● Negotiation:

◕ Archive conversations

◕ Mute interlocutor

◕ Stickers

3. Getting started

Getting started

● Why and how to switch to Scala ● Scala and Akka crash course ● Takeaways

Why and how to switch to Scala

We want a WhatsApp inside

Letgo

LOL

I’ve payed $22 Billion for WhatsApp

Getting started Why and how to switch to Scala

● Realtime (WebSockets) ● Akka ● Scale!

Why How

● Learning a lot ● External consultancy ● Akka :) ● Backwards Compatible

Scala quick start

Getting started Scala quick start

● Case classes ● Functional ● Optionals ● Futures ● OOP

class User { private $id; private $name; public function __construct(Uuid $id, string $name) { $this!→id = $id; $this!→name = $name; } public function id() : Uuid { return $this!→id; } public function name() : string

Case Class

{ return $this!→id; } public function name() : string { return $this!→name; } public function setId(Uuid $id) : self { return new static($id, $this!→name); } public function setName(string $name) : self { return new static($this!→id, $name); }}

Case Class

rafa.name = "Santi"val santi = rafa.copy(name = "Santi")println(santi.name) #$ Santi

val rafa = User(UUID.randomUUID(), "Rafa")println(rafa.name) #$ Rafa

case class User(id: UUID, name: String)

Case classes

Does not compile

Usage

Immutability

val users = List( User(UUID.randomUUID(), "Rafa"), User(UUID.randomUUID(), "Santi"), User(UUID.randomUUID(), "Jaime"), User(UUID.randomUUID(), "Diyan"))

Functional

Mutable state

val names = users.map(user %& user.name)val names = users.map(_.name)

List[String] names = new ArrayList();for (User user: users) { names.add(user.name)}

Procedural

Option

Option[A]

Some(x: A) None

def searchUser(id: UUID): Option[User] = { #$ …search user in database (blocking) Some(rafa)}

Option

searchUser(userId) match { case Some(user) %& #$ do stuff case None %& #$ user not found}

Usage (pattern matching)

Option usage (functional)

searchUser(userId) match { case Some(user) %& #$ do stuff case None %& #$ user not found}

searchUser(userId).map { userFound %& #$ do stuff}

Futures

def searchUser(id: UUID): Future[Option[User]] = { Future { Thread.sleep(1000) Some(rafa) } }

Futures usage

searchUser(userId).onComplete { case Success(Some(user)) %& #$ do stuff case Success(None) %& #$ user not found case Failure(exception) %& #$ future has crashed}searchUser(userId).map { case Some(user) %& #$ do stuff case None %& #$ user not found}

searchUser(userId).map(_.map(_.name))

trait UserRepository { def search(id: UUID): Future[Option[User]]}trait ConsoleLogger { def warning(message: String) = { println(message) }}

OOP - DIP

OOP - DIP

class MysqlUserRepository extends UserRepository with ConsoleLogger { def search(id: UUID): Future[Option[User]] = { #$ implementation warning("user not found") Future(Some(rafa)) }}

OOP - Companion object

object UserId { def random: UserId = { UserId(UUID.randomUUID()) }}case class UserId(id: UUID)

val userId = UserId.randomprintln(userId.id) case class User(id: UserId, name: String)

Usage

Akka (actor model)

Scala quick start Akka (actor model)

● Concept ● Introductory examples ● Chat actors architecture

Scala quick start Akka (actor model) - Concept

● Mailbox (1 each time) ● receive to handle incoming messages ● ActorRef ● Tell or ask methods to interact with the ActorRef ● Location transparency

final class ConnectionActor extends Actor { }

object ConnectionActor { def props: Props = Props(new ConnectionActor)}

Building our first actor

Instantiationval connection: ActorRef = context.actorOf(ConnectionActor.props)

object ConnectionActor { def props: Props = Props(new ConnectionActor)}

final class ConnectionActor extends Actor {

override def receive: Receive = { case PingQuery %& } }

Building our first actor

Instantiationval connection: ActorRef = context.actorOf(ConnectionActor.props)

final class ConnectionActor(webSocket: ActorRef) extends Actor { override def receive: Receive = { case PingQuery %& webSocket ! PongResponse } }

Tell (Fire & forget)

object ConnectionActor { def props(webSocket: ActorRef): Props = Props(new ConnectionActor(webSocket)) }

Building our first actor

Instantiationval connection: ActorRef = context.actorOf(ConnectionActor.props(webSocket))

case class ConnectionActorState( lastRequestSentAt: Option[DateTime]) { def requestSent: ConnectionActorState = copy(lastRequestSentAt = Some(DateTime.now))}

Dealing with state

case class ConnectionActorState( lastRequestSentAt: Option[DateTime]) { def requestSent: ConnectionActorState = copy(lastRequestSentAt = Some(DateTime.now))}

final class ConnectionActor(webSocket: ActorRef) extends Actor { var state = ConnectionActorState(lastRequestSentAt = None) override def receive: Receive = { case PingQuery(requestId) %& state = state.requestSent webSocket.actorRef ! PongResponse }

Dealing with state

State model

Akka: 1 message at a time (no race conditions)

final class ConnectionActor(webSocket: ActorRef) extends Actor { var state = ConnectionActorState(lastRequestSentAt = None) override def preStart(): Unit = { context.system.scheduler.schedule( initialDelay = 1.minute, interval = 1.minute, receiver = self, message = CheckWebSocketTimeout ) } override def receive: Receive = { case PingQuery(requestId) %& state = state.requestSent

Lifecycle

override def preStart(): Unit = { context.system.scheduler.schedule( initialDelay = 1.minute, interval = 1.minute, receiver = self, message = CheckWebSocketTimeout ) } override def receive: Receive = { case PingQuery(requestId) %& state = state.requestSent webSocket ! PongResponse() case CheckWebSocketTimeout %& if (state.hasBeenIdleFor(5.minutes)) { self ! PoisonPill } }}

Lifecycle

override def receive: Receive = { case PingQuery %& Future { Thread.sleep(1000) sender() ! PongResponse }}

Akka and Futures - SHIT HAPPENS

sender() could have changed

Be careful dealing with futures - sender()

override def receive: Receive = { case PingQuery %& Future { Thread.sleep(1000) PongResponse }.pipeTo(sender())}

sender() outside Future

Same happens with self

Chat actors architecture

TalkerS

ConversationSJ

TalkerJ

ConnectionS1 ConnectionJ1

Maintains consistency between 2 talkers : 1 conversation

Kill connections if shit happens

Chat actors architecture

TalkerS

ConversationSJ

ConnectionS1CS2CS3

Connection Supervisor

Talker Provider

Conversation Provider

Maintains consistency between N connections : 1 talker

“Singleton” actor

Non “singleton” actor

Takeaways

Scala quick start Takeaways

● Neophyte guide for scala ● Raúl Raja ● Akka Concurrency book ● Scala for the impatient ● Lightbend webinars ● Lightbend activator

4. Pain Points

Pain Points

● MaxScale ● Slick ● Deploy ● Dependency Injection ● Sync between chats

Chat protocol

SERVER TO CLIENT

Events

Commands

Queries

ACK

RESPONSE

Events

CLIENT TO SERVER

SERVER TO CLIENTCommands

typing_started

typing_stopped

interlocutor_typing_started

interlocutor_typing_stopped interlocutor_message_sent

interlocutor_reception_confirmed

interlocutor_read_confirmed

Events

Queries

Events

fetch_conversations

fetch_conversation_details

fetch_messages fetch_messages_newer_than_id

fetch_messages_older_than_id

confirm_reception

confirm_read

archive_conversations

unarchive_conversations

CLIENT TO SERVER

Chat protocol

authenticate

create_conversation

send_message

DB initial import

DB initial import

Legacy events

workerN

Legacy events

worker…

Legacy events

worker2

Scaling domain events workers

Legacy events

worker1

Auto scaling supervisor actor

SQS

Scaling domain events workers

Legacy events

worker1

Legacy events

worker2

Legacy events

worker…

Legacy events

workerN

Auto scaling supervisor actor

SQS

5. From PHP to Scala

From PHP to Scala

● Language community ● Composer vs SBT

◕ Semantic Versioning (scalaz, play…) ● Developer eXperience

◕ Not descriptive errors

◕ Scala and IntelliJ ● Learning Curve ● Loving and hating the compiler ● Another set of problems

Questions?

Thanks!Contact

@JavierCane@SergiGP@GVico46

Credits

● Presentation base template by SlidesCarnival ● Graphics generated using draw.io