london scala meetup - omnia

WELCOME TO THE LONDON SCALA UG

MEET UP @

WILLIAM HILL

Luis Ángel Vicente Sánchez Software Engineer @ Innovation Team

OMNIA DISTRIBUTED & REACTIVE PLATFORM FOR DATA MANAGEMENT

Site https://developer.williamhill.com

Twi$er https://twitter.com/lvicentesanchez

WHY BUILDING A DATA MANAGEMENT PLATFORM?

•  Lots of information flowing in the system.

•  What if we could make sense of all of it...?

•  ... and on real-time?

•  We would be in a great position to innovate.

WHY BUILDING A DATA MANAGEMENT PLATFORM?

BUT…

BUT… AVAILABILITY

BUT…

BUT… SCALABILITY

BUT…

BUT… FAULT TOLERANCE

REQUIREMENTS

Availability

Faulttolerance

Scalability

REQUIREMENTS

REACTIVE MANIFESTO

REACTIVE MANIFESTO Respondina+melymanner


Stayresponsiveunderdifferentworkloads


Stayresponsiveunderdifferentworkloads Stayresponsiveinfaceoffailures


Stayresponsiveunderdifferentworkloads Stayresponsiveinfaceoffailures

Loosecoupling,isola+on,loca+ontransparency,anderrorhandlingdelega+on

•  It’s a lambda architecture.

OMNIA

•  It’s a lambda architecture.

OMNIA

InputC

hron

osD

ata

Sour

ce

NeoCortexSpeed Layer

FatesBatch Layer

Her

mes

Serv

ing

Laye

r Output

CHRONOS

•  Collects data from multiple data sources.

•  Initially a PUSH system…

•  ... but that introduces dependencies on 3rd parties.

•  Evolved to a PUSH-PULL system.

CHRONOS

CHRONOS

Chr

onos

Dat

a So

urce

TCP

HTTP

WS

Bets

Depo

sits

Prices

CHRONOS

JMS

SSE

Chr

onos

Dat

a So

urce

TCP

HTTP

WS

…HTTP

Poll

Bets

Depo

sits

Prices

CHRONOS

JMS

SSE

Chr

onos

Dat

a So

urce

TCP

HTTP

WS

…HTTP

Poll

Streams

Bets

Depo

sits

Prices

CHRONOS STREAM

CHRONOS

Adapter

STREAM

CHRONOS

Adapter

STREAM

PUSH:•  HTTP•  TCP•  WEBSOCKETS

PULL:

•  HTTPPOLL•  JMS•  SSE

CHRONOS

Adapter Converter

STREAM


PULL:


CHRONOS

Adapter Converter

STREAM


PULL:


INCIDENT: { type: “bet”, version: “1.0”, time: “2015-09-03 06:00:10”, acquisitionTime: “2015-09-03 06:00:06”, source: “BetSystem”, payload: {…. Any valid JSON …} }

CHRONOS

Adapter Converter

STREAM


PULL:


INCIDENT: { type: “bet”, version: “1.0”, time: “2015-09-03 06:00:10”, acquisitionTime: “2015-09-03 06:00:06”, source: “BetSystem”, payload: {…. Any valid JSON …} } O

bservable[Incide

nt]

CHRONOS

Adapter Converter Persistor

STREAM


PULL:


INCIDENT: { type: “bet”, version: “1.0”, time: “2015-09-03 06:00:10”, acquisitionTime: “2015-09-03 06:00:06”, source: “BetSystem”, payload: {…. Any valid JSON …} } O

bservable[Incide

nt]

CHRONOS

CHRONOS

Chronos 1 Chronos 2 Chronos 3

Chronos N

….…

CHRONOS

Chronos 1(SSE, Bets placed)

(SSE, Twitter)

Chronos 2(JMS, Deposits)

Chronos 3(HTTP, Events)

Chronos N

….…

CHRONOS

Chronos 1(SSE, Bets placed)

(SSE, Twitter)

Chronos 3(HTTP, Events)

Chronos N(SSE, Twitter)

….…

Chronos 2(JMS, Deposits)

(SSE, Bets placed)

FATES

Views

Jobs

Fates

•  Long term memory.

•  Records all incidents as timelines, i.e. customer bets, customer payments, everything that happens in a match…

•  … or any combination of those.

FATES

FATES

Bets DepositsEvents Session

FatesBatch Layer

FATES

Timelines


FatesBatch Layer

FATES

Customer: 123

Login

Deposit

Bet placed

…

LogoutTimelines


FatesBatch Layer

FATES

Customer: 123

Login

Deposit

Bet placed

…

Logout

Event: 78

Started

Bet placed

Penalty

…

GoalTimelines


FatesBatch Layer

•  Creates views using the timelines and/or other views.

•  These views are created using: – Timeline analysis – Logical reasoning… – ... or any other ML technique

FATES

FATES

LogicalReasoning• DeducDon•  InducDon• AbducDon

Timelineanalysis• Trends• Cycles• Seasonality

OtherML• ClassificaDon• Clustering• PredicDons

•  Timelines and views persisted to Cassandra.

•  Column family name: <category>_tl or <category>_vw

•  Column family key: ((entityId, date), timestamp)

FATES

NEOCORTEX

NEOCORTEX

Bets

Depo

sits

Events

Session

Micro Services

Outpu

t

•  Short term memory

•  Built on top of Spark: performant, distributed, real-time computations.

•  Built on top of abstractions that hide low level details: Timeline, View, Stream, Sink, Monoid, Group, Ring…

NEOCORTEX

NEOCORTEX

HERMES

•  Full duplex serving layer.

•  Can be queried from any backend…

•  ... and we provide a JS library to provide efficient data access from the browser. Inspired by Falcor and GraphML

HERMES

HERMES

B2CBrowser

B2BLoad

bal

ance

r

Push Server

DistributedCache Push

Server

Push Server

…

TCP

WS

HTTP

JS A

PI WH AppsC

ache

Cac

heApps

HERMES

HermesDistributedCache

HermesNode

LocalCache

SubscripDo

nManager

ClientM

anager

Authen

DcaD

onHandler

Dispatcher

HTTP

WS

TCP

Browser

Herm

esJS

WHAp

ps

INFRASTRUCTURE

INFRASTRUCTURE

Omnia

Docker

Marathon

Mesos

Node Node Node Node Node

USE CASE: BET SUGGESTIONS

•  To provide customers with bet suggestions to make their experience more relevant/personal

•  Offered to new and existing customers.

•  Hot bets and personalized suggestions.


USE CASE: GAMIFICATION

•  To provide customers with a more engaging experience by gamifying parts of the customer journey.

•  Appeal to attributes like status, mastery, belonging.

•  Achievements and rewards, leveling up, leaderboard, …


https://careers.williamhillplc.com

QUESTIONS

74

Twitter: @WHTechjobs