erlang factory layered architecture - final

A Layered ArchitectureRobustness & Efficiency

dennis.docter@spilgames.comMarch 22nd, 2013

2

A legacy of growth

4

Social and casual gaming platform

50+ portals190+ countries

Casual games

Social games

Multi player games

Native games

0"

50"

100"

150"

200"

250"1/1/07"

5/1/07"

9/1/07"

1/1/08"

5/1/08"

9/1/08"

1/1/09"

5/1/09"

9/1/09"

1/1/10"

5/1/10"

9/1/10"

1/1/11"

5/1/11"

9/1/11"

1/1/12"

5/1/12"

0"

10"

20"

30"

40"

50"

60"

70"

80"

90"

1/1/08" 1/1/09" 1/1/10" 1/1/11" 1/1/12"

Traffic growth over 1600%

Tech dept. growth over 2000%

200+ million unique usersper month

7

XMLInterfacing

Business

logic

SQL

Auth

oriz

atio

n +

Auth

entic

atio

n

Caching

Json Interfacing + plain GET

Businesslogic

Auth

oriz

atio

n

SQL

SwiftAPI

CachingCaching

MemcacheCluster

Mysql DB

Intefacing

Businesslogic

Caching

CachingBusiness

logic

Intefacing

Businesslogic

Schema

Caching

CachingBusiness

logic

Businesslogic

Mysql DBMysql DB

Mysql DB

Mysql Write DB

MysqlReader

MysqlReaders

SQL

SQL

SQL

SchemaSchemaSchema

SQL

SQL

• Inconsistent  design  &  interfaces• Unclear  responsibili4es• Technical  debt  • Independent  teams  developing  

the  same  features.• 7  layers  of  caching  hell• Scaling  

Problems

• Inconsistent  design  &  interfaces• Unclear  responsibili4es• Technical  debt  • Independent  teams  developing  

the  same  features.• 7  layers  of  caching  hell• Scaling  

Problems

not a problem

:)

9

A new approach

11

• Flexible

• Predictable

• Scalable

New architecture expectations

Buzzword

CompliantTM

Choices made

• Simple• Clean  separa4on• No  horizontal  dependencies• Fault  tolerant• Distributed

12

13

• Caching• Persistence• Sharding

Layers Responsibilities

Interface

Service

Storage

Client • Integra4on• State

• Authen4ca4on• Aggrega4on• Valida4on

• Authoriza4on• Composi4on• Transforma4on

13

• Caching• Persistence• Sharding

Layers Responsibilities

Interface

Service

Storage

ClientS

tateless• Integra4on• State

• Authen4ca4on• Aggrega4on• Valida4on

• Authoriza4on• Composi4on• Transforma4on

14

Application• Small & Simple• Limited scope• Quick to deploy

Platform• Abstracts complexity• Low change rate• Long lifetime

Building blocks

Walkthrough

16

3rd PartyBackend

https post/user/get/

UserInterface

UserService Awards

Service

get(me)

1

1 FriendsService

1

count

UserService

2

get([friends])

{“auth”: {“token”: “UwAA_AWeff...”}}

Interfaces

list

ApplicationInterface

HighscoreService

1 list

2

/application/highscores/

17

Widgets

17

logo widget profile widget

search widget

navigation widget

recommendation widget

recent_games widget

new_games widget most_played widget

social_games widget

Widgets

18

profile

definition

logic

template

UserService

AwardService

html,[properties]

+

18

profile

definition

logic

template

UserService

AwardService

html,[properties]

+navigation

header

search

logo

home_page

http://www.agame.com/

...

...preprocess

wid

get p

latfo

rm

body

footer

new games

...

18

profile

definition

logic

template

UserService

AwardService

html,[properties]

+navigation

header

search

logo

Servicesuser

interface{“context”:{ “guid”: 62623423, “siteid” : 88, “locale” : “en_US”},”guid”: 324112535}

user service

profilestorage

check fetch filter modify

{ “gid”: “324112535”, “username” : “-fooba-”, “firstName” : “john” “avatarId” : 99142}

{ “gid”: “324112535”, “username” : “-fooba-”, “avatarId” : “http://agame.com/avatars/99142.png”}

get

Storage platform

• API  instead  of  SQL• Buckets  of  data  =  Key-­‐Value  with  Schema• Abstracts  storage

20

Shard 1 Shard 2 Shard ..

Storage Node 1

Bucket Bucket Bucket

Cache Storage Node 2

Bucket Bucket Bucket

Cache

Job handlers

21

AccountInterface

UserService

Profilebucket

interface-account.register{“guid”:7734}

AuditHandler

WelcomeHandler

sendemail

Audit Logservice-user.register

{“result”:”success”,”guid”:”73324”}

rabbitmq

Fitting it together

23

Widget  PlaGorm

Profilewidget

Gamewidget

Authin@erface

Gameservice

Awardinterface

Userinterface

Userservice

Awardservice

Storage  PlaGorm

Gamesbucket

UserAwardbucket

Profilesbucket

Awardservice

Userservice

inte

rface

serv

ice

stor

age

Load  balancer

GameAwardbucket

Profilesbucket

Protobuf

Protobuf

JSON

24

• Strictly  define  EVERYTHING  • input/output  format,  urls,  • rpc  func4ons,  types,  valida4on

• Strict  rules  on  changes• Everything  op4onal• Unless  you’re  really,  really  sure  (and  have  proof!)

• Generate  code  and  documenta4on• Interface  derived  from  specifica4on• Documenta4on  derived  from  specifica4on

How we do it

25

• XML:  Defini4on

• XSLT:  Transforma4on

• Piqi:  • Protobuf  &  Json  encoding/decoding

• HTTP  RPC(h`p://piqi.org/)

26

• Small  applica4ons  running  on  their  own  nodes

• Scale  up  where  necessary

• Many  of  them  on  the  same  machine

• Small  cluster  already  contains  hundreds  of  nodes

What we ended up with…

27

• Connec4ng  all  nodes  is  a  bad  idea:• Not  needed• Number  of  open  connec4ons

• Solu4on:  use  hidden  nodes• Each  node  specifies  the  nodes  it  needs.• How  does  it  scale?

Keeping it connected

28

Number of servers (~30 nodes / server)

Num

ber o

f TC

P co

nnec

tions

Hidden nodes vs. visible nodes

240$ 320$ 400$ 480$ 560$ 640$ 720$1770$7140$

16110$

28680$

44850$

64620$

87990$

114960$

145530$

0"

20000"

40000"

60000"

80000"

100000"

120000"

140000"

160000"

2" 4" 6" 8" 10" 12" 14" 16" 18"

Hidden" Visible"

29

• Each  applica4on  has  it’s  own  router

• Scans  &  Monitors  

• Balances  requests  across  nodes

• Parallelizes  requests  as  much  as  possible

• Applica4on  can  only  talk  to  what  it  has  defined

Router

30

• Use  proper  transports  (Erlang  RPC  vs.  HTTP)

• Yes  there  is  some  transport  overhead

• Typically  <  1ms  per  layer  (+network  latency)

• But  each  layer  has  predictable  response  4mes

• As  long  as  enough  downward  resources  are  available

But what about response times?

31

The old ...

32

.. and the new.

Closing Words

34

• Interface  layer  has  been  defined  and  largely  implemented

• ImplemenMng  the  service  &  storage  layer  step  by  step• Meanwhile  we  have  adapters  in  place

• Several  of  the  new  API’s  are  in  use  in  producMon

• First  portals  on  the  new  widget  plaGorm  will  be  deployed  in  the  coming  months

Where are we now..

35

• Deploying  to  mul4ple  datacenters  this  year

• Bring  elas4city  to  our  deployments

• Decommission  our  “old  architecture”  

• Erlang  game  plahorm

... And where we’re going

36

• An  architectural  vision  is  crucial• Integra4ng  into  a  mess  is  a  mess• Layers  with  clear  responsibili4es• Strict  communica4on• Package  small• Change  is  the  only  constant• Op4mize  for  predictability  first  

Lessons Learned

Questions

37

38

Thank you!