erlang factory layered architecture - final
DESCRIPTION
TRANSCRIPT
2
A legacy of growth
4
Social and casual gaming platform
50+ portals190+ countries
Casual games
Social games
Multi player games
Native games
0"
50"
100"
150"
200"
250"1/1/07"
5/1/07"
9/1/07"
1/1/08"
5/1/08"
9/1/08"
1/1/09"
5/1/09"
9/1/09"
1/1/10"
5/1/10"
9/1/10"
1/1/11"
5/1/11"
9/1/11"
1/1/12"
5/1/12"
0"
10"
20"
30"
40"
50"
60"
70"
80"
90"
1/1/08" 1/1/09" 1/1/10" 1/1/11" 1/1/12"
Traffic growth over 1600%
Tech dept. growth over 2000%
200+ million unique usersper month
7
XMLInterfacing
Business
logic
SQL
Auth
oriz
atio
n +
Auth
entic
atio
n
Caching
Json Interfacing + plain GET
Businesslogic
Auth
oriz
atio
n
SQL
SwiftAPI
CachingCaching
MemcacheCluster
Mysql DB
Intefacing
Businesslogic
Caching
CachingBusiness
logic
Intefacing
Businesslogic
Schema
Caching
CachingBusiness
logic
Businesslogic
Mysql DBMysql DB
Mysql DB
Mysql Write DB
MysqlReader
MysqlReaders
SQL
SQL
SQL
SchemaSchemaSchema
SQL
SQL
• Inconsistent design & interfaces• Unclear responsibili4es• Technical debt • Independent teams developing
the same features.• 7 layers of caching hell• Scaling
Problems
• Inconsistent design & interfaces• Unclear responsibili4es• Technical debt • Independent teams developing
the same features.• 7 layers of caching hell• Scaling
Problems
not a problem
:)
9
A new approach
11
• Flexible
• Predictable
• Scalable
New architecture expectations
Buzzword
CompliantTM
Choices made
• Simple• Clean separa4on• No horizontal dependencies• Fault tolerant• Distributed
12
13
• Caching• Persistence• Sharding
Layers Responsibilities
Interface
Service
Storage
Client • Integra4on• State
• Authen4ca4on• Aggrega4on• Valida4on
• Authoriza4on• Composi4on• Transforma4on
13
• Caching• Persistence• Sharding
Layers Responsibilities
Interface
Service
Storage
ClientS
tateless• Integra4on• State
• Authen4ca4on• Aggrega4on• Valida4on
• Authoriza4on• Composi4on• Transforma4on
14
Application• Small & Simple• Limited scope• Quick to deploy
Platform• Abstracts complexity• Low change rate• Long lifetime
Building blocks
Walkthrough
16
3rd PartyBackend
https post/user/get/
UserInterface
UserService Awards
Service
get(me)
1
1 FriendsService
1
count
UserService
2
get([friends])
{“auth”: {“token”: “UwAA_AWeff...”}}
Interfaces
list
ApplicationInterface
HighscoreService
1 list
2
/application/highscores/
17
Widgets
17
logo widget profile widget
search widget
navigation widget
recommendation widget
recent_games widget
new_games widget most_played widget
social_games widget
Widgets
18
profile
definition
logic
template
UserService
AwardService
html,[properties]
+
18
profile
definition
logic
template
UserService
AwardService
html,[properties]
+navigation
header
search
logo
home_page
http://www.agame.com/
...
...preprocess
wid
get p
latfo
rm
body
footer
new games
...
18
profile
definition
logic
template
UserService
AwardService
html,[properties]
+navigation
header
search
logo
Servicesuser
interface{“context”:{ “guid”: 62623423, “siteid” : 88, “locale” : “en_US”},”guid”: 324112535}
user service
profilestorage
check fetch filter modify
{ “gid”: “324112535”, “username” : “-fooba-”, “firstName” : “john” “avatarId” : 99142}
{ “gid”: “324112535”, “username” : “-fooba-”, “avatarId” : “http://agame.com/avatars/99142.png”}
get
Storage platform
• API instead of SQL• Buckets of data = Key-‐Value with Schema• Abstracts storage
20
Shard 1 Shard 2 Shard ..
Storage Node 1
Bucket Bucket Bucket
Cache Storage Node 2
Bucket Bucket Bucket
Cache
Job handlers
21
AccountInterface
UserService
Profilebucket
interface-account.register{“guid”:7734}
AuditHandler
WelcomeHandler
sendemail
Audit Logservice-user.register
{“result”:”success”,”guid”:”73324”}
rabbitmq
Fitting it together
23
Widget PlaGorm
Profilewidget
Gamewidget
Authin@erface
Gameservice
Awardinterface
Userinterface
Userservice
Awardservice
Storage PlaGorm
Gamesbucket
UserAwardbucket
Profilesbucket
Awardservice
Userservice
inte
rface
serv
ice
stor
age
Load balancer
GameAwardbucket
Profilesbucket
Protobuf
Protobuf
JSON
24
• Strictly define EVERYTHING • input/output format, urls, • rpc func4ons, types, valida4on
• Strict rules on changes• Everything op4onal• Unless you’re really, really sure (and have proof!)
• Generate code and documenta4on• Interface derived from specifica4on• Documenta4on derived from specifica4on
How we do it
25
• XML: Defini4on
• XSLT: Transforma4on
• Piqi: • Protobuf & Json encoding/decoding
• HTTP RPC(h`p://piqi.org/)
26
• Small applica4ons running on their own nodes
• Scale up where necessary
• Many of them on the same machine
• Small cluster already contains hundreds of nodes
What we ended up with…
27
• Connec4ng all nodes is a bad idea:• Not needed• Number of open connec4ons
• Solu4on: use hidden nodes• Each node specifies the nodes it needs.• How does it scale?
Keeping it connected
28
Number of servers (~30 nodes / server)
Num
ber o
f TC
P co
nnec
tions
Hidden nodes vs. visible nodes
240$ 320$ 400$ 480$ 560$ 640$ 720$1770$7140$
16110$
28680$
44850$
64620$
87990$
114960$
145530$
0"
20000"
40000"
60000"
80000"
100000"
120000"
140000"
160000"
2" 4" 6" 8" 10" 12" 14" 16" 18"
Hidden" Visible"
29
• Each applica4on has it’s own router
• Scans & Monitors
• Balances requests across nodes
• Parallelizes requests as much as possible
• Applica4on can only talk to what it has defined
Router
30
• Use proper transports (Erlang RPC vs. HTTP)
• Yes there is some transport overhead
• Typically < 1ms per layer (+network latency)
• But each layer has predictable response 4mes
• As long as enough downward resources are available
But what about response times?
31
The old ...
32
.. and the new.
Closing Words
34
• Interface layer has been defined and largely implemented
• ImplemenMng the service & storage layer step by step• Meanwhile we have adapters in place
• Several of the new API’s are in use in producMon
• First portals on the new widget plaGorm will be deployed in the coming months
Where are we now..
35
• Deploying to mul4ple datacenters this year
• Bring elas4city to our deployments
• Decommission our “old architecture”
• Erlang game plahorm
... And where we’re going
36
• An architectural vision is crucial• Integra4ng into a mess is a mess• Layers with clear responsibili4es• Strict communica4on• Package small• Change is the only constant• Op4mize for predictability first
Lessons Learned
Questions
37
38
Thank you!