mongo db in gaming industry

MongoDB in KIXEYE Analytics Ben Goswami

©2011 Kixeye, Private and Confidential

Game Play

BP Video clips ( http://www.youtube.com/watch?v=uUHzppVn8WM)

http://www.youtube.com/watch?v=uUHzppVn8WM


Agenda

• Data scaling problems at KIXEYE analytics and what technologies we use

• Example application: Cheater Detection Engine

• How MongoDB helped us to solve it


Game Play

BP Attack Video (https://www.youtube.com/watch?v=qud_-18D2MU)

https://www.youtube.com/watch?v=qud_-18D2MU


Game Play and Data

• Games generate lots of click data

• Click = evento time of the evento what happened before or aftero how the events are correlated

_id" : ObjectId("506e23001d4524122c00004b"), "ip" : "10.54.72.206", "dt" :{ "_id" : ObjectId("506e23001d4524dd2b0000bd"), "ip" : "10.54.72.206", "&attacker_level=42&defender=0&defender_ext=salvage&defender_level=25&attack_type=PVE&attacking_units=%7b%224071%22%3a%5b5%2c3%2c0%5d%7d&defending_units=%7b%221040%22%3a%5b4%2c0%2c4%5d%2c%221060%22%3a%5b1%2c0%2c1%5d%7d&attacking_shipconfig=%7b%22ships%22%3a%5b%7b%22armors%22%3a%5b333%2c333%2c333%2c333%5d%2c%22weapons%22%3a%5b134%2c134%2c193%2c134%2c134%2c134%2c134%2c134%5d%2c%22specials%22%3a%5b542%2c561%2c571%2c501%5d%2c%22hullID%22%3a4071%2c%22tacticalModules%22%3a%5b%5d%2c%22actives%22%3a%7b%22hp%22%3a9317.68457%2c%22rank%22%3a4%2c%22fltp%22%3a5%2c%22bid%22%3a5483479%2c%22f%22%3a0%2c%22id%22%3a110%2c%22flt%22%3a5%2c%22vxp%22%3a10.858466%7d%7d%2c%7b%22armors%22%3a%5b333%2c333%2c333%2c333.. "stage" : "new" }

Attack: { attacker : { user:123 player_level: 31 fleet: { 1 : { Weapons: {

name:“Ripper Cannon IV” damage:306

accuracy:55 }

armors:{}…

} 2: { } } } defender: { } attack_time: 100354563 attack_duration:290 sector:7…}


How much data?

•Multi TBs / month (4-5x in next 6 months)

•Critical : Volume reads with speed

•< 1% data has actionable intelligence

Search it Fast stupid!!

MongoDB


MongoDB in KIXEYE

•Why:

– fast read + writes for high volume – Document based architecture, schema less– Indexing , auto failover, auto sharding

•Where:– Games , KIXEYE Platform– Analytics, Logging


Data technologies at Analytics

Technology Data Volume Query Type I/O speed

MySQL Small< 300 GB

Adhoc + Pre defined

Read fast, only Aggregated data

Mongo Mid< 2 TB

Predefined Read + Write speed fast

Hadoop(Hive+HDFS)

X-Large 2 TB+

Adhoc Read slowwrites: only insert no update


An Example Application

Cheater Detection engine for games: How mongoDB helped us

A simple backyard at start in BYM


Currency is time or $ or both


Sample record

• 4kb nested jason block

• base = {base_name: “sweet_mango_db”

resources {

1: { name: "storage silo"

level: 5, mode:“built" time:123456902

2: { …

..}

...}

...} ..}


How people cheat?

• Each upgrade or repair takes pre defined time

• if time+payment < predefined time = cheater

• victims chain create snowball effect


..and why they cheat...


• 16Mil * 50 * 10 * 3 *....

• multiple time based snapshots

• 0.01 % cheaters • Detect < 5 mins• That's searching

through lot of data fast

Data volume for cheater detection


.. and boss told

• Catch and auto ban cheaters

• Don't label a good guy a cheater


..team was ready for the challenge

• Speed+Accuracy of application

• Technology evalauation


MongoDB fits the requirement

• read writes are fast because memory mapped files compared to mysql+memcache or hbase

• Unstructured data => need schema less

• indexing on a nested data element


.. more on why mongo

• Auto purging thru TTL

• Auto sharding during traffic spikes

• Binary data support, fast uncompression

• Capped collection: writes faster


Development decision points

• Old map-reduce (prior to 2.2) vs New Aggregation framework

• Global write locks removal in 2.2

• Lazy and Non lazy inserts


Result

• 100+ cheaters caught per hour

• Better game balancing

• $$ saved


Wrap up : A good partnership

•Good partnership with KIXEYE and 10gen•Consulting/Training and MMS service was very helpful

•More mongoDB usages in future at KIXEYE


{Status: “The End”}

•{ next_action: “Q&A” }

• Like this jason? meet us outside – KIXEYE recruiting{ Company : “KIXEYE”,

“Engineers/Total _employee” : “65%”,

Culture : [“Fun”, “Innovative”, “Passionate”]

Analytics_engg : { technology : “mongo, hadoop , mysql”,

tag : “big data, geek” }

}

•Love numbers and big data? We have a job for you at analytics engg at KIXEYE. Talk to our recruiting team

Extra slides


Game Play

BP Attack Video (https://www.youtube.com/watch?v=qud_-18D2MU)

Usage of data

• Data mining and aggregation to

oOptimize User AcquisitionoOptimize game analyzing game play

behavior and

Data

Algo + Optimize

r

Feedback


Mongo - pain points

• Diskspace handling

• No sql syntax support

• Capped collection - no deletes allowed

• No good client GUI based tool

Damaged building is useless


Data related to player

• building has 5-10 levels

• building can have 3 states [built, damaged, repair in progress)

• an average base can have 50+ buildings

• 16 million such bases

• 45k people at any second [i.e base gets updated]

• Each level up or repair has time delay. People pay to reduce delay


Map-reduce in cheater detection

function mapf()

{

emit(this.userid, {userid:this.userid, total_time:this.length, bldg_id:1} ;

}

function reducef(key, values)

{

var r = {userid:key, total_time:0, count:0, avg_time:0};

values.forEach(function(v)

{r.total_time += v.total_time; }); return r;

}

function finalizef(key, value) { ... }


The End

•Q&A

•Love numbers and big data? We have a job for you at analytics engg at KIXEYE. Talk to our recruiting team


To scale data handling

• Write speed for large volume data

• Read speed + volume [aggregate+find]

• MongoDB provides good balance compared to other sql+nosql solutions for certain use cases

mysql BI Mongo Clustermongo

ETL

mySQL Mongo Hadoop mySQL/mongo

non Real time >= 1 hr real time 5 mins

Games+Platforms devices

Dashboards Hive sql query API Scheduler,

Data access tools

API

Admin console

Analytics Components - Birds' Eye View

mongo db in gaming industry

Technology

saved2011 kixeye

faster2011 kixeye

logging2011 kixeye

cheater2011 kixeye

kixeye2011 kixeye

tool2011 kixeye

correlated2011 kixeye

both2011 kixeye