Understanding and

improving games through

machine learning

Natasha Latysheva

Data Scientist

Jagex Games Studio

This talk:1. Introduction to Jagex2. Recap of main machine learning

fields + examples3. Fun ML in video games in general4. ML projects at Jagex Data Science

• Identifying player personas• Deep learning player lifecycle

sequences• Automatic bot and abuse

detection• Trading card deck analysis• Quest recommender systems

Jagex Game Studios

• Video game development company based in Cambridge

• Best known for Runescape, the world's largest free-to-play massively multiplayer online role-playing game (MMORPG). First released in 2001!

• Extremely community oriented game

• Old School Runescape was released in 2013

RuneScape

Main..

.. and

RuneScape

Old School

TF2 meets

Minecraft!

A quick machine

learning recap

Machine learning recap

Essentially learning patterns from data.

Branches of ML:

• 1. Supervised learning.

• Most familiar names, e.g. linear regression, logistic regression, decision trees, random forests, SVMs, many neural nets.

• Algorithm is presented with input data and explicit labels on that input. Learn signals associating input data with labels.

• Everything from good ole iris classification to detecting diseased patients from complex medical images.

Machine learning recap

2. Unsupervised learning.

• No explicitly labelled data, instead often seeking to find natural sub-groups of similar data points within the data.

• Describing the structure of the dataset.• Objective evaluation of accuracy often

difficult/not possible• Common methods:

• Clustering methods like k-means and hierarchical clustering

• Dimensionality reduction methods like principal components analysis

• Outlier/anomaly detection• Some NNs, e.g. generative adversarial

networks (GANs)

Larsen et al. 2016

Machine learning recap

3. Reinforcement learning.

• Agent must maximise cumulative reward by learning to optimally interact with an environment by choosing ‘good’ actions• E.g. robot must come up with correct limb

positions and joint torque to move efficiently/quickly

• Manufacturing program must maximise total units of product produced each day by controlling/tuning sub-processes

• Video game player agent must process the information from pixels on the screen to operate virtual keyboard + mouse to choose when to turn, apply brakes, gas, etc.

DuskDrive (OpenAI Universe, 2016)

Houthooft et al. 2017

Fun machine learning in

video games examples

1. DeepMind

• Classic Atari games• Learning from raw pixel feeds to

maximise future cumulative reward (i.e. score), no previous information encoded

• Ability to generalise skill playing across games

• In some cases, outperforming human players

• Deep Q-network approach• Combining deep learning of the

value function Q(s, a) with RL + experience replay

• Deep NN is Q(s, a) approximator, trained with Bellman updates Compulsory DeepMind demo:

https://www.youtube.com/watch?v=V1eYniJ0Rnk

Mnih et al., 2015

DeepMind handling quite complex games

DeepMind handling quite complex games

• 2. Hitman: NPC locomotion learning with RL

• 3. Hearthstone (collectible card game)

• 3. Hearthstone (collectible card game)

Bursztein, 2014

5. Predicting outcome: Domain experts (like casters) generate metrics (per turn + cumulative) that seem important to winning a game:• Mana advantage (delta mana spent in turn)• Board advantage (delta # minions)• Hand size advantage• Board mana advantage (delta total mana spent), etc.• No info on hero class or health

• 4. Candy Crush (King)

Recent and upcoming ML

projects at Jagex

1. Identifying player

personas

.

Identifying player personas

• Understand different archetypes of players

in Runescape

• Ultimately we would like to have a better

sense of our customer base

• Helps games devs

• Helps us understand the motivations, churn

behaviours, and value of our customers

.

Identifying player personas

• Draw inspiration from Bartel types

• Generated ~280 features to quantify our player behaviour

• Loosely based on Bartle Types (Killers, Achievers, Socializers, Explorers) • Quickly abandoned

• Example behavioural features• Chat Rank Score• Skill Heterogeneity• Forum Interactions • Economic Activity• Social Clustering Coefficients• Abilities Used During Play

.

Identifying player personas

• Data cleaning:

• Normalising by total and that year play time,

logging certain variables

• Dimensionality reduction using principal

components analysis

• Identifying optimal number of clusters using

silhouette plots or tSNE

Then… doing the clustering in PC space!

Life stages

• Differences in playtime both overall and recently are such a strong differentiating factor between

players that you first need to cluster on life cycle stages

Life stages

Name % Median Lifetime Playtime

Median YearlyPlaytime

Little ActivityShort Lifetime Playtime

30.7% 3h 3h

Little ActivityLong Lifetime Playtime

22.1% 590h 4h

ActiveMid Lifetime Playtime

22.2% 75h 35h

Extremely ActiveExtreme Lifetime Playtime

25% 2130h 320h

• The extremely dedicated players make up a huge proportion of total revenue.

Playstyles interacting with lifecycle stages

• Can observe which play styles emerge

or dominate at which lifecycle stages

• Can characterise transitions between

the different personas in consecutive

years

• Importantly: Quantify which types of

players tend to abandon the game

2. Deep learning player

lifecycle sequences

Player lifecycle sequencesMatt Project

• Players can undergo a variety of business relevant lifecycle

transitions.

Player lifecycle sequencesMatt Project

• Number of possible combinations, complexity of transitions,

scale of data, etc. leads to a difficult modelling problem

• Goal: We want to learn more about the player base and

how their lifecycle stages change.

• Two approaches:

• 1. Building transition probability matrices to examine

trends and project player states forward into the future

Matt Project

• Transition matrix approach does a surprisingly good job.

Deep learning player lifecycle sequences Matt Project

• 2. Building a recurrent neural network (specifically, an LSTM RNN) to learn the types of transitions that

happen and project these forward into the future

• Inherent memory to the system.

• But - difficult and time-consuming to train! GPUs help.

• On the plus side, can generate sequences seeded with multiple states. Also, in theory can learn

quite complex patterns.

• Models operational but need to tune hyperparameters and compare performance to transition

matrices.

[Colah’s blog]

3. Sentiment, bot and

abuse detection work

Matt Project

Sentiment, bot and abuse detection workMatt Project

• At the moment we:

• Help community team understand the popularity of

updates using sentiment detection

• Very simple model – Naïve Bayes classifier run over

in-game chat to identify emotional valence

• Detect instances of harassment

• Search through text with preset list of regular

expressions

• Try to learn more complex language models

• Using word2vec to identify words with similar

semantic meanings, identify harassment using

learned vector embeddings

Matt Project

[Saed Sayad]

In progress…

Training deep convolutional neural nets to identify bot-like mouse movements

[Adit Deshpande]

Common CNN architecture

[Also working to automate bot removal in other ways]

.

4. Trading card deck analysis

• Play a sequence of cards in a sequence to

create a quest for your legend

• Many cards have additional effects that buff

your character, provide weapons, alter

creature power, or grief / damage your

opponent

• Games span 5 chapters where each player

places up to 4 cards per chapter

• The gameplay is combo-oriented where card

play sequence is very important

Miro Project

.

Build card vectors with word2vec

• Learn card vectors using Word2Vec algorithm: • Given a card vector, predict the

remaining cards that were played by the player in that chapter.

This worked surprisingly well!

• Cards with similar effects clustered together

• Legend specific cards clustered together• Common cards that combo well with

Legend specific cards are located in a region of space close to the Legend

Miro Project

.

Build deck vectors with word2vec

• Tweak Word2Vec algorithm (more akin to Paragraph2Vec) to learn vector representation for custom decks

• Deck vectors naturally learned archetype clusters without being explicitly taught

• Decks from different legends that play by similar strategies tend to cluster in the same region of space

• Deck vectors can be used down-stream in other ML tasks • Matchmaking• assessing deck quality• recommending cards• predicting churn

Miro Project

• Card and deck vectors userful for game devs for exploring the space of decks real players use• Interactive web application for exploring decks by hovering over

decks, filtering by cards, player rank, and win rate

• Built clustering tool on top of deck vectors to allow devs to quickly identify common cards within a region of ‘deck space’• View summary statistics on the clusters (win rates, play rates,

cluster sizes)

• This is used to identify balance issues in game, identify how players are playing decks, and categorize player play-styles

Miro ProjectChronicle project conclusions

5. Quest recommender

system

Matt and Miro Project

• Ultimate goal: help guide players towards appropriate and interest-relevant content

• Starting point: recommend appropriate quests to players.

• Modelling using 2 approaches:

• 1. Restricted Boltzmann Machine (RBM) trained on past quest data

• 2. Matrix factorisation on past quest data with a REST API built to interface with game

Miro ProjectQuest recommender system

Matt and Miro Project

[deeplearning4j]

Acknowledgements

• Data Science Team!

• Miroslaw Horbal

• Matt Dixon

• Babis Georgiadis

• Katie Scott

• Joel Graham

• Ilian Mitev

• Analytics Team

Questions?