uk giaf: summer 2015

UPCOMING TALKSWhat to do with data from 1 billion smart devices in China?Peter Warman - CEO & Co-founder, Newzoo From Data Science to Data Impact: On many ways to segment your players & more Volodymyr (Vlad) Kazantsev - Head of Data Science, Product Madness Trends in game analytics: What’s happening (and why)?Heather Stark - Analyst, Kirnan Limited

35 2015 © All rights reserved to

From Data Science to Data Impact:

On many ways to segment your players

Volodymyr (Vlad) KazantsevHead of Data Science at Product Madness

jobs@productmadness.comvolodymyrk

36

What we do?jobs@productmadness.com volodymyr.kazantsev@productmadness.com volodymyrk

37

Heart of Vegas in (public) Numbers

iPad US - #13 top grossingiPhone US - #32 top grossingAndroid - #44 top grossing

US (games) AustraliaiPad - #1 top grossingiPhone - #1 top grossingAndroid -#3 top grossing

jobs@productmadness.com volodymyrk

38

Data Impact Team

Ad-hoc analytics; dashboards

Deep dive analysis; Predictive analytics

ETL, R&D

jobs@productmadness.com volodymyrk

39

Data Impact Team

Insights

Data

Science

Data

Engineering

7 people; 4 in London office

jobs@productmadness.com volodymyrk

Ad-hoc analytics; dashboards

Deep dive analysis; Predictive analytics

ETL, R&D

40

Ad-hoc analytics; dashboards

Deep dive analysis; Predictive analytics

ETL, R&D

Data Impact Team

Insights

Data

Science

Data

Engineering

7 people; 4 in London office

We Are Hiring !jobs@productmadness.com

jobs@productmadness.com volodymyrk

41

Technology Stack

ETL orchestration

Transformation& Aggregation

SQL

Data Products

Reports

Dashboards

+

42

few examples ..

A B

A/B TestsCustomer Lifetime Value

days

$ va

lue

Segmentation

group 1 group 2 group 3 group 4

43

Segmentation Basics1

44

Successful segmentation is the product of a detailed understanding of your market and will therefore take time

- Market Segmentation: How to Do it and Profit from it, 4th edition: Malcolm McDonald

45

BasicsCustomers have different needs and meansSegmentation can help to understand those differencesWhich can help to deliver on those needsAnd drive higher profitability

46

What is a Player Segment?

A segment is a group of customers who display similarities to each other...

Customers move in and out of segments over time

47

How many segments are there?

There is no one right way to segment (not should there be):

Many different approaches and techniques

Mix of art, science, common sense, experience and practical

knowledge

Depends on business needs and availability of data

Don’t aim to build one holistic model to meet all needs

48

Strategic Management

Product Development

Marketing Operations

Comments

Geography /Demographics

Loyalty / Length of Relationship

Behavioural

Needs-based

Value Based

49

Strategic Management

Product Development

Marketing Operations

Comments

Geography /Demographics

✭✭ ✭✭ ✭✭Separates players by country, city, city-district, distance from land-based casinos. By generational profile: boomers, Gen-Y, Gen-X.

Loyalty / Length of Relationship

✭✭✭ ✭ ✭✭✭ New players, on-boarding, engaged, lapsed, re-engaged, cross-promoted.

Behavioural ✭ ✭✭✭ ✭✭✭Based on identifying player’s behaviour characteristics that help to understand why customer behave the way they do

Needs-based ✭ ✭✭✭ ✭ Divide customers based on needs which are being fulfilled by playing Online Slots

Value Based ✭✭✭ ✭ ✭✭ Based on present and future value of the customer (RFM/CLV)

50

Land-based Slots Player segmentation

<10%

>50%

51

Segmentation = building a taxonomyAll Players

New(<28 days)

Established (>28d)

Payer Non Payer0-2 days 3-7d 8-27

<30 spins >30 … High V Med V Low V Engaged Casual…VIP Concierge

52

..and simplifying it daily useAll Players

New(<28 days)

Established (>28d)

Payer Non Payer0-2 days 3-7d 8-27

<30 spins >30 … High V Med V Low V Casual…

New High Value Med Value Low Value Engaged Casual

Engaged

53

How to profit from Segmentation?2

54

Clients of Segmentation

○ Strategy and Finance

○ Product development

○ Marketing operations

55

Strategy and FinanceThis Month

high-value med-value low-value super free-rider casual slotter recently lapsed

high-value 55.27% 30.06% 4.81% 5.54% 2.00% 2.32%

med-value 11.11% 42.50% 25.25% 10.92% 6.20% 4.02%

low-value 0.59% 7.72% 36.02% 30.59% 17.12% 7.96%

super free-rider 0.04% 0.30% 2.76% 70.50% 22.22% 4.18%

casual slotter 0.01% 0.10% 0.96% 8.98% 51.37% 38.58%

recently lapsed 0.05% 0.22% 1.01% 8.93% 13.00% n/a

New 0.01% 0.08% 0.67% 3.22% 31.05% 64.97%

This Month 0.15% 0.54% 2.13% 21.56% 31.22% 23.03%

Last Month 0.11% 0.43% 2.03% 21.09% 37.19% 27.20%

Last

Mon

th

56

Strategy and FinanceThis Month

high-value med-value low-value super free-rider casual slotter recently lapsed

high-value 6.80% -0.45% -1.66% -2.39% -1.07% -1.24%

med-value 3.09% 2.60% -2.81% -2.12% -0.60% -0.16%

low-value 0.11% 0.90% -1.63% 1.99% -0.54% -0.82%

super free-rider 0.01% 0.05% -0.05% -2.05% 2.58% -0.54%

casual slotter 0.00% 0.02% 0.05% -1.26% 2.71% -1.54%

recently lapsed -0.01% -0.05% -0.35% -4.21% -8.43% N/A

New 0.01% 0.04% 0.36% 1.59% 16.17% 1.21%

Manage transitions, not churn

Last

Mon

th

57

Product Development

New Slot Game Released

Coins Spent

58

Product Development

Geo: AustraliaValue: Low-valueBehaviour: Prefer Medium bet

New Slot Game Released

Coins Spent

59

Marketing

Objective Behavioral RFM/CLV geo/demographic Lifecycle

Sale Events

Monetization campaigns

Retention campaigns

Re-engagement

VIP management

60

How to actually do segmentation?3

61

Pillars of Successful Segmentation Project

Business knowledge

Data knowledge

Analytical skillsPeople

Process

Technology

ETL

Machine Learning

Business Intelligence

Product Integration

Marketing

Product

Data Services

62

Top-down approach to segmentation

1. Define objectives and therefore customer characteristicsa. dd

2. Choice method to split usersa. d

3. Prioritise segments to targeta. d

4. Operationalise segmentationa. s

5. ‘land’ the segmentation within the organization

63

Bottom-up approach360o player

view

Segmentation

Player transitions

Tailored interventions

Prioritisation and testing

● Build database to provide 360o view of the customer● Demographic, behavioural, payments, etc.● Add predictive attributes, such as conversion probability, churn risk, LTV, etc.

● Segment customers by desired attributes: more than one approach● Use robust statistical techniques for clustering or validation of empirical segmentation● Ensure segmentation is intuitive for the business and can be used across business functions

● Identify how players are moving from one segment to another (segment transition matrix)● Determine value levers and identify potential improvement ideas

● Create tailored interventions (CRM, push ..), aimed at moving customers to more valuable segments● Build predictive models to detect best offer and prevent undesirable transitions

● Prioritise interventions based on expected LTV uplift and ease of implementation● Test interventions through experimentation

64

How to actually do segmentation?

Just Look at Data Clustering Decision Trees

Player Attributes

de-correlate

Normalise Scale

65

de-correlate and normalise

Player 1 more similar to Player 2 ?Player 3 more similar to Player 2 ?

Weekly Play Summary

66

de-correlate and normaliseWeekly Play Summary

67

de-correlate and normalise

Player 1 more similar to Player 2 !

68

de-correlate and normalise

Player 1 more similar to Player 2 !

69

de-correlate and normalise

Player 1 more similar to Player 2, isn’t he?

70

de-correlate and normalise

Player 3 more similar to Player 2 !

71

What now?

K-meansHierarchical ClusteringDecision Trees.. and many more

72

Decision Tree for ClusteringAll Payers

500 (next month>$100): 4.7%10000 did not: 95.3%

Last_months_dollars <=$22 (next month>$100): 0.04%

5000 did not: 99%

Last_months_dollars >$2498 (next month>$100) > $100: 9%

5000 did not: 91%

Transactions <=10243 (next month>$100): 5.5%

4200 did not: 94.5%

Transactions > 10255 (next month>$100): 24%

800 did not: 76%

73

Decision Tree for ClusteringAll Payers

500 (next month>$100): 4.7%10000 did not: 95.3%

Last_months_dollars <=$22 (next month>$100): 0.04%

5000 did not: 99%

Last_months_dollars >$2498 (next month>$100) > $100: 9%

5000 did not: 91%

Transactions <=10243 (next month>$100): 5.5%

4200 did not: 94.5%

Transactions > 10255 (next month>$100): 24%

800 did not: 76%

Low Value

Medium ValueHigh Value

74

Segmentation at Product Madness4

75

Lifestage Segmentation

On-Boarding

Disengaged

Engaged

not played game

Churned

76

On-boarding segment

On-Boarding

Disengaged

Engaged

not played game

Churned

77

On-boarding segment

On-Boarding

Disengaged

Engaged

not played game

Churned

78

Lifestage Segmentation

On-Boarding

Disengaged

Engaged low riskhigh risk

low riskhigh risk

low riskhigh risk

not played game

churned

churned

Churned

79

Behavioural Segmentation

Average BetGifts per DayBonuses per DayMachine StickinessDays PlayedSpins per DayPreference for New Machines%% of spin on High-Roller machinesBig Win Stickinessetc.

Hierarchical Clustering

80

Behavioural Segmentation

81

Infrastructure

Data Warehouse

Segmentation Engine

CRM Email GAME Reporting Ad Hoc Analytics

Predictive Analytics

82

Segmentation for A/B tests

A B

83

Segmentation for A/B tests

A B

84

Bonferroni correction:

Bayesian Hierarchical Model

Combine stats with Market Intuition!

Adjustment for multiple testing

𝛼adjustted = 𝛼desired/M

85 2015 © All rights reserved to

Thank You!

jobs@productmadness.comvolodymyr.kazantsev@productmadness.com

volodymyrk

Kinran@

HASt

ark

What’s going on?

and why

Kinran@

HASt

ark

Market forces

Tech enablers

Frontier zones  

Failure prevention

Kinran@

HASt

ark

Kawanabe Kyosai

Kinran@

HASt

ark

Market forces

Kinran@

HASt

ark

Kinran@

HASt

ark

...[according to VB research] most mobile-first companies are trying to pay between $1 and $1.50 for users, but they are only getting quality users at multiples of those numbers...

August 12 2015 http://venturebeat.com/2015/08/12/this-service-tells-you-what-supercell-machine-zone-and-other-big-publishers-spend-on-user-acquisition/

Kinran@

HASt

ark

...[according to AppScotch] Machine Zone is currently spending somewhere around $12 per user with AdColony, InMobi, and Unity Ads, up to $20 per user with Vungle, and between $2 and $30 per user with Chartboost...

August 12 2015 http://venturebeat.com/2015/08/12/this-service-tells-you-what-supercell-machine-zone-and-other-big-publishers-spend-on-user-acquisition/

Kinran@

HASt

ark

LTV modelling

attributionprediction

Kinran@

HASt

ark

Engagement insight

play patternspend pattern

Kinran@

HASt

ark

Kinran@

HASt

ark

Tech enablers

Kinran@

HASt

ark

Kinran@

HASt

ark

Kevin Schmidt and Luis Vicente, Mind CandyPractical real-time approximations using Spark Streaming

hyperloglogs to count uniquesbloom filters to count revenuestream-summary for top-k

(metwally agrawal abbadi 2005)

from nucl.ai Data Science track (to be published)earlier version from huguk available nowhttp://www.slideshare.net/huguk/fast-perfect-practical-realtime-approximations-using-spark-streaming

Kinran@

HASt

ark

Image: re-work.co Deep Learning Summit Boston

Kinran@

HASt

ark

http://ufldl.stanford.edu/wiki/index.php/Visualizing_a_Trained_Autoencoder

Kinran@

HASt

ark

https://www.flickr.com/photos/darronb/6502473781

Kinran@

HASt

ark

Frontier zones

Kinran@

HASt

ark

Kinran@

HASt

ark

Miloš Milošević, Nordeus

Early Churn Prediction and Personalised Interventions In Top 11later detection is more accurate - but less usefultried many techniques – logistic regression good!cluster users based on first day gameplaycustomise messaging based on clustersincreased D1 retention (and downstream metrics)

from nucl.ai 2015 Data Science track (to be published)writeup available on gamasutra now:http://www.gamasutra.com/blogs/MilosMilosevic/20150811/250913/How_data_scientists_slashed_early_churn_in_Top_Eleven.php

Kinran@

HASt

ark

http://mba-suits.com/first-set-pink-white-grey-suit/http://www.ahipster.com/hipster-fashion-knitted-jumpers/hipster-clothes-wooly/

Kinran@

HASt

ark

Failure prevention

Kinran@

HASt

ark

Meta S. Brown Analytics failure and how to avoid it

Analytics programs fail....... because they lack a viable plan for success

Imperial College Data Science Institute 24 June 2015http://www.slideshare.net/metabrown/analytics-failure-how-to-avoid-it

Kinran@

HASt

ark

Meta S. Brown Analytics failure and how to avoid it

Analytics programs fail....... because they lack a viable plan for success

Define success, and who decides on it

Imperial College Data Science Institute 24 June 2015http://www.slideshare.net/metabrown/analytics-failure-how-to-avoid-it

Kinran@

HASt

ark

Meta S. Brown Analytics failure and how to avoid it

Start with a business problema small oneunderstand the business problem really well

As you scale up – pay attention to process replicable! replicable! replicable!

Imperial College Data Science Institute 24 June 2015http://www.slideshare.net/metabrown/analytics-failure-how-to-avoid-it

Kinran@

HASt

ark

Meta S. Brown Analytics failure and how to avoid it

Use only as much data as you need to

The best use case for Big is personalisation

Imperial College Data Science Institute 24 June 2015http://www.slideshare.net/metabrown/analytics-failure-how-to-avoid-it

Kinran@

HASt

ark

What next?

JOIN IN THE CONVERSATION PARTICIPATE IN THE NEXT GIAF

Analytics for Games www.deltadna.com/giaf

events@deltadna.com

#UKGIAF