a large-scale, longitudinal study of user profiles in world of warcraft

@_jon_bell_WiC May 14, 2013

A Large-Scale, Longitudinal Study of User Profiles in

World of WarcraftJonathan Bell, Swapneel Sheth, Gail Kaiser

Columbia University, New York, NY USA

PROGRAMMING SYSTEMS LAB

COLUMBIA UNIVERSITY

http://www.psl.cs.columbia.edu/

enable(vt):to make possible, practical, or easy

1

@_jon_bell_WiC May 14, 2013

Motivation

2

@_jon_bell_WiC May 14, 2013

Motivation

• We’re a group of Software Engineers.

2

@_jon_bell_WiC May 14, 2013

Motivation

• We’re a group of Software Engineers.

• Recently studying Gamification of Software Development

2

@_jon_bell_WiC May 14, 2013

Motivation

• We’re a group of Software Engineers.

• Recently studying Gamification of Software Development

• What principles can we carry over from game design to apply it to non-game contexts?

2

@_jon_bell_WiC May 14, 2013

Motivation

• We’re a group of Software Engineers.

• Recently studying Gamification of Software Development

• What principles can we carry over from game design to apply it to non-game contexts?

• Why not look to real games?

2

@_jon_bell_WiC May 14, 2013

Game Studies

3

@_jon_bell_WiC May 14, 2013

Game Studies

• Qualitative Research:

• 100’s-1000’s of samples [Debeuvais; Yee]

• Personalities, demographics, etc

3

@_jon_bell_WiC May 14, 2013

Game Studies

• Qualitative Research:

• 100’s-1000’s of samples [Debeuvais; Yee]

• Personalities, demographics, etc

• Quantitative Research:

• 10,000’s-100,000’s of samples [Duchenaut; Lewis]

• In-game demographics, etc

3

@_jon_bell_WiC May 14, 20134

@_jon_bell_WiC May 14, 2013

WoW: Quantitative Data

5

@_jon_bell_WiC May 14, 2013

WoW: Quantitative Data

• In-game “Demographics”

• Guild, Class, Race, Level

5

@_jon_bell_WiC May 14, 2013

WoW: Quantitative Data

• In-game “Demographics”

• Guild, Class, Race, Level

• Quests

5

@_jon_bell_WiC May 14, 2013

WoW: Quantitative Data

• In-game “Demographics”

• Guild, Class, Race, Level

• Quests

• Achievements

5

@_jon_bell_WiC May 14, 2013

WoW: Quantitative Data

• In-game “Demographics”

• Guild, Class, Race, Level

• Quests

• Achievements

• Professions

5

@_jon_bell_WiC May 14, 2013

WoW: Quantitative Data

• In-game “Demographics”

• Guild, Class, Race, Level

• Quests

• Achievements

• Professions

• Raids

5

@_jon_bell_WiC May 14, 2013

WoW: Quantitative Data

• In-game “Demographics”

• Guild, Class, Race, Level

• Quests

• Achievements

• Professions

• Raids

• Companions

5

@_jon_bell_WiC May 14, 2013

WoW: Quantitative Data

• In-game “Demographics”

• Guild, Class, Race, Level

• Quests

• Achievements

• Professions

• Raids

• Companions

• Mounts

5

@_jon_bell_WiC May 14, 2013

WoW: Quantitative Data

• In-game “Demographics”

• Guild, Class, Race, Level

• Quests

• Achievements

• Professions

• Raids

• Companions

• Mounts

• Titles

5

@_jon_bell_WiC May 14, 2013

WoW: Quantitative Data

• In-game “Demographics”

• Guild, Class, Race, Level

• Quests

• Achievements

• Professions

• Raids

• Companions

• Mounts

• Titles

• Reputation5

@_jon_bell_WiC May 14, 20136

@_jon_bell_WiC May 14, 20136

@_jon_bell_WiC May 14, 20136

@_jon_bell_WiC May 14, 20136

@_jon_bell_WiC May 14, 20136

@_jon_bell_WiC May 14, 20136

@_jon_bell_WiC May 14, 2013

Methodology

7

@_jon_bell_WiC May 14, 2013

Accessing the Data

• Blizzard’s API allows you to retrieve this per-character

• No option to list all characters

• How do we discover the characters?

8

@_jon_bell_WiC May 14, 2013

Character Discovery

9

@_jon_bell_WiC May 14, 2013

Character Discovery

• Blizzard does list characters by guild

9

@_jon_bell_WiC May 14, 2013

Character Discovery

• Blizzard does list characters by guild

• Several popular websites have listings of guilds and characters [e.g. WarcraftRealms]

9

@_jon_bell_WiC May 14, 2013

Character Discovery

• Blizzard does list characters by guild

• Several popular websites have listings of guilds and characters [e.g. WarcraftRealms]

• Solution: Find all guilds belonging to these characters. And find all members of all identified guilds

9

@_jon_bell_WiC May 14, 2013

Fetching the Data

• Blizzard API is rate-limited per-IP

• Solution: use a cluster of 60+ IPs to crawl the API

10

@_jon_bell_WiC May 14, 2013

Fetching the Data

• Blizzard API is rate-limited per-IP

• Solution: use a cluster of 60+ IPs to crawl the API

MySQL

Crawler Nodes (60+)

ManagementConsole (Normal PC)

Retrieve "todo-list"

Store resultsStarts Nodes

Runs Analysis

10

@_jon_bell_WiC May 14, 2013

0"

200000"

400000"

600000"

800000"

1000000"

1200000"

1400000"

9/29/2011"

9/30/2011"

10/1/2011"

10/2/2011"

10/3/2011"

10/4/2011"

10/5/2011"

10/6/2011"

10/7/2011"

10/8/2011"

10/9/2011"

10/10/2011"

10/11/2011"

10/12/2011"

10/13/2011"

10/14/2011"

Characters)

Date)

Crawler)Performance)

11

@_jon_bell_WiC May 14, 2013

Raw Data

• Found approximately 12 million characters with basic information

• Complete data for half of these

Basic information

12

@_jon_bell_WiC May 14, 2013

Other MMORPG Study Sizes

1"

10"

100"

1,000"

10,000"

100,000"

1,000,000"

10,000,000"

100,000,000"

WoW:"Debeauvais"

2011"

EverQuest:"Williams"[2008]"

Everquest,"Ul<ma,"

Camelot,"SW"Galaxies:"Yee"

[2006]"

WoW:"Harrison"[2011]"

WoW:"Lewis"[2010]"

WoW:"Bell"[2013]"(1)"

WoW:"Bell"[2013]"(2)"

Sample'Size'(Log'Scale)'

Sample'Size'of'MMORPG'Studies'

13

@_jon_bell_WiC May 14, 2013

Other MMORPG Study Sizes

1"

10"

100"

1,000"

10,000"

100,000"

1,000,000"

10,000,000"

100,000,000"

WoW:"Debeauvais"

2011"

EverQuest:"Williams"[2008]"

Everquest,"Ul<ma,"

Camelot,"SW"Galaxies:"Yee"

[2006]"

WoW:"Harrison"[2011]"

WoW:"Lewis"[2010]"

WoW:"Bell"[2013]"(1)"

WoW:"Bell"[2013]"(2)"

Sample'Size'(Log'Scale)'

Sample'Size'of'MMORPG'Studies'

Closest: 136,047

13

@_jon_bell_WiC May 14, 2013

Other MMORPG Study Sizes

1"

10"

100"

1,000"

10,000"

100,000"

1,000,000"

10,000,000"

100,000,000"

WoW:"Debeauvais"

2011"

EverQuest:"Williams"[2008]"

Everquest,"Ul<ma,"

Camelot,"SW"Galaxies:"Yee"

[2006]"

WoW:"Harrison"[2011]"

WoW:"Lewis"[2010]"

WoW:"Bell"[2013]"(1)"

WoW:"Bell"[2013]"(2)"

Sample'Size'(Log'Scale)'

Sample'Size'of'MMORPG'Studies'

Closest: 136,047

Us: 6 (12) Million

13

@_jon_bell_WiC May 14, 2013

Research Questions

14

@_jon_bell_WiC May 14, 2013

Research Questions1. Can we cluster characters into profiles

based on the type of goals that they complete in game?

14

@_jon_bell_WiC May 14, 2013

Research Questions1. Can we cluster characters into profiles

based on the type of goals that they complete in game?

2. Do characters' play profile evolve as they level?

14

@_jon_bell_WiC May 14, 2013

Research Questions1. Can we cluster characters into profiles

based on the type of goals that they complete in game?

2. Do characters' play profile evolve as they level?

3. Do characters with different play profiles play through the game content at different rates?

14

@_jon_bell_WiC May 14, 2013

RQ1: Creating Player Profiles

15

@_jon_bell_WiC May 14, 2013

RQ1: Creating Player Profiles

• Within WoW, players complete Achievements

15

@_jon_bell_WiC May 14, 2013

RQ1: Creating Player Profiles

• Within WoW, players complete Achievements

• Our dataset contains each achievement each player earns, with a timestamp

15

@_jon_bell_WiC May 14, 2013

RQ1: Creating Player Profiles

• Within WoW, players complete Achievements

• Our dataset contains each achievement each player earns, with a timestamp

• Profile players based on the types of achievements they complete

15

@_jon_bell_WiC May 14, 2013

Categorizing Achievements

• Draw on motivation categorizations described by Yee

Achievement Social ImmersionAdvancement Socializing Discovery

Mechanics Relationship Role-Playing

Competition Teamwork Customization

Escapism

Categorization of play motivations, Yee

16

@_jon_bell_WiC May 14, 2013

Categorizing Achievements

17

@_jon_bell_WiC May 14, 2013

Categorizing Achievements

• 1000 Conquest Points

17

@_jon_bell_WiC May 14, 2013

Categorizing Achievements

• 1000 Conquest Points

• Earn 1000 conquest points -> Achievement

17

@_jon_bell_WiC May 14, 2013

Categorizing Achievements

• 1000 Conquest Points

• Earn 1000 conquest points -> Achievement

• Archavon the Stone Watcher

17

@_jon_bell_WiC May 14, 2013

Categorizing Achievements

• 1000 Conquest Points

• Earn 1000 conquest points -> Achievement

• Archavon the Stone Watcher

• Defeat Archavon the Stone Watcher in 10 player mode -> Social

17

@_jon_bell_WiC May 14, 2013

Categorizing Achievements

• 1000 Conquest Points

• Earn 1000 conquest points -> Achievement

• Archavon the Stone Watcher

• Defeat Archavon the Stone Watcher in 10 player mode -> Social

• It’s Happy Hour Somewhere

17

@_jon_bell_WiC May 14, 2013

Categorizing Achievements

• 1000 Conquest Points

• Earn 1000 conquest points -> Achievement

• Archavon the Stone Watcher

• Defeat Archavon the Stone Watcher in 10 player mode -> Social

• It’s Happy Hour Somewhere

• Drink 25 different types of beverages -> Immersion

17

@_jon_bell_WiC May 14, 2013

Categorizing Achievements

• 1000 Conquest Points

• Earn 1000 conquest points -> Achievement

• Archavon the Stone Watcher

• Defeat Archavon the Stone Watcher in 10 player mode -> Social

• It’s Happy Hour Somewhere

• Drink 25 different types of beverages -> Immersion

• Arrested Development

17

@_jon_bell_WiC May 14, 2013

Categorizing Achievements

• 1000 Conquest Points

• Earn 1000 conquest points -> Achievement

• Archavon the Stone Watcher

• Defeat Archavon the Stone Watcher in 10 player mode -> Social

• It’s Happy Hour Somewhere

• Drink 25 different types of beverages -> Immersion

• Arrested Development

• Allow all three of Corla’s zealots to evolve, then defeat Corla after slaying the evolved zealots in Blackrock Caverns on Heroic Difficulty -> All three

17

@_jon_bell_WiC May 14, 2013

Four Distinct Player Profiles

Generalists Achievement Loners Socialists

Size (Millions of players)

Social

Immersion

Achievement

1.6 1.4 1 1

12% ± 6 7% ± 8 1% ± 3 28% ± 6

38% ± 6 20% ± 7 38% ± 29 31% ± 5

66% ± 4 87% ± 6 35% ± 24 61% ± 4

18

@_jon_bell_WiC May 14, 2013

RQ2: Achievements per level

0"

20"

40"

60"

80"

100"

120"

140"

160"

180"

0(9"

10(19"

20(29"

30(39"

40(49"

50(59"

60(69"

70(79"

80(84"

85"

Number'of'Achievements2

Level2

Achievement"

Immersion"

Social"

19

@_jon_bell_WiC May 14, 2013

RQ2: Achievements per level

0"

20"

40"

60"

80"

100"

120"

140"

160"

180"

0(9"

10(19"

20(29"

30(39"

40(49"

50(59"

60(69"

70(79"

80(84"

85"

Number'of'Achievements2

Level2

Achievement"

Immersion"

Social"

Mostly get achievements after finishing most game content

19

@_jon_bell_WiC May 14, 2013

RQ2: Achievements per level

0"

20"

40"

60"

80"

100"

120"

140"

160"

180"

0(9"

10(19"

20(29"

30(39"

40(49"

50(59"

60(69"

70(79"

80(84"

85"

Number'of'Achievements2

Level2

Achievement"

Immersion"

Social"

Very few do social achievements early on

19

@_jon_bell_WiC May 14, 2013

RQ2: Achievements per level

0"

20"

40"

60"

80"

100"

120"

140"

160"

180"

0(9"

10(19"

20(29"

30(39"

40(49"

50(59"

60(69"

70(79"

80(84"

85"

Number'of'Achievements2

Level2

Achievement"

Immersion"

Social"

Had to be social to progress?

19

@_jon_bell_WiC May 14, 2013

RQ3: Leveling by Primary Motivation

• “Primary” motivation - whatever the player did most

• Look purely at time to reach level 85 (maximum level)

20

@_jon_bell_WiC May 14, 2013

RQ3: Leveling by Primary Motivation

0!1000!2000!3000!4000!5000!6000!7000!8000!9000!10000!

Achievement! Immersion! Social! Overall!

Tota

l Hou

rs to

Lev

el!

Character Classification!

21

@_jon_bell_WiC May 14, 2013

RQ3: Leveling by Primary Motivation

0!1000!2000!3000!4000!5000!6000!7000!8000!9000!10000!

Achievement! Immersion! Social! Overall!

Tota

l Hou

rs to

Lev

el!

Character Classification!

Players that “stop to smell the roses”

21

@_jon_bell_WiC May 14, 2013

Bonus RQ: Has Blizzard made the game easier?

22

@_jon_bell_WiC May 14, 2013

Studying Game Difficulty

• Modeled based on how long it takes to level

• Measured in real-world time

• Compare time-to-level from before Nov 2011 to post Nov 2011

23

@_jon_bell_WiC May 14, 2013

Leveling by Start Date

0!500!1000!1500!2000!2500!3000!3500!

20! 30! 40! 50! 60! 70! 80! 90*!

Hou

rs b

etw

een

leve

ls!

Level!

Pre-Cataclysm!Post-Cataclysm!

24

@_jon_bell_WiC May 14, 2013

Leveling by Start Date

0!500!1000!1500!2000!2500!3000!3500!

20! 30! 40! 50! 60! 70! 80! 90*!

Hou

rs b

etw

een

leve

ls!

Level!

Pre-Cataclysm!Post-Cataclysm!

Abandoned characters

24

@_jon_bell_WiC May 14, 2013

Leveling by Start Date

0!500!1000!1500!2000!2500!3000!3500!

20! 30! 40! 50! 60! 70! 80! 90*!

Hou

rs b

etw

een

leve

ls!

Level!

Pre-Cataclysm!Post-Cataclysm!

Abandoned characters

24

@_jon_bell_WiC May 14, 2013

Leveling by Start Date

0!500!1000!1500!2000!2500!3000!3500!

20! 30! 40! 50! 60! 70! 80! 90*!

Hou

rs b

etw

een

leve

ls!

Level!

Pre-Cataclysm!Post-Cataclysm!

Abandoned characters

24

@_jon_bell_WiC May 14, 2013

And there’s so much more.

25

@_jon_bell_WiC May 14, 2013

What’s the most popular character name?

Name Count  (Normalized)Shadow 922Nightmare 839Nemesis 831Loki 772Hades 722Pandora 707Lilith 672Lucian 660Wolverine 657Holycow 656Luna 644Kratos 642

26

@_jon_bell_WiC May 14, 2013

WoW: Social Data

• Guild

• Achievements (surprise!)

• Co-completions of group achievements

• Huge network of players

• Over 100 million edges

27

@_jon_bell_WiC May 14, 2013

Contributions

28

@_jon_bell_WiC May 14, 2013

Contributions

• Gathered the first massive dataset from the MMORPG, World of Warcraft

28

@_jon_bell_WiC May 14, 2013

Contributions

• Gathered the first massive dataset from the MMORPG, World of Warcraft

• Defined player profiles in WoW

28

@_jon_bell_WiC May 14, 2013

Contributions

• Gathered the first massive dataset from the MMORPG, World of Warcraft

• Defined player profiles in WoW

• Showed that these profiles are not static, and players evolve over time

28

@_jon_bell_WiC May 14, 2013

Contributions

• Gathered the first massive dataset from the MMORPG, World of Warcraft

• Defined player profiles in WoW

• Showed that these profiles are not static, and players evolve over time

• Showed that these players advance through content at different paces

28

@_jon_bell_WiC May 14, 2013

Contributions

• Gathered the first massive dataset from the MMORPG, World of Warcraft

• Defined player profiles in WoW

• Showed that these profiles are not static, and players evolve over time

• Showed that these players advance through content at different paces

• API Scraper: http://code.psl.cs.columbia.edu/

28

@_jon_bell_WiC May 14, 2013

A Large-Scale, Longitudinal Study of User Profiles in

World of WarcraftJonathan Bell, Swapneel Sheth, Gail Kaiser

Columbia University, New York, NY USA

PROGRAMMING SYSTEMS LAB

COLUMBIA UNIVERSITY

http://www.psl.cs.columbia.edu/

enable(vt):to make possible, practical, or easy

29