a/b testing with air patriots - gdc 2014
DESCRIPTION
A/B Testing is about using data to challenge assumptions and test new ideas, often leading to increased engagement and monetization. In addition to showing you the latest cross-platform testing services at your disposal, we'll discuss the "happy accident" that the Air Patriots team made. It was unexpected and led to an increase in both retention and monetization.TRANSCRIPT
![Page 1: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/1.jpg)
1
@PaulCutsinger
Russ Carroll
A|B Testing
![Page 2: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/2.jpg)
2
![Page 3: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/3.jpg)
3
![Page 4: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/4.jpg)
4
Surface the Value
Image from Diner Dash by PlayFirst
Source: Amazon Appstore, January 2013
+75% ARPPU
![Page 5: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/5.jpg)
5
Show Users How to “Consume”
Image from Cut the Rope by ZeptoLab
Source: Amazon Appstore, January 2013
+65%
repeat orders
![Page 6: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/6.jpg)
6
Experiment and react quickly
7 day retention
Time
Active Players
![Page 7: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/7.jpg)
7
A|B Testing
![Page 8: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/8.jpg)
8
This is Air Patriots
![Page 9: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/9.jpg)
Case Study: Store Optimization
![Page 10: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/10.jpg)
10
• Goal – Easier to access store content
• Platform – Android & iOS
• Test – Make it possible to unlock content on Main Menu
STORE OPTIMIZATION
![Page 11: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/11.jpg)
11
LOCKED ON MAIN MENU
UNLOCKABLE >
<
![Page 12: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/12.jpg)
12
STORE OPTIMIZATION
Test Case Conversion to
Customer 7-Day Retention
Locked on Main Menu
(Control) – –
Unlockable on Main
Menu 20.93% 4.17%
![Page 13: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/13.jpg)
13
• Goal – Drive players to our other games
• Platform – Android
• Test – Which image is clicked on more
CROSS PROMOTION
![Page 14: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/14.jpg)
CROSS PROMOTION
![Page 15: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/15.jpg)
15
CROSS PROMOTION
![Page 16: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/16.jpg)
16
Air Patriots: Cross Promotion
B
A 1.77%
0.93%
Clic
k T
hro
ug
h
![Page 17: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/17.jpg)
19.94% 1. 3 day Retention -1.07% +0.83%
8.06% 2. 7 day Retention +0.74% -10.11%
2.05% 3. Units Purchased +0.34% -12.07%
0% 4. Ads Clicked +1.89% +3.07%
43,120
The Producer’s Perspective Ads on menu or in game
![Page 18: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/18.jpg)
19.94% 1. 3 day Retention -1.07% +0.83%
8.06% 2. 7 day Retention +0.74% -10.11%
2.05% 3. Units Purchased +0.34% -12.07%
0% 4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In Game On Menu No Ads
The Producer’s Perspective
![Page 19: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/19.jpg)
19.94% 1. 3 day Retention -1.07% +0.83%
8.06% 2. 7 day Retention +0.74% -10.11%
2.05% 3. Units Purchased +0.34% -12.07%
0% 4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In Game On Menu No Ads Goals
The Producer’s Perspective
![Page 20: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/20.jpg)
19.94% 1. 3 day Retention -1.07% +0.83%
8.06% 2. 7 day Retention +0.74% -10.11%
2.05% 3. Units Purchased +0.34% -12.07%
0% 4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In Game On Menu No Ads Goals
The Producer’s Perspective
![Page 21: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/21.jpg)
19.94% 1. 3 day Retention -1.07% +0.83%
8.06% 2. 7 day Retention +0.74% -10.11%
2.05% 3. Units Purchased +0.34% -12.07%
0% 4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In Game On Menu No Ads Goals
The Producer’s Perspective
![Page 22: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/22.jpg)
22
Retrieve a variation
Objective C
![Page 23: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/23.jpg)
23
Record a view and conversion Objective C
bit.ly/ABTestXPlat
bit.ly/ABTestiOS
bit.ly/ABTestJava
More details at
![Page 24: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/24.jpg)
Case Study: Junk Race by GameNug
![Page 25: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/25.jpg)
Junk Race Overview
![Page 27: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/27.jpg)
27
Tips: Producers in control: “With two-way services you can request a value from the
server that you control. Two-way services let you make changes to your live game
instantly.
Free: Amazon is the only free two-way service that we have found.”
Multi-variable: “We don't test variables, we test user experiences.”
Watch sample size: “If a couple users can change your conclusion, you need a
larger sample”
Deploy Instantly Cross Platform: “A/B testing on iOS has the advantage of
skipping the week long review period ”
-John Goforth
![Page 28: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/28.jpg)
28
Case Study: Difficulty Level
![Page 29: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/29.jpg)
29 Amazon Services Confidential
GAME DIFFICULTY TEST
7 day retention
Time
Active Players
![Page 30: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/30.jpg)
30 Amazon Services Confidential
GAME DIFFICULTY TEST
• Goal – A Dramatic Increase!
• Platform – Android & iOS
• Test – A/B/n Test – 5 Difficulty levels!
OR?
![Page 31: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/31.jpg)
31 Amazon Services Confidential
GAME DIFFICULTY TEST
![Page 32: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/32.jpg)
32 Amazon Services Confidential
Test 1:
A
Control
B
2% easier
C
4% easier
D
6% easier
E
8% easier
7-Day Retention – tbd tbd tbd tbd
Conversion to customer – -1.71% 16.11% 19.23% 4.84%
GAME DIFFICULTY TEST
![Page 33: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/33.jpg)
33 Amazon Services Confidential
GAME DIFFICULTY TEST
• How much would you pay to increase retention by 20%?
• How much time would you invest?
Our Take-Aways!
• Never launch a game in future w/o AB difficulty test!
• Low-hanging fruit!
• Opportunities abound!
![Page 34: A/B Testing with Air Patriots - GDC 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042815/55856183d8b42ab5228b49ec/html5/thumbnails/34.jpg)
34 Amazon Services Confidential
A|B Testing
• Monetization – minimize luck and maximize
skill
• Delight your customers
• test customer experiences
• AB testing tools are easy to use
• Cross Platform
• Deploy changes instantly
• Free
• Low-hanging fruit is waiting to be harvested
Paul Cutsinger Russell Carroll Find me at: • @PaulCutsinger • @AmazonAppDev • developer.amazon.com/blog • bit.ly/ABTestXPlat