a/b testing in mobile
TRANSCRIPT
A/B TESTING IN MOBILE
MAURICIO T ZAQUIA@mtzaquia
MAURICIO TZAQUIA
APPLE DISTINGUISHED EDUCATOR, CLASS OF 2015
iOS DEVELOPER @
MOBILEMARKET
NEW USER RETENTION
0x
1x
2x
Web Apps
MOBILE TRANSACTIONS
Apps 54 %
Web 46 %
MOBILE CONVERSION
Web
Apps
0x 1x 2x 3x
”A/B TESTING IS A TERM FOR A RANDOMIZED EXPERIMENT WITH TWO VARIANTS, A AND B, WHICH ARE THE
CONTROL AND VARIATION IN THE CONTROLLED EXPERIMENT.”
Wikipedia, 2017
HYPOTHESIS EXPECTATION
IMPLEMENTATION
EXECUTION
ANALYSIS
DECISION
/* code */
button.setTitle("Buy", for: .normal)button.setTitleColor(UIColor.green, for: .normal)
// controller code...
// controller code...
// controller code...
if (Experiments.ios_button_urgency.variant == 1) { button.setTitle("Buy before it's too late!", for: .normal) button.setTitleColor(UIColor.red, for: .normal) } else { button.setTitle("Buy", for: .normal) button.setTitleColor(UIColor.green, for: .normal) }
// controller code...
35% 42%
button.setTitle("Buy before it's too late!", for: .normal) button.setTitleColor(UIColor.red, for: .normal)
// controller code...
// controller code...
Experiments.ios_button_urgency.variant
FEATURE FLAGS
Experiments.ios_button_urgency.variant
WHAT HAPPENS IF A WEBPAGE CRASHES?
WHAT HAPPENS IF
AN APP CRASHES?
”47% OF THE APPS CRASH AT LEAST 1% OF THE TIME, AND 32% OF THE APPS HAVE A
CRASH RATE OVER 2%”
Crittercism, 2016
EVERYBODY BUGS
FINDING CRASHES IS ”EASY”…
…BUT WITH THE RIGHT TOOLS
NETWORKLOAD
SCREENISSUES
MEMORYWARNING
NETWORKLOAD
Experiments.ios_deep_search
DAT
A E
XCH
AN
GE
OVE
R TI
ME
BASE VARIANT
NETWORKLOAD
Experiments.ios_deep_search
APP
ABA
ND
ON
MEN
T O
VER
TIM
E
BASE VARIANT
MEMORYWARNING
Experiments.ios_revamped_card
MEM
ORY
WA
RNIN
GS
OVE
R TI
ME
BASE VARIANT
SCREEN ISSUES
Experiments.ios_table_refactor
1.26% ± 1.64%
BASE
VARIANT
SCREEN FREEZE OVER 1,5 SECONDS
SCREEN ISSUES
Experiments.ios_layout_flow
-95.78% ± 7.19%
BASE
VARIANT
SCREEN VIEWS
SCREEN FREEZE
Experiments.ios_table_refactor
BASE VARIANT
FRAMEWORKS
HTTPS://GITHUB.COM/KEEPSAFE/SWITCHBOARD HTTPS://GITHUB.COM/MATTT/SKYLAB
COMPANIES
Q&A
THANK YOUMAURICIO T ZAQUIA@mtzaquia