devtribe a/ b testing on multiple platforms with recurring and paying users
TRANSCRIPT
A/B testing (on multiple platforms with recurring and paying users)
at Spotify
DevTribe Gathering 32016-12-06. Oscar Carlsson, [email protected]
A/B testing at Spotify(on multiple platforms with recurring and paying users)
Oscar CarlssonData Scientist
Feature toggling+
A/B testing
picture from https://www.elegantthemes.com/blog/tips-tricks/easy-ab-testing-with-divi-leads
The classic example of A/B testing
Winner if CTR/Conversion is up.
Oftenly no requirement to understanding the
why
New users
Existingusers
Multiple platforms
Premium& Free
New usersExisting users
Platforms
New usersNewly registered users with no habits or routines in the app. They have never seen the UI and any impression or interaction is guaranteed to be the first.
Relatively low traffic.
Activation:How many registered users use the app on first day?
Engagement:How much are they using the product?
Retention: How many registered users use the app second week/month?
Conversion:Do they turn premium?
New usersNewly registered users with no habits or routines in the app. They have never seen the UI and any impression or interaction is guaranteed to be the first.
Relatively low traffic.
Activation:How many registered users use the app on first day?
Retention: How many registered users use the app second week/month?
Engagement:How much are they using the product?
Conversion:Do they turn premium?
Existing usersUsers who are using your product already. They probably have habits and routines and they might have seen the surface before.
Relatively high traffic.
Engagement:How much are they using the product?
Conversion:Do they turn premium?
Existing usersUsers who are using your product already. They probably have habits and routines and they might have seen the surface before.
Relatively high traffic.
Engagement:How much are they using the product?
Conversion:Do they turn premium?
Platforms(Often) test separately
Not fully aligned - We’re testing in different phase on all platforms
Users can use multiple platforms and usage can shift
AndroidiPhone Desktop Web playerand more.
Every team should be testing!
.. could there be challenges with that?
A/B testing
A/Btesting
A/Btesting
A/Btesting
I want long term effect!
I want your change andI want long term effect!
I also want those changes andI want long term effect!
Me too andI want long term effect!
I want long term effect!
I want your change andI want long term effect!
I also want those changes andI want long term effect!
Me too andI want long term effect!
Dependencies!&
Complicatedcode?
New bold fontCenter header
Remove card backgroundCenter follower count
Fully scientific? Quick product development?
Fully scientific? Quick product development?
Fully scientific? Quick product development?
A mix of both?
How many tests can a user live through?
How do you track, and do you track what
happens to the users when the test is over?
picture from http://codevoyagers.com/2016/03/16/design-like-youre-right-test-like-youre-wrong/
Thanks for listeningOscar Carlsson