product madness - a/b testing
DESCRIPTION
Andy Toben from Product Madness' presentation on A/B Testing, from GIAF.TRANSCRIPT
What is an A/B test ?“A/B testing is a methodology in advertising of using randomized experiments with two variants, A and B, which are the control and treatment in the controlled experiment.”
- wikipedia.org
What Can We A/B Test?Basically anything:• Page Colour • Layout• Call to action • Images• If you can change it, you can test it!
Step by Step OptimizationTesting one variable at the time works, but can create issues:
- Time consuming:Needs a lot of traffic or data
- Local verses Global Optimum:Can lead to accepting the best variant for that test, but it is not optimised against all possible variants
Multi Variant TestingThis allows you to test all combinations at once however:
- Requires massive data sets – much more than Step by Step
- Requires mathematical tools
- No intuitive / building insights
How do we know who’s winning ?
How do we know who’s winning ?
Advertising is a game, so how about games ?
Strategies
What can we use A/B testing to discover?
• Search - Option A / Option B• Optimize - Option A / Option A+ / Option A-• Change management - treatment verses control• Measure - No treatment verses treatment
A/B Testing User ExperienceWhen A/B testing user experience you can test out the following and create a big impact… for better or worse!
- Order of game levels / features- Bonus system- Available content- P2P assignment - Pricing and Economy
Challenges We FaceSome A/B tests are very expensive - You may need to commit development or art resource for something that may not work.
Variant assignment needs to persistent - In some cases you can’t just change a player’s track to the
“winning” branch What to measure:
• Responsiveness (CTR)• Engagement• Retention • Monetization
Branching user experience
Understanding Results
Feature cost 1st day retention 7th day retention User value
10,000 coins 37% 10% 10 USD
25,000 coins 36% 6% 12 USD
Which metric is the best one to look at, and which result should we take action on?
But after all, It’s just a tool
Allocate Variant
No Change
No
Should we test the User?
User
NoIs there an
active Experiment?
Yes
Yes
