a/b testing that matters

@commagereBlake Commagere

A/B Testing

Technically Split or Bucket Testing, but nobody calls it that.

Blake Commagere @commagere

Who the hell am I? It’s a fair question

●Started seven companiesoFive of those have been bought

●Raised over 12M Angel / VC●I build viral things

oPlaxooCausesoVampires

●I’m obsessed with human behavior

2


My Fucking TalksAnalogies Rule

Dave McClure Blake Commagere

●Makes sailors blush with his swearing

●Occasionally uses hyperbole in his talks

●Is funny

●Makes Dave McClure blush with his swearing

●The most egregious abuse of hyperbole

●Thinks he’s funny

3


This talk is for you.

Interrupt at any point. Don’t be shy.

If it sucks and you never say anything, I’m blaming you.

4


What I’m covering

●What the hell is A/B testing?

●Why should you care?

●How should you A/B test?

●How can A/B testing fuck you over?

●A few ancillary benefits of A/B testing

5


What I’m NOT covering AKA “Shit you should Google if you don’t already know”

●Familiarize yourself with concepts likeoConfidence IntervalsoComparative ErroroStatistical SignificanceoSample Size requirements

FYI, there are online calculators for all these things

●How to fucking do mathoYes there’s a lot of math in A/B TestingoChances are, the tools you use will do the math for you

If not, your code better do the math for you

6


What the hell is A/B testing?

7


What the hell is A/B Testing? Will it hurt?

●Randomized ExperimentoControl & VariantoTests against a specific goaloTypically you want to establish Statistical Significance with a Confidence Level of 95%

●Bucket or Split Testing oTechnically, this is what you want most of the timeoMultiple variantsoPeople usually say A/B when they mean Bucket/Split oJust go with it

8


Why should you care?

9


Do I really need A/B Testing? Technically, you may not need it

●Do you have users?●If not, do you plan on having users?●If not… congrats! You don’t need A/B Testing because you don’t have a business!

●For everyone else, you need A/B Testing.

10


You are not your user Your users are much, much stupider

You Your User

●Busy as hell

●Dedicated

●Very knowledgeable

●Eager to learn

●Not Stupid

●Lazy

●Impatient

●Stupid

●Stupid

●Real. Stupid.

11


The Result? You’re building the product for you

●You are not your user

●Your instincts are probably wrong

●Your wants/needs are probably different

●You view the market differently

12


Still not convinced?

●Create and Make are synonyms

●“Make an Ad” vs “Create an Ad”

●Software Prefers ‘Create’ because REST, CRUD

●Make outperformed by up to 20% for non-artists

●There is a mental barrier to Create - it sounds more involved/complex. Making is easier.

13

Create vs Make - one word makes all the difference


How should you A/B Test

14


Blockers on your A/B testing If this is happening, all your tests will suck

●Shit is brokenoFIX IT NOW WHY ARE YOU FUCKING READING THIS

●Shit is slowoPageSpeed on frontend, oProfilers on backend / mobile

●Official definitions of slow*oA webpage that takes > ~2 seconds & no progress

indicatoroAn app that isn’t immediately responsive

*according to me

15


Ensuring tests maximize impact AKA the most obvious advice ever

1)ABT. Always Be Testing 2)Win as frequently as possible3)In areas that matter

●Any moron can get #1 right●Most morons can’t do #2 or #3●Essentially, you need to know what to test and how to test.

16


Only a few features matter That’s it

●Users are either engaged or not ●Data on your userbase tells you which 2-3 features matter

●Optimize these features●Make Onboarding focus on these features●Facebook example:oFriends, Photos, Status UpdatesoNew User Experience focused on theseoNetwork leverage to help new users

17


Similarly, only a few flows matter●Is a flow critical?odoes it get a user engaged with a critical featureodoes it help another user engage with a critical feature

●Is every step necessary?oAlmost always, the answer is no

●Where are the leaks?oAlmost always, this is the result of bad messagingoFind the worst leak, fix it, then move to the next one

18

That’s also it


Your messaging is never perfect AKA test your fucking messaging

●In most cases it is better to:oUse fewer words - your users are lazy oBe Colloquial - your users are dumboTarget the user - your users are selfish

●You can always target betteroSome data is implicit to your existing dataoYour users only care about what they get

●Language is always evolvingoLanguage in every channel is evolving as welloChannels change & language in a channel changes!

19


Tools of the trade

●Web:oGoogle Analytics (Content Experiments)oOptimizelyoMixPanel

●Mobile:oMixPaneloOptimizelyoSwrve

20


Which tool should I use?

●The best one for your company:oIt dependsoOn. so. many. things.

●You’ll end up doing some coding too probably.

●The worst one for your company?oNot having oneoCreating your own from scratch

21


How A/B Testing can fuck you over

22


Pitfalls Some very common mistakes

●Find statistical significance sample size in advanceoYou want 95% confidence leveloWith 20,000 users, this can be 377 people. NOT 50/50

●Do not take convenience sampleso2-4 weeks is ideal

●Test in 7, 14, 28 day intervalsoHuman behavior is day dependent, tied to weekoYour results will skew if you do 8 day tests

23


The Lies We Tell Ourselves Sometimes we like lying because we’re lazy

●You can make the numbers lie

●Avoiding a channel that you hateoEmail worksoSEO worksoTwitter worksoFacebook worksoIf a channel doesn’t work for you, you’re probably doing it wrong

24


Farmville How to optimize for suck

Farmville on Launch

Farmville after ~2 yrs

25


Ancillary Benefits

26


Designer vs Engineer Your users are much, much stupider

Designer Engineer

●Better not suck at design

●Better have design exp

●Better respect design

●Will not always agree with engineers on design

●Not Stupid

●Probably sucks at design

●May not have design exp

●Should respect design

●Will not always agree with designers on design

●Not Stupid

27


Using A/B Testing for Team Bonding●The loudest voice usually gets their way

●This can stifle good ideas

●Sometimes it’s ok to test a bad idea

●Becomes a teachable moment

●You could be wrong (GASP!)

28


Want more swearing and/or advice?

[email protected]

@commagere

29

mailto:[email protected]

https://twitter.com/commagere

a/b testing that matters

Technology