a/b testing - in data we trust
TRANSCRIPT
A/B TESTINGIn data we trust!
Pedro Marques CI&T Digital Day
PEDRO MARQUESUX Designer @ booking.com @pmarquees
700kProperties
135Offices
207Countries
42Languages
74Currencies
8.5kEmployees
8kServers
perl + html templatesCore
3º MAIORe-commerce do mundo
A/B TestingWTF?
Emojiana
Emojiana Hippo
50%
50%
11%
2%
Born at GoogleEarly 00’s
PharmaceuticalTo test new drugs
But why A/B Test?
No Regression
But bear in mind:
A/B Test will prove you wrong
Data never lies
Define your KPI’s
Correlation does NOT imply causation
Conver$ion
User performs a given action
Define your KPI's
What should I test?
Test all your hypothesis
Assumption is the mother of all fuck ups
Framing Effect
A/B Testing Kills creativity
Junior designers X Sr. Designes
https://medium.com/the-year-of-the-looking-glass/junior-designers-vs-senior-designers-fbe483d3b51e
Would like fries with that?http://blog.booking.com/hamburger-menu.html
No significant impact.At all.
NICE!But how do I do that?
Coin flip
Now what?Data analysis is the key
Some advicesThese are mine, test and see if it works for you =)
Shit X ShitNope, a big nope.
Ask questionsEven before starting the experiment
Dont trust the gurusThis is science, not religion
Flip the coin properlyCalculate the amount of time and sample that you need
https://vwo.com/ab-split-test-duration/
A/B Testing is a big subject
Thank you folks! <[email protected]@pmarquees