15 a/b testing mistakes i see all the damn time (and what to do about it) by peep laja, conversionxl...
TRANSCRIPT
15 A/B testing mistakes I see all the damn time (and what to do about it)
by Peep Laja, ConversionXL Call to Action Conference 2014
“You should test that!”
“Dude wtf?”
@peeplaja
Typical scenario: A company runs 100 tests /
year. After 12 months their
conversion rate is still the same.
@peeplaja
Testing mistake #1:Precious time
wasted on stupid tests
@peeplaja
Testing mistake #2:
You don’t have a data-driven,
learning-oriented hypothesis
@peeplaja
Testing mistake #3:You think you
know what will work
@peeplaja
Testing mistake #4:You run tests
when you have no traffic
@peeplaja
Testing mistake #5:
Your absolute conversions are
way too low
@peeplaja
You need *at least* 250-400 conversions PER
variation.
Way more if you want to segment the data
@peeplaja
Our "thumb rule" is: 3000-4000 conversions per variation and 3-4 week test duration. That is enough traffic so we can even talk about valid data if we drill down into segments.
Testing "sin" nr.1: search in segments for uplifts although you have no statistical validity, e.g. 85 versus 97 conversions. That’s bullshit.
@peeplaja
Testing mistake #6:
Your tests don’t run long enough
@peeplaja
Testing mistake #7:
You don’t test full weeks at a time
@peeplaja
Testing mistake #8:
Test data is not sent to third-party
analytics
@peeplaja
Pro tip: If you have over 500,000
pageviews / mo, you don’t (necessarily) need to spend
$100,000 on GA Premium. Get unsampled data with Analytics
Canvas
@peeplaja
Testing mistake #9:
You give up after your first test for
a hypothesis fails.
Full case study: http://conversionxl.com/case-study-how-we-improved-landing-page-conversion/
79.3% uplift in conversions
@peeplaja
Testing mistake #11:
You’re not aware of validity threats
@peeplaja
History effectInstrumentation effect
Selection effect
@peeplaja
Testing mistake #12:
You’re ignoring small gains
@peeplaja
5% monthly increase in your conversion rate will result in
80% uplift for the year. That’s how math works.
@peeplaja
Testing mistake #13:
You don’t know html/css/jquery -
and thus only test the simple stuff
@peeplaja
Testing mistake #14:
You’re not running tests at all times
@peeplaja
Testing mistake #15:
You’re late for your lunch break!
Follow me @peeplaja
@peeplaja
Keep reading ConversionXL!
Thank you