how many visitors do you need for your a/b test?
DESCRIPTION
This presentation will help you understand which factors are important while calculate number of visitors needed for an A/B test.TRANSCRIPT
How many visitors do you need for your A/B test?
testing simplified!
© Wingify Software Pvt. Ltd.
Disclaimer
• We can never say that we need X number of visitors for an A/B test– Theoretically, an A/B test can take infinite number
of visitors before producing statistically significant result
• Instead, what we can say is that we need to at least test X number of visitors– With Y probability (usually 80%) of detecting a
statistical difference in results (if there is any)
© Wingify Software Pvt. Ltd.
Phew! Explanations please..
© Wingify Software Pvt. Ltd.
• Conversion rate is never an exact number, it is always a range. That is, we can never say conversion rate is 10%, although we always say it is 10% ± 1%
– This is because as we collect data, we are estimating what real conversion rate is (in statistical terms, we are estimating population mean from sample mean)
– Initially, our guesses are raw (as we only have few data points) but as we test more visitors, the error range decreases and we have better estimates of conversion rate
© Wingify Software Pvt. Ltd.
First, let’s define statistical significance
But, still, conversion rates are always estimates…• So, we now have two conversion rate estimates for control
and variation, say following:
• Observe how these ranges are overlapping, and so even if conversion of variation appears to be worse, we cannot say for sure (until these ranges are non overlapping, as following)
© Wingify Software Pvt. Ltd.
• There are two scenarios in A/B test:– Variation is performing better (or worse) as
compared to control• Difference in conversion rate is statistically significant
– Variation is performing similarly as compared to control• Difference in conversion rate is not statistically
significant
© Wingify Software Pvt. Ltd.
So, how many visitors to A/B test?
So, how many visitors to A/B test?
• Aim of A/B test calculations is to make sure we test enough visitors in order to know with certain confidence whether there is any statistical difference in control and variation conversion rate
• As stressed earlier, we can never be 100% sure that after testing X number of visitors we will know if test has a statistically better or worse performing variation– If we test even more visitors, there are even better chances of
finding a statistical difference if it really is there
© Wingify Software Pvt. Ltd.
Factors important in calculations
• Suppose we want to calculate X, which is the number of visitors we need to test in order to find out whether statistical significance is there
• There are various factors which help us calculate X:– Statistical Power (usually 80%): it is the probability with
which you expect to find out statistical significance after testing X visitors. (There are 80% chances that after testing X visitors, we will find statistical significant results if they are there)• As statistical power increases, your chances of finding a statistical
difference gets better (but of course you need to test more visitors)
© Wingify Software Pvt. Ltd.
Factors important in calculations– Statistical confidence (normally 95%): once statistical difference is found, it is the
confidence we have in that difference. (There is 5% chance that the difference in conversion rate is not real and is due to randomness)• If you need higher statistical confidence, we need to test more visitors
– Existing conversion rate of website: for lower conversion rates websites (say ones with 1% conversion), we need to test many more visitors as compared to situation if average conversion rate is higher (say 10%)
– Difference in conversion rate you want to detect: if you want to detect even a small difference in conversion rate (say you want to know if variation differs from control by even 0.1%), you need to test many more visitors. If you are only concerned with detecting a large differences (say only >10%), you need to test lesser number of visitors
– Number of variations you are testing: obviously, if you are testing 4 variations you need twice the number of visitors as compared to situation when you are just testing 2 variations.
© Wingify Software Pvt. Ltd.
You need a thumb rule?
Sorry, there is no thumb rule to find out how many visitors you need to
test :(
© Wingify Software Pvt. Ltd.
Not all is lost, though…
• You don’t have to be a statistician in order to do these calculations, you can use an online calculator to find out number of visitors to test:
http://visualwebsiteoptimizer.com/ab-split-test-duration/
© Wingify Software Pvt. Ltd.