Download - How many visitors do you need for your A/B test?

How many visitors do you need for your A/B test?

testing simplified!

© Wingify Software Pvt. Ltd.

Disclaimer

• We can never say that we need X number of visitors for an A/B test– Theoretically, an A/B test can take infinite number

of visitors before producing statistically significant result

• Instead, what we can say is that we need to at least test X number of visitors– With Y probability (usually 80%) of detecting a

statistical difference in results (if there is any)


Phew! Explanations please..


• Conversion rate is never an exact number, it is always a range. That is, we can never say conversion rate is 10%, although we always say it is 10% ± 1%

– This is because as we collect data, we are estimating what real conversion rate is (in statistical terms, we are estimating population mean from sample mean)

– Initially, our guesses are raw (as we only have few data points) but as we test more visitors, the error range decreases and we have better estimates of conversion rate


First, let’s define statistical significance

But, still, conversion rates are always estimates…• So, we now have two conversion rate estimates for control

and variation, say following:

• Observe how these ranges are overlapping, and so even if conversion of variation appears to be worse, we cannot say for sure (until these ranges are non overlapping, as following)


• There are two scenarios in A/B test:– Variation is performing better (or worse) as

compared to control• Difference in conversion rate is statistically significant

– Variation is performing similarly as compared to control• Difference in conversion rate is not statistically

significant


So, how many visitors to A/B test?

So, how many visitors to A/B test?

• Aim of A/B test calculations is to make sure we test enough visitors in order to know with certain confidence whether there is any statistical difference in control and variation conversion rate

• As stressed earlier, we can never be 100% sure that after testing X number of visitors we will know if test has a statistically better or worse performing variation– If we test even more visitors, there are even better chances of

finding a statistical difference if it really is there


Factors important in calculations

• Suppose we want to calculate X, which is the number of visitors we need to test in order to find out whether statistical significance is there

• There are various factors which help us calculate X:– Statistical Power (usually 80%): it is the probability with

which you expect to find out statistical significance after testing X visitors. (There are 80% chances that after testing X visitors, we will find statistical significant results if they are there)• As statistical power increases, your chances of finding a statistical

difference gets better (but of course you need to test more visitors)


Factors important in calculations– Statistical confidence (normally 95%): once statistical difference is found, it is the

confidence we have in that difference. (There is 5% chance that the difference in conversion rate is not real and is due to randomness)• If you need higher statistical confidence, we need to test more visitors

– Existing conversion rate of website: for lower conversion rates websites (say ones with 1% conversion), we need to test many more visitors as compared to situation if average conversion rate is higher (say 10%)

– Difference in conversion rate you want to detect: if you want to detect even a small difference in conversion rate (say you want to know if variation differs from control by even 0.1%), you need to test many more visitors. If you are only concerned with detecting a large differences (say only >10%), you need to test lesser number of visitors

– Number of variations you are testing: obviously, if you are testing 4 variations you need twice the number of visitors as compared to situation when you are just testing 2 variations.


You need a thumb rule?

Sorry, there is no thumb rule to find out how many visitors you need to

test :(


Not all is lost, though…

• You don’t have to be a statistician in order to do these calculations, you can use an online calculator to find out number of visitors to test:

http://visualwebsiteoptimizer.com/ab-split-test-duration/





Questions?

Paras Chopra, CEO, [email protected]


mailto:[email protected]

Download - How many visitors do you need for your A/B test?

Top Related