the finishing line
DESCRIPTION
One of the most commonly asked questions is “when is an MVT experiment or AB test finished?” Is it at 30 days...? 100 conversions...? 10,000 visitors...? The short answer is... it depends.TRANSCRIPT
![Page 1: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/1.jpg)
The Finishing Line- when is a Multivariate Test finished?
By Joe Doveton
Oban Digital
![Page 2: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/2.jpg)
When Is A Test Finished?
• One of the most commonly asked questions is “when is an MVT experiment or AB test finished?”
• Is it at 30 days...? 100 conversions...? 10,000 visitors...?
• The short answer is... it depends
![Page 3: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/3.jpg)
Known and Unknown VariablesA test is made up of known and unknown
variables...
![Page 4: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/4.jpg)
Things you know:– Likely visitors to
test page(s)
– Likely conversion rate for the original (control)
–Numbers of design variations in the test
Things you don’t:
– The improvement (or deterioration) in the conversion rate generated by the challenger designs
![Page 5: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/5.jpg)
Understanding Test Data
Conversion rate for the original is 51.68%
The leading challenger is Combination 7with an 80% probability of beatingthe original
This is a 2.40% uplift in Conversion Rate vsthe original
![Page 6: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/6.jpg)
Margin of erroris important because it effectively tells you if you have a big enough
sample. In an ideal world one would wait until the margin of error
was 0%
![Page 7: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/7.jpg)
Margin of error
• In this test there is 0.7% margin of error in the sample
• This means that the original is converting at 51.68% plus or minus 0.7%
• The best case scenario for the original is therefore to add 0.7% to the conversion rate making 52.38%
• The worst case scenario for the test leader is to subtract 0.7% from that conversion rate, making a CR of 52.22%
• This means the test is not quite statistically valid-we could run the test a little longer to get a big enough sample
• Despite this, there is enough evidence to apply Combination 7 as a test winner- 80% confidence and 2.40% uplift over the original plus £10k in increased revenue
![Page 8: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/8.jpg)
In some tests it is not practical to wait until you have 0% margin of
error, especially on low traffic pages.
![Page 9: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/9.jpg)
Sometimes you just have to make a call. Here are some tips to help you decide when your test is finished.
![Page 10: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/10.jpg)
Fit test type to visitor profile
• With low traffic pages make small changes and test via AB testing. You will get a quicker result and you can iterate
• With higher traffic pages you can use MVT. MVT has the advantage of allowing you to test many variations at once but still isolate the impact of individual on-page elements
![Page 11: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/11.jpg)
Brainstorm
• A test is only as good as the ideas that go into it
• Manually go through funnels
• Use analytics
• Analyse competitors
• Group and ungroup elements
• Prioritise tests- do the low hanging fruit first
• Set yourself cut off points
![Page 12: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/12.jpg)
Test goals
Match your test goals as closely as possible with your regular goals set up in analytics
Test goals should be consistent with the elements that you want to change- for example if you want to change CTR changing the footer navigation isn’t likely to contribute much to this
![Page 13: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/13.jpg)
Keep Things Manageable
• If using MVT keep your test down to a manageable number of combinations
• Test only a few elements together concurrently
• Each time you add in a new element, you double the length of time it takes to get a statistically valid result (same with segments)
• No-one wants to read a test with 128 possible test combinations (sorry Mr. Taguchi)
![Page 14: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/14.jpg)
Check The “Elements” View
• With MVT, one can check both the interaction between elements (the ‘combination’) and the individual elements in isolation
• Often, the elements view will give you a different view of the data
• In a test with no clear winners, there may be evidence of specific elements driving conversions that allows you to plan a follow-on experiment
![Page 15: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/15.jpg)
Confidence Level
• 100% confidence results in test are rare. Very rare- even in AB tests.
• Set yourself an acceptable confidence level and go with it
• Anything over 70% is pretty good- 80% and above is exceptional
• PS this also works in reverse with losing tests
• Be prepared to call winners and losers
• Accept that inconclusive results are part of the game
![Page 16: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/16.jpg)
Be Prepared To Iterate
• So your first test result was inconclusive. Great! Why great? It gives you an opportunity to re-evaluate the design
• You start to gain insight into what elements and what changes affect user behaviour
• Inconclusive test results give you the chance to try something else
• Each test is a stepping stone-iterate, test and move on
![Page 17: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/17.jpg)
Lifetime Value of Test Result
• A small uplift over a two week test period might leave you feeling underwhelmed
• However, what is that uplift worth when extrapolated over a whole year?
• Even a modest test result could transform your business
![Page 18: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/18.jpg)
There’s No Such Thing As Bad Data
• Everybody loses tests
• It’s part of the game
• Losing tests stop you from going off with design changes in the wrong direction
• Losing tests help to point you back in the right direction
![Page 19: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/19.jpg)
Follow The Money
• ... and finally... follow the money
• Many platforms allow you to track revenue as well as conversions- if not, integrate with Google Analytics
• Even if you increase conversions it may not increase revenue- your design could make it easier for customers to buy your lower priced items
• Track the test impact on your bottom line either through your test tool or your analytics
![Page 20: The Finishing Line](https://reader034.vdocuments.mx/reader034/viewer/2022042817/559debff1a28ab3f148b46f7/html5/thumbnails/20.jpg)
Contact Oban Digital
Phone: +44 1273 936095
Email: [email protected]
Website: www.obandigital.com
Twitter: twitter.com/ObanDigital