practical a/b testing statistics: 5 tips to help you get reliable data
TRANSCRIPT
![Page 1: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/1.jpg)
![Page 2: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/2.jpg)
Your%Presenter
Michael%Aagaard
4 Senior%Conversion%Op:mizer%at%Unbounce%
4 Full4:me%CRO%for%the%lat%seven%years%
4 Interna:onal%speaker%
4 Founder%of%ContentVerve.com
![Page 3: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/3.jpg)
Join%the%conversa:on
4%Include%the%hashtag%#thinkppc)in%your%TwiJer%tweets.%4%Or%use%the%webinar%ques:on%box%to%send%us%ques:ons.
![Page 4: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/4.jpg)
“When can I stop my A/B Test?”
![Page 5: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/5.jpg)
“Why run an A/B Test?”
![Page 6: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/6.jpg)
To get a representative picture of how a variant would perform if implemented permanently “in the wild”.
![Page 7: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/7.jpg)
To get a representative picture of how a variant would perform if implemented permanently “in the wild”.
![Page 8: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/8.jpg)
Do Not Use 95% Significance Level as a Stopping Rule
1
![Page 9: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/9.jpg)
After 1 day
+151.4% / significance 97%
655 visitors / 18 conversions
![Page 10: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/10.jpg)
After 2 days
+98.4% / significance 95%
988 visitors / 23 conversions
![Page 11: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/11.jpg)
After 1 week
+0.2% / significance 50%
3031 visitors / 58 conversions
![Page 12: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/12.jpg)
After 2 weeks
+18.1% / significance 77%
5277 visitors / 78 conversions
![Page 13: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/13.jpg)
After 4 weeks
+36.2% / significance 96%
Total sample: 9396 visitors / 122 conversions
![Page 14: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/14.jpg)
What we have now: 9,396
What we need: 34,304 visitors in total
What we’re missing:24,908 visitors 73% of the required sample size.
http://www.evanmiller.org/ab-testing/sample-size.html
![Page 15: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/15.jpg)
http://abtestguide.com/calc/
![Page 16: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/16.jpg)
“Formulas don’t know whether they are being used properly, and they don’t warn you when your results are incorrect.”
- Deborah J. Rumsey, PhD
![Page 17: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/17.jpg)
“Real accuracy depends on the quality of data as well as on the sample size.”
- Deborah J. Rumsey, PhD
![Page 18: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/18.jpg)
Worth Getting Familiar With:
- Power Level - P-Value - Margin of Error - Confidence Interval - Null hypothesis / alternative hypothesis
![Page 19: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/19.jpg)
Make Sure You Collect Enough Data to Yield Meaningful Insight
2
![Page 20: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/20.jpg)
Full weeks Full business cycles
![Page 21: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/21.jpg)
Weekly Business Cycle
![Page 22: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/22.jpg)
Full weeks Full business cycles
Week One Week Two Week Three Week Four
![Page 23: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/23.jpg)
Test duration: 2-4 weeks (2 or more business cycles)
Sample size: Pre-calculated sample size (users)
400+ conversions (goals)Significance: 95% or higher
Power: 80% P-Value: 0%
What I look for:
![Page 24: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/24.jpg)
Be Careful With Averages - They Hide the Truth
3
![Page 25: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/25.jpg)
Devices
Browsers
Channels
Country
Days of the week
Hours ofthe day
User types
Motivation
Campaigns
Age
Gender
Gender
BrowsersDevices
Browsers
Channels
Countries
Days of the week
Hours ofthe day
User types
Motivation
Campaigns
Age
Days of the month
Gender
Devices
Experiment
![Page 26: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/26.jpg)
![Page 27: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/27.jpg)
![Page 28: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/28.jpg)
Integrate Test Data with Your Web Analytics Setup
And Get the Full Picture
![Page 29: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/29.jpg)
Device?Channel?
Campaign?User Type?
![Page 30: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/30.jpg)
Formulate a Clear Test Hypothesis
4
![Page 31: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/31.jpg)
Why do we think we need to make a change?
What is it that we want to change?
What impact do we expect to see?
How will we measure this impact?
![Page 32: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/32.jpg)
1. Because we saw [data/feedback]
2. We expect that [change] will cause [impact]
3. We’ll measure this using [data metric]
![Page 33: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/33.jpg)
Because [exit surveys indicated that saving money is most important to our users.]
We expect that [tweaking our value prop to reflect this] will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our key metric.]
![Page 34: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/34.jpg)
Because [exit surveys indicated that saving money is most important to our users.]
We expect that [tweaking our value prop to reflect this] will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our key metric.]
![Page 35: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/35.jpg)
Because [exit surveys indicated that saving money is most important to our users.]
We expect that [tweaking our value prop to reflect this] will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our key metric.]
![Page 36: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/36.jpg)
Because [exit surveys indicated that saving money is most important to our users.]
We expect that [tweaking our value prop to reflect this] will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our key metric.]
We expect to see reliable results in [four weeks.]
![Page 37: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/37.jpg)
Learn Statistics!!!
5
![Page 38: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/38.jpg)
http://amzn.to/1RFNBwA
![Page 39: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/39.jpg)
Live%Q&A%Time
![Page 40: Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data](https://reader031.vdocuments.mx/reader031/viewer/2022022413/58efdf131a28abd67d8b457b/html5/thumbnails/40.jpg)
Have%more%questions?
Contact%Us%Directly
• Webinar%Feedback:%[email protected]
____________________________________________
Thank'you'for'attending' our'webinar!'#thinkppc