optimizely stats engine: an overview and practical tips for running experiments
TRANSCRIPT
Optimizely Stats EngineLeo Pekelis
Darwish Gani Robin Pam
Housekeeping notes
• Chat box is available for questions • There will be time for Q&A at the end • We will be recording the webinar for future viewing • All attendees will receive a copy of slides after the webinar
Your speakers
Darwish Gani Product manager
Robin Pam Product marketing
Leo Pekelis Statistician
Objectives
Understand why Optimizely built Stats Engine
Introduce the methods Stats Engine uses to calculate results
Get practical recommendations for how to test with Stats Engine
Why make a new Stats Engine?
Meet Joe: A farmer who uses a traditional t-test
Joe wants to try a new fertilizer this year
With his original fertilizer, 10% of plants survive the winter
He thinks that this new fertilizer might helpmore survive.
Joe has a hypothesis
Joe calculates a sample size for his experiment in advance, given how much better he thinks the new fertilizer might be
He waits through the winter…
10% of plants survive the winter
15% of plants survive
96% statistical significance!
…and he is rewarded for his patience
Does that work today?
Meet Kyle: Head of Optimization at Optimizely
Kyle doesn’t know what improvement to expect
Kyle also gets data from Optimizely all the time
Kyle wants to test many goals and variations at once, instead of just one hypothesis
vs.
Actually that’s a lot of work. It’s cumbersome and error-prone.
What’s your chance of making incorrect decision?
30%
Objectives
Understand why Optimizely built Stats Engine
Introduce the methods Stats Engine uses to calculate results
Get practical recommendations for how to test with Stats Engine
Introducing Stats Engine
How we did it
• Partnered with Stanford statisticians
• Talked with customers • Examined historical experiments
• Found the best methods for real-time data
What does Stats Engine do?
Provides a principled and mathematical way to calculate your chance of making an incorrect decision.
Sequential Testing False Discovery Rate• First used in 1940s for military weapons testing
• Sample size is not fixed in advance
• Data is evaluated as it’s collected
• First used in 1990s in genetics
• Correct error rates for multiple goals and variations
• Expected number of false discoveries
Sequential Testing
False Discovery Rate control
+
=
Statistical Significance for Digital Experimentation
Statistical Significance for Digital ExperimentationContinuously Evaluate Test Results
Run many goals and variations
Don’t worry about estimating a MDE upfront
Sequential Testing Finding the right stopping rule
Variation #1
Variation #2
Declare a winner?
500
Visitors
50%
65%
Is this lift big enough for the visitors I saw?
?
Desired Stopping Rule: I will be “wrong” only 5% of the time.
Variation #1
Variation #2
1000
Visitors
55%
59%
Traditional Error Rates 5%
Find a stopping rule, so I declare a winner incorrectly <5% of the
time
Variation #1
Variation #2
Traditional Error Rates
Visitors
500
50%
65%
5%
1000
55%
59%
5%
5000
52%
57%
5%
10000
54%
59%
5%
Variation #1
Variation #2
Traditional Error Rates
500 1000 5000 10000
Visitors
50% 55% 52% 54%
65% 59% 57% 59%
5% 5% 5% 5%
Look only once: 5% Error rate
Variation #1
Variation #2
Traditional Error Rates
500 1000 5000 10000
Visitors
50% 55% 52% 54%
65% 59% 57% 59%
5% 5% 5% 5%
Look only once: 5% Error rate
Look more than once: >5% Error rate
Variation #1
Variation #2
Traditional Error Rates
500 1000 5000 10000
Visitors
50% 55% 52% 54%
65% 59% 57% 59%
5% 5% 5% 5%
<5% Error rate for the entire test!
Sequential Testing Error Rate 1% .5% 1.5% 1.5% < 5%
“P-Hacking” “Continuous Monitoring”
“Repeated Significance Testing”
These are not new problems!
Steven Goodman, Stanford Physician & Statistician, nature.com
“The P value was never meant to be used the way it's used today.”
Source: Evan Miller, How not to Run an A/B Test
Sample Size + Power Calculations Focus on creating and running tests
Sequential Testing
• Continuously Evaluate Test Results
• Don’t worry about estimating a MDE upfront
Framework of hypothesis testing that was created to allow the experimenter to evaluate test results as they come in
False discovery rate control Error rates for a world with many goals and
variations
Variations Goal 1 Goal 2 Goal 3 Goal 4 Goal 5
Control
Variation 1
Variation 2
Variations Goal 1 Goal 2 Goal 3 Goal 4 Goal 5
Control
Variation 1
Variation 2
Significance Level 90 (False Positive Rate 10%)
Variations Goal 1 Goal 2 Goal 3 Goal 4 Goal 5
Control
Variation 1
Variation 2
Significance Level 90 (False Positive Rate 10%)
1 False Positive!
Variations Goal 1 Goal 2 Goal 3 Goal 4 Goal 5
Control
Variation 1
Variation 2
1 False Positive!
Significance Level 90 (False Positive Rate 10%)
1 other variation x goal has a large improvement.
Variations Goal 1 Goal 2 Goal 3 Goal 4 Goal 5
Control
Variation 1
Variation 2
1 False Positive!
Significance Level 90 (False Positive Rate 10%)
1 other variation x goal has a large improvement.
1 True Positive!
My Report
• Variation 2 is improving on Goal 1
• Variation 1 is improving on Goal 4
“10% of what I
report could be wrong.”X50%
• Variation 2 is improving on Goal 1• Variation 1 is improving on Goal 4
Furthermore, I found the following results.This leads me to conclude that …
Statistical Significance:
The chance that any variation on any goal
that is reported as a winner or loser
is correct.
• “New York Times has a feature in its Tuesday science section, Take a Number … Today’s column is in error … This is the old, old error of confusing p(A|B) with p(B|A).”
• Andrew Gelman, Misunderstanding the p-value
• “If I were to randomly select a drug out of the lot of 100, run it through my tests, and discover a p<0.05 statistically significant benefit, there is only a 62% chance that the drug is actually effective.”
• Alex Reinhart, The p value and the base rate fallacy
• “In this article I’ll show that badly performed A/B tests can produce winning results which are more likely to be false than true. At best, this leads to the needless modification of websites; at worst, to modification which damages profits.”
• Martin Goodson, Most Winning A/B Test Results are Illusory
• “An unguarded use of single-inference procedures results in a greatly increased false positive (significance) rate”
• Benjamini, Yoav, and Yosef Hochberg. "Controlling the false discovery rate: a practical and powerful approach to multiple testing." Journal of the Royal Statistical Society. Series B (Methodological) (1995): 289-300. APA
• “New York Times has a feature in its Tuesday science section, Take a Number … Today’s column is in error … This is the old, old error of confusing p(A|B) with p(B|A).”
• Andrew Gelman, Misunderstanding the p-value
• “If I were to randomly select a drug out of the lot of 100, run it through my tests, and discover a p<0.05 statistically significant benefit, there is only a 62% chance that the drug is actually effective.”
• Alex Reinhart, The p value and the base rate fallacy
• “In this article I’ll show that badly performed A/B tests can produce winning results which are more likely to be false than true. At best, this leads to the needless modification of websites; at worst, to modification which damages profits.”
• Martin Goodson, Most Winning A/B Test Results are Illusory
• “An unguarded use of single-inference procedures results in a greatly increased false positive (significance) rate”
• Benjamini, Yoav, and Yosef Hochberg. "Controlling the false discovery rate: a practical and powerful approach to multiple testing." Journal of the Royal Statistical Society. Series B (Methodological) (1995): 289-300. APA
False Discovery Rate controlFramework for controlling errors that arise from running multiple
experiments at once.
Run many goals & variations
What Stats Engine means for you
• You see fewer, but more accurate conclusive results.
• You can implement winners as soon as significance is reached.
• You get • easy experiment workflow. • reduced unforeseen, and hidden errors.
Objectives
Understand why Optimizely built Stats Engine
Introduce the methods Stats Engine uses to calculate results
Get practical recommendations for how to test with Stats Engine
And now, for some practical guidance
First, some vocabulary• Baseline conversion rate
The control group’s expected conversion rate.
• Minimum detectable effect The smallest conversion rate difference it is possible to detect in an A/B Test.
• Statistical significance The likelihood that the observed difference in conversion rates is not due to chance.
• Minimum sample size The smallest number of visitors required to reliably detect a given conversion rate difference
Sample size calculators and statistical power
How many visitors do you need to see significant results?
Visitors needed to reach significance with Stats Engine
Improvement
5% 10% 25%
Baseline conversion rate
1% 458,900 101,600 13,000
5% 69,500 15,000 1,800
10% 29,200 6,200 700
25% 8,100 1,700 200
Lower conversion rate, lower effects = more visitors
One example of calculating your opportunity cost
12% minimum detectable effect
Original
Variation
Should you stop or continue a test?
Should you stop or continue a test?
Is my test significant?
Congrats
Can I afford to wait?
Continue
Stop
Accept lowersignificance
Concede inconclusive
Yes
NoNo
Yes
Should you stop or continue a test?
Is my test significant?
Congrats
Can I afford to wait?
Continue
Stop
Accept lowersignificance
Concede inconclusive
Yes
NoNo
Yes
Difference intervals
Difference intervals
15.4
11.6 Middle Ground
Best Case
Worst case
7.3
Seasonality• We DO take into account seasonality while a test is running.
• We DO NOT take into account future seasonality after an experiment is stopped.
Should you stop or continue a test?
Is my test significant?
Congrats
Can I afford to wait?
Continue
Stop
Accept lowersignificance
Concede inconclusive
Yes
NoNo
Yes
• Use Difference Intervals to understand the types of lifts you could see.
Should you stop or continue a test?
Is my test significant?
Congrats
Can I afford to wait?
Continue
Stop
Accept lowersignificance
Concede inconclusive
Yes
NoNo
Yes
A reasonable number of visitors left, relative to your traffic?
Visitors Remaining Explained
Improvement Increases in Magnitude
Improvement Stays the Same
Improvement Decreases in Magnitude
A good idea to wait
5761 + 3200 Note!
Should you stop or continue a test?
Is my test significant?
Congrats
Can I afford to wait?
Continue
Stop
Accept lowersignificance
Concede inconclusive
Yes
NoNo
Yes
• Use Visitors Remaining to evaluate if waiting makes sense.
Should you stop or continue a test?
Is my test significant?
Congrats
Can I afford to wait?
Continue
Stop
Accept lowersignificance
Concede inconclusive
Yes
NoNo
Yes
If you’re an organization that can
• Iterate quickly on new variations • Run lots of experiments • Have little downside risk of implementing non-winning variations
then you can likely tolerate a higher error rate.
Difference intervals can guide your decision
Should you stop or continue a test?
Is my test significant?
Congrats
Can I afford to wait?
Continue
Stop
Accept lowersignificance
Concede inconclusive
Yes
NoNo
Yes • Use Difference Intervals to measure risk you take on
> 100,000 visitors? What next?
Time to move on to the next idea
Should you stop or continue a test?
Is my test significant?
Congrats
Can I afford to wait?
Continue
Stop
Accept lowersignificance
Concede inconclusive
Yes
NoNo
Yes
• Use Visitors Remaining to evaluate if waiting makes sense.
Recap
Is my test significant?
Congrats
Can I afford to wait?
Continue
Stop
Accept lowersignificance
Concede inconclusive
Yes
NoNo
Yes
• Use Difference Intervals to understand the types of lifts you could see.
Recap
Is my test significant?
Congrats
Can I afford to wait?
Continue
Stop
Accept lowersignificance
Concede inconclusive
Yes
NoNo
Yes
• Use Difference Intervals to understand the types of lifts you could see.
Can I afford to wait?
Continue
Stop
Concede inconclusive
Yes
NoNo
• Use Visitors Remaining to evaluate if waiting makes sense.
Recap
Is my test significant?
Congrats
Can I afford to wait?
Continue
Stop
Accept lowersignificance
Concede inconclusive
Yes
NoNo
Yes
• Use Difference Intervals to understand the types of lifts you could see.
Can I afford to wait?
Continue
Stop
Concede inconclusive
Yes
NoNo
• Use Visitors Remaining to evaluate if waiting makes sense.
Stop
Accept lowersignificance
Concede inconclusive
Yes
No
Can I afford to wait?
• Use Difference Intervals to measure risk you take on
Recap
Is my test significant?
Congrats
Can I afford to wait?
Continue
Stop
Accept lowersignificance
Concede inconclusive
Yes
NoNo
Yes
• Use Difference Intervals to understand the types of lifts you could see.
Can I afford to wait?
Continue
Stop
Concede inconclusive
Yes
NoNo
• Use Visitors Remaining to evaluate if waiting makes sense.
Stop
Accept lowersignificance
Concede inconclusive
Yes
No
Can I afford to wait?
• Use Difference Intervals to measure risk you take on
• Use Visitors Remaining to evaluate if waiting makes sense.
Tuning your testing strategy for your traffic and business
1%6% 4% 3% inconclusive inconclusive inconclusiveinconclusive 18%
Looking for small effects?
• Tests take longer to reach significance
• Might find more winners, if you are willing to wait long enough
inconclusive
Testing for larger effects?
• Run more tests, faster
• Know when it’s time to move on to the next idea
Frequently asked questions
Do I need to re-run my historical tests?
Is this a one-tailed or two-tailed test? Why did you switch?
Why does Stats Engine report 0% Statistical Significance when other tools
report higher values?
Why does Statistical Significance increase step-wise?
If I take the results I see in Optimizely and plug them into any other statistics
calculator, the statistical significance is different. Why?
How does Stats Engine handle revenue calculations?
Questions?