Download - The Optimizely Experience - Kaiser Fung
KAISER FUNGAug. 26, 2014
New York Citykaiserdatamatter@gmail
Q4
~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung
Testing has played a big part in my career
(c) Kaiser Fung
Q4
~ Testing
U wanted 2
But Didn’t
1. Why worry about statistical significance?2. Can I use a simple sample size formula?3. Is 90% confidence too risk-averse?4. What is the value of testing?
(c) Kaiser Fung
Varying Copy on a Conversion Page
(c) Kaiser Fung
Not Statistically Significant
(c) Kaiser Fung
Q*Q3
~ Testing
U wanted 2
But Didn’t
1Why should we care about statistical significance?
(c) Kaiser Fung
(c) Kaiser Fung
(c) Kaiser Fung
Statistical Noise
(c) Kaiser Fung
(c) Kaiser Fung
(c) Kaiser Fung
(c) Kaiser Fung
(c) Kaiser Fung
Statistical Noise
(c) Kaiser Fung
Icon by Laura Begg
40,000
(c) Kaiser Fung
20,000
(c) Kaiser Fung
30,000
(c) Kaiser Fung
50:50
(c) Kaiser Fung
15k 15k(c) Kaiser Fung
125
(c) Kaiser Fung
2Can we use a simple sample sizing formula?
Q*Q*Q2
~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung
10%90%(c) Kaiser Fung
More Samples
More Precision
(c) Kaiser Fung
(c) Kaiser Fung
3Is 90% confidence too risk averse?
Q2*Q*Q~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung
OMG
(c) Kaiser Fung
90% Confidence means:If the analyst tells me the test is
significant, there is a 90% chance this effect is really there
(c) Kaiser Fung
90% Confidence means:If the analyst tells me the test is
significant, there is a 90% chance this effect is really there
Good Question – Wrong Definition
(c) Kaiser Fung
90% Confidence means:If the effect is really there, there is a 90% chance the analyst will tell me
the test is significant
(c) Kaiser Fung
90% Confidence means:If the effect is really there, there is a 90% chance the analyst will tell me
the test is significant
Correct Definition – Wrong Question
(c) Kaiser Fung
Positive Predictive Value:If the analyst tells me the test is
significant, what is the probability this effect is really there?
(c) Kaiser Fung
90% Confidence = 63% PPV
(c) Kaiser Fung
PPV = 45-60%(c) Kaiser Fung
Chapter 4:
Timid Testers / Magic Lassos
(c) Kaiser Fung
4What’s the value of testing?
Q3*Q~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung
BEFORE AFTERxkcd (c) Kaiser Fung
B.T.
30%
70% 63%
37%
A.T.
(c) Kaiser Fung
1. Why statistical significance2. How sample sizing3. What confidence level4. Why test
Statistical NoiseSafe LandingMeasure of RiskPPV Lift
A/A TestSq-root LawPPVBayes
Q4
~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung
KAISER FUNGStrategic data advisory services,
including test planning & analysisSpeaking on business analytics & dataviz
Training in statistical thinking (c) Kaiser Fung