Transcript
Page 1: The Optimizely Experience - Kaiser Fung

KAISER FUNGAug. 26, 2014

New York Citykaiserdatamatter@gmail

Q4

~ Testing

U wanted 2

But Didn’t

(c) Kaiser Fung

Page 2: The Optimizely Experience - Kaiser Fung

Testing has played a big part in my career

(c) Kaiser Fung

Page 3: The Optimizely Experience - Kaiser Fung

Q4

~ Testing

U wanted 2

But Didn’t

1. Why worry about statistical significance?2. Can I use a simple sample size formula?3. Is 90% confidence too risk-averse?4. What is the value of testing?

(c) Kaiser Fung

Page 4: The Optimizely Experience - Kaiser Fung

Varying Copy on a Conversion Page

(c) Kaiser Fung

Page 5: The Optimizely Experience - Kaiser Fung

Not Statistically Significant

(c) Kaiser Fung

Page 6: The Optimizely Experience - Kaiser Fung

Q*Q3

~ Testing

U wanted 2

But Didn’t

1Why should we care about statistical significance?

(c) Kaiser Fung

Page 7: The Optimizely Experience - Kaiser Fung

(c) Kaiser Fung

Page 8: The Optimizely Experience - Kaiser Fung

(c) Kaiser Fung

Page 9: The Optimizely Experience - Kaiser Fung

Statistical Noise

(c) Kaiser Fung

Page 10: The Optimizely Experience - Kaiser Fung

(c) Kaiser Fung

Page 11: The Optimizely Experience - Kaiser Fung

(c) Kaiser Fung

Page 12: The Optimizely Experience - Kaiser Fung

(c) Kaiser Fung

Page 13: The Optimizely Experience - Kaiser Fung

(c) Kaiser Fung

Page 14: The Optimizely Experience - Kaiser Fung

Statistical Noise

(c) Kaiser Fung

Page 15: The Optimizely Experience - Kaiser Fung

Icon by Laura Begg

40,000

(c) Kaiser Fung

Page 16: The Optimizely Experience - Kaiser Fung

20,000

(c) Kaiser Fung

Page 17: The Optimizely Experience - Kaiser Fung

30,000

(c) Kaiser Fung

Page 18: The Optimizely Experience - Kaiser Fung

50:50

(c) Kaiser Fung

Page 19: The Optimizely Experience - Kaiser Fung

15k 15k(c) Kaiser Fung

Page 20: The Optimizely Experience - Kaiser Fung

125

(c) Kaiser Fung

Page 21: The Optimizely Experience - Kaiser Fung

2Can we use a simple sample sizing formula?

Q*Q*Q2

~ Testing

U wanted 2

But Didn’t

(c) Kaiser Fung

Page 22: The Optimizely Experience - Kaiser Fung

10%90%(c) Kaiser Fung

Page 23: The Optimizely Experience - Kaiser Fung

More Samples

More Precision

(c) Kaiser Fung

Page 24: The Optimizely Experience - Kaiser Fung

(c) Kaiser Fung

Page 25: The Optimizely Experience - Kaiser Fung

3Is 90% confidence too risk averse?

Q2*Q*Q~ Testing

U wanted 2

But Didn’t

(c) Kaiser Fung

Page 26: The Optimizely Experience - Kaiser Fung

OMG

(c) Kaiser Fung

Page 27: The Optimizely Experience - Kaiser Fung

90% Confidence means:If the analyst tells me the test is

significant, there is a 90% chance this effect is really there

(c) Kaiser Fung

Page 28: The Optimizely Experience - Kaiser Fung

90% Confidence means:If the analyst tells me the test is

significant, there is a 90% chance this effect is really there

Good Question – Wrong Definition

(c) Kaiser Fung

Page 29: The Optimizely Experience - Kaiser Fung

90% Confidence means:If the effect is really there, there is a 90% chance the analyst will tell me

the test is significant

(c) Kaiser Fung

Page 30: The Optimizely Experience - Kaiser Fung

90% Confidence means:If the effect is really there, there is a 90% chance the analyst will tell me

the test is significant

Correct Definition – Wrong Question

(c) Kaiser Fung

Page 31: The Optimizely Experience - Kaiser Fung

Positive Predictive Value:If the analyst tells me the test is

significant, what is the probability this effect is really there?

(c) Kaiser Fung

Page 32: The Optimizely Experience - Kaiser Fung

90% Confidence = 63% PPV

(c) Kaiser Fung

Page 33: The Optimizely Experience - Kaiser Fung

PPV = 45-60%(c) Kaiser Fung

Page 34: The Optimizely Experience - Kaiser Fung

Chapter 4:

Timid Testers / Magic Lassos

(c) Kaiser Fung

Page 35: The Optimizely Experience - Kaiser Fung

4What’s the value of testing?

Q3*Q~ Testing

U wanted 2

But Didn’t

(c) Kaiser Fung

Page 36: The Optimizely Experience - Kaiser Fung

BEFORE AFTERxkcd (c) Kaiser Fung

Page 37: The Optimizely Experience - Kaiser Fung

B.T.

30%

70% 63%

37%

A.T.

(c) Kaiser Fung

Page 38: The Optimizely Experience - Kaiser Fung

1. Why statistical significance2. How sample sizing3. What confidence level4. Why test

Statistical NoiseSafe LandingMeasure of RiskPPV Lift

A/A TestSq-root LawPPVBayes

Q4

~ Testing

U wanted 2

But Didn’t

(c) Kaiser Fung

Page 39: The Optimizely Experience - Kaiser Fung

KAISER FUNGStrategic data advisory services,

including test planning & analysisSpeaking on business analytics & dataviz

Training in statistical thinking (c) Kaiser Fung

Page 40: The Optimizely Experience - Kaiser Fung

Thank you

Kaiser [email protected]

Twitter: @junkchartsLinkedIn

(c) Kaiser Fung


Top Related