# The Optimizely Experience - Kaiser Fung

Post on 01-Nov-2014

209 views

Category:

Embed Size (px)

DESCRIPTION

This presentation was given at the Optimizely Experience New York City by data scientist, Kaiser Fung. The title of his presentation is 4 questions about testing you wanted to ask but didn't.

TRANSCRIPT

• 1. KAISER FUNG Aug. 26, 2014 New York City kaiserdatamatter@gmail Q4 ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
• 2. Testing has played a big part in my career (c) Kaiser Fung
• 3. Q4 ~ Testing U wanted 2 But Didnt 1. Why worry about statistical signicance? 2. Can I use a simple sample size formula? 3. Is 90% condence too risk-averse? 4. What is the value of testing? (c) Kaiser Fung
• 4. Varying Copy on a Conversion Page (c) Kaiser Fung
• 5. Not Statistically Signicant (c) Kaiser Fung
• 6. Q*Q3 ~ Testing U wanted 2 But Didnt 1 Why should we care about statistical signicance? (c) Kaiser Fung
• 7. (c) Kaiser Fung
• 8. (c) Kaiser Fung
• 9. Statistical Noise (c) Kaiser Fung
• 10. (c) Kaiser Fung
• 11. (c) Kaiser Fung
• 12. (c) Kaiser Fung
• 13. (c) Kaiser Fung
• 14. Statistical Noise (c) Kaiser Fung
• 15. Icon by Laura Begg 40,000 (c) Kaiser Fung
• 16. 20,000 (c) Kaiser Fung
• 17. 30,000 (c) Kaiser Fung
• 18. 50:50 (c) Kaiser Fung
• 19. 15k 15k (c) Kaiser Fung
• 20. 125 (c) Kaiser Fung
• 21. 2 Can we use a simple sample sizing formula? Q*Q*Q2 ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
• 22. 10%90% (c) Kaiser Fung
• 23. More Samples More Precision (c) Kaiser Fung
• 24. (c) Kaiser Fung
• 25. 3 Is 90% condence too risk averse? Q2*Q*Q ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
• 26. OMG (c) Kaiser Fung
• 27. 90% Condence means: If the analyst tells me the test is signicant, there is a 90% chance this effect is really there (c) Kaiser Fung
• 28. 90% Condence means: If the analyst tells me the test is signicant, there is a 90% chance this effect is really there Good Question Wrong Definition (c) Kaiser Fung
• 29. 90% Condence means: If the effect is really there, there is a 90% chance the analyst will tell me the test is signicant (c) Kaiser Fung
• 30. 90% Condence means: If the effect is really there, there is a 90% chance the analyst will tell me the test is signicant Correct Definition Wrong Question (c) Kaiser Fung
• 31. Positive Predictive Value: If the analyst tells me the test is signicant, what is the probability this effect is really there? (c) Kaiser Fung
• 32. 90% Condence = 63% PPV (c) Kaiser Fung
• 33. PPV = 45-60% (c) Kaiser Fung
• 34. Chapter 4: Timid Testers / Magic Lassos (c) Kaiser Fung
• 35. 4 Whats the value of testing? Q3*Q ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
• 36. BEFORE AFTER xkcd (c) Kaiser Fung
• 37. B.T. 30% 70% 63% 37% A.T. (c) Kaiser Fung
• 38. 1. Why statistical signicance 2. How sample sizing 3. What condence level 4. Why test Statistical Noise Safe Landing Measure of Risk PPV Lift A/A Test Sq-root Law PPV Bayes Q4 ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
• 39. KAISER FUNG Strategic data advisory services, including test planning & analysis Speaking on business analytics & dataviz Training in statistical thinking (c) Kaiser Fung
• 40. Thank you Kaiser Fung kaiserdatamatter@gmail.com Twitter: @junkcharts LinkedIn (c) Kaiser Fung