The Optimizely Experience - Kaiser Fung
Post on 01-Nov-2014
209 views
Embed Size (px)
DESCRIPTION
This presentation was given at the Optimizely Experience New York City by data scientist, Kaiser Fung. The title of his presentation is 4 questions about testing you wanted to ask but didn't.TRANSCRIPT
- 1. KAISER FUNG Aug. 26, 2014 New York City kaiserdatamatter@gmail Q4 ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
- 2. Testing has played a big part in my career (c) Kaiser Fung
- 3. Q4 ~ Testing U wanted 2 But Didnt 1. Why worry about statistical signicance? 2. Can I use a simple sample size formula? 3. Is 90% condence too risk-averse? 4. What is the value of testing? (c) Kaiser Fung
- 4. Varying Copy on a Conversion Page (c) Kaiser Fung
- 5. Not Statistically Signicant (c) Kaiser Fung
- 6. Q*Q3 ~ Testing U wanted 2 But Didnt 1 Why should we care about statistical signicance? (c) Kaiser Fung
- 7. (c) Kaiser Fung
- 8. (c) Kaiser Fung
- 9. Statistical Noise (c) Kaiser Fung
- 10. (c) Kaiser Fung
- 11. (c) Kaiser Fung
- 12. (c) Kaiser Fung
- 13. (c) Kaiser Fung
- 14. Statistical Noise (c) Kaiser Fung
- 15. Icon by Laura Begg 40,000 (c) Kaiser Fung
- 16. 20,000 (c) Kaiser Fung
- 17. 30,000 (c) Kaiser Fung
- 18. 50:50 (c) Kaiser Fung
- 19. 15k 15k (c) Kaiser Fung
- 20. 125 (c) Kaiser Fung
- 21. 2 Can we use a simple sample sizing formula? Q*Q*Q2 ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
- 22. 10%90% (c) Kaiser Fung
- 23. More Samples More Precision (c) Kaiser Fung
- 24. (c) Kaiser Fung
- 25. 3 Is 90% condence too risk averse? Q2*Q*Q ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
- 26. OMG (c) Kaiser Fung
- 27. 90% Condence means: If the analyst tells me the test is signicant, there is a 90% chance this effect is really there (c) Kaiser Fung
- 28. 90% Condence means: If the analyst tells me the test is signicant, there is a 90% chance this effect is really there Good Question Wrong Definition (c) Kaiser Fung
- 29. 90% Condence means: If the effect is really there, there is a 90% chance the analyst will tell me the test is signicant (c) Kaiser Fung
- 30. 90% Condence means: If the effect is really there, there is a 90% chance the analyst will tell me the test is signicant Correct Definition Wrong Question (c) Kaiser Fung
- 31. Positive Predictive Value: If the analyst tells me the test is signicant, what is the probability this effect is really there? (c) Kaiser Fung
- 32. 90% Condence = 63% PPV (c) Kaiser Fung
- 33. PPV = 45-60% (c) Kaiser Fung
- 34. Chapter 4: Timid Testers / Magic Lassos (c) Kaiser Fung
- 35. 4 Whats the value of testing? Q3*Q ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
- 36. BEFORE AFTER xkcd (c) Kaiser Fung
- 37. B.T. 30% 70% 63% 37% A.T. (c) Kaiser Fung
- 38. 1. Why statistical signicance 2. How sample sizing 3. What condence level 4. Why test Statistical Noise Safe Landing Measure of Risk PPV Lift A/A Test Sq-root Law PPV Bayes Q4 ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
- 39. KAISER FUNG Strategic data advisory services, including test planning & analysis Speaking on business analytics & dataviz Training in statistical thinking (c) Kaiser Fung
- 40. Thank you Kaiser Fung kaiserdatamatter@gmail.com Twitter: @junkcharts LinkedIn (c) Kaiser Fung
