The Optimizely Experience - Kaiser Fung

Download The Optimizely Experience - Kaiser Fung

Post on 01-Nov-2014

209 views

Category:

Business

0 download

Embed Size (px)

DESCRIPTION

This presentation was given at the Optimizely Experience New York City by data scientist, Kaiser Fung. The title of his presentation is 4 questions about testing you wanted to ask but didn't.

TRANSCRIPT

  • 1. KAISER FUNG Aug. 26, 2014 New York City kaiserdatamatter@gmail Q4 ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
  • 2. Testing has played a big part in my career (c) Kaiser Fung
  • 3. Q4 ~ Testing U wanted 2 But Didnt 1. Why worry about statistical signicance? 2. Can I use a simple sample size formula? 3. Is 90% condence too risk-averse? 4. What is the value of testing? (c) Kaiser Fung
  • 4. Varying Copy on a Conversion Page (c) Kaiser Fung
  • 5. Not Statistically Signicant (c) Kaiser Fung
  • 6. Q*Q3 ~ Testing U wanted 2 But Didnt 1 Why should we care about statistical signicance? (c) Kaiser Fung
  • 7. (c) Kaiser Fung
  • 8. (c) Kaiser Fung
  • 9. Statistical Noise (c) Kaiser Fung
  • 10. (c) Kaiser Fung
  • 11. (c) Kaiser Fung
  • 12. (c) Kaiser Fung
  • 13. (c) Kaiser Fung
  • 14. Statistical Noise (c) Kaiser Fung
  • 15. Icon by Laura Begg 40,000 (c) Kaiser Fung
  • 16. 20,000 (c) Kaiser Fung
  • 17. 30,000 (c) Kaiser Fung
  • 18. 50:50 (c) Kaiser Fung
  • 19. 15k 15k (c) Kaiser Fung
  • 20. 125 (c) Kaiser Fung
  • 21. 2 Can we use a simple sample sizing formula? Q*Q*Q2 ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
  • 22. 10%90% (c) Kaiser Fung
  • 23. More Samples More Precision (c) Kaiser Fung
  • 24. (c) Kaiser Fung
  • 25. 3 Is 90% condence too risk averse? Q2*Q*Q ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
  • 26. OMG (c) Kaiser Fung
  • 27. 90% Condence means: If the analyst tells me the test is signicant, there is a 90% chance this effect is really there (c) Kaiser Fung
  • 28. 90% Condence means: If the analyst tells me the test is signicant, there is a 90% chance this effect is really there Good Question Wrong Definition (c) Kaiser Fung
  • 29. 90% Condence means: If the effect is really there, there is a 90% chance the analyst will tell me the test is signicant (c) Kaiser Fung
  • 30. 90% Condence means: If the effect is really there, there is a 90% chance the analyst will tell me the test is signicant Correct Definition Wrong Question (c) Kaiser Fung
  • 31. Positive Predictive Value: If the analyst tells me the test is signicant, what is the probability this effect is really there? (c) Kaiser Fung
  • 32. 90% Condence = 63% PPV (c) Kaiser Fung
  • 33. PPV = 45-60% (c) Kaiser Fung
  • 34. Chapter 4: Timid Testers / Magic Lassos (c) Kaiser Fung
  • 35. 4 Whats the value of testing? Q3*Q ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
  • 36. BEFORE AFTER xkcd (c) Kaiser Fung
  • 37. B.T. 30% 70% 63% 37% A.T. (c) Kaiser Fung
  • 38. 1. Why statistical signicance 2. How sample sizing 3. What condence level 4. Why test Statistical Noise Safe Landing Measure of Risk PPV Lift A/A Test Sq-root Law PPV Bayes Q4 ~ Testing U wanted 2 But Didnt (c) Kaiser Fung
  • 39. KAISER FUNG Strategic data advisory services, including test planning & analysis Speaking on business analytics & dataviz Training in statistical thinking (c) Kaiser Fung
  • 40. Thank you Kaiser Fung kaiserdatamatter@gmail.com Twitter: @junkcharts LinkedIn (c) Kaiser Fung