strata new york 2012: continuous experimentation with continuous deployment
DESCRIPTION
Continuous deployment is core to Etsy, and we push public facing code over 30 times per day. Evaluating an experiment amidst this shifting landscape is a difficult task as our traditional methods of monitoring operational metrics don’t provide enough information to make product-level decisions. To this end, we have developed internal tooling for deep analytics that enables us to systematically analyze our experimental results in a continuously changing environment. This talk will focus on the analysis framework that we have built – from the raw logging data, to our elastic mapreduce-based data transformations, to the final dashboards and underlying statistics that drive decision making.TRANSCRIPT
![Page 1: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/1.jpg)
Continuous Experimentationwith Continuous
Deployment
Steve MardenfeldEtsy
Monday, November 5, 2012
![Page 2: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/2.jpg)
Monday, November 5, 2012
![Page 3: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/3.jpg)
World’s Handmade Market
Monday, November 5, 2012
![Page 4: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/4.jpg)
Continuous Deployment
Monday, November 5, 2012
![Page 5: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/5.jpg)
Monday, November 5, 2012
![Page 6: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/6.jpg)
Monday, November 5, 2012
![Page 7: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/7.jpg)
Metrics
Monday, November 5, 2012
![Page 8: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/8.jpg)
Graph Factory
Monday, November 5, 2012
![Page 9: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/9.jpg)
Oops
Monday, November 5, 2012
![Page 10: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/10.jpg)
Oops
Monday, November 5, 2012
![Page 11: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/11.jpg)
Good Enough?
Monday, November 5, 2012
![Page 12: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/12.jpg)
What is Better?
Monday, November 5, 2012
![Page 13: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/13.jpg)
Sometimes it’s Obvious
Monday, November 5, 2012
![Page 14: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/14.jpg)
Most Times it’s Not
Monday, November 5, 2012
![Page 15: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/15.jpg)
Most Times it’s Not
Cart Page Restyled
Monday, November 5, 2012
![Page 16: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/16.jpg)
Most Times it’s Not
Etsy Mentioned on Oprah
Internet Goes Down in Brooklyn
Justin Beiber Sneezes
New Cat Meme
Cart Page Restyled
Monday, November 5, 2012
![Page 17: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/17.jpg)
A/B Testing?Monday, November 5, 2012
![Page 18: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/18.jpg)
Monday, November 5, 2012
![Page 19: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/19.jpg)
Monday, November 5, 2012
![Page 20: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/20.jpg)
Test Them Both
Monday, November 5, 2012
![Page 21: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/21.jpg)
Statistics?
Monday, November 5, 2012
![Page 22: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/22.jpg)
First Pass
HADOOP
Monday, November 5, 2012
![Page 23: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/23.jpg)
First Pass
CASCADING
Monday, November 5, 2012
![Page 24: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/24.jpg)
First Pass
R
Monday, November 5, 2012
![Page 25: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/25.jpg)
First Pass
RHADOOP
CASCADING
Monday, November 5, 2012
![Page 26: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/26.jpg)
Automate It
A/B Analyzer
Monday, November 5, 2012
![Page 27: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/27.jpg)
A/B Analyzer
•Framework for looking at experiments
•Done for every experiment beforehand
Monday, November 5, 2012
![Page 28: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/28.jpg)
A/B Analyzer
Monday, November 5, 2012
![Page 29: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/29.jpg)
A/B Analyzer
Monday, November 5, 2012
![Page 30: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/30.jpg)
Monday, November 5, 2012
![Page 31: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/31.jpg)
3 Types of Metrics
•Co-occurence
•visits that did X (searched)
•average number of X (searches)
•Funnels
•Special Cases*
Monday, November 5, 2012
![Page 32: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/32.jpg)
Co-Occurence
•Co-occurence of all actions with the variant
•i.e. - how many people saw this variant and then purchased
•i.e. - average number of purchases per variant
Monday, November 5, 2012
![Page 33: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/33.jpg)
Funnels
BuySearch Listing Buy
Monday, November 5, 2012
![Page 34: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/34.jpg)
Funnels
Buy
BuySearch Listing Buy
Search Listing BuyHome
Monday, November 5, 2012
![Page 35: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/35.jpg)
Funnels
BuyBuy
BuySearch Listing Buy
Search Listing BuyHome
Monday, November 5, 2012
![Page 36: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/36.jpg)
Special Cases
•Exiting pages
•Bounce rate
•Visit length
Monday, November 5, 2012
![Page 37: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/37.jpg)
Pipeline
HDFS Events Visits MySql
A/B
PROCESSING
Client
Logs
DB
COLLECTION
Monday, November 5, 2012
![Page 38: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/38.jpg)
Bucketing
Home Search Listing Buy
Search A/B Test
Monday, November 5, 2012
![Page 39: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/39.jpg)
Bucketing
Home Search Listing Buy
Search A/B Test
Monday, November 5, 2012
![Page 40: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/40.jpg)
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
![Page 41: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/41.jpg)
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
![Page 42: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/42.jpg)
Idea
Monday, November 5, 2012
![Page 43: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/43.jpg)
Changing Search
Monday, November 5, 2012
![Page 44: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/44.jpg)
Related Searches?
Monday, November 5, 2012
![Page 45: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/45.jpg)
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
![Page 46: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/46.jpg)
Be Specific
•What does an improvement look like?
•More clicks?
•Less searches?
•Shorter visits?
•Longer visits?
Monday, November 5, 2012
![Page 47: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/47.jpg)
Bet Driven Development
Monday, November 5, 2012
![Page 48: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/48.jpg)
Is this even worth it?
Monday, November 5, 2012
![Page 49: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/49.jpg)
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
![Page 50: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/50.jpg)
Code It Up
Monday, November 5, 2012
![Page 51: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/51.jpg)
Code
Monday, November 5, 2012
![Page 52: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/52.jpg)
Turn it On
Monday, November 5, 2012
![Page 53: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/53.jpg)
Process
•Idea
•Hypothesis
•Deploy
•Wait
•Analysis
Monday, November 5, 2012
![Page 54: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/54.jpg)
Realtime?
Monday, November 5, 2012
![Page 55: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/55.jpg)
Wait
Monday, November 5, 2012
![Page 56: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/56.jpg)
Process
•Idea
•Hypothesis
•Code
•Wait
•Analysis
Monday, November 5, 2012
![Page 57: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/57.jpg)
Making Sense of ResultsMonday, November 5, 2012
![Page 58: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/58.jpg)
Back To Related Searches
Monday, November 5, 2012
![Page 59: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/59.jpg)
Back To Related Searches
Monday, November 5, 2012
![Page 60: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/60.jpg)
Monday, November 5, 2012
![Page 61: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/61.jpg)
Monday, November 5, 2012
![Page 62: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/62.jpg)
Final Thoughts
•Bet driven development - be explicit about your changes
•Easy experimental setup and quick analysis makes for more experiments
•An experiment is successful if you learned something, not if you launched
Monday, November 5, 2012
![Page 63: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/63.jpg)
Monday, November 5, 2012
![Page 64: Strata New York 2012: Continuous Experimentation with Continuous Deployment](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552c3b0b4c90581158b496e/html5/thumbnails/64.jpg)
http://www.slideshare.net/powerlexis/retro-slides-from-ibm
Monday, November 5, 2012