continuous delivery and rapid experimentation
DESCRIPTION
A presentation I gave at Agile Australia 2013 on Continuous DeliveryTRANSCRIPT
![Page 1: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/1.jpg)
Continuous Delivery
The Path to Rapid Experimentation
Paul Coia
![Page 2: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/2.jpg)
Deploy
Experiment
Example
Lessons
![Page 3: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/3.jpg)
![Page 4: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/4.jpg)
Global platform for independent artists and designers to sell
their work more easily, more oen.
![Page 5: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/5.jpg)
![Page 6: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/6.jpg)
![Page 7: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/7.jpg)
![Page 8: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/8.jpg)
10M designs on 40M products
500,000 artists all over the world.
40M pageviews from 7M Users/month
![Page 9: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/9.jpg)
Deploy
Experiment
Example
Lessons
![Page 10: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/10.jpg)
2000 releases 4 years
700 in 2012
Average 3 per day
![Page 11: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/11.jpg)
cap production deploy
![Page 12: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/12.jpg)
![Page 13: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/13.jpg)
1. Continuous Integration success
2. Deploy code via Capistrano
3. Monitor vital signs
![Page 14: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/14.jpg)
Jenkins Continuous Integration Server
![Page 15: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/15.jpg)
New Relic Application Monitoring
![Page 16: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/16.jpg)
New Relic Key Transaction Monitoring
![Page 17: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/17.jpg)
AirbrakeError Reporting
![Page 18: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/18.jpg)
Circonus and StatsdBusiness Metrics Monitoring
![Page 19: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/19.jpg)
StatsdClient.increment("Payment OK")
![Page 20: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/20.jpg)
if enrolled_in?(“new”).variant? // show variantelse // show normalend
![Page 21: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/21.jpg)
![Page 22: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/22.jpg)
Minimal cost of deploying
Rapid pace of change possible
Confidence make lots of changes
Try ideas quicker
Why we like it
![Page 23: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/23.jpg)
Deploy
Experiment
Example
Lessons
![Page 24: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/24.jpg)
1.Observe
2.Form a hypothesis
3.Devise an experiment to test it
4.Draw conclusions
Scientific Method
![Page 25: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/25.jpg)
Quantitive Research - What?
Qualitative Research - Why?
![Page 26: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/26.jpg)
“If the SEARCH box was bigger, then users will be more likely to search”
![Page 27: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/27.jpg)
![Page 28: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/28.jpg)
![Page 29: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/29.jpg)
![Page 30: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/30.jpg)
![Page 31: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/31.jpg)
Split users at point of variation
Isolate experiments
Persist the decision across visits
Exclude crawlers and ‘old’ users
Enrollment
![Page 32: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/32.jpg)
// ON LANDING PAGEif new_visitor() @guinea_pig = trueend
// ON EXPERIMENT PAGEif @guinea_pig && !retrieve_enrolment() experiment = enrol_in_experiment(“new”) persist_enrolment(experiment)end
if enrolled_in?(“new”).variant? // experiment logicelse // normal logicend
![Page 33: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/33.jpg)
RUNNING_EXPERIMENTS ={ ... new: {description: 'New feature', enrolment_percent: 20}, ...}
![Page 34: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/34.jpg)
Google AnalyticsData Collection
![Page 35: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/35.jpg)
‘R’ Statistical LanguageGoogle Analytics API
![Page 36: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/36.jpg)
![Page 37: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/37.jpg)
Respect Statistics
Establish your test duration up front
Wait for the test to complete
Results
![Page 38: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/38.jpg)
Model your Product funnel
Greater volume earlier in the funnel
Higher conversion rate between steps
Add lots of instrumentation
Useful metrics
![Page 39: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/39.jpg)
Homepage
Sign up page
Signed up
![Page 40: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/40.jpg)
1%
Homepage
Sign up page
Signed up
![Page 41: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/41.jpg)
1%
Homepage
Sign up page
Signed up
20%
5%
![Page 42: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/42.jpg)
‘Landing Page’ Optimizers
Client-side changes only
Simple experiments
![Page 43: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/43.jpg)
Trivial deployment
Feature toggle
Experimentation platform
= Trivial feature experimentation
![Page 44: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/44.jpg)
Deploy
Experiment
Example
Lessons
![Page 45: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/45.jpg)
22% less likely to Add to Cart
50% less likely to Checkout
Mobile Visitors
![Page 46: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/46.jpg)
![Page 47: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/47.jpg)
No change to Checkout Completion
![Page 48: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/48.jpg)
![Page 49: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/49.jpg)
Improvement in Checkout Completion
![Page 50: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/50.jpg)
![Page 51: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/51.jpg)
Increase in Add to Cart rate
![Page 52: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/52.jpg)
![Page 53: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/53.jpg)
Further Increase in Add to Cart rate
![Page 54: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/54.jpg)
Multiple lightweight experiments
A handful of weeks
Verified 30% increase in Purchasing
![Page 55: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/55.jpg)
Deploy
Experiment
Example
Lessons
![Page 56: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/56.jpg)
Invest in your deployment tools
![Page 57: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/57.jpg)
Experiment on the small scale
![Page 58: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/58.jpg)
Expect to be wrong
![Page 59: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/59.jpg)
Build on the successes
![Page 60: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/60.jpg)
Design for incremental change
![Page 62: Continuous Delivery and Rapid Experimentation](https://reader034.vdocuments.mx/reader034/viewer/2022051514/54b73ee94a7959be4c8b4877/html5/thumbnails/62.jpg)