Download - Continuous Delivery: The Dirty Details
![Page 2: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/2.jpg)
a.k.a. “Continuous Deployment”
![Page 3: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/3.jpg)
![Page 4: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/4.jpg)
www. .com
![Page 5: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/5.jpg)
![Page 6: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/6.jpg)
AUGUST 20121.4 Billion page viewsUSD $76 Million in transactions3.8 Million items sold
http://www.etsy.com/blog/news/2012/etsy-statistics-august-2012-weather-report/
![Page 7: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/7.jpg)
~170 Committers, everyone deploys
credit: martin_heigan (flickr)
![Page 8: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/8.jpg)
Very end of 2009Today
30
20
10
40
![Page 9: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/9.jpg)
Continuous delivery is a pattern language in growing use in software development to improve the process of software delivery. Techniques such as automated testing, continuous integration, and continuous deployment allow software to be developed to a high standard and easily packaged and deployed to test environments, resulting in the ability to rapidly, reliably and repeatedly push out enhancements and bug fixes to customers at low risk and with minimal manual overhead. The technique was one of the assumptions of extreme programming but at an enterprise level has developed into a discipline of its own, with job descriptions for roles such as "buildmaster" calling for CD skills as mandatory. ~wikipedia
![Page 10: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/10.jpg)
+ DevOps+ Working on mainline, trunk, master+ Feature flags+ Branching in code
![Page 11: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/11.jpg)
An Apology
![Page 12: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/12.jpg)
We build primarily in PHP.Please don’t run away!
An Apology
![Page 13: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/13.jpg)
“Continuous Deploymentin Practice at Etsy”
The Dirty Details of...
![Page 14: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/14.jpg)
![Page 15: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/15.jpg)
2010-today2009Then Now
Just before we started using CD
![Page 16: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/16.jpg)
15 mins6-14 hoursThen
1 person“Deployment Army”
Now
Rapid releasecycle
Highly orchestratedand infrequent
![Page 17: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/17.jpg)
Commonplace andhappens so often
we cannot keep up
Special event andhighly disruptive
Then Now
![Page 18: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/18.jpg)
Blocked for15 minutes,
next deploy willonly take
15 minutes
Config flags <5 mins
Blocked for6-14 hours,
plus minimum of6 hours toredeploy
Then Now
![Page 19: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/19.jpg)
Mainline,minimal linking
and building,rsync,site up
Release branch,database schemas,
data transforms,packaging,
rolling restarts,cache purging,
scheduled downtime
Then Now
![Page 20: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/20.jpg)
FastSimple
Common
SlowComplexSpecial
Then Now
![Page 21: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/21.jpg)
Deploying code is the very first thingengineers learn to do at Etsy.
![Page 22: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/22.jpg)
1st dayAdd your photo to Etsy.com.
![Page 23: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/23.jpg)
2nd dayComplete tax, insurance, and benefits forms.
1st dayAdd your photo to Etsy.com.
![Page 24: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/24.jpg)
![Page 25: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/25.jpg)
WARNING
![Page 26: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/26.jpg)
![Page 27: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/27.jpg)
Continuous DeploymentSmall, frequent changes.
Constantly integrating into production.30 deploys per day.
![Page 28: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/28.jpg)
“Wow... 30 deploys a day.How do you build features so quickly?”
![Page 29: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/29.jpg)
Software Deploy ≠ Product Launch
![Page 30: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/30.jpg)
Deploys frequently gated by config flags(“dark” releases)
![Page 31: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/31.jpg)
$cfg[‘new_search’] = array('enabled' => 'off');$cfg[‘sign_in’] = array('enabled' => 'on');$cfg[‘checkout’] = array('enabled' => 'on');$cfg[‘homepage’] = array('enabled' => 'on');
![Page 32: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/32.jpg)
$cfg[‘new_search’] = array('enabled' => 'off');
![Page 33: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/33.jpg)
$cfg[‘new_search’] = array('enabled' => 'off');
// Meanwhile...
# old and boring search$results = do_grep();
![Page 34: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/34.jpg)
$cfg[‘new_search’] = array('enabled' => 'off');
// Meanwhile...
if ($cfg[‘new_search’] == ‘on’) { # New and fancy search $results = do_solr();} else { # old and boring search $results = do_grep();}
![Page 35: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/35.jpg)
$cfg[‘new_search’] = array('enabled' => 'on');
// or...
$cfg[‘new_search’] = array('enabled' => 'staff');
// or...
$cfg[‘new_search’] = array('enabled' => '1%');
// or...
$cfg[‘new_search’] = array('enabled' => 'users', 'user_list' => 'mike,john,kellan');
![Page 36: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/36.jpg)
Validate in production, hidden from public.
![Page 37: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/37.jpg)
Small incremental changes to the applicationNew classes, methods, controllersGraphics, stylesheets, templatesCopy/content changes
Turning flags on/off, or ramping up
What’s in a deploy?
![Page 38: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/38.jpg)
Security, bugs, traffic, load shedding,adding/removing infrastructure.
Tweaking config flags or releasing patches.
Quickly Responding to issues
![Page 39: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/39.jpg)
http://www.flickr.com/photos/flyforfun/2694158656/
![Page 40: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/40.jpg)
http://www.flickr.com/photos/flyforfun/2694158656/
OperatorConfig flags
Metrics
![Page 41: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/41.jpg)
![Page 42: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/42.jpg)
![Page 43: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/43.jpg)
“How do you continuously deploy database schema changes?”
![Page 44: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/44.jpg)
Code deploys: ~ every 15-20 minutesSchema changes: Thursday
![Page 45: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/45.jpg)
Our web application is largely monolithic.
![Page 46: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/46.jpg)
Etsy.com, support tools, developer API,back-office, analytics
![Page 47: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/47.jpg)
External “services” are not deployed with the main application.
![Page 48: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/48.jpg)
Databases, Search, Photo storage
![Page 49: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/49.jpg)
For every config flag, there are two stateswe can support — forward and backward.
![Page 50: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/50.jpg)
Expose multiple versions in each service.Expect multiple versions in the application.
![Page 51: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/51.jpg)
Example: Changing a Database Schema
![Page 52: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/52.jpg)
Prefer ADDs over ALTERs (“non-breaking expansions”)
![Page 53: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/53.jpg)
Altering in-place requires couplingcode and schema changes.
![Page 54: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/54.jpg)
Merging “users” and “users_prefs”
![Page 55: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/55.jpg)
1. Write to both versions2. Backfill historical data3. Read from new version4. Cut-off writes to old version
![Page 56: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/56.jpg)
0. Add new version to schema1. Write to both versions2. Backfill historical data3. Read from new version4. Cut-off writes to old version
![Page 57: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/57.jpg)
0. Add new version to schemaSchema change to add prefs columns to “users” table.
“write_prefs_to_user_prefs_table” => “on”“write_prefs_to_users_table” => “off”“read_prefs_from_users_table” => “off”
![Page 58: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/58.jpg)
1. Write to both versionsWrite code for writing prefs to the “users” table.
“write_prefs_to_user_prefs_table” => “on”“write_prefs_to_users_table” => “on”“read_prefs_from_users_table” => “off”
![Page 59: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/59.jpg)
2. Backfill historical dataOffline process to sync existing data from “user_prefs”to new columns in “users”
![Page 60: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/60.jpg)
3. Read from new versionData validation tests. Ensure consistency both internallyand in production.
“write_prefs_to_user_prefs_table” => “on”“write_prefs_to_users_table” => “on”“read_prefs_from_users_table” => “staff”
![Page 61: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/61.jpg)
3. Read from new versionData validation tests. Ensure consistency both internallyand in production.
“write_prefs_to_user_prefs_table” => “on”“write_prefs_to_users_table” => “on”“read_prefs_from_users_table” => “1%”
![Page 62: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/62.jpg)
3. Read from new versionData validation tests. Ensure consistency both internallyand in production.
“write_prefs_to_user_prefs_table” => “on”“write_prefs_to_users_table” => “on”“read_prefs_from_users_table” => “5%”
![Page 63: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/63.jpg)
3. Read from new versionData validation tests. Ensure consistency both internallyand in production.
“write_prefs_to_user_prefs_table” => “on”“write_prefs_to_users_table” => “on”“read_prefs_from_users_table” => “on”
(“on” == “100%”)
![Page 64: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/64.jpg)
4. Cut-off writes to old versionAfter running on the new table for a significant amount of time, we can cut off writes to the old table.
“write_prefs_to_user_prefs_table” => “off”“write_prefs_to_users_table” => “on”“read_prefs_from_users_table” => “on”
![Page 65: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/65.jpg)
“Branch by Astraction”
Controller Controller
Users Model
“users” (old) “user_prefs” “users”
old schema new schema
(Abstraction)
http://paulhammant.com/blog/branch_by_abstraction.htmlhttp://continuousdelivery.com/2011/05/make-large-scale-changes-incrementally-with-branch-by-abstraction/
![Page 66: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/66.jpg)
1. Write to both versions2. Backfill historical data3. Read from new version4. Cut-off writes to old version
“The Migration 4-Step”
![Page 67: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/67.jpg)
1. Write to both versions2. Backfill historical data3. Read from new version4. Cut-off writes to old version5. Clean up flags, code, columns (when?)
“The Migration 4-Step”
![Page 68: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/68.jpg)
Architecture and Process
![Page 69: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/69.jpg)
Deploying is cheap.
![Page 70: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/70.jpg)
Some philosophies on product development...
![Page 71: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/71.jpg)
Gathering data should be cheap, too.
staff, opt-in prototypes, 1%
![Page 72: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/72.jpg)
Treat first iterations as experiments.
![Page 73: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/73.jpg)
Get into code as quickly as possible.
![Page 74: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/74.jpg)
Architecture largely doesn’t matter.
![Page 75: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/75.jpg)
Kill things that don’t work.
![Page 76: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/76.jpg)
“Terminate with extreme predjudice.”
![Page 77: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/77.jpg)
Is the dumb solution enough to build a product?How long will the dumb solution last?
![Page 78: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/78.jpg)
Your assumptions will be wrongonce you’ve scaled 10x.
![Page 79: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/79.jpg)
“We don’t optimize for being right. We optimize for quickly detecting when we’re wrong.”
~Kellan Elliott-McCrea, CTO
![Page 80: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/80.jpg)
Become really good at changingyour architecture.
![Page 81: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/81.jpg)
Invest time in architecture by the2nd or 3rd iteration.
![Page 82: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/82.jpg)
Integration and Operations
![Page 83: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/83.jpg)
Continuous DeploymentSmall, frequent changes.
Constantly integrating into production.30 deploys per day.
![Page 84: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/84.jpg)
Code review before commit
![Page 85: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/85.jpg)
Automated tests before deploy
![Page 86: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/86.jpg)
Why Integrate with Production?
![Page 87: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/87.jpg)
Dev ≠ Prod
![Page 88: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/88.jpg)
Verify frequently and in small batches.
![Page 89: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/89.jpg)
Integrating with production is a test in itself.We do this frequently and in small batches.
![Page 90: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/90.jpg)
"Production is truly the only place youcan validate your code."
![Page 91: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/91.jpg)
"Production is truly the only place youcan validate your code."
~ Michael Nygard, about 40 min ago
![Page 92: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/92.jpg)
More database servers in prod.Bigger database hardware in prod.More web servers.Various replication schemes.Different versions of server and OS software.Schema changes applied at different times.Physical hardware in prod.More data in prod.Legacy data (7 years of odd user states).More traffic in prod.Wait, I mean MUCH more traffic in prod.Fewer elves.Faster disks (SSDs) in prod.
![Page 93: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/93.jpg)
Using a MySQL database to test an application that will eventually be deployed on Oracle:
![Page 94: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/94.jpg)
Using a MySQL database to test an application that will eventually be deployed on Oracle: Priceless.
![Page 95: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/95.jpg)
Verify frequently and in small batches.
![Page 96: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/96.jpg)
Dev ≠ Prod
![Page 97: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/97.jpg)
Dev ⇾ QA ⇾ Staging ⇾ Prod
![Page 98: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/98.jpg)
Dev ⇾ QA ⇾ Staging ⇾ Prod
![Page 99: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/99.jpg)
Dev ⇾ Pre-Prod ⇾ Prod
![Page 100: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/100.jpg)
Test and integrate where you’ll see value.
![Page 101: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/101.jpg)
Config flags (again)
off, on, staff, opt-in prototypes, user list, 0-100%
![Page 102: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/102.jpg)
Config flags (again)
off, on, staff, opt-in prototypes, user list, 0-100%
“canary pools”
![Page 103: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/103.jpg)
Automated tests after deploy
![Page 104: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/104.jpg)
Real-time metrics and dashboardsNetwork & Servers, Application, Business
![Page 105: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/105.jpg)
![Page 106: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/106.jpg)
Release Managers: 0
![Page 107: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/107.jpg)
![Page 108: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/108.jpg)
Is it Broken? Or , is it just better?
![Page 109: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/109.jpg)
![Page 110: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/110.jpg)
Metrics + Configs ⇾ OODA Loop
![Page 111: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/111.jpg)
“Theoretical” vs. “Practical”
![Page 112: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/112.jpg)
Surprise!!!Turning off multi-language supportimproves our page generation times by up to 25%.
Homepage (95th perc.)
![Page 113: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/113.jpg)
Nope. It’s really broken.
![Page 114: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/114.jpg)
http://www.flickr.com/photos/flyforfun/2694158656/
OperatorConfig flags
Metrics
![Page 115: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/115.jpg)
Thursday, Nov 22 - ThanksgivingFriday, Nov 23 - “Black Friday”
Monday, Nov 26 - “Cyber Monday”
~30 days out from Christmas
![Page 116: Continuous Delivery: The Dirty Details](https://reader034.vdocuments.mx/reader034/viewer/2022052522/554bb179b4c905ae618b5921/html5/thumbnails/116.jpg)
30
20
10
40