Download - Continuous Deployment: The Dirty Details
![Page 2: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/2.jpg)
CONTINUOUS DEPLOYMENT
The Dirty Details
“OK, sounds cool. But I have some questions...”
![Page 3: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/3.jpg)
CD 100- & 200-levels
- CI environment for automated tests- Committing to trunk- Branching in code- Config flags (a.k.a. feature flags)- DevOps mentality- Metrics and alerting- Automated deploy scripts
credit: photobookgirl (flickr)
![Page 4: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/4.jpg)
CD 100- & 200-levels
- CI environment for automated tests- Committing to trunk- Branching in code- Config flags (a.k.a. feature flags)- DevOps mentality- Metrics and alerting- Automated deploy scripts
- Deploys vs. releases- Decoupled systems, schema changes- How we work: Arch. & Process- Integration and Operations
CD 300 level
credit: photobookgirl (flickr)
![Page 5: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/5.jpg)
![Page 6: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/6.jpg)
www. .com
![Page 7: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/7.jpg)
GROSS MERCHANDISE SALES
http://www.etsy.com/blog/news/2013/notes-from-chad-2012-year-in-review/
![Page 8: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/8.jpg)
DECEMBER 20121.5 Billion page views$117 Million of goods sold6 Million items sold
http://www.etsy.com/blog/news/2013/etsy-statistics-december-2012-weather-report/Items by anjaysdesigns, betwixxt, OneStarLeatherGoods, mediumcontrol, TheDesignPallet
![Page 9: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/9.jpg)
175+ Committers, everyone deploys
credit: martin_heigan (flickr)
![Page 10: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/10.jpg)
Very end of 2009Today
DEPLOYMENTS PER DAY
![Page 11: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/11.jpg)
Continuous delivery is a pattern language in growing use in software development to improve the process of software delivery.
~wikipedia
Techniques such as automated testing, continuous integration, and continuous deployment allow software to be developed to a high standard and easily packaged and deployed to test environments, resulting in the ability to rapidly, reliably and repeatedly push out enhancements and bug fixes to customers at low risk and with minimal manual overhead.
credit: Stewart, redgen (flickr)
![Page 12: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/12.jpg)
Linux, Apache, MySQL, PHPArchitecture Stack
Memcache, Gearman, Postgresql, Solr, Java, Traffic Server, Hadoop, HBase
credit: Saire Elizabeth (flickr)
Git, Jenkins
![Page 13: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/13.jpg)
![Page 14: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/14.jpg)
2010-today2009
Then Now
Just before we started using CD
![Page 15: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/15.jpg)
15 mins6-14 hours1 person“Deployment Army”
Then Now
Part of everydayworkflow
Special event andhighly orchestrated
![Page 16: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/16.jpg)
Blocked for15 minutes.
15 minutes toredeploy
Blocked for6-14 hours.
6+ hours toredeploy
Then Now
![Page 17: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/17.jpg)
Mainline,minimal linking
and building,rsync,site up
Release branch,database schemas,
data transforms,packaging,
rolling restarts,cache purging,
scheduled downtime
Then Now
![Page 18: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/18.jpg)
1!" #$%Put your face on Etsy.com.
![Page 19: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/19.jpg)
2&# #$%Complete tax, insurance, and
benefits forms.
credit: ktpupp (flickr)
![Page 20: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/20.jpg)
![Page 21: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/21.jpg)
WARNING
![Page 22: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/22.jpg)
GROSS MERCHANDISE SALES
![Page 23: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/23.jpg)
Continuous DeploymentSmall, frequent changes.
Constantly integrating into production.30+ deploys per day.
![Page 24: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/24.jpg)
“Wow... 30 deploys a day.How do you build features so quickly?”
![Page 25: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/25.jpg)
Software Deploy ≠ Product Launch
![Page 26: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/26.jpg)
Deploys frequently gated by config flags(“dark” releases)
![Page 27: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/27.jpg)
$cfg[‘new_search’] = array('enabled' => 'off');$cfg[‘sign_in’] = array('enabled' => 'on');$cfg[‘checkout’] = array('enabled' => 'on');$cfg[‘homepage’] = array('enabled' => 'on');
![Page 28: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/28.jpg)
$cfg[‘new_search’] = array('enabled' => 'off');
![Page 29: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/29.jpg)
$cfg[‘new_search’] = array('enabled' => 'off');
// Meanwhile...
# old and boring search$results = do_grep();
![Page 30: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/30.jpg)
$cfg[‘new_search’] = array('enabled' => 'off');
// Meanwhile...
if ($cfg[‘new_search’] == ‘on’) { # New and fancy search $results = do_solr();} else { # old and boring search $results = do_grep();}
![Page 31: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/31.jpg)
$cfg[‘new_search’] = array('enabled' => 'on');
// or...
$cfg[‘new_search’] = array('enabled' => 'staff');
// or...
$cfg[‘new_search’] = array('enabled' => '1%');
// or...
$cfg[‘new_search’] = array('enabled' => 'users', 'user_list' => 'mike,john,kellan');
![Page 32: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/32.jpg)
Validate in production, hidden from public.
![Page 33: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/33.jpg)
Small incremental changes to the applicationNew classes, methods, controllersGraphics, stylesheets, templatesCopy/content changes
Turning flags on, off, or % ramp up
What’s in a deploy?
![Page 34: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/34.jpg)
Latent bugs and security holesTraffic management, load sheddingAdding and removing infrastructure
Tweaking config flags or releasing patches.
Low MTTR (response times)
![Page 35: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/35.jpg)
http://www.flickr.com/photos/flyforfun/2694158656/
![Page 36: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/36.jpg)
http://www.flickr.com/photos/flyforfun/2694158656/
OperatorConfig flags
Metrics
![Page 37: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/37.jpg)
Favorites“on”
![Page 38: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/38.jpg)
Favorites“off”
![Page 39: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/39.jpg)
Many deploys eventually lead to a product launch.
![Page 40: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/40.jpg)
“How do you continuously deploy database schema changes?”
![Page 41: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/41.jpg)
Code deploysSchema changes
~15-20 minutes
![Page 42: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/42.jpg)
Code deploysSchema changes
~15-20 minutesTHURSDAYS!
![Page 43: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/43.jpg)
Our web application is largely monolithic.
Etsy.com, Support & Back-office tools, Developer API, Gearman (async work)
![Page 44: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/44.jpg)
Our web application is largely monolithic.
Etsy.com, Support & Back-office tools, Developer API, Gearman (async work)
PHP, Apache, Memcache
![Page 45: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/45.jpg)
External “services” are not deployed with the main application.
e.g. Databases, Search, Photo storage, Payments
![Page 46: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/46.jpg)
External “services” are not deployed with the main application.
e.g. Databases, Search, Photo storage, Payments
MYSQL(schema changes)
SOLR, JVM(rolling restarts)
PROXY CACHE, FILERS, AMAZON S3
(specialized infra.)
PCI(controlled access)
![Page 47: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/47.jpg)
For every config flag, there are two stateswe can support — present and future.
![Page 48: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/48.jpg)
For every config flag, there are two stateswe can support — present and future.
... or past and present.
![Page 49: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/49.jpg)
“Non-Breaking Expansions”Expose new version in a service interface;
support multiple versions in the consumer.
![Page 50: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/50.jpg)
Example: Changing a Database SchemaMerging “users” and “users_prefs”
![Page 51: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/51.jpg)
RULE OF THUMB:Prefer ADDs over ALTERs (non-breaking expansion)C
![Page 52: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/52.jpg)
1. Write to both versions2. Backfill historical data3. Read from new version4. Cut-off writes to old version
![Page 53: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/53.jpg)
0. Add new version to schema1. Write to both versions2. Backfill historical data3. Read from new version4. Cut-off writes to old version
![Page 54: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/54.jpg)
0. Add new version to schemaSchema change to add prefs columns to “users” table.
“write_prefs_to_user_prefs_table” => “on”“write_prefs_to_users_table” => “off”“read_prefs_from_users_table” => “off”
![Page 55: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/55.jpg)
1. Write to both versionsWrite code for writing prefs to the “users” table.
“write_prefs_to_user_prefs_table” => “on”“write_prefs_to_users_table” => “on”“read_prefs_from_users_table” => “off”
![Page 56: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/56.jpg)
2. Backfill historical dataOffline process to sync existing data from “user_prefs”to new columns in “users”
![Page 57: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/57.jpg)
3. Read from new versionData validation tests. Ensure consistency both internallyand in production.
“write_prefs_to_user_prefs_table” => “on”“write_prefs_to_users_table” => “on”“read_prefs_from_users_table” => “staff”
![Page 58: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/58.jpg)
3. Read from new versionData validation tests. Ensure consistency both internallyand in production.
“write_prefs_to_user_prefs_table” => “on”“write_prefs_to_users_table” => “on”“read_prefs_from_users_table” => “1%”
![Page 59: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/59.jpg)
3. Read from new versionData validation tests. Ensure consistency both internallyand in production.
“write_prefs_to_user_prefs_table” => “on”“write_prefs_to_users_table” => “on”“read_prefs_from_users_table” => “5%”
![Page 60: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/60.jpg)
3. Read from new versionData validation tests. Ensure consistency both internallyand in production.
“write_prefs_to_user_prefs_table” => “on”“write_prefs_to_users_table” => “on”“read_prefs_from_users_table” => “on”
(“on” == “100%”)
![Page 61: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/61.jpg)
4. Cut-off writes to old versionAfter running on the new table for a significant amount of time, we can cut off writes to the old table.
“write_prefs_to_user_prefs_table” => “off”“write_prefs_to_users_table” => “on”“read_prefs_from_users_table” => “on”
![Page 62: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/62.jpg)
“Branch by Astraction”
Controller Controller
Users Model
“users” (old) “user_prefs” “users”
old schema new schema
(Abstraction)
http://paulhammant.com/blog/branch_by_abstraction.htmlhttp://continuousdelivery.com/2011/05/make-large-scale-changes-incrementally-with-branch-by-abstraction/
![Page 63: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/63.jpg)
1. Write to both versions2. Backfill historical data3. Read from new version4. Cut-off writes to old version
“The Migration 4-Step”
![Page 64: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/64.jpg)
“When do you clean up all of those config flags?
![Page 65: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/65.jpg)
We might remove config flags for the old version when...
It is no longer valid for the business.It is no longer stable, maintained, or trusted.It has poor performance characteristics.The code is a mess, or difficult to read.We can afford to spend time on it.
![Page 66: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/66.jpg)
Promote “dev flags” to “feature flags”
![Page 67: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/67.jpg)
// Feature flag$cfg[‘mobilized_pages’] = array('enabled' => 'on');
// Dev flags$cfg[‘mobile_templates_seller_tools’] = array('enabled' => 'on');$cfg[‘mobile_templates_account_tools’] = array('enabled' => 'on');$cfg[‘mobile_templates_member_profile’] = array('enabled' => 'on');$cfg[‘mobile_templates_search’] = array('enabled' => 'off');$cfg[‘mobile_templates_activity_feed’] = array('enabled' => 'off');
...
if ($cfg[‘mobilized_pages’] == ‘on’ && $cfg[‘mobile_templates_search’] == ‘on’) { // ... // ...}
![Page 68: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/68.jpg)
// Feature flags$cfg[‘search’] = array('enabled' => 'on');$cfg[‘developer_api’] = array('enabled' => 'on');$cfg[‘seller_tools’] = array('enabled' => 'on');
$cfg[‘the_entire_web_site’] = array('enabled' => 'on');
![Page 69: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/69.jpg)
![Page 70: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/70.jpg)
// Feature flags$cfg[‘search’] = array('enabled' => 'on');$cfg[‘developer_api’] = array('enabled' => 'on');$cfg[‘seller_tools’] = array('enabled' => 'on');
$cfg[‘the_entire_web_site’] = array('enabled' => 'on');$cfg[‘the_entire_web_site_no_really_i_mean_it’] = array('enabled' => 'on');
![Page 71: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/71.jpg)
Architecture and Process
![Page 72: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/72.jpg)
Deploying is cheap.
![Page 73: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/73.jpg)
Releasing is cheap.
![Page 74: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/74.jpg)
Some philosophies on product development...
![Page 75: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/75.jpg)
Gathering data should be cheap, too.staff, opt-in prototypes, 1%
![Page 76: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/76.jpg)
Treat first iterations as experiments.
![Page 77: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/77.jpg)
Get into code as quickly as possible.
![Page 78: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/78.jpg)
“Where a new system concept or new technology is used, one has to build a system to throw away, for even the best planning is not so omniscient as to get it right the first time. Hence plan to throw one away; you will, anyhow.”
~ Fred Brooks, The Mythical Man-Month
![Page 79: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/79.jpg)
Architecture largely doesn’t matter.
![Page 80: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/80.jpg)
Kill things that don’t work.
![Page 81: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/81.jpg)
Your assumptions will be wrongonce you’ve scaled 10x.
![Page 82: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/82.jpg)
“We don’t optimize for being right. We optimize for quickly detecting when we’re wrong.”
~Kellan Elliott-McCrea, CTO
![Page 83: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/83.jpg)
Become really good at changingyour architecture.
![Page 84: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/84.jpg)
Invest time in architecture by the2nd or 3rd iteration.
![Page 85: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/85.jpg)
Integration and Operations
![Page 86: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/86.jpg)
WARNING
REMEMBER THIS?
![Page 87: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/87.jpg)
Continuous DeploymentSmall, frequent changes.
Constantly integrating into production.30 deploys per day.
![Page 88: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/88.jpg)
Code review before commit
![Page 89: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/89.jpg)
Automated tests before deploy
![Page 90: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/90.jpg)
Why Integrate with Production?
![Page 91: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/91.jpg)
Dev ≠ Prod
![Page 92: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/92.jpg)
Verify frequently and in small batches.
![Page 93: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/93.jpg)
Integrating with production is a test in itself.We do this frequently and in small batches.
![Page 94: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/94.jpg)
More database servers in prod.Bigger database hardware in prod.More web servers.Various replication schemes.Different versions of server and OS software.Schema changes applied at different times.Physical hardware in prod.More data in prod.Legacy data (7 years of odd user states).More traffic in prod.Wait, I mean MUCH more traffic in prod.Fewer elves.Faster disks (SSDs) in prod.
![Page 95: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/95.jpg)
Using a MySQL database in dev for an application that will be runningon Oracle in production: Priceless
![Page 96: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/96.jpg)
Verify frequently and in small batches.
![Page 97: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/97.jpg)
Dev ⇾ QA ⇾ Staging ⇾ Prod
![Page 98: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/98.jpg)
Dev ⇾ QA ⇾ Staging ⇾ Prod
![Page 99: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/99.jpg)
Dev ⇾ Pre-Prod ⇾ Prod
![Page 100: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/100.jpg)
Test and integrate where you’ll see value.
![Page 101: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/101.jpg)
Config flags (again)
off, on, staff, opt-in prototypes, user list, 0-100%
![Page 102: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/102.jpg)
Config flags (again)
off, on, staff, opt-in prototypes, user list, 0-100%
“canary pools”
![Page 103: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/103.jpg)
Automated tests after deploy
![Page 104: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/104.jpg)
Real-time metrics and dashboardsNetwork & Servers, Application, Business
![Page 105: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/105.jpg)
![Page 106: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/106.jpg)
SERVER METRICS
Apache requests/sec, Busy processes,CPU utilization, Script exec time (med. & 95th)
APPLICATION METRICS
Logins, Registrations, Checkouts, Listings created, Forum posts
Time and event correlated.
![Page 107: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/107.jpg)
![Page 108: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/108.jpg)
Real humans reporting trouble!
![Page 109: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/109.jpg)
“Theoretical” vs. “Practical” Engineering
![Page 110: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/110.jpg)
http://www.flickr.com/photos/flyforfun/2694158656/
OperatorConfig flags
Metrics
![Page 111: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/111.jpg)
Thanksgiving, “Black Friday,” “Cyber Monday” ➔ Christmas(~30 days)
Managing risk during Holiday Shopping season
Code Freeze?
![Page 112: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/112.jpg)
“Code Slush”
DEPLOYMENTS PER DAY
![Page 113: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/113.jpg)
Tighten your feedback cyclesIntegrate with production and validate early in cycle.Use tools that allow you to detect issues early.Optimize for quick response times.
Applied to both feature development and operability.
![Page 114: Continuous Deployment: The Dirty Details](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554bb00fb4c905b8618b5930/html5/thumbnails/114.jpg)
Thank you
Mike Brittain
... and questions?
These slides will be available later today at http://mikebrittain.com/talks
ENGINEERING DIRECTOR