workflow design: extracting signal from the noise - chefconf 2015
TRANSCRIPT
Workflow Design
Extracting Signal from the Noise
@jonlives
Jon Cowie
Staff Operations Engineer
@jonlives
There is no correct way to use Chef.
workflow_design
@jonlives
…but I just want to get stuff done!
@jonlives
Absorb what is useful.
Discard what is useless.
@jonlives
“I am not smart enough to build an ontology … that
can encompass all the variations in infrastructure.
Nobody is, the world moves too fast.”
@jonlives
There is no magic pill.
@jonlives
You are the expert.
@jonlives
Beware Argument Culture!
Kate Heddleston: bit.ly/argument-culture
@jonlives
[An] argument culture urges us to approach the world—and the people in it—in an adversarial
frame of mind Deborah Tannen “The Argument Culture”
@jonlives
Argument Culture• Useful in “regulated” environments like law
• Engineering is not one of those environments
• Weed out weak logic, keep strong ideas
• Everybody is on one side or the other
• We attack their position and defend our own
• We can win or lose arguments—just like battles
@jonlives
Argument Culture - Effects• Emotional interactions
• Winning becomes paramount
• The loud dominate
• People are afraid to speak
• Best practice == repeating the loudest arguments
• Sucks for diversity in cultures and orgs
@jonlives
What a Workflow is…workflow noun ( also work-flow) /ˈwɜːkfləʊ/
the way that a particular type of work is organised, or the order of the stages in a particular work process
Cambridge Business English Dictionary
@jonlives
What a Workflow is not…• “The Chef Way”
• “We need to use X tool”
• “We need to be more DevOps”
• “We need our process to be more like Y”
• “Z in Chef is bad. Never do Z.”
@jonlives
A workflow is the key to solving a problem.
@jonlives
There is no correct way to use Chef…
@jonlives
There are only lessons learned by others in the process of solving their
specific problems.
@jonlives
Case Studies
@jonlives
Case Studies• Anonymized
• Companies of varying sizes
• First, a quick summary
• Then we look at their motivations
• Finally we look at their solutions
@jonlives
Case Study #1 - Summary• Small internet video company
• Small Ops team
• EC2 based infrastructure
• Semi-immutable infrastructure • Packer used to build AMI containers
• Chef run on instances to correct config & service status
• Previously required rolling a new AMI
@jonlives
Case Study #1 - Motivation• They’re trying to make it as easy to get started as
possible
• Chat or comments in PRs since everybody is remote
• They aren't interested in letting some third party change stuff unexpectedly
• No auto-deploying master - they had an accidental push
@jonlives
Case Study #1 - Solution• Cookbooks live in chef repo
• Dependencies in Berksfile
• “Pull request” development model
• Merge to master & Jenkins runs foodcritic & rubocop
• Merge into “jenkins” branch, triggers upload to server
@jonlives
Case Study #2 - Summary• One project team inside extremely large company
• On-premise cloudstack infrastructure • VMs created, then bootstrapped with Chef
• Dev, QA, Staging, Prod environments
• Legacy cookbooks maintained alongside new
• 1 workflow for new cookbooks, one for legacy
@jonlives
Case Study #2 - Motivation• Too easy to make an innocuous change that broke
several features without knowing about it
• Desire to catch syntax errors or undeclared attributes
• Complete test coverage with minitest, kitchen-ci for easier PR validation, and local cookbook development
@jonlives
Case Study #2 - Solution• 1 monolithic project cookbook
• Add dependencies to Berksfile & “berks install”
• Create recipes & write tests
• “kitchen converge” until it works
• Push to git & chef server
• Test in dev, then on to prod
@jonlives
Case Study #3 - Summary• Mid-sized online marketplace
• Mainly on-premise infrastructure
• Servers often long-lived & not immutable
• Large number of Dev & Ops Chef users
• One main environment
@jonlives
Case Study #3 - Motivation• Many Chefs
• Many small changes by often inexperienced users
• Historical cookbooks make using community cookbooks difficult & time consuming
• App deployed using separate deploy tool
• Continuous delivery in “DNA”
@jonlives
Case Study #3 - Solution• Cookbooks live in monolithic repository
• Roles == attribs, environments == constraints
• Tooling geared for efficiency & iteration vs testing
• Testing done locally with chef-zero & why-run
• …or in prod with whitelists
• Versions always pinned in prod environment
@jonlives
Case Study #4 - Summary• Mid-sized ecommerce service provider
• Mainly on-premise infrastructure
• Multiple product-based teams
• Multiple environments to support those teams
@jonlives
Case Study #4 - Motivation• Teams based around the Products they build and run
• Different opinions & requirements for workflow
• Most effort was spent on new greenfield projects
• Trying to avoid “veneering tech debt”
• Chef was new to most people
• Pressure to produce visible & demonstrable results
@jonlives
Case Study #4 - Solution• 85% of cookbooks in monolithic repo
• Start with community cookbooks
• Wrap as needed
• Team or Product specific environments
• Roles used to set permissions etc
• Optional cookbook pinning
@jonlives
So, how do we create a Workflow?
@jonlives
Workflow Design• What problem are you solving?
• Who are you solving it for?
• How does your ideal solution look?
• Choose your Tooling
• Implement your workflow
@jonlives
What Problem are you Solving?• Write down a simple problem
statement
• “I want to automate app deploys”
• “I want to automate server builds”
• “I want us to test our cookbooks more”
• Break that down into simple pieces
What Who How Tooling Implement
@jonlives
Who are you solving it for?• Understand your stakeholders
• Understand your business priorities
• What do *they* care about most?
• How often will they work with your solution?
• Conway’s law
What Who How Tooling Implement
@jonlives
Conway’s Law
“Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations”
Melvin Conway, 1968
@jonlives
How does your ideal solution look?• Have an ideal “end goal” in mind
• Agree with stakeholders
• You won’t get there in one go
• But you’ll know what you’re working towards
What Who How Tooling Implement
@jonlives
Choose your Weapons!• Pick candidate tools
• Probe strengths and weaknesses
• Understand their motivations
• Extensibility if tweaking needed?
• Is anybody already using it in the same way?
What Who How Tooling Implement
@jonlives
…or make your own Weapons• Sometimes, you have to invent
• Legitimate approach
• Many popular workflow tools started this way
• Be sure you have time / resources
• Clear mission statement is vital
What Who How Tooling Implement
@jonlives
Implementation• Have realistic expectations
• Start slowly
• Efficiency-Thoroughness Trade-Off (ETTO)
• Iterate often
• Learn from failures. They will happen.
• Improve
What Who How Tooling Implement
@jonlives
Don’t Fear the Code!
@jonlives
Don’t Fear the Code!• Understanding == Effectiveness
• If you know how it works, you can figure out why it breaks.
• Research known bugs & limitations
• Be prepared to report or fix issues
@jonlives
It’s OK to try stuff!
@jonlives
http://jonliv.es/book
Discount Code: AUTHD
40% off Print 50% off Digital
