operations: a developer's guide
Post on 13-Feb-2017
3.559 Views
Preview:
TRANSCRIPT
Operations: a developer’s guideAnna Shipman @annashipman
@annashipman
Technical architect at the Government Digital Service
@annashipman
My background
@annashipman
Previous career in publishing
@annashipman
Self taught programmer: HTML, CSS, terrible JS
@annashipman
First employed gig, 2005: backend, Java
@annashipman
First job at GDS, 2012: redirecting URLs
@annashipman
@annashipman
Wrote perl to generate nginx config
@annashipman
Didn’t understand the emails from the infrastructure team
@annashipman
So joined the infrastructure team
@annashipman
Which led to where I am now…
@annashipman
Technical architect on a large infrastructure project
@annashipman
What I learned has made me a better developer
@annashipman
Knowing these things might help you
@annashipman
Wrangling servers Virtualisation Containerisation Some tools to make you a better developer
@annashipman
Each section will have a take-home
@annashipman
1. Wrangling servers
@annashipman
How the internet works
Wrangling servers
@annashipman
“It’s a series of tubes”—Sen. Ted Stevens
Wrangling servers
@annashipman
Servers
Wrangling servers
@annashipman
Where are your servers?
Wrangling servers
@annashipman
Where are your servers?You own them (e.g. in the office)
Wrangling servers
@annashipman
Where are your servers?You own them (e.g. in the office) Shared hosting (e.g. Dreamhost)
Wrangling servers
@annashipman
Where are your servers?You own them (e.g. in the office) Shared hosting (e.g. Dreamhost) The cloud (e.g. AWS)
Wrangling servers
@annashipman
Where are your servers?You own them (e.g. in the office) Shared hosting (e.g. Dreamhost) The cloud (e.g. AWS) PaaS/application hosting (e.g. Heroku)
Wrangling servers
@annashipman
Where are your servers?You own them (e.g. in the office) Shared hosting (e.g. Dreamhost) The cloud (e.g. AWS) PaaS/application hosting (e.g. Heroku) Something else/don’t know
Wrangling servers
@annashipman
You need to make sure the server has the software you need
Wrangling servers
@annashipman
Handcrafting servers
Wrangling servers
@annashipman
But what happens if your server dies?
Wrangling servers
@annashipman
It’s also easy to make a mistake
Wrangling servers
@annashipman
Configuration management
Wrangling servers
@annashipman
Tools that use configuration you’ve written to build servers
Wrangling servers
https://www.getfilecloud.com/blog/2014/08/top-8-configuration-management-tools-for-sys-admins/
@annashipman
GOV.UK uses Puppet, my current project uses Ansible
Wrangling servers
@annashipman
Config management tools automate building your servers
Wrangling servers
@annashipman
So it is reliable and repeatable
Wrangling servers
@annashipman
Some getting started guides at the end
Wrangling servers
@annashipman
If nothing else, just write a script
Wrangling servers
@annashipman
“Cattle not pets”
Wrangling servers
@annashipman
You should not be afraid to lose your servers
Wrangling servers
@annashipman
2. Virtualisation
@annashipman
Creating logical computing resources from available physical resources
Virtualisation
@annashipman
Virtual machines
Virtualisation
@annashipman
What’s a hypervisor?
Virtualisation
@annashipman
Hypervisor is the software that runs the VMs
Virtualisation
@annashipman
@annashipman
Slight digression into cloud computing
Virtualisation
@annashipman
“Someone else’s computers”
Virtualisation
@annashipman
Your servers are in a data centre
Virtualisation
@annashipman
A layer of abstraction that pools the resources
Virtualisation
@annashipman
@annashipman
Some advantages of cloud computing
Virtualisation
@annashipman
Some advantages of cloud computingIncreased uptime & disaster recovery
Virtualisation
@annashipman
Some advantages of cloud computingIncreased uptime & disaster recovery Can get a VM straight away
Virtualisation
@annashipman
Some advantages of cloud computingIncreased uptime & disaster recovery Can get a VM straight away Charged for what you use
Virtualisation
@annashipman
That’s the cloud – now your computer
Virtualisation
@annashipman
How is virtualisation useful to you?
Virtualisation
@annashipman
Used to be software = slow, resource-intensive
Virtualisation
@annashipman
From 2005 Intel & AMD started doing hardware-accelerated virtualisation
Virtualisation
@annashipman
So can run lots of VMs on your computer
Virtualisation
@annashipman
Vagrant is a lightweight way to create VMs
Virtualisation
@annashipman
Vagrant.configure(2) do |config| config.vm.box = "puppetlabs/centos-7.0-64-puppet" config.vm.provider :virtualbox |vb| vb.customize ["modifyvm", :id, "--memory", "2048"] end end
Virtualisation
@annashipman
Vagrant.configure(2) do |config| config.vm.box = "ubuntu/trusty64" end
Virtualisation
@annashipman
$ vagrant up
Virtualisation
@annashipman
$ vagrant ssh
Virtualisation
@annashipman
$ cd /vagrant
Virtualisation
@annashipman
Links to documentation and simple examples at the end
Virtualisation
@annashipman
You can develop locally using the same software as is deployed remotely
Virtualisation
@annashipman
You can use configuration management to build your VM
Virtualisation
@annashipman
It makes it easy to collaborate
Virtualisation
@annashipman
Vagrant makes it easy to get started/pick up where you left off
Virtualisation
@annashipman
Currently Otto uses Vagrant
Virtualisation
@annashipman
Vagrant can improve your project tomorrow
Virtualisation
@annashipman
3. Containerisation
@annashipman
(Docker is not the only containerisation tech)
Containerisation
@annashipman
Docker is based on Linux Containers
1. Put netting on garlic and put back outside
Containerisation
@annashipman
2008: LXC 2004: Solaris zones 2000: BSD jails 1982: chroot (tech based on)
Containerisation
@annashipman
You can run multiple containers
Containerisation
@annashipman
Containers are isolated from each other
Containerisation
@annashipman
Why would you use containers?
Containerisation
@annashipman
Configuring a VM can be slow
Containerisation
@annashipman
You might instead decide to take a snapshot
Containerisation
@annashipman
And that is what you deploy to production
Containerisation
@annashipman
Containers are immutable
Containerisation
@annashipman
So you can be sure what you’ve tested is what is running in production
Containerisation
@annashipman
With all the same dependencies
1. Put netting on garlic and put back outside
Containerisation
@annashipman
However…
Containerisation
@annashipman
The immutability can make developing locally difficult
Containerisation
@annashipman
You can’t run them everywhere
Containerisation
@annashipman
Possibly a harder concept to grasp
Containerisation
@annashipman
Right now, it’s not essential that you understand Docker unless you are interested
Containerisation
@annashipman
2 + 3: Virtualisation + Containerisation
Turtles
@annashipman
Locally, it’s Vagrant or Docker
Turtles
@annashipman
But if you deploy to the cloud…
Turtles
@annashipman
@annashipman
How do you know where the problem is?
Turtles
@annashipman
It’s turtles all the way down
Turtles
@annashipman
BREAK
@annashipman
Wrangling servers“Cattle not pets”
@annashipman
VirtualisationUse Vagrant
@annashipman
ContainerisationYou don’t need to learn Docker right now
@annashipman
4. Some tools to make you a better developer
@annashipman
“Should all my tools be written in JavaScript (warning: if the answer is ‘no’, there might be a revolution!!!)”—Remy
Tools
@annashipman
Build systems – Grunt, Gulp…
Tools
@annashipman
Don’t just work out of the box
Tools
@annashipman
Dependencies and plug-ins
Tools
@annashipman
You could use NPM
Tools
@annashipman
Make
Tools
@annashipman
Can create tasks to do anything: clean, compile, build…
Tools
@annashipman
Advantages of Make
Tools
@annashipman
Advantages of MakeDependency tracking and resolution
Tools
@annashipman
Advantages of MakeDependency tracking and resolution Only does what it needs to
Tools
@annashipman
all: styles.css print.css
styles.css: styles.scss sass styles.scss:styles.css
print.css: print.scss sass print.scss:print.css
Tools
@annashipman
all: styles.css print.css
styles.css: styles.scss sass styles.scss:styles.css
print.css: print.scss sass print.scss:print.css
Compiles your SCSS to CSS
Tools
@annashipman
all: styles.css print.css
styles.css: styles.scss sass styles.scss:styles.css
print.css: print.scss sass print.scss:print.css
This is what Grunt, Gulp etc are doing
Tools
@annashipman
all: styles.css print.css
styles.css: styles.scss sass styles.scss:styles.css
print.css: print.scss sass print.scss:print.css
$ make styles.css
Tools
@annashipman
all: styles.css print.css
styles.css: styles.scss sass styles.scss:styles.css
print.css: print.scss sass print.scss:print.css
Only runs if SCSS is newer than CSS
Tools
@annashipman
all: styles.css print.css
styles.css: styles.scss sass styles.scss:styles.css
print.css: print.scss sass print.scss:print.css
$ make print.css
Tools
@annashipman
all: styles.css print.css
styles.css: styles.scss sass styles.scss:styles.css
print.css: print.scss sass print.scss:print.css
$ make all
Tools
@annashipman
all: styles.css print.css
styles.css: styles.scss sass styles.scss:styles.css
print.css: print.scss sass print.scss:print.css
$ make all
Tools
@annashipman
all: styles.css print.css
styles.css: styles.scss sass styles.scss:styles.css
print.css: print.scss sass print.scss:print.css
$ make
Tools
@annashipman
all: styles.css print.css
styles.css: styles.scss sass styles.scss:styles.css
print.css: print.scss sass print.scss:print.css
If print.css doesn’t need updating, only runs styles.cssTools
@annashipman
Advantages of MakeDependency tracking and resolution Only does what it needs to
Tools
@annashipman
Advantages of MakeDependency tracking and resolution Only does what it needs to Included in your OS*
*Linux, Unix, MacOS
Tools
@annashipman
Has everything you need
Tools
@annashipman
It just isn’t written in JS
Tools
@annashipman
Unix tools
Tools
@annashipman
grep
Tools
@annashipman
grepSearch given input files for patterns
Tools
@annashipman
cat
Tools
@annashipman
catConcatenate and print files
Tools
@annashipman
awk
Tools
@annashipman
awkScan input files for patterns; perform action
Tools
@annashipman
And the most useful: man
Tools
@annashipman
manformat and display manual pages
Tools
@annashipman
$ man grep
Tools
@annashipman
$ man man
Tools
@annashipman
Being unixy
Tools
@annashipman
Each tool does one thing very well
Tools
@annashipman
Composable
Tools
@annashipman
|
Tools
@annashipman
| (Pipe)
Tools
@annashipman
Read a file of text, determine the n most frequently used words, and print out a sorted list of those words along with their frequencies.
Tools
@annashipman
1 tr -cs A-Za-z '\n' | 2 tr A-Z a-z | 3 sort | 4 uniq -c | 5 sort -rn | 6 sed ${1}q
Tools
@annashipman
1 tr -cs A-Za-z '\n' | 2 tr A-Z a-z | 3 sort | 4 uniq -c | 5 sort -rn | 6 sed ${1}q
Tools
@annashipman
1 tr -cs A-Za-z '\n' | 2 tr A-Z a-z | 3 sort | 4 uniq -c | 5 sort -rn | 6 sed ${1}q
Tools
@annashipman
1 tr -cs A-Za-z '\n' | 2 tr A-Z a-z | 3 sort | 4 uniq -c | 5 sort -rn | 6 sed ${1}q
Tools
@annashipman
1 tr -cs A-Za-z '\n' | 2 tr A-Z a-z | 3 sort | 4 uniq -c | 5 sort -rn | 6 sed ${1}q
Tools
@annashipman
1 tr -cs A-Za-z '\n' | 2 tr A-Z a-z | 3 sort | 4 uniq -c | 5 sort -rn | 6 sed ${1}q
Tools
@annashipman
1 tr -cs A-Za-z '\n' | 2 tr A-Z a-z | 3 sort | 4 uniq -c | 5 sort -rn | 6 sed ${1}q
Tools
@annashipman
1 tr -cs A-Za-z '\n' | 2 tr A-Z a-z | 3 sort | 4 uniq -c | 5 sort -rn | 6 sed ${1}q
Tools
@annashipman
1 tr -cs A-Za-z '\n' | 2 tr A-Z a-z | 3 sort | 4 uniq -c | 5 sort -rn | 6 sed ${1}q
Tools
@annashipman
1 tr -cs A-Za-z '\n' | 2 tr A-Z a-z | 3 sort | 4 uniq -c | 5 sort -rn | 6 sed ${1}q
Tools
@annashipman
1 tr -cs A-Za-z '\n' | 2 tr A-Z a-z | 3 sort | 4 uniq -c | 5 sort -rn | 6 sed ${1}q
Tools
@annashipman
Read a file of text, determine the n most frequently used words, and print out a sorted list of those words along with their frequencies.
Tools
@annashipman
You might not want to write all your programs in Unix
Tools
@annashipman
It can be really useful for doing complex tasks
Tools
@annashipman
It’s worth getting familiar with Unix tools locally
Tools
@annashipman
So that the first time isn’t when you’re debugging on the server
Tools
@annashipman
$ rake -T | grep "some detail"
Tools
@annashipman
Read the output
Tools
@annashipman
Git instructions
Tools
@annashipman
$ git add accidental_new_file.txt $ git status On branch master Your branch is up-to-date with 'origin/master'. Changes to be committed: (use "git reset HEAD <file>..." to unstage)
new file: accidental_new_file.txt
Tools
@annashipman
$ git add accidental_new_file.txt $ git status On branch master Your branch is up-to-date with 'origin/master'. Changes to be committed: (use "git reset HEAD <file>..." to unstage)
new file: accidental_new_file.txt
Tools
@annashipman
Unix error messages
Tools
@annashipman
$ grep -r anna
Tools
@annashipman
$ grep -r anna grep: warning: recursive search of stdin
Tools
@annashipman
$ grep -r anna .
Tools
@annashipman
Unix tools can help you
Tools
@annashipman
The main three things you should take home:
@annashipman
Use Vagrant
@annashipman
Unix tools are your friends
@annashipman
Read the output
Thank you!Anna Shipman @annashipman
Configuration managementhttps://www.scriptrock.com/articles/the-7-configuration-management-tools-you-need-to-know http://gettingstartedwithchef.com/first-steps-with-chef.html https://docs.vagrantup.com/v2/getting-started/provisioning.html
Virtualisationhttp://searchvirtualdatacentre.techtarget.co.uk/definition/Virtualisation http://searchservervirtualization.techtarget.com/definition/server-virtualization http://www.infoworld.com/article/2621446/server-virtualization/server-virtualization-top-10-benefits-of-server-virtualization.html
Using Vagranthttps://www.vagrantup.com/ http://blog.bennycornelissen.nl/otto-a-modern-developers-new-best-friend/ https://github.com/patrickdlee/vagrant-examples (Useful getting started examples)
Dockerhttp://patg.net/containers,virtualization,docker/2014/06/05/docker-intro/ https://zeltser.com/security-risks-and-benefits-of-docker-application/
Containerisation vs Virtualisationhttp://www.slideshare.net/bcantrill/docker-and-the-future-of-containers-in-production https://www.scriptrock.com/articles/docker-vs-vagrant
Make instead of Grunt/Gulphttp://blog.keithcirkel.co.uk/why-we-should-stop-using-grunt/ https://www.youtube.com/watch?v=0RYETb9YVrk (Talk on using NPM as a build tool) https://blog.jcoglan.com/2014/02/05/building-javascript-projects-with-make/
Tools for better devhttp://www.leancrew.com/all-this/2011/12/more-shell-less-egg/ (More detail on the 6-line Unix program)
top related