24 months - a devops retrospective
TRANSCRIPT
![Page 1: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/1.jpg)
2 4 M O N T H S
@ S _ M C L E O D @ I N F O X C H A N G E
![Page 2: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/2.jpg)
I N F O X C H A N G E
![Page 3: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/3.jpg)
A N O T- F O R - P R O F I T C O M M U N I T Y O R G A N I S AT I O N T H AT D E L I V E R S
T E C H N O L O G Y F O R S O C I A L J U S T I C E
![Page 4: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/4.jpg)
T E C H N O L O G I E S
![Page 5: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/5.jpg)
I N F O X C H A N G E2 0 1 2
![Page 6: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/6.jpg)
I N F O X C H A N G E O P E R AT I O N S T E A M 2 0 1 2
![Page 7: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/7.jpg)
We were in a bad place
![Page 8: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/8.jpg)
I N F R A S T R U C T U R E
• 6+ Year old servers
• 6+ Brands of network switches
• 'Core routers' = Desktop computers
• Partially puppetted Infrastructure
• Many 'legacy' servers
• More SPOF than you could dream up
![Page 9: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/9.jpg)
A P P L I C AT I O N D E P L O Y M E N T S
Major applications only deployed monthly at most
Large, complex deployments with many changes
No form of continuous integration or delivery
Releases were fragile & high risk
![Page 10: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/10.jpg)
C U LT U R E
• Change Adverse
• Witch Hunting
• Blame
• Negativity
• A lot of Ass-Covering
![Page 11: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/11.jpg)
I almost quit on my first week
![Page 12: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/12.jpg)
I N F O X C H A N G E2 0 1 4
![Page 13: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/13.jpg)
![Page 14: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/14.jpg)
P L AT F O R M I N F R A S T R U C T U R E
• In-house "PAAS" for our applications
• Almost all applications containerised in Docker
• Modern and supportable hardware platform & fault tolerant network redesign
• Every server (300~) is puppet controlled
• Excellent visibility into operational state
![Page 15: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/15.jpg)
A P P L I C AT I O N D E P L O Y M E N T S
• Docker images built and deployed by CI
• Prod app images == test app images
• Devs release to prod
• Up to 32 successful deployments to prod per day
![Page 16: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/16.jpg)
C U LT U R E
• Dev & Ops teams together facilitate product delivery
• Ops involved in application architecture
• Devs involved in platform architecture
• No more 'It worked in Dev'
![Page 17: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/17.jpg)
H O W D I D W E D O I T ?
![Page 18: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/18.jpg)
– L A R R Y W A L L
“Easy things should be easy, and hard things should be possible.”
![Page 19: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/19.jpg)
W E K N E W O U R D E S I R E D S TAT E
• Eliminate all current technical debt
• Increase operational supportability
• Increase operation and deployment resiliency
• Recover from failure quicker
• Stop fighting fires
• The ability to perform 10+ production deploys a day of any application
![Page 20: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/20.jpg)
Pick highly visible & easy to achieve changes first
![Page 21: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/21.jpg)
PAY D O W N T E C H D E B T
• It's hard to eliminate tech debt but it MUST be brought under control to reduce reactive work
• P2V your physicals
• chroot anything you can't modernise & puppet the underlying OS
• Set hard sunset dates & communicate them
• Demonstrate & praise modern environments
![Page 22: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/22.jpg)
The journey from here
to here
is important
![Page 23: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/23.jpg)
We had to make it easy to do the right thing
![Page 24: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/24.jpg)
• Dev VMs often out of sync with
puppet
• External package repos often
broken
• Devs didn’t have visibility into
deployment processes
• Ops needed to be involved in
every deployment
2012 Release Process
![Page 25: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/25.jpg)
U S I N G P U P P E T T O D E P L O Y C O M P L E X A P P L I C AT I O N S ( I S A B A D I D E A )
• High complexity deployment
• Large chains of execs
• Changes are regularly required to manifests
• Puppet is not designed to manage large directories of files
• Deployments are limited to how often you can run puppet
![Page 26: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/26.jpg)
2014 Release Process
1. Code pushed to git
2. CI builds Docker image
3. Tests run
4. MCollective triggers
Puppet to deploy Docker
image
![Page 27: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/27.jpg)
I N F O X C H A N G E2 0 1 4
Firefighting to a minimum
Deploys to a maximum
Happy Devs, Happy Ops
![Page 28: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/28.jpg)
L E S S O N S
• Know when to start fresh - Hindsight is 20/20
• Fail forward and aim to fail fast - It’s often cheaper
• Communication is key - Start DevOps lunches!
• Management must buy in - Make them read The Phoenix Project
![Page 29: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/29.jpg)
– J O H N V I N C E N T
“Here’s the secret. I’ll tell you EXACTLY what devops means.
Devops means giving a shit about your job enough to not pass the buck. Devops means giving a shit
about your job enough to want to learn all the parts and not just your little world.
Developers need to understand infrastructure. Operations people need to understand code.
People need to fucking work with each other and not just occupy space next to each other..”
![Page 30: 24 Months - A DevOps Retrospective](https://reader034.vdocuments.mx/reader034/viewer/2022051516/55a417e41a28aba26f8b462b/html5/thumbnails/30.jpg)
Q U E S T I O N S ?
@ S _ M C L E O D @ I N F O X C H A N G E