managing short lived kubernetes (production) deployments
TRANSCRIPT
![Page 1: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/1.jpg)
Managing short livedKubernetes deployments
Martin Danielsson (@donmartin76)
Solution Architect @ Haufe Group
![Page 2: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/2.jpg)
$ whoami
C:\> WINDOWS.EXE
C/C++/C# Background
15+ years
$ docker ps
Containers & Kubernetes
Since ~2 years
wicked.haufe.io maintainer
OSS API Management
“Solution
Architect”
Developer
since 2006
![Page 3: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/3.jpg)
Setting The Scene
![Page 4: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/4.jpg)
Strategic move
to containers Modular
Architecture
Without Container
Experience
Current Occupation – A Cloud Journey
Hosted with Hoster
Long Release
cycles
(LOTS of) Manual
Work for Releases
Little Operations
Insight
Error tracking
very difficult
Non-Parity
Dev/Test/Prod
(Cost!)
Legacy Web App
(Java based)
![Page 5: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/5.jpg)
Solution – Let’s go DevOps in the cloud!
![Page 6: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/6.jpg)
A Process Pattern
![Page 7: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/7.jpg)
Enabling CI/CD
Automatic Provisioni
ngFull Insight
Minimize Ops
Top Priorities
![Page 8: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/8.jpg)
Chosen Solution Outline
Kubernetes
Azure Container Services
Azure as IaaS provider
![Page 9: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/9.jpg)
Alternative Solution Outline
kops
Kubernetes
kops (kubernetes operations)
AWS as IaaS provider
![Page 10: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/10.jpg)
Steps to DevOps Happiness (for us)
Provision
Deploy CI/CD
Weekly for Production, Daily for Dev/Test
Ship when ready!
![Page 11: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/11.jpg)
But… Why?
Target
“No-Ops”
No long-running
systems
Enable validation of
3rd Party component
upgrades
Incremental
changes
Practice Disaster
Recovery Daily
100% Reproducible
Deployments
On-demand Production
Identical Environments
![Page 12: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/12.jpg)
Dem geneigten Zuhörer mag aufgefallensein…
Stateless Components
Stateful Components
![Page 13: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/13.jpg)
Adding State (Persistence)
![Page 14: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/14.jpg)
Full Provisioning
Create backupProvision new infrastructure
• From backups
• Same as disaster recovery!
Deploy components
• Using deployment pipelines
• Partly parallelized
Top level DNS switch
• Using DNS traffic manager
Destroy old infrastructure
• If tests succeed
![Page 15: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/15.jpg)
Persistence Options
Roll your own persistence Persistence “as a service”
Self managed VMs (incl. NFS) Managed Disks
(AWS EBS, Azure Managed Disks)
DBaaS (many options)
Files as a service
(AWS EFS, Azure
Files)
Gluster/Ceph FS (cluster)
![Page 16: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/16.jpg)
Persistence Requirements
A) Backup on demand (or auto)
B) Restore to other instance
AB) Clone on demand
![Page 17: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/17.jpg)
On-demand Environments
ProdDev/Test
Load Testing…
![Page 18: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/18.jpg)
Example – SQL Schema Update
Create backupProvision new infrastructure
Deploy components
Top level DNS switch
Destroy old infrastructure
Test/Validate
![Page 19: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/19.jpg)
Advantages
On Demand Dev/
Test EnvsEnables Test
Of Risky Updates
Built-In Disaster
Recovery
![Page 20: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/20.jpg)
-as-a-Service
Less Complex
No Operations Overhead
Supports A+B, or AB?If not: Can I live without Prod
Data in Dev/Test Envs?
Do I trust Service Provider
to live up to SLA?
In case of
What can I do?
![Page 21: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/21.jpg)
Limitations
![Page 22: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/22.jpg)
Possible Constraints
Implementation
EffortSLA Requirements
(Downtime)
Data SizeBackup/Restore Time
Team Size
Ops Skills Needed
![Page 23: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/23.jpg)
Our Solution Vector
![Page 24: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/24.jpg)
Resource GroupKubernetes Cluster
Solution Architecture (Infrastructure)
ks8 Master
ks8 Agent
ks8 Agent n
…
NFSVM(s)
Postgres VM(s)
Database as a Service
Disks
Disks
![Page 25: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/25.jpg)
NFS Storage/Postgres Storage
• Backup – Cloning disks from running system
• Restore – Cloning from backups
• Very much a transient technology!• But it works…
• Moving to DBaaS (e.g. Cosmos DB) over time
![Page 26: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/26.jpg)
Endless Variants…
![Page 27: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/27.jpg)
Conclusion and Takeaways
k8s Ops possible
as a Team
Requires full (test)
automation
Team dedicationRethinking ops is
challenging
No Silver Bullet
Assess your requirements
![Page 28: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/28.jpg)
Q & A
![Page 29: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/29.jpg)
Thanks!
Twitter donmartin76GitHub donmartin76
linkedin.com/in/martindanielsson/
www.haufegroup.comwork.haufegroup.io
wicked.haufe.io
![Page 30: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/30.jpg)
Backup Slides
![Page 31: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/31.jpg)
Persistence problems and possible solutions
Data Type Solution Technology Backup/Restore Complexity
Plain Files NFS AB Low
CephFS/GlusterFS A+B High
SQL Database Azure SQL Server A+B Medium
Azure Postgres-aaS AB Low
AWS RDS for Postgres AB Low
NoSQL Azure Cosmos DB A+B Medium
AWS DynamoDB A+B (via tools) Medium
![Page 32: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/32.jpg)
Integration & e2e Test
Build & Unit Test
Docker Image
Deploy
Building blocks of CI/CD pipelines
• E.g., Blue/Green• Rolling Updates• Also used for initial
deployment
![Page 33: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/33.jpg)
Incremental Frontend Deployment
Merge feature to master
•After code review
• Including test suite changes
Build master branch
• Includes unit testing
•First integration tests
Deploy to integration system
•Run integration tests
•Rollback if failing
Deploy to Production
•Run e2e integration tests
•Rollback if failing
![Page 34: Managing short lived Kubernetes (Production) deployments](https://reader030.vdocuments.mx/reader030/viewer/2022021500/5a6693287f8b9ac5128b564d/html5/thumbnails/34.jpg)
Incremental Backend Deployment
Merge feature to master
•After code review
• Including test suite changes
Build master branch
• Includes unit testing
•First integration tests
Deploy to integration system
•Blue/Green with integration tests
Deploy to Production
•Blue/Green with integration tests