persistence in the cloud with bosh
TRANSCRIPT
Persistence in the Cloud
with BOSH
Michael Richardson - @m_richo
Agenda
1.What is BOSH?
2.Demo - ELK in AWS
What is BOSH?
* BOSH is an open source tool for
release engineering, deployment,
lifecycle management, and monitoring
of distributed systems.
* source https://bosh.io
What is BOSH?
* BOSH is an open source tool for
release engineering, deployment,
lifecycle management, and monitoring
of distributed systems.
* source https://bosh.io
What is BOSH?
WTF??
Packaging
Config Management Monitoring
Application
Deployment
Infrastructure Provisioning
Packaging
Config Management Monitoring
Application
Deployment
Infrastructure Provisioning
fpm
/bin/bash
Packaging
Config Management Monitoring
Application
Deployment
Infrastructure Provisioning
fpm
/bin/bashBOSH
1.Packaging
2.Infrastructure Provisioning
3.Application Deployment
4.Configuration Management
5.Monitoring
Packaging
Packaging
Piece of code or pre-compiled binaries
required at either build-time or run-time
Infrastructure
Provisioning
Infrastructure Provisioning - Resources
Terraform CloudFormation
100s of cloud and
SaaS resources
100s of cloud and
SaaS resources
Infrastructure Provisioning - Resources
Terraform CloudFormation BOSH
100s of cloud and
SaaS resources
100s of cloud and
SaaS resources
● Compute
● Block store
● ELB & EIP
integration
Infrastructure Provisioning - Cloud Support
Terraform CloudFormation
Multi-cloud
● AWS
● Openstack
● Azure
● VMWare
● And many
more
AWS only
Infrastructure Provisioning - Cloud Support
Terraform CloudFormation BOSH
Multi-cloud
● AWS
● Openstack
● Azure
● VMWare
● And many
more
AWS only Multi-cloud
● AWS
● Openstack
● Azure
● VMWare
Application
Deployment
Application Deployment
Canary releases (canaries)
Rolling Updates (max_in_flight)
Configuration
Management
Configuration Management - Resources
Puppet/Chef/Ansible
100s of resources
Configuration Management - Resources
Puppet/Chef/Ansible BOSH
100s of resources 3 resources
● packages
● file template
● service
Configuration Management
Traditional CM tool (puppet, chef, ansible)
Configuration management for a server
Configuration Management
Traditional CM tool (puppet, chef, ansible)
Configuration management for a server
BOSH
Configuration Management for a service
Monitoring
Process monitoring (monit)
Node (VM) monitoring
Notifications
Datadog
PagerDuty
AWS CloudWatch
Monitoring
Process monitoring (monit)
Node (VM) monitoring
Notifications
Datadog
PagerDuty
AWS CloudWatch
Monitoring
Demo
ELK in AWS
Demo - ELK stack in AWS
AWS VPC
Availability zone a
BOSH subnet
Availability zone b
Public subnet
Private subnet
Public subnet
Private subnet
NAT
Gateway
NAT
Gateway
Internet Gateway
S3 bucket - bosh artifacts
BOSH Director
https://github.com/AusDTO/aws-infrastructure-provision
Demo - ELK stack in AWS
https://github.com/logsearch/logsearch-boshrelease
parser
ingestor
queue
ls-router
haproxy
Logstash
redis
logstash
ES cluster
ES data
EBS
EBS
EBS ES data EBS
ES master EBS
kibana
ES cluster monitor
ES maintenance
EBS
Logstash
S3 upload
Demo - Choose your own Adventure
1. Scale vertically
2. Scale horizontally
3. More disk space please
4. Kill a node with data
5. Add some more monitoring for your service
6. Patch VM image (AMI)