autoscaling distributed system with bosh (cloud foundry summit 2014)
DESCRIPTION
Technical Track presented by Yudai Iwasaki, Lead Engineer, Software Innovation Center, NTT. Yudai Iwasaki is a research engineer at NTT Software Innovation Center. He is a core member of the development team of Cloudn PaaS, which is a public PaaS solution provided by NTT Communications, and is leading the development of their deployment system for Cloud Foundry. He is also the leader of community relationships at NTT and is a member of the Japan Cloud Foundry Group, in which capacity he gives lectures on the structure of Cloud Foundry for Japanese Cloud Foundry developers. He is a developer of Nise BOSH, which is a light weight BOSH emulator, BOSH CloudStack CPI, and BOSH AutoScaler.TRANSCRIPT
© 2014 Nippon Telegraph and Telephone Corporation
AUTOSCALING CLOUD FOUNDRY WITH BOSH Yudai Iwasaki NTT Service Innovation Laboratory Group
CF Summit 2014
© 2014 Nippon Telegraph and Telephone Corporation
• Core developer of Cloudn PaaS • Working on Cloud Foundry since 2012
– Nise BOSH, BOSH CloudStack CPI and BOSH AutoScaler • Twitter: @i_yudai
© 2014 Nippon Telegraph and Telephone Corporation
Agenda • Problem – System load is not fixed
• Solution – BOSH AutoScaler
• Getting Started
© 2014 Nippon Telegraph and Telephone Corporation
Problem: System Load is
Not Fixed
© 2014 Nippon Telegraph and Telephone Corporation
Daily, Weekly, Monthly, and Yearly Peaks
© 2014 Nippon Telegraph and Telephone Corporation
Daily, Weekly, Monthly, and Yearly Peaks
© 2014 Nippon Telegraph and Telephone Corporation
Manually Scaling? Always monitor load?
© 2014 Nippon Telegraph and Telephone Corporation
Impossible No one wants to do such a boring job
© 2014 Nippon Telegraph and Telephone Corporation
Solution: BOSH AutoScaler
© 2014 Nippon Telegraph and Telephone Corporation
BOSH AutoScaler • Extension for BOSH
• Flexible scaling policies in deployment manifest files
• Special support for Cloud Foundry
© 2014 Nippon Telegraph and Telephone Corporation
Architecture Overview
© 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler (with CF Plugin)
Heartbeat
Heartbeat (subscribe)
process (BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy (PUT /deployments)
matching rules against log periodically Logging metrics（@buffers） run()
Agent Agent
Agent Agent
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF
CF Component CF Component
CF Component
Collector
TSDB
Varz
Load policies (GET /deployments)
© 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler (with CF Plugin)
Heartbeat
Heartbeat (subscribe)
process (BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy (PUT /deployments)
matching rules against log periodically Logging metrics（@buffers） run()
Agent Agent
Agent Agent
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF
CF Component CF Component
CF Component
Collector
TSDB
Varz
Load policies (GET /deployments)
© 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler (with CF Plugin)
Heartbeat
Heartbeat (subscribe)
process (BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy (PUT /deployments)
matching rules against log periodically Logging metrics（@buffers） run()
Agent Agent Agent Agent Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF CF Component
CF Component CF Component
Collector
TSDB
Varz
Load policies (GET /deployments)
© 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler (with CF Plugin)
Heartbeat
Heartbeat (subscribe)
process (BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy (PUT /deployments)
matching rules against log periodically Logging metrics（@buffers） run()
Agent Agent Agent Agent Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF CF Component
CF Component CF Component
Collector
TSDB
Varz
Load policies (GET /deployments)
© 2014 Nippon Telegraph and Telephone Corporation
Policy Definitions
© 2014 Nippon Telegraph and Telephone Corporation
Defining policies
• Adding “scale” section • Defining conditions for
each job • Various condition classes
to define flexible policies
-‐-‐-‐ name: cf .... scale: jobs: -‐ name: router cooldown: 300 out: limit: 10 unit: 2 conditions: -‐ class: CpuAverage larger_than: 80 duration: 300 -‐ class: MemoryAverage larger_than: 90 duration: 300 in: limit: 3 conditions: -‐ class: CpuAverage smaller_than: 10 duration: 300 -‐ class: MemoryAverage smaller_than: 20 duration: 300
© 2014 Nippon Telegraph and Telephone Corporation
Conditions: BOSH Heartbeat • CpuAverage
– Average CPU percentage for duration • MemoryAverage
– Average memory percentage for duration • LoadAverage1
– Latest Load Average in 1 minute • LoadAverage5
– Latest Load Average in 5 minutes • LoadAverage15
– Latest Load Average in 15 minutes
© 2014 Nippon Telegraph and Telephone Corporation
Conditions: CF Plugin • CFVarzAverage
– Average Varz value for duration
DEA available_memory_ratio available_disk_ratio
Router total_routes latency.1m
HM9000 NumberOfRunningInstances NumberOfCrashedInstances
etcd SendingRequestRate ReceivingRequestRate
Loggregator Server receivedMessageCount numberOfWebsocketSinks
Cloud Controller connection_count threadqueue.num_waiting
© 2014 Nippon Telegraph and Telephone Corporation
Web UI
© 2014 Nippon Telegraph and Telephone Corporation
© 2014 Nippon Telegraph and Telephone Corporation
© 2014 Nippon Telegraph and Telephone Corporation
© 2014 Nippon Telegraph and Telephone Corporation
Getting Started It’s easy to plug in
© 2014 Nippon Telegraph and Telephone Corporation
https://github.com/nttlabs/bosh-scaler
© 2014 Nippon Telegraph and Telephone Corporation
-‐-‐-‐ # BOSH NATS settings nats: &nats uri: mbus://192.168.50.4:21084 user: nats pass: nats # BOSH Director REST API settings rest: &rest endpoint_uri: https://192.168.50.4:25555 user: scaler password: scaler disable_verify_certification: true
collectors: -‐ class: BoshNatsCollector bosh_nats: *nats bosh_rest: *rest -‐ class: CfVarzTsdbCollector port: 4567 listeners: -‐ class: BoshScaler bosh_rest: *rest interval: 60 # seconds buffer_size: 1000 ui: enable: true port: 8888
Configuration
© 2014 Nippon Telegraph and Telephone Corporation
... properties: collector: use_tsdb: true deployment_name: cf opentsdb: # your AutoScaler address address: 192.168.15.139 port: 4567
Cloud Foundry Manifest
© 2014 Nippon Telegraph and Telephone Corporation
Optimize Running Costs with BOSH AutoScaler