autoscaling openstack natively with heat, ceilometer and lbaas

201 WalkthroughAutoscaling OpenStack natively with Heat, Ceilometer and LBaaS

Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.

SpeakersJason GrimmConsulting Systems EngineerCisco OpenStack Private Cloud

Sharmin ChokseyTechnical LeaderCisco Cloud Services

Shixiong ShangChief Technology OfficerNephos6

AgendaIntroductions <

About The Workshop <

Environment Setup <

Background & Use Cases <

Services Enabling Autoscale <

Hands-On <

About The Workshop Tips and LogisticsContent and Schedule


▪ We all work for or with Cisco, but this is not a marketing workshop▪ Highly technical and all content is 100% FOSS (Free and Open-Source Software)▪ All Materials available at https://github.com/grimmtheory/autoscale

▪ Please connect to http://cisco.webex.com/rooms/jasgrimm▪ Chat, voting, questions, recording, etc.▪ Allows us to see when everyone is ready to proceed▪ Session recording will be made available after the summit

▪ Please work with your peers at your table!▪ Bandwidth constraints and technical issues mean might not get a test environment up in time▪ Some of you may have more experience with this technology and can assist others▪ General collaboration and discussion

Workshop Tips and Logistics

https://github.com/grimmtheory/autoscale


http://cisco.webex.com/rooms/jasgrimm


Content and ScheduleIntroductions

About The Workshop** Environment Setup **

Autoscale Background and Use CaseServices Enabling Autoscale

Hands-On WorkshopLab 1

Lab 2.1Lab 2.2Lab 3

Q & AOpen Discussion

Close

30 Minutes(or less)

45 Minutes(or more)

15 Minutes(or less, as needed)

Environment SetupVirtual Box, Ubuntu, DevStack

Install test environmentInstall content location and examplesTest environment logical architectureTest environment configurationTest environment build process


Install Test Environment…InstructionsBefore we can begin working with LBaaS, autoscale, etc. We need an environment.

For this purpose we’ve chosen a locally installed environment due to past experiences with WIFI coverage and bandwidth at OpenStack summits.

1) Mount media

2) Copy data folder over locally

3) Install VirtualBox

4) Install Vagrant (Optional)

5) Boot Virtual Machine

DVD Local Repo USB Key Internet

HTTP HTTPMOUNTMOUNT

Step 1Mount Data

Step 2Copy Data

Step 3Install Apps

Step 4Boot VM


Install content location and examples

Also available at:• My VM - http://192.168.1.100/autoscale• My Cloud Server - http://38.84.67.227/autoscale• VirtualBox - https://www.virtualbox.org/wiki/Downloads

Vagrantfile

Git clone example

Wget example

http://192.168.1.100/autoscale

http://38.84.67.237/autoscale

https://www.virtualbox.org/wiki/Downloads


Test environment logical architecture


Test environment configurationDisabled

Trove

Sahara

Cinder

Nova Network

Tempest

Swift

Enabled

MySQL

RabbitMQ

Keystone

Horizon

Glance

Neutron

Ceilometer

Heat

OS & OpenStack Configuration$HOME = /home/vagrant/devstack

Operating System = Ubuntu 14.04.xOpenStack version = “stable/kilo”

eth0 = NAT, default GWeth1 = Management / API networketh2 = Neutron / Bridge network

Open vSwitchL3 / DHCP / Floating IP

“Flat” networking also availableNo tenant VLANs, FWaaS or VPNaaS

Vagrant / VirtualBox ConfigurationVagrant v.1.7.2

VirtualBox 4.3.28eth0 = NAT

eth1 = Host Only 192.168.33.2 /24eth2 = Host Only 0.0.0.0

VT-x off, APIC-IO off, promiscuous on eth2


Test environment build processMethodology• Tools-based install and config of the OS,

DevStack and post-install tasks.• Automate difficult configuration of services

and pre-requisites (Neutron, Heat, Ceilometer, OVS, custom images, etc.)

Build Options• Base - OS install & DevStack configured• Staged - Base + DevStack install completed

and basic configuration (security groups, networks and subnets created, etc.)• Complete - Staged + all objects created

(LBs, VIPs, Ceilometer alarms, Heat templates, etc.)

CREATE MACHINE2 vCPU, 4 GB RAM, VT-x, APIC-IO

3 NICs - eth0 NAT, eth1 API, eth3 VMsPromiscuous mode for Neutron Bridge

DEVSTACK PRE INSTALLInstall pre-requisitesClone git repositories

Configure local.conf, bridges, routes users, groups,iptables, ovs, sysctl, permissions, etc.

INSTALL OSBase Install

Patch / UpdateInstall VBox extensions

DEVSTACK INSTALLInstall and monitorDevStack Install

DEVSTACK POST INSTALL BASIC

Add DNS to subnetsEdit security groups

Create / install SSH keys

DEVSTACK POST INSTALL ADV.

Create load balancersAdd members to pool

Create VIPs

COMPLETESnapshot the

virtual machine

Backgroundand Use Cases

What is Autoscaling and when / how is it used?


Autoscale Background and Use CasesWhat is Autoscaling…?“…a method used in cloud computing, whereby the amount of resources in a server farm scales automatically based on the load on the farm. It is closely related to, and builds upon, the idea of load balancing.”

Who uses Autoscaling…?Any workload that benefits from programmatic scaling up or down as load dictates.

Particularly true from a cost perspective when high workloads are very infrequent.

Some examples are:• Retail - Black Friday, holidays, video game / movie launch, etc.• Academic / Research - HPCC• Media - Video / Audio rendering• Analytics - Big data, data mining, data modeling• Security - Federal or private data heuristic, cyber defense, etc.

How does Autoscaling work…?

ServerLoad

StressMeter Alarm Action Server

Result+ + + +

Services Enabling Autoscaling

An overview of:HeatCeilometerNeutron w/ LBaaS

Heat Overview


HEATOpenStack Orchestration Service

Heat makes “clouds” rise

heat The heat tool is a CLI which communicates with the heat-api to execute AWS CloudFormation APIs. End developers could also use the heat REST API directly.

heat-api The heat-api component provides an OpenStack-native REST API that processes API requests by sending them to the heat-engine over RPC.

heat-api-cfn The heat-api-cfn component provides an AWS Query API that is compatible with AWS CloudFormation and processes API requests by sending them to the heat-engine over RPC.

heat-engine The heat engine’s main responsibility is to orchestrate the launching of templates and provide events back to the API consumer.

Infrastructure resource abstraction and orchestration:• Physical compute Power• Physical switches• Physical block Storage• …

OpenStack resource abstraction and orchestration:• VM Instances• Virtual Networks• Virtual Routers• …

“Typically” Virtual(See Ironic Project)

Ceilometer & Neutron LBaaS OverviewCeilometer & Heat Overview


Ceilometer Overview

1"

Polling"agents"

Meter" Transformer(s)"

New"Meter"

Publishing"Pipeline"

API" Database"

AMQP"

Receiver"

Collector"

Other"OpenStack""Service"

alarm"evaluator"

Publisher(s)"

alarm"evaluator"

Alarm"


Ceilometer and Neutron LBaaS

2"

Polling"agents"

Meter"

Neutron"(LBaaS)"

Neutron"API"

LBaaS"Default"Meters" Type" Unit"

network.services.lb.ac0ve.connec0ons1 gauge1 connec0on1

network.services.lb.incoming.bytes1 cumula0ve1 B1

network.services.lb.total.connec0ons1 cumula0ve1 connec0on1

network.services.lb.vip1 gauge1 vip1



3"

Transformer(s)"

New"Meter"


Publisher(s)"

sinks:&&&&&'&name:&lb_total_conn&&&&&&&transformers:&&&&&&&&&&&'&name:&"rate_of_change"&&&&&&&&&&&&&parameters:&&&&&&&&&&&&&&&&&source:&&&&&&&&&&&&&&&&&&&&map_from:&&&&&&&&&&&&&&&&&&&&&&&&name:&"network.services.lb.total.connec;ons"&&&&&&&&&&&&&&&&&&&&&&&&unit:&"connec;on"&&&&&&&&&&&&&&&&&target:&&&&&&&&&&&&&&&&&&&&&map_to:&&&&&&&&&&&&&&&&&&&&&&&&&name:&"network.services.lb.total.connec;ons.rate"&&&&&&&&&&&&&&&&&&&&&&&&&unit:&"connec;ons/s"&&&&&&&&&&&&&&&&&&&&&type:&"gauge"&&&&&&&publishers:&&&&&&&&&&&'&no;fier://&

sources:&&&&&'&name:&lb_total_conn&&&&&&&interval:&60&&&&&&&meters:&&&&&&&&&&&'&"network.services.lb.total.connec;ons"&&&&&&&sinks:&&&&&&&&&&&'&lb_total_conn&

Configuration file: /etc/ceilometer/pipeline.yaml



4"

API" Database"

Receiver"

Collector"

ceilometer sample-list -m network.services.lb.total.connections.rate !+--------------------------------------+--------------------------------------------+-------+--------+---------------+---------------------+ !| Resource ID | Name | Type | Volume | Unit | Timestamp | !+--------------------------------------+--------------------------------------------+-------+--------+---------------+---------------------+ !| 9d944aec-7d66-46b5-a71b-434e4e9fa98f | network.services.lb.total.connections.rate | gauge | 0.0 | connections/s | 2015-05-13T02:45:02 | !| 9d944aec-7d66-46b5-a71b-434e4e9fa98f | network.services.lb.total.connections.rate | gauge | 0.0 | connections/s | 2015-05-13T02:44:02 | !| 9d944aec-7d66-46b5-a71b-434e4e9fa98f | network.services.lb.total.connections.rate | gauge | 0.0 | connections/s | 2015-05-13T02:43:02 | !+--------------------------------------+--------------------------------------------+-------+--------+---------------+---------------------+ !!!!ceilometer statistics -m network.services.lb.total.connections.rate !+--------+---------------------+---------------------+-----+-----+-----+-----+-------+----------+---------------------+---------------------+ !| Period | Period Start | Period End | Max | Min | Avg | Sum | Count | Duration | Duration Start | Duration End | !+--------+---------------------+---------------------+-----+-----+-----+-----+-------+----------+---------------------+---------------------+ !| 0 | 2015-05-13T02:16:02 | 2015-05-13T02:47:02 | 0.0 | 0.0 | 0.0 | 0.0 | 32 | 1860.0 | 2015-05-13T02:16:02 | 2015-05-13T02:47:02 | !+--------+---------------------+---------------------+-----+-----+-----+-----+-------+----------+---------------------+---------------------+ !



5"

API"

alarm"evaluator"

alarm"evaluator"

Alarm"

ceilometer alarm-threshold-create --name lb_conn_rate_hi \ !--description 'lb connection rate is too high' --severity critical --enabled True \ !--meter-name network.services.lb.total.connections.rate \ !--period 60 --evaluation-periods 3 --statistic avg --comparison-operator gt --threshold 2.0 \ !--repeat-actions True !

ceilometer alarm-list !+----------+-----------------+--------+----------+---------+------------+-----------------------------------------------------------------+ !| Alarm ID | Name | State | Severity | Enabled | Continuous | Alarm condition | !+----------+-----------------+--------+----------+---------+------------+-----------------------------------------------------------------+ !| uuid | lb_conn_rate_hi | ok | critical | True | True | network.services.lb.total.connections.rate > 2.0 during 3 x 60s | !+----------+-----------------+--------+----------+---------+------------+-----------------------------------------------------------------+ !


Ceilometer and Heat

6"

Polling"agents"

Meter" Transformer(s)"

New"Meter"


API" Database"

AMQP"

Receiver"

Collector"

alarm"evaluator"

Publisher(s)"

alarm"evaluator"

Alarm"

Heat"

Hands-On Workshop

Heat feature progressionBasic Heat OperationsAdvanced Heat OperationsAutoscale

‹#›

25

Auth Model• Icehouse• Security Isolation• Keystone Domain “heat”• Stack Domain Users managed by

heat_admin_domain• Pre-configured in devstack• Requires Keystone v3

Resource Groups• Icehouse• De-coupling hierarchical nested

stacks

Stack Snapshot/Rollback

Provider Resources• Icehouse• Useful in Template composition

TripleO Heat Templates• Leverages Resource Groups,

Provider Resources and SoftwareConfig Resource

Key enhancements in Icehouse and

• Kilo

SoftwareConfig Resources• Icehouse• Useful in Template composition

‹#›26

Heat Autoscaling - Lab Exercise - 1 (Simple-Server)

Spawn a VMInjects an ssh keySecurity groupsVM port fixed ip + security groupFloating ip associated it with the fixed ipInstalls a simulated http server via the -init user-data script

heat stack-create simple-stack -f simple-server.yaml --parameters \"key_name=<key_name>\;node_name=<node_name>\;node_server_flavor=<node_server_flavor>\;node_image_name=<node_image_name>;\;floating_net_id=<floating_net_id>;\;private_net_id=<private_net_id>;\;private_subnet_id=<private_subnet_id>"

User-definedConstrained and/or default valuesType/Description

Fundamental blocks

Consumed by the Heat engine to

orchestrate

‹#›27

Heat Autoscaling - Lab Exercise - 2 Part 1 (Load balancer)

lb vip :: floating ip

curl -X GET http://128.107.2.111503 Service Not Available

‹#›28

Heat Autoscaling - Lab Ex - 2 Part 2 (Adding Members)web-server.yaml environment.yaml

heat stack-create lb-members-stack -f lb-members.yaml \-e environment.yaml --parameters \"key_name=<key_name>\;node_name=<node_name>\;node_server_flavor=<node_server_flavor>\;node_image_name=<node_image_name>;\;floating_net_id=<floating_net_id>;\;private_net_id=<private_net_id>;\;private_subnet_id=<private_subnet_id>;\;pool_id=<pool_id>\;capacity_count=<capacity_count>"

curl -X GET http://128.107.2.111Welcome to 192.168.5.116

curl -X GET http://128.107.2.111Welcome to 192.168.5.117

Nested Resource

Identically Configured

lb-members.yaml

Provider Resource

‹#›

Scales arbitrary # of resources

Resource to be scaled

29

Heat Autoscaling - Lab Ex - 3 (Autoscaling)lb-autoscale-members.yaml

Creates an Alarm

on a

Action to be taken upon Alarm

Raise

Action to be executed against

resource

Connection rate crosses 3 per second for a consecutive period of 60 seconds each, the lb_connection_rate_high will be raised and a scaleup_policy will get executed

ReferencesWeb Links(Content we used as referencewhen building this lab)Extra slides


Summit Lab / Workshop

▪ Materials for the presentation are available for download at https://github.com/grimmtheory/autoscale

DevStack

▪ DevStack Configuration Guide – http://docs.openstack.org/developer/devstack/configuration.html

Ceilometer

Heat

▪ Heat main page - https://wiki.openstack.org/wiki/Heat

▪ Heat developer documentation -http://docs.openstack.org/developer/heat/ Heat operations documentation

Heat main github repository

Heat templates github respository

Heat user template guide

Heat developer template guide

▪ Heat Orchestration Template (HOT) guide – http://docs.openstack.org/developer/heat/template_guide/hot_guide.html

▪ Heat on DevStack guide – http://docs.openstack.org/developer/heat/getting_started/on_devstack.html

References



http://docs.openstack.org/developer/devstack/configuration.html

http://docs.openstack.org/developer/devstack/configuration.html

https://wiki.openstack.org/wiki/Heat

http://docs.openstack.org/developer/heat/

http://docs.openstack.org/developer/heat/template_guide/hot_guide.html

http://docs.openstack.org/developer/heat/template_guide/hot_guide.html

http://docs.openstack.org/developer/heat/getting_started/on_devstack.html

http://docs.openstack.org/developer/heat/getting_started/on_devstack.html


References

Title LinkAutoscale workshop repository https://github.com/grimmtheory/Heat: main page https://wiki.openstack.org/wiki/HeatHeat: developer documentation http://docs.openstack.org/developer/heat/ Heat: operations documentation Heat main github repository


https://wiki.openstack.org/wiki/Heat

http://docs.openstack.org/developer/heat/

autoscaling openstack natively with heat, ceilometer and lbaas

Internet

cisco andor

cisco confidential

test environment instructions

installed environment

environment setup virtual

autoscale background

marketing workshop

content location