rackspace private cloud presentation for chefconf 2014

1

April 16, 2014

Ops and Chefs: Lessons learned while working on OpenStack deployment cookbooks

#chefconf 2014

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Who Are We?

Justin Shepherd

Principal Architect

Rackspace Private Cloud

github.com/galstrom21

Joseph Breu

Software Dev Team Lead US

Rackspace Private Cloud

github.com/rackerjoe

@rackerjoe

3RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Support Team

•Manages and Operates OpenStack clouds for our customers

•Troubleshoots any issues a customer might encounter with their cloud

Dev Team

•Creates tools for deploying and configuring OpenStack clouds

•Makes initial decision on when a new project is “stable” enough to enable in the customer clouds

A little context


In the beginning….

The Dev Team were experts at deployments

Best Practices from the Ops Team


Over time….

The Ops team took the reigns and became the experts.

New expertise breeds new Best Practices!



In the end….

How do you codify this new knowledge?



Nova

• Has 1000+ configuration options

• has support for 4 hypervisor platforms

Neutron

• has support for ~15 network platforms

Cinder

• has support for ~20 storage platform

Flexibility vs. Standardization


Limit platform options based on current expertise

• We chose KVM because it had the best support in the community, and we have lots of Linux expertise in house

• We chose NetApp initially and leveraged our existing storage team

Add more options as needed based on demand

• We need compelling reasons to add features

Each addition increases QE test matrices exponentialy!



Even limiting scope it takes 61,480 lines of cookbook to implement a working OpenStack cluster



Issues with cookbooks• Setting an incorrect value in nova’s config file

• Absence of a useful attribute

• Cookbook dependencies and upstream breakage (COOK-2676)

Issues in the upstream OpenStack projects• Hopefully these are known issues, and the community is addressing them

Issues in the upstream OS packages• Vendor X breaks a package (i.e. kernel and openvswitch)-where X is all of them!

Issues in foundation technologies• MySQL

• RabbitMQ

Issue Tracking


The usual suspects

• Ubuntu LTS

• RHEL

• CentOS

The usual problems

• Independent packaging process

• Separate bug tracking process

• Packages are cut at different points in time

Multiple Base Operating Systems


OpenStack releases a new “stable” version every 6 months.

• There are also 3 scheduled minor version releases for the stable version during the next 6 month cycle.

• Although none of the upstream packages match these releases

Deciding on your Release Schedule


We support the current “stable” version and one prior

• So our customers do not have to update every 6 months

• Upgrades are currently disruptive

We currently release minor version updates every other month

• Includes cookbook bug fixes

• Includes latest available packages



This is probably one of the easiest decision to make and the hardest to get right

• We have changed our model multiple times in the last 3 years

• We will probably change it again

Go ask @claco Thursday 2:50pm Regency A/B about this



Major changes can occur every 6 months

• Package renames

• Service renames

• New/Deprecated config options

Services get split into their own projects

• Nova-Volumes -> Cinder

• Nova-Network -> Neutron (kind of)

• Nova-Scheduler -> Gantt

Managing Chef deployed OpenStack


Inter-release upgrades

• Sweeping attribute key renames

• node[‘quantum’] -> node[‘neutron’]

• Have to remap customers environments

• https://github.com/rcbops/mungerator

• New BUGS!

• That never happens.. it is a new version.. it must be a feature

OS Upgrades

• Terrible, terrible, and more terrible

Managing Chef deployed OpenStack


Jenkins is used for testing and gating of our cookbooks

• Syntax Verification (pep8, foodcritic)

• Full deployment of OpenStack utilizing the cookbooks with the proposed changeset applied

• Functional Tests of OpenStack

• OpenStack API Testing

• OpenStack CLI Testing

Upstream opscode cookbooks are tested before inclusion

Testing

rackspace private cloud presentation for chefconf 2014

Internet