puppet for production in webex - puppetconf 2013

Cisco Confidential© 2011 Cisco and/or its affiliates. All rights reserved. 1

Puppet at Cisco CCATG

Aug 23, 2013

Reinhardt Quelle, Cloud Services Architect

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2

CCATG Cloud Services Hosts and Manages SaaS Applications

Millions of Meetings

for

10s of Millions of Users

totaling

Billions of Minutes

each month

7x24x365

Cisco Social

WebEx

Connect


US

UK

India

Australia

China

Hong Kong

Amsterdam

Japan

Global Footprint

~ 7K Hosts~ 8 Data Centers> 12 iPOPsPrivate Backbone

Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 4Cisco Confidential 4© 2011 Cisco and/or its affiliates. All rights reserved.

Deployment Automation: Keeping these screens green

while evolving the service


Systems (or services) may look complicated


…but they are made of simple parts


…composed into modules


…and assembled into a system


Our systems are similar:Puppet manages “Resources”

Files

Packages

=

Users

Services

…Etc.


Which are composed into systems

Manifests - nodes.pp - site.pp

Classes, Modules

=


So what’s missing?


TIME


Software versions, and often configuration versions, evolve as time passes:

BaseOS_Hardening v1.1ElasticSearch v0.20.6

JRE v1.7.0_25

BaseOS_Hardening v1.1ElasticSearch v0.90.2-1

JRE v1.7.0_25

• Some systems can simply be knocked over the head and recreated with fresh versions

• Others – notably most database servers – cannot; updates are performed in-place

• “Big Bang” upgrades don’t often happen; we step methodically through groups of machines


In other words, SEQUENCE


We never run one of anything

DC1 DC2

Multiple DC PairsMultiple Clusters of each Service type• By Customer Class• By Lifecycle Stage• By Special Needs


We must manage versions across these clusters, through time

By DC

Or by Node

v1v1 v2

v2

v1 v1v2v2


Cluster management includes a layer of definition and control above Puppet

“Blueprints” or “Models”

• JSON/YAML• TOSCA• CMDB *

Orchestration

• Fabric• SLiM• Mcollective


One Puppet Master?

Puppet Master

Manifests & Modules

DC1 DC2


One Puppet Master?

Puppet Master

Manifests & Modules

DC1 DC2

Guess when you’ll need to push infrastructure changes the most!


So, maybe one per DC?

Puppet Master

Manifests & Modules

DC1 DC2

Puppet Master

Manifests & Modules


Perhaps Many…

WebEx Meetings

WebEx Connect


Or, how about masterless?

puppet apply \ -–modulepath=/opt/puppet_local \ --execute “include servertype::front-end”

Manifests & Modules copy [/etc/puppet/*] to each node


Our OS’s have robust packaging systems…

Manifests & Modules .rpm or .deb

yum install app_pp_v1 && puppet apply …’

privatepackage repository


Orchestration tools provide a means of applying the changes

ssh node81 ‘yum install app_pp_v1 && puppet apply …’

fab dfw-frontends pp_apply:latest’

FabricMcollectiveSaltAnsible


Cisco/WebEx uses multiple solutions

• Application Stacks/Deployment are NOT Homogenous

• The “right” solution for one stack not always right for another

• Share as much as possible, but don’t force it

• Tightly coupled systems are often rigid, brittle

• Solving big, general problems is hard; small bites are easily digested

“A foolish consistency is the hobgoblin of little minds” – Emerson


Whether one, many, or no servers,Puppet is CODE

• Every artifact (module, manifest, Hiera file) is checked into version control

• Versions are packaged and released and should go through same promotion process as application code

• All good coding practices applyModular

Well defined interfaces

Tested

Shared


Design Pattern: ServerType/Profile/Module• Modules are the atomic packages of configuration

• “Profiles” bundle modules into commonly used sets for ease of consumption:

BaseOS

JavaApp

Tomcat App

• A given machine has exactly one “ServerType”

• Inspired by Chef’s “roles”, and similar to Craig Dunn’s Role/Profile/Modules

• At the code level, these are actually all just modules


Design Pattern: Cooperative Modules• Build loosely coupled modules that can work together if installed together,

but that can stand on own, too

• Example: standard monit config includes /etc/monit/conf.d/*Application that wants to be monitored just drops file in this location

• logrotate, collectd, apache, nginx, etc all support

class elasticsarch {

…

if $monit::include_dir != undef { validate_absolute_path($monit::include_dir) file { "${monit::include_dir}/${monit_config_file_name}": ensure => present, content => template("elasticsearch/${monit_config_file_name}.erb"), notify => Service['monit'], } }}


Design Pattern: Assembling Configuration• The singleton ServerType defines which profiles and modules are

included; structure and order

• Puppet Librarian and its Puppetfile describe which version of a module is used, and where it comes from

• After Puppet Librarian has run and downloaded all required assets, FPM is called upon to build the package

• Dev cycle includes doing local builds/tests against Vagrant in both develpers personal machines as well as the Jenkin’s build farm

• Upon successful build, packages are uploaded to repositories


Miscellaneous Tidbits

• Dependencies in Puppet, not RPM/DEB

• Packaging works for deploying to Puppetmasters, too.

• Modules are designed for transparency, simplicity: “4AM-proofing”

• Composition usually trumps inheritance

• Tim Bell and the CERN folks talk of “Pets” and “Cattle”You can only shoot a system in the head if you can create another at will

• “Fried” or “Baked”? YES.

Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 36Cisco Confidential© 2011 Cisco and/or its affiliates. All rights reserved. 36

Photo placeholder

Thank you

Please tell your friends we’re hiring DevOps Engineers!

puppet for production in webex - puppetconf 2013

Technology

cisco andor

cisco confidential

cisco ccatg

hardening v1

jre v1

node v1 v1 v2 v2 v1

manifests nodes

webex meetings webex