Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 1

3 Years of Puppet at Cisco: The Secrets to Our Success

PuppetConf 2013

Keith Chambers & Ryan Uber

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2

•  Worked at Cisco for 13 years -  7 years in Technical Assistance Center (support)

-  2 years in Product Marketing

-  4 years as WebEx Social Platform Architect

•  Passionate about technology

•  Passionate about people

•  Always looking to raise the bar

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3

•  Worked at Cisco for 3 years -  Software Engineer

-  Has played a critical role in our success

•  Work at a managed host provider for 5 years -  Designed and implemented most of the infrastructure

-  Supported everything he built

•  Puppet user for 6 years

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4

•  Overview

•  Experience

•  Plumbing

•  History & Evolution

•  Secrets to our success

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6

•  Enterprise Social Media -  Create content -  Communities -  Follow -  Like -  Activity feed -  Recommendations

•  On-premise deployment model

•  Later added cloud hosted

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7

Persistence

•  Traditional tiered web service

•  Large number of services -  App Server -  Worker -  OpenFire -  RabbitMQ -  Memcached -  MongoDB -  Oracle -  Solr

Caching

Application

Web Server

Worker

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8

•  Virtual appliance delivery model

•  Roles based architecture

-  1 service per VM -  25 VM deployments are

common

•  Director provides system wide:

-  Configuration -  Software maintenance

-  Diagnostics and logging

-  Health monitoring

•  Everything works out of the box

Oracle

OS MongoDB

OS

Solr

OS

Worker

OS

Memcached

OS App

OS

RabbitMQ

OS OpenFire

OS

Virtualized Compute & Storage

Director

OS

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10

1.  Customer downloads a single OVA from Cisco

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11

1.  Customer downloads a single OVA from Cisco

2.  Customer deploys the Director VM from the OVA by selecting the Director configuration option

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13

1.  Customer downloads a single OVA from Cisco

2.  Customer deploys the Director VM from the OVA by selecting the Director configuration option

3.  Customer configures Director VM network settings

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14

1.  Customer downloads a single OVA from Cisco

2.  Customer deploys the Director VM from the OVA by selecting the Director configuration option

3.  Customer configures Director VM network settings

4.  Customer browses to the Director UI and configures system settings and defines the topology

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17

1.  Customer downloads a single OVA from Cisco

2.  Customer deploys the Director VM from the OVA by selecting the Director configuration option

3.  Customer configures Director VM network settings

4.  Customer browses to the Director UI and configures system settings and defines the topology

5.  Customer deploy all VMs defined in the topology from the OVA by selecting the appropriate configuration

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18

1.  Customer downloads a single OVA from Cisco

2.  Customer deploys the Director VM from the OVA by selecting the Director configuration option

3.  Customer configures Director VM network settings

4.  Customer browses to the Director UI and configures system settings and defines the topology

5.  Customer deploy all VMs defined in the topology from the OVA by selecting the appropriate configuration

•  System handles all other setup and configuration

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19

1.  Customer download a single release image from file from Cisco

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20

1.  Customer download a single release image from file from Cisco

2.  Customer uploads the release image to the Director

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22

1.  Customer download a single release image from file from Cisco

2.  Customer uploads the release image to the Director

3.  Customer clicks upgrade

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23

1.  Customer download a single release image from file from Cisco

2.  Customer uploads the release image to the Director

3.  Customer clicks upgrade

•  System handles the upgrade

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27

OVF

OS

Bootstrap Config

Network Settings Director FQDN

Glue Code

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29

•  Package everything as an RPM

•  RPMs distributed from Director via Yum

•  Controller is a Puppet Master

Director

OS

Analytics Store

OS

Puppet Master Yum

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30

•  Versioned build artifact containing: -  All RPMs for all roles -  All Puppet modules

•  Upgrade mechanics: -  New image uploaded to Director -  Old image unmounted and new

image mounted -  Director calls Puppet run for all

hosts -  On success the updated version

is stored in on VM file system -  Salt used to fetch version

number from all VMs

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31

Director

OS

Analytics Store

QUAD OS

JSON Store

QUAD OS

RDBMS Store

QUAD OS

Graph Store

QUAD OS

Rsyslog

•  Rsyslog -  High performance -  Reliable -  Part of CentOS 6

Rsyslogx Rsyslogx Rsyslogx Rsyslogx

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32

Director

OS

Analytics Store

OS

BSON Store

OS

RDBMS Store

OS

Social Graph

OS

monit monit monit monit

Nagios

•  Nagios: -  Reliable -  Extremely flexible -  Available in EPEL

•  Monit -  Reliable -  Flexible and extensible -  Enables self healing -  HTTP API -  Available in EPEL

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33

Director

OS

Analytics Store

OS

BSON Store

OS

RDBMS Store

OS

Social Graph

OS

Graphite

collectd collectd collectd collectd

•  Graphite: -  Reliable -  Extremely flexible -  Available in EPEL

•  Collectd -  Reliable -  100+ plugins -  Extensible -  In EPEL

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35

Phase 1 Mid 2010 •  20GB VM •  CentOS 5 •  deployer.zip •  Monitoring 1.0

•  100% proprietary •  Time to market focused •  No DevOps tooling

experience in the team

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36

Phase 1 Mid 2010 •  20GB VM •  CentOS 5 •  deployer.zip •  Monitoring 1.0

•  100% proprietary •  Time to market focused •  No DevOps tooling

experience in the team

Phase 2 Early 2011 •  RPM + Yum •  Puppet •  Release Image •  Reduce VM footprint

•  Major pushback on Puppet •  Your team matters

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37

Phase 1 Mid 2010 •  20GB VM •  CentOS 5 •  deployer.zip •  Monitoring 1.0

•  100% proprietary •  Time to market focused •  No DevOps tooling

experience in the team

Phase 2 Early 2011 •  RPM + Yum •  Puppet •  Release Image •  Reduce VM footprint

•  Major pushback on Puppet •  Your team matters

Phase 3 Mid 2011 •  Director REST API •  OVF Properties •  Unified Logging w/

Rsyslog + Scribe

•  Rsyslog was resisted due to name association with syslog

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38

Phase 1 Mid 2010 •  20GB VM •  CentOS 5 •  deployer.zip •  Monitoring 1.0

•  100% proprietary •  Time to market focused •  No DevOps tooling

experience in the team

Phase 2 Early 2011 •  RPM + Yum •  Puppet •  Release Image •  Reduce VM footprint

•  Major pushback on Puppet •  Your team matters

Phase 3 Mid 2011 •  Director REST API •  OVF Properties •  Unified Logging w/

Rsyslog + Scribe

•  Rsyslog was resisted due to name association with syslog

Phase 4 Early 2012

•  CentOS 6 upgrade •  Monit •  Salt •  Remove Scribe •  Maintain fewer libraries •  Further reduce VM footprint

•  Started replacing Monitoring 1.0 •  Salt proved buggy

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39

Phase 1 Mid 2010 •  20GB VM •  CentOS 5 •  deployer.zip •  Monitoring 1.0

•  100% proprietary •  Time to market focused •  No DevOps tooling

experience in the team

Phase 2 Early 2011 •  RPM + Yum •  Puppet •  Release Image •  Reduce VM footprint

•  Major pushback on Puppet •  Your team matters

Phase 3 Mid 2011 •  Director REST API •  OVF Properties •  Unified Logging w/

Rsyslog + Scribe

•  Rsyslog was resisted due to name association with syslog

Phase 4 Early 2012

•  CentOS 6 upgrade •  Monit •  Salt •  Remove Scribe •  Maintain fewer libraries •  Further reduce VM footprint

•  Started replacing Monitoring 1.0 •  Salt proved buggy

Phase 5 Late 2012 •  Replaced Monitoring 1.0 with

Nagios, Collectd, Graphite •  Last minute decision to go with

Graphite •  Puppet code spaghetti •  VM tampering undercutting upgrade

reliability •  Other Cisco groups want to reuse

our code

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40

Phase 1 Mid 2010 •  20GB VM •  CentOS 5 •  deployer.zip •  Monitoring 1.0

•  100% proprietary •  Time to market focused •  No DevOps tooling

experience in the team

Phase 2 Early 2011 •  RPM + Yum •  Puppet •  Release Image •  Reduce VM footprint

•  Major pushback on Puppet •  Your team matters

Phase 3 Mid 2011 •  Director REST API •  OVF Properties •  Unified Logging w/

Rsyslog + Scribe

•  Rsyslog was resisted due to name association with syslog

Phase 4 Early 2012

•  CentOS 6 upgrade •  Monit •  Salt •  Remove Scribe •  Maintain fewer libraries •  Further reduce VM footprint

•  Started replacing Monitoring 1.0 •  Salt proved buggy

Phase 5 Late 2012 •  Replaced Monitoring 1.0 with

Nagios, Collectd, Graphite •  Last minute decision to go with

Graphite •  Puppet code spaghetti •  VM tampering undercutting upgrade

reliability •  Other Cisco groups want to reuse

our code

Phase 6 Started 2012 •  Generic programmable

Controller •  Deploy VMs via IaaS API •  Orchestrate deploy and upgrade

with Fabric •  Baked VM images •  Hiera for module parameters

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42

Keep it simple • 80/20 rule • Build only what you need • What works is boring

Learn and adapt • Don’t fall in love

with what you’ve built

Unix tool chain pattern • Loosely couple with

best of bread tools • End-to-end

solutions lock you in

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43

•  Excellent at configuration management

•  Bulletproof reliability

•  Long term investment protection

•  Vibrant community and ecosystem

•  Commitment to open source

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44

•  Alex Honor of DTO Solutions and Simplify Ops -  Insightful program leadership -  Walking encyclopedia of proven patterns -  Always there when I’m in a jam

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45