Story of Puppet @eBay Classifieds Group or how we came to the setup we use

Some bio or “Who is Vladimir” ! Sysadmin for eighteen years

!  Currently in Solutions team of eCG’s marktplaats.nl

!  All-round nerd

!  Hobbyist programmer

Puppet at eBay Classifieds: !  9 countries

!  13 platforms

!  1 codebase

!  1 change management process

Puppet at eBay Classifieds: !  We started adopting puppet in 2008

!  Since then we had about 4 different iterations of “this is bloated and doesn’t work, let’s think more and re-do”

!  Currently we believe we achieved a workable scenario where we: !  Enable teams to share and re-use puppet code

!  Enforce the same code style and standards

!  Allow for peer-reviews of the code, also enforce them

“If it’s not in Puppet – it does not exist.”

What we are going to talk about: •  Different iterations of puppet implementation at eCG

•  Go in-depth on the structure of the current one

•  Process around the codebase

•  Tools

“Bucket of recipes”

Puppet at eBay Classifieds (1st implementation): !  A bunch of modules without visible structure (server-role based)

!  Relations between modules on any levels (includes, instantiations)

!  One Subversion repository, where everyone works on trunk

What were the lessons learned?

ü  As puppet adoption grew, modules for different platforms started clashing (same, yet a bit different functionality)

ü  As number of modules and their dependencies grew people were afraid of making changes - dependency tree was very hard to follow

ü  Resulted in tons of duplicate code

ü  Change management was very risky – you commit something and it’s live

“Buckets of recipes”

Puppet at eBay Classifieds (2nd implementation): !  What did we improve?

!  Release management with Subversion branching

!  Better module grouping

What were the lessons learned?

ü  Waiting on releases is killing. If you need a change - you need it now, not in 2 or 3 days

ü  Provokes people to lock machines to their own puppet environment

ü  Surprises at times of the releases. For some people it was daytime, for others it was middle of the night

Puppet at eBay Classifieds (3rd implementation): !  What did we improve?

!  Switched to Git

!  Completely redesigned modules structure (more on this later)

!  Introduced extlookup for data separation vs configuration logic

!  Release more often

What were the lessons learned?

ü  When release process is human-dependent it will be a bottleneck

ü  Extlookup is nice, but in many places unreadable

Puppet at eBay Classifieds (4th implementation): !  What did we improve?

!  Introduced Gerrit for code reviews

!  Implemented automatic puppet-lint, template syntax and PEP8 checkers

!  Introduced automated quorum votes for peer reviews

!  Release as soon as your code passed checks and YOU need it live

!  Further improved module tree structure

!  Replaced extlookup with hiera

!  Introduced puppetdb

So how does it look now?

Shared

eCG

Platform

Config

!  Four module levels

!  One include direction

!  No includes on the same level

Shared modules

Shared

eCG

Platform

Config!  Shared modules are very basic, deploy one piece

of software in its standard configuration (no customization)

!  Standalone. Includes no other modules

!  Installs needed packages, basically

!  Parameterized to allow configuration override !  Examples: apache, ntpd, activemq, mysql, postgres

eCG modules

Shared

eCG

Platform

Config! eCG modules add our specific eCG-wide

approved customizations. Rule: it must play well with majority of teams to make it

!  Mostly include shared modules

!  Never include each other

!  Override default shared module configuration with eCG specific one via parameters

!  May be parameterized to allow for configuration tweaks or overrides !  Examples: puppetmaster, nagios checks, collectd

plugins

Platform modules

Shared

eCG

Platform

Config

!  Platform level introduces company-specific modules and configuration customizations

!  May include eCG modules

!  May include shared modules

!  Never include each other

Config modules

Shared

eCG

Platform

Config

! Config level introduces server-role specific modules

!  Are the starting point for our servers configuration

!  May include platform, eCG or shared modules

!  Never include each other

!  Node definition must include config module

Why did we do it like this?

Shared

eCG

Platform

Config

!  Easy starting point. Look up a node, see its class, open the class – you see on one page how your server is configured on a global level

!  If you need to know more, dive deeper

!  We enable code sharing, we do not enforce it !  If you see a module in shared you can use – use

it. If you can’t use it, extend it in a backwards-compatible way or write other extension on an appropriate level

Hiera

!  Great way to separate code from data

!  Company-based structure

!  Lookup path based on facts to allow company data separation while keeping the same source tree :datadir: /var/lib/hiera/%{environment}/%{companyprefix}

!  Inside company dirs a variety of fact-based lookups to allow for fine-grained data structures

PuppetDB

!  Much faster catalog compilations for huge catalogs (nagios server with 200k resources)

!  REST API to talk to for inventory and searches

!  Many brilliant ways to (ab)use it !  Stores all facts (can be used to feed CMDBs)

!  Stores most recent catalog (can be used for inspections)

!  POC we are running - internal Puppet naginator module is very slow, collection of exported resources takes about 15 minutes. We use PuppetDB to generate the same nagios configs in just under 8 seconds

!  “If there is data, it can be used”. PuppetDB is a central data store. Love.

So all good, but how does this work in practice?

!  Per-user environments on puppetmasters

!  Make the change in your own env

!  Test it from your own env

!  Push to Gerrit

!  Ask team members to review

!  Merge

Code reviewPush to GerritTest the change with --environment

Local environment Merge

Gerrit

!  Great code review tool (https://code.google.com/p/gerrit/)

!  Acts as a Git server with fine-grained access control

!  Each commit is a separate change, that requires votes for approvals

!  Votes can be +1 for Code Review, +2 for change approval

!  No +2 – no merge

!  We use voting system with auto +2 !  Depending on platform size or number of platforms affected by the

change more votes are needed to get a change through

!  Each change can be “pushed through” in case of an emergency

More Gerrit goodies

!  Allows for post-push hooks !  Puppet-lint

!  Ruby templates parsing

!  PEP8 (we love Python)

!  Each change is separately checked out by hooks and tested for syntax or other obvious errors

!  Concept of one commit – one change forces people to create atomic changes that are easy to revert in case of errors

!  Keeping it all in the same system, even if you decide to split Git repositories still makes work done by colleagues VISIBLE

Gerrit and puppetmasters

!  Master is automatically pulled by crons on puppetmasters

! Gerrit disallows pushing directly to master

!  Even if you force-merge a change, it makes your change VISIBLE and easily REVERTIBLE

!  Takes about half a minute extra time

!  Ensures you didn’t slack out on syntax checkers

To sum it all up

!  There is no single structure that fits every company

!  Puppet code is like any other code !  Write

!  Use

!  Refactor if needed

!  Layered modules structure can work well in many cases

!  Complementary tooling around Puppet is a life- (and time-) saver !  Lint(s), Gerrit, PuppetDB

Work to be done in our house

!  Approval process improvements !  Better quorum schemas

!  Easier ways of code style and syntax checking

!  See that people don’t leave changes “hanging”

!  Integrate Gerrit with corporate GitHub

!  Work more with community !  Open source some generic shared modules

!  Offer changes to PuppetForge modules

!  POC with Vagrant for puppet code testing

Questions?

Thank you!

You can reach me by: •  Mailing to favoretti@gmail.com •  Following me on GitHub (

https://github.com/favoretti)

Special thanks to:

•  PuppetLabs and eCG for the opportunity to speak here •  Teams of eCG for bearing with me all these years •  You all for attending and listening

We are hiring! Visit http://ebayclassifiedsgroup.com/careers for more info.