September 24, 2014

Erik Dalén

Experiences from running masterless Puppet

whoami• System engineer at Spotify since almost 3 years.

• Community contributor to Puppet.

• Author of Puppet Explorer web UI.

• Author of the puppetdbquery, dnsquery and puppet ls modules.

2

Agenda• Background about our puppet setup

• How we implemented masterless

• Why we did the switch to masterless

• New workflows and future improvements

3

Background• Puppet users since 4 years

• Two large Puppet installations, 20+ Puppet masters.

• A single repository with all modules

• Git branches mapped to Puppet environments

• Code review in Gerrit to merge into production branch

• Tests using AWS virtual machines on every code review

4

Section nameSection name

ImplementationStarted with just

switching to the same workflow but masterless.

Using a custom Ruby “wrapper” around puppet apply.

5

SITE X

git mirror git mirror

SITE X

git mirror git mirror

SITE Y

PuppetDB

NodeNode

Node

Gerrit

● ServerDB● LDAP

Gerrit replication Gerrit replication

git pullgit pullfailover if pod mirrors are unavailable

Queries

* Queries* Store facts* Store catalog* Store report

hierasecrets

hiera secrets

HTTPS GET

auth by cert

Puppet CArequest

certificate

Benefits of this setup• Debug logging of compilation and hiera lookups

• Ability to get traceback of custom functions directly on the node

• Deprecation warnings from compilation visible on the node

$ sppuppet hiera resolvconf::servers 8.8.8.8 8.8.4.4

6

Drawbacks• Requires more configuration, and can’t use puppet to manage it.

• Quite specialised.

7

Secret management• Small web service where nodes authenticates using their SSL cert

• Calculates the facts for the node from the certificate

• Sends the secret hiera data that applies to that particular node

8

PuppetDB access control• Very binary per default, allow everyone or a whitelist

9

Node Apache PuppetDB

mod_ext_filter

jq

Why switch to masterless?• Scaling the Puppet masters not an issue

• But scaling the workflow is an issue

10

Growing amount of committers11

Num

ber

of m

onth

ly c

omm

itter

s

0

55

110

165

220

2 years ago 1 year ago now

Growing amount of modules12

Num

ber

of m

odul

es

0

150

300

450

600

2 years ago 1 year ago now

More complex dependencies• 126 modules using the apache module

• 91 modules using the nginx module

• 92 modules using the postgresl module

Backwards incompatible changes to shared modules almost impossible to do.

13

Flexible workflow needed• Allow one set of modules per Puppet run determined by hiera

• Easier ways to do continuous delivery of both application and configuration

• Gradual rollout of new module versions

14

Why not r10k or librarian-puppet?Both great utilities that allow building a environment dynamically. But it is still a fixed environment for many nodes.

production_new_apache production_new_apache_new_postgres production_old_apache_new_postgres production_new_apache_old_postgres

15

Our solution to the problemAt each puppet

run we look in hiera data which modules to install.

16

hiera/common.yaml: modules: spotify-hostbase: latest hiera/role/spotify_web.yaml modules: spotify-apache: latest hiera/role/puppetexplorer.yaml modules: puppetlabs-apache: latest

Internal forge• A simple forge implementation using the new V3 forge API

• Mirrors the upstream forge

• Will be used for distributing internal modules as well

github.com/jhaals/puppet-anvil

17

Questions?