Puppet at Bazaarvoice

Hi, my name is Dave.

• DevOps Engineer at Bazaarvoice.• Started working with puppet in 2008 while working at

Bioware. First version was 0.24.• At Bioware, puppet configured over 14k nodes that

comprised of web servers, databases and game servers.• All 5 datacenters (in California, Virginia, Ireland,

Australia and Texas) housed puppet managed nodes that all reported back to a centralized puppet dashboard.

• My contact info:– http://www.linkedin.com/in/jamesbarcelo

Bazaarvoice Plug

• We do embedded DevOps!• Application teams are responsible for their

application’s operation success.• 2.0 stack is 100% in Amazon!• Conferences!• Work on awesome projects with spiffy tech like

Cassandra or ElasticSearch.• Send me your resume!

Dave.barcelo@bazzarvoice.com

Preview presentation

• Puppet in the legacy stack.• Puppet in the Data Infrastructure Team.– Focus on client/server.

• Puppet in the Data Services Team.– Focus on masterless puppet.

Puppet in the Legacy Infrastructure

• Traditional puppet use with client/server.• Multiple levels of inheritance.• Puppet managed instances were configured

according to DNS naming convention.

node /my-hostname/ { … …}

• Some issues encountered:– Very hard to work with. Very complex.– Large codebase. Adds to complexity. – No confidence in making changes. Side effects

feared after code change. A jinga tower of puppet code.

– Too many pivot points. Many places to configure. Adds to complexity.

– Lots of code rot. Had not been refactored.

Puppet master/client in Data Infrastructure teams

Architecture

• Each server type we care about will be referenced by its role. We only care about roles, not hostnames.

• Centered around an uber IT tools server that runs everything ops (including puppet) to do work in an environment. The Mothership.

• Hiera and parameterized classes will be used to create generic puppet modules that can be reused for different roles.

• Development will be centered on using puppet environments on the Mothership to protect devs from stepping on each other.

Mothership

• Contains a cocktail of different application tools for doing work in the environment. Tools included:– Mcollective/ActiveMQ– Puppet server– Puppet managed operation scripts.

• Motherships configured to be highly available in regular AWS fashion(Autoscaling, cluster multiple activeMQ, etc).

• Advertises multiple puppet environments that clients can switch between via –environments.

What is a Universe?

Anatomy of a Mothership

Methods of passing in data

Getting environment data into puppet configuration. • Hiera datastore.• Puppet stdlib/tags.txt• Cloud formation parameters – Universe, VPC

Puppet Stdlib/facts.d

• Bootstrap process (EC2 user data) populates /etc/facter/facts.d/tags.txt with mappings. These mappings become facters.

• Example of data in tags.txt:– Universe value.– Ec2 metadata.

/etc/facter/facts.d/tags.txt:universe=devTag_region=us-east-1

Hiera datastore

• Hiera is used extensively where different data needs to be passed into puppet according to context. Different contexts would include:– Amazon region.– Instance role.– Universe.

• Example usage:$app_version = hiera(‘app_version’, nil)

/etc/hiera.yaml--:logger: console

:hierarchy:

- %{fqdn} - 10-team/%{team}/10-region/%{tag_region}/10-universe/%{universe}/10-roles/%{role} - 10-team/%{team}/10-region/%{tag_region}/10-universe/%{universe}/20-common - 10-team/%{team}/10-region/%{tag_region}/20-roles/%{role} - 10-team/%{team}/10-region/%{tag_region}/30-common - 10-team/%{team}/20-universe/%{universe}/10-roles/%{role} - 10-team/%{team}/20-universe/%{universe}/20-common - 10-team/%{team}/30-roles/%{role} - 10-team/%{team}/40-common

- 40-common

- environments

:backends: - yaml - json

:yaml: :datadir: /etc/puppet_env/%{environment}/manifests/hieradata

:json: :datadir: /etc/puppet/env/global_hieradata

Puppet Environments/Universe

Mothership Execution Flow

Puppet code on the Mothership• The files that do the things:– /etc/puppet/puppet.conf– /etc/puppet/env/global_hieradata/

environments.json– /etc/puppet/puppetmaster.conf– /etc/puppet_env/{puppet_env}/…• manifests/…• modules/…

/etc/puppet/puppet.conf# File managed by Puppet.

[main] vardir = /var/lib/puppet logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl

usecacheonfailure = true pluginsync = true factpath = $vardir/lib/facter preferred_serialization_format = yaml

[user] vardir = /var/lib/puppet logdir = /var/log/puppet rundir = /var/run/puppet

ssldir = $vardir/ssl

[agent] runinterval = 1800 ca_server = <%= ca_srv %> server = <%= logical_srv %> certificate_revocation = False environment = <%= environment %> report = true

/etc/puppet/env/global_hieradata/environments.json

{ "environments": [ { "cert": [ { "modulepath": "/etc/puppet_env/cert/modules" }, { "manifestdir": "/etc/puppet_env/cert/manifests" }, { "manifest": "/etc/puppet_env/cert/manifests/site.pp" } ] }}

/etc/puppet/puppetmaster.conf[main] vardir = /var/lib/puppet logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl

usecacheonfailure = true pluginsync = true factpath = $vardir/lib/facter preferred_serialization_format = yaml syslogfacility = local1

[master] certname=<%= certname %> ca = True certificate_revocation=False dns_alt_names=<%= logical_srv %> ssl_client_header = SSL_CLIENT_S_DN ssl_client_verify_header = SSL_CLIENT_VERIFY autosign = true

# For puppet dashboard reporting. reports = store, datadog_reports

<% if store_configs == true %> # Puppetdb.

storeconfigs = true storeconfigs_backend = puppetdb <% end %>

[user] vardir = /var/lib/puppet logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl

# Environments<% environments.each do |env_val| -%><% env_val.keys.each do |env_key| -%>[<%= env_key -%>]<% env_val[env_key].each do |env_data| -%><% env_data.each_pair do |k, v| -%><%= k %> = <%= v -%><% end %><% end %><% end %><% end %>

/etc/puppet_env/{env}/manifests/site.pp

import 'roles/*.pp'

node default { class { 'basenode_role': }

class { "$tag_role": }}

/etc/puppet_env/{env}/manifests/role/00_basenode.pp

class basenode_role { class { ‘security’: } class { ‘monitoring’: } …..}

/etc/puppet_env/{env}/manifests/role/mothership_role.pp

class mothership_role {

class { 'puppet': master => true, ca_srv => $tag_caserver, logical_srv => $tag_puppet_server, }

}

Mothership Dev Workflow

Masterless Puppet in Data Services Teams

Architecture

Still keeping bits of the Mothership project:• Applications/Services scoped in zookeeper by

Universe.• Emphasis is put on making things simpler.

Puppet code will not be monolithic. Individual application teams will only need to maintain there own modules/manifests.

• Changes to modules/manifests will not impact other teams.

Methods of passing in data

• The usual suspects:– Puppet stdlib/tags.txt.– Hiera.– Cloud formation parameters – Universe, VPC

• Some new ones:– EC2 data/metadata -> facter.– Zookeeper.– Cloud formation parameters - DeployTag

getEC2data_cache.rb

• Script runs out of /etc/facts.d that converts EC2 data/metadata into facts.

Zookeeper/Ostrich

• Custom functions to pull data from zookeeper the same way applications do discovery.

Masterless Execution Flow

Puppet code in Masterless

• No more Mothership. All work is done via puppet apply.– /etc/hiera.yaml– /etc/puppet/manifests/{role}.pp– /etc/puppet/manifests/00_common.pp– /etc/puppet/manifests/01_users.pp– /etc/puppet/modules/…

/etc/hiera.yaml--:logger: console

:hierarchy: - %{fqdn}

- 10-universe/%{universe}/10-roles/%{role} - 10-universe/%{universe}/20-common - 20-roles/%{role} - 30-common

:backends: - yaml

:yaml: :datadir: /etc/puppet/manifests/hieradata

/etc/puppet/manifests/{role}.ppimport '00_common'

node default {

# This class contains common modules that should be used by all roles. class { 'common': }

class { 'activemq’: } -> class { 'mcollective': server => true, client => true, }

}

/etc/puppet/manifests/00_common.ppimport '01_users'###################################### Common#####################################

class common {

class { 'stdlib': }

file { '/opt/bazaarvoice': ensure => directory, }

# Authorized keys for project developers. class { 'user_setup': stage => setup, }

host { 'internal_ip': ensure => 'present', name => $fqdn, ip => $ipaddress, }

class { 'prompt': }}

/etc/puppet/manifests/01_users.ppclass user_setup {

include users

users::user { 'dbarcelo': groups => 'wheel', sshKey => 'ssh-dss AAAAB3NzaC1kc3MAAACBANL1zoZdYJp/6vQ4G5iNQXjdJ7NGmK0J2eqHbztvuD0CBPyqMuEtuYKRg14tFd4iwp5EpnT4UWpv8kXF/dkEN3b5xgN/R+1hYq7/3mnRLchMFTl0tyryLuARC9zTI003mQrXd/W9jzXaNlCTpxh8Ihj2Ov3lvAAX65tN9nijxhCTAAAAFQCgMU0obmTLo5CRYtRwDCkj1mb2hQAAAIAiZF9axkCvMa9vwigDiAf3rNMbut1gtqtwdzux8c9T1inApKV5sccjg5POKm+4WmWTBOtQfYR8cNot2Mn/mO+MRiKH8sYapYnU2es+KRBmhdARE+N7EqdD0WqoP7NrsNVbObHwDQBNkODuc3ZPyTQuqv/w4poTXaS5u5M1XZbgZwAAAIEAjt4r7SN1I/m0V/TvedTVxJvKln4wZkFxyI5CAgpsAr435kwSLM08R9Hd0/5Vy9LfhYpH1aZTBaoTqmTCtnv3mp1coXoscEp5nE0llfm+4DX3YvWnR80S/OeMUe71Ucm1ORwFpST/K4WKQoZ30TAVVsc8nYy2hyD7hyozjzsS09o= dave.barcelo@dbarcelo-mbpro' } users::user { 'lwadhwani': groups => 'wheel', sshKey => 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAysFCPpffw9LIOqAEFZxOOb52m2FbHhumBFc07o8sm3c4cmdLq/bBtr5TyuQp89zVNEaTGRbw1nMpQCDno4i5ipTvCLoKkOE1PRdtyJw6PGu6VV/0U1ghK+1xmveM2jDX/otj5hjnQiRm1+Fx/orYwNBkywDlDHZQCGxalWaFgXVyReCRUqq0jBwj3EKJfsQgoxuSrh7F6GjsQ6DUOsA3wBfewS25hPmhulEqvga4/P58BMHemL9d4Ugu98Vg7fgaur/b1adX+LzbmE6C2T4Gn1kzAOEct6bFgLPRj3n5/EaspdOsZ/Nnik0LUvIwZNHgDCLgkS0D8aMIsiUrB4OqSw== luveen@Pantalaimon' }}

/etc/puppet/modules/…

• Do stuff!• Code is still generic but it does not have to be.

Masterless Dev Workflow