puppet at github - puppetconf 2013

Puppet at GitHub

@wfa r rGitHub

Operations

Known Aliases:

King of Kebabs

The Chairman

Mr. Caremad !

The State of Puppet at GitHub

"

" The State of Puppet at GitHub

github/boxen


~1.5 years old


open-sourced ~7 months ago


~240 open-source puppet modules


puppet 3.latest


supports ruby 1.8.7, 1.9.3, 2.0.0


hiera


OS X


Linux support in-progress


#tomorrow @ 2:20pm


github/puppet


$


~5 years old


0.24.x — 2.7.x


121 modules


~280k lines of code


every single employee has commit access


15.5k commits to master past 12 months


by 86 contributors


% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %

% %


% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %

% %

42.5%


% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %% % % % % % % % % % % % % % % % % % % %

% %

now with ops taken out


0

100

200

300

40020

12-0

8-25

2012

-10-2

0

2012

-12-15

2013

-02-

09

2013

-04-

06

2013

-06-

01

2013

-07-

27

commits on master per week, last 12 months


0

1000

2000

3000

4000total commits by author past year


0

175

350

525

700total commits by author past year, except ops


0

175

350

525

700total commits by authors with >10 commits, past year, except ops


single puppetmaster


rubygems


ruby 1.8.7


unicorn


puppet 2.7.latest


~600 nodes


run hourly via crond


puppetdb


nagiosdb


"yo puppetdb, gimme all your nagios::object::* resources so I can

update this nagios config"


filtergendb


"yo puppetdb, gimme all the filtergen::rule resources I would realize

so I can update this filtergen config"


⚡ puppetdb ⚡


as it turns out, an api call is faster than running puppet on a host


gpanel


"Imagine Puppet Dashboard meets Razor and went on a weekend trip to the

beach with the Heroku API and drank epic amounts of blue drink"


aka we reinvented our own, smaller version of Foreman


inventory


app configuration


versioning of configuration values


Create nil => 1Update 1 => 2Delete 2 => nil


let's make credential rolling less awful


provisioning


provisioning is typically awful


we sprinkled in some ChatOps


a little bit later...


enc


we have the dumbest ENC out there


⋆


# /usr/local/sbin/fetch_gpanel_enc \ fe1.rs.github.com

---parameters: gpanel_cabinet: D20-13 gpanel_enabled: true gpanel_monitored: false


we never specify classes via the ENC


any variables we pass through are prefixed with gpanel_

How GitHub writes Puppet

(

( How GitHub Writes Puppet

rodjek/puppet-lint


if you aren't using puppet-lint to audit your puppet codebase,

you are doing it wrong


puppet-lint enforces the Puppet Labs style guide


puppet-lint can even fix a ton of linter errors for you


put it in a pre-commit hook


$ git commit -am "can't lint this"

modules/github/manifests/role/redis.pp: syntax okmodules/github/manifests/role/redis.pp - WARNING: => is not properly aligned on line 118

1 errors found, aborting commit.


and then buy rodjek a beer


rodjek/rspec-puppet


if you aren't writing tests for your puppet code before

running it on a server,you are doing it wrong


use whatever framework/library


rspec-puppet just happens to be a pretty good one


put it in a pre-commit hook


$ git commit -am "tests dont pass but whatever lol"

1) Expected redis::server would include class "more_than_a_single_c_thread"

1 failures encountered, aborting commit.


and then buy rodjek another beer


node definitions


we don't use an ENC to describe node classes


node /^github-redis\d+/ { class { 'github::role::polling_redis': enabled => $::gpanel_enabled, environment => $::gpanel_environment, private_ipv4 => $::ipaddress, }}


abstractions all the way down


treat your site classes as cascades down to your dist classes


class redis::server( # params) {

class { 'redis::config': ... } -> class { 'redis::package': ... } ~> class { 'redis::service': ... }

}


class github::redis( # params) inherits github::defaults {

$memory = $environment ? { 'stg' => '2G', default => $half_memory_gb }

class { 'redis::server': ... }

}


class github::role::polling_redis( # params) {

class { 'github::redis': # overrides based on specific node }

}


augeas


you know what's not awesome?


an erb template that requires your class to take 52876423 parameters so you can configure every possible

value in my.cnf


augeas { 'my.cnf/performance': context => '/files/etc/mysql/my.cnf/mysqld', changes => [ # automatic dump/restore 'set innodb_auto_lru_dump 18000',

# innodb "set innodb_buffer_pool_size ${innodb_buffer_pool_size}", 'set innodb_log_file_size 256M', 'set innodb_log_buffer_size 8M', 'set innodb_lazy_drop_table 1', ],require => Percona::Server[$::fqdn]

}


it can seem complex and scary


it is an amazing tool

How GitHub ships Puppet

)

) How GitHub Ships Puppet

continuous integration


commit gets pushed


jenkins runs the test suite


status gets posted back to GitHub.com


tmm1/test-queue


remembers how to better parallelize tests


continuous deployment


tests passed for commit, Hubot auto-deploys


branch deploy everything


a lot of shops have a few environments


testing

staging

production


currently we have 181 environments


that's not the same as 181 nodesnot running production


we only have 5 of those


Hubot automatically merges the master branch before allowing any

branch to deploy


ChatOps


202 employees100% access to Puppet

100% trust


the list of roles you'd never think would touch production puppet

webkit developercore git developers

core ruby developerssvn developers

search developerfrontend designers

windows developersmac developers

supportenterprise sales


every puppet run happens in chat


all puppet output goes to chat


everyone can see everything


everyone can do anything


hands-on learning, by accident


eliminate disruptive questions

The Future of Puppet at GitHub

*

* The Future of Puppet at GitHub

puppet 3.x


upgrading a large, old codebase from 2.7 to 3 is really painful


we've been talking about it for a year


we're still trying to push forward, but...


helping maintain security fixes for 2.7


goal is to be on 3.2 by end of year


ruby 2.0


moar faster rubby


we get this for free when we move to 3


mcollective


aka the story of github/shell and the wonders of ssh in a for-loop


there are a lot of cool things about mcollective


I think it will be a part of our stack in the future


but you can still go pretty damn far with bash and ssh


even more puppetdb tooling


waiting for multiple runs to converge exported resources is painful


use the puppetdb API to skip all that


even more gpanel integration


"databags"


"node search"


steal good ideas from other tools

closing thoughts

"all software is terrible"

— anyone who's worked with software long enough

"no software is better than no software"— rtomayko

the plumbing doesn't matter when all you care about is the porcelain

write some damned good porcelain

THIS IS RODJEK

HE'S NORMALLY NOT THIS BLURRY

HE IS ALSO NOT MY BROTHER, BUT YOU REALLY SHOULD BUY

HIM A BEER

YOU SHOULD BUY THIS MAN A LOT OF BEER

THANKS

https://speakerdeck.com/wfarr/

puppet-at-github-puppetconf-2013

!

puppet at github - puppetconf 2013

Technology

state of puppet

puppet dashboard

github puppetdb

github gpanel

github provisioning

github enc

github githubboxen

github hiera