puppet module reusability - what i learned from shipping to the forge

Puppet Module ReusabilityLearning from shipping to the Forge

Gareth Rushgrove

Who(Who is this person?)

@garethr

UK Government Digital Service

Last codeI wrote

7. Gareth Rushgrove

Last puppet module I wrote

The Problem(Sharing modules is hard)

Not invented here syndrome

1

A search for puppet-apache

145Gareth Rushgrove

145Gareth Rushgrove

Puppetlabsapache

67 contributors

Gareth Rushgrove

281forks

Gareth Rushgrove

281Gareth Rushgrove

Vendor everything pattern

2

Not publishing to the Forge

3

Puppet Forge

Undocumented modules

4

Official docs guidelines

(Too) opinionated configuration

5

Sometimes I just want to write nginx configuration

Gareth Rushgrove

Duplicate resources6

Package { 'build-essential’: ensure => installed,}

Gareth Rushgrove

Stdlib(Start here)

Standard Library from Puppetlabs

Fail fast

Gareth Rushgrove

Validation

Gareth Rushgrove

Useful error messages

Gareth Rushgrove

fail("${::operatingsystem} not supported")

Gareth Rushgrove

validate_string

Gareth Rushgrove

validate_string($version)

Gareth Rushgrove

<Puppet::Error: true is not a string.

Gareth Rushgrove

validate_slength

Gareth Rushgrove

validate_re

Gareth Rushgrove

validate_hash

Gareth Rushgrove

Gareth Rushgrove

validate_cmd

validate_bool

Gareth Rushgrove

Avoid duplicate resources

Gareth Rushgrove

Package { 'build-essential’: ensure => installed,}

Gareth Rushgrove

include 'gcc'

Gareth Rushgrove

package{[ 'python-pip', 'python-ldap',]: ensure => installed}

Gareth Rushgrove

ensure_packages

Gareth Rushgrove

ensure_packages([ 'python-pip', 'python-ldap',])

Gareth Rushgrove

ensure_resource

Gareth Rushgrove

Nicer interfaces

Gareth Rushgrove

str2bool

Gareth Rushgrove

any2array

Gareth Rushgrove

And much more

Gareth Rushgrove

Dependency management is

everywhere(Else)

NPM, Bundler, Pip, Mix, Leiningen

Gareth Rushgrove

Librarian Puppet

r10k

echo 'modules' >> .gitignore

Gareth Rushgrove

dependency 'puppetlabs/stdlib'dependency 'garethr/erlang'dependency 'maestrodev/wget'

Gareth Rushgrove

forge "http://forge.puppetlabs.com"

mod 'puppetlabs/ruby'mod 'puppetlabs/ntp'mod 'puppetlabs/git'mod 'puppetlabs/vcsrepo'mod 'puppetlabs/apt'mod 'puppetlabs/gcc'

Gareth Rushgrove

librarian-puppet install

Gareth Rushgrove

A Nice Module Pattern(Structuring modules for reuse)

R.I.Pienaar

install, config, service, params

Gareth Rushgrove

manifests!"" config.pp!"" init.pp!"" install.pp!"" params.pp#"" service.pp

Gareth Rushgrove

class sample () inherits sample::params {

class { 'sample::install': } -> class { 'sample::config': } ~> class { 'sample::service': } -> Class['sample']}

Gareth Rushgrove

anchor { 'sample::begin': } ->class { 'sample::install': } ->class { 'sample::config': }class { 'sample::service': } ->anchor { 'sample::end': }

Gareth Rushgrove

Anchor['sample::begin'] ~> Class['sample::service']

Class['sample::install'] ~> Class['sample::service']

Class['sample::config'] ~> Class['sample::service']

Gareth Rushgrove

Puppet module tool

Gareth Rushgrove

puppet module generate name

Gareth Rushgrove

Default module skeleton

Gareth Rushgrove

In puppet source code

!"" manifests!"" spec!"" tests!"" Modulefile!"" .gitignore#"" README

Gareth Rushgrove

Creating your own module skeleton

Gareth Rushgrove

~/.puppet/var/puppet-module/skeleton

Gareth Rushgrove

puppet module skeleton

!"" manifests!"" spec!"" tests!"" templates!"" files!"" lib!"" Gemfile

Gareth Rushgrove

!"" Rakefile!"" .nodeset.yml!"" .fixtures.yml!"" .travis.yml!"" Modulefile!"" .gitignore#"" README.md

Gareth Rushgrove

Creates install, config, service, params classes

Gareth Rushgrove

Dependency management with Bundler

Gareth Rushgrove

Better unit testing setup using rspec-puppet-helper

Gareth Rushgrove

Adds syntax and lint checking

Gareth Rushgrove

Adds Travis CI configuration

Gareth Rushgrove

Adds integration tests with rspec-system

Gareth Rushgrove

Adds module management with maestrodev blacksmith

Gareth Rushgrove

Focus on the interface

Gareth Rushgrove

Minimize number of entry points

Gareth Rushgrove

Allow overriding provided templates

Gareth Rushgrove

Multi-OS Support(Where to abstract the differences)

Vary default parameters

Gareth Rushgrove

Keep logic in one place

Gareth Rushgrove

case $::osfamily { 'Debian': {  }  'RedHat', 'Amazon': { } default: { fail("${::operatingsystem} not su}

Gareth Rushgrove

ARM-9 Data in Modules

garethr-erlang

0.0.x supported Ubuntu only

Gareth Rushgrove

0.1.x supports Ubuntu, Debian, Redhat, Suse

Gareth Rushgrove

Gareth Rushgrove

Pull requests to the rescue

Insist on tests with open source contributions

Gareth Rushgrove

Module Testing(Code should have tests)

Why test a declarative language?

Gareth Rushgrove

Modules increasingly contain logic

Gareth Rushgrove

Modules increasingly take arguments

Gareth Rushgrove

Modules increasingly have interfaces with other modules

Gareth Rushgrove

Modules increasingly used in many Ruby and Puppet version combinations

Gareth Rushgrove

Good news. The tools got better

Gareth Rushgrove

puppet-lint

Gareth Rushgrove

Puppetstyle guide

Availableas a gem

puppet-lint --with-filename /etc/puppet/modulesfoo/manifests/bar.pp: trailing whitespace found on line 1apache/manifests/server.pp: variable not enclosed in {} on line 56

Gareth Rushgrove

puppet-syntax

Gareth Rushgrove

Validate Puppet and ERB syntax

require 'puppet-syntax/tasks/puppet-syntax'

Gareth Rushgrove

bundle exec rake syntax---> syntax:manifests---> syntax:templates

Gareth Rushgrove

rspec-puppet

Gareth Rushgrove

Unit testing for Puppet

context "epel enabled" do let(:params) {{ :epel_enable => true }} it { should contain_class('epel') }end

Gareth Rushgrove

context "epel disabled" do let(:params) {{ :epel_enable => false }} it { should_not contain_class('epel') }end

Gareth Rushgrove

Fixtures, matchers

Test the interface not the implementation

Gareth Rushgrove

All of the above

Gareth Rushgrove

task :test => [ :syntax, :lint, :spec,]

Gareth Rushgrove

bundle exec rake test

Gareth Rushgrove

Gareth Rushgrove

Nice continuous integration

Test pull request branches too

---language: rubybefore_install: rm Gemfile.lock || truervm: - 1.8.7 - 1.9.3script: bundle exec rake testenv: matrix: - PUPPET_VERSION="~> 2.7.0" - PUPPET_VERSION="~> 3.1.0" - PUPPET_VERSION="~> 3.2.0"

Gareth Rushgrove

A matrix of possibilities

Gareth Rushgrove

rspec-system

Gareth Rushgrove

Integration test against real systems

Puppet helpers for rspec-system

Vagrant driver

VSphere provider

Test that Puppet runs without errors

Gareth Rushgrove

it 'should run without errors' do pp = "class { 'sample': }"

puppet_apply(pp) do |r| r.exit_code.should == 2 endend

Gareth Rushgrove

Test runs are idempotent

Gareth Rushgrove

it 'should run without errors' do pp = "class { 'sample': }"

puppet_apply(pp) do |r| r.exit_code.should == 2 r.refresh r.exit_code.should be_zero endend

Gareth Rushgrove

Test that the module installs relevant binaries

Gareth Rushgrove

it 'should install the erl binary' do shell 'which erl' do |r| r.stdout.should =~ /\/usr\/bin\/e r.stderr.should be_empty r.exit_code.should be_zero endend

Gareth Rushgrove

Test against different operating systems

Gareth Rushgrove

default_set: 'centos-64-x64'sets: 'centos-64-x64': nodes: "main.foo.vm": prefab: 'centos-64-x64' 'fedora-18-x64': nodes: "main.foo.vm": prefab: 'fedora-18-x64' 'debian-607-x64': nodes: "main.foo.vm":

Gareth Rushgrove

Room for improvements

(Making reuse easier in Puppet)

Run your own Forge1

A more secure Forge2

Bless a dependency management tool

3

Optional dependencies

4

dependency 'puppetlabs/stdlib'

optional_dependency 'puppetlabs/apache'

Gareth Rushgrove

Private classes5

private class elixir::install {

Gareth Rushgrove

Parameter naming conventions

6

example42Proposal

Interfaces in Puppet7

interface packageandservice { $version $enable $package_name $template}

Gareth Rushgrove

class diamond ( $version,

$enable, $package_name, $template,) { implements packageandservice

Gareth Rushgrove

Questions?(And thanks for listening)