puppet modules for fun and profit

PUPPET MODULESFOR FUN AND PROFIT

Puppet Conf San Francisco 2012Alessandro Franceschi

Lab42 / GrandSla

PUPPET @ Lab 422007 - Meet Puppet. Managed the Bank of Italy webfarm

2008 - First generation of Lab42 Puppet Modules

2009 - Multi OS support and standardization

2010 - A redesigned and coherent Example42 Module setPuppet Modules Standards and Interoperability (PuppetCamp 2010 - Belgium)Re-Use your Modules! (PuppetCamp 2010 - San Francisco)

2011 - Introducing PuppiPuppi: Puppet strings to the shell (PuppetCamp Europe 2011 - Amsterdam)

2012 - Example42 Next Gen modules - GrandSla: Puppet driven Infrastructure and Support

Developing IT Infrastructures with Puppet (CodeMotion 2012 - Rome)A Holistic approach to Puppet modules (PuppetCamp Dublin and Geneva 2012)

“Job Driven” modules development

MODULES PATTERNS...Data Separation

Configuration data is defined outside the module (or even Puppet manifests)Module’s behavior is managed via APIsAllow module’s extension and override via external data

ReusabilityCustomize behavior without changing module codeDo not force author’s idea on how configurations should be providedSupport different OS. Easily allow new additions

StandardizationFollow PuppetLabs style guidelines (puppet-lint)Have coherent, predictable and intuitive interfacesProvide contextual documentation (puppet-doc)

InteroperabilityLimit cross-module dependenciesAllow easy modules’ cherry pickingBe self contained, do not interfere with other modules’ resources

... AND ANTI-PATTERNSData Mixed with Logic

Configuration data mixed inside the module’s logicModule’s behavior defined in many different placesModule’s logic is rigid and can’t be defined externally

Works for me (here and now)Module just works for the current setupCan’t easily be re-used in other places for other projectsWorks just for the currently used OSes

Code chaosNo layout rules, no standard styleNot standardized and predictable parametersWho said “documentation”?

InteroperabilityWho cares.

Basically whatever is quick and dirty... but is this really an anti-pattern?*

*(IMHO, yes)

Example42 modules: 10 design rulesRule 1 - Provide alternatives for Data Separation

Rule 2 - Provide choice on Configuration Files supply

Rule 3 - Configure everything but provide OS defaults.

Rule 4 - Allow management of general module’s behavior

Rule 5 - Allow Custom Options for endless parameters

Rule 6 - Permit easy extension with custom classes

Rule 7 - Offer easy removal of the module’s resources

Rule 8 - Limit cross-dependencies. Prerequisites as options.

Rule 9 - Automatically monitor and firewall resources

Rule 10 - Puppi integration: Puppet knowledge to the CLI

DATA SEPARATION ALTERNATIVESSet (Top Scope/External Node Classifier) variables and include classes:

$::openssh_template = 'site/openssh/openssh.conf.erb'include openssh

Use Hiera:hiera('openssh_template')include openssh

Use Parametrized Classes:class { 'openssh':  template => 'site/openssh/openssh.conf.erb',}

Happily mix different patterns:$::monitor = true$::monitor_tool = [ 'nagios' , 'munin' , 'puppi' ]class { 'openssh':  template => 'site/openssh/openssh.conf.erb',}

Ex

am

ple

42

Ru

le #

1

PARAMS_LOOKUP EVERYWHEREEach parameter is processed by the params_lookup functionclass openssh ([...] # openssh module specific parameters ...  $my_class = params_lookup( 'my_class' ),  $source = params_lookup( 'source' ),  $source_dir = params_lookup( 'source_dir' ),  $source_dir_purge = params_lookup( 'source_dir_purge' ),  $template = params_lookup( 'template' ),  $service_autorestart = params_lookup( 'service_autorestart' , 'global' ),  $options = params_lookup( 'options' ),  $version = params_lookup( 'version' ),  $absent = params_lookup( 'absent' ),  $disable = params_lookup( 'disable' ),  $disableboot = params_lookup( 'disableboot' ),  $monitor = params_lookup( 'monitor' , 'global' ),  $monitor_tool = params_lookup( 'monitor_tool' , 'global' ),  $monitor_target = params_lookup( 'monitor_target' , 'global' ),[...] # Other common parameters  ) inherits openssh::params {[...]}

Flexibility on booleans: they are sanitized by the any2bool function  You set: $absent => “yes” # (or “1”, ‘Y’, “true”, true ...) The module internally uses:  $bool_absent = any2bool($absent)

Ex

am

ple

42

Ru

le #

1

PARAMS LOOKUP ORDERThe function params_lookup is provided by the Puppi moduleIt allows data to be defined in different ways:

Via Hiera, if availableAs Top Scope variable (as provided by External Node Classifiers)Via defaults set in the module’s params class

The “global” argument is used to define site_wide behavior# If there’s a direct param that’s the value class { ‘openssh’: monitor => true}

# Otherwise, If Hiera is available: hiera(“monitor”) # If global lookup is sethiera(“openssh_monitor”) # A specific value overrides the global one

# If variable is still not evaluated, Top Scope is looked up:$::monitor # If global lookup is set$::openssh_monitor # If present, overrides $::monitor

# Module’s params are used as last option defaults:$openssh::params::monitor

Ex

am

ple

42

Ru

le #

1

CUSTOMIZE: CONFIGURATION FILEProvide Main Configuration as a static file ...

class { 'openssh':  source => 'puppet:///modules/site/ssh/sshd.conf'}

... an array of files looked up on a first match logic ...class { 'openssh':  source => [ "puppet:///modules/site/ssh/sshd.conf-${fqdn}",              "puppet:///modules/site/ssh/openssh.conf"],}

... or an erb template:class { 'openssh':  template => 'site/ssh/sshd.conf.erb',}

Config File Path is defined in params.pp (can be overriden):config_file => '/etc/ssh/sshd_config',

Ex

am

ple

42

Ru

le #

2

CUSTOMIZE: CONFIGURATION DIRYou can manage the whole Configuration Directory:

class { 'openssh':  source_dir => 'puppet:///modules/site/ssh/sshd/',}This copies all the files in lab42/files/ssh/sshd/* to local config_dir

You can purge any existing file on the destination config_dir which are not present on the source_dir path:

class { 'openssh':  source_dir => 'puppet:///modules/site/ssh/sshd/',  source_dir_purge => true, # default is false}

WARNING: Use with care

Config Dir Path is defined in params.pp (can be overriden):  config_dir => '/etc/ssh',

Ex

am

ple

42

Ru

le #

2

CUSTOMIZE: PATHS AND NAMESCustomize Application Parameters. An example:Use the puppet module to manage pe-puppet!

class { 'puppet':  template => 'lab42/pe-puppet/puppet.conf.erb',  package => 'pe-puppet',  service => 'pe-puppet',  service_status => true,  config_file => '/etc/puppetlabs/puppet/puppet.conf',  config_file_owner => 'root',  config_file_group => 'root',  config_file_init => '/etc/sysconfig/pe-puppet',  process => 'ruby',  process_args => 'puppet',  process_user => 'root',  config_dir => '/etc/puppetlabs/puppet/',  pid_file => '/var/run/pe-puppet/agent.pid',  log_file => '/var/log/pe-puppet/puppet.log',  log_dir => '/var/log/pe-puppet',}

Ex

am

ple

42

Ru

le #

3

DEFAULTS IN PARAMS.PPEach module has a params class with defaults for different OSclass openssh::params { ### Application related parameters  $package = $::operatingsystem ? {    default => 'openssh-server',  }  $service = $::operatingsystem ? {    /(?i:Debian|Ubuntu|Mint)/ => 'ssh',    default => 'sshd',  }  $process = $::operatingsystem ? {    default => 'sshd',  } [...] $port = '22'  $protocol = 'tcp'

# General Settings  $my_class = ''  $source = ''  $source_dir = ''  $source_dir_purge = '' [...]

### General module variables that can have a site or per module default  $monitor = false  $monitor_tool = ''  $monitor_target = $::ipaddress  $firewall = false  $firewall_tool = ''  $firewall_src = '0.0.0.0/0' [...]

Ex

am

ple

42

Ru

le #

3

MANAGE BEHAVIOREnable Auditing:

class { 'openssh':  audit_only => true, # Default: false}No changes to configuration files are actually made and potential changes are audited

Manage Service Autorestart:class { 'openssh':  service_autorestart => false, # Default: true}No automatic service restart when a configuration file / dir changes

Manage Software Version:class { 'foo':  version => '1.2.0', # Default: unset}Specify the package version you want to be installed.Set => ‘latest’ to force installation of latest version

Ex

am

ple

42

Ru

le #

4

CUSTOM OPTIONSWith templates you can provide an hash of custom options:

class { 'openssh':  template => 'site/ssh/sshd.conf.erb',  options => {    'LogLevel' => 'INFO',    'UsePAM' => 'yes',  },}

The Hash values can be used in your custom templates:- Allow management of any kind of configuration parameter- Provide endless configuration values without adding new parameters

- Works only for parameters used in templates on in custom classes

Ex

am

ple

42

Ru

le #

5

CUSTOM OPTIONS IN TEMPLATESAlternative ways to use the options hash in an erb template:

Direct but not safe (you must always provide all the used options)UsePAM <%= options['UsePAM'] %>

Failsafe with defaults (verbose but safe)<% if scope.lookupvar("openssh::options['UsePAM']") then -%>UsePAM <%= options['UsePAM'] %><% else -%>UsePAM no<% end -%>

Show what you have (useful for config files has defaults for every option)<% scope.lookupvar("openssh::options").sort_by {|key, value| key}.each do |key, value| -%><%= key %> <%= value %><% end -%>

The smart way: options_lookup (Use the option value or set a default)UsePAM <%= scope.function_options_lookup(['UsePAM',‘no’]) %>

Ex

am

ple

42

Ru

le #

5

CUSTOMIZE: CUSTOM CLASSProvide added resources in a Custom Class:

class { 'openssh':  my_class => 'site/my_openssh',}This autoloads: site/manifests/my_openssh.pp

Custom class can stay in your site module:class site::my_openssh {  file { 'motd':    path => '/etc/motd',    content => template('site/openssh/motd.erb'),  }}You hardly need to inherit openssh: there are parameters for everythingDo not call your class site::openssh, naming collisions could happen.

Ex

am

ple

42

Ru

le #

6

EASY DECOMMISSIONINGDisable openssh service:

class { 'openssh':  disable => true}

Deactivate openssh service only at boot time:class { 'openssh':  disableboot => true}Useful when a service is managed by another tool (ie: a cluster suite)

Remove openssh (package and files):class { 'openssh':  absent => true}

Monitoring and firewalling resources removal is automatically managed

Ex

am

ple

42

Ru

le #

7

CROSS-MODULE INTEGRATIONSIntegration with other modules sets and conflicts management is not easy.

Strategy 1: Provide the option to use the module’s prerequisite resources:class { 'logstash':  install_prerequisites => false, # Default true}The prerequisites resources for this module are installed automatically BUT can be managed by third-party modules

Strategy 2: Use if ! defined when defining common resourcesif ! defined(Package['git']) {  package { 'git': ensure => installed } }Not a definitive solution, but better than nothing.

Strategy 3: Always define in Modulefile the module’s dependenciesdependency 'example42/puppi', '>= 2.0.0'

Strategy 4: Never assume your resource defaults are set for othersExec { path => "/bin:/sbin:/usr/bin:/usr/sbin" }

Ex

am

ple

42

Ru

le #

8

EXTEND: MONITORManage Abstract Automatic Monitoring:

class { 'openssh':  monitor => true,  monitor_tool => [ 'nagios','puppi','monit' ],  monitor_target => $::ip_address # Default}

Monitoring is based on these parameters defined in params.pp:  port => '22',  protocol => 'tcp',  service => 'ssh[d]', # According to OS   process => 'sshd',   process_args => '',  process_user => 'root',  pid_file => '/var/run/sshd.pid',

Abstraction is managed in the Example42 monitor moduleHere “connectors” for different monitoring tools are defined and can be added (also using 3rd party modules).

Ex

am

ple

42

Ru

le #

9

EXTEND: FIREWALLManage Automatic Firewalling (host based):

class { 'openssh':  firewall => true,  firewall_tool => 'iptables',  firewall_src => '10.0.0.0/8',  firewall_dst => $::ipaddress_eth1, # Default is $::ipaddress}

Firewalling is based on these parameters defined in params.pp:  port => '22',  protocol => 'tcp',

Abstraction is managed in the Example42 firewall moduleCurrently only the “iptables” firewall_tool is defined, it uses Example42 iptables module to manage local iptables rules

Ex

am

ple

42

Ru

le #

9

EXTEND: PUPPIManage Puppi Integration:

class { 'openssh':  puppi => true, # Default: false  puppi_helper => 'standard', # Default}

The Puppi module is a prerequisite for all Example42 modulesIs required because it provides common libs, widely used in the modulesBUT the actual puppi integration is optional (and disabled by default)

Puppi integration allows CLI enrichment commands like:puppi info opensshpuppi log opensshpuppi check opensshNote: puppi support for info/log commands for NextGen modules is under development

Puppi helpers allow you to customize Puppi behavior

Ex

am

ple

42

Ru

le #

10

DOWNLOAD

Example42 Puppet Modules Site:http://www.example42.com

GitHub repositories:http://github.com/example42

Git Download:git clone -r http://github.com/example42/puppet-modules-nextgen

Note on GitHub repos:puppet-modules-nextgen contains only NextGen modules (as git submodules)puppet-modules contains both NextGen and older modules

One more thing...How to make a NextGen module

git clone -r http://github.com/example42/puppet-modules-nextgencd puppet-modules-nextgenExample42-tools/module_clone.shThis script creates a skeleton for a new module based on different Example42 foo module templates. Run it from the directory that contains the foo module (moduledir).By default it uses the "foo" module as template.Specify -t <source_module> to use a different template.Example:Example42-tools/module_clone.sh -t foo_webapp

Source module template is foo

Enter the name of the new module based on foo: mynewmoduleEdit mynewmodule/manifests/params.pp to manage dif ferent OS

A new, basic, NextGen module based on the foo template is done. Add features and application specific resources to enrich it

ad maiora

Graphics: www.tatlin.net

Questions?

@alvagante