PUPPETCustom Modules & Using the Forge

INTRODUCTIONAaron Bernstein – Aspiring Engineer

▪ One of the lead developers in the WorkspaceTeam at GoDaddy

▪ Been working with GoDaddy for about a year and a half.

▪ Specifically tasked on our FaxThruEmail product.

▪ Experience includes:

▪ Fifteen years of programming in a multitude of languages…

▪ Responsible for system administration over the last decade.

▪ Currently working with operations/engineering on:

▪ Automating our configuration and package management (RPM/YUM, Puppet);

▪ Improving our build (deploy) process (Jenkins, Spacewalk, Koji/Sake);

▪ Implementing continuous integration efforts for our product (Github, Unit Testing).

Scott Eichner – Aspiring Engineer

▪ One of the lead developers in the WorkspaceTeam at GoDaddy

▪ Been working with GoDaddy for about 9 years.

▪ Specifically tasked on our Webmail product for the past 6 years.

▪ Experience includes:

▪ Nine years of programming in a multitude of languages…

▪ Recently tasked with DevOps role for workspace platform team.

▪ Beginning to work on the Cloud Platform team.

PRESENTATION STYLES& DISCLAIMERS• Bold = Keywords• Italicized = Applicable Terms• Courier = Code Samples• Green = GoDaddy• Underlined = Links

!! All contents within this presentation have come from open sources, all credit goes to those who created it. I have merely become a messenger of the knowledge. It’s open source, pass it around !!

The MIT License (MIT)

Copyright (c) 2013 Aaron Bernstein & Scott Eichner

Permission is hereby granted, free of charge, to any person obtaining a copy of this presentation and associated documentation files (the “Presentation"), to deal in the Presentation without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Presentation, and to permit persons to whom the Presentation is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Presentation.

THE PRESENTATION IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS ORIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR INCONNECTION WITH THE PRESENTATION OR THE USE OR OTHER DEALINGS IN THE PRESENTATION.

WHAT ARE MODULES?

▪ The smallest unit of measure that Puppet builds from.

▪ Generally thought of as a configuration containing each of our three core requirements, installation, configuration and monitoring.

▪ Typically breaks down to classes,

▪ Classes are singleton collections of resources.

▪ Directories with files, arranged in a specific, predictable structure.

▪ Manifest files within a module have to obey certain naming restrictions.

▪ This set of directories is known as the modulepath, which is a configurable setting.

▪ If a class is defined in a module, you can declare that class by name in any manifest. Puppet will automatically find and load the manifest that contains the class definition.

GIVE YOUR MODULES PURPOSE▪ Classes are Puppet’s way of separating

out chunks of code, and modules are Puppet’s way of organizing classes so that you can refer to them by name.

▪ What task do you need your module to accomplish?

▪ What work is your module addressing?

▪ What higher function should your modulehave within your Puppet environment?

▪ Puppet Labs best practice strongly recommends creating multiple moduleswhen and where applicable. The practice of having many small, focused modules is encouraged, as it promotes code reuse and turns modules into building blocks rather than full solutions.

Fax Software

Modems

Configurations

Fax

Tomcat

Caching Systems

DSN Broker

Configurations

Application

Webserver

API Client

Caching Systems

DSN Broker

Configurations

Web

User Profiles

Basic System Packages

Security Patches

Base

DESIGN PATTERN:NODE/ROLE/PROFILES/MODULES/RESOURCES

▪ A node includes one role, and one only;

▪ A role includes one or more profiles to define the type of server;

▪ A profile includes and manages modules to define a logical technical stack;

▪ Modules manage resources and should only be responsible for managing aspects of the component they are written for.

MODULE DIRECTORY STRUCTURE▪ my_module — This outermost directory’s name matches the name of the module.

▪ manifests/ — Contains all of the manifests in the module.

▪ init.pp — Contains a class definition. This class’s name must match the module’s name.

▪ other_class.pp — Contains a class named my_module::other_class.

▪ my_defined_type.pp — Contains a defined type named my_module::my_defined_type.

▪ implementation/ — This directory’s name affects the class names beneath it.▪ foo.pp — Contains a class named my_module::implementation::foo.

▪ bar.pp — Contains a class named my_module::implementation::bar.

▪ files/ — Contains static files, which managed nodes can download.

▪ service.conf — This file’s URL would be puppet:///modules/my_module/service.conf.

▪ lib/ — Contains plugins, like custom facts and custom resource types.

▪ templates/ — Contains templates, which the module’s manifests can use.

▪ component.erb — A manifest can render this template with template('my_module/component.erb').

▪ tests/ — Contains examples showing how to declare the module’s classes and defined types.

▪ init.pp

▪ other_class.pp — Each class or type should have an example in the tests directory.

▪ spec/ — Contains spec tests for any plugins in the lib directory.

DESIGN PATTERN:PACKAGE/FILE/SERVICE

▪ Also referred to as “THE TRIFECTA”

▪ Learn it, live it, love it.

▪ If you can only do this, you can still do a lot.

# Package/File/Service SSH Example

package { 'openssh-server':

ensure => installed,

}

file { '/etc/ssh/sshd_config':

source => 'puppet:///modules/sshd/sshd_config',

owner => 'root',

group => 'root',

mode => '640',

# will restart whenever modified.

notify => Service['sshd'],

require => Package['openssh-server'],

}

service { 'sshd':

ensure => running,

enable => true,

hasstatus => true,

hasrestart => true,

}

MODULE CLASSES▪ Naming Convention

▪ The file must be named the same as the class or definition that is contained within (with the sole exception of the main class), and classes must be named after their function.

▪ Modules

▪ The main class of a module is its interface point, and ought to be the only parameterized class if possible. Limiting the parameterized classes to just the main class allows you to control usage of the entire module with the inclusion of a single class.

▪ The main class of any module must share the name of the module and be located in the `init.pp` file.

▪ The install class must be located in the `install.pp` file, and should contain all of the resources related to getting the software the module manages onto the node.

▪ The resources related to configuring the installed software should be placed in a config class and must be located in the `config.pp` file.

▪ The remaining service resources, and anything else related to the running state of the software, should be contained in the serviceclass. The service class must located in the `service.pp` file.

▪ Once a class is stored in a module, there are actually several ways to declare or assign it

▪ You can declare classes by putting `include ntp` in your main manifest. The include function declares a class, if it hasn’t already been declared somewhere else. If a class HAS already been declared, include will notice that and do nothing.

CLASS STRUCTURE

MODULE PARAMETERS

▪ Parameters form the public API of your module. They are the most important interface you expose, and care should be taken to give a good balance to the number and variety of parameters so users can customize their interactions with the module.

▪ Naming consistency is imperative for community comprehension and assists in troubleshooting and collaborating on module development.

▪ Best practices recommend the pattern of `thing_property` for naming parameters.

▪ Puppet Standard Module Initiative

▪ A community driven effort to standardize on parameters.

▪ If you want to maximize the usability of your modulewithout requiring users to modify it, you have to make it more flexible through the addition of parameters. Adding parameters enables more customized use of your module.

class ntp::params {

$config_template = 'ntp/ntp.conf.erb'

$package_ensure = 'present'

$service_enable = true

$service_ensure = 'running'

$service_manage = true

case $::osfamily {

'RedHat': {

$config = '/etc/ntp.conf'

$driftfile = '/var/lib/ntp/drift'

$keys_file = '/etc/ntp/keys'

$package_name = [ 'ntp' ]

$service_name = 'ntpd'

$servers = [

'0.centos.pool.ntp.org',

'1.centos.pool.ntp.org',

'2.centos.pool.ntp.org',

]

}

default: {

fail("The ${module_name} is not supported on an

${::osfamily} based system.")

}

}

}

PUPPET DYNAMIC DATAParams.pp

Hiera (./data/operatingsystem/RedHat.yaml)

ntp::config_template: 'ntp/ntp.conf.erb'ntp::package_ensure: 'present'ntp::service_enable: = truentp::service_ensure: = 'running'ntp::service_manage: = truentp::config: '/etc/ntp.conf'ntp::driftfile: '/var/lib/ntp/drift'ntp::keys_file: '/etc/ntp/keys'ntp::package_name: [ 'ntp' ]ntp::service_name: 'ntpd'ntp::servers: [

'0.centos.pool.ntp.org','1.centos.pool.ntp.org','2.centos.pool.ntp.org',

]

class ntp::params {$config_template = 'ntp/ntp.conf.erb'$package_ensure = 'present'$service_enable = true$service_ensure = 'running'$service_manage = true

case $::osfamily {'RedHat': {

$config = '/etc/ntp.conf'$driftfile = '/var/lib/ntp/drift'$keys_file = '/etc/ntp/keys'$package_name = [ 'ntp' ]$service_name = 'ntpd'$servers = [

'0.centos.pool.ntp.org','1.centos.pool.ntp.org','2.centos.pool.ntp.org',

]}default: {

fail("The ${module_name} module is not supported on an ${::osfamily} based system.")

}}

}

MODULE INHERITANCE

▪ Modules can inherit from other modules using the `inherits` in the class construct.

class ntp::params { … }

class ntp (

$config_template = $ntp::params::config_template,

$package_ensure = $ntp::params::package_ensure,

$service_enable = $ntp::params::service_enable,

$service_ensure = $ntp::params::service_ensure,

$service_manage = $ntp::params::service_manage

) inherits ntp::params { … }

class ntp::verne inherits ntp { … }

MODULE DEPENDENCIES

▪ If your module's functionality depends on another module, then you must list these dependenciesand include them directly.

▪ This means you must `include x` in your `init.pp` to ensure the dependency is included in the catalog.

▪ You must also add the dependencyto the Modulefile and `.fixtures.yml`.

class openssh::install {…package { 'openssh-server':ensure => installed,

}…

}

class openssh::config {…file { '/etc/ssh/sshd_config':source => 'puppet:///modules/sshd/sshd_config',owner => 'root',group => 'root',mode => '640',# will restart whenever modified.notify => Service['sshd'], require => Package['openssh-server'],

}…

}

class openssh::service {…service { 'sshd':ensure => running,enable => true,hasstatus => true,hasrestart => true,

}…

}

MODULE ORDERING

▪ Class-based ordering allows you to shield the implementation details of each class from the other classes.

▪ Due to a long-standing Puppetbug (#8040), there is a need to anchor classes within a moduleto stop them floating around in the ordering graph and potentially doing things in the wrong order.

▪ To this end we recommend you anchor classes in the`init.pp` file for your module.

# modules/module/manifests/init.pp

Anchor['begin'] ->

Class['module::install'] ->

Class['module::config'] ~>

Class['module::service']->

Anchor['end']

(Note: anchoring requires puppetlabs-stdlib.)

TESTING PUPPET MODULES

▪ Rspec-Puppet

▪ A unit-testing framework for Puppet. It extends RSpec to allow the testing framework to understand Puppetcatalogs, the artifact it specializes in testing.

▪ RSpec lets you provide facts, like `osfamily`, in order to test the module in various scenarios.

▪ Rspec-system

▪ An acceptance/integration testing framework that provisions, configures, and uses various Vagrantvirtual machines to apply your puppet module to a real machine and then test things from the command line within the VM.

▪ serverspec

▪ Serverspec provides additional testing constructs (such as `be_running` and `be_installed`) for rspec-system, and allows you to abstract away details of the underlying distribution when testing.

USING THE MODULEFILE▪ Modulefiles support the following pieces of metadata:

▪ name — REQUIRED. The full name of the module, including the username

▪ version — REQUIRED. The current version of the module. This should be a semantic version.

▪ summary — REQUIRED. A one-line description of the module.

▪ description — REQUIRED. A more complete description of the module.

▪ dependency — A module that this module depends on. Unlike the other fields, the dependency method accepts up to three comma-separated arguments:

▪ a module name (with a slash between the user and name, not a hyphen), a version requirement, and a repository.

▪ project_page — The module’s website.

▪ license — The license under which the module is made available.

▪ author — The module’s author. If not provided, this field will default to the username portion of the module’s name field.

▪ source — The module’s source. This field’s purpose is not specified.

# Modulefile

name ‘examplecorp-mymodule’

version ‘0.0.1’

summary ‘My module is an example of a

summary’

description “This is a full description of

the module, and is being written as a

multi-line string.“

dependency ‘puppetlabs/mysql’, ‘1.2.3’

In your module’s main directory, create a text file named Modulefile.

The Modulefile resembles a configuration or data file, but is actually a simple Ruby domain-specific language (DSL), which is executed when you build a tarball of the module.

DOCUMENTING PUPPET MODULES

▪ You are encouraged to document any module you write, whether you intend the module solely for internal use or for publication on the Puppet Forge.

▪ Documenting your module will help future-you remember what your module was built to do and can help explain why you chose to do things one way versus another. And anyone else using your module will definitely appreciate it.

▪ If you use the `puppet module generate`command it will create a default `README.md` template.

▪ Refer to the Puppet module style guide for more on documentation.

#modulename

####Table of Contents

1. [Overview](#overview)

2. [Module Description - What the module does and why it is

useful](#module-description)

3. [Setup - The basics of getting started with

[Modulename]](#setup)

* [What [Modulename] affects](#what-[modulename]-affects)

* [Setup requirements](#setup-requirements)

* [Beginning with [Modulename]](#beginning-with-

[Modulename])

4. [Usage - Configuration options and additional

functionality](#usage)

5. [Reference - An under-the-hood peek at what the module is

doing and how](#reference)

5. [Limitations - OS compatibility, etc.](#limitations)

6. [Development - Guide for contributing to the

module](#development)

7. [Release Notes/Contributors/Etc] - **Optional**

##Overview

##Module Description

##Setup

###What [Modulename] affects

VERSIONING PUPPET MODULES

▪ Once you've decided on the new versionnumber, you must increase the versionnumber metadata in the Modulefile.

▪ Versioning within the Modulefile allows you to build Puppet environments by picking and choosing specific versions of a module or set of modules.

▪ It also allows you to create a list of dependencies in the Modulefile of your modules with specific versions of dependent modules, which ensures your module isn't used with an ancient dependency that won't work.

▪ Versioning also enables workflow management by allowing you to easily use different versions of modules in different environments.

A Note on Semantic Versioning

▪ When writing your Modulefile, you’re setting a version for your own module and optionally expressing dependencies on others’ moduleversions.

▪ We strongly recommend following the Semantic Versioning specification. Doing so allows others to rely on your modules without unexpected change.

▪ Many other users already use semantic versioning, and you can take advantage of this in your modules’ dependencies.

▪ For example, if you depend on puppetlabs/stdlib and want to allow updates while avoiding breaking changes, you could write the following line in your Modulefile(assuming a current stdlib version of 2.2.1):

▪ dependency 'puppetlabs/stdlib', '2.x'

THE FORGE

▪ REUSE

▪ Puppet Forge, a repository of moduleswritten by the community for Puppet Open Source and Puppet Enterprise IT automation software.

▪ These modules solve a wide variety of problems so using them can save you time and effort.

▪ SHARE

▪ Register an account, create a module, upload a release of it and your automation code is now shared with the Puppet community.

▪ Learn how to create and share modules using the puppet module tool.

MANAGING PUPPET MODULES

▪ The `puppet module` subcommand, which ships with Puppet, is a tool for finding and managing new modules from the Forge. Its interface is similar to several common package managers, and makes it easy to search for and install new modules from the command line.

▪ If you have used a command line package manager tool (like gem, apt-get, or yum) before, these actions will generally do what you expect.

/* Use the --tree option with module list to view the modules arranged by dependency instead of by location on disk*/# puppet module list

# puppet module search apache

# puppet module install puppetlabs-apache --version 0.0.2

# puppet module upgrade puppetlabs-apache --version 0.0.3

# puppet module uninstall puppetlabs-apache

GENERATE & BUILD PUPPET MODULES

▪ Use the `puppet module generate` action to generate a template layout.

▪ An example Modulefile and README, and will also include a copy of the spec_helper tool for writing rspec-puppet tests.

▪ Once your content and Modulefile are ready, you can build a package of your module.

▪ This will generate a .tar.gz package, which will be saved in the module’spkg/ subdirectory.

# puppet module generate <USERNAME>-<MODULE NAME>

# puppet module build <MODULE DIRECTORY>

PUBLISHING PUPPET MODULES

▪ Puppet Forge requires module names to have a username prefix.

▪ Because many users have published their own versions of modules with common names (“mysql,” “bacula,” etc.)

▪ Be sure to use this long name in your module’s Modulefile.

▪ Now that you have a compiled tar.gz package, you can upload it to the Puppet Forge. There is currently no command line tool for publishing; you must use the Puppet Forge’s web interface.

▪ Your module has now been published to the Puppet Forge. The Forge will pull your README, Changelog, and License files from your tarball to display on your module’s page. To confirm that it was published correctly, you can install it on a new system using the puppetmodule install action.

To publish your module, you will need to:

1. Create a Puppet Forge account (if you don’t already have one)

2. Prepare your module

3. Write a Modulefile with the required metadata

4. Build an uploadable tarball of your module

5. Upload your module using the Puppet Forge’s web interface.

▪ If you have never published this module before, you must create a new page for it.

SHOWCASE

▪ Roles/Profiles Repository.

▪ Present shared repository for reusable roles/profiles.

▪ User Profile Module.

▪ Present a way to manage configurations on many different systems using Puppet.

▪ Using Vagrant and Puppet

▪ Demonstration of a complete build out of a VM

▪ Current efforts in configuring and managing our systems using Puppet.

CONTACT INFORMATIONProvide feel free to contact me at the following addresses.

m: abernstein@godaddy.com

gd_github: abernstein

twitter: @bernstein_aaron

m: seichner@godaddy.com

gd_github: seichner

Thank you! Questions?

Need additional guidance or reference material?

Download or print a copy of our Core Types Cheat

Sheet and Modules Cheat Sheet for fast reference

when writing your first manifests.

Experienced Puppet users spend most of their time

in the type reference and the language guide.

Get acquainted with them early!

RESOURCES (DOCUMENTATION)

▪ Official Site

▪ Download Learning Puppet VM

▪ Learning Guide

▪ Style Guide

▪ Glossary

▪ Puppet Patterns

▪ Modules Documentation

▪ Beginners Guide to Modules

▪ Puppet 2.7.x Reference Manual

▪ Puppet Standard Module Initiative

RESOURCES (DOCUMENTATION)

▪ Why Puppet Should Manage Your Infrastructure

▪ Puppet Forge

▪ PuppetLabs Github

▪ Writing Custom Types

▪ Git Workflow and Puppet Environments

▪ Rspec-Puppet

▪ Rspec-system

▪ serverspec

▪ Vagrant

▪ Mcollective

▪ Scaling Recommendations

RESOURCES (PRESENTATIONS)

▪ Slideshare - Puppetlabs

▪ Slideshare - PuppetConf 2013

▪ PuppetLabs - Roles/Profiles

▪ Puppet and Windows

▪ Getting Started with the Learning Puppet VM

▪ What’s New and Awesome in Puppet Enterprise 3

▪ Introduction to Mcollective

▪ Troubleshooting the Puppet Enterprise Stack