puppet getting started by dirk götz

70
www.netways.de // blog.netways.de // @netways Make IT do more with less 27 NOVEMBER 2013 | PUPPET CAMP PUPPET GETTING STARTED DIRK GÖTZ | NETWAYS GMBH

Upload: netways

Post on 08-May-2015

1.134 views

Category:

Technology


4 download

DESCRIPTION

Puppet getting started will show the different components used in puppet environments, starting with facter and puppet to different webinterfaces like puppet enterprise console and foreman. It will also cover an exemplary design for scaling the puppet master and for development livecycle of modules. Furthermore an example for design of modules will be given.

TRANSCRIPT

Page 1: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

27 NOVEMBER 2013 | PUPPET CAMP

PUPPET GETTING STARTED

DIRK GÖTZ | NETWAYS GMBH

Page 2: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

AGENDA

■ Brief introduction

■ Configuration management

■ Components

■ Design your environment

■ Design your workflow

■ Design your module

Page 3: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

BRIEF INTRODUCTION

Page 4: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

BRIEF INTRODUCTION TO NETWAYS

• Founded in 1995

• Open source since 1997

• 40 employees

• Specialised in open source systems

management and open source data

center infrastructure

Page 5: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

Puppet Camp 2014

• 11 April 2014

• 110 attendees (April 2013)

• 170 attendees (November 2014)

Open Source Data Center Conference

• 8 – 10 April 2014

• 120 attendees (2013)

• 2 tracks of presentations & workshops

NETWAYS CONFERENCES

Page 6: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

CONFIGURATION MANAGEMENT

Page 7: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

Golden images

Software solutions

CONFIGURATION MANAGEMENT

Manual configuration

Self-made scripts

for i in $(cat host.cfg)dossh user@$i uname -a

done

Types of „Automation“

Page 8: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

THE PUPPET WAY

1. Define

2. Simulate

3. Enforce

4. Report

Desired StateCurrent State

Page 9: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

THE PUPPET WAY - DEFINE

■ Using Puppet Domain Specific Language (DSL)

■ Describe relationship between resources

■ Create modular reuseable code

Page 10: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

THE PUPPET WAY - SIMULATE

■ Simulate deployment

■ Without changes to your system

■ Test and fix it before enforcing

$ sudo puppet apply ntp.pp --verbose --noopInfo: Loading facts in /var/lib/puppet/lib/facter/root_home.rbInfo: Loading facts in /var/lib/puppet/lib/facter/pe_version.rbInfo: Loading facts in /var/lib/puppet/lib/facter/vmware.rbInfo: Loading facts in /var/lib/puppet/lib/facter/last_run.rbInfo: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rbInfo: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rbInfo: Loading facts in /var/lib/puppet/lib/facter/puppi_projects.rbInfo: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rbError: Could not find template 'ntp/ntp.conf.erb' at /tmp/ntp.pp:9 on node puppet.localdomainError: Could not find template 'ntp/ntp.conf.erb' at /tmp/ntp.pp:9 on node puppet.localdomain

Page 11: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

THE PUPPET WAY - ENFORCE

■ Compares current state and desired state

■ Changes to desired state

■ Idempotency

$ sudo puppet agent -tInfo: Retrieving pluginInfo: Loading facts in /var/lib/puppet/lib/facter/root_home.rbInfo: Loading facts in /var/lib/puppet/lib/facter/pe_version.rbInfo: Loading facts in /var/lib/puppet/lib/facter/vmware.rbInfo: Loading facts in /var/lib/puppet/lib/facter/last_run.rbInfo: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rbInfo: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rbInfo: Loading facts in /var/lib/puppet/lib/facter/puppi_projects.rbInfo: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rbInfo: Caching catalog for puppet.localdomainInfo: Applying configuration version '1384768191‚Notice: Finished catalog run in 0.54 seconds

Desired StateCurrent State

Page 12: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

THE PUPPET WAY - REPORT

■ Report changes and metrics

■ Many different presentation formats

Page 13: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

COMPONENTS – BASIC TOOLS

Page 14: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET WORKFLOW

SSL-Encryption

Page 15: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

FACTER

■ Open source Ruby library by Puppet Labs

■ Provides information on your host

■ CLI returns key-value pairs

$ facterarchitecture => x86_64augeasversion => 0.9.0blockdevice_sr0_model => QEMU DVD-ROMblockdevice_sr0_size => 1073741312blockdevice_sr0_vendor => QEMUblockdevices => sr0,vdadomain => localdomainfacterversion => 1.7.3filesystems => ext4,iso9660fqdn => puppet.localdomain

$ facter osfamilyRedHat

Page 16: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET

■ Open source Ruby library by Puppet Labs

■ Client server based

• REST-API• X509 certificates

■ Platform independent

• Puppet Domain Specific Language• Providers for different platforms

Page 17: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET DOMAIN SPECIFIC LANGUAGE

■ Abstracts resources

■ Resource types

• user, group, • file, package, service, • exec, • …and many more• possible to create your own

■ Type and title/name pairs must be unique

Page 18: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET PROVIDERS

■ Resource Abstraction Layer

• Resource types• Providers

Page 19: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET – MORE ABSTRACTION

■ Class groups resources in one manifest

■ Syntax constructs possible

Page 20: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET – SYNTAX CONSTRUCTS

■ Variables (but are more like constants)

■ Logical structures

• selectors• case statements• if/elseif/else statements

■ Dependencies

• require/before• subscribe/notify

■ Inheritance

Page 21: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET – MORE ABSTRACTION

■ Module groups classes and corresponding files

modulename|-- files – static files|-- lib – custom facts, functions, resources|-- manifests – manifests containing classes|-- spec – tests for rspec-puppet|-- templates – dynamic files|-- tests – examples declaring classes

Page 22: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET FILESERVER

■ File transfer from master

• puppet:///modules/modulename/filename

■ “Magic“ mount point for modules

■ Additional mount points possible

■ Recursion for directories possible

Page 23: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET PLUGIN SYNC

■ Option on the agent

■ Syncs from all modules

• custom facts• custom resources (types and providers)

■ Before the actual Puppet run occurs

Page 24: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET TEMPLATES

■ Function executed on server

■ ERB (Plain text with embedded Ruby)

• variables (including facts)• conditionals• iterations• access to tags and classes• usage of Puppet functions

■ Combination of multiple templates possible

Page 25: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET NODE DECLARATION

■ Connects system (node) with functionality (classes)

• Single hosts

• Mulitiple hosts (by regular expression)

• Default

• Inheritance possible

Page 26: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

COMPONENTS – BASIC GUI

Page 27: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

BASIC GUI – ADDED FUNCTIONALITY

■ Reporting target

• Collects reports• Graphical presentation

■ External Node Classifier (ENC)

• Graphical node declaration• Adds groups

Page 28: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET - REPORTING

■ Enable on agent

■ Choose target on server

• http/https• log• tagmail• store• rrdgraph• puppetdb

■ Multiple targets possible

■ Create your own

Page 29: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET - ENC

■ External source for node declaration

• Script returning yaml• LDAP

■ Merged with internal node declaration

$/etc/puppet/node.rb puppet.localdomain---environment: productionparameters: foreman_env: productionowner_name: Admin Userdomainname: ""owner_email: root@localdomainroot_pw: $1$default$hCkak1kaJPQILNmYbUXhD0puppetmaster: ""

classes: profiles::default:

dn: cn=testserver,ou=Hosts,dc=madstop,dc=comobjectClass: deviceobjectClass: ipHostobjectClass: puppetClientobjectClass: topcn: testserverenvironment: testingipHostNumber: 192.168.0.50description: My test serverl: dc1puppetClass: testingpuppetVar: owner_name=„Admin user“

Page 30: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET DASHBOARD

■ Open source ruby web interface

• Version <= 1.2 by Puppet Labs• Version >= 2 by community

http://puppetlabs.com/presentations/story-dashboard-20

Page 31: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

COMPONENTS – ORCHESTRATION

Page 32: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

ORCHESTRATION

■ Centralized execution on multiple systems

■ Different solutions

• ssh loops• func• fabric• capistrano• mcollective• … and many more

Page 33: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

MCOLLECTIVE

■ Open source Ruby framework by Puppet Labs

■ Facter for grouping

■ Middleware for communication

$ mco package status httpd -F kernel=Linux

* [ ============================================================> ] 3 / 3

centos63a version = httpd-2.2.15-15.el6.centos.1ubuntu1204a version = httpd-purgedmaster version = httpd-purged

---- package agent summary ----Nodes: 3 / 3

Versions: 1 * 2.2.15-15.el6.centos.1, 2 * purgedElapsed Time: 0.12 s

Page 34: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET ENTERPRISE CONSOLE

■ Ruby web interface based on dashboard

• Auditing• Live management

Page 35: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

COMPONENTS – DEPLOYMENT

Page 36: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

DEPLOYMENT

■ Config management needs running agent

■ (Agent needs running operating system)

■ Different solutions:

• Kickstart / Autoyast / Preseed / Jumpstart• Cobbler / kickstand• Razor• Puppet Cloud Provisioner• Foreman• … and many more

Page 37: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

RAZOR

■ Open source Ruby library by Puppet Labs/EMC

■ Client

• micro-kernel for PXE boot

■ Server

• REST-API• CLI

■ Policy connects

• host profiles• operating system• config management

{"name": "centos-for-small","repo": { "name": "centos-6.4" },"installer": { "name": "centos" },"broker": { "name": "noop" },"enabled": true,"hostname": "host${id}.example.com","root_password": "secret","max_count": "20","rule_number": "100","tags": [{ "name": "small", "rule": ["<=", ["num", ["fact",

"processorcount"]], 2]}]}

Page 38: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

CLOUD PROVISIONER

■ Open source Ruby library by Puppet Labs

■ Instantiates cloud instances

• EC2• VMware• OpenStack• Google Compute Engine

■ Installs Puppet using SSH

Page 39: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

FOREMAN

■ Open source Ruby web interface by Ohad Levy

(Red Hat)

• Reporting target• ENC• Provisioning

Page 40: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

FOREMAN – SMART PROXIES

■ Connects GUI and Backends

• puppet• puppetca• tftp• dhcp• dns

Page 41: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

FOREMAN – COMPUTE RESOURCES

■ Integrates virtualization and cloud platforms

• Libvirt• oVirt / RHEV• VMware• EC2• Google Compute Engine• OpenStack• Rackspace

Page 42: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

COMPONENTS – MORE FEATURES

Page 43: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

EXPORTED RESOURCES

■ One node creates resource

■ Another node realizes that resource

■ Resource needs to be stored:

• Stored configs (deprecated)• PuppetDB

■ Use cases:

• host entries• sshkey management• monitoring / backup• other centralized services

Page 44: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPETDB

■ PostgreSQL and Java based data warehouse solution

by Puppet Labs

■ Collects

• Facts• Catalogs• Reports (optional)

■ Used for

• Inventory service• Exported resources

Page 45: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

HIERA

■ Open source Ruby library by Puppet Labs

■ Hierarchical data look-up separates code and data

■ Integrated in Puppet >3 / addon for Puppet <3

$cat hiera.yaml---:backends:- yaml

:yaml::datadir: /etc/puppet/hieradata

:hierarchy:- `hosts/${::fqdn}´- `location/${::location}´- common

$cat hosts/specialhost.localdomainntp::server = 192.168.23.23

$ cat location/rz2ntp::server = 192.168.2.23yum:mirror = 192.168.2.42

$ cat commonntp::server = 192.168.0.23yum::mirror = 192.168.0.42proxy::server = [`192.168.0.237´,`192.168.0.238´]

Page 46: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

COMPONENTS – SOFTWARE MANAGEMENT

Page 47: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

SOFTWARE MANAGEMENT

■ Provider needs defined sources for software

■ Local mirror reduces traffic

■ Many different tools

• rsync / createrepo• updian• Spacewalk / Red Hat Satellite / Suse Manager• pulp / katello• … and many more

Page 48: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PULP

■ Open source Python repository management

by Red Hat

■ Server:

• Import and upload of content• rpm and puppet modules• Publish web-based or ISO images

■ Client:

• Server-side management and reporting

Page 49: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

KATELLO

■ Java web interface by Red Hat

■ Combines:

• candlepin (subscription management)• pulp (software management)• Foreman + Puppet (configuration management)

Page 50: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

DESIGN YOUR ENVIRONMENT

Page 51: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

CHOICES

■ Support / packages needed?

■ Best method to report?

■ Resources to export?

■ Deployment needed?

■ Software management needed?

■ Node declaration or ENC?

Page 52: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

SCALE UP

■ Puppet easily scales up

■ Start simple, grow with your environment

■ Rule of thumb:

number of nodes * catalog compile time (in seconds)number of masters = ------------------------------------------------------------------------------

cores per master * run interval (in seconds)

Page 53: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

SCALE UP

■ Only one certficate authority

• Only needed for registration

■ Load balance Puppet traffic

■ GUI only required for users

• But ENC / Reporting always required

■ Orchestration

Page 54: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

SCALE UP – EXAMPLE: PUPPET ENTERPRISE

Page 55: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

DESIGN YOUR WORKFLOW

Page 56: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

EDITOR

■ vim – text editor

• vim-puppet – synthax highlight• tabular – style guide conformity• puppet-lint – style guide conformity• syntastic – validation

■ Geppetto – eclipse based IDE

synthax hightlight, style guide conformity, module creation

Page 57: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

STYLEGUIDE

■ Official guidehttp://docs.puppetlabs.com/guides/style_guide.html

• readability• work with and without features• simple and robust• shareable / useable by others

■ Create your own

• based on official guide• make your own rules

Page 58: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

TESTING

■ puppet parser validate – syntax

■ puppet-lint – style guide conformity

■ puppet apply --noop – simulate

■ vagrant – simulate

■ rspec-puppet – expected results

Page 59: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

VERSION CONTROL SYSTEM

■ Pre-Commit / Pre-Receive

• use for validation and review

■ Post-Commit

• checkout in environment

■ Versioning for change management

■ Solutions:

• Version control: svn, git, bazaar• Validation & Review: gerrit, jenkins/hudson

Page 60: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

STAGING

■ Puppet environments

• Different versions of modules

■ Use your own versioning

• config_version: script returns version string

Page 61: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

DOCUMENTATION

■ Inline

• RDoc markup• Console output• HTML generation

■ README

• Markdown

■ Modulefile

• used by Puppet module tool

Page 62: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

DESIGN YOUR MODULE

Page 63: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

CONSIDERATIONS

■ Where to start?

■ Use cases?

• different platforms• different roles• small adjustments• one-time or multiple objects

■ ‚Part of it‘ or separate?

• Dependencies

■ Templates or files?

■ Write your own?

Page 64: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET FORGE

■ Community platform for modules

• Web platform• Command line tool

■ Module information:

• Author• Project homepage and issue tracker• Tags• Releases and download count• Test results

Page 65: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

PUPPET FORGE – EXAMPLE

example42

■ 96 modules on forge / more on github

■ always same layout

■ supports: Red Hat, CentOS, Fedora, Ubuntu, Debian,

Mint, SLES, OpenSuSE, FreeBSD

■ adds monitoring, firewall and puppi (own script library)

Page 66: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

MODULE DESIGN – MORE ABSTRACTION

■ Classes abstract resources

■ Modules abstract classes

■ Nodes contain the logic

Not maintainable!

node 'basil.puppetlabs.vm' {class { 'apache’:version => 'latest’,

}class { 'motd': }class { 'ssh’: }if $::operatingsystem == 'solaris' {class { 'users’:default_shell => '/bin/false’,

}}else {class { 'users': }

}Class['ssh’] -> Class['users’]

}

Page 67: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

MODULE DESIGN – MORE ABSTRACTION

■ Classes abstract resources

■ Modules abstract classes

■ Profiles contain the logic

• abstract the modules• separate implementation

from technology

class profiles::application {include tomcatinclude mysqlinclude componenta

}class profiles::application::x inheritsprofiles::application {include componentbcomponentb::resource { 'name':ensure => present,

}}class profiles::application::y inheritsprofiles::application {include componentcinclude componentd

}class profiles::application::z inheritsprofiles::application {include componentbinclude componentdinclude dependencyClass['dependency'] -> Class['componentd']

}

Page 68: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

MODULE DESIGN – MORE ABSTRACTION

■ Classes abstract resources

■ Modules abstract classes

■ Profiles contain the logic

■ Roles contain business logic

• no logic, just profiles• separate business role from implementation

■ Node has exactly one role

• No Puppet know-how neededfor node declaration

class role::webapp {include profiles::baseinclude profiles::customappinclude profiles::test_tools

}

node ‘web1.example.com’ {include role:webapp

}

Page 69: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

QUESTIONS & ANSWERS

Page 70: Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

NETWAYS GmbH

Deutschherrnstrasse 15-19

90429 Nürmberg

Phone: +49 911 92885-0

Fax: +49 911 92885-77

Email: [email protected]

Website: www.netways.de

Twitter: twitter.com/netways

Facebook: facebook.com/netways

Blog: blog.netways.de

QUESTIONS & ANSWERS