puppet getting started by dirk götz
DESCRIPTION
Puppet getting started will show the different components used in puppet environments, starting with facter and puppet to different webinterfaces like puppet enterprise console and foreman. It will also cover an exemplary design for scaling the puppet master and for development livecycle of modules. Furthermore an example for design of modules will be given.TRANSCRIPT
www.netways.de // blog.netways.de // @netways
Make IT do more with less
27 NOVEMBER 2013 | PUPPET CAMP
PUPPET GETTING STARTED
DIRK GÖTZ | NETWAYS GMBH
www.netways.de // blog.netways.de // @netways
Make IT do more with less
AGENDA
■ Brief introduction
■ Configuration management
■ Components
■ Design your environment
■ Design your workflow
■ Design your module
www.netways.de // blog.netways.de // @netways
Make IT do more with less
BRIEF INTRODUCTION
www.netways.de // blog.netways.de // @netways
Make IT do more with less
BRIEF INTRODUCTION TO NETWAYS
• Founded in 1995
• Open source since 1997
• 40 employees
• Specialised in open source systems
management and open source data
center infrastructure
www.netways.de // blog.netways.de // @netways
Make IT do more with less
Puppet Camp 2014
• 11 April 2014
• 110 attendees (April 2013)
• 170 attendees (November 2014)
Open Source Data Center Conference
• 8 – 10 April 2014
• 120 attendees (2013)
• 2 tracks of presentations & workshops
NETWAYS CONFERENCES
www.netways.de // blog.netways.de // @netways
Make IT do more with less
CONFIGURATION MANAGEMENT
www.netways.de // blog.netways.de // @netways
Make IT do more with less
Golden images
Software solutions
CONFIGURATION MANAGEMENT
Manual configuration
Self-made scripts
for i in $(cat host.cfg)dossh user@$i uname -a
done
Types of „Automation“
www.netways.de // blog.netways.de // @netways
Make IT do more with less
THE PUPPET WAY
1. Define
2. Simulate
3. Enforce
4. Report
Desired StateCurrent State
www.netways.de // blog.netways.de // @netways
Make IT do more with less
THE PUPPET WAY - DEFINE
■ Using Puppet Domain Specific Language (DSL)
■ Describe relationship between resources
■ Create modular reuseable code
www.netways.de // blog.netways.de // @netways
Make IT do more with less
THE PUPPET WAY - SIMULATE
■ Simulate deployment
■ Without changes to your system
■ Test and fix it before enforcing
$ sudo puppet apply ntp.pp --verbose --noopInfo: Loading facts in /var/lib/puppet/lib/facter/root_home.rbInfo: Loading facts in /var/lib/puppet/lib/facter/pe_version.rbInfo: Loading facts in /var/lib/puppet/lib/facter/vmware.rbInfo: Loading facts in /var/lib/puppet/lib/facter/last_run.rbInfo: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rbInfo: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rbInfo: Loading facts in /var/lib/puppet/lib/facter/puppi_projects.rbInfo: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rbError: Could not find template 'ntp/ntp.conf.erb' at /tmp/ntp.pp:9 on node puppet.localdomainError: Could not find template 'ntp/ntp.conf.erb' at /tmp/ntp.pp:9 on node puppet.localdomain
www.netways.de // blog.netways.de // @netways
Make IT do more with less
THE PUPPET WAY - ENFORCE
■ Compares current state and desired state
■ Changes to desired state
■ Idempotency
$ sudo puppet agent -tInfo: Retrieving pluginInfo: Loading facts in /var/lib/puppet/lib/facter/root_home.rbInfo: Loading facts in /var/lib/puppet/lib/facter/pe_version.rbInfo: Loading facts in /var/lib/puppet/lib/facter/vmware.rbInfo: Loading facts in /var/lib/puppet/lib/facter/last_run.rbInfo: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rbInfo: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rbInfo: Loading facts in /var/lib/puppet/lib/facter/puppi_projects.rbInfo: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rbInfo: Caching catalog for puppet.localdomainInfo: Applying configuration version '1384768191‚Notice: Finished catalog run in 0.54 seconds
Desired StateCurrent State
www.netways.de // blog.netways.de // @netways
Make IT do more with less
THE PUPPET WAY - REPORT
■ Report changes and metrics
■ Many different presentation formats
www.netways.de // blog.netways.de // @netways
Make IT do more with less
COMPONENTS – BASIC TOOLS
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET WORKFLOW
SSL-Encryption
www.netways.de // blog.netways.de // @netways
Make IT do more with less
FACTER
■ Open source Ruby library by Puppet Labs
■ Provides information on your host
■ CLI returns key-value pairs
$ facterarchitecture => x86_64augeasversion => 0.9.0blockdevice_sr0_model => QEMU DVD-ROMblockdevice_sr0_size => 1073741312blockdevice_sr0_vendor => QEMUblockdevices => sr0,vdadomain => localdomainfacterversion => 1.7.3filesystems => ext4,iso9660fqdn => puppet.localdomain
$ facter osfamilyRedHat
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET
■ Open source Ruby library by Puppet Labs
■ Client server based
• REST-API• X509 certificates
■ Platform independent
• Puppet Domain Specific Language• Providers for different platforms
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET DOMAIN SPECIFIC LANGUAGE
■ Abstracts resources
■ Resource types
• user, group, • file, package, service, • exec, • …and many more• possible to create your own
■ Type and title/name pairs must be unique
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET PROVIDERS
■ Resource Abstraction Layer
• Resource types• Providers
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET – MORE ABSTRACTION
■ Class groups resources in one manifest
■ Syntax constructs possible
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET – SYNTAX CONSTRUCTS
■ Variables (but are more like constants)
■ Logical structures
• selectors• case statements• if/elseif/else statements
■ Dependencies
• require/before• subscribe/notify
■ Inheritance
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET – MORE ABSTRACTION
■ Module groups classes and corresponding files
modulename|-- files – static files|-- lib – custom facts, functions, resources|-- manifests – manifests containing classes|-- spec – tests for rspec-puppet|-- templates – dynamic files|-- tests – examples declaring classes
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET FILESERVER
■ File transfer from master
• puppet:///modules/modulename/filename
■ “Magic“ mount point for modules
■ Additional mount points possible
■ Recursion for directories possible
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET PLUGIN SYNC
■ Option on the agent
■ Syncs from all modules
• custom facts• custom resources (types and providers)
■ Before the actual Puppet run occurs
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET TEMPLATES
■ Function executed on server
■ ERB (Plain text with embedded Ruby)
• variables (including facts)• conditionals• iterations• access to tags and classes• usage of Puppet functions
■ Combination of multiple templates possible
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET NODE DECLARATION
■ Connects system (node) with functionality (classes)
• Single hosts
• Mulitiple hosts (by regular expression)
• Default
• Inheritance possible
www.netways.de // blog.netways.de // @netways
Make IT do more with less
COMPONENTS – BASIC GUI
www.netways.de // blog.netways.de // @netways
Make IT do more with less
BASIC GUI – ADDED FUNCTIONALITY
■ Reporting target
• Collects reports• Graphical presentation
■ External Node Classifier (ENC)
• Graphical node declaration• Adds groups
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET - REPORTING
■ Enable on agent
■ Choose target on server
• http/https• log• tagmail• store• rrdgraph• puppetdb
■ Multiple targets possible
■ Create your own
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET - ENC
■ External source for node declaration
• Script returning yaml• LDAP
■ Merged with internal node declaration
$/etc/puppet/node.rb puppet.localdomain---environment: productionparameters: foreman_env: productionowner_name: Admin Userdomainname: ""owner_email: root@localdomainroot_pw: $1$default$hCkak1kaJPQILNmYbUXhD0puppetmaster: ""
classes: profiles::default:
dn: cn=testserver,ou=Hosts,dc=madstop,dc=comobjectClass: deviceobjectClass: ipHostobjectClass: puppetClientobjectClass: topcn: testserverenvironment: testingipHostNumber: 192.168.0.50description: My test serverl: dc1puppetClass: testingpuppetVar: owner_name=„Admin user“
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET DASHBOARD
■ Open source ruby web interface
• Version <= 1.2 by Puppet Labs• Version >= 2 by community
http://puppetlabs.com/presentations/story-dashboard-20
www.netways.de // blog.netways.de // @netways
Make IT do more with less
COMPONENTS – ORCHESTRATION
www.netways.de // blog.netways.de // @netways
Make IT do more with less
ORCHESTRATION
■ Centralized execution on multiple systems
■ Different solutions
• ssh loops• func• fabric• capistrano• mcollective• … and many more
www.netways.de // blog.netways.de // @netways
Make IT do more with less
MCOLLECTIVE
■ Open source Ruby framework by Puppet Labs
■ Facter for grouping
■ Middleware for communication
$ mco package status httpd -F kernel=Linux
* [ ============================================================> ] 3 / 3
centos63a version = httpd-2.2.15-15.el6.centos.1ubuntu1204a version = httpd-purgedmaster version = httpd-purged
---- package agent summary ----Nodes: 3 / 3
Versions: 1 * 2.2.15-15.el6.centos.1, 2 * purgedElapsed Time: 0.12 s
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET ENTERPRISE CONSOLE
■ Ruby web interface based on dashboard
• Auditing• Live management
www.netways.de // blog.netways.de // @netways
Make IT do more with less
COMPONENTS – DEPLOYMENT
www.netways.de // blog.netways.de // @netways
Make IT do more with less
DEPLOYMENT
■ Config management needs running agent
■ (Agent needs running operating system)
■ Different solutions:
• Kickstart / Autoyast / Preseed / Jumpstart• Cobbler / kickstand• Razor• Puppet Cloud Provisioner• Foreman• … and many more
www.netways.de // blog.netways.de // @netways
Make IT do more with less
RAZOR
■ Open source Ruby library by Puppet Labs/EMC
■ Client
• micro-kernel for PXE boot
■ Server
• REST-API• CLI
■ Policy connects
• host profiles• operating system• config management
{"name": "centos-for-small","repo": { "name": "centos-6.4" },"installer": { "name": "centos" },"broker": { "name": "noop" },"enabled": true,"hostname": "host${id}.example.com","root_password": "secret","max_count": "20","rule_number": "100","tags": [{ "name": "small", "rule": ["<=", ["num", ["fact",
"processorcount"]], 2]}]}
www.netways.de // blog.netways.de // @netways
Make IT do more with less
CLOUD PROVISIONER
■ Open source Ruby library by Puppet Labs
■ Instantiates cloud instances
• EC2• VMware• OpenStack• Google Compute Engine
■ Installs Puppet using SSH
www.netways.de // blog.netways.de // @netways
Make IT do more with less
FOREMAN
■ Open source Ruby web interface by Ohad Levy
(Red Hat)
• Reporting target• ENC• Provisioning
www.netways.de // blog.netways.de // @netways
Make IT do more with less
FOREMAN – SMART PROXIES
■ Connects GUI and Backends
• puppet• puppetca• tftp• dhcp• dns
www.netways.de // blog.netways.de // @netways
Make IT do more with less
FOREMAN – COMPUTE RESOURCES
■ Integrates virtualization and cloud platforms
• Libvirt• oVirt / RHEV• VMware• EC2• Google Compute Engine• OpenStack• Rackspace
www.netways.de // blog.netways.de // @netways
Make IT do more with less
COMPONENTS – MORE FEATURES
www.netways.de // blog.netways.de // @netways
Make IT do more with less
EXPORTED RESOURCES
■ One node creates resource
■ Another node realizes that resource
■ Resource needs to be stored:
• Stored configs (deprecated)• PuppetDB
■ Use cases:
• host entries• sshkey management• monitoring / backup• other centralized services
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPETDB
■ PostgreSQL and Java based data warehouse solution
by Puppet Labs
■ Collects
• Facts• Catalogs• Reports (optional)
■ Used for
• Inventory service• Exported resources
www.netways.de // blog.netways.de // @netways
Make IT do more with less
HIERA
■ Open source Ruby library by Puppet Labs
■ Hierarchical data look-up separates code and data
■ Integrated in Puppet >3 / addon for Puppet <3
$cat hiera.yaml---:backends:- yaml
:yaml::datadir: /etc/puppet/hieradata
:hierarchy:- `hosts/${::fqdn}´- `location/${::location}´- common
$cat hosts/specialhost.localdomainntp::server = 192.168.23.23
$ cat location/rz2ntp::server = 192.168.2.23yum:mirror = 192.168.2.42
$ cat commonntp::server = 192.168.0.23yum::mirror = 192.168.0.42proxy::server = [`192.168.0.237´,`192.168.0.238´]
www.netways.de // blog.netways.de // @netways
Make IT do more with less
COMPONENTS – SOFTWARE MANAGEMENT
www.netways.de // blog.netways.de // @netways
Make IT do more with less
SOFTWARE MANAGEMENT
■ Provider needs defined sources for software
■ Local mirror reduces traffic
■ Many different tools
• rsync / createrepo• updian• Spacewalk / Red Hat Satellite / Suse Manager• pulp / katello• … and many more
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PULP
■ Open source Python repository management
by Red Hat
■ Server:
• Import and upload of content• rpm and puppet modules• Publish web-based or ISO images
■ Client:
• Server-side management and reporting
www.netways.de // blog.netways.de // @netways
Make IT do more with less
KATELLO
■ Java web interface by Red Hat
■ Combines:
• candlepin (subscription management)• pulp (software management)• Foreman + Puppet (configuration management)
www.netways.de // blog.netways.de // @netways
Make IT do more with less
DESIGN YOUR ENVIRONMENT
www.netways.de // blog.netways.de // @netways
Make IT do more with less
CHOICES
■ Support / packages needed?
■ Best method to report?
■ Resources to export?
■ Deployment needed?
■ Software management needed?
■ Node declaration or ENC?
www.netways.de // blog.netways.de // @netways
Make IT do more with less
SCALE UP
■ Puppet easily scales up
■ Start simple, grow with your environment
■ Rule of thumb:
number of nodes * catalog compile time (in seconds)number of masters = ------------------------------------------------------------------------------
cores per master * run interval (in seconds)
www.netways.de // blog.netways.de // @netways
Make IT do more with less
SCALE UP
■ Only one certficate authority
• Only needed for registration
■ Load balance Puppet traffic
■ GUI only required for users
• But ENC / Reporting always required
■ Orchestration
www.netways.de // blog.netways.de // @netways
Make IT do more with less
SCALE UP – EXAMPLE: PUPPET ENTERPRISE
www.netways.de // blog.netways.de // @netways
Make IT do more with less
DESIGN YOUR WORKFLOW
www.netways.de // blog.netways.de // @netways
Make IT do more with less
EDITOR
■ vim – text editor
• vim-puppet – synthax highlight• tabular – style guide conformity• puppet-lint – style guide conformity• syntastic – validation
■ Geppetto – eclipse based IDE
synthax hightlight, style guide conformity, module creation
www.netways.de // blog.netways.de // @netways
Make IT do more with less
STYLEGUIDE
■ Official guidehttp://docs.puppetlabs.com/guides/style_guide.html
• readability• work with and without features• simple and robust• shareable / useable by others
■ Create your own
• based on official guide• make your own rules
www.netways.de // blog.netways.de // @netways
Make IT do more with less
TESTING
■ puppet parser validate – syntax
■ puppet-lint – style guide conformity
■ puppet apply --noop – simulate
■ vagrant – simulate
■ rspec-puppet – expected results
www.netways.de // blog.netways.de // @netways
Make IT do more with less
VERSION CONTROL SYSTEM
■ Pre-Commit / Pre-Receive
• use for validation and review
■ Post-Commit
• checkout in environment
■ Versioning for change management
■ Solutions:
• Version control: svn, git, bazaar• Validation & Review: gerrit, jenkins/hudson
www.netways.de // blog.netways.de // @netways
Make IT do more with less
STAGING
■ Puppet environments
• Different versions of modules
■ Use your own versioning
• config_version: script returns version string
www.netways.de // blog.netways.de // @netways
Make IT do more with less
DOCUMENTATION
■ Inline
• RDoc markup• Console output• HTML generation
■ README
• Markdown
■ Modulefile
• used by Puppet module tool
www.netways.de // blog.netways.de // @netways
Make IT do more with less
DESIGN YOUR MODULE
www.netways.de // blog.netways.de // @netways
Make IT do more with less
CONSIDERATIONS
■ Where to start?
■ Use cases?
• different platforms• different roles• small adjustments• one-time or multiple objects
■ ‚Part of it‘ or separate?
• Dependencies
■ Templates or files?
■ Write your own?
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET FORGE
■ Community platform for modules
• Web platform• Command line tool
■ Module information:
• Author• Project homepage and issue tracker• Tags• Releases and download count• Test results
www.netways.de // blog.netways.de // @netways
Make IT do more with less
PUPPET FORGE – EXAMPLE
example42
■ 96 modules on forge / more on github
■ always same layout
■ supports: Red Hat, CentOS, Fedora, Ubuntu, Debian,
Mint, SLES, OpenSuSE, FreeBSD
■ adds monitoring, firewall and puppi (own script library)
www.netways.de // blog.netways.de // @netways
Make IT do more with less
MODULE DESIGN – MORE ABSTRACTION
■ Classes abstract resources
■ Modules abstract classes
■ Nodes contain the logic
Not maintainable!
node 'basil.puppetlabs.vm' {class { 'apache’:version => 'latest’,
}class { 'motd': }class { 'ssh’: }if $::operatingsystem == 'solaris' {class { 'users’:default_shell => '/bin/false’,
}}else {class { 'users': }
}Class['ssh’] -> Class['users’]
}
www.netways.de // blog.netways.de // @netways
Make IT do more with less
MODULE DESIGN – MORE ABSTRACTION
■ Classes abstract resources
■ Modules abstract classes
■ Profiles contain the logic
• abstract the modules• separate implementation
from technology
class profiles::application {include tomcatinclude mysqlinclude componenta
}class profiles::application::x inheritsprofiles::application {include componentbcomponentb::resource { 'name':ensure => present,
}}class profiles::application::y inheritsprofiles::application {include componentcinclude componentd
}class profiles::application::z inheritsprofiles::application {include componentbinclude componentdinclude dependencyClass['dependency'] -> Class['componentd']
}
www.netways.de // blog.netways.de // @netways
Make IT do more with less
MODULE DESIGN – MORE ABSTRACTION
■ Classes abstract resources
■ Modules abstract classes
■ Profiles contain the logic
■ Roles contain business logic
• no logic, just profiles• separate business role from implementation
■ Node has exactly one role
• No Puppet know-how neededfor node declaration
class role::webapp {include profiles::baseinclude profiles::customappinclude profiles::test_tools
}
node ‘web1.example.com’ {include role:webapp
}
www.netways.de // blog.netways.de // @netways
Make IT do more with less
QUESTIONS & ANSWERS
www.netways.de // blog.netways.de // @netways
Make IT do more with less
NETWAYS GmbH
Deutschherrnstrasse 15-19
90429 Nürmberg
Phone: +49 911 92885-0
Fax: +49 911 92885-77
Email: [email protected]
Website: www.netways.de
Twitter: twitter.com/netways
Facebook: facebook.com/netways
Blog: blog.netways.de
QUESTIONS & ANSWERS