the road to lazy monitoring with icinga2 & puppet - … · the road to lazy monitoring with...

The road to lazy monitoringThe road to lazy monitoringwith Icinga2 & Puppetwith Icinga2 & Puppet

Tom De Vylder@tomdevylder

whoiswhois

● Ops guyOps guy

● Rusty, old, Bavarian vehicles enthousiastRusty, old, Bavarian vehicles enthousiast

● The one who always gets the patdown at checkpointsThe one who always gets the patdown at checkpoints

● Open Source Consultant & Enthousiast at InuitsOpen Source Consultant & Enthousiast at Inuits

● Team Packages & Tools at IcingaTeam Packages & Tools at Icinga

Monitoring!Monitoring!

Monitoring!Monitoring!

Image credits: http://mta.hu/data/cikk/13/00/73/cikk_130073/20120611-144546.jpgImage credits: http://mta.hu/data/cikk/13/00/73/cikk_130073/20120611-144546.jpg

Monitoring :((Monitoring :((

Monitoring is...Monitoring is...

● BoringBoring

● TediousTedious

● RepetitiveRepetitive

● Production = Production = deploymentdeployment + monitoring + backup + monitoring + backup

● Often neglectedOften neglected

Why can't Why can't someone else do it?someone else do it?

Someone else!Someone else!

● TediousTedious


● People make mistakesPeople make mistakes

● Nobody likes Nagios configsNobody likes Nagios configs

Someone else!Someone else!

● TediousTedious


● People make mistakesPeople make mistakes

● Nobody likes Nagios configsNobody likes Nagios configs

● Monitor servers not peopleMonitor servers not people

Image credits: https://openclipart.org/image/2400px/svg_to_png/219371/You-Are-Being-Monitored.pngImage credits: https://openclipart.org/image/2400px/svg_to_png/219371/You-Are-Being-Monitored.png

Single source of truthSingle source of truth

Config management!Config management!

Config management!Config management!

PuppetPuppet RudderRudder

JujuJujuChefChef

AnsibleAnsible

Saltstack SaltstackCFEngineCFEngine

TrueConfigTrueConfig

PuppetPuppet

PuppetPuppet● Config management toolConfig management tool

● Exported resourcesExported resources

● PuppetDBPuppetDB

● Nagios_* {} resource typesNagios_* {} resource types

node “web.example.org” { @@nagios_host { $::fqdn: ensure => present, alias => $::hostname, address => $::ipaddress, use => "generic-host", }

@@nagios_service { "check_ping_${::fqdn}": check_command => "check_ping", use => "generic-service", host_name => $::fqdn", notification_period => "24x7", service_description => "${::fqdn}_check_ping", }}

Nagios_* {} resource typesNagios_* {} resource types

node icinga.example.org { Nagios_host <<||>> Nagios_service <<||>>}


@@nagios_host { $::fqdn: ensure => absent,}

@@nagios_service { "check_ping_${::fqdn}": ensure => absent,}


Nagios_host <<||>>Nagios_service <<||>>


/etc/nagios/




● Nagios_* resource typesNagios_* resource types




● Nagios_* resource typesNagios_* resource types

● https://github.com/Icinga/puppet-icinga2https://github.com/Icinga/puppet-icinga2

● https://github.com/Icinga/puppet-icingaweb2https://github.com/Icinga/puppet-icingaweb2

Puppet-Icinga2Puppet-Icinga2

puppet-icinga2 (develop) ☃ git log | tail -4Author: Nick ChappellDate: Fri May 2 19:36:32 2014 -0700

Initial commit.

Other tools...Other tools...● https://github.com/Icinga/puppet-icinga2https://github.com/Icinga/puppet-icinga2

● https://github.com/Icinga/puppet-icingaweb2https://github.com/Icinga/puppet-icingaweb2

● https://github.com/Icinga/icinga2-ansiblehttps://github.com/Icinga/icinga2-ansible

● https://github.com/Icinga/docker-icinga2https://github.com/Icinga/docker-icinga2

● https://github.com/Icinga/chef-icinga2https://github.com/Icinga/chef-icinga2

Puppet as Puppet as central source of truthcentral source of truth

Exported resourcesExported resources

Collected resourcesCollected resources

Icinga2Icinga2

Icinga2Icinga2● ApplyApply rules rules

● Assign rulesAssign rules

template Service "generic-service" { max_check_attempts = 3 check_interval = 5min retry_interval = 5min enable_perfdata = true}

apply Service "ping4" { import "generic-service" check_command = "ping4" assign where host.address}

apply Service "ping6" { import "generic-service" check_command = "ping6 assign where host.address6}

Icinga2Icinga2

apply Service "ssh" { import "generic-service" check_command = "ssh" assign where host.address && host.vars.os == "Linux"}

Icinga2Icinga2

object HostGroup "database" { display_name = "Database Servers" assign where match("mysql*", host.name) assign where match("psql*", host.name) assign where match("db*", host.name)}

Icinga2Icinga2

Icinga2Icinga2● ApplyApply rules rules

● Assign rulesAssign rules


node 'web.example.org' { @@::icinga2::object::host { $::fqdn: groups => ['puppet', $::operatingsystem], target_dir => '/etc/icinga2/objects/hosts', target_file_name => "${::fqdn}.conf", template_to_import => 'linux-host', vars => { distro => $::operatingsystem, distro_version => $::operatingsystemrelease, virtual_machine => $::is_virtual, puppet_version => $::puppetversion, puppet_environment => $::environment, }, }}

node 'web.example.org' { @@::icinga2::object::host { $::fqdn: groups => ['puppet', $::operatingsystem], target_dir => '/etc/icinga2/objects/hosts', target_file_name => "${::fqdn}.conf", template_to_import => 'linux-host', vars => { distro => $::operatingsystem, distro_version => $::operatingsystemrelease, virtual_machine => $::is_virtual, puppet_version => $::puppetversion, puppet_environment => $::environment, puppet_classes => $::puppet_classes, puppet_modules => $::puppet_modules, }, }}

classes_file = '/var/lib/puppet/classes.txt'classes_hash = {}modules_array = []

File.foreach(classes_file) do |l| modules_array << l.chomp.gsub(/::.*/, '')end

modules_array = modules_array.sort.uniqmodules_array.each do |i| classes_array = [] classes_array << i File.foreach(classes_file) do |l| classes_array << l.chomp if l =~ /^#{i}/ classes_array = classes_array.sort.uniq end classes_hash[i] = classes_arrayend

Facter.add(:puppet_modules, :timeout => 10) do confine :kernel => 'Linux' setcode do modules_array.sort.uniq.join(', ').to_s endend

Facter.add(:puppet_classes, :timeout => 10) do confine :kernel => 'Linux' setcode do classes_hash.map { |_k, v| v }.sort.uniq.join(', ').to_s endend

# facter -p puppet_modules --yaml--- puppet_modules: - apache - collectd - concat - dnsclient - firewall - grafana - icinga2 - logrotate - mysql - postfix - ssh - stdlib - sudo - sysctl - ulimit

# facter -p puppet_classes --yaml--- puppet_classes: mysql: - mysql - "mysql::client" - "mysql::client::install" - "mysql::server" - "mysql::server::config" - "mysql::server::install" - "mysql::server::installdb" - "mysql::server::service" icinga2: - icinga2 - "icinga2::server" - "icinga2::server::config" - "icinga2::server::features" - "icinga2::server::install" - "icinga2::server::service"

@@::icinga2::object::host { $::fqdn: groups => ['puppet', $::operatingsystem], target_dir => '/etc/icinga2/objects/hosts', target_file_name => "${::fqdn}.conf", template_to_import => 'linux-host', vars => { distro => $::operatingsystem, distro_version => $::operatingsystemrelease, virtual_machine => $::is_virtual, puppet_version => $::puppetversion, puppet_environment => $::environment, puppet_classes => inline_template( '<%= puppet_classes.collect { |k, v| v }.sort.join(" ") %>' ), puppet_modules => inline_template( '<%= puppet_modules.sort.join(" ") %>' ), },}

icinga2::object::apply_service_to_host { 'mysql_server': check_command => 'by_ssh', display_name => 'MySQL', target_dir => '/etc/icinga2/objects/applys', assign_where => '"match(*mysql::server*", host.vars.puppet_classes)"', vars => { by_ssh_command => "check_tcp -H localhost -p 3306", by_ssh_logname => 'root', by_ssh_quiet => true, by_ssh_timeout => 60, },}

Collected resourcesCollected resources

There's a better wayThere's a better way

Query PuppetDBQuery PuppetDB

curl \ -s \ -X GET \ http://puppetdb:8080/pdb/query/v4/resources \ --data-urlencode \ 'query=["extract", "certname", ["and", ["=", "type", "Class"], ["~", "title", "^Settings"] ] ]'

Query PuppetDBQuery PuppetDB

PuppetDBqueryPuppetDBquery

$nodes = query_nodes('Class[Settings]', 'fqdn')


$nodes = query_nodes('Class[Settings]', 'fqdn')notify { "${nodes}": }

# puppet agent --test –noopInfo: Applying configuration version '1447591034'Notice: /Stage[main]/Main/Notify[web.example.org,db.example.org, icinga2.example.org,puppet.example.org]/message: current_value absent, should be [web.example.org, db.example.org,icinga2.example.org, puppet.example.org] (noop)Notice: Applied catalog in 0.21 seconds


$nodes = query_nodes('Class[Mysql::Server]', 'fqdn')notify { "${nodes}": }

# puppet agent --test –noopInfo: Applying configuration version '1447591391'Notice:/Stage[main]/Main/Notify[db.example.org]/message:current_value absent, should be db.example.org (noop)Notice: Applied catalog in 0.21 seconds


$nodes = query_nodes('Class[Settings]', 'fqdn')

define monitor::host ($host_name) { $foo.each |Integer $index, String $host| { ::icinga2::object::host { $host: groups => ['puppet', $::operatingsystem], target_dir => '/etc/icinga2/objects/hosts', target_file_name => "${host}.conf", template_to_import => 'linux-host', } }}

::monitor::host { 'all': host_name => $nodes, }


SummarySummary● No more exported resourcesNo more exported resources

● Automagically worksAutomagically works

● Production = deployment + Production = deployment + monitoringmonitoring + backup + backup

● Purge decommissioned nodesPurge decommissioned nodes

ContactContactTom De VylderTom De Vylder

@tomdevylder@[email protected]@inuits.eu

INUITS bvbaINUITS bvbaHeidestraat 257Heidestraat 2572070 Zwijndrecht2070 ZwijndrechtBelgiumBelgium

+32 475 961221+32 475 [email protected]@inuits.eu

the road to lazy monitoring with icinga2 & puppet - … · the road to lazy monitoring with...

Documents