have you been stalking your servers?
DESCRIPTION
A presentation for DrupalCon Prague 2013TRANSCRIPT
![Page 1: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/1.jpg)
Have you been stalking your servers?
![Page 2: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/2.jpg)
Have you been stalking your servers?
Marji CermakSysadmin & DevOps Engineer at Morpht
[email protected]@cermakm
![Page 3: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/3.jpg)
The rule of 3 things
picture: http://www.flickr.com/photos/helenaperezgarcia/5692392667/
![Page 4: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/4.jpg)
The rule of 3 things
1. What is monitoring and why do you want to monitor
2. Some monitoring tools available for you
3. It is easy to start with monitoring.
![Page 5: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/5.jpg)
Part 1
What is monitoring and why do you want to monitor
![Page 6: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/6.jpg)
photo: http://www.flickr.com/photos/tiagopadua/7903366470/
![Page 7: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/7.jpg)
Monitoring
Monitoring is an intermittent (regular or irregular) series of observations in time, carried out to show the extent of compliance with a formulated standard or degree of deviation from an expected norm.
J. M. Hellawell (1991), modified by A. Brown (2000), http://jncc.defra.gov.uk/page-2268nature conservation area
![Page 8: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/8.jpg)
Why you need to monitor
● to know about the bad news before your customers (or your boss)
![Page 9: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/9.jpg)
Why you need to monitor
● to know about the bad news before your customers (or your boss)
● to scale up your server in advance
![Page 10: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/10.jpg)
Why you need to monitor
● to know about the bad news before your customers (or your boss)
● to scale up your server in advance
● to tune up your app
![Page 11: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/11.jpg)
Why you need to monitor (cont.)
● to prove your uptime of 99.999 :)
![Page 12: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/12.jpg)
The fun of the nines
Source: http://en.wikipedia.org/wiki/High_availability
Nines: http://en.wikipedia.org/wiki/List_of_unusual_units_of_measurement#Nines
![Page 13: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/13.jpg)
Why you need to monitor (cont.)
● to prove your uptime of 99.999 :)
● to minimise downtime (expensive)
![Page 14: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/14.jpg)
Why you need to monitor (cont.)
● to prove your uptime of 99.999 :)
● to minimise downtime (expensive)
● to capture customer information
![Page 15: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/15.jpg)
Why you need to monitor (cont.)
● to have data / metrics to diagnose
![Page 16: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/16.jpg)
Diagnosing your collected data
watch out for:● trends
![Page 17: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/17.jpg)
Diagnosing your collected data
watch out for:● trends● spikes
![Page 18: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/18.jpg)
Diagnosing your collected data
watch out for:● trends● spikes● irregularities
![Page 19: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/19.jpg)
Diagnosing your collected data
watch out for:● trends● spikes● irregularities● thresholds
![Page 20: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/20.jpg)
Areas to monitor
● network
photo: http://www.flickr.com/photos/misja_klimov/2120956405/
![Page 21: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/21.jpg)
Areas to monitor
● network● server
photo: http://www.flickr.com/photos/johnjack/3666997634/
![Page 22: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/22.jpg)
Areas to monitor
● network● server● services
photo: http://www.flickr.com/photos/agustingodet/3691794089/
![Page 23: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/23.jpg)
Areas to monitor
● network● server● services
photo: http://www.flickr.com/photos/agustingodet/3691792393/
![Page 24: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/24.jpg)
Areas to monitor
● network● server● services● applications photo: http://www.flickr.com/photos/cheerfulstoic/942211994/
![Page 25: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/25.jpg)
Areas to monitor
● network● server● services● applications● users
photo: http://www.flickr.com/photos/jimmysmith/99528596/
![Page 26: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/26.jpg)
Drupal Areas to monitor?
● network● server● services● applications● users
![Page 27: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/27.jpg)
Drupal Areas to monitor
● network● server ● services● applications● users
![Page 28: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/28.jpg)
Drupal Areas to monitor
● network● server ● services● applications● users
![Page 29: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/29.jpg)
Drupal Areas to monitor
● network● server● services
○ webserver○ database
● applications● users
![Page 30: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/30.jpg)
Drupal Areas to monitor
● network● server● services
○ webserver○ database
● applications - your Drupal site(s)● users
![Page 31: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/31.jpg)
Drupal Areas to monitor
● network● server● services
○ webserver○ database
● applications - your Drupal site(s)● users
![Page 32: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/32.jpg)
Part 2
Some monitoring tools available for you
![Page 33: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/33.jpg)
Meet Nagios, Munin and others
● Nagios● Munin● APC dashboard
● related Drupal modules
![Page 34: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/34.jpg)
![Page 35: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/35.jpg)
Nagios /ˈnɑːɡiːoʊs/
● system, network and infrastructure monitoring software application
● monitors and alerts● many plugins
![Page 36: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/36.jpg)
Nagios /ˈnɑːɡiːoʊs/
Name and Pronunciation:● NetSaint -> "Nagios Ain't Gonna Insist On
Sainthood"● Agios' a transliteration of the Greek word
άγιος (saint)
![Page 37: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/37.jpg)
Nagios /ˈnɑːɡiːoʊs/
● alerts by email/pager/IM...● alerts to different contacts● notification escalation● service / host dependencies● soft / hard states
![Page 38: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/38.jpg)
Nagios /ˈnɑːɡiːoʊs/
![Page 39: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/39.jpg)
Drupal and Nagios
![Page 40: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/40.jpg)
![Page 41: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/41.jpg)
Munin
● network/system monitoring application● outputs graphs through a web interface● many plugins
![Page 42: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/42.jpg)
Munin
● master / node architecture● connects to all nodes at regular intervals ● it uses the RRDtool (round robin database
tool, handles time-series data)
![Page 43: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/43.jpg)
![Page 44: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/44.jpg)
![Page 45: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/45.jpg)
Munin Example
![Page 46: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/46.jpg)
Drupal and Munin
![Page 47: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/47.jpg)
Drupal and Munin
![Page 48: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/48.jpg)
● they complement each other● nagios normally alerts on one “service” ● munin can be used to correlate different
things
Nagios & Munin
![Page 49: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/49.jpg)
APC - what is it?
The Alternative PHP Cache (APC) is a free and open opcode cache for PHP.
![Page 50: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/50.jpg)
APC - what is it?
The Alternative PHP Cache (APC) is a free and open opcode cache for PHP.
Its goal is to provide a free, open, and robust framework for caching and optimising PHP intermediate code.
Inside your webserver (not a webcache)
![Page 51: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/51.jpg)
Monitoring APCMemory Usage, Hit & Misses
![Page 52: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/52.jpg)
Monitoring APCFragmentation
![Page 53: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/53.jpg)
Monitoring APCmemory usage
![Page 54: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/54.jpg)
Monitoring APCfiles in cache
![Page 55: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/55.jpg)
Other monitoring tools
● Collectd● Graphite● Shinken● Sensu● NewRelic● Pingdom
![Page 56: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/56.jpg)
Part 3
It is easy to start with monitoring.
![Page 57: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/57.jpg)
How to install these tools?
Muninsudo apt-get install munin munin-node
Nagiossudo apt-get install nagios3
APC dashboardphp.apc script from php-apc package
![Page 58: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/58.jpg)
How to configure these?
● It is a bit fiddly● There are many guides targeting beginners● You don’t want to do it again and again
![Page 59: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/59.jpg)
puppet – a quick way to start
system for automating system administration tasks
![Page 60: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/60.jpg)
puppet – a quick way to start
● a declarative language for expressing system configuration,
![Page 61: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/61.jpg)
puppet – a quick way to start
● a declarative language for expressing system configuration,
● a client and server for distributing it
![Page 62: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/62.jpg)
puppet – a quick way to start
● a declarative language for expressing system configuration,
● a client and server for distributing it
● and a library for realising the configuration.
![Page 63: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/63.jpg)
puppet – a quick way to start
package { 'munin-node': ensure => installed }
service { 'munin-node':
enable => true,
ensure => running,
require => Package['munin-node'],
}
![Page 64: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/64.jpg)
puppet – a quick way to start
1. clone the stalk-your-box repo
2. run puppet apply on the code
3. monitor!
![Page 65: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/65.jpg)
A quick way to start
$ git clone git://github.com/morpht/stalk-your-box.git /tmp/stalk-your-box
Cloning into '/tmp/stalk-your-box'...remote: Counting objects: 23, done.remote: Compressing objects: 100% (19/19), done.remote: Total 23 (delta 1), reused 23 (delta 1)Receiving objects: 100% (23/23), 11.35 KiB, done.Resolving deltas: 100% (1/1), done.
![Page 66: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/66.jpg)
A quick way to start
$ cd /tmp/stalk-your-box/$ sudo puppet apply --modulepath=modules manifest.pp
notice: /Stage[main]/Nagios::Server/Package[nagios3]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]/Nagios::Server/File[/etc/nagios3/htpasswd.users]/ensure: created
notice: /Stage[main]/Nagios::Server/Exec[update-nagios-htpasswd]/returns: Adding password for user nagiosadmin
notice: /Stage[main]/Nagios::Server/Exec[update-nagios-htpasswd]/returns: executed successfully
notice: /Stage[main]/Munin::Node/Package[libcache-cache-perl]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]/Munin::Node/Package[munin-node]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]/Munin::Node/File[munin-node.conf]/content: content changed '{md5}e486786f866d7d7e025dea401c300e7b' to '{md5}dbf97a87a8da86ef68155815ecae3c1c'
notice: /Stage[main]/Munin::Server/Service[apache2]: Triggered 'refresh' from 1 events
notice: Finished catalog run in 44.26 seconds
![Page 67: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/67.jpg)
What this gives you
![Page 68: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/68.jpg)
What this gives you
![Page 69: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/69.jpg)
What this gives you
![Page 70: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/70.jpg)
Manifest.pp# Execute apt-get update before any package is installed:
exec { 'apt-update':
command => 'apt-get update',
# but don't execute it more than once a day:
unless => 'test $(find /var/cache/apt/pkgcache.bin -mtime 0 | wc -l ) -eq 1',
}
Exec['apt-update'] -> Package <| |>
# Include minimal apache2 installation. Munin server, nagios
# and APC dashboard depend on it.
include 'apache2'
![Page 71: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/71.jpg)
Manifest.pp# Install munin node and munin server:
class { 'munin::node': }
class { 'munin::server':
htuser => 'munin', # Username for basic access auth.
htpass => 'Prague2013' # Password for basic access auth.
}
# Install nagios:
class { 'nagios::server':
contact_email => 'root@localhost', # Email to send alerts to.
htpass => 'Prague2013', # Password for the nagiosadmin username.
}
![Page 72: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/72.jpg)
Manifest.pp
# Deploys APC dashboard - install php-apc package and
# deploy the apc.php script from it.
package { 'php-apc': ensure => installed }
exec { 'deploy-apc-dashboard':
path => '/bin:/usr/bin',
command => 'gzip -dc /usr/share/doc/php-apc/apc.php.gz > /var/www/apc.php',
notify => Service['apache2'],
unless => '[ -f /var/www/apc.php ]',
require => [ Package['php-apc'], Package['apache2'] ]
}
![Page 73: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/73.jpg)
Summary
It is easy to start with monitoring.
![Page 74: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/74.jpg)
The fun part - what’s wrong?
What’s wrong here?
![Page 75: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/75.jpg)
The fun part - what’s wrong?
![Page 76: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/76.jpg)
Questions
Here is the get started monitoring repo:https://github.com/morpht/stalk-your-box
Marji CermakSysadmin & DevOps Engineer at Morpht
[email protected]@cermakm
![Page 77: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/77.jpg)
ResourcesRule of Three: en.wikipedia.org/wiki/Rule_of_three_(writing)Nagios: http://www.nagios.org/Munin: http://munin-monitoring.org/Nagios module: https://drupal.org/project/nagiosMunin module: https://drupal.org/project/muninMunin plugins (experimental): https://drupal.org/sandbox/murrayw/2084281Sensu: http://sensuapp.orgMySQLTuner: http://MySQLTuner.pl
![Page 78: Have you been stalking your servers?](https://reader033.vdocuments.mx/reader033/viewer/2022052522/554f82c6b4c9052a518b4da7/html5/thumbnails/78.jpg)
THANK YOU!
WHAT DID YOU THINK?
Locate this session at the DrupalCon Prague website:http://prague2013.drupal.org/schedule
Click the “Take the survey” link