monitoring the elk stack using zabbix and grafana (dennis kanbier / 26-11-2015)

Xifeo ICT | Dennis Kanbier

Monitoring the ELK stack using Zabbix (and

Grafana)Dennis Kanbier

26 Nov 2015 - NLZGG


Overview• Introduction

• What is ELK and why do I want it?

• How to monitor ELK with Zabbix

• Impress people with pretty graphs

• Demo time ( please tweet using #nlzgg! )


Who am I?

• Dennis Kanbier

• [email protected]

• Linux Consultant with Xifeo ICT

• “dkanbier” on Zabbix forums and IRC

mailto:[email protected]


What is the ELK stack• Elasticsearch

• Search server

• Based on Apache Lucene

• Logstash

• Data pipeline

• Processes logs and other data

• Plugins

• Kibana

• Web frontend for Elasticsearch


Basic Logstash Pipeline


Apache log example• 83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/

logstash-monitorama-2013/images/ahiruyaki.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36”

• A few interesting fields:

• client_ip: 83.149.9.216

• status code: 200

• timestamp: 04/Jan/2015:05:13:42 +0000

• and much more


{ "clientip" : "83.149.9.216", "ident" : , "auth" : , "timestamp" : "04/Jan/2015:05:13:42 +0000", "verb" : "GET", "request" : “…/a-2013/images/ahiruyaki.png”, "httpversion" : "HTTP/1.1", "response" : "200", "bytes" : "203023", "referrer" : “http://semicomplete.com …“, "agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X…” }

Logstash processes the log message and produces output:

http://semicomplete.com


Now what?• Use the processed data to do fun things like:

• plot all entries using the [timestamp] field to see how busy the web server is during the day

• which part of your website gets requested the most using the [request] field

• monitor the status of your web server using the [response] field

• Or do even more fun things using plugins…


logstash-filter-geoip• Use the geo-ip filter plugin while logstash is processing the

message

• Looks up the geographical location of an IP address and adds this information to the message

• Use Kibana to create a heat map based on the geographical information in each message to see from which part in the world your website gets visited

• Very nice to have in case of DDOS attacks originating from places you don’t expect visitors from

• Or to determine where to up your advertising game


Example heat map


Developer issues


Not always the developers fault


Why developers love ELK• Logs from all components and servers are available in one central place, easily searchable using

the Kibana web interface

• No more begging the scary Unix administrator for log files

• Log message are enriched with meaningful fields which enables users to:

• query on stack traces

• query on specific thread id’s

• query on log_level

• query on customer_id

• list faults per server/application/cluster/log_type/etc..

• You can easily plot logs, see how fast requests are handled

• Timestamps are normalised, even log entries from different time zones are searchable chronologically


Example Kibana screen


Monitoring Logstash with Zabbix

• Used 3 components to generate data and send it to the Zabbix server:

• logstash-input-heartbeat

• logstash-filter-metrics

• logstash-output-zabbix


logstash-input-heartbeat• Plugin to generate a heartbeat message at a

configurable interval

• The message contains a field called “clock” with the epoch time of the moment it was send

• Assign a type of “heartbeat” to the message so we can use that to filter out the message further in the pipeline and send it to the Zabbix server


logstash.confinput { heartbeat { message => "epoch" interval => 20 type => "heartbeat" }}

filter {}

output {}


logstash-output-zabbix

• Used to send data to Zabbix

• Requires a configured Zabbix host and trapper item to receive the data

• These fields must be added to the [@metadata] field of the message you want to send to the Zabbix server


logstash.confinput { heartbeat { message => "epoch" interval => 20 type => "heartbeat" }}

filter { if [type] == "heartbeat" { mutate { add_field => { "[@metadata][zabbix_key]" => "logstash_heartbeat" } add_field => { "[@metadata][zabbix_host]" => "logstash-indexer-1" } } }}

output { zabbix { zabbix_server_host => “zabbix-server.example.com” zabbix_host => “[@metadata][zabbix_host]” zabbix_key => “[@metadata][zabbix_key]" zabbix_value => clock}

http://zabbix-server.example.com


Results in Zabbix• Zabbix host logstash-indexer-1 should already have the item

with key “logstash_heartbeat” configured as a Zabbix trapper item.

• Logstash now sends the epoch value every 20 seconds to the logstash_heartbeat trapper item

• You can use this to further process the data in Zabbix:

• trigger when there is too much time between epoch values

• trigger when you do not receive any values for a period of time


Example item in Zabbix


logstash-filter-metrics• Plugin to count messages and flush the result on a

specific interval

• Used to monitor logstash performance by counting messages and store their count value in Zabbix as Delta (speed per second)

• which gives us the messages this logstash server handles per second


logstash.confinput { …}

filter { metrics { meter => "events" add_tag => "metric" add_field => { "[@metadata][zabbix_key]" => "logstash_events" } add_field => { "[@metadata][zabbix_host]" => “logstash-indexer-1“ } flush_interval => 20 }}

output { if "metric" in [tags] { zabbix { zabbix_server_host => “zabbix-server.example.com” zabbix_host => “[@metadata][zabbix_host]” zabbix_key => “[@metadata][zabbix_key]" zabbix_value => “[events][count]” } }}

http://zabbix-server.example.com


More results!• We now have the item “logstash_events” which

shows how many messages we process per second:


Now about that last slide….

• Presenting reports and graphs are not exactly Zabbix’ strongest points

• Works all right to view single values like on the previous slide

• But try to align multiple graphs nicely on one screen to put on a wall and impress people


Grafana• Build to present metric data in a simple and nice

manner

• Supports multiple data sources like Graphite and InfluxdB

• Alexander Zobnin created grafana-zabbix, which enables us to use the Zabbix database as a Grafana backend through the Zabbix API

• https://github.com/alexanderzobnin/grafana-zabbix

https://github.com/alexanderzobnin/grafana-zabbix


Example dashboard


Things to consider• Grafana queries the Zabbix database through the

API

• If you have a lot of graphs with a lot of items these queries are quite expensive

• Especially if you have multiple dashboards refreshing automatically on a high interval

• Still learning on how to best counter these challenges


Live demo time!Did you remember to tweet things with hashtag #nlzgg?


Questions?


Contact details

• dkanbier on Freenode IRC

• https://github.com/dkanbier

• http://denniskanbier.nl/blog for tutorials

• [email protected]

Resources• http://www.zabbix.com • https://github.com/alexanderzobnin/grafana-zabbix • https://www.elastic.co • http://www.xifeo.nl

http://denniskanbier.nl/blog

mailto:[email protected]

http://www.zabbix.com

https://github.com/alexanderzobnin/grafana-zabbix

https://www.elastic.co

http://www.xifeo.nl

monitoring the elk stack using zabbix and grafana (dennis kanbier / 26-11-2015)

Technology