Log Management Systems

A comparison of message and metric management solutions

Presenter: Mehdi Hamidi( @eXtrem0us )

Introduction

What is LOG?

Combination of Time, Tag and Message

Indicates State of Applications (?)

Human and Machine Readable Messages (?)

Level of logs in syslog standard

From Debug to Panic

rsyslog, syslog, syslog-ng

/var/log/syslog/var/log/rsyslog

Importance of logs

Companies and Businesses

Even Personal Use!
(Twitter, Sensors,... )

LogAnalyzer: a simple solution :)

LogAnalyzer: a simple solution :)

Importance of Logging Systems

WHAT Actually We NEED?

Collect

Messages

Metrics

Store

Visualize

Alert

Importance of Logging Systems

Heterogeneous Environment

Write our own script for each type of log (?)

Not in an enterprise environment with lots of devices and services!

Technical Fragility and dependency to Individuals

Strong Dependency to knowledge about underlying process

Commercial Solutions

Splunk
(500M/Day is Free, then: 5,000,000 $)

Nagios
Everything is restricted to Nagios Concept
No separation between metrics and messages
No stylish diagrams (in free solution)
Problems in cloud infrastructure
No realtime monitoring
No manipulating messages
(1,995 $ for commercial solution)

Online Services

Good logging system Specifications

Have a common interface

Decouple data sources from data outputs

Prevent mentioned dependencies

No effect of adding new data source/output

Reliability

Persistent Buffering

Extensibility

High Availability

Load Balancing

Robustness

Lots of OpenSource Bricks (OSB!)

Logging Systems:Fluentd

LogStash

GrayLog

Logalice

Rsyslog

Scribe

Message Stores:ElasticSeach

Hadoop

MongoDB

File

RDBMS

Redis

...

Visualization
(Dashboards):Kibana

Grafana

Gaylog-WebUI

PacketBeat

Chronograph

...

Metric Stores:InfluxDB

Prometheus

Graphite

...

Alerting:Kapacitor

Skyline

Oculus

Cabot

Log Nature

Semistructured or UnstructuredGenerated MassivelyMore Written and less Read

(That's why we use NoSQL)

Popular Stacks (metrics): TICK Stack

Popular Stacks (Messages) : ELK Stack

Fluentd VS LogStash

Overview

Fluentd:

Written in Cruby

Used in Google Cloud Platform and Kubernetes

Maintained by Tresure Data

Logstash:

Written in Jruby

Used in ELK Stack

Maintained by Elastic Co.

Both use their own RubyGems Repo

Out of the box nature, less dependencies

Configuration

Fluentd:Each Input is taggedLogs are routed by tags

Logstash:All inputs are Gathered and ScatteredConditional Outputs, No tags

Configuration

Transport and Buffering

Fluentd: built-in

LogStash: bundled Redis
version 5.3: persistent buffering

Full Buffer or Output Exception occurrence

Fluentd:
Exception:streamingBlock input plugin:batchDrop oldest chunk:monitoring

LogStash:RetryDiscardDead Letter Queuing

High Availability and Load balancing

High Availability and Load balancing

High Availability and Load balancing

High Availability and Load balancing

Memory

Fluentd:40 MLogstash:120 M

(in big clusters matter)

Forwarders

Fluentd:

Fluentbit(Written in C)

Fluentd-Forwarder(Written in Go)

(all in one)

LogStash:

Filebeat

Metricbeat

Packetbeat

Winlogbeat(beat family: separated component for each purpose)

Community and Support

Fluentd:Poor
Japanese Blogs
Google Group

Logstash:Rich
DocumentsBlogsIRCMeetups and Certs

Plugins

FluentdPluginsVerified

Input/Output55444

Filter908

Parser302

Formatter60

Obsolete 80

Plugins are maintained more by other people.

Plugins

All Plugins are in a Single GitHub Repo.

LogStashPlugins

Input52

Filter46

Output55

Questions?

Thanks You!

ahamidi66@gmail.com
@eXtrem0us

Resources

http://logz.io/blog/fluentd-logstash
http://docs.fluentd.org/articles/buffer-plugin-overvie
https://prometheus.io/docs/introduction/comparison
http://logz.io/blog/elk-stack-5-0
https://www.youtube.com/watch?v=1ye0-sityBw
https://www.youtube.com/watch?v=0lAHrspviIs&list=PL62pIycqXx-TPwtk4JDd0wMuFAyP0gU1y
https://www.youtube.com/watch?v=mfb0R7azKZc
https://www.youtube.com/watch?v=_BAWi9Zhmic