©2015 GlobalLogic Inc. CONFIDENTIAL

The ELK stack - get to know logs

Igor RudykDevOps / System Integrator

2 CONFIDENTIAL

Agenda• Introduction. What is ELK, and why do we need it?• The ELK stack

LogstashElasticSearchKibana

•Architecture•Demo

3 CONFIDENTIAL

Can you check the errors from yesterday between 9:09 and 9:27

4 CONFIDENTIAL

So what’s a log

5 CONFIDENTIAL

Log = timestamp + data

6 CONFIDENTIAL

Lifecycle of a log

Rec

ord

Transmit Analyze

Store

Delete

Real Real

7 CONFIDENTIAL

Default problem

• Multiple log time formats•Apr 28 20:21:59

•[27/Apr/2015:07:05:28 +0000]

•071012 09:27:32

•Mon, 27-Apr-15 06:27:02 UTC

•2015-04-28 20:07:51 +0000

• Starts not with timestamp or without timestamp•Error messages with really unhelpful info•No rotation•No scaling

Tools?

• grep

• awk / sed / cut

•less / tail

•vi / vim

•regular expression

•...

8 CONFIDENTIAL

Logging SolutionsSolutions

Collections Transport Parsing Storage Analysis Alerting

Visualizer

Commercial

Logstash

Logstash shipper or logstash-forwarder (Lumberjack)

RPM installation

Logstash shipper or logstash-forwarder (Lumberjack, encrypted transport is the default)

RPM installation

Output plugins

Central server-master with a hot-standby in case of failure

Codecs plugins

Grok debugger

ElasticSearch,

MongoDB,

AWS S3

and much more

Kibana,

graylog2

Riemann Kibana,

graylog2

NO

fluentd Input plugins

Install from source or via gem

Output plugins

Load-balance between multiple hosts or have a master with a hot-standby in case of failure

Plugins Doesn’t provide any storage tier itself but allows you to easily configure where your logs should be collected

Kibana,

graylog2

Riemann Kibana,

graylog2

NO

splunk Splunk Universal Forwarder

RPM installation

Splunk Universal Forwarder

SSL security

Splunk Splunk Splunk Splunk Splunk YES

Graylog2

Graylog2

Logstash

Graylog2

Logstash

Graylog2 ElasticSearch Kibana,

graylog2

graylog2 graylog2 NO

loggly loggly loggly loggly Hosted loggly loggly loggly YES

9 CONFIDENTIAL

What is ELK, and why do we need it?

• ELK is a stack of programs that help dealing with logs.

• Includes:

– Aggregation of logs

– Search capabilities

– Aggregation of statistics

– Visualizations

10 CONFIDENTIAL

Logstash

Unstructured

Filters Outputs

Documents

11 CONFIDENTIAL

Inputs➔ Logs:

● Lumberjack - resilient, compressed, secure (logstash-forwarder)● Remote syslog● Files

➔ Devices:● Event log, Collectd● Netflow, WMI

➔ Event Queue:● Redis, RabbitMQ● Kafka, ZeroMQ

➔ Streaming APIs:● Twitter

➔ Email (IMAP)➔ Amazon S3, ganglia, sqs, varnishlog, etc➔ ....

http://logstash.net/docs/1.4.2/ - Full list

12 CONFIDENTIAL

Filters➔ grep➔ date➔ json➔ grok➔ ....

http://logstash.net/docs/1.4.2/ - Full list

Why Do I like Logstash?It uses Grok filter for parsing standard and non standard logs:

Log Line:27/10/14 07:39:28 [localhost-startStop-1] [] INFO com.vidmind.config.LoggingPropertyPlaceholderConfigurer - streams.limit.general = 0

Pattern:

%{DATESTAMP} %{SYSLOG5424SD} ?? %{WORD:ErrorLevel} %{JAVACLASS}

13 CONFIDENTIAL

Outputs

➔ Storage:● ElasticSearch● MongoDB● S3● Graphite● File● ...

➔ Notification: ● Zabbix ● Nagios● Riemann● PagerDuty● Email

➔ Event Queue:● Redis, RabbitMQ● Kafka, ZeroMQ● tcp/udp

➔ SaaS:● AWS CloudWatch● Hipchat● Jira

➔ ....http://logstash.net/docs/1.4.2/ - Full list

14 CONFIDENTIAL

Logstash - Forwarder (Shipper)Configuration file

{ "network": { "servers": [[logstash_indexers]] "timeout": 15, "ssl ca": "logstash-forwarder.crt" }, "files": [ { "paths": [ "/usr/share/tomcat7/logs/*.json.log" ], "fields": { "type": "tomcat", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]" } },

{ "paths": [ "/usr/share/tomcat7/logs/*.activities.log" ], "fields": { "type": "activities", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]"

} }

]}

15 CONFIDENTIAL

Logstash-IndexerConfiguration file

input { lumberjack { codec => json{} port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" }}

output {elasticsearch { host => "127.0.0.1" protocol => "http" cluster => "[[elasticsearch_cluster_name]]" manage_template => false index => "logstash-%{system}-%{type}-%{+YYYY.MM.dd}" }}

16 CONFIDENTIAL

ElasticSearchConfiguration file (yaml-based configuration)

cluster.name: [[elasticsearch_cluster_name]]node.name: "[[node_name]]"

node.master: false / truenode.data: false / true

index.number_of_replicas: 1

#Securitydiscovery.zen.ping.multicast.enabled: falsediscovery.zen.ping.unicast.hosts: [[elasticsearch_servers]]

action.disable_close_all_indices: trueaction.disable_delete_all_indices: trueaction.disable_shutdown: truescript.disable_dynamic: true

17 CONFIDENTIAL

The ELK stackGeneral Architecture

Logstash ElasticSearch Kibana

18 CONFIDENTIAL

KibanaElasticSearch

Logstash-Forwarder

Logstash-Indexer

Logstash-Indexer

Logstash-Indexer

The ELK stackOur Scaled Architecture

ElasticSearch

ElasticSearch

Kibana

Kibana

©2015 GlobalLogic Inc. CONFIDENTIAL

DEMO