Log Aggregation and AnalysisUsing Elastic Stack

1Logging - What ?Human readable and machine parseable format

Record now, analyze later

Various sources of eventsWebservers

User activity on a website

Application logs

Node metrics

Other operational data

Mobile / IoT devices

2Logging - Why ?Record keeping

Operational InsightsMonitor and optimize resource consumption / utilization

Early detection find out before it goes wrong

Incident forensicsWhere did it go wrong ?

Consistency related bugs Failing system is better than an incosistent system

Answer Questions in (near)real-timeFunctional metrics How many users logged in within last hour?, Which location is most active right now?, What's the average response time for X page?

3Available toolsSplunkGraylogElastic stack

LicensePaidFreeFreemium

Setup complexityEasyMediumMedium

HostingOn premise / HostedOn premise / HostedOn-premise / Hosted

CapabilitiesIngestion / Storage / Analytics / AlertsIngestion / Storage / Analytics / AlertsIngestion / Storage / Analytics / Alerts

Scalable ?YesYesYes

Architecture

MonolithMonolithDivided into different components Each can be used separately

And many more - Grafana, Logsearch, MS Azure Log Analytics, Loggly

4Elastic Stack

Explore and visualize your data. Search, Dashboards and many more.

Lightweight data shippers

Parse, Enrich & Transport DataStore, search, and analyze your data.

5Step 1 - Gathering logs

Lightweight Data ShippersBeats is the platform for single-purpose data shippers. They install as lightweight agents and send data from hundreds or thousands of machines to Logstash or Elasticsearch. Using log appenders / handlers

Beats

6Beats (by Dre Elastic) FilebeatReads from file

Non-intrusive

MetricbeatCollects metrics from systems and services

Modules available for Apache, nginx, Docker, Kafka, PostgreSQL and more

PacketbeatLightweight network packet analyzer

Modules available for HTTP, DNS, AMQP and more

WinlogbeatCollects windows event logs

Add your ownDozens of community developed beats available

Extensible architecture Easy to create on our own

Written in Go

7Filebeat

filebeat: prospectors: - paths: - /var/log/*.log input_type: log - paths: - /var/log/apache2/* input_type: logoutput: elasticsearch: hosts: ["localhost:9200"]

Sample confiuration :

{ "@timestamp": "2016-01-06T00:00:00Z", "type": "log", "message": ""}

Sample published event:

8Metricbeat

metricbeat.modules:- module: redis metricsets: - info hosts: ["127.0.0.1:6379"]output.elasticsearch: hosts: ["localhost:9200"]

Sample configuration:

{ "@timestamp": "2016-01-06T00:00:00Z", "type": "metricsets", "redis": { ... }}

Sample published event:

9Step 2 - Processing logs using LogstashIngest-process-output pipeline

Ingest Data of All Shapes, Sizes, and SourcesBeats, log4j, redis, tcp/udp, HTTP

ProcessTransform unstructured data to structured data using grok filter

Filter out unnecessary data

Mutate data (calculate fields, add extra context, get geo co-ordinates from IP address, etc)

Stash it awayData stores (elasticsearch, files, mongoDB, redis), other services (email, pagerduty, redmine, irc, jira), brokers (kafka, rabbitMQ) and many more

Scalable, Durable

10Logstash contd.

GeoIP Lookup /
Data Mutation /Structuring data

input { beats {port => 5044}}

filter{ if[type] == "nginx-logs" { grok { match => {"message" => "%{PATTERN}"} } geoip { source => "remote_addr" } }}

output { elasticsearch { hosts => ["localhost:9200"] index => "logstash-%{+YYYY.MM.dd}"}}

Sample confiuration :

11Step 3 - Storing logs in ElasticsearchDistributed RESTful search and analytics engine (JSON/HTTP)

Fast get your answers instantly

Scalable Run on your laptop or hundreds of servers

Resilient and Highly Available Clustering, Failure detection

Full text search, Aggregation, Geo filtering (within x mile radius), Suggestions (show more like this), Fuzzy search, Scripting

12Step 4 - Explore and Visualize using KibanaWorks seamlessly with Elasticsearch

Easy yet powerful search interface

Supports histograms, line graphs, pie charts and many more

Visualize geospatial data

Extensible Create your own visualization

Create and share dashboards

13Elastic Stack in Action

14

Demo 1 Parsing nginx logs Nginx Access Logs

Filebeat conf

FilebeatRead file(s) push to Logstash

LogstashParse logs
GeoIP lookup
User agent parsingPush to Elasticsearch

ElasticsearchIndex and store

KibanaSearch and Analyze

15Demo 2 Logs from a Django application

ElasticsearchIndex and store

KibanaSearch and Analyze

LogstashCollect logs
Push to Elasticsearch

Django App
Logstash handler

16Demo 3 Capture and monitor node metricsMetricbeat conf

MetricbeatRead metrics
from nodes

ElasticsearchIndex and store

KibanaSearch and Analyze

17Q & A

18Thank You

Ahmedabad Java Meetup Grouphttps://www.meetup.com/Ahmedabad-Java-Meetup-Group/ @JavaMeetup AhmedabadJavaMeetup

Dhaval Mehtadmehta17@gmail.com @mehtadhaval07