Listen to the sounds of your application
Originally authored by: Maciej Biłas
Krzysztof Ciesielski
Presented atJ-Day
Lbn.sc
Agenda
• Monitoring, huh?
• Introducing Graphite
• Log analysis – the whys
• Logstash architecture & use cases
• Exploring logs with Kibana
Monitoring, huh?
source: codeascraft.com
Types of measurements
Network
Machine
Application
source: codeascraft.com
measurement > prediction
measurement >> prediction
Our stack
Server
App
Server
App
…
Graphite Logstash
Yammer Metrics
http://graphite/render?target=server.web1.load&height=800&width=600
Graphite
echo "local.random.diceroll 4 `date +%s`" |\ nc graphite-server.your.org 2003
successful.login.attempt 1 1384471287 successful.login.attempt 1 1384471297
successful.login.attempt = 1
Yammer metrics
private final Meter successfulLogins = metrics.meter(name(LoginHandler.class, "successful")); !public void login(String user, String password) { if (canLogin(user,password)) { successfulLogins.mark(); // ... } else { // ... } }
MetricsTypes:
• Gauges
• Counters
• Meters
• Histograms
• Timers
• Health Checks
Reporters:
• STDOUT
• CSV
• SLF4J
• JMX
• Graphite
• Ganglia
Counters, meters… vs.
Gauges
Aggregationapp00.webservice.requestTime app01.webservice.requestTime app02.webservice.requestTime
…
source: codeascraft.com
You can also check out• collectd https://collectd.org
• StatsD https://github.com/etsy/statsd/
• Riemman http://riemann.io/
• Twitter’s Ostrich https://github.com/twitter/ostrich
• Ganglia http://ganglia.sourceforge.net/
• Dashboards: http://shopify.github.io/dashing/https://github.com/obfuscurity/descarteshttps://github.com/obfuscurity/dusk
Log aggregation?
Event Sourcing
Tracing and storing all the events
Current app state: replaying the stream
Great for data mining and analysis
Log aggregation
• Less invasive way to build an “event stream”
• Logs are data with plenty of value
Log aggregation
Log source
Log source
Log source
Central storage
Query engine
Web view
Logstash
• Open source
• Written in JRuby
• Gathers logs from various inputs
• Parses and extracts metadata
• Writes to various outputs
Architecture
Inputscollectd
drupal_dblog
elasticsearch
eventlog
exec
file
ganglia
gelf
gemfire
generator
graphite
heroku
imap
irc
log4j
lumberjack
pipe
rabbitmq
redis
relp
s3
snmptrap
sqlite
sqs
stdin
stomp
syslog
tcp
udp
unix
varnishlog
websocket
wmi
xmpp
zenoss
zeromq
Grok
time 2013-11-10 19:41:25.321
thread main
loglevel INFO
source o.a.camel.impl.DefaultCamelContext
message Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
Thin layer on top of a regular expression
Grok
grok { patterns_dir => “./some-dir“ match => ["message", "%{TIMESTAMP_ISO8601:time} \[%{DATA:thread}\] %{LOGLEVEL:loglevel} %{DATA:source} - % {MULTILINE_GREEDYDATA:message}"] }
2013-11-10 19:41:25.321 [main] INFO o.a.camel.impl.DefaultCamelContext - Apache Camel 2.11.1 (CamelContext: camel-1) started in 4.450 seconds
Outputsboundary
circonus
cloudwatch
csv
datadog
datadog_metrics
elasticsearch
elasticsearch_http
elasticsearch_river
exec
file
ganglia
gelf
gemfire
google_bigquery
google_cloud_storage
graphite
graphtastic
hipchat
http
irc
jira
juggernaut
librato
loggly
lumberjack
metriccatcher
mongodb
nagios
nagios_nsca
null
opentsdb
pagerduty
pipe
rabbitmq
redis
riak
riemann
s3
sns
solr_http
sqs
statsd
stdout
stomp
syslog
tcp
udp
websocket
xmpp
zabbix
zeromq
Kibana
Checking app activity
Spotting anomalies
Regular events
How often is a feature used?
Plotting multiple event types
Other tools
• Splunk
• Graylog2
• Fluentd
• log.io
References• Graphite
http://graphite.readthedocs.org/en/latest/
• Yammer Metricshttp://metrics.codahale.com/
• Logstashhttp://logstash.net/
• Surfing the event stream by Sam Newman at Geeconhttp://www.slideshare.net/spnewman/surfing-the-event-stream
• Lessons from Building and Scaling LinkedIn by Jay Krepshttp://www.infoq.com/presentations/linkedin-architecture-stack
• Code as Craft http://codeascraft.com/
Thank you!Questions?