![Page 1: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/1.jpg)
Log all the things!
Honza Král @honzakral
![Page 2: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/2.jpg)
Logs?
![Page 3: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/3.jpg)
Log lines
Twitter feed
Invoices
Metrics
Events!
![Page 4: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/4.jpg)
Why?
![Page 5: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/5.jpg)
What happened last Tuesday?
![Page 6: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/6.jpg)
Multiple machines
Multiple logs
Analysis/Discovery
Time period
Grep?
![Page 7: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/7.jpg)
Time? Time?! Time!
apache
unix timestamp
log4j
postfix.log
ISO 8601
[23/Jan/2014:17:11:55 +0000]
1390994740
2009-01-01T12:00:00+01:00
[2014-01-29 12:28:25,470]
Feb 3 20:37:35
![Page 8: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/8.jpg)
Web Server logs VS Load Balancer see immediately that caching is off static files leaking to gunicorn
Web Server VS Database 500s VS Deploys new version has a bug
Traffic VS Ad Campaigns
Correlate events
![Page 9: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/9.jpg)
Central storage Even for data from different systems
Enriched data IP -> location, hostname URL -> author, product, category
Search user:honza status:404
Analysis Visualisations for easy pattern discovery
Ideal state
![Page 10: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/10.jpg)
Centralised Logging
![Page 11: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/11.jpg)
Steps
Collect data
Parse data
Enrich data
Store data
Search and aggregate
Visualize data
![Page 12: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/12.jpg)
Elastic Stack
![Page 13: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/13.jpg)
Steps in Elastic Stack
Collect data
Parse data
Enrich data
Store data
Search and aggregate
Visualize data
![Page 14: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/14.jpg)
Steps in Elastic Stack
Collect data
Parse data
Enrich data
Store data
Search and aggregate
Visualize data
![Page 15: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/15.jpg)
![Page 16: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/16.jpg)
metricbeat: modules: - module: redis metricsets: ["info"] hosts: ["host1"] period: 1s enabled: true - module: apache metricsets: ["info"] hosts: ["host1"] period: 30s enabled: true
filebeat: prospectors: - paths: - "logs/access.log" document_type: access multiline: pattern: ^# negate: true match: after
protocols: http: ports: [80, 8000]
mysql: ports: [3306]
redis: ports: [6379]
pgsql: ports: [5432]
thrift: ports: [9090]
output: logstash: hosts: ["localhost:5044"]
![Page 17: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/17.jpg)
![Page 18: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/18.jpg)
Inputs
Monitoring collectd, graphite, ganglia, snmptrap, zenoss
Datastores elasticsearch, redis, sqlite, s3
Queues kafka, rabbitmq, zeromq
Logging beats, eventlog, gelf, log4j, relp, syslog, varnish log
Platforms drupal_dblog, gemfire, heroku, sqs, s3, twitter
Local exec, generator, file, stdin, pipe, unix
Protocol imap, irc, stomp, tcp, udp, websocket, wmi, xmpp
![Page 19: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/19.jpg)
Filters
aggregate alter anonymize collate csv cidr clone cipher checksum date dns drop elasticsearch extractnumbers environment elapsed fingerprint geoip grok i18n json json_encode kv mutate metrics multiline metaevent prune punct ruby range syslog_pri sleep split throttle translate uuid urldecode useragent xml zeromq ...
![Page 20: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/20.jpg)
Outputs
Store elasticsearch, gemfire, mongodb, redis, riak, rabbitmq, solr
Monitoring ganglia, graphite, graphtastic, nagios, opentsdb, statsd, zabbix
Notification email, hipchat, irc, pagerduty, sns
Protocol gelf, http, lumberjack, metriccatcher, stomp, tcp, udp, websocket, xmpp
External service google big query, google cloud storage, jira, loggly, riemann, s3, sqs, syslog, datadog
External monitoring boundary, circonus, cloudwatch, librato
Local csv, dots, exec, file, pipe, stdout, null
![Page 21: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/21.jpg)
![Page 22: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/22.jpg)
Open SourceDocument-basedBased on Lucene JSON over HTTP
Distributed Search Engine
![Page 23: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/23.jpg)
Cluster Collection of Nodes
Index Collection of Shards
Shard Unit of scale Distributed across cluster Primary and replica
Data Management
node 1orders
products
21
4
1
node 2orders
products
2
2
node 3orders
3 4
1
3
products
![Page 24: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/24.jpg)
Time based data flow
Current replicas to speed up search on stronger boxes
Week old snapshot keep only 1 replica
Month old move to weaker boxes
2 months close the indices
3 months delete
![Page 25: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/25.jpg)
![Page 26: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/26.jpg)
![Page 27: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/27.jpg)
![Page 28: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/28.jpg)
![Page 29: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/29.jpg)
Architecture
Enrich VisualizeCollect Store
![Page 30: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/30.jpg)
Logging and Python
![Page 31: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/31.jpg)
Track metrics execution time query time # of queries
Include metadata user_id content
Log as JSON
Enhance your logs
![Page 32: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/32.jpg)
Add structured info
Track info through services
Log to file
Add filebeat to read the file
Structlog
![Page 33: Log all the things! - EuroPython€¦ · Centralised Logging. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data. Elastic Stack. Steps in Elastic](https://reader033.vdocuments.mx/reader033/viewer/2022060212/5f0511dc7e708231d4111bda/html5/thumbnails/33.jpg)
Thanks!
Honza Král @honzakral