Upload File

metrics, logs, transaction traces, anomaly detection at scale

41
Processing Metrics, Logs & Traces … at Scale Otis Gospodnetić

Upload: sematext-group-inc

Post on 28-Jul-2015

2.070 views

Category:

Data & Analytics


0 download

Report

Tags:

TRANSCRIPT

Page 1: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

Processing Metrics, Logs & Traces

… at ScaleOtis Gospodnetić

http://sematext.com/
Page 2: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHO WHY

WHAT HOW

Page 3: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHO

HQ: BrooklynPeople: Everywhere

Page 4: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHO Otis Gospodnetić

Sematext founderApache memberBook authorex-Lucene/Solr dev

Page 5: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHO Services

Solr Elasticsearch* Kafka Spark HBase Cassandra...

* We’ve got serious Solr & Elasticsearch ninjas on the team!

Page 6: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHY

Page 7: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHO Clients want

Performance Bottlenecks Tuning Scaling

WHY

Page 8: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHO

Before you can fix things need to know what to fix

WHY

Page 9: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHY We need….INSIGHT

Performance Metrics! Anomalies! Logs!

Page 10: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHY i.e. Need Tools!

Metrics monitoring

Log searching Anomaly alerting

Page 11: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

OSS

Use the (Open) Source, Luke

Page 12: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

OSS

OpenTSDB InfluxDB Ganglia Graphite Nagios ELK ...

Page 13: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

OSS

http://blog.sematext.com/2015/04/22/monitoring-stream-processing-tools-cassandra-kafka-and-spark/

http://blog.sematext.com/2015/04/22/monitoring-stream-processing-tools-cassandra-kafka-and-spark/
http://blog.sematext.com/2015/04/22/monitoring-stream-processing-tools-cassandra-kafka-and-spark/
Page 14: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

OSS

“I have an ELK stack that has been suffering as of late. The logstash service will continually crash, the elasticsearch cluster is hardly in the green, and it is taking a constant amount of maintenance.”

Page 15: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHAT

Page 16: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHAT

SPM → monitoring

Logsene → logging

On PremisesCloud

http://sematext.com/spm http://sematext.com/logsene

http://sematext.com/spm
http://sematext.com/logsene
http://sematext.com/spm
Page 17: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHAT

http://blog.sematext.com/2015/04/22/monitoring-stream-processing-tools-cassandra-kafka-and-spark/

http://blog.sematext.com/2015/04/22/monitoring-stream-processing-tools-cassandra-kafka-and-spark/
http://blog.sematext.com/2015/04/22/monitoring-stream-processing-tools-cassandra-kafka-and-spark/
Page 18: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHAT SPM

Logsene

Page 19: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

HOW

Page 20: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
Page 21: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHAT Agent

Java Node.jsWant Traces? Embed it!Collectd ⇒ SIGAR for OSFlume SpilloverChannelES API

Page 22: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHAT Interesting finds

Variable Collectd supportCollectd ⇒ SIGARApache Flume Elasticsearch Stats APIMetrics 2nd class citizen

Page 23: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHAT Transaction Tracing

Java Bytecode Instrumentation

Bottleneck finderAppMap maker

Page 24: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
Page 25: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
Page 26: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHAT Custom Pointcuts

<method signature="java.lang.String com.company.example.Service#getUserName(com.company.model.Company company)"/>

Page 27: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
Page 28: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

Write-agg vs. Read-agg

Page 29: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

Anomalies > Thresholds

Page 30: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHAT Alerts

HeartbeatsThresholdsAnomalies

Page 31: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHAT Anomaly Detection

ExponentialSTDFromMAKNN ...

boolean result = anomalyCount / (notAnomalyCount + anomalyCount) >= 3d / 4d;

Page 32: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

WHAT Anomaly issues

Warn early / create noiseNormal abnormalitiesSlow change

Page 33: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

Scalable Data Stores

Page 34: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

http://blog.sematext.com/2015/06/09/docker-monitoring-support/

Page 35: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

Logging

Page 36: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

Hot vs. Cold

HOT COLD

Page 37: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

Drop, don’t Delete

HOT COLDdrop

Page 38: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

Pull, don’t Push

GET QUEUEpull

ES

Page 39: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

Beware of Aggregations

Circuit Breakers

Page 40: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

http://blog.sematext.com/2014/10/06/top-5-most-popular-log-shippers/

Page 41: Metrics, Logs, Transaction Traces, Anomaly Detection at Scale

Thank you!

@[email protected]

@sematexthttp://sematext.com

https://twitter.com/otisg
https://twitter.com/sematext

System Logs - Cisco€¦ · System Logs Thischapterdescribeshowtoconfigureparametersrelatedtothevarioustypesofloggingandhowtoviewing theircontent.Itincludesthefollowingsections:

DeepLog: Anomaly Detection and Diagnosis from …lifeifei/papers/deeplog.pdfDeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning Min Du, Feifei Li, Guineng

DeepLog: Anomaly Detection and Diagnosis from System Logs ...lifeifei/papers/deeplog.pdf · „is work proposes DeepLog, a data-driven approach for anom-aly detection that leverages

DATAMINING2 Anomaly & Outliers Detectiondidawiki.di.unipi.it/...dm2_anomaly_outliers_2020.pdf · General Issues: Anomaly Scoring •Many anomaly detection techniques provide only

Jurassic Plant Fossils from Cubaredciencia.cu/geobiblio/paper/2016_Ceballos_Fossil News Sum16.pdf · These fossil traces were given the name Teredolites clavatus. The logs on which

Real-time Event Processing using LINQ for Logs & Traces

On the Utility of Anonymized Flow Traces for Anomaly Detection

LOGS...............THE BACK LOGS

Large Scale Anomaly Detection in Data Center Logs and Metrics · Large Scale Anomaly Detection in Data Center Logs and Metrics Anomaly Detection in Metrics Discord Discovery Improvements

Betriebsanleitung / Instructions COMpact 5200/5200R/5500R · Centralino per telefonia analogica/ISDN/via Internet ... gemeint, sondern auch Service-Logs von Routern oder Wireshark-Traces

Logs, Logs, Logs, from Natural Element Homes

FAQ – Traces online - Evira · EU TRACES Team – [email protected] 1/72 FAQ V2 - TRACES online TABLE OF CONTENTS FAQ V2 - TRACES ONLINE.....1

DeepLog: Anomaly Detection and Diagnosis from System Logs ...€¦ · DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning Min Du, Feifei Li, Guineng Zheng,

COTSEN Virtual Cotsen Textile Traces TEXTILE TRACES Global

DeepLog: Anomaly Detection and Diagnosis from … · DeepLog: Anomaly Detection and Diagnosis from System Logs ... log data is universally available in nearly all computer systems

DeepLog: Anomaly Detection and Diagnosis from System Logs ...lifeifei/papers/dl_ccs.pdf · DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning Min Du,

EXAD: A System for Explainable Anomaly Detection on Big ...yanlei/publications/icdm2018.pdf · in time-series data obtained from traces of Apache Spark jobs. Apache Spark has become

Anomaly Detection and Threat Hunting in Splunk UBA · 2020. 11. 6. · UBA detected ”sethc.exe” as an anomaly that had run once out of 7.5 million logs ”sethc.exe” is a valid

SOLUTIONS UNIFIED THREAT MANAGEMENT ET NEXT … · outils de reporting et d’analyse de logs embarqués, Rapports interactifs et personnalisables, envoi des traces en syslog, Agent

06 Resistivity Logs Induction Logs

Data Analysis, Machine Learning, Broand You!Pandas to Scikit-Learn Example: Anomaly Detection Bro DNS and HTTP logs Categorical and Numeric Data Clustering Isolation Forests Scikit-Learn

The Equatorial Anomaly - Miles Mathismilesmathis.com › equat.pdfThe Equatorial Anomaly by Miles Mathis First published August 18, 2013 The equatorial anomaly is an anomaly in the

Integrating Traffics with Network Device Logs for Anomaly ...downloads.hindawi.com/journals/scn/2019/5695021.pdf · SecurityandCommunicationNetworks ROUTE1 ROUTE2 SERVER1 Set traffic

Combining Gamma Ray Logs With Boring Logs

4.4 Resistivity logs and Induction logs

d2vkrkwbbxbylk.cloudfront.netTelegraf Plugin for Connext DDS enables monitoring architecture with DDS and InfluxDB Logs & Traces Grafana Web-based Interface for Monitoring and Analytics

APPENDIX D SOIL BORING LOGS / CORE LOGS

Tools for Traces, Logs, and Plug-Ins - cisco.com · Tools for Traces, Logs, and Plug-Ins • TraceandLogCentral,page1 • ImportCertificates,page2 • ViewTraceandLogCentralOptions,page2

Predicting the Black Swan - ITU...Jun 17, 2019  · •Network management data: CM, FM, PM, UE measurements •Cloud resource KPIs •Logs •Traces •Context data •Weather •(Road)

Mesh Options A Comparision of Service...A pluggable policy layer and configuration API supporting access controls, rate limits and quotas. Automatic metrics, logs, and traces for all

Anomaly detection using logs and metrics analysis for ... · operation logs and resource metrics data for near real-time anomaly detection. We propose a set of methods to map the

Traces, traces d'interactions, traces d'apprentissages : définitions, … · 2018-07-23 · numériques d’interactions et nous montrerons en quoi les trois usages des traces :

SpiceWorks Webinar: Whose logs, what logs, why logs

Lego for Microservices · Secrets Streamlined and secure access to application secrets Observability Automatic view logs, metrics, and traces across components and networked

Logs – Solve USING LOGS METHOD