bots, outliers and outages… · and outliers in real time. 3 what’s elastic? 4 search. 5 search...
TRANSCRIPT
Matteo RebeschiniSolutions Architect @ [email protected]
2018 Phoenix Data Conference
Bots, Outliers and Outages… Do you know what's lurking in your data?
Abstract
With the mass amounts of data that are being ingested daily it is nearly impossible by traditional means to understand what is hidden in your data. How do you separate the ordinary from the un-ordinary in a timely fashion?
Unsupervised machine learning on time series data enables real-time discovery of those interesting and possibly costly data anomalies. Matteo will describe, build and run several types of machine learning jobs in Elasticsearch that can detect and alert on these anomalies and outliers in real time.
6
100,000+ Meetup
Members
250M+Product Downloads
5000+Subscription Customers
Statistics since 2012, founding of Elastic
How Elastic Stack Components Work Together
Kibana
Beats
LogstashElasticsearch
LogsMetricsPackets...
Datastore Web APIs
11 All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement.
ES-Hadoop
Deployment in the Enterprise
Data store Web APIs
Social Sensors
Kafka
Redis
MessagingQueue
Logstash
Workers (2+)
LDAP
Authentication
AD
Notification
SSO
Kibana
Custom UI
Elasticsearch Clients
Elasticsearch
Master (3)
Ingest (X)
Data – Hot (X)
Data – Warm (X)
Machine Learning (2+)
Coordinating (X)
Alerting (X)
HEARTBEAT
Beats
FILEBEAT
METRICBEAT
PACKETBEAT
WINGLOGBEAT
AUDITBEAT
SAML
Elastic Stack
Store, Search, & Analyze
Visualize & Manage
Ingest
Metrics
Logging
APM
SiteSearch
Application Search
BusinessAnalytics
EnterpriseSearch
SecurityAnalytics
Future Solutions
SaaS
Elastic Cloud
Self Managed
Elastic CloudEnterprise Standalone
Deployment
● Algorithms that
○ Learn from Data
○ Using Statistical Techniques
○ Without Explicit Programming
What’s Machine Learning?
Elastic Machine Learning Scope
Anomaly DetectionAutonomous cars Voice Recognition
Fraud detection
Speech RecognitionLanguage Translation Entity Resolution
Predictive Medicine
Learn to Rank
RecommendationsImage Classification
Elastic Machine Learning Scope
Method
Unsupervised
Data
Time Series
Supervised
Panel
Cross SectionProblem
Classification
Regression
Elastic ML
Anomaly Detection
Challenges that Anomaly Detection Solves
18
● IT Operations○ How do I know my systems are behaving normally?○ Where to set thresholds for good alerting?○ How to find the root cause of problems when I don’t know what to look for?
● IT Security○ Do I have systems that are compromised with malware?○ Which users could be an insider threat?
● IoT / SCADA / Other○ Is my factory working normally?○ What do I do with thousands of time-series data points?○ Which traffic incidents are causing the most delay?
19
Detecting (noteworthy) anomalies is hard!
• Data is complex, high dimensional, fast moving• Human inspection is not practical• Easy to miss things
Visual inspection is not practical
Where’s the anomaly?
20
Detecting (noteworthy) anomalies is hard!
• Defining “normal” via static thresholds is hard• Rules don’t evolve with data / infrastructure• Rules can be bypassed
Rule-based alerts are
insufficient
What’s the right threshold?
Elastic Machine Learning• Uses unsupervised machine learning techniques to ▪ Learn what’s “normal” by modeling historic behavior▪ Detect anomalies when data falls outside expected bounds ▪ Use models to predict future behavior (prediction)▪ Use predictions to make decisions
21
Elastic Machine Learning
• Unsupervised techniques - no manual training / input needed• Evolves with the data - “online” model learns continuously• Influencer detection - accelerates root cause identification
22
Detect anomalies of different types
• Time series - single / multiple metric(s)
• Outliers in population (using entity profiling)
• Rare / unusual events
23
Technology Behind Elastic ML
The technology behind Elastic’s Machine Learning (ML) is a bespoke amalgamation of different machine learning methods and techniques that brings sophisticated real-time automated anomaly detection for time series data to users that may not be able to employ data science on their own.
Using techniques such as clustering, various types of time-series decomposition, Bayesian distribution modeling, and correlation analysis, Elastic Machine Learning takes a 100% unsupervised machine learning approach to statistically model data’s time-based characteristics merely by observing its historical behavior.
Behind the scenes, a dynamic, ever-learning statistical model is built and stored, per unique time-series. Real-time data being analyzed both contributes to this model’s maturity and is assessed against the model so that it can be judged for its level of unusualness. If the data’s behavior is seen as being within the low probability range, an anomaly record is created, persisted, and scored proportional to the probability. This score is normalized on a user-friendly dynamic scale between 0 and 100, where 100 is the most unusual thing ever detected for the data set.