machine learning deep dive - mdc 2020 · machine learning deep dive. agenda 2 1 about elastic 2...
TRANSCRIPT
![Page 1: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/1.jpg)
Thomas GrabowskiPrincipal Product Manager for Machine Learning
Machine Learning Deep Dive
![Page 2: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/2.jpg)
Agenda
2
1 About Elastic2 Machine Learning background3 Demo4 Q&A
![Page 3: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/3.jpg)
140M+ Product
Downloads
3,000+ Subscription Customers
85,000+ Community Members
Statistics since 2012, founding of Elastic
3
![Page 4: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/4.jpg)
2014
Mill
ions
of D
ownl
oads
40.
100.
201620152012 2013
Cumulative downloads of the Elastic Stack (Elasticsearch, Kibana, Beats, Logstash) and X-Pack
4
![Page 5: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/5.jpg)
Horizontal Scale Real-Time Data Availability
Flexible Data Model
Rapid Query Execution Sophisticated Query Language Schemaless
5
Today’s Developer Requirements
![Page 6: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/6.jpg)
6
Elastic Stack100% open source
No enterprise edition All new versions with 5.0
![Page 7: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/7.jpg)
7
X-PackSingle install
Extensions for the Elastic Stack Subscription pricing
Security
Alerting
Monitoring
Reporting
Graph
Machine Learning
![Page 8: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/8.jpg)
8
Elastic CloudHosted Elasticsearch & Kibana
Includes X-Pack features Starts at $45/mo
Available in AWS today and Google CloudMachine Learning and Azure platform available soon
Available as a private cloud/on-premise solution (Elastic Cloud Enterprise)
![Page 9: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/9.jpg)
75% of our customers use our products for multiple use cases
SEARCH
SECURITY
CUSTOM APPS
METRICS
OPERATIONAL ANALYTICS
LOG ANALYTICS
9
![Page 10: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/10.jpg)
![Page 11: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/11.jpg)
11
![Page 12: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/12.jpg)
12
Dashboards aren’t enough
![Page 13: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/13.jpg)
13
Rules Don’t Scale
• Where do you set the threshold?
• Who updates the rules?
• False positives are costly
![Page 14: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/14.jpg)
14
Search
Aggregations
Visualization
Machine Learning
Extracting useful, valuable information is hard
![Page 15: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/15.jpg)
Machine Learning1Algorithms and methods for data driven prediction, decision making, and modeling
Supervised Learning
Prediction based on examples of correct behavior
1Machine Learning Overview, Tommi Jaakkola, MIT
Unsupervised Learning
No explicit target, only data, goal to model/discover
Semi-supervised Learning
Supplement limited annotations with unsupervised learning
Active Learning
Learn to query the examples actually needed for learning
Transfer Learning
How to apply what you have learned from A to B
Reinforcement Learning
Learning to act, not just predict; goal to optimize the consequences of
actions
Other! …
![Page 16: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/16.jpg)
Machine Learning1Algorithms and methods for data driven prediction, decision making, and modeling
Supervised Learning
Prediction based on examples of correct behavior
1Machine Learning Overview, Tommi Jaakkola, MIT
Unsupervised Learning
No explicit target, only data, goal to model/discover
Semi-supervised Learning
Supplement limited annotations with unsupervised learning
Active Learning
Learn to query the examples actually needed for learning
Transfer Learning
How to apply what you have learned from A to B
Reinforcement Learning
Learning to act, not just predict; goal to optimize the consequences of
actions
Other! …
Time Series Anomaly Detection
![Page 17: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/17.jpg)
© Elasticsearch BV
Has my order rate dropped significantly?
![Page 18: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/18.jpg)
Has my order rate dropped significantly?
• Learn models from past behavior (training, modeling)
• Use models to predict future behavior (prediction)
• Use predictions to make decisions
Expected value @ 15:05 = 1859
Actual value @ 15:05 = 280
Probability = 0.0000174025
![Page 19: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/19.jpg)
19
Machine Learning Technical DebtAll the additional plumbing that is required for Machine Learning systems
From Google Paper: Scully, D., et al. “Machine learning - The high-interest credit card of technical debt.” (2014)
![Page 20: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/20.jpg)
20
It All Begins with DataDiscovering information in NGINX logs
68.75.44.178, 172.68.146.54, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/company_profile_cover_crop/public/1500x500_1_10.jpg?itok=RUgim2UQ&sc=297009042628d7de3f0eb50e807d29e4 HTTP/1.1" 200 92763 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36”
![Page 21: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/21.jpg)
21
68.75.44.178, 172.68.146.54, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/company_profile_cover_crop/public/1500x500_1_10.jpg?itok=RUgim2UQ&sc=297009042628d7de3f0eb50e807d29e4 HTTP/1.1" 200 92763 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"221.247.242.171, 162.158.166.51, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/company_profile_logo/public/company_logos/aaeaaqaaaaaaaawvaaaajdk3n2vkzme0lte0zjctngy3ms1inmm4lta4ntnhzwqymzvmoq.png?itok=H2B05xX0 HTTP/1.1" 200 9296 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"192.228.32.190, 108.162.246.21, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /jobs/24237/it-back-end HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://68.75.44.178, 172.68.146.54, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/company_profile_cover_crop/public/1500x500_1_10.jpg?itok=RUgim2UQ&sc=297009042628d7de3f0eb50e807d29e4 HTTP/1.1" 200 92763 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"221.247.242.171, 162.158.166.51, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/company_profile_logo/public/company_logos/aaeaaqaaaaaaaawvaaaajdk3n2vkzme0lte0zjctngy3ms1inmm4lta4ntnhzwqymzvmoq.png?itok=H2B05xX0 HTTP/1.1" 200 9296 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"192.228.32.190, 108.162.246.21, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /jobs/24237/it-back-end HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"137.56.184.63, 162.158.165.50, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/company_profile_cover/public/1500x500_1_10.jpg?itok=1cNqdGYK HTTP/1.1" 200 102268 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"92.222.165.172, 162.158.167.202, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "POST /jstats.php HTTP/1.0" 200 13 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" 68.75.44.178, 172.68.146.54, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/company_profile_cover_crop/public/1500x500_1_10.jpg?itok=RUgim2UQ&sc=297009042628d7de3f0eb50e807d29e4 HTTP/1.1" 200 92763 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"221.247.242.171, 162.158.166.51, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/company_profile_logo/public/company_logos/aaeaaqaaaaaaaawvaaaajdk3n2vkzme0lte0zjctngy3ms1inmm4lta4ntnhzwqymzvmoq.png?itok=H2B05xX0 HTTP/1.1" 200 9296 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
![Page 22: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/22.jpg)
22
Ingest, Enrich, Visualize, Analyze, Alert
Elasticsearch
X-pack
Master Nodes (3)
Ingest Nodes (X)
Data Nodes - Hot (X)
Data Nodes - Warm (X)
Beats
Log Files Metrics
Wire Data your(beat)
Filebeat Module
NGINX
Kibana
X-pack
Instances (X)
![Page 23: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/23.jpg)
23
Getting Started Resources
Try it out yourself: Download Elastic Stack & X-Pack
Get the product tour in Elastic’s Machine Learning webinar
Try Elastic’s Machine Learning video series: Single metric jobs Multi metric jobs Detect outliers in a population
Take Elastic’s Machine Learning training course
![Page 24: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/24.jpg)
Demo Time!
![Page 25: Machine Learning Deep Dive - MDC 2020 · Machine Learning Deep Dive. Agenda 2 1 About Elastic 2 Machine Learning background 3 Demo 4 Q&A. 140M ... s 40. 100. 2012 2013 2015 2016 Cumulative](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55cdd53bc5812231f5646/html5/thumbnails/25.jpg)
Questions?