the devops toolbox: open source log analytics
TRANSCRIPT
![Page 1: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/1.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
The DevOps Toolbox: Open Source Log Analytics
Tomer Levy & Asaf YigalCofounders, Logz.io
![Page 2: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/2.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
Is anyone using ELK to process logs?
![Page 3: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/3.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
Is anyone using the public cloud? AWS?
![Page 4: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/4.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
Is anyone doing kite-surfing?
Windsurfing?
Paragliding?
Sailing?
![Page 7: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/7.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
And this is the view from the office
![Page 8: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/8.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
When should we leave everything and go
kitesurfing?
This is our challenge
![Page 12: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/12.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
Wind analytics — next items
• Alerts
• Wind forecast combined with real wind
• Wind predictions!
![Page 13: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/13.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
Who are we?
• Logz.io Insights: Behavioural intelligence to pinpoint
what actually matters in logs
• ELK++ as a Service – Infinitely scalable
– Secured
– Highly Available
– Additional Features (alerts, role-based access)
![Page 14: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/14.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
ELK implementation —creating the right architecture
![Page 15: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/15.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
ELK implementation —creating the right architecture
Curator
Curator
3x Master Nodes + 1 data
![Page 16: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/16.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
ELK implementation —creating the right architecture
Curator
3x Master Nodes + 1 data
Index Failures Handler
![Page 17: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/17.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
ELK basic implementation —find the weak spots
AZ-1
AZ-2
ELBLB
![Page 18: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/18.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
• Grok – parse logs to extract the relevant fields…
• Try our blog for some help on grok/plugins etc’
• blog.logz.io
ELK basic implementation — configuration
![Page 19: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/19.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
1. Use Elasticsearch AWS Plugin
2. EBS are challenging for big environment Use PIOPS if you can afford
3. Don’t run AWS cluster on the same AZ (but don’t run them on different zones!) - use Shard allocation awareness
4. S3 Snapshots are cool! Things tend to break…
Elasticsearch basic implementation —configuration
![Page 20: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/20.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
Let’s see a high-level view of how we process logs
Demo
![Page 21: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/21.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
Want to try our product?
• Email us with questions:
• Visit our website for more information!
![Page 22: The DevOps Toolbox: Open Source Log Analytics](https://reader034.vdocuments.mx/reader034/viewer/2022042701/55a985b11a28ab250a8b46c0/html5/thumbnails/22.jpg)
logz.io | @logzio | @tomerlevy | @asafyigal
We’re hiring — big time!
• Elasticsearch experts
• Java developers
• Machine-learning experts
• See our job listings page