red hat summit 2017 - lt107508 - better managing your red hat footprint with log aggregation
TRANSCRIPT
![Page 1: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/1.jpg)
LOG AGGREGATION
To better manage your Red Hat footprint
Miguel Pérez ColinoStrategic Design Team - ISBU2017-05-03
@mmmmmmpc
![Page 2: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/2.jpg)
AgendaManaging your Red Hat footprint with Log Aggregation
● The Situation● The Challenge ● The Solution
![Page 3: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/3.jpg)
THE SITUATION
![Page 4: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/4.jpg)
Cloud DeploymentsThey do really scale ...
https://www.cncf.io/blog/2016/08/23/deploying-1000-nodes-of-openshift-on-the-cncf-cluster-part-1/
● Higher scalability● More workloads per physical
machine (multi-tenant)● Network and Storage also
Software Defined● Containers and Microservices
providing more granularity
![Page 5: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/5.jpg)
Cloud DeploymentsAct as one single thing …
… and need to be managed and operated as one
Source: https://commons.wikimedia.org/wiki/File:Auklet_flock_Shumagins_1986.jpg
![Page 6: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/6.jpg)
THE CHALLENGE
![Page 7: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/7.jpg)
Data (What)Data + Information flow in Log Aggregation
ProcessIngest StoreCollect Query ViewGenerate
Derived from: http://www.dataintensive.info/
![Page 8: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/8.jpg)
Personas (Who)That can use Log Aggregation
Log Aggregation
MonitoringProvides Events, Consumes Logs
Cloud OpsRoot Cause Analysis
DeveloperApp Analysis & Debug
Security EngineerSec Analysis, Audits
User / MarketingAccess to stats
Service DesignerIT Manager
Access to aggregated data, i.e. SLA, usage
![Page 9: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/9.jpg)
Personas (Motivation)That need Log Aggregation
Cloud Ops (Apps)
“I want to proactively know about active or potential degradation of service”
Cloud Ops (OpenStack)
“User reports that their VM request failed and returned error”
Developer (OpenShift)
“My recent commit resulted in Jenkins test failure”
“Application (multi-tiered) launched from CloudForms returns error”
Cloud Suite User
![Page 10: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/10.jpg)
Situational Awareness (Why)Or the need of it!
Source: https://en.wikipedia.org/wiki/Situation_awareness
![Page 11: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/11.jpg)
THE SOLUTION
![Page 12: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/12.jpg)
ArchitectureProposed General Architecture
Real TimeAnalytics
and Response
Host
Bus
N N N
Archive
Data Store
GeneralVisualization
MC
MC
Storage
Legend
MC
N
Message Client
Normalizer
C
C
C Collector
Slide Credit: Tushar Katarki [@tkatarki]
![Page 13: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/13.jpg)
ImplementationIntroduction to EFK
Kibana ElasticSearch Log Source Fluentd
User Interface for:● Search● Graph● Dashboard
Index and store data and metadata
making search fast and reliable
● Parsing● Filtering● Enriching● Deleting● Output
Buffering
● TCP/UDP● HTTP● File: Text● Stdout: CSV,
JSON, MessagePack
● syslog/journal
Slide Credit: Tushar Katarki [@tkatarki]
![Page 14: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/14.jpg)
Current StatusBeing delivered and supported
OpenShift Container Platform 3.5
● Full EFK stack provided as containers
OpenStack Platform 10
● Fluentd as log collector
Red Hat Virtualization
● Coming Soon!
Log files
Journal Fluentd
KuberentesServices
Syslog
Master Nodes
Elasticsearch
Kibana
...Application Nodes
Log files
Journal Fluentd
App inside container
Syslog
Infra Nodes
Elasticsearch
Kibana
host logs
App inside container
Elasticsearch
Curator
Multi-Tenant Access
Diagram Credit: Tushar Katarki [@tkatarki]
![Page 15: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/15.jpg)
BEYOND ...
![Page 16: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/16.jpg)
Common Data ModelTo ensure integration and interoperability
What Is It?
● A Data Model for Logs (and other data) to identify and tag data (i.e. log fields)
Why?
● Alignment/Correlation with different RH products● Improved maintainability of Data● Better presentation/data consumption● Enables 3rd party ecosystem● Facilitates deep learning analysis of data
Ingestion pipeline
Consumption pipeline
Indexing and Storage
![Page 17: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/17.jpg)
Common Data ModelExample ...
Data extracted:
● Container name● Pod name● Namespace name● Docker container ID
K8S data queried:
● Pod UID● Pod labels● Pod host● Namespace UID.
All merged into output log in JSON Format
Images Credit: Anton Sherkhonov [@peatz]
CDM
A → 1B → 2C → 3
![Page 18: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/18.jpg)
User ExperiencePrototyping and validating dashboards for users
Slide Credits: Peter Portante & Vince Conzola
![Page 19: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/19.jpg)
Exploring different approachesPrototyping with alternative toolsets with partners
Slide Credits: Luca Rosellini (Keedio)
![Page 20: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/20.jpg)
ACTION!
![Page 21: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/21.jpg)
How are you doing it?Please, provide your feedback ...
http://bit.ly/log-aggregation
![Page 22: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/22.jpg)
THANK YOUplus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews
![Page 23: Red Hat Summit 2017 - LT107508 - Better Managing your Red Hat footprint with log aggregation](https://reader031.vdocuments.mx/reader031/viewer/2022030318/5a65bbfe7f8b9a0a5f8b4abf/html5/thumbnails/23.jpg)