log forwarding at scale - scale 18x | 18x...about fluentd more than 600 plugins available pluggable...
TRANSCRIPT
![Page 2: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/2.jpg)
/about
● Open Source Engineer at Treasure Data
● Repositories / Projects
○ github.com/edsiper○ fluentbit.io○ duda.io○ monkey-project.com
![Page 3: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/3.jpg)
“Logging is Simple”
![Page 4: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/4.jpg)
“Logging is Simple”
![Page 5: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/5.jpg)
“Logging is Simple”
Logs App Analysis
Logging exists because of Analysis needs
![Page 6: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/6.jpg)
Before Analysis
Logs Database
Someone have to do some work
?
![Page 7: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/7.jpg)
In a galaxy not so far away...
Logs Database
![Page 8: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/8.jpg)
Analysis
![Page 9: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/9.jpg)
Internally, Logging is not Simple
![Page 10: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/10.jpg)
Scale Logging Requires Understanding
![Page 11: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/11.jpg)
Logging Pipeline
![Page 12: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/12.jpg)
Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
![Page 13: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/13.jpg)
Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
● Log files● Journald● TCP
![Page 14: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/14.jpg)
Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Format / Structure:
● JSON● Regex / Named Capture
![Page 15: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/15.jpg)
Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Alter content: Grep, Exclude, Metadata
![Page 16: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/16.jpg)
Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Memory / Filesystem
![Page 17: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/17.jpg)
Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
OthersDeliver buffers to N destinations
Logging Pipeline
![Page 18: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/18.jpg)
Logging Pipeline
Log Messages Parse Filter Buffer
Routing
Elasticsearch
InfluxDB
Others
Logging Pipeline
![Page 19: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/19.jpg)
Logging Pipeline
● How to deal with the Logging Pipeline ?
● Is there any solution around ?
![Page 20: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/20.jpg)
![Page 21: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/21.jpg)
About Fluentd
● Created by
● Now hosted at
![Page 22: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/22.jpg)
About Fluentd
● More than 600 plugins available
● Pluggable Architecture
● Built-in Reliability
● Full integration with Docker and Kubernetes
● Written in Ruby + C
![Page 23: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/23.jpg)
Fluentd Modes
● Log Forwarder
● Log Aggregator
![Page 24: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/24.jpg)
Log Aggregator = (Forwarder + Buffering Capabilities)
![Page 25: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/25.jpg)
Edge Nodes / Forward to Aggregators
App
Node 1
DB
App DB
![Page 26: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/26.jpg)
Edge Nodes & Costs
![Page 27: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/27.jpg)
Edge Nodes & Costs
● Fluentd requires ~40MB as minimum
● Deploying a few hundred could be expensive
● Can we make Forward cheaper ?
![Page 28: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/28.jpg)
Forwarder & Aggregator
Log Forwarder Log Aggregator
![Page 29: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/29.jpg)
![Page 30: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/30.jpg)
About Fluentd
● Written in C
● Pluggable Architecture
● Built-in Reliability
● Event Driven - Async I/O
![Page 31: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/31.jpg)
Why Fluent Bit as a Forwarder
● Features
○ Input, Filter and Output Plugins
○ Built-in parsing support
○ Minimum memory required 450KB
![Page 32: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/32.jpg)
Edge Nodes / Forward to Aggregators
![Page 33: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/33.jpg)
Cheap Forwarding
![Page 34: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/34.jpg)
Cloud Native Features
● Docker & Kubernetes Support
● Buffering fully controled○ pause() / resume() for input plugins
● Easy to containerize○ Small memory footprint○ No dependencies (all are built-in)
![Page 35: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/35.jpg)
Hands on!
DEMO #1
Unstructured vs Structured data
![Page 36: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/36.jpg)
Unstructured v/s Structured
● Why
○ Structured data have a schema
○ Easy to convert to different representations
○ It can be filtered
![Page 37: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/37.jpg)
Hands on!
DEMO #2
Process Docker Logs
![Page 38: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/38.jpg)
Kubernetes use case
● Applications runs in Containers
● Containers runs in a POD
● Multiple PODs can exists in a Node
● How to solve logging ?
![Page 39: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/39.jpg)
Hands on!
DEMO #3Kubernetes: parse logs and append Metadata
API Server Nodes
![Page 40: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/40.jpg)
Kubernetes
Metadata Support Status
The new kubernetes filter takes care of the following metadata handling:
○ Local data: POD Name, Namespace, Container Name and Container ID.
○ Remote (API Server): Labels and Annotations
![Page 41: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/41.jpg)
Fluent Bit, what else ?
Networking and Co-routines
Easier implementation of output plugins that interact with networking operations like socket(), connect(), read(), write(), etc.
Fluent Bit provides non-blocking networking API that uses the event-loop with co-routines to implement:
● Network I/O● TLS/SSL usage● HTTP Client
![Page 42: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/42.jpg)
Kubernetes DaemonSet
Github Repository
● https://github.com/fluent/fluent-bit-kubernetes-daemonset
Docker Image (ubuntu-slim)
● quay.io/fluent/fluent-bit-kubernetes-daemonset
![Page 43: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/43.jpg)
Roadmap
Next Release v0.11 (March 2017)
● Kubernetes support (filter_kubernetes)● Parsers & Filters● Memory optimizations
Release v0.12 (May 2017)
● in_tail + Multiline support● Monitoring - re-enable HTTP service end-point: memory, records
flow, others.
![Page 44: Log Forwarding at Scale - SCALE 18x | 18x...About Fluentd More than 600 plugins available Pluggable Architecture Built-in Reliability Full integration with Docker and Kubernetes Written](https://reader030.vdocuments.mx/reader030/viewer/2022040608/5ec55e99419eb03a8221966c/html5/thumbnails/44.jpg)
Thanks!
Project information
● Web site Fluentbit.io● Documentation http://fluentbit.io/documentation/● Github http://github.com/fluent/fluent-bit
Contact
● Slack http://slack.fluentd.org (fluent-bit channel)● Twitter @fluentbit