logstash family introduction
DESCRIPTION
Logstash IntroductionTRANSCRIPT
![Page 1: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/1.jpg)
Logstash Family Introduc4on
Owen
![Page 2: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/2.jpg)
What is a log
• Oxford Dic4onary – a thick piece of wood that is cut from or has fallen from a tree
– (also logbook) an official record of events during a par4cular period of 4me, especially a journey on a ship
• 4me + data
![Page 3: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/3.jpg)
In theory, life cycle of log
Record
Transmit
Analyze Store
Delete
![Page 4: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/4.jpg)
In design, life cycle of log
Record
Transmit
Store
Delete
![Page 5: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/5.jpg)
In fact, life cycle of log
Record
Delete
![Page 6: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/6.jpg)
Problems
• Logging to a database or filesysytem • Logging has placed a load on the database and filesystem
• Mul4ple log formats • No easy way to search logs • No easy method to gather sta4s4cs
![Page 7: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/7.jpg)
Find the logs of 16 computers 6 months ago ?
![Page 8: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/8.jpg)
Why use Logstash?
• A lot choices! �• But we want a free & high-integrality &
easy to use solution • splunk (finding your faults, just like mom) • facebookarchive/scribe (2682 ★) • Graylog2(Server+WUI 1683 ★) • fluentd (2038 ★) • logstash (2689 ★)
![Page 9: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/9.jpg)
logstash and other things
hRps://www.youtube.com/watch?v=RuUFnog29M4
![Page 10: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/10.jpg)
Logstash
• Open Source, Apache Licence • WriRen in JRuby, Runs on JVM • Plugins easily wriRen in Ruby • Process mul4ple format ( input, output ) • Logstash Family! ( Elas4cSearch , Kibana)
![Page 11: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/11.jpg)
LogStash Family architecture
![Page 12: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/12.jpg)
Elas4cSearch
• A response to the claim : “Search is hard” • Powerful indexing & search tool • search & index data available Rescully as JSON over HTTP
![Page 13: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/13.jpg)
Kibana
![Page 14: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/14.jpg)
All-‐in-‐one!
![Page 15: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/15.jpg)
How logstash works?
• logstash process events, not (only) loglines!
• “The logstash agent is a processing pipeline with 3 stages: – inputs -> filters -> outputs.” – separate threads
• “Inputs generate events, filters modify them, outputs ship them elsewhere.”
• -- [the life of an event in logstash] �
![Page 16: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/16.jpg)
In my thinking, Event Life Cycle
Input
filter output
![Page 17: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/17.jpg)
In fact, Event Life Cycle
event (Input -‐> output)
event -‐-‐-‐-‐-‐-‐-‐-‐-‐ input filter output
![Page 18: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/18.jpg)
Logstash is a wooden tube
Input
Input
Input
filter
output
codec
filter filter
output
output
![Page 19: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/19.jpg)
Logstash plugins Workflow
• inputs – How events get into LogStash.
• codecs – convert an incoming format into an internal representa4on
• filters – processing ac4ons on events : modify events or drop events
• outputs – How output events from LogStash
![Page 20: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/20.jpg)
Logstash plugins
![Page 21: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/21.jpg)
What is an event!?
• A @4mestamp (ISO 8601 4mestamp) • A messsage field ( data ) • A @version • host ( the host of sender) • type( syslog, irc, etc)
![Page 22: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/22.jpg)
Exersice: Hello Word!
java -‐jar logstash-‐1.1.12-‐flatjar.jar agent -‐f hello.conf
java -‐jar logstash.jar agent -‐f hello.conf
![Page 23: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/23.jpg)
Input
• tcp • udp • unix • file • syslog • redis • logstash-‐fowarder(former Lumberjack)
![Page 24: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/24.jpg)
Codecs
• plain • json • rubydebug • mul4line
![Page 25: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/25.jpg)
Outputs
• mongodb • elas4cSearch • email • file • jira
![Page 26: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/26.jpg)
Exercise: Mul4ple input & output
![Page 27: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/27.jpg)
logstash-‐forwader
• ♫ I'm a lumberjack and I'm ok! I sleep when idle, then I ship logs all day! I parse your logs, I eat the JVM agent for lunch! ♫
• WriRen in Go • lumberjack is reserved for protocol • Resource Usage Concerns • Need an SSL CA to verify the server
![Page 28: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/28.jpg)
lumberjack
• Encryp4on & Authen4ca4on (TLS) • Compression ( reduce bandwidth) • Sequence & ack behavior like TCP • Low latency • Reliable Aplica4on-‐Level message transport
![Page 29: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/29.jpg)
Forwarder Sample
![Page 30: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/30.jpg)
Filters
• date • grok • drop • geoIP • mutate • mul4line
![Page 31: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/31.jpg)
Exercise: Parse Data
![Page 32: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/32.jpg)
filter config
![Page 33: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/33.jpg)
powerful grok • Parse arbitrary text and structure it. • The syntax for a grok paRern is – %{SYNTAX:SEMANTIC}
• 55.3.244.1 GET /index.html 15824 – %{IP:client} – %{WORD:method} – %{URIPATHPARAM:request} – %{NUMBER:bytes}
• hRps://github.com/elas4csearch/logstash/blob/v1.4.2/paRerns/grok-‐paRerns
![Page 34: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/34.jpg)
grok sample
![Page 35: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/35.jpg)
drop
![Page 36: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/36.jpg)
mutate
• Muta4ons on fields. – rename – remove – replace – join – split – upper – lower
![Page 37: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/37.jpg)
mul4line
• Codecs & filter
![Page 38: Logstash family introduction](https://reader033.vdocuments.mx/reader033/viewer/2022042518/554bb6d5b4c90530298b4827/html5/thumbnails/38.jpg)
Reference
• hRps://www.digitalocean.com/community/tutorials/how-‐to-‐use-‐logstash-‐and-‐kibana-‐to-‐centralize-‐and-‐visualize-‐logs-‐on-‐ubuntu-‐14-‐04
• hRp://www.vmdoh.com/blog/centralizing-‐logs-‐lumberjack-‐logstash-‐and-‐elas4csearch
• hRp://jpmens.net/2012/08/09/i-‐grok-‐how-‐to-‐mutate-‐a-‐file-‐with-‐logstash/
• hRp://gleenders.blogspot.tw/2014/02/logstash-‐glassfish.html