atmosphere 2014: centralized log management based on logstash and kibana - case study - dariusz...
DESCRIPTION
Nowadays cloud enviroments are primary platform for applications. We no longer have multipurpose machines, rather multiple smaller virtual servers with dedicated roles. Therefore there is a need to have one place where we can manage applications and system logs. I wish to share my experience gained while building centralized log managment system using Nxlog, Logstash and Kibana. With that tools we are building cost effective and scalable log managment platform. Dariusz Eliasz - Works in Allegro Group as a Solution Architect and is responsible for organizing cooperation with infrastructure teams, also leads some of the infrastructure projects. Earlier as an Expert System Administratorhe was related with building and maintaining the infrastructure shared services (i.e. image hosting platform) within Allegro Group.TRANSCRIPT
Centralized log management based on
Logstash and Kibana - case study
Dariusz Eliasz
20.05.2014 Atmosphere Conference
• What’s the problem ?
• Solutons
• Transport format
• Architecture
– Sender
– Log router
– Log collector
– Full text search engine
– GUI
• Use case
Agenda
What’s the problem ??
Solutons ??
Transport format - syslog
• RFC3164 (BSD syslog )
• limited size - 1kB
• format of a syslog message:
– PRIORITY (calculated from severity and facility)
– HEADER (tmestamp + hostname or IP)
– MSG (tag + content)
<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8
Transport format - json
• JavaScript Object Notaton
• lightweight text-data interchange format
• language independent
• self-describing
Transport format - json
{
"LogType": "access_log",
"Vhost": "atmosphere-conference.com",
"HtpsOn": "false",
"Xrealip": "1.2.3.5",
"Clientp": "91.17.13.28",
"UserAgent": "Mozilla/4.0 (compatble; MSIE 6.0; Windows NT 5.1)",
}
Architecture
Sender
Sender
• nxlog (htp://nxlog-ce.sourceforge.net/)
• multple input types:
– tcp socket
– udp socket
– fle input
– unx socket
• multple parser types:
– bsd syslog
– json
Sender
Good practce:
• make as much as possible processing on sender site, eg: apache access logs in
json format
• automate confguraton management
Log router
Log router
nxlog
nxlog logstash
logstash
redis
redissyslog-ng
syslog-ng redis
Log router
### JSON PARSER
parser p_json {
json-parser (prefx("_json."));
};
### FILTERS
flter f_someflter {
("${_json.SomeJsonField}” == ”abc.com”)
};
### INPUTS & OUTPUTS
### LOG PATHS
log {
source(s_network_json);
destnaton(d_udp_logstash);
destnaton(d_tcp_hadoop);
#fags(fow-control); # disabled to separate destnatons
};
Log router
Good practce:
• good separaton of destnatons
• calculate enough redis size – it’s yor bufer
• batch events writes to redis
Log collector
Log collector
• Logstash htp://logstash.net/
• collectng, parsing and storing logs tool
• plugins:
Inputs
• fle
• gelf
• tcp
• log4j
• redis
• varnishlog
Codecs
• json
• line
• msgpack
• netlow
• multline
Filters
• grok
• alter
• cidr
• geoip
• grep
• mutate
Outputs
• elastcsearch
• graphite
• jira
• tcp
• zeromq
• zabbix
Log collector
Good practce:
• keep up2date version of java & logstash
• use batch & multthread read from redis
• read logs
• bulk writes to elastcsearch
Full text search engine
• Elastcsearch htp://www.elastcsearch.org/
• distributed, real-tme search and analytcs engine
• store documents as a JSON
• high availability
• schema free
• index mult-tenancy
• on top of Lucene
Full text search engine
• every index is replicated
• every index sharded
• index parttoning – tme based
• data retenton – tme based
Full text search engine
Good practce:
• half memory for ES (<30GB), half for system cache
• bootstrap.mlockall: true
• gateway.recover_afer_nodes
• indices.felddata.cache.size
• authorizaton via proxy
• curator
• Marvel plugin
GUI
• Kibana 3 htp://www.elastcsearch.org/overview/kibana/
• search, graph & analyze logs
• JavaScript based (AngularJS)
• only simple htp server needed
Q&A
Sources
Images:
htp://www.datalife7.com/2014/01/einsteins-secret-to-problem-solving-and.html
htp://www.formengifs.com/victorinox-swiss-army-swiss-champ-multtool-knife/
htp://www.slashgear.com/google-data-center-hd-photos-hit-where-the-internet-lives-gallery-17252451/