infiniflux collector
TRANSCRIPT
InfiniFlux Collector
www.infiniflux.com
Table of Contents
What is Collector
Create and Start Collector
Setup Client - log aggregator
Setup Client - rsyslog
Setup Client - logstash
1
2
3
4
5
Setup Client with shell script6
Tag7
Add custom log8
What is Collector
3
InfiniFlux Collector is the process that collects various forms of log files from remote servers by using log collector and insert into InfiniFlux DB.
• Syslog
• Apache access
• Tomcat server
• Java, PHP, etc
• Application log
• InfiniFlux log
• Custom log
</>
</>
</>Send via TCP InfiniFluxAppend
Client
ifluxcollector
Create and Start Collector
4
InfiniFlux sever must be operated before creating and executing collector.
• Create and execute InfiniFlux DB
$ ifluxadmin -c Create InfiniFlux DB
$ ifluxadmin -u Execute InfiniFlux server
• Create and execute collector
$ ifluxcollector –c –p 5657 Create collector through 5657 port
$ ifluxcollector –u –p 5657 Execute collector through 5657 port
Default port number: 5657Stand by in order to receive TCP data through designated port number.
Setup Client - log aggregator
5
Currently InfiniFlux provides two types of log collectors; rsyslog and logstash.
- Rsyslog reads and sends log files, and immediately process when events occur.- Logstash reads and sends log files.
ifluxcollectorSend via TCP
Log Aggregator
File rsyslogRead
File logstashRead
Event rsyslogRaise
Setup Client - log aggregator
6
Below are the values to be set in the client file that sends logs.There are four required properties and two selected properties.
Collector host(HOST)
Collector port(PORT)
Table name(TABLE)
Tag name(TAG)
Path of log file(FILE_PATH)
Port that receives data(RECV_PORT)
• Required Property
• Selected Property
Setup Client - logstash
7
This is how to send logs to collector by using logstash.
• Verify versions of JavaIt is required to verify the version of Java since logstash can be operated on version of Java 7 or more.
$ java -version
• Checkout and download logstash• Check whether logstash is installed, and download it from the website below if not installed.
https://www.elastic.co/products/logstash
• Create a configuration file of logstash.Name a configuration file of “apache_access” as “apache_access.conf”.Detailed information on configuration file is described on the next page.
• Check if the configuration file is valid.$ logstash –f apache_access.conf –configtest
• Execute logstash.$ logstash –f apache_access.conf
Setup Client - logstash
8
This is the template and example of logstash. Set values of FILE_PATH, TABLE_NAME, TAG, HOST, and PORT to your environment.
input {file {
path => “FILE_PATH”}
}
filter {mutate {
add_field => {“table” => “TABLE_NAME”“tag” => “TAG”
}}
}output {
tcp {codec => line {
format => “[[table:%{[table]} tag:%{[tag]} message:%{[message]}]]”
}host => “HOST”port => “PORT”
}}
input {file {
path => “/var/log/httpd/access_log”}
}
filter {mutate {
add_field => {“table” => “apache_access_table”“tag” => “apache_access”
}}
}output {
tcp {codec => line {
format => “[[table:%{[table]} tag:%{[tag]} message:%{[message]}]]”
}host => “192.168.0.10”port => “5657”
}}
Template Example
Setup Client - rsyslog
9
This is how to set configuration file when log file is sent by using rsyslog.
• Rsyslog is basic syslog protocol so that, in general, most of Linux has it, but if not, it can be downloaded from the website below.
http://www.rsyslog.com/
• Create a configuration file for the log.Create the configuration file to “/etc/rsyslog.d” directory in the form of <name of configuration file>.conf.
• Re-execute rsyslog.Create a configuration file and re-execute rsyslog in order to reflect the newly created configuration file.
$ sudo service rsyslog restart
• Verify data entry.To verify whether the data is properly inserted, select data in the table by using ifluxsql.
Setup Client - rsyslog
10
This is the configuration file example of rsyslog.Create the file in “/etc/rsyslog.d/syslog.conf.”
$template TABLE_NAME_Format, “[[table:TABLE_NAME tag:TAG message:%msg%]]\n”
$ModLoad imfile$InputFileName FILE_PATH$InputFileTag TAG_file:$InputFileStateFile stat-TAG$InputFilePollInterval 1$InputRunFileMonitor
If $programname == ‘TAG_file’ then @@HOST:PORT;TABLE_NAME_Formatif $programname == 'TAG_file' then ~
$template syslog_table_Format, “[[table:syslog_table tag:syslog message:%msg%]]\n”
$ModLoad imfile$InputFileName /var/log/syslog$InputFileTag syslog_file:$InputFileStateFile stat-syslog$InputFilePollInterval 1$InputRunFileMonitor
If $programname == ‘syslog_file’ then @@localhost:5657;syslog_table_Formatif $programname == 'syslog_file' then ~
Template
Example
11
Shell scripts are provided for creating configuration files easily.
• Location of shell scripts$ cd $IFLUX_HOME/collector/samples/conf
• There are three shell script files.file-config-for-logstash.sh : script for setting up logstashfile-config-for-rsyslog.sh : script for setting up rsyslogsyslog-config-for-rsyslog.sh : script for setting up syslog file
• Create a configuration file by using scripts.$ sudo ./file-config-for-rsyslog.sh <COLLECTOR_HOST> <COLLECTOR_PORT> <TABLE_NAME> <TAG> <FILE_PATH>$ sh ./file-config-for-rsyslog.sh 192.168.0.10 5657 apache_access_table apache_access /var/log/httpd/apache_access
• Verify the created configuration file.$ cd /etc/rsyslog.dCreated file: 192.168.0.10:5657_apache_access_table_from_file.conf
Setup Client with Shell Script
Tag
12
Tag type Description
syslog It is able to receive standard format of syslog which is created as a file.
apache_access It receives combined format of apache access log.
infinifluxIt receives trace log file of InfiniFlux.Trace logs can be found in the “$IFLUX_HOME/trc” directory.
JSONIt dynamically process data in the form of JSON.Data must be inserted in the form of {“key”:“value”}, and all the column will be created in the VARCHAR format if a table is not created beforehand.
UnparseWhen “unparsed” logs were received, it inserts the log in a VARCHAR column rather than parsing it.
• Supported tag types
• To add a tag
• All the tags will be saved in the “$IFLUX_HOME/collector/regex” directory as files.
• The same name should be given to a file and tag, and need a declaration for a each column and regular expression should be
defined.
• To reflect modified information of tag file, re-execute the collector when contents of the file are changed.
Need to specified tags in order to separate log types, and parse log data that were inserted in accordance with defined tags.
When there is a new kind of log entry, a new tag must be added as well.
Add Custom Log
13
Item Description
REGEX_NOIt determines nth token among the tokens that were specified in the regular expression.To check the number of token, use ifluxregex utility.
NAME When there is no table to be inserted, a new one is created. “NAME” will name the new table.
TYPE It specifies data type of table column.
SIZEIt specifies the size of table column.For the case of VARCHAR type, it needs to be specified precisely.
DATA_FORMAT If a column type set to “datetime”, it specifies the format of datetime of input log.
USE_INDEXIt specifies whether to create an index for the column. If it is 1, create an index.For the formats of VARCHAR and text, it creates keyword index, and bitmap index for the rest.Automatically, <table name>_idx_<field name> name will be created.
• COL_LIST
• REGEX
• Write regular expression for the log.
• Each log will be inserted as a sentence unit except a new line.
When there is a new kind of log is received, a new tag file for the log should be added in the “$IFLUX_HOME/collector/regex” directory. A tag file is consisted of COL_LIST and REGEX.
14
COL_LIST= ((
REGEX_NO = 0NAME = tmTYPE = datetimeSIZE = 8DATE_FORMAT="%b %d %H:%M:%S"),
(REGEX_NO = 4NAME = hostTYPE = varcharSIZE = 128USE_INDEX = 1),
(REGEX_NO = 5NAME = msgTYPE = varcharSIZE = 512USE_INDEX = 1)
)REGEX="(([a-zA-Z]+)\s+([0-9]+)\s+([0-9:]*))\s(\S*)\s+((?:[^\0])*)"
Add Custom Log This is the contents of syslog tag file.Tag file can be found in the “$IFLUX_HOME/collector/regex/syslog”.
