MT-Logging with WSO2 BAM

Amani SoysaSenior Software Engineer

How it Works??

Logs are published to BAM via Log4JAppender

Published Logs are Stored in Cassandra (per tenant per day)

When the day is over Logs are published to HDFS file system daily using BAM analyzers

To view Logs :-Application Logs – Log events on web applications/services filter by the application name System Logs – View the system log of each product {With filtering categories}

Log Viewers get logs from cassandra <<real time log>> and HDFS file system <<Archive logs>>

Log Publisher - Log4JAppender

Setting Up LogEventAppender

log4j.rootLogger=INFO, LOGEVENT

# LOGEVENT is set to be a LogEventAppender using a PatternLayout to send logs to LOGEVENT log4j.appender.LOGEVENT=org.wso2.carbon.logging.appender.LogEventAppenderlog4j.appender.LOGEVENT.url=tcp://localhost:7611log4j.appender.LOGEVENT.layout=org.wso2.carbon.utils.logging.TenantAwarePatternLayoutlog4j.appender.LOGEVENT.columnList=%T,%S,%A,%d,%c,%p,%m,%H,%I,%Stacktracelog4j.appender.LOGEVENT.userName=adminlog4j.appender.LOGEVENT.password=adminlog4j.appender.LOGEVENT.processingLimit=1000log4j.appender.LOGEVENT.maxTolerableConsecutiveFailure=20

streamId = dataPublisher.defineStream("{" + "'name':'log" + "." + tenantId + "."+ severName + "." + currDateStr + "'," + " 'version':'1.0.0',"+ " 'nickName': 'Logs'," + " 'description': 'Logging Event',"+ " 'metaData':[" + " {'name':'clientType','type':'STRING'}" + " ],"+ " 'payloadData':[" + " {'name':'tenantID','type':'STRING'},"+ " {'name':'serverName','type':'STRING'},"+ " {'name':'appName','type':'STRING'},"+ " {'name':'logTime','type':'LONG'},"+ " {'name':'priority','type':'STRING'},"+ " {'name':'message','type':'STRING'},"+ " {'name':'logger','type':'STRING'},"+ " {'name':'ip','type':'STRING'},"+ " {'name':'instance','type':'STRING'},"+ " {'name':'stacktrace','type':'STRING'}" + " ]" + "}");

Stream Definition

Log Analyzer

Logging-Config.xml

Basic Information

<isDataFromCassandra>true</isDataFromCassandra>

Information Regarding Cassandra

<cassandraHost>localhost:9160</cassandraHost> <userName>admin</userName> <password>admin</password>

Information Regarding HDFS Server

<archivedHost>hdfs://localhost:9000</archivedHost> <archivedHDFSPath>/stratos/logs</archivedHDFSPath>

Logging Summarizer

Archive logs daily at a given time and store in hdfs file system

summarizer-config.xml

Basic Information

<isDataFromCassandra>true</isDataFromCassandra><deleteColFamily>false</deleteColFamily>

Information Regarding Cassandra <cassandraHost>localhost:9160</cassandraHost> <userName>admin</userName> <password>admin</password>Information Regarding BAM <publisherURL>localhost:7611</publisherURL> <userName>admin</userName> <password>admin</password> <cronExpression>0 0 1 ? * * *</cronExpression>

Information Regarding HDFS Server

<archivedHost>hdfs://localhost:9000</archivedHost> <archivedLogLocation>/stratos/logs</archivedLogLocation>

Advantages

Asynchronous and None Blocking Data publishing Receives and Stores Log Events Cassandra

Cluster which is high scalable and a big Data Repository

Rich tools set for analytics Can be shared with CEP for real time Log Event

analysis. Can provide Logging tool boxes and dashboards

for system administrators using WSO2 BAM High performance and non-intrusiveness

Log Viewer

Get Logs From Cassandra - <<Latest>> Get Archived Logs From HDFS File system

Log Viewer – Archived Logs

Log Viewer – Application Logs