mt logging with_bam
Post on 22-May-2015
291 Views
Preview:
DESCRIPTION
TRANSCRIPT
MT-Logging with WSO2 BAM
Amani SoysaSenior Software Engineer
How it Works??
Logs are published to BAM via Log4JAppender
Published Logs are Stored in Cassandra (per tenant per day)
When the day is over Logs are published to HDFS file system daily using BAM analyzers
To view Logs :-Application Logs – Log events on web applications/services filter by the application name System Logs – View the system log of each product {With filtering categories}
Log Viewers get logs from cassandra <<real time log>> and HDFS file system <<Archive logs>>
Log Publisher - Log4JAppender
Setting Up LogEventAppender
log4j.rootLogger=INFO, LOGEVENT
# LOGEVENT is set to be a LogEventAppender using a PatternLayout to send logs to LOGEVENT log4j.appender.LOGEVENT=org.wso2.carbon.logging.appender.LogEventAppenderlog4j.appender.LOGEVENT.url=tcp://localhost:7611log4j.appender.LOGEVENT.layout=org.wso2.carbon.utils.logging.TenantAwarePatternLayoutlog4j.appender.LOGEVENT.columnList=%T,%S,%A,%d,%c,%p,%m,%H,%I,%Stacktracelog4j.appender.LOGEVENT.userName=adminlog4j.appender.LOGEVENT.password=adminlog4j.appender.LOGEVENT.processingLimit=1000log4j.appender.LOGEVENT.maxTolerableConsecutiveFailure=20
streamId = dataPublisher.defineStream("{" + "'name':'log" + "." + tenantId + "."+ severName + "." + currDateStr + "'," + " 'version':'1.0.0',"+ " 'nickName': 'Logs'," + " 'description': 'Logging Event',"+ " 'metaData':[" + " {'name':'clientType','type':'STRING'}" + " ],"+ " 'payloadData':[" + " {'name':'tenantID','type':'STRING'},"+ " {'name':'serverName','type':'STRING'},"+ " {'name':'appName','type':'STRING'},"+ " {'name':'logTime','type':'LONG'},"+ " {'name':'priority','type':'STRING'},"+ " {'name':'message','type':'STRING'},"+ " {'name':'logger','type':'STRING'},"+ " {'name':'ip','type':'STRING'},"+ " {'name':'instance','type':'STRING'},"+ " {'name':'stacktrace','type':'STRING'}" + " ]" + "}");
Stream Definition
Log Analyzer
Logging-Config.xml
Basic Information
<isDataFromCassandra>true</isDataFromCassandra>
Information Regarding Cassandra
<cassandraHost>localhost:9160</cassandraHost> <userName>admin</userName> <password>admin</password>
Information Regarding HDFS Server
<archivedHost>hdfs://localhost:9000</archivedHost> <archivedHDFSPath>/stratos/logs</archivedHDFSPath>
Logging Summarizer
Archive logs daily at a given time and store in hdfs file system
summarizer-config.xml
Basic Information
<isDataFromCassandra>true</isDataFromCassandra><deleteColFamily>false</deleteColFamily>
Information Regarding Cassandra <cassandraHost>localhost:9160</cassandraHost> <userName>admin</userName> <password>admin</password>Information Regarding BAM <publisherURL>localhost:7611</publisherURL> <userName>admin</userName> <password>admin</password> <cronExpression>0 0 1 ? * * *</cronExpression>
Information Regarding HDFS Server
<archivedHost>hdfs://localhost:9000</archivedHost> <archivedLogLocation>/stratos/logs</archivedLogLocation>
Advantages
Asynchronous and None Blocking Data publishing Receives and Stores Log Events Cassandra
Cluster which is high scalable and a big Data Repository
Rich tools set for analytics Can be shared with CEP for real time Log Event
analysis. Can provide Logging tool boxes and dashboards
for system administrators using WSO2 BAM High performance and non-intrusiveness
Log Viewer
Get Logs From Cassandra - <<Latest>> Get Archived Logs From HDFS File system
Log Viewer – Archived Logs
Log Viewer – Application Logs
top related