scalable log analysis with wso2 bam

Scalable Log Analysis with WSO2 BAM

Anjana Fernando

Senior Technical LeadWSO2 Inc.

Agenda

• Logging Requirement• Thinking Big• Log Publishing• Log Storage• Log Analysis• Log Search• Custom Dashboards / Alerts• Demo• Future Improvements

Logging Requirement

• Record interested events of a system• Analyse the log events• Take appropriate actions with the analysis

Thinking Big

• Not your typical logger• WSO2’s logging solution is made

from ground up for large deployments

• Utilizes a big data architecture for logging

• WSO2 BAM as the framework

Source: fishingforsoul.wordpress.com

Log Publishing

• Asynchronous• Thrift protocol based event streams• Custom Log4J pattern layout for capturing:

• Tenant information• Server information• Application information

• Not only for Java applications, you can write your own log publisher using any language,

Log Storage

• Multi-level storage• Most recent logs stored in Cassandra• Archived logs store in HDFS

Source: www.carolinasit.com

Log Analysis

• Hive/Hadoop based log summarisation and archiving

• Log archiving daily stored as a hierarchy of tenants and applications

• Custom analytics possible via custom Hive scripts and CEP integration

Log Search

• Search the most recent logs• System Logs

• View logs from all applications, search by log level, keywords• System Logs

• View logs from a specific application, search by log level, keywords

Custom Dashboards / Alerts

• Google Gadgets / Jaggery application based dashboards• Alerting support using in-built CEP features in BAM

• Supports output adaptors such as Email, SMS, HTTP, JMS with message types text, XML, JSON etc..

WSO2 Distributed Logging Architecture

Demo

Future Improvements

• Out of the box support for many other log formats• A mechanism to define custom formats

• Improved searching capabilities• Better indexing for archived logs

• Support for additional data stores, i.e. not only Cassandra, but MongoDB, RDBMS etc.., required for better embeddability with other products

Questions?

Thank You