Analysis Farm:A Cloud-based Scalable Aggregation and

Query Platform for Network Log Analysis

wei.jianwen@gmail.comDec 12th, 2011

Jianwen Wei, Yusu Zhao, Kaida Jiang, Rui Xie, Yaohui JinSchool of Electronic Information and Electrical Engineering, SJTU

Network and Information Center, SJTUShanghai Jiaotong University

The 2011 International Workshop on Data Cloud (D-CLOUD 2011), Hong Kong

Outline

• Background

• Design and Implementation

• Experimental Results

• Summary

Outline

•Background

• Design and Implementation

• Experimental Results

• Summary

BackgroundMotivation: An Overview of SJTU Networks

• Serving 50,000 people• 10Gb WDM, MPLS• Network Monitoring

BackgroundMotivation: An Overview of SJTU Networks (cont.)

Applications such as BT etc. use too much BORDER bandwidth!

BackgroundDeployment of the Network Log Analysis System

Mirrored Traffic(Raw Data)

DPIBorder Router

Syslog(plain text)

Analysis Farm

Syslog Collector

6Gbps ~3MBytes/s 5000 per sec

BackgroundNetwork Log Analysis System

Mirrored Traffic(Raw Data)

DPIBorder Router

Syslog(plain text)

Analysis Farm

Syslog Collector

6Gbps ~3MBytes/s 5000 per sec

BackgroundNetwork Log Analysis System: Border Router

Mirrored Traffic(Raw Data)

DPIBorder Router

Syslog(plain text)

Analysis Farm

Syslog Collector

6Gbps ~3MBytes/s 5000 per sec

•Handle all incoming and outgoing traffic•Connecting to multiple ISPs• Traffic at 6Gbps

BackgroundNetwork Log Analysis System: DPI Engine

Mirrored Traffic(Raw Data)

DPIBorder Router

Syslog(plain text)

Analysis Farm

Syslog Collector

6Gbps ~3MBytes/s 5000 per sec

• Input: 6Gbps raw network traffic•Output: 3MBytes/s syslog messages•Running on an x86 server•Analyze every network session

BackgroundNetwork Log Analysis System: Syslog Collector

Mirrored Traffic(Raw Data)

DPIBorder Router

Syslog(plain text)

Analysis Farm

Syslog Collector

6Gbps ~3MBytes/s 5000 per sec

• Java-written syslog collector•Running on a virtual machine• Insertion rate: 5000/s on average, 12000/s at peak

BackgroundNetwork Log Analysis System: Analysis Farm

Mirrored Traffic(Raw Data)

DPIBorder Router

Syslog(plain text)

Analysis Farm

Syslog Collector

6Gbps ~3MBytes/s 5000 per sec

• Store log•Analyze log

BackgroundLog Analysis Tasks

• Aggregating

• Get the overall usage of network border

• Querying

• Inspect network activities

http.tcp 1320155721-1320155731202.120.2.102:54285-8.8.4.4:80 374 24021

BackgroundLog Analysis Tasks

400 million log records per day (350GByte) !

BackgroundResearch Challenges

• Storage Scalability

• Computation Scalability

• Query Agility

BackgroundRelated Work

• loggly.com

• “Logging as a Service”

• Yottaa.com

• Log-based Website performance analysis

• They use cloud-based solutions for scalability

Outline

• Background

•Design and Implementation

• Experimental Results

• Summary

Design and ImplementationOur Approach: Cloud Computing + NoSQL

• Cloud Computing

• manageable, scalable, on demand resources

• OpenStack open source toolset for building clouds

• NoSQL (Not Only SQL)

• weaken ACID to improve performance

• MongoDB document-oriented distributed database

Design and ImplementationThe Architecture of Analysis Farm

CPU Memory iSCISStorage

Network

Users

Application Layer

IaaS Layer

Hardware Resource Pool

mongos

mongod mongod mongod mongod

VM VM VM VM

Configuration server

Request

Design and ImplementationHow we tackle the three challenges?

• Storage Scalability

• On line Storage Expansion

• Computation Scalability

• MongoDB Scale out

• Query Agility

• MongoDB Handles ad hoc queries effectively

Design and ImplementationAddress the Storage Scalability

1.The application servers ask the IaaS layer for more disk space.

2.The IaaS layer asks the hardware resource pool to attach new block devices.

3.The application servers execute on line filesystem expansion.

On Line Storage Expansion

No service interruption

Design and ImplementationAddress the Computation Scalability

1.The IaaS provides a new server to the cluster.

2.The MongoDB cluster rebalances data automatically.

MongoDB Scale out

combiner

mapper, combiner mapper, combiner mapper, combiner mapper, combiner

MapReduceRequest

mongos

mongod mongod mongod mongod

No service interruption

Design and ImplementationAddress the Query Agility

• Expressive Data Model

• Building Blocks for Compound Queries

• Aggregating tools such as Group, MapReduce

• Effective Optimization Methods, such as index

MongoDB handles ad hoc queries effectively

Outline

• Background

• Design and Implementation

• Experimental Results

• Summary

Experimental ResultsAggregating and Querying

• Aggregating Log

• Ad hoc Querying

SPEED is our primary focus.

Experimental ResultsExperimental Setup for Aggregating

• Method

• Aggregate 10-min log with MongoDB MapReduce

• Dataset

• One day’s log records, ~400million records

• Configurations for Comparison

• 1x farm: 4 mongod threads on a single server

• 4x farm: 4 mongod threads on four servers

• 8x farm: 8 mongod threads on eight servers

Experimental ResultsExperimental Results for Aggregating

Type Records Processed TimeRate

(records/s)

1x

4x

8x

3201454 523s 6119

3103742 200s 15568

3317013 111s 29883

Experimental Results for 10-minute Log Aggregating

Experimental ResultsExperimental Setup for ad hoc Querying

• Method

• Execute ad hoc querying

• Dataset

• One day’s log records, ~400million records

• Index

• (start_t, end_t, src_IP, dst_IP, app)

• Configuration for Analysis Farm

• 8x farm: 8 mongod threads on eight servers

Experimental ResultsExperimental Setup for ad hoc Querying (cont.)

• Query Types

• IP-initial Query src_IP == IP

• IP-engaging Query src_IP == IP OR dst_IP == IP

• IP-pair Query IP-pair engaging AND app == HTTP

• Time Scopes

• 10 minutes, 30 minutes, 60 minutes

Experimental ResultsExperimental Results for IP-initial Query

Time Scope Execution Time Records ScannedRate

(records/s)

10min

30min

60min

3.085s 227581 73770

8.816s 643259 72965

18.517s 1370443 73795

Experimental Results for IP-initial Query(src_IP == IP)

Experimental ResultsExperimental Results for IP-engaging Query

Time Scope Execution Time Records ScannedRate

(records/s)

10min

30min

60min

18.012s 1234582 68542

54.708s 3673304 67144

119.034s 7912644 66474

Experimental Results for IP-engaging Query(src_IP == IP OR dst_IP == IP)

Experimental ResultsExperimental Results for IP-pair Query

Time Scope Execution Time Records ScannedRate

(records/s)

10min

30min

60min

5.670s 296772 52340

6.267s 324813 51829

19.327s 1027513 53165

Experimental Results for IP-pair Query(the IP-pair engages AND app == http)

Outline

• Background

• Design and Implementation

• Experimental Results

• Summary

Summary

• Analysis Farm is built on OpenStack and MongoDB

• Analysis Farm is easy-to-manage and easy-to-scale-out

• Feasibility in aggregating and querying is verified

• We use Analysis Farm to analyze 400 million, or 350GB log records every day

Acknowledgement

• 973 program and NFSC

• My partners in Shanghai Jiaotong Univ.

• Dr. Lin Gu in HKUST

• Workshop organizers and reviewers

Thank you!

The 2011 International Workshop on Data Cloud (D-CLOUD 2011), Hong Kong

Analysis Farm: A Cloud-based Scalable Aggregation and Query Platform for Network Log AnalysisShanghai Jiaotong University

wei.jianwen@gmail.com @JianwenWEI