distributed network traffic monitoring and analysis using load

WebTrafMon II POSTECHDP&NM Lab.

Distributed Network Traffic Monitoring and Analysis using Load Balancing Technology

Distributed Network Traffic Monitoring and Analysis using Load Balancing Technology

Soon-Hwa Hong, Jae-Young Kim, Bum-Rae Choand James W. Hong

DP&NM Lab. Dept. of Computer Science and Engineering

POSTECH, Pohang KoreaEmail: {padosori, jay, brcho, jwkhong}@postech.ac.kr

http://dpnm.postech.ac.kr/


(2/16)

Introduction?Network traffic (text, image, software, audio, video) is

increasing continuously both on the Internet and Intranet.? A simple, accurate and efficient network traffic monitoring

and analysis is required to understand the current usage as well as to plan for future.

?Many shortcomings exist in currently available monitoring systems.? cannot analyze long-term traffic.? do not have monitoring capability from multiple network points.? capture, analysis and presentation all in one machine.? cannot prevent packet drops from the system overload.

?WebTrafMon II attempts to overcome these shortcomings using distributed architecture and load balancing technique.


(3/16)

Related WorkUserInterface

Distributed Architecture

Support for Multiple network points

Analysis Scope Analysis Interval

Analysis Method

CaptureMethod

Textnonolayer 7currentreal timelibpcaptcpdump

Webyesyeslayer 7current, hourly, daily, monthly, yearly

real time, batch

libpcapWebTrafMon II

Webnonolayer 7current, hourlyreal time, batch

libpcapWebTrafMon

Webyesyeslayer 25 minute, hourly, daily, weekly, monthly

batchsnmpagent

MRTG

X-Windowsnonolayer 7

current, user-specified time

real time, batch

libpcapethereal

Webnonolayer 75 second, hourlybatchlibpcapntop


(4/16)

WebTrafMon II Requirements

?analyze various types of information: host information, network, transport, and application layer protocols. ?analyze present real-time, hourly, daily, monthly

and yearly network traffic data automatically.?analyze multiple network points traffic.?no packet drops.?web-based graphical user interface.


(5/16)

Centralized vs. DistributedCentralized Traffic Analysis Architecture

presentation

analysis

capture

network point user

■ system overload occurs frequently and manypackets are dropped

■ cannot support for multiple network points■ presentation time is slow

Distributed Traffic Analysis Architecture

analysis

capture presentation

usernetwork point

■ capture, analysis and presentation modules execute on separate machines to minimize system overload

■ can support for multiple network points■ presentation time is fast

network traffic datapacket header information

packet header information network traffic data

analyzed information

distributedenvironment

analyzed information


(6/16)

WebTrafMon II: Design

database

analyzer web viewer

… … … … … … …

1. multiple network point packet capture and analyze packet header

(real-time, hourly, daily, monthly, yearly)

statistics network traffic data

2. analyze packet header and save into DB and make short term and long term traffic data

3. query to database from user request and give information to user

probe

network point

promiscuous mode packet capture

probe

network point


analyze packet header and save into DB make real-time, short term,

long term traffic data

user

distributedenvironment

request

response

packet header information


(7/16)

Packet Capture Module (Probe)

probe(analysis packet header)

network point

Log File

log format

… … … … … … … … … … …

log format

log format

log format

?Probe captures packet with promiscuous mode, analyzespacket header and saves into log file?No packet drops from system overload using an independent

packet capture module.

Log format : time, length, frame_type, source ip, destination ip,protocol, source port and destination port information

save packet header information into log file



(8/16)

Log Format

6 6 2 4data CRC

46-1500 bytes

TCP data

Ethernet

IP

TCP

Log format2 2

time length frame_type src_ip dst_ip protocol src_port dst_port4 4 42 2 2 21 (bytes)

frame capture time

total Ethernet frame size(Ethernet header + data + CRC)

Dst addr

Srcaddr Type

4 1 IP data4Dst addrSrc addr Protocol

dst portsrc port


(9/16)

Packet Analysis Module (Analyzer)

… … … … …

log file Log Transformerlog data

log file

log file

database

IP-based, non IP-based data

DB Analyzer

IP-based, non IP-based data

statistical real-time, hourly, daily, monthly and yearly data.

?An Analyzer is divided into a Log Transformer module and DB Analyzer module. ?Log Transformer assorts log files into IP-based data, and non IP-based data (e.g., ARP, RARP, IPX). ?Log Transformer saves these assorted data to database. ?DB Analyzer analyzes assorted data in database and makes statistical real-time,hourly, daily, monthly and yearly data.


(10/16)

Data Translation by Analyzer for Long-Term Traffic

Log files

hourly_application

hourly_data_sent

hourly_data_received

hourly_data_exchanged

hourly_ network

hourly_transport

daily_*_table monthly_*_table yearly_*_table

raw_ip_table raw_non_ ip_table

hourly

monthly yearly

daily


(11/16)

DB Schema

SMALLINT UNSIGNEDdst_port

SMALLINT UNSIGNEDsrc_port

TINYINT UNSIGNED

protocol

INTdst_ip

INTsrc_ip

SMALLINT UNSIGNED

frame_type

SMALLINT UNSIGNEDlength

INTtime

Type Name

SMALLINT UNSIGNED

ether_type

SMALLINT UNSIGNED

length

INTtime

TypeName

BIGINTcount

BIGINTbytes

VARCHAR(100)frame_name

VARCHAR(20)frame_type

TINYINT UNSIGNEDday

TypeName

BIGINTcount

BIGINTbytes



TINYINT UNSIGNEDyear

TypeName

BIGINTcount

BIGINTbytes



TINYINT UNSIGNED

month

TypeName

BIGINTcount

BIGINTbytes



TINYINT UNSIGNED

hour

TypeName

raw_ip_table

raw_non_ ip_table

hourly_network_table

daily_network_tablemonthly_network_table

yearly_network_table

hourly

dailyyearly

monthly

This is a *_network_table example. network_table has IP information from raw_ip_table and ARP and so on information from raw_non_ip_table.


(12/16)

Web Viewer Module Design

raw_ip, non_ip table

data_sent table

data_received table

data_exchanged table

network table

transport table

application table

Database

Web Browser(user)

Web Viewer

Database

Client

Web

Serverquery

reply

CGI

requestresponse


(13/16)

http

Database

Implementation

Binary Log File

packet capture using libpcap

probe

analyzer

save packet header information into log file

save packet header information into database raw ip, non-ip, table

raw_ip, non_ip_table

data sent table

data received table

data exchanged table

network table

transport table

application table

MySQLClient

NFS distributed environmentNFS distributed environment

MySQLServer

web viewerC-CGI,MySQL Client

query

reply

C-CGI, Apache web-server

Internet

make real-time, hourly, daily, monthly and yearly host, protocol and application information table

user


(14/16)

Web-based User Interface: Main View

menuanalysis time

hourly total traffic

detailed views

time interval menu


(15/16)

Detailed Views1

3

2

4

5

6


(16/16)

Summary & Future Work

?WebTrafMon II overcomes many shortcomings of existing monitoring and analysis systems.

? can analyze real-time and hourly, daily, monthly and yearly network traffic data.

?With load balancing, independent packet capture prevents packet drops from monitoring system overload.

?WebTrafMon II can analyze multiple network points traffic.

? Future work on WebTrafMon II?More analysis on host and application relationships.? Adapt to monitor and analyze other types of IP networks (IPoA,

IPoWDM, etc.)? Traffic analysis based on contents (video, audio, etc.)

distributed network traffic monitoring and analysis using load

Documents