traffic measurements modified from carey williamson

Traffic Measurements

Modified from Carey Williamson

Network Traffic Measurement

• A focus of networking research for 20+ years• Collect data or packet traces showing packet activity on the

network for different applications• Study, analyze, characterize Internet traffic

• Goals:– Understand the basic methodologies used– Understand the key measurement results to date

2

Why Network Traffic Measurement?

• Understand the traffic on existing networks• Develop models of traffic for future networks• Useful for simulations, capacity planning studies

3

Measurement Environments

• Local Area Networks (LAN’s)– e.g., Ethernet LANs

• Wide Area Networks (WAN’s)– e.g., the Internet

• Wireless LANs• …

4

Requirements

• Network measurement requires hardware or software measurement facilities that attach directly to network

• Allows you to observe all packet traffic on the network, or to filter it to collect only the traffic of interest

• Assumes broadcast-based network technology, superuser permission

5

Measurement Tools (1 of 3)

• Can be classified into hardware and software measurement tools

• Hardware: specialized equipment– Examples: HP 4972 LAN Analyzer, DataGeneral Network Sniffer,

others...

• Software: special software tools– Examples: tcpdump, xtr, SNMP, others...

6

Measurement Tools (2 of 3)

• Measurement tools can also be classified as active or passive• Active: the monitoring tool generates traffic of its own during

data collection (e.g., ping, pchar)• Passive: the monitoring tool is passive, observing and

recording traffic info, while generating none of its own (e.g., tcpdump)

7

Measurement Tools (3 of 3)

• Measurement tools can also be classified as real-time or non-real-time

• Real-time: collects traffic data as it happens, and may even be able to display traffic info as it happens, for real-time traffic management

• Non-real-time: collected traffic data may only be a subset (sample) of the total traffic, and is analyzed off-line (later), for detailed analysis

8

Potential Uses of Tools (1 of 4)

• Protocol debugging– Network debugging and troubleshooting– Changing network configuration– Designing, testing new protocols– Designing, testing new applications– Detecting network weirdness: broadcast storms, routing loops, etc.

9

Potential Uses of Tools (2 of 4)

• Performance evaluation of protocols and applications– How protocol/application is being used– How well it works– How to design it better

10

Potential Uses of Tools (3 of 4)

• Workload characterization– What traffic is generated– Packet size distribution– Packet arrival process– Burstiness– Important in the design of networks, applications, interconnection

devices, congestion control algorithms, etc.

11

Potential Uses of Tools (4 of 4)

• Workload modeling– Construct synthetic workload models that concisely capture the

salient characteristics of actual network traffic– Use as representative, reproducible, flexible, controllable workload

models for simulations, capacity planning studies, etc.

12

13

Traffic Measurement Time Scales

• Performance analysis– representative models

• throughput, packet loss, packet delay

– Microseconds to minutes

• Network engineering– network configuration– capacity planning– demand forecasting– traffic engineering– Minutes to years

• Different measurement methods14

Properties

• Most basic view of traffic is as a collection of packets passing through routers and links

• Packets and Bytes– One can capture/observe packets at some location– Packet arrivals

• interarrivals• count traffic at timescale T

– Captures workload generated by traffic on a per-packet basis

– Packet Size• time series of Byte count

– Captures the amount of consumed bandwidth

• packet size distribution– router design etc.

15

Higher-level Structure

• Transport protocols and applications

• ON/OFF process– bursty workload

– Packet-level– Packet Train

• interarrival threshold

– Session• single execution of an application• Human generated

16

Flows

• Set of packets passing an observation point during a time interval with all packets having a set of common properties– Header field contents, packet characteristics, etc.

• IP flows– source/destination addresses– IP or transport header fields– prefix

• Network-defined flow– network’s workload– ingress and egress– Traffic matrix and Path matrix

17

Semantically Distinct Traffic Types

• Control Traffic– Control plane

• Routing protocols– BGP, OSPF, IS-IS

• Measurement and management– SNMP

• General control packets– ICMP

– Data plane

• Malicious Traffic

18

19

Challenges

• Practical issues– Observability

• Core simplicity– Flows– Packets

• Distributed Internetworking• IP Hourglass

– Data volume

– Data sharing

20

Challenges

• Statistical difficulties– Long tails and High variability

• Instability of metrics• Modeling difficulty• Confounding intuition

– Stationarity and stability• Stationarity: joint probability distribution does not change when shifted in

time• Stability: consistency of properties over time

– Autocorrelation and memory in system behavior

– High dimensionality

21

Tools

• Packet Capture– General purpose systems

• libpcap• tcpdump• ethereal• scriptroute• …

– Special purpose system

– Control plane traffic• GNU Zebra• Routeviews

22

Data Management

• Full packet capture and storage is challenging• Limitations of commodity PC

• Data stream management

• Big Data platforms– Hadoop, etc.

23

Data Reduction

• Lossy compression

• Counters– SNMP Management Information Base

• Flow capture– Packet trains– Packet flows

24

Data Reduction

• Sampling– Basic packet sampling

• Random: with fixed probability• Deterministic: periodic samples• Stratified: multi step sampling

– Trajectory sampling• Chose a randomly sampled packet at all locations

25

Data Reduction

• Summarization– Bloom filters– Sketches: Dimension reducing random projections– Probabilistic counting– Landmark/sliding window models

26

Review: Bloom Filters

• Given a set S = {x1,x2,x3,…xn} on a universe U, want to answer queries of the form:

• Bloom filter provides an answer in– “Constant” time (time to hash).– Small amount of space.– But with some probability of being wrong.

• Alternative to hashing with interesting tradeoffs.

SyIs

Review: Bloom FiltersStart with an m bit array, filled with 0s.

Hash each item xj in S k times. If Hi(xj) = a, set B[a] = 1.

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0B

0 1 0 0 1 0 1 0 0 1 1 1 0 1 1 0B

To check if y is in S, check B at Hi(y). All k values must be 1.

0 1 0 0 1 0 1 0 0 1 1 1 0 1 1 0B

0 1 0 0 1 0 1 0 0 1 1 1 0 1 1 0B

Possible to have a false positive; all k values are 1, but y is not in S.

n items m = cn bits k hash functions

Review: Bloom Filters

• Tradeoffs

• Three parameters.– Size m/n : bits per item.– Time k : number of hash functions.– Error f : false positive probability.

29

Review: Bloom Filters

• False Positive Probability• Pr(specific bit of filter is 0) is

• If r is fraction of 0 bits in the filter then false positive probability is

• Approximations valid as r is concentrated around E[r]. – Martingale argument suffices.

• Find optimal at k = (ln 2)m/n by calculus.– So optimal fpp is about (0.6185)m/n

kckkkk pp )e1()1()'1()1( /

n items m = cn bits k hash functions

pmp mknkn /e)/11('

Data Reduction

• Dimensionality reduction– Clustering

– Principal Component Analysis

• Probabilistic models– Distribution models– Dependence structure

• Inference– Traffic Matrix estimation

31

Curse of Dimensionality.

• A major problem is the curse of dimensionality.• If the data x lies in high dimensional space, then an

enormous amount of data is required to learn distributions or decision rules.

• Example: 50 dimensions. Each dimension has 20 levels. This gives a total of cells. But the no. of data samples will be far less. There will not be enough data samples to learn.

Dimension Reduction

• One way to avoid the curse of dimensionality is by projecting the data onto a lower-dimensional space.

• Techniques for dimension reduction:– Principal Component Analysis (PCA)– Fisher’s Linear Discriminant – Multi-dimensional Scaling. – Independent Component Analysis.– …

Principal Component Analysis

• PCA is the most commonly used dimension reduction technique.– Also called the Karhunen-Loeve transform

• PCA – data samples

• Compute the mean

• Computer the covariance:

• Compute the eigenvalues and eigenvectors of the matrix

• Solve

• Order them by magnitude:

• PCA reduces the dimension by keeping direction such that

Principal Component Analysis

Principal Component Analysis

• For many datasets, most of the eigenvalues \lambda are negligible and can be discarded.

The eigenvalue measures the variationIn the direction e

Example:

Why Principal Component Analysis?

• Motive– Find bases which has high variance in data– Encode data with small number of bases with low MSE

0

5

10

15

20

25

PC1 PC2 PC3 PC4 PC5 PC6 PC7 PC8 PC9 PC10

Vari

an

ce (

%)

Dimensionality Reduction

Can ignore the components of less significance.

You do lose some information, but if the eigenvalues are small, you don’t lose much– n dimensions in original data – calculate n eigenvectors and eigenvalues– choose only the first p eigenvectors, based on their eigenvalues– final data set has only p dimensions

Dimensionality Reduction

Variance

Dimensionality

• PCA may not find the best directions for discriminating between two classes.

• Example: suppose the two classes have 2D Gaussian densities as ellipsoids.

• 1st eigenvector is best for representing the probabilities.

• 2nd eigenvector is best for discrimination.

PCA and Discrimination

Linear methods..

• Principal Component Analysis (PCA)

One DimensionalManifold

Nonlinear Manifolds..

A

Unroll the manifold

PCA and MDS see the Euclideandistance

What is important is the geodesic distance

To preserve structure preserve the geodesic distance and not the euclidean distance.

Two methods

• Tenenbaum et.al’s Isomap Algorithm– Global approach.– On a low dimensional embedding

• Nearby points should be nearby.• Farway points should be faraway.

• Roweis and Saul’s Locally Linear Embedding Algorithm– Local approach

• Nearby points nearby

Isomap

• Estimate the geodesic distance between faraway points.• For neighboring points Euclidean distance is a good approximation

to the geodesic distance.• For farway points estimate the distance by a series of short hops

between neighboring points.– Find shortest paths in a graph with edges connecting neighboring data points

Once we have all pairwise geodesic distances use classical metric MDS

Isomap - Algorithm

• Determine the neighbors.– All points in a fixed radius.– K nearest neighbors

• Construct a neighborhood graph.– Each point is connected to the other if it is a K nearest neighbor.– Edge Length equals the Euclidean distance

• Compute the shortest paths between two nodes– Floyd’s Algorithm– Djkastra’s ALgorithm

• Construct a lower dimensional embedding.– Classical MDS

Isomap

Observations

48

Overview of Traffic Analysis

49

Traffic Samples from Internet2

50

Packet Trains and Autocorrelation

51

Observation #1

• The traffic model that you use is extremely important in the performance evaluation of routing, flow control, and congestion control strategies– Have to consider application-dependent, protocol-dependent, and

network-dependent characteristics– The more realistic, the better

52

Observation #2

• Characterizing aggregate network traffic is hard– Lots of (diverse) applications– Just a snapshot: traffic mix, protocols, applications, network

configuration, technology, and users change with time

53

Observation #3

• Packet arrival process is not Poisson– Packets travel in trains– Packets travel in tandems– Packets get clumped together (ack compression)– Interarrival times are not exponential– Interarrival times are not independent

54

Observation #4

• Packet traffic is bursty– Average utilization may be very low– Peak utilization can be very high– Depends on what interval you use!!– Traffic may be self-similar

• bursts exist across a wide range of time scales

– Defining burstiness (precisely) is difficult

55

Observation #5

• Traffic is non-uniformly distributed amongst the hosts on the network– Example: 10% of the hosts account for 90% of the traffic (or 20-80)– Why?

• Clients versus servers, geographic reasons, popular ftp sites, web sites, etc.

56

Observation #6

• Network traffic exhibits ‘‘locality’’ effects– Pattern is far from random– Temporal locality– Spatial locality– Persistence and concentration– True at host level, at gateway level, at application level

57

Observation #7

• Well over 90% of the byte and packet traffic on most networks is TCP/IP– By far the most prevalent– Often as high as 95-99%– Most studies focus only on TCP/IP for this reason

58

Observation #8

• Most conversations are short– Example: 90% of bulk data transfers send less than 10 kilobytes of

data– Example: 50% of interactive connections last less than 90 seconds– Distributions may be ‘‘heavy tailed’’

• i.e., extreme values may skew the mean and/or the distribution

59

Observation #9

• Traffic is bidirectional– Data usually flows both ways– Not just acks in the reverse direction– Usually asymmetric bandwidth though– Pretty much what you would expect from the TCP/IP traffic for most

applications

60

Observation #10

• Packet size distribution is bimodal– Lots of small packets for interactive traffic and acknowledgements – Lots of large packets for bulk data file transfer type applications– Very few in between sizes

61

Network Security Monitoring and Analysis based on Big Data Technologies

Bingdong Li

62

Objectives

• A network security monitor and analysis system based on Big Data technologies to

– Measures the network

– Real time continuous monitoring and interactive visualization

– Intelligent network object classification and identification based on role behavior as context

Big Data Machine Learning

Network Security

Objectives

65

System Design

• Data Collection

66

System Design

• Online Real Time Process

67

System Design

• NoSQL Storage

68

System Design

• User Interfaces

69

70

Monitoring and Visualization

• Real Time response within a time constraint

• Interactive involve user interaction

• Continuously “continue to be effective overtime in light

of the inevitable changes that occur” (NIST)

71

Network Status

72

Top N

73

HadoopDhruba Borthakur

74

Hadoop, Why?

• Need to process Multi Petabyte Datasets• Expensive to build reliability in each application.• Nodes fail every day

– Failure is expected, rather than exceptional.– The number of nodes in a cluster is not constant.

• Need common infrastructure– Efficient, reliable, Open Source Apache License

• The above goals are same as Condor, but– Workloads are IO bound and not CPU bound

Commodity Hardware

Typically in 2 level architecture– Nodes are commodity PCs– 30-40 nodes/rack– Uplink from rack is 3-4 gigabit– Rack-internal is 1 gigabit

Goals of HDFS

• Very Large Distributed File System– 10K nodes, 100 million files, 10 PB

• Assumes Commodity Hardware– Files are replicated to handle hardware failure– Detect failures and recovers from them

• Optimized for Batch Processing– Data locations exposed so that computations can move to where data resides– Provides very high aggregate bandwidth

• User Space, runs on heterogeneous OS

SecondaryNameNode

Client

HDFS Architecture

NameNode

DataNodes

1. filename

2. BlckId, DataNodes

o

3.Read data

Cluster Membership

Cluster Membership

NameNode : Maps a file to a file-id and list of MapNodesDataNode : Maps a block-id to a physical location on diskSecondaryNameNode: Periodic merge of Transaction log

Distributed File System

• Single Namespace for entire cluster• Data Coherency

– Write-once-read-many access model– Client can only append to existing files

• Files are broken up into blocks– Typically 128 MB block size– Each block replicated on multiple DataNodes

• Intelligent Client– Client can find location of blocks– Client accesses data directly from DataNode

NameNode Metadata

• Meta-data in Memory– The entire metadata is in main memory– No demand paging of meta-data

• Types of Metadata– List of files– List of Blocks for each file– List of DataNodes for each block– File attributes, e.g creation time, replication factor

• A Transaction Log– Records file creations, file deletions. etc

DataNode

• A Block Server– Stores data in the local file system (e.g. ext3)– Stores meta-data of a block (e.g. CRC)– Serves data and meta-data to Clients

• Block Report– Periodically sends a report of all existing blocks to the NameNode

• Facilitates Pipelining of Data– Forwards data to other specified DataNodes

Data Flow

Web Servers Scribe Servers

Network Storage

Hadoop ClusterOracle RAC MySQL