Differences between In- and Outbound Internet Backbone Traffic

Wolfgang John and Sven TafvelinDept. of Computer Science and Engineering

Chalmers University of TechnologyGöteborg, Sweden

2007-05-23TNC 2007

Overview

1. Introduction

2. Highlights of directional differences on• IP level• TCP level• UDP level

3. Summary of results

4. Conclusions

2007-05-23TNC 2007

Introduction: Motivation

• Why measuring on Internet links?– to understand the nature of Internet traffic– quantify deployment of protocol features

• Interesting for– Network engineers and protocol developers– Network modeling and simulation community– Network security and intrusion detection

2007-05-23TNC 2007

Introduction: Related work

• Directional differences on backbone traffic– Evident on simple packet header analysis– Correlation of packets might reveal reasons

• Related work:– Mainly unidirectional flow data (NetFlow)– Either low or very high aggregation level– Marginal discussion on directional differences

2007-05-23TNC 2007

Introduction: Our contribution

• Complete view on different levels

• Contemporary data

• Packet level analysis

• Bi-directional TCP connections

• Specific measurement location– Medium aggregation level– Suitable for highlighting directional differences

2007-05-23TNC 2007

Introduction: Measurement location

Internet

Internet

Region

al ISPsRegion

al ISPs

Gbg

Sthlm

Göteborgs Univ.

Chalmers Univ.

Stud-Net

• 2x 10 Gbit/s (OC-192)• 2x DAG6.2SE Cards• tightly synchronized• capturing headers

2007-05-23TNC 2007

Introduction: General traffic characteristics

• Data from 20 days in April 2006

• 146 traces, 10.7 billion frames, 7.5 TB

• 99.99% IPv4 data

• 93% TCP packets

• 97% TCP data

• Data and packet counts equal on inbound and outbound links!

2007-05-23TNC 2007

Inside Outside Source Dest. Dest. SourceTotal 0.63 22.0 0.27 19.2 0.49 19.8TCP 0.41 5.0 0.18 4.3 0.31 4.5UDP 0.48 19.2 0.18 16.4 0.38 16.9Rest 0.15 1.9 0.02 1.1 0.15 1.0

Outbound InboundTotal

Highlights: IP level

• Distinct IP addresses seen (in Millions)

TotalTCPUDPRest

Inside Outside Source Dest. Dest. SourceTotal Outbound Inbound

2007-05-23TNC 2007

• Distinct IP addresses seen (in Millions)

• Surprisingly large numbers

• Inbound destinations >> outbound sources

• Outside hosts primarily due to UDP

Highlights: IP level

TotalTCPUDPRest

Inside Outside Source Dest. Dest. SourceTotal Outbound Inbound

0.63 22.0 0.27 19.2 0.49 19.80.41 5.0 0.18 4.3 0.31 4.50.48 19.2 0.18 16.4 0.38 16.9

2007-05-23TNC 2007

rejected connections 44.3 12.3 32.0scanning - no reply 35.0 8.3 26.7

asymetric traffic 4.8 2.2 2.6scanning - RST reply 4.5 1.8 2.7

total outbound inboundTCP connections 72.6 28.0 44.6

rejected 44.3 12.3 32.0established 28.3 15.7 12.6

Highlights: TCP level

• Connection attempt breakdown (Millions)

total outbound inboundTCP connections

rejectedestablished

72.6 28.0 44.6

44.3 12.3 32.0rejected connectionsscanning - no replyasymmetric traffic

scanning - RST reply

2007-05-23TNC 2007

Highlights: TCP level

• Connection attempt breakdown (Millions)

• Inbound connections mainly scans!

72.6 28.0 44.6

44.3 12.3 32.0

44.3 12.3 32.0TCP connections

rejectedestablished

total outbound inbound

35.0 8.3 26.7rejected connections

scanning - no replyasymmetric traffic

scanning - RST reply

2007-05-23TNC 2007

total outbound inboundestablished connections 28.3 15.7 12.6proper closing (2xFIN) 19.0 11.4 7.6FIN and RST outbound 3.2 0.6 2.6

FIN and RST inbound 1.7 0.7 1.0single RST 2.2 1.6 0.6

FIN, RST in counter dir. 1.2 0.9 0.3unclosed 1.0 0.5 0.5

Highlights: TCP level (2)

established connectionsproper closing (2xFIN)FIN and RST outbound

FIN and RST inboundsingle RST

FIN, RST in counter dir.unclosed

total outbound inbound28.3 15.7 12.6

• TCP termination behavior (Millions)

2007-05-23TNC 2007

Highlights: TCP level (2)

• TCP termination behavior (Millions)

• Only 67% close properly (2xFIN)• Inbound: 20% of conn. closed by FIN and RST!

established connectionsproper closing (2xFIN)FIN and RST outbound

FIN and RST inboundsingle RST

FIN, RST in counter dir.unclosed

total outbound inbound28.3 15.7 12.619.0 11.4 7.6 3.2 0.6 2.6

2.2 1.6 0.6

2007-05-23TNC 2007

Highlights: TCP level (3)

• Statistical properties of established TCP connections– Lifetime, data volume, packet count

• Inbound connections more likely to:– show lifetimes between 1 and 5 seconds– be long lasting (>10 minutes)– carry more data and more packets– show higher asymmetry (client-server pattern)

2007-05-23TNC 2007

TCP level: P2P traffic

• Quantification according to port-numbers

• Missing payload

→ underestimated by factor 2-3 [*,**]

– 13% of data in outbound connections– 25% of data in inbound connections

* S. Sen et al, “Accurate, Scalable in-network identification of P2P traffic across large networks”, IMW 2002

** T. Karagiannis et al, “Transport layer identification of P2P Traffic”, ACM SIGCOMM 2004

2007-05-23TNC 2007

Highlights: UDP level

• 68 million UDP flows• 51 million carry less than 3 packets!• DNS: 5%; NTP 1.7%• Incoming scanning: > 8%• P2P overlay traffic: > 20%• Signaling Traffic

– Distributed Hash Table (DHT) like Kademlia– Update routing tables in decentralized way– Periodic “ping” queries and replies– P2P overlay networks span entire globe– High fluctuation in peering partners → lots of IPs

2007-05-23TNC 2007

Summary of results

• Besides equal counts and volumes on both links, directional differences were found in:– IP packet sizes– IP fragmentation– Number of TCP connections– TCP connection establishment & termination– TCP option usage– TCP connection properties– UDP scanning traffic

2007-05-23TNC 2007

Conclusion

• High level analysis does not necessarily show differences → detailed analysis does!

• 2 main reasons for directional differences: – Malicious traffic

• the Internet is “unfriendly”

– P2P• Göteborg is a P2P source• P2P is changing traffic characteristics

e.g. packet sizes, TCP termination, TCP option usage

Thank you very much for you attention!

Questions?

2007-05-23TNC 2007

BACKUP

• BACKUP SLIDES

2007-05-23TNC 2007

Common P2P port numbers

TCP:688[0-9] bittorrent

49200 bittorrent32459 bittorrent (µtorrent)49152 bittorrent (µtorrent)

41[1-2] direct connect (dc++)1412 direct connect (dc++)6346 gnutella (limewire)6348 gnutella (bearshare)

466[0-8] overnet (edonkey)14662 overnet (edonkey)14663 overnet (edonkey)

UDP:688[0-9] bittorrent

49200 bittorrent32459 bittorrent (µtorrent)

41[1-2] direct connect (dc++)1412 direct connect (dc++)9183 dc++ kademlia6346 gnutella (limewire)6348 gnutella (bearshare)

466[0-8] overnet (edonkey)4672 overnet (edonkey)

14672 overnet (edonkey)

2007-05-23TNC 2007

TCP level (4)

• TCP options (in %)

SYN SYN/ACK both SYN SYN/ACK bothoutbound 100.00 99.59 99.59 19.36 15.46 15.46inbound 99.94 99.92 99.85 24.33 23.85 23.83

MSS WS

SYN SYN/ACK both SYN SYN/ACK bothoutbound 93.67 69.70 69.70 16.50 12.32 12.32inbound 97.22 90.40 90.38 19.72 18.51 18.50

SACK TS

SYN SYN/ACK both SYN SYN/ACK bothMSS WS

SYN SYN/ACK both SYN SYN/ACK bothSACK TS

outboundinbound

outboundinbound

2007-05-23TNC 2007

SYN SYN/ACK both SYN SYN/ACK bothoutbound 93.67 69.70 69.70 16.50 12.32 12.32inbound 97.22 90.40 90.38 19.72 18.51 18.50

SACK TS

16.50 12.32 12.32

TCP level (4)

• TCP options (in %)

SYN SYN/ACK both SYN SYN/ACK bothoutbound 100.00 99.59 99.59 19.36 15.46 15.46inbound 99.94 99.92 99.85 24.33 23.85 23.83

MSS WSSYN SYN/ACK both SYN SYN/ACK both

MSS WS

SYN SYN/ACK both SYN SYN/ACK bothSACK TS

outboundinbound

100.00 99.59 99.5999.94 99.92 99.85

outboundinbound 24.33 23.85 23.83

19.36 15.46 15.46

93.67 69.70 69.7019.72 18.51 18.5097.22 90.40 90.38

15.4623.83

69.7090.38

12.3218.50

2007-05-23TNC 2007

IP level (2)

• Packet size distribution on the 2 links

Packet size total outbound inbound20-39 0.14% 0.18% 0.11%40-60 39.25% 38.41% 40.02%

576 0.98% 0.63% 1.30%628 1.79% 2.12% 1.49%

1300 1.13% 1.25% 1.01%1400-1500 38.53% 38.62% 38.45%

Packet size total outbound inbound20-3940-60

576628

13001400-1500

2007-05-23TNC 2007

IP level (2)

• Packet size distribution on the 2 links

Packet size total outbound inbound20-39 0.14% 0.18% 0.11%40-60 39.25% 38.41% 40.02%

576 0.98% 0.63% 1.30%628 1.79% 2.12% 1.49%

1300 1.13% 1.25% 1.01%1400-1500 38.53% 38.62% 38.45%

Packet size total outbound inbound20-3940-60

576628

13001400-1500

39.25% 38.41% 40.02%

38.53% 38.62% 38.45%

1.79% 2.12% 1.49%

2007-05-23TNC 2007

IP level (3)

• IP fragmentation on the 2 links

Total outbound inboundTotal 0.065% (100.0%) 0.014% (100.0%) 0.113% (100.0%)TCP (4.5%) (18.0%) (2.9%)UDP (88.6%) (18.8%) (97.1%)ESP (6.8%) (63.1%) (0.0%)

Total outbound inboundTotalTCPUDPESP

0.065% (100.0%) 0.014% (100.0%) 0.113% (100.0%)

2007-05-23TNC 2007

Malicous traffic / P2P traffic

• Connection properties

1E-3

1E-2

1E-1

1E+0

1E+1

1E+2

0 5 10 15 20 25

1E-4

1E-3

1E-2

1E-1

1E+0

1E+1

1E+2

0 100 200 300 400 500 600size in Kbytes

% c

on

nec

tio

ns

outboundinbound

lifetime in sec

Property mean σ median P80out 18.2 60.7 1.8 16.6in 17.3 65.8 0.6 24.8out 61.0 2362 1.1 2.9in 81.5 3298 1.9 8.9out 81.5 2289 11.5 22.0in 113.0 3538 11.5 21.0

Lifetime in sec

Packet Count

Size in Kbytes

Property mean σ median P80

Lifetime in sec

Packet Count

Size in Kbytesout 61.0 2362 1.1 2.9in 81.5 3298 1.9 8.9