caida bandwidth estimation meeting san diego june 2002 r. hughes-jones manchester udpmon and...

CAIDA Bandwidth Estimation Meeting San Diego June 2002R. Hughes-Jones Manchester

UDPmon and TCPstreamTools to understand Network Performance

Richard Hughes-JonesThe University of Manchester

www.hep.man.ac.uk/~rich/net/


UDPmon: The Measurements (1)

Latency Round trip times measured using Request-Response

UDP frames Latency as a function of frame size

Slope gives sum of individual data transfer rates end-to-endMem copy + pci + Gig Ethernet + pci + mem copy

Intercept indicates processing times + HW latencies

Histograms of ‘singleton’ measurements Tells us about:

Behavior of the IP stack The way the HW operates Performance of the LAN / MAN / WAN


UDPmon: The Measurements (2) UDP Throughput Send a controlled stream of UDP frames spaced at regular intervals Vary the frame size and the frame transmit spacing At the receiver record

The time of first and last frames received The number packets received, the number lost, number out of order The received inter-packet spacing is histogramed The time each packet is received provides packet loss pattern CPU load, Number of interrupts

Use the Pentium CPU cycle counter for times and delay Few lines of user code

Tells us about: Behavior of the IP stack The way the HW operates Capacity and Available throughput of the LAN / MAN / WAN


UDPmon: The Works How it works:

The ‘Zero_stats’ request also provides an interlock against concurrent tests.

Zero stats OK done

●●●

Get remote statisticsSend statistics

Send data frames atregular intervals ●●●

Time to send Time to receive

Inter-packet time

Signal end of testOK done

n bytes

Number of packets

Wait timetime


Latency: Tests B2B PCs with Gigabit Ethernet

y = 0.0252x + 46.887

y = 0.0127x + 64.183

0

20

40

60

80

100

0 500 1000 1500 2000 2500 3000

Message length bytes

Late

ncy u

s

PC=PC UDP SysKonnet

0

50

100

150

200

250

300

0 2000 4000 6000 8000 10000 12000 14000 16000


Late

ncy u

s

ave time

NIC SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE CPU: PIII 800 MHz PCI:64 bit 33 MHz RedHat 7.1 Kernel 2.4.14 Latency low & well behaved Latency Slope 0.0252 us/byte Expect:

PCI 0.00758

GigE 0.008

PCI 0.00758

0.0236 us/byte

www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt


Latency: Tests B2B PCs with Gigabit Ethernet

y = 0.0187x + 167.86

0

50

100

150

200

250

0 200 400 600 800 1000 1200 1400


Late

ncy u

s

Intel pro/1000 ave time

0

50

100

150

200

250

300

350

400

450

0 2000 4000 6000 8000 10000 12000 14000 16000


Late

ncy u

s

Intel pro/1000 ave time

NIC Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 Latency high but well behaved Indicates Interrupt coalescence Slope 0.0187 us/byte Expect:

PCI 0.00188

GigE 0.008

PCI 0.00188

0.0118 us/byte


Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE CPU: PIII 800 MHz PCI:32 bit 33 MHz 100Mbit NICs RedHat 5.2 Kernel 2.2.14 LAN 2 PC B2B FWHM ~3 s No change with packet size

Man – UCL LAN – MAN – SuperJANET4 – MAN – LAN London Man at 130Mbit on ATM 155Mbit links

Latency: Histograms

64 bytes UDP PC=PC req-resp

0

500

1000

1500

2000

2500

3000

0 50 100 150 200 250 300

Latency us

N(t

)

1400 bytes UDP PC=PC req-resp

0

1000

2000

3000

4000

5000

0 50 100 150 200 250 300

Latency us

N(t

)1024 bytes SJ4 man_UCL

0

100

200

300

400

500

600

6000 8000 10000 12000 14000 16000 18000

Latency us

N(t

)

1400 bytes SJ4 man_UCL

0

100

200

300

400

500

600

6000 8000 10000 12000 14000 16000 18000Latency us

N(t

) TCP1400

UDP 1400


UDP IntelPro1000 : P4DP6 64bit 66MHz PCI slot4

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40

Transmit Time per frame usR

ecv

Wir

e ra

te M

bit

s/s

50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes

UDP Throughput: Intel Pro/1000 on B2B P4DP6

Max throughput 950Mbit/s Some throughput drop for packets

>1000 bytes

Loss NIC dependent Loss not due to user Kernel

moves

Traced to discards in the receiving IP layer ???

Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 4: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel 2.4.14

UDP IntelPro1000 : P4DP6 64bit 66MHz PCI slot4

0

10

20

30

40

50

60

70

80

90

100

0 5 10 15 20 25 30 35 40

Transmit Time per frame us

% P

acket

loss



UDP Throughput: SysKonnect SK-9843 B2B Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 33 MHz RedHat 7.1 Kernel 2.4.14

UDP SysKonnect

0

100

200

300

400

500

600

700

800

0 5 10 15 20 25 30 35 40


Rec

v W

ire

rate

Mb

its/

s


Max throughput 690Mbit/s No packet loss

Packet loss during dropUDP SysKonnnect : 370DLE 64bit 33MHz

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 5 10 15 20 25 30 35 40


% P

acket

loss



European Topology: NRNs & Geant

SuperJANET4

CERN

UvA

Manc

SURFnet


Gigabit Throughput on the Production WAN

Manc - RAL 570 Mbit/s 91% of the 622 Mbit access link

between SuperJANET4 and RAL 1472 bytes propagation ~21s

Manc-UvA (SARA) 750 Mbit/s SJANET4 + Geant + SURFnet

Manc – CERN 460 Mbit/s CERN PC had a 32 bit PCI bus

U D P M an -C E R N G ig 19 M ay 02

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40

Transm it T im e per fram e us

Re

cv

Wir

e r

ate

Mb

its

/s


U D P M an -R AL G ig 21 Ap r 02

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40


Re

cv

Wir

e r

ate

Mb

its

/s


U D P M an -U vA G ig 19 M ay 02

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40


Re

cv

Wir

e r

ate

Mb

its

/s

50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1472 bytes


UDP Throughput: SLAC - Man

UDP SLAC - Man 31May 02

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40

Transmit Time per frame usR

ecv

Wir

e ra

te M

bits

/s


UDP SLAC - Man 31May 02

0

10

20

30

40

50

60

70

80

90

100

0 5 10 15 20 25 30 35 40


% P

acke

t lo

ss


SLAC – Manc 470 Mbit/s 75% of the 622 Mbit access link SuperJANET4 peers with ESnet

at 622Mbit in NY


Packet Loss: Amsterdam – Chicago Lambda

(16b of 18)

WS WSL2

fast->slowL2

slow->fastfast fasthigh RTT

Cees de Laat & R. H-JParis Mar 02

BW ~500 Mbit/s

Keep 2 Drop 1L2 Buffers full

Packet Number

Time s


UDP Frame Jitter (Nov 01)UDP Frame Jitter (Nov 01)

1400 bytes man_ral w=80us Jitter

0

50

100

150

200

250

300

350

0 20 40 60 80 100 120 140 160 180 200

Inter-frame spacing us

N(t

)

1400 bytes man_ucl w=80us Jitter

0100200300400500600700800

0 20 40 60 80 100 120 140 160 180 200


N(t

)

1400 bytes man_cern w=80us Jitter

0

50

100

150

200

250

300

0 100 200 300 400 500 600 700


N(t

)

1400 bytes man_lyon w=80us Jitter

0

10

20

30

40

50

0 100 200 300 400 500 600 700


N(t

)

1400 bytes man_sara w=80us Jitter

0

100

200

300

400

500

0 20 40 60 80 100 120 140 160


N(t

)

1400 bytes man_sara w=80us Jitter

1

10

100

1000

0 200 400 600 800 1000 1200 1400 1600


N(t

)

Interrupt Coalescence Lon MAN~130 Mbit

Via CERN


TCPstream TCP Throughput vs : msg size, transmit spacing, n-streams,

window size 1 control TCP link n test streams Memory – memory Only 1 process – no threads – no context switching Streams treated equally not round Robin but next available stream At the receiver record

The time of first and last message received The amount of data ( CPU load, Number of interrupts )

Use the Pentium CPU cycle counter for times and delay Few lines of user code

Tells us about: Behavior of the IP stack Available TCP throughput of the LAN / MAN / WAN


Gigabit TCP on the Production WAN (01)

Throughput vs TCP buffer size

TCP window sizes in Mbytes calculated from RTT*bandwidth

0

100

200

300

400

500

600

0 1000000 2000000 3000000 4000000 5000000

T C P B uffe r s ize B y tes

Re

cv

. U

se

r d

ata

ra

te M

bit

/s

M an-UvA M an-C E R N

Link RTT ms BW Measured by UDP

Expected BW for 1 Gbit/s

750 Mbit/s 460 Mbit/s

Man – Ams

14.5 1.81 1.36

Man – CERN

21.4 2.68 1.23


Gigabit TCP on the Production WAN (10)

Throughput vs n-streams Default buffer size slope = ~25 Mbit/s/stream up to 9 streams then 15 Mbit/s/stream With larger buffers rate of increase per stream is larger Plateaus at about 7 streams giving a total throughput of ~400 Mbit/s

0

50

100

150

200

250

300

350

400

450

500

0 5 10 15 20 25

Num ber T C P s tream s

Re

ce

ive

d t

hro

ug

hp

ut

Mb

it/s 6 5536

524288

1048576

1600512

2097152

3145728

4194304


More Information Some URLs

UDPmon / TCPstream kit + writeuphttp://www.hep.man.ac.uk/~rich/net

ATLAS Investigation of the Performance of 100Mbit and Gigabit Ethernet Components Using Raw Ethernet Frames

http:// www.hep.man.ac.uk/~rich/atlas/atlas_net_note_draft5.pdf DataGrid WP7 Networking:

http://www.gridpp.ac.uk/wp7/index.html Motherboard and NIC Tests:

www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.pptPPNCG Home page with Stop Press:

http://ppncg.rl.ac.uk/PPNCG Page for monitoring Grid, HEP & Astronomy Sites http://icfamon.dl.ac.uk/ppncg/astronomy.htmlIEPM PingER home site:

http://www-iepm.slac.stanford.edu/IEPM-BW site:

http://www-iepm.slac.stanford.edu/bw


The Measurements (4)

PCI Activity Logic Analyzer with

PCI Probe cards in sending PC Gigabit Ethernet Fiber Probe Card PCI Probe cards in receiving PC

GigabitEthernet

ProbeCPU

mem

chipset

NIC

CPU

mem

NIC

chipset

Logic AnalyserDisplay


PCI: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14 SK300 1400 bytes sent Wait 100 us ~8 us for send or receive

Gigabit Ethernet frame 11.6 s

Send PCI

Receive PCI

Send data mem NIC

Receive data NIC mem


PCI: SysKonnect SK-9843 Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14

SK301 1400 bytes sent Wait 20 us

Sk303 1400 bytes sent Wait 10 us Frames are back-to-back Can drive at line speed Cannot go any faster !

Gig Eth frames back to back


PCI: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz RedHat 7.1 Kernel 2.4.14

IT66M212 1400 bytes sent Wait 11 us ~4.7us on send PCI bus PCI bus ~45% occupancy ~ 3.25 us on PCI for data recv

IT66M212 1400 bytes sent Wait 11 us Packets lost Action of pause packet?


PCI: Intel Pro/1000 on P4DP6

ITP4010 1400 bytes sent Wait 8 us ~5.14us on send PCI bus PCI bus ~68% occupancy ~ 2 us on PCI for data recv

Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 3-5: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel 2.4.14

caida bandwidth estimation meeting san diego june 2002 r. hughes-jones manchester udpmon and...

Documents

hughesjones manchester

time time slide

usbyte slide

lan man wan slide

university of manchester

regular intervals time

behaved latency slope

latency low