eslea pmb, manchester, 23 nov 2006, r. hughes-jones manchester 1 protocols progress with current...

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester1

Protocols

Progress with Current Work.

Richard Hughes-Jones The University of Manchester

www.hep.man.ac.uk/~rich/ then “Talks”

http://www.hep.man.ac.uk/~rich/


iGrid2002 monolithic code Convert to use pthreads

control Data input Data output

Code branch for Simon’s file transfer tests

Work on vlbi_recv: Output thread polled for data in the ring buffer – burned CPU Input thread signals output thread when there is work to do – else wait on

semaphore – packet loss at high rate, variable thoughput Output thread uses sched_yield() when no work to do – CPU used

Add code for: MarkV card and PCEVN interface

Measure: throughput, packet loss, re-ordering, 1-way delay

Multi-flow Network performance – being set up Nov/Dec06

vlbi_udp: UDP on the WAN


vlbi_udp_recv_k269_110d_w12_29Aug06

980

985

990

995

1000

0 500 1000 1500 2000 2500 3000

Time between frames us

Wir

e R

ate

Mb

it/s

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Packet

loss

Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 99.7% id, 0.3% wa, 0.0% hi, 0.0% si Cpu1 : 11.3% us, 88.7% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si Cpu2 : 0.3% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.0% hi, 0.0% si Cpu3 : 9.3% us, 15.6% sy, 0.0% ni, 37.5% id, 0.0% wa, 1.3% hi, 36.2% si

vlbi_udp: B2B UDP Tests Kernel 2.6.9 vlbi_recv sched_yield() Wait 12 us Stable throughput

999 Mbit/s variation less than 1 Mbit/s No packet loss

Inter-packet time

Processing time mean 0.1005 sigma 0.1438

CPU load:


1

100

10000

1000000

100000000

10000000000

0 20 40 60 80 100 120 140 160

Time between frames us

N(t

)


1

100

10000

1000000

100000000

10000000000

0 20 40 60 80 100 120

Packet process time us

N(t

)


Dedicated

Gbit link

OnsalaSweden

Gbit link

Jodrell BankUK

DwingelooDWDM linkMedicina

Italy

Chalmers University

of Technolog

y, Gothenbur

g

TorunPoland

Gbit link

Metsähovi

vlbi_udp: Multi-site Streams


TCP: tcpdelay How does TCP move CBR data? Want to examine how TCP moves Constant Bit Rate Data VLBI Application Protocol tcpdelay a test program:

instrumented TCP program emulates sending CBR Data.

Records relative 1-way delay Web100 Record TCP Stack activity

n bytes

Number of packets

Wait timetime

Data1

●●●

Timestamp1

Time

TCP & Network Receiver

Timestamp2

Sender

Data2Timestamp4

Timestamp5

Data4

Timestamp3

Data3

Packet loss

And now with Packet Loss


TCP: tcpdelay Visualising the Results

If Throughput NOT limited by TCP buffer size / Cwnd maybe we can re-sync with CBR arrival times.

Need to store CBR messages during the Cwind drop in the TCP buffer Then transmit Faster than the CBR rate to catch up

Message number / Time

Packet lossDelay in stream

Expected arrival time at CBR

Arrival timethroughput

1Slope

Stephen Kershaw


TCP: tcpdelay JB-Manc Message size: 1448 Bytes Wait time: 22 us Data Rate: 525 Mbit/s Route: JB-Man RTT ~1 ms TCP buffer 2MB Drop 1 in 10,000 packets ~2.5-3 ms increase in time

for about 2000 messagesie ~ 44 ms

Classic Cwnd behaviour Cwnd dip corresponds to

~ 1.2M bytes data “Delayed” (~810 packets)

Peak Throughput ~ 620 Mbit/s

gig7-jbgig1 rtt:1ms buf:64k

0.00

1.00

2.00

3.00

4.00

5.00

6.00

0 10 20 30 40 50 60 70

Message No 1000s

1 w

ay

de

lay

ms

0

200

400

600

800

1000

0 500 1000 1500 2000 2500 3000time ms

TC

PA

chiv

e M

bit

/s

0

20000

40000

60000

80000

100000

0 500 1000 1500 2000 2500 3000time ms

Cw

nd


Arrival Times: UKLight JB-JIVE-Manc Message size: 1448 Bytes Wait time: 22 us Data Rate: 525 Mbit/s Route:

JB-UKLight-JIVE-UKLight-Man

RTT ~27 ms

TCP buffer 32M bytes

BDP @512Mbit 1.8Mbyte Estimate catchup possible

if loss < 1 in 1.24M

Data needed forJIVE-Manc ~27msChicago-Manc ~120 ms

Have ~30 GBytes!!!

0 1 2 3 4 5 6 7 8 9 10

x 104

5

10

15

20

25

30

35

40

45

50

Message number

Tim

e /

s

Effect of loss rate on message arrival time

Drop 1 in 5k

Drop 1 in 10k

Drop 1 in 20kDrop 1 in 40k

No loss

Stephen Kershaw

Presented at the Haystack Workshop


TCP: TCP Stacks, Sharing, Reverse Traffic

Delayed by Provision of UKLight link Manc - Starlight PC installed in Starlight and Manchester Sep06 Udpmon tests Good

Plateau ~990 Mbit/s wire rate No packet Loss Same in both directions

TCP studies: Work Now in progress

gig7-chi1_10Sep06

0100200300400500600700800900

1000

0 10 20 30 40Spacing between frames us

Rec

v W

ire r

ate

Mbi

t/s

50 bytes

100 bytes

200 bytes

400 bytes

600 bytes

800 bytes

1000 bytes

1200 bytes

1400 bytes

1472 bytes


DCCP: The Application View Stephen & Richard with help from Andrea Had problems with Fedora Core 6 using stable kernel 2.6.19-rc1

DCCP data packets never reached the receiving TSAP ! Verify with tcpdump Using 2.6.19-rc5-g73fd2531-dirty

Ported udpmon to dccpmon Some system calls don’t work

dccpmon tests Plateau ~990 Mbit/s wire rate No packet Loss Receive system crashed!

Iperf tests 940Mbps, back-to-back

Need more instrumentation in DCCP Eg a line in /proc/sys/snmp

zeus15-atb79_29Sep06

0100200300400500600700800900

1000

0 10 20 30 40Spacing between frames us

Rec

v W

ire r

ate

Mbi

t/s

800 bytes

1000 bytes

1200 bytes

1400 bytes

1424 bytes


10 Gigabit Ethernet Lab 10 Gigabit Test Lab now set up in Manchester

Cisco 7600 Cross Campus λ <1ms Neterion NICs 4 Myricom 10 Gbit NICs – delivery this week Chelsio being purchased

Boston/Supermicro X7DBE PCs Two Dual Core Intel Xeon

Woodcrest 5130 2 GHz PCI-e and PCI-X

B2B performance so far SuperMicro X6DHE-G2 Kernel (2.6.13) & Driver dependent! One iperf TCP data stream 4 Gbit/s Two bi-directional iperf TCP data streams 3.8 & 2.2 Gbit/s UDP Disappointing

Installed Fedora Core5 Kernels 2.6.17 & 2.6.18 (+web100 + packet drop) & 2.6.19 on the Intel dual-core PCs


ESLEA-FABRIC:4 Gbit flows over GÉANT Set up 4 Gigabit Lightpath Between GÉANT PoPs

Collaboration with Dante GÉANT Development Network London – Amsterdam

and GÉANT Lightpath service CERN – Poznan PCs in their PoPs with 10 Gigabit NICs

VLBI Tests: UDP Performance

Throughput, jitter, packet loss, 1-way delay, stability Continuous (days) Data Flows – VLBI_UDP and multi-Gigabit TCP performance with current kernels Experience for FPGA Ethernet packet systems

Dante Interests: multi-Gigabit TCP performance The effect of (Alcatel) buffer size on bursty TCP using BW limited

Lightpaths


Options Using the GÉANT LightPaths Set up 4 Gigabit Lightpath Between GÉANT PoPs

Collaboration with Dante PCs in Dante PoPs

10 Gigabit SDH backbone Alkatel 1678 MCC Node location:

Budapest Geneva Frankfurt Milan Paris Poznan Prague Vienna

Can do traffic routingso make long rtt paths

Ideal: London Copenhagen


Network/PC Booking System Based on Meeting Room Booking System Divide into Links and End systems Hard work by Stephen Kershaw Tesing with VLBI


Any Questions?


Backup Slides

eslea pmb, manchester, 23 nov 2006, r. hughes-jones manchester 1 protocols progress with current...

Documents

ms tcp buffer

cbr data

cbr rate

tcp stacks

tcp bufferthen

tcp buffer size cwnd

instrumented tcp program

uklight jbjive