qos testing between cnaf and pisa test results nov 29 2005
TRANSCRIPT
QoS Testing between CNAF and Pisa
Test results
Nov 29 2005
Outline
• Objectives
• QoS scenarios
• Best-effort TCP performance results
• Test on differentiation of outgoing traffic from INFN-Pisa to CNAF
• Best-effort UDP performance results
• Future work
Objectives
• Test traffic differentiation techniques on the customer edge routers of the INFN local area networks, for allocation of a minimum guaranteed bandwidth to a number of traffic classes– Traffic classification and marking– Scheduling: Weighted Round Robin (WRR)– No policing (for the moment)
Testbed configuration
• CNAF:– Juniper M10 (dedicated to testing)– GigaEthernet switch Extreme Summit 400– Two end-nodes (64 bit PCI slot network interface, 1 GEthernet),
connected to the Service Challenge GigaEthernet switch– Capacity to/from GARR: 2 Gbit (boundling of two GEthernet
interfaces)
• PISA– Juniper M7 (production router)– Two end-node (64 bit PCI-X slot network interface, 1 GEthernet;
1 Fast-Ethernet interface )– Capacity to/from GARR: 1 Gbit
Traffic Classes1. User traffic:
• Bandwidth range: [300, 1000] Mb/s• Minimum 30% of link capacity guaranteed in case of congestion• TOS Preference codepoint: 000 (best-effort)
2. Service Challenge traffic:• Bandwidth range: [700, 1000] Mb/s• Minimum 70% of link capacity guaranteed in case of congestion• TOS Preference codepoit: 001 (assured-rate)
• Purposes of traffic differentiation:– Allocation of minimum guaranteed bandwidth to input/output legacy
and Service Challenge traffic classes in case of congestion– Fair distribution of link capacity in case of congestion– Possibility to get more bandwidth than the minimum guaranteed in
case of spare link capacity
Scenario 1: differentiation of outgoing traffic (from Pisa)
GARR
GARR
CNAF INFN Pisa
70%
30%
Juniper M10 Juniper M7
1 Gb/s2.0 Gb/sService Challenge
Service Challenge
bottleneck
Users
Scenario 2: differentiation of incoming traffic (to Pisa)
GARR
GARR
CNAF INFN Pisa
70%
30%
Juniper M10 Juniper M7
1 Gb/s2.0 Gb/sService Challenge
Service Challenge
bottleneck
Users
Scenario 3: differentiation of outgoing traffic (from CNAF)
GARR GARR
CNAF
PisaJuniper M10
Torino
Legnaro
MilanoBari
2.0 Gb/s
20%
20%
20%
20%
20%
bottleneck
Scenario 4: differentiation of incoming traffic (to CNAF)
GARR
GARRCNAF
PisaJuniper M10
Torino
Legnaro
Milano
Bari
2.0 Gb/s
1Gb/s
1Gb/s
1Gb/s
100Mb/s1Gb/s
bottleneck
Best-effort TCP performance
• Initially asymmetric throughput to/from Pisa (probably due to misconfiguration in the PISA MAN), now solved
• Performance (with network configuration fixed):
Best-effort TCP performance:test CNAF INFN Pisa
Test TCP CNAF --> Pisa
860870
880890
900910
920930
940
1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 103 109 115
n° istances
Th
rou
gp
ut
(Mb
/s)
Test TCP Pisa --> CNAF
850860870880890900910920930940
1 9 17 25 33 41 49 57 65 73 81 89 97 105 113
n° istances
Th
rou
gh
pu
t (M
b/s
)
Differentiation of outgoing traffic (from Pisa)
GARR
GARR
CNAF INFN Pisa
SC1
70%
Juniper M10 Juniper M7
1 Gb/s2.0 Gb/s
Service Challenge
bottleneck
Users
30%SC2
BE1
BE2
BE2
BE1
AR
AR
Test on differentiation of outgoing traffic (1/2)
Differentiation of outgoing traffic in router M7 (Pisa)
0%
20%
40%
60%
80%
100%
1 2 3 4
n° test
thro
ug
hp
ut
%
Traffico AR
Traffico BE2
Traffico BE1
N° test
Sender(command - line) [Mb/s]
Riceiver[Mb/s]
ProtocolDroppedPackets
Percentuage
1BE1 = 90
BE2 = 250AR = 750
BE1 = 72BE2 = 198AR = 670
UDPUDPUDP
√BE1 = 8
BE2 = 21AR = 71
2BE1 = 90
BE2 = 280AR = 700
BE1 = 67BE2 = 208AR = 663
UDPUDPUDP
√BE1 = 7
BE2 = 22AR = 71
3BE1AR
TCPTCP × BE1 = 10,5
AR = 89,5
4BE1 =
BE2 = 280AR =
BE1 = 68,5BE2 = 280AR = 565
TCPUDPTCP
×BE1 = 7,5
BE2 = 30,7AR =61,8
Differentiation of outgoing traffic on router M7 (Pisa)
0%
20%
40%
60%
80%
100%
5 6 7 8 9
n° test
Thro
ughp
ut % Traffico AR
Traffico BE2
Traffico BE1
N° test
Sender(command
- line) [Mb/s]
Riceiver[Mb/s]
ProtocolDroppedPackets
Percentuage
5BE1 = 90
BE2 =AR = 750
BE1 = 90,5BE2 = 47AR = 783
UDPTCPUDP
×BE1 = 9,9BE2 = 5,1AR = 85
6BE1 = 90
BE2 = 250AR =
BE1 = 90,4BE2 = 250AR = 583
UDPUDPTCP
×BE1 = 10,6BE2 = 27AR = 62,4
7BE1 = BE2 =
AR = 750
BE1 = 76,3BE2 = 68AR = 784
TCPTCPUDP
×BE1 = 9,7BE2 = 7,2AR = 83,1
8BE1 = 80
BE2 = 250AR =
BE1 = 80BE2 = 250AR = 600
UDP UDPTCP (*20)
×BE1 = 8,4 BE2 = 26,9AR = 64,5
9BE1 = 80
BE2 = 250AR =
BE1 = 80BE2 = 250AR = 600
UDP UDPTCP
(*100)
×BE1 = 8,4 BE2 = 26,9AR = 64,5
Test on differentiation of outgoing traffic (2/2)
Test Marker
Traffic Tos binary Tos hexadecimal
Best-effort 000 (0x00)
Assured-rate 001 0x20
[datatag1test] /home/bencivenni > tcpdump -c 20 host 212.189.152.177 -i eth0
• tcpdump: listening on eth0• 11:18:33.254874 192.135.23.200.20000 > qos1.pi.infn.it.33233: .
ack 2468187583 win 18432 (DF)• 11:18:33.254878 qos1.pi.infn.it.33233 > 192.135.23.200.20000: .
1:1461(1460) ack 0 win 22 (DF) [tos 0x20]
Best-effort UDP performance (1/3)
• Performance issues in the LAN CNAF, probably due to the NIC hardware used
Test LAN CNAF TCP/UDP
0
200
400
600
800
1000
1 13 25 37 49 61 73 85 97 109
n° istances
Th
rou
gh
pu
t (M
b/s
)
UDP b=800 Mb/s
TCP
Best-effort UDP performance 1/3:traffic profile
• Purpose of test: verify the inpact of production traffic on the UDP constant bit rate traffic profile generated at the source (constant inter-packet gap)
• Production traffic mixing with test traffic on most of the end-to-end path under test
Best-effort UDP performance 3/4
Test UDP INFN Pisa --> CNAFbandwidth = 800 Mb/s
0
200
400
600
800
1000
1 17 33 49 65 81 97 113
n° istances
Th
rou
gh
pu
t (M
b/s
)
HOST PISA (Sender)
HOST CNAF (Riceiver)
Differences between CNAF and INFN Pisa
PISA
• Operating System:
Linux 2.4.21-32.0.1.ELsmp
• iperf version 1.7.0 • MTU = 1500 Byte• BUS PCI
CNAF
• Operating System:
Linux 2.4.21- 20.EL.cernsmp
• iperf version 2.0.1• MTU = 5000 Byte• BUS PCI-X
Xeon
GARR GARR
Xeon
opteron
opteron
datatag1test
test7200a
nettetst1
nettest2
192.135.23.200
Eth0 1Gb/s 64 bit
192.135.23.201
Eth0 1Gb/s 64 bit
192.135.23.202
Eth1 1Gb/s 64bit
192.168.1.11
Eth1 1Gb/s 32 bit
192.168.1.14
Eth11Gb/s 32 bit
Eth2 10Gb/s 64 bit/133 Mhz
192.168.1.101
192.168.1.102
Eth2 10Gb/s 64 bit/133 Mhz
Switch Extreme Summit 400-48pt
Jumbo frames 9216
Switch Extreme Summit 400-48pt
Jumbo frames 9216
RouterJuniper M10
131.154.99.32
131.154.99.201
131.154.100.1
131.154.100.2
Test SC
2 Gb/s Balanced Boundling
SC
Eth2
Eth2
Eth0
Eth0
Athlon
RouterJuniper M7
INFNPisa
212.189.152.177 Eth0
1gb/s 64 bit/133 Mhz
1 Gb/s
Iperf: Command line option
• -l (iperf_len): The length of buffers to read or write. Iperf works by writing an array of len bytes a number of times. Default is 8 KB for TCP, 1470 bytes for UDP. Note for UDP, this is the datagram size and needs to be lowered when using IPv6 addressing to 1450 or less to avoid fragmentation.
UDP Test : Pisa CNAFSender
(command line) (Mb/s)
-l
(Byte)
Sender
(real) (Mb/s)
Riceiver
(Mb/s)
-l
(Byte)Packet loss
800 × 840470-819
× 20%
900 × 905422-800
× 16%
800 1500 800 800 1500 0%
900 1690 901 900 1690 0%*
900 1690 901 900 2000 0%
UDP Test : CNAF Pisa
Sender
(command line) (Mb/s)
-l
(Byte)
Sender
(real) (Mb/s)
Riceiver
(Mb/s)
-l
(Byte)Packet loss
800 ×463-805
449-788
× 2,2%
900 ×500-785
482-780
× 2,2%
800 1500 800 × 1500 ×
900 1690 901 × 1690 ×
900 1690 901 × 2000 ×
Sender
(command line) (Mb/s)
-l
(Byte)
Sender
(real) (Mb/s)
Riceiver
(Mb/s)
-l
(Byte)Packet loss
800 ×466-696
460-648
× 0,2%
900 ×509-739
503-718
× 16%
800 1500 800 670 1500 14%
900 1690 901 702 1690 22%
900 1690 901 702 2000 22%
UDP Test : LAN CNAF
Router Configuration: Class of Service (1/2)
class-of-service { classifiers { # Definisco traffico best-effort tutto ciò inet-precedence inet-precedence-service-challenge{ che entra con un TOS = 000 # forwarding-class best-effort { loss-priority high code-points 000; } } forwarding-classes { # Definisco le 2 code # queue 0 best-effort; queue 2 ip-premium; } interfaces { ge-y/y/y { scheduler-map service-challenge-ar-be; unit 0 { rewrite-rules { # All’interfaccia d’uscita marco i pacchetti del inet-precedence ar-mark; traffico assured-rate con il TOS opportuno } che verrà poi specificato nelle rewrite-rules # } ............ # Associo alle interfacce fastethrnet il traffico best-effort #
Router Configuration: Class of Service (2/2)
rewrite-rules { inet-precedence ar-mark { forwarding-class assured-rate { loss-priority low code-point 001; # TOS = 001 → Assured forwarding # } } } scheduler-maps { service-challenge-ar-be { forwarding-class assured-rate scheduler sch-assured-rate; forwarding-class best-effort scheduler sch-best-effort; } } schedulers { # Servo per il 70% del tempo la coda del sch-assured-rate traffico assurde-srate e per il 30% quella transmit-rate percent 70; del traffico best-effort # buffer-size percent 70; priority high; } sch-best-effort { transmit-rate percent 30; buffer-size percent 30; priority low;
Router Configuration: Firewall
firewall { filter assured-rate-traffic { #Etichetto come assured-rate- term term-192.135.23.200{ traffic il traffico generato da from { 212.189.152.177 e destinato source-address { a 192.135.23.200 # 212.189.152.177/32; } destination-address { 192.135.23.200/32; } protocol [ udp tcp ]; } then { accept; forwarding-class assured-rate; } } term term-192.135.23.201{ #Etichetto come best-effort- from { traffic il traffico generato da source-address { 212.189.152.177 e destinato
212.189.152.177/32; a 192.135.23.201 # } destination-address { 192.135.23.201/32; } protocol [ udp tcp ];then { accept; forwarding-class best-effort; } }
term default-action { # Tutto ciò che non è stato specificato
then { precedentemente viene accettato e accept; classificato come traffico best forwarding-class best-effort; effort # } } } }