jumbo frames in ams-ix
DESCRIPTION
TRANSCRIPT
![Page 1: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/1.jpg)
Jumbo Frames in AMS-IXversion 0.5
AMS-IX GM33. Amsterdam16 November 2011
Maksym [email protected]
Take part in survey: https://www.surveymonkey.com/s/TGS5T2D
![Page 2: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/2.jpg)
2
Content
1. AMS-IX survey and possible
implementations
2. Benefits of Jumbo Frames
3. Obstacles to Jumbo Frames
4. Our statistics
5. Summary
![Page 3: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/3.jpg)
1. AMS-IX survey andpossible implementations
Jumbo Framesin AMS-IX
Maksym [email protected]
AMS-IX GM33. Amsterdam16 November 2011
![Page 4: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/4.jpg)
4
AMS-IX survey
115 participants
![Page 5: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/5.jpg)
5
Possible solutions
Two ways to implement Jumbo Frames:1.Change MTU for the existing VLAN
2.Make a new VLAN with Jumbo Frame support
![Page 6: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/6.jpg)
6
1. Modify existing VLAN
Cons1.Customers don’t like changes in existing VLAN
2.No official standards
3.No MTU negotiation protocol
4.Network upgrade
Pros1.Technical possibility: our equipment supports it
![Page 7: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/7.jpg)
7
AMS-IX survey
115 participants
![Page 8: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/8.jpg)
8
Cons1.Customers don’t like changes in existing VLAN
2.No official standards
3.No MTU negotiation protocol
4.Network upgrade
Pros1.Technical possibility: our equipment supports it
![Page 9: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/9.jpg)
9
2. Make a new VLAN
Cons1.New port for each customer: will anyone pay for it?
2.No official standards: what size of Jumbo Frames?
Pros1.Technical possibility: our equipment supports it
![Page 10: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/10.jpg)
10
Cons1.New port for each customer: will anyone pay for it?
2.No official standards: what size of Jumbo Frames?
Research is needed
2. Make a new VLAN
![Page 11: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/11.jpg)
2. Benefits of Jumbo Frames
Jumbo Framesin AMS-IX
Maksym [email protected]
AMS-IX GM33. Amsterdam16 November 2011
![Page 12: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/12.jpg)
12
2.1. CPU load
Alteon Networks.
Extended Frame
sizes for Next
Generations
Ethernets
http://staff.psc.edu/mathis/MTU/AlteonExtendedFrames_W0601.pdf
![Page 13: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/13.jpg)
13
2.1. CPU load
Dell Inc. Internal Report
![Page 14: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/14.jpg)
14
2.2. Network overhead
Theoretical maximum throughout on 1Gbit
9000 bytes
TCP: 990.042 Mbps
UDP: 992.697 Mbps
1500 bytes
TCP: 941.482 Mbps
UDP: 957.087 Mbps
TCP: +4.8%, UDP: +3.5%
http://sd.wareonearth.com/~phil/net/overhead/
![Page 15: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/15.jpg)
15
2.3. TCP performance
Frame size = 9000 bytes
Throughput = 40Mbit/s
Frame size = 1500 bytes
Throughput = 6.5Mbit/s
http://www2.rad.com/networks/2003/largemtu/tcperf.htm
Double MTU size - Double TCP throughout
MSS = MTU – 40 RTT = 40ms Loss = 0.01%
![Page 16: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/16.jpg)
16
2.1. Benefits
1. Less CPU load
2. Less network overhead
3. Better TCP performance
![Page 17: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/17.jpg)
3. Obstacles to Jumbo Frames
Jumbo Framesin AMS-IX
Maksym [email protected]
AMS-IX GM33. Amsterdam16 November 2011
![Page 18: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/18.jpg)
18
3.1. Standards: header
1. IEEE 802.3 specification – 1518 bytes
2. 802.1Q (VLANs) – 1522 bytes
3. 802.1ad (Provider Bridge) – 1526 bytes
4. 802.1AS – 2000 bytes
5. 802.3AE – 1582 bytes
6. MPLS – 1518 bytes + N * 4 bytes
But they say about Ethernet header only i.e. payload still 1500 bytes
![Page 19: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/19.jpg)
19
3.1. Standards: payload
1. Ethernet (IEEE 802.3 specification) – 1518 bytes
2. FCoE (T11 specification) – 2166 bytes
3. iSCSI (VMWare de-facto ?) – 9000 bytes
What should be used for AMS-IX?
![Page 20: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/20.jpg)
20
3.1. Standard: terms
1. Baby Giant – MPLS, 802.1Q, 802.1ad, 802.3AE
2. Mini Jumbo – FCoE
3. Giant Jumbo – payload more that 1500 bytes
4. Payload MTU – size of payload
5. Link MTU – size headers plus payload
![Page 21: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/21.jpg)
21
3.2. Delay and jitter
Double MTU size - Double delay
http://www.ethernetalliance.org/files/static_page_files/DCB whitepaper.pdf
![Page 22: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/22.jpg)
22
3.3. Buffer size
Port_Buffers = 2 * MTU + link_delay * link_speed
Switch_Buffers = Num_Ports * Num_Queues * Port_Buffer
link_delay = 0, Num_Ports = 24
MTU = 1518
Num_Queues = 8
Buffers = 582 912
MTU = 9018
Num_Queues = 8
Buffers = 3 462 912
Double MTU size - Double buffers
http://www.ethernetalliance.org/files/static_page_files/DCB whitepaper.pdf
![Page 23: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/23.jpg)
23
3.4. PMTUD
1. PMTUD – Path MTU Discovery (RFC 1191)
2. Easy to break (firewalls, wrong stack implementation)
3. Difficult to debug (RFC 2923)
4. TCP timeouts if it doesn’t work
![Page 24: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/24.jpg)
24
3. Obstacles
1. No standard/agreement for size of Jumbo Frames
2. Increased transmission time, packet delay, jitter, etc
3. Require bigger buffers on equipment
4. Path MTU Discovery (PMTUD) doesn’t work
![Page 26: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/26.jpg)
26
4.1. Frame size statistics
https://www.ams-ix.net/sflow-stats/size/
Twice more traffic with minimal size that with maximum size
![Page 27: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/27.jpg)
27
4.2. RIPE63 feedback
Only 7 answers Please, take part https://www.surveymonkey.com/s/TGS5T2D
1. Maximum MTU size is enabled on your backbone?
2. Does your backbone transfer packets with payload more that 1500 bytes?
3. Do you like an idea of two interfaces?
4. Do you use Jumbo Frames with your customers?
1. 2 – 1500 , 2 – 9000, 3 – 9216
2. 6 – yes, 1 – no
3. 1 – yes, 6 – no
4. 2 – yes, 5 – no
![Page 29: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/29.jpg)
29
5. Theory
Obstacles1.No standard/agreement for size of Jumbo Frames
2.Increase transmission time, packet delay, jitter, etc.
3.Require bigger buffers on equipment
4.Path MTU Discovery doesn’t work
Benefits1.Less CPU load
2.Less network packet overhead
3.Better TCP performance
![Page 30: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/30.jpg)
30
5. Applications
Obstacles1.Inter-process communication (IPC)
2.Protocols using small packets (DNS, VoIP, etc)
3.Interoperability (no standards, broken PMTUD)
Benefits1.Data transfer (Backups/Clusters/NFS/NNTP)
2.VPNs with payload 1500 bytes
3.SAN (FCoE/iSCSI)
![Page 31: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/31.jpg)
31
5. Next steps1. Postpone for now
2. Get feedback from our customers
![Page 32: Jumbo frames in AMS-IX](https://reader036.vdocuments.mx/reader036/viewer/2022081422/553992444a795979798b496b/html5/thumbnails/32.jpg)
Take part in survey: https://www.surveymonkey.com/s/TGS5T2D
Jumbo Framesin AMS-IX
Maksym [email protected]
AMS-IX GM33. Amsterdam16 November 2011
Comments?