a 50-gb/s ip routerweb.engr.illinois.edu/~dhekne2/50gbps.pdf · growth of the internet in 2014 •...
TRANSCRIPT
![Page 1: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/1.jpg)
A 50-Gb/s IP Router~2Gb/s per author
Presenter: Ashutosh DheknePhD student, Computer Science,
University of Illinois at Urbana Champaign
![Page 2: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/2.jpg)
Why so many authors?
• Router developing is hard
• Lot of hardware and assembly level software expertise
• Complete understanding of architectural techniques
• Helps shorten the acknowledgement section
• Okay, let's get a bit more serious!
![Page 3: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/3.jpg)
In 2014Growth of the Internet
• Number of users and bandwidth requirement did not double every year.
• Cisco 7609 has 256 Gbps bandwidth• Alcatel Lucent and BT tested 1.4Tbps
link speed on 21 Jan 2014
• Paper claims that number of Internet users are doubling every year.
• The paper is from 1997 so over-estimates the growth a bit.
• Need for 50Gbps router, however, was real.
Cisco 7609 Alcatel-Lucent BT
![Page 4: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/4.jpg)
In 2014Average Packet Size
• The paper uses 1000bits as average packet size
• What to optimize?
• Number of packets processed or number of bytes processed?
• Avg 500-700
• Sharp rise at 1500
![Page 5: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/5.jpg)
Router Components – A simplistic view
*Routing Fabric
Routing Processor
Input Ports Output Ports
![Page 6: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/6.jpg)
Router Components – Innovations
*Routing Fabric
Network Processor
Input Ports Output Ports
Forwarding Engine
![Page 7: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/7.jpg)
Innovations – Forwarding Tables
*Routing Fabric
Network Processor
Input Ports
Forwarding Engine
• Complete set of routing tables at the forwarding engine
• Reduces probability of cache miss to a great degree
• Next Hop info only
![Page 8: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/8.jpg)
Innovations – Switched Backplane
*Routing Fabric
Network Processor
Input Ports
Forwarding Engine
• Switched backplane instead of shared bus
• Improves speed
![Page 9: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/9.jpg)
Innovations – separate board forward engine
*Routing Fabric
Network Processor
Input Ports
Forwarding Engine
• Forwarding engines on distinct boards
• Allows flexibility
• Allows various link layer protocols on the link card
![Page 10: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/10.jpg)
Innovations – QoS
*Routing Fabric
Network Processor
Input Ports
Forwarding Engine
• QoS function is added
• Forwarding engine classifies by assigning it to a flow
• QoS processor on the outbound card scedulesthe packet
QoSProcessor
![Page 11: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/11.jpg)
The Forwarding Engines – Hardware Overview
• Interesting tweaks• Place two pairs of int instructions
together
• Or, place 1 pair of int and 1 pair of float instruction
• Fit entire fwd code in 8kB Icache
• Fit 12000 routes in Scache [95%]
• Fit all routes in Bcache
• Allow the network processor to update this cache
• Alpha 21164 Processor• 415MHz, 64 bit, 32 regs
• 2 int, 2 float exec units
• 4 instruction groups
• 3 internal, 1 ext cache
• Very high data speeds
• Large instruction cache
• Large secondary cache
• Control over r/w sequencing
![Page 12: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/12.jpg)
Forwarding Engine – Hardware Operation
Alpha 21164 Processor
Request FIFO
Reply FIFO
In Packet ID Out Packet ID
Link Layer Len
IP Header (16B)Link Layer
IDSrc
CardSrc
PortType cast error unused
![Page 13: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/13.jpg)
Forwarding Engine – Software
• Confirm that packet header is from IPv4 datagram
• Confirm packet length and header length is reasonable
• Confirm no IP options are used
• Compute hash offset into route cache and load route
• Start loading the next header
Stage 1: Sanity Check Stage 2: Extended Route Stage 2: TTL, CRC
![Page 14: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/14.jpg)
Forwarding Engine – Software
• Does cached route match the destination?
• If not, consult the Bcache
• Is packet destined for this router? Then don’t update TTL
• Update TTL otherwise
Stage 1: Sanity Check Stage 2: Extended Route Check Stage 2: TTL, CRC
![Page 15: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/15.jpg)
Forwarding Engine – Software
• Updated TTL is written to IP header
• New checksum is written
• Routing info is extracted
• New link layer info computed
• Entire packet is written out
Stage 1: Sanity Check Stage 2: Extended Route Stage 2: TTL, CRC
![Page 16: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/16.jpg)
Instructions in Fast Path
Instructions Count % Exec Unit
And, bic, bis, ornot, xor 24 28 E0/E1
ext*, ins*, sll, srl, zap 23 27 E0
add*, sub*, s*add 8 9 E0/E1
branches 8 9 E1
ld* 6 7 E0/E1
addt, cmpt*, fcmov* 6 7 FA
st* 4 5 E0
fnop 4 5 FM
wmb 1 1 E0
nop 1 1 E0/E1
• Some hardware specific tweaks are employed
• nop and fnop used to pad instructions – eq to efficiency gains in x86
• Many bit operations required to extract data
![Page 17: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/17.jpg)
Tricks and Exceptions
• IP header checksum is not checked• Saves 9 cycles – about 21% • Errors are rare. They are corrected by TCP anyways• IPv6 does not implement error checking • What do modern routers do?
• What is not handled by fast code?• Destinations that miss the route cache• Headers with errors• Headers with IP options• Datagrams that require fragmentation• Multicast datagrams
![Page 18: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/18.jpg)
Switched Bus
• 15 port crossbar type switch is used
• Connects one source port to one destination port
• Multicast requires lowering of throughput and possible inherent fairness issues at any moment
• During a transfer 15 simultaneous transfers can happen
• Remember, only one output packet per port
![Page 19: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/19.jpg)
Head of line Blocking
• Head of line blocking occurs when the first packet in a queue is blocked because its destination interface is busy
• This causes all subsequent packets to be blocked even if their dest interfaces are free
• Avoided in a scheme where an interface gets to disclose all interfaces it is interested in
To 3 To 2 To 1
To 5 To 4 To 1
3
2
5
4
1x
From 3
From 4
![Page 20: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/20.jpg)
Matrix of Interests
• Disclose all interfaces for which your queue has at least one packet
• Allows transfer of a packet even from behind the queue
To 3 To 2 To 1
To 5 To 4 To 1
3
2
5
4
1From 3
From 4
x 1 2 3 4 5
1 x x
2 x
3 x
4 x
5 x
![Page 21: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/21.jpg)
Wavefront Method
• By allowing a particular transfer to go through, the algorithm disallows few other possible transfers
• Therefore, instead of rasterizing through all entries look at only the allowed entries.
• This speeds up the algo by reducing the number of comparisons
• 3 is allowed to send to 1. No need to check 41 and 51
x 1 2 3 4 5
1 x x
2 x
3 x
4 x
5 x
![Page 22: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/22.jpg)
Scheduling of switch
Epoch 1
Epoch 2
Epoch 3
Epoch 4
Source card has data for the destination card
Switch allocator decides to schedule transfer for forth epoch
Cards are informed about the imminent transfer; data path cards are informed to configure themselves
Data transfer takes place
16
cyc
les
16
cyc
les
16
cyc
les
16
cyc
les
![Page 23: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/23.jpg)
Line Card Design, Inbound, Outbound Packets
InputLine Card
OC-48c2.4Gbps
OC-12c622Mbps
HIPPI 800Mbps
Ethernet100Mbps
64 bytes pages
Alpha 21164 Processor
Forwarding Engine
ORCA 2C04A FPGA (QoS Processor)
OutputLine Card
Alpha 21064 Processor
Network Processor
![Page 24: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/24.jpg)
Conclusions
• Provides a huge impetus to the router industry to start working towards faster core routers.
• It shows that it is possible to examine every packet header and make decisions fast enough for multi Gbps routers
• It is unique in discussing the actual assembly level tricks to achieve the required throughput
• The paper creates an appreciation of the issues faced by the core routers on the Internet
Routers can keep up!Right selection of hardware, placement and software is crucial.
![Page 25: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/25.jpg)
Open Questions
• The network processor ARPs all possible addresses at a low frequency. Is doing such a thing acceptable on the Internet backbone?
• When the Network Processor writes the newly discovered routes to the B-cache, how does it synchronize with possible reads from the forwarding engine?
• How often do the S-cache flush because of a change in B-cache?
• How expensive was it to build this router?
• No security aspects such as mitigation of DoS attacks not discussed
• No audit trails
![Page 26: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/26.jpg)
Open Questions
• The network processor ARPs all possible addresses at a low frequency. Is doing such a thing acceptable on the Internet backbone?
• When the Network Processor writes the newly discovered routes to the B-cache, how does it synchronize with possible reads from the forwarding engine?
• How often do the S-cache flush because of a change in B-cache?
• How expensive was it to build this router?
• No security aspects such as mitigation of DoS attacks not discussed
• No audit trails
![Page 27: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/27.jpg)
Today’s State of the Art
• Routing using GPU [link to paper]
• Cisco nPower X1, with 336 multi-threaded processor cores [link]
• The Alcatel-Lucent 7950 Extensible Routing System has 16Tbps bandwidth [link]
• Cisco’s Carrier Routing System routes at 400Gbps per link [link]
![Page 28: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/28.jpg)
[reference]
Asia-Pacific6007Gbps
US-Canada12100Gbps
Europe27718Gbps
Africa397Gbps
Latin-America3567Gbps
![Page 29: A 50-Gb/s IP Routerweb.engr.illinois.edu/~dhekne2/50Gbps.pdf · Growth of the Internet In 2014 • Number of users and bandwidth requirement did not double every year. • Cisco 7609](https://reader031.vdocuments.mx/reader031/viewer/2022022423/5aa024e27f8b9a84178dc2bd/html5/thumbnails/29.jpg)
Thank You!