Download - Backbone Routers
-
8/2/2019 Backbone Routers
1/18
Internet Backbone Routers and Evolving Internet Design s 1
Internet Backbone Routers and Evolving Internet Design
Chuck SemeriaMarketing EngineerJuniper NetworksSeptember 16, 1998
Contents Introduction 1Im pact of Rapid Grow th on ISPs 2
Delivering Solutions that Permit ISPs to Grow Their Networks 3
Essent ial Elements of a Routing System 4
Key Attributes for an Internet Backbone Router 5
M40 Internet Backbone Router Architecture 6
Routing Software: JUNOS Internet Software 6
Packet Lookup: The Internet Processor ASIC 7
Route Lookup 7
Programmability 8
Perform an ce In su ran ce 8
Atomic Updates 9
Tr affic Visib ility 10
Switch Fabric: Distributed Buffer Manager ASIC, I/ O Man ager ASIC,
and Shared Memory 10
Distributed Buffer Manager ASIC and Shared Memory 11
I/ O Manager ASIC 12
Line Cards 12
How Packets Traverse the M40 Packet Forwarding Engine 14
The M40 Internet Backbone Router in an ISP Network 15
Rock-Solid Reliabili ty Under Failure Conditions 15
M40 System Dep loym en t 17
Conclusion 18
Introduction
As we ap proach th e twen ty-first centu ry, the Internet continu es to experience extraord inary
growth . Any w ay one measures it, the grow th is remarkable on all frontsthe nu mber of
hosts, the num ber of users, the amou nt of traffic, the nu mber of links, the band wid th of
individual links, or the grow th rates of Internet Service Provider (ISP) networks.
-
8/2/2019 Backbone Routers
2/18
Introduction
2 s Internet Backbone Routers and Evolving Internet Design
Formerly, the ISP commu nity adap ted general-pu rpose equ ipmen t for use in the core of the
Internet. The explosive growth of the Internet has created a market for networ king equipmen t
that is specially built to solve the unique problems confronting Internet backbone providers.
This new class of equipmen t is required to scale the Internet no t only in terms of aggregate
bandw idth and raw throu ghpu t, but also in terms of software richness and control.
To achieve the objectives of providing increased band wid th and software richness, the
designers of new routing systems must achieve the forwarding p erformance p reviously foundonly in switches. However, delivering w ire-speed performance for variable-sized packets with
longest-match lookups is considerably more comp lex than th e relatively straightforw ard
switching goal of sup porting wire-speed p erformance for fixed-sized cells with fixed-length
lookups. In particular, packet processing can no longer remain a m icroprocessor-based or
microprocessor-assisted function, but mu st transition to an ASIC-based ap proach that
supp orts the evolution of routing software as the Internet environment changes.
The system cha llenges facing this new class of Internet backbone rou ter also are comp lex. New
routing system s must be deployed into existing OC-3 and OC-12 based cores with their related
intra-POP infrastructure, and they also must supp ort the transition to OC-48 based cores with
OC-12 and Gigabit Ethernet based intra-POP infrastructures. In add ition, new rou ting systems
mu st accelerate the evolu tion of the Internet from a best-effort service to a fun dam entally
reliable service. With the Internet em erging in its role as the new pu blic netw ork, users aredem and ing and expecting increased reliability and stability to sup por t mission-critical
app lications. The Internet cannot continu e to p rovide erratic best-effort service that generally
works but at other times fails. As they hav e come to expect with the Plain Old Telephone
Service (POTS), Internet users w ant to h ear d ial-tone an d receive quality service whenever
they wish to commun icate.
The Jun iper Netw orks M40 Internet backbone rou ter is the worlds most complete system for
Internet routing . It is the first system that has been specially built to combine Intern et scale,
Internet control, and u np aralleled forw arding performa nce. By integrating th e flexibility and
control of router software with the p acket-forward ing performance of a switch and p roviding
rock-solid stability during exceptional cond itions, the M40 system is designed to serve as the
found ation of the op tical Internet, while sup porting the Internets transition from a best-effort
to a fundamentally reliable communications system.
Impact of Rapid Growth on ISPs
The rapid growth of the Internet has had a profound impact on the way ISPs deploy and
maintain services to their customers. As ISPs struggle to expand their networks at phenomenal
rates, they are constantly challenged to overcome a nu mber of complex scaling issues:
s The challenge of enhancing reliability (7x24 service) during a time o f rapid ly increasing
traffic and customer exp ectations
s The need for traffic engineering tools to op timize the utilization of available band wid th
and maintain network reliability
s The challenge of maintaining pace with the rapid evolution of Internet technologynewprotocols, new routing hardware, new switching hardware, and fiber optic devices
s The logistics of find ing sufficient space and adequate p ower to d eploy equipmen t at a rate
commensurate w ith circuit availability and consumer d emand
s The challenge of providing the u nd erlying reliability required to offer differentiated
services to distingu ish themselves from their competition
-
8/2/2019 Backbone Routers
3/18
Introduction
Internet Backbone Routers and Evolving Internet Design s 3
Any solution intended for the ISP market must add ress all these growth issues. If a new system
cannot enhance performance and contribute to the fundamental reliability of the network, the
solution is doomed to failure.
Delivering Solutions that Permit ISPs to Grow Their Networks
ISPs require solutions that scale in terms of both rou ting richness and packet-forwarding
performan ce (Figure 1). The dem and for a single system that ad dresses both these issues has
created frustration with the ISP commu nity, because the vend or commu nity has been un able to
deliver a satisfactory solution tha t meets bo th th ese requiremen ts. Formerly, if an ISP needed
software richness for enhan ced control in a complex netw ork design, it deployed rou ters and
accepted relatively slow forwarding performance.
Figure 1: ISP Dilemma
To overcome these limitations, several ISPs decided to use an overlay ap proach, designing
their networ ks with IP routers at the edges to provide software richness and ATM switches at
the core to deliver speed . During the transition to OC-12 based cores, ISPs examined theavailable tools, put them together in complex ways, and achieved the d esired result. How ever,
the resulting overlay network solutions create their own set of management problems:
s The physical topology does not match th e logical topology of the netw ork.
s The ATM cell tax results in inefficient use of prov isioned band wid th.
s The full mesh of PVCs creates n-squared scalability problems.
s The overlay approach requires coordinated managem ent of two distinct networks: the
ATM network and the overlay IP network.
Despite these limitations, many ISPs have been w illing to accept the challenges of dep loying
an overlay solution because it has been their only option to create a high-bandw idth n etwork.
How ever, it certainly w ould be preferable to have a specially bu ilt system th at simp lifies thecreation of high-bandw idth netw orks wh ile eliminating the cost and com plexity of the overlay
solution. The new class of Internet backbone routers seeks to prov ide this tool by allowing
network designers to get away from deploying redundant devices at different layers just to
meet the fund amen tal need for speed w ith control. Before discussing the features that mu st be
sup ported by the new gen eration of Internet backbone routers, lets take a mom ent to examine
the essential elemen ts of a routing system .
Packet-forwarding
performance
Software
richness
Internet growth
Switch
Router
M40
-
8/2/2019 Backbone Routers
4/18
Introduction
4 s Internet Backbone Routers and Evolving Internet Design
Essential Elements of a Routing System
All routers must perform two fundamental tasks, routing and packet forwarding (Figure 2).
The routing process collects information about the network top ology and creates a forward ing
table. The packet-forw arding process copies a packet from an inpu t interface of the router to
the proper outpu t interface based on information contained in the forwarding table.
Figure 2: Routing and Packet-Forwarding Processes
Any routing system requires four essential elements to implement the routing and
packet-forward ing p rocesses: routing software, packet processing, a switch fabric, and line
cards (Figure 3). For any system that is designed to operate in th e core of the Internet, all four
elements mu st be equally powerful because a high-perform ance router can be only as strong as
its weakest elemen t.
Figure 3: Essential Elements of a Routing System
Routing software is the part of the system th at performs th e routing fun ction. It is responsible
for maintaining peer relationships, running the rou ting protocols, building the routing table,
and creating the forw arding table that is accessed by the packet-forward ing part of the system.Software also p rovides system control features includ ing traffic engineering, the u ser interface,
policy, and network m anagement.
Routing process
Packet-forwarding
process
Packets in Packets out
Routing
process
Packet-forwarding
processLine card
Line card
Line card
Line card
Packet
processing
Routing software
Switch fabric
Packet processing may be distributed to each line card, executed in a centralized fashion,
or performed by both the line cards and a centralized processor
-
8/2/2019 Backbone Routers
5/18
Introduction
Internet Backbone Routers and Evolving Internet Design s 5
Regardless of a router s architecture, each packet entering the system requ ires a certain am oun t
of processing tha t is comp letely indep end ent of the packets length. The incoming
encapsulation mu st be removed, a longest-match route lookup n eeds to be p erformed, the
packet needs to be queued on the ou tput p ort, and the outgoing encapsulation mu st be
provid ed. The tasks of perform ing longest-match lookup s and the related packet processing at
a high rate are the most d ifficult challenges to overcome when d eveloping a high-perform ance
routing system.
The switch fabric provides the infrastructure for moving packets between router line cards.
Designers have been w orking w ith switch fabrics for a number of decades, and the issues are
well und erstood by th e vend or comm un ity. There even are a nu mber of off-the-shelf chip sets
available to router vend ors to build a sw itch fabric. These solutions might be crossbar switches,
banyan netw orks, Clos netw orks, perfect shuffle networks, and so forth.
A line card termina tes circuits of different physical med ia types, implemen ting the Layer 1 and
Layer 2 techn ologies su ch as DS-3, ATM, SONET, Frame Relay, and PPP. Issues involving th e
design and development of router line cards also are well und erstood by the vend or
commu nity. Line cards simply have to be built according to the p revailing stand ards that
define p hysical interface types, optical characteristics, electrical levels, and so forth.
Key Attributes for an Internet Backbone Router
The next generation of Internet backbone routers m ust be sp ecifically designed to deliver
Internet scale, Internet control, and un par alleled p erformance over an optical infrastructu re.
The key attributes supp orted by an y Internet backbone router mu st include:
s Stable and complete routing software that has been written by Internet experts and has
successfully passed extensive interop erability testing in large ISP networks
s Traffic engineering features that offer fund amen tally new an d soph isticated control to
sup port the efficient utilization of available resources in large ISP networks
s Packet processing capable of performing incoming decapsulation, route lookup, queuing,
and outgoing en capsu lation at wire speed, regardless of packet size or system
configuration
s A switch fabric that has been oversized to prov ide an effective aggregate band wid th of
40 Gbps (8xOC-48) to support th e transition to OC-48 based cores.
s A wid e variety of interface types cap able of delivering wire-rate performance
s A chassis capable of providing a p ort den sity of at least one slot per r ack-inch
s Mechanicals, serviceability, and m anagement th at make th e system very d eployable in the
core of a large ISP network
s An ability to maintain overa ll netw ork stability and adapt to a highly fluctuating
environment without impa cting other p arts of the network
-
8/2/2019 Backbone Routers
6/18
M40 Internet Backbone Router Architecture
6 s Internet Backbone Routers and Evolving Internet Design
M40 Internet Backbone Router Architecture
The fun damen tal architecture of the M40 systemcomplete separation betw een the routing
and packet-forwarding functionshas been implemented by designing a system that consists
of two indep endent components, a Routing Engine and a Packet Forward ing Engine. The goal
of this separation is to best ensure that h igh levels of route instability do no t impact the
performance of the Packet Forwarding Engine. Likewise, extremely large volum es of traffic do
not imp act the ability of the Routing Eng ine to maintain p eer relationsh ips and calculate
routing tables. The clean separation of these tw o functions perm its the delivery of both
sup erior forw arding p erformance and a highly reliable operating system (Figure 4).
Figure 4: M40 System Architecture
The remaind er of this section focuses on how the M40 Internet backbone rou ter provid es
leading-edge solutions for each of the key elements in a router s architecture: routing software,
ASIC-based packet processing an d looku p, switch fabric, and line cards. By prov iding a system
wh ere all four componen ts are equally pow erful, the M40 system d elivers a comp lete solution
that is un ique in its ability to operate successfully in the core of the Internet.
Routing Software: JUNOS Internet Software
Juniper N etworks developed the JUNOS software from the ground up to meet the exacting
requiremen t of ISPs. The software incorporates th e following features:
s A modular software architecture run ning in protected mem ory provides system reliability
and scalability.
s Indust rial-strength rou ting protocol imp lemen tationsBGP4, IS-IS, OSPF, rou te reflection,
confederations, commu nities, and so forthdeliver the features required by ISPs to
control and m anage their networks.
s An extremely flexible policy definition langu age simp lifies the man agemen t of routing
policy for tens of thousan ds of routes.
s Traffic engineering, using Multiprotocol Label Switching (MPLS) as the technology,
maximizes the efficient u se of scarce network resou rces and p rovides the founda tion for
differentiated serv ices, as well as evolving new services.
Routing
Engine
Packet
Forwarding
Engine
Router operating system
(Internet Control,
Internet Scale)
Wire speed
40-Mpps packet processor
Computer-scale ASICs
JUNOS
Routing
table
Line
cardLine
card
Packet processing
Forwarding
table
Switch fabric
-
8/2/2019 Backbone Routers
7/18
M40 Internet Backbone Router Architecture
Internet Backbone Routers and Evolving Internet Design s 7
s The user interface provides mu ltiple u ser access levels, configu ration change control,
sup port for ASCII files, and the ability to return to previou s versions of a configur ation. To
minimize the chan ce of software configur ation errors, it sup por ts the ability to make
mu ltiple configuration changes to a new configura tion in a single step.
s System security is prov ided by Secure Shell (SSH) access to the u ser interface, TCP MD5
auth entication for BGP sessions, architectural safeguard s aga inst d enial-of-service attacks,
and so forth.
Jun iper Netw orks under stand s the importan ce of routing software for ISPs to control and
man age their netw orks. Accordingly, the JUNOS routing software has alread y been tested and
qualified by the w orlds largest ISPs.
For a complete d iscussion of the features an d benefits of the JUNOS Internet softw are,
includ ing traffic engineering, refer to the Jun iper Netw orks wh ite pap er entitled Op timizing
Routing Software for Reliable Internet Grow th.
Packet Lookup: The Internet Processor ASIC
Packet processing is an area in wh ich Juniper N etworks has fund amentally advanced the state
of the art for Internet backbone rou ters. All route lookups are performed by a single,compu ter-scale ASIC that is significantly m ore complex than ASICs used in any other
commu nications d evices. Jun iper N etworks, staffed by man y of Silicon Valleys leading
experts in high-speed computing and ASIC design, is the industry pioneer in advancing ASIC
designs in communications devices to levels previously found only in leading-edge
computers.
The Internet Processor ASIC is the fun dam ental technical advan ce in the M40 system in term s
of performance, program mability, and the coherence between routing and forward ing. The
Internet Processor contains more th an 6.5 million transistors and over 1 million ga tes. As a
comparison, the new Intel Pentium II has 7.5 million transistors. It is the pow er of the Internet
Processor ASIC combined with the JUNOS software th at enables the M40 system to
successfully bridge the performance gap between routers and switches in the core of the
Internet.
Route Lookup
The Internet Processor ASIC performs longest-match looku ps a t a rate of 40 million rou te
lookup s per second, w hich is one hun dred times faster than m icroprocessor-based lookups
currently dep loyed in the Internet. A single Internet Processor ASIC provides more than
enough power to perform wire-speed lookups for an 8xOC-48 system.
In add ition, the Internet Processor ASIC can be configured to p erform longest-match lookup s
with p er-prefix accounting. Per-prefix accoun ting prov ides statistics describing the n um ber of
bytes and packets forward ed to each p refix in the IP forwarding table. ISPs can use these
statistics to obtain a p icture of how traffic is flowing across their netw ork.
-
8/2/2019 Backbone Routers
8/18
M40 Internet Backbone Router Architecture
8 s Internet Backbone Routers and Evolving Internet Design
Programmability
The Internet Processor is both a fully generic and a fully programm able lookup en gine. In the
initial release, it su pp orts IPv4 (includ ing IP mu lticast) and MPLS. Because of the Internet
Processors program mab ility, sup port can be extended to includ e other protocols such as IPv6
and Frame Relay, simply by programming the Packet Forwarding Engine, developing new
routing software for the Routing Engine, and communicating the new forwarding table to the
Internet Processor.
The programm ability of the Internet Processor allows the existing JUNO S softwa re to continue
to evolve and remain fully supp orted in hard ware. As routing continues to develop in a
nu mber of un pred ictable ways, the progr amm ability of the Internet Processor ASIC allows
Jun iper Netw orks to futu re-proof the Packet Forward ing Engine by allowing it to supp ort new
functionality in hardware.
Performance Insurance
A key attribu te facilitated by the Internet Processor ASIC is performan ce insurance.
Performance insuran ce safeguard s system stability by segregating, and thu s protecting, the
operation of the routing and forwarding fun ctions. Whenever a netw ork failure strikes, three
events occur:
s The routing p rocess experiences stress because of the amou nt of routing change. The
routing stress is manifest in the need for each router to m aintain peer relationship s,
transmit and process routing u pd ate messages, calculate a new shortest-path tree, apply
policy to complete the route selection p rocess, and mod ify the forward ing table used by
the Packet Forwarding Engine.
s The packet-forwarding process experiences stress because rou ters that are still connected
to active network links sudd enly can find themselves forced to operate at peak capacity or
at even greater capacity rather than the usu al background level.
s The links an d routers forced to carry the ad ditional traffic resulting from the failure
condition become critical to the stability of the netw ork as a wh ole. If these netw ork
elements are u nable to carry the increased traffic load and fail, a relatively simple local
failure can cascade across the entire service provider s netw ork.
The challenge facing service provid ers is that trad itional rou ter architectures fail to deliver the
performance insurance that p rovides rock-solid system stability during failure cond itions.
Jun iper Netw orks believes that the M40 system prov ides the best solution in the indu stry for
accomplishing this critical design objective, supp orting perform ance insuran ce by:
s Implementing an architecture that distinctly separates the functions performed by the
Routing Engine and the Packet Forw arding EngineThis design segregates each
compon ent of the M40 system so that the stress experienced by each part of the system
does not negatively imp act the performance of the other part of the system.
s Ensuring that the lookup performance of the Internet Processor ASIC is never
comprom isedThe Internet Processor ASIC is fully sized to p erform lookup s at a rate of
40 Mpp s regardless of how long the lookup o r how large the routing table. The 40 millionlookups per second is achieved w ith 80,000 routes an d 80,000 distinct destina tion
add resses as opposed to artificial benchmarks that d o not reflect the current state of the
Internet.
s Allowing the Packet Forwarding Engine to m aintain forwarding performance when there
are high rates of up dates to the forwarding table, by supp orting th e revolutionary concept
of atomic updates to its forwarding table.
-
8/2/2019 Backbone Routers
9/18
M40 Internet Backbone Router Architecture
Internet Backbone Routers and Evolving Internet Design s 9
Atomic Updates
The Internet Processor ASIC sup ports atom ic up dates to its centralized forward ing table
(Figure 5). In the M40 system, the rou ting table contains the rou tes learned from rou ting
protocol exchanges w ith neighbors and through static configuration. The forwarding table is
derived from the rou ting table and contains an index of IP prefixes (or MPLS labels) that are
actively u sed to associate a p refix (or MPLS label) with an outgoing interface. The Packet
Forward ing Engine uses the contents of the forw arding tab le, not the routing table, to make itsforwarding decisions.
Typ ically, mod ifying a specific route affects only a sm all portion of the forw arding tables d ata
structure. This means that the Routing Engine simply needs to u pd ate a portion of the binary
tree in free mem ory and switch a pointer, and the up date is instantan eous. Because forward ing
information does not have to be distributed to m ultiple line cards, the Internet Processor does
not requ ire the forward ing table to be locked as the table is mod ified. Any attemp t to perform a
route lookup gets either the old tree or the new tree, dep end ing on wh ether the location is read
before or after the pointer has been changed. The benefit of this design is that the forward ing
table remains consistent at all times. This mean s that d uring period s of route instability, the
Packet Forwarding Engine can simultaneously accept updates to its forwarding table while
continuing to make forward ing decisions at an extremely rapid r ate.
Figure 5: Atomic Updates
Other high-performance router architectures perform packet lookups on each individual line
card. This means that w hen a routing change occurs and the forwarding table must be
mod ified, the u pd ate mu st be distributed to each of the ind ividu al line cards. During the
up date and distribution process, the forwarding table must be locked to maintain route
consistency as the centralized table is mod ified. As a result, packets are required to queu e up
and wait until the forwarding table is updated and then un locked. Only after the forward ing
table is unlocked can packet forwarding resum e. Clearly, locking the forward ing table can have
a negative impact on system performance during exceptional conditions when a high rate of
routing chan ge is coup led with a d rama tic increase in traffic flowing throu gh the router.
Forwarding
table
RoutingEngine
Packet
ForwardingEngine Internet Processor
Update
Routing
table
New subtreeDeleted subtree
New pointerOld pointer
Forwarding table
(Binary tree data structure)
-
8/2/2019 Backbone Routers
10/18
M40 Internet Backbone Router Architecture
10 s Internet Backbone Routers and Evolving Internet Design
Traffic Visibility
With backbone rou ters forwarding tra ffic at 40-Mpps rates, an ISP needs to und erstand the
traffic trend s that are tak ing place in its networ k. For examp le, new app lications can affect
overall network performa nce or the volum e of traffic can shift in d ifferent parts of the netw ork.
The traffic sampling featu re of the Internet Processor allows an ISP to sam ple trafficsay 1 ou t
of 1,000 packets.
Traffic sampling occurs by send ing a n otification w ith a sp ecial flag to another par t of the
system, a process that occurs withou t impacting the lookup performance of the Internet
Processor. A notification is a Juniper N etworksdefined data structure that contains all the
information need ed to process a packet after it has been stored in shared m emory. Based on th e
information contained in the notification, the packet then can be retrieved from shared
memory an d forwarded to a user process executing in the Routing Engine without imp acting
the performance of the Packet Forwarding Engine.
Switch Fabric: Distributed Buffer Manager ASIC, I/O Manager ASIC, and Shared Memory
The M40 system prov ides a conservatively rated 40-Gbps switch fabric that is imp lemented as
a shared mem ory system. In ad dition to the Internet Processor ASIC, the sw itch fabric is
composed of the Distributed Buffer Manager ASICs, the I/ O Manager ASICs, and the shared
mem ory system (Figure 6). In the illustra tion, note that each p acket is fragm ented into 64-byte
blocks for efficient storage in the sha red m emory system, wh ile a notification d escribing the
head er of the packet is forwarded to the Internet Processor ASIC for route lookup .
Figure 6: M40 Packet Forwarding Engine
From a vendor s persp ective, the chief limitation of a shared mem ory interconnect is that it is
technically difficult to design and implem ent. However, this does not m ean that ven dor s
shou ld avoid designing switch fabrics based on this architecture. For a system of this size, that
is 8xOC-48, or even a few times larger, a shared mem ory architecture presents a solid app roach
that results in numerous system benefits.
Distributed BufferManager
Shared memory
Distributed BufferManager
Inputinterface
Outputinterface
(Packet)
I/OManager
I/OManager
Linecard
Linecard
(Blocks)
(Packet)
Route lookup
Internet
Processor(Notification) Notification
Packets
Blocks
-
8/2/2019 Backbone Routers
11/18
M40 Internet Backbone Router Architecture
Internet Backbone Routers and Evolving Internet Design s 11
Distributed Buffer Manager ASIC and Shared Memory
Each of the M40 systems two Distributed Buffer Manager ASICs contains m ore than
10.5 million tra nsistors and 1.7 million ga tes. Like the Internet Processor ASIC, the Distributed
Buffer Manager ASIC offers comp lete flexibility, because it can be p rogram med by Jun iper
Networks to parse Layer 3 headers and generate route lookup keys based on current or future
Internet requirements.
The Distributed Buffer Manager ASIC man ages the M40 Internet backbone rou ters shared
mem ory system. Each Flexible PIC Concentrator prov ides 128 megabytes of shared m emory
and four Physical Interface Concentrator (PIC) line card slots. The Distributed Buffer Manager
treats the mem ory on each Flexible PIC Concentrator as p art of the systems shared mem ory
pool.
The Distributed Buffer Manager ASICs, combined with a shared m emory architecture,
contribute to the performance and robustness of the M40 switch fabric:
s The Distributed Buffer Manager ASIC allows Jun iper Netw orks to dev elop a system that
offers the simp licity of single-stage buffering; that is, a single write and a single read from
shared m emory. Shared mem ory is totally nonblocking as compared to a crossbar switch
architecture, which requires mu ltiple inpu t queu es to prevent head -of-line blocking.
s The capacity of the M40 shared m emory interconn ect is oversized so th at it easily can
sup port eight full-du plex line-rate OC-48s. This design allows the M40 to defer all queuing
and drop algorithms to th e outpu t interface, so the M40 never has to drop packets on an
inpu t interface.
s In add ition to th e benefits of single-stage bu ffering to m aximize memory u tilization, each
Flexible PIC Concentrator p rovides 128 megabytes of shared p acket buffer memory, thu s
supp orting large bandw idthdelay designs. Large band widthdelay designs permit end
station TCP sessions to keep the pipe full of data and achieve better performance.
s The use of shared m emory w ith single-stage buffering permits the constru ction of a
low-latency system. How ever, it should be emp hasized that latency is predominately an
issue for LAN systems, not WAN system s, for wh ich the bandw idthdelay design is the
primary concern.s The integrity of data stored in the SDRAM shared m emory is guar anteed by the use of
error-correcting code (ECC) memory. ECC memory gu arantees that ra nd om bit errors do
not affect data integrity.
s Because the system architecture keeps the Routing Engine completely ind ependen t from
the switch fabric, the Routing Engine does not consume switch bandwidth.
-
8/2/2019 Backbone Routers
12/18
M40 Internet Backbone Router Architecture
12 s Internet Backbone Routers and Evolving Internet Design
I/O Manager ASIC
Each I/ O Manager ASIC contains more than 10 million transistors and 1 million gates. The I/ O
Manager ASIC is required to p erform different functions for incoming and outgoing packets.
Hence, its primary resp onsibilities in the p acket-forward ing process have been a llocated to
three independent modules operating within the ASIC: the Input Processor, the Memory
Access Interface, and th e Outp ut Processor:
s The Inpu t Processor acts on incoming packets. It is responsible for receiving packets from
the line card , decoding the Layer 2 head er, and performing the packet-to-block
segmentation.
s The Memory Access Interface acts on both incom ing and outgoing packets. It
commu nicates with the Distributed Buffer Manager ASIC and hand les buffer mem ory for
a specific Flexible PIC Concentrator. For incoming packets, it is responsible for servicing
write block requests mad e by the Distributed Buffer Manager ASIC. For outbou nd
packets, the I/ O Manager ASIC services read block requests mad e by the Distributed
Buffer Manager ASIC.
s The Outp ut Processor acts on outbou nd packets. It is responsible for receiving packet
notifications from the Distributed Buffer Manager ASIC, performing class-of-service
selection, output notification queuing, weighted round-robin queue service, and RandomEarly Discard (RED and WRED); maintaining statistical counters; performing
block-to-packet reassembly; and providing the outbound Layer 2 encapsulation.
Similar to both th e Internet Processor ASIC and the Distributed Buffer Manager ASIC, the I/ O
Manager ASIC offers maximu m flexibility because it also is highly p rogram mable. It can be
programmed by Juniper N etworks to provide sup port for Layer 2 decoding and encapsulation
schemes su ch as PPP, Frame Relay, and MPLS. As far as sup por ting class of service, the I/ O
Manager ASIC offers num erous op tions for assigning a n otification to a queue and servicing
the outp ut qu eues, as well as controlling drop profiles and the packet discard process. Finally,
the I/ O Manager p lays an integral role in supp orting the write-once, read-many facility
required for the efficient forw ard ing of IP m ulticast traffic.
Line Cards
The M40 system line cards are imp lemented with m edia-specific ASICs. For example, Jun iper
Netw orks has integrated full SONET/ SDH processing on a single highly integrated A SIC. On
other vendor s systems SONET/ SDH p rocessing typically is performed by a n um ber of
different comp onents, not a single, highly integra ted ASIC that performs a ll requ ired
functions.
The benefits delivered by the M40 system line card ASICs are increased p ort d ensity, higher
performance, lower pow er draw s, and enhan ced reliability. Sup er-POPs require a router that
coinciden tly supp orts a large num ber of different interface types and high p ort densities so
that the router can grow and evolve as ISP requiremen ts change. As a result, the M40 system
can function in a broad range of sup er-POP environments d uring an y stage of an ISPs
development.
-
8/2/2019 Backbone Routers
13/18
M40 Internet Backbone Router Architecture
Internet Backbone Routers and Evolving Internet Design s 13
The wide variety of line cards and the nu mber of slots enhance the configuration flexibility of
the M40 system in a sup er-POP environment. A fully pop ulated M 40 system prov ides
32 slotsan ind ustry-leading p ort dens ity of one slot per rack-inchoffering comp lete
mix-and-match flexibility for line card installation. Because the switch fabric has been
oversized, all line cards op erate at w ire rate for all packet sizes. The line card s available in 1998
include:
s OC-48 IP over SONET/ SDH
s OC-12 IP over SONET/ SDH
s OC-3 IP over SONET/ SDH
s OC-12 IP over ATM
s OC-3 IP over ATM
s DS-3 with an internal DSU
s Gigabit Ethernet
The line cards can be mixed and matched in each slot as desired. (The exception is the OC-48
line card, wh ich u ses four slots.) Table 1 illustrates the maximu m den sities for each line card
type in a fully popu lated M40 system.
Table 1: M40 Interface Densities
Interface Type Maximum Ports Per M40 System Maximum Ports Per 7-Foot Rack
OC-48 SONET/ SDH 8 16
OC-12 SONET/ SDH 32 64
OC-12 ATM 32 64
OC-3 SONET/ SDH 128 256
OC-3 ATM 64 128
DS3 128 256
-
8/2/2019 Backbone Routers
14/18
How Packets Traverse the M40 Packet Forwarding Engine
14 s Internet Backbone Routers and Evolving Internet Design
How Packets Traverse the M40 Packet Forwarding Engine
Figure 7 illustrates the path a packet takes as it is processed by the Packet Forward ing Engine.
A packet arrives at the M40 system on the inp ut interface (Step 1). The m edia-specific ASIC at
that interface performs all the med ia-specific details, such as rem oving the payload from the
SON ET frame, HDLC framing, and checksum verification. Jun iper Netw orks has designed an d
developed three m edia-specific ASICs: one for SONET, a second for ATM, and a third for DS3
interfaces.
Figure 7: Packet Forwarding in the M40 System
From the med ia-specific stage, a serial stream of bytes is passed to the I/ O Manager ASIC(Step 2). The I/ O Manager d etermines wh ether the frame is IPv4 (includ ing IP multicast) or
MPLS and identifies the beginn ing of the Layer 3 packet. The I/ O Manager also sets flags in
the p ackets notification tha t might be used for differentiated serv ices. Finally, the I/ O
Manager chops the p acket into 64-byte blocks and p asses each of the blocks to the Distributed
Buffer Manager ASIC (Step 3). These blocks are sized for efficient storage and retrieval from
shared mem ory an d are un related to 53-byte ATM cells. The Distributed Buffer Manager ASIC
distributes the blocks evenly in a round -robin fashion across the shared mem ory (Step 4).
In para llel to distributing each of blocks to shared mem ory, the Distributed Buffer Manager
ASIC also extracts the rou te lookup key from the blocks it receives and constructs a packet
notification. Recall that a packet notification is a Juniper Netw orksdefined (and
programmable) data structure that contains all the information needed to process a packet
after the packet has been stored in shared mem ory. For a unicast IPv4 packet, the Distributed
Buffer Manager ASIC determines the incoming interface, destination IP ad d ress, source IP
add ress, and p rotocol value, as well as the source and d estination TCP/ UDP port num bers.
For an MPLS frame, the Distributed Buffer Manager extracts a route lookup key that con tains
the incoming interface and the value of the MPLS label. After collecting this information, the
Distributed Buffer Manager forwards the notification to the Internet Processor ASIC (Step 5) so
that it can m ake a forward ing decision for the packet.
I/OManager
mediaspecific
Distributed BufferManager
Shared memorymediaspecific
InternetProcessor
IP forwardingtable
Distributed BufferManager
1 2
3
5
4
6
7a
8
9 10
Inputinterface
Outputinterface
7b 8
I/OManager
Notifications
Packets
Blocks
-
8/2/2019 Backbone Routers
15/18
The M40 Internet Backbone Router in an ISP Network
Internet Backbone Routers and Evolving Internet Design s 15
The Internet Processor ASIC perform s the rou te lookup . For an IPv4 packet, it searches the IP
forward ing table (Step 6) looking for the longest m atch for the d estination p refix. For an MPLS
frame, the Internet Processor performs an exact match looku p in the MPLS forwarding tab le.
After performing the looku p, the Internet Processor passes the n otification message containing
the forwarding decision to the Distributed Buffer Manager ASIC (Step 7a). The Distributed
Buffer Manager forward s the notification to the I/ O Manager ASIC for the ou tpu t interface
(Step 7b). For an IP mu lticast fram e, the Internet Processor forwards a notification to the I/ OManager for each outpu t port.
The I/ O Manager ASIC on the outp ut interface is respon sible for managing packet queues. The
packet itself is not queu ed, but instead the notification for the packet is queued , while the
actual packet remains stored as blocks in shared mem ory. In the specific case of an IP m ulticast
packet, the I/ O Manager for each outp ut interface independ ently queues a packet notification.
For every outpu t port there are four queu es, each of wh ich has some configured share of the
physical links band width. The I/ O Manager on the ou tput interface can take a num ber of
factors into accoun t when d eciding to qu eue a packet, includ ing the value of the IP precedence
bits, utilization of the inpu t interface, destination ad dress, and RED/ WRED algorithms.
When th e packet notification reaches the front of its queu e and is ready for transmission, the
I/ O Manager issues a request through the Distributed Buffer Manager to read the packetsblocks from sha red m emory (Step 8). The I/ O Manager reassembles the blocks into the packet
and forward s the frame to th e med ia-specific ASIC on the ou tpu t interface (Step 9).
The med ia-specific ASIC on the outp ut interface perform s the necessary med ia-specific
operations, such as PPP-over-SON ET scrambling and H DLC framing , places the bits in a
SONET frame, iden tifies the beginning of the payload in the SONET frame, and then serializes
the bits on the fiber (Step 10). At this p oint, the packet leaves the Packet Forwarding Engine for
the next hop along the p ath towards the destination.
The M40 Internet Backbone Router in an ISP Network
In add ition to providing the raw performance and righ t features for an OC-48 Internetbackbone rou ter, the M40 system ad dresses other issues that ISPs demand : rock-solid
reliability under failure conditions and ea sy dep loyability in super-POP environments.
Rock-Solid Reliability Under Failure Conditions
The prim ary concern for any ISP is the stability of the netw ork as a w hole, not just the
reliability of an individu al system. Even though the M40 system is designed to be extremely
reliable from both a ha rdw are and a software p erspective, the basic objective for all ISPs is to
localize the imp act of any n etwork ou tage. ISPs do not w ant their routers behaving in ways
that allow a local failure to cascade across the netw ork, creating ad ditional outages.
-
8/2/2019 Backbone Routers
16/18
The M40 Internet Backbone Router in an ISP Network
16 s Internet Backbone Routers and Evolving Internet Design
To p rovide an app ropriate level of stability, ISPs traditionally design th eir networ ks
conservatively and try to an ticipate the impossible so that customers never become
par titioned , resulting in a complete loss of service. To do this, ISPs dep loy redu nd ant system s,
install numerous backup links, and ru n the network a round 50 percent capacity du ring normal
operating conditions. The ability of the netw ork to accommod ate some nu mber of failures
wh ile still being able to sw itch all traffic is a critical design objective. ISPs follow th is
conservative app roach for several reasons:
s If a rou ter or link fails, the rest of the n etwork still has su fficient capacity to carry the
add itional traffic load. During exceptional cond itions, the ma jority of the network can
continue to run at 50 percent capacity, but certain network elemen ts might need to operate
at close to 100 percent capacity as the traffic load increases.
s Conservative designs provide ISPs with a bit of wiggle room during p eriods of high
growth and rap id evolution. If an ISP has failed to prov ision adequ ate capacity in the
network topology, it might be willing to accept some add itional risk and let through pu t at
that point dr ift up to 60 or 70 percent until adequate transm ission capacity is available.
The challenge w ith this conservative app roach is that existing routers generally operate well
und er low load conditions, but they are not engineered to provide stable performance wh en
the traffic load jump s d rama tically to 100 percent. The M40 Internet backbone rou ter is un ique
in its ability to provide the rock-solid stab le performance that other systems lack du ring
periods of extreme stress:
s The M40 Internet backbone rou ter is fully sized with resp ect to both route processing and
packet forward ing. During exceptional cond itions, the Routing Engine continu es to
receive and transmit routing updates, perform route calculations, maintain peer
relationship s, react to interfaces going down , and so forth. Similarly, the Packet
Forward ing Engine continues to sw itch packets at a rate of 40 Mpps regard less of packet
size or load on the system .
s Complementing the architectural separation of the routing and packet-forward ing
processes, atom ic up da tes perm it the state of the Packet Forw arding Eng ine to concur w ith
the state of the Routing Engine without impacting forwarding performance. During
exceptional cond itions, atomic upda tes allow th e M40 system to avoid destab ilizing thelinks that still remain up , thus eliminating the p rimary reason for cascad ing failures.
s The traffic engineering featu res sup ported by the JUNOS Internet software allow ISPs to
man age around network failures. Lacking the p refailure transm ission capacity, the M40
system provid es tools that permit an ISP to determ ine the best method for d istributing the
current traffic load over available resources without creating congestion and further
destabilizing the network.
-
8/2/2019 Backbone Routers
17/18
The M40 Internet Backbone Router in an ISP Network
Internet Backbone Routers and Evolving Internet Design s 17
M40 System Deployment
The M40 system has been specifically designed to ad dresses the num erous challenges facing
ISPs in the su per-POP environm ent (Figure 8) as they transition their networ ks from existing
OC-3 and O C-12 based cores to OC-48 based cores. The M40 Internet backbone rou ter prov ides
indu stry-leading metrics in terms of:
sSizeEach M40 Internet backbone rou ter chassis is 35 in height, so tw o chassis can beinstalled in a single seven -foot-high rack.
s Power d raw The M40 system ad dresses the logistical issue of adequa te pow er
availability by draw ing less than 1 amp per rack-inch.
s Performance den sityThe M40 system p rovides forward ing performance of greater than
1 Mpps p er rack-inch, wh ich is ten times the performance currently available from other
systems targeted a t the Internet core.
s Slot densityA fully popu lated M40 system prov ides one slot per rack-inch, wh ich is
three to four times the slot density ava ilable from oth er systems targeted for sup er-POP
environments.
The M40 Internet backbone rou ter represents an en tirely new class of service provider system ,
with sp ecialized features that are familiar to those working in a telecommu nicationsenvironm ent. For examp le, a craft interface perm its a techn ician to m onitor the statu s of the
system, troubleshoot the system with help from a remote Network Operations Center (NOC),
and perform a nu mber of system functions.
Figure 8: Super-POP Environment
Router M40Switch
M40Switch
OC-48
OC-48
OC-48
OC-3, OC-12, or
Gigabit Ethernet
Legacy
DS0, DS1
Access
Access Intra-POP Core Backbone
Dense
DS3
Customer
Access
Router
Gigabit Ethernet
or OC-12
OC-3, OC-12, or
Gigabit Ethernet
Gigabit Ethernet
or OC-12
-
8/2/2019 Backbone Routers
18/18
Conclusion
18 s Internet Backbone Routers and Evolving Internet Design
Conclusion
The M40 Internet backbone rou ter represents an en tirely new class of routing system that has
been sp ecifically d esigned to help ISPs negotiate the transition from O C-3 and OC-12 to OC-48
based backbones. The core of the Internet is constan tly developing along tw o dimen sions,
software richness and ban dw idth. The M40 system prov ides all the features that a next
generation routing system m ust supp ort:
s The Packet Forwarding Engine is oversized w ith packet processing and a switch fabric to
effortlessly sup port 8xOC-48 interfaces at full wire-speed. The M40 system provid es the
forward ing performance previously found only in switches, withou t sacrificing the
elements of network control.
s The Routing Engine executes indu strial-strength, full-featured rou ting p rotocol and traffic
engineering software designed and written by acknowledged ind ustry experts.
s The fun damen tal architecture of the M40 systemcomplete separation betw een the
routing and packet-forwarding functionshas been d esigned with the goal of providing
perform ance insuran ce to enhance the stability of large ISP networ ks du ring exceptiona l
conditions. The M40 system is un ique in its ability to contribu te to netw ork stability and
adap t to highly fluctuating environments without impacting other parts of the network.
s Programm able, comp uter-scale ASICs allows the JUNOS Internet software to continu e to
evolve while futu re-proofing the Packet Forwarding Engine by allowing it to sup port new
functionality w ithout hard ware changes.
s The variety and den sity of interfaces, as well as the m echanicals and ser viceability, make
the M40 Internet backbone rou ter eminen tly dep loyable in the core of large ISP netw orks.
s By p roviding a routing system w here all four fundam ental comp onents are equally
pow erful routing software, packet processing, the switch fabric, and line cardsthe
M40 system d elivers a comp lete solution th at is uniqu e in its ability to operate su ccessfully
in the core of the Internet.
Junip er Netw orks is a registered trad emarks of Jun iper Netw orks, Inc. All other trad emark s, service marks, registered trad emar ks, or registere
service marks mentioned in this document are the property of their respective owners.
Copyrigh t 1997, Jun iper N etwork s, Inc.
All rights reserved. Printed in USA.