backbone routers

8/2/2019 Backbone Routers

1/18

Internet Backbone Routers and Evolving Internet Design s 1

Internet Backbone Routers and Evolving Internet Design

Chuck SemeriaMarketing EngineerJuniper NetworksSeptember 16, 1998

Contents Introduction 1Im pact of Rapid Grow th on ISPs 2

Delivering Solutions that Permit ISPs to Grow Their Networks 3

Essent ial Elements of a Routing System 4

Key Attributes for an Internet Backbone Router 5

M40 Internet Backbone Router Architecture 6

Routing Software: JUNOS Internet Software 6

Packet Lookup: The Internet Processor ASIC 7

Route Lookup 7

Programmability 8

Perform an ce In su ran ce 8

Atomic Updates 9

Tr affic Visib ility 10

Switch Fabric: Distributed Buffer Manager ASIC, I/ O Man ager ASIC,

and Shared Memory 10

Distributed Buffer Manager ASIC and Shared Memory 11

I/ O Manager ASIC 12

Line Cards 12

How Packets Traverse the M40 Packet Forwarding Engine 14

The M40 Internet Backbone Router in an ISP Network 15

Rock-Solid Reliabili ty Under Failure Conditions 15

M40 System Dep loym en t 17

Conclusion 18

Introduction

As we ap proach th e twen ty-first centu ry, the Internet continu es to experience extraord inary

growth . Any w ay one measures it, the grow th is remarkable on all frontsthe nu mber of

hosts, the num ber of users, the amou nt of traffic, the nu mber of links, the band wid th of

individual links, or the grow th rates of Internet Service Provider (ISP) networks.


2/18

Introduction

2 s Internet Backbone Routers and Evolving Internet Design

Formerly, the ISP commu nity adap ted general-pu rpose equ ipmen t for use in the core of the

Internet. The explosive growth of the Internet has created a market for networ king equipmen t

that is specially built to solve the unique problems confronting Internet backbone providers.

This new class of equipmen t is required to scale the Internet no t only in terms of aggregate

bandw idth and raw throu ghpu t, but also in terms of software richness and control.

To achieve the objectives of providing increased band wid th and software richness, the

designers of new routing systems must achieve the forwarding p erformance p reviously foundonly in switches. However, delivering w ire-speed performance for variable-sized packets with

longest-match lookups is considerably more comp lex than th e relatively straightforw ard

switching goal of sup porting wire-speed p erformance for fixed-sized cells with fixed-length

lookups. In particular, packet processing can no longer remain a m icroprocessor-based or

microprocessor-assisted function, but mu st transition to an ASIC-based ap proach that

supp orts the evolution of routing software as the Internet environment changes.

The system cha llenges facing this new class of Internet backbone rou ter also are comp lex. New

routing system s must be deployed into existing OC-3 and OC-12 based cores with their related

intra-POP infrastructure, and they also must supp ort the transition to OC-48 based cores with

OC-12 and Gigabit Ethernet based intra-POP infrastructures. In add ition, new rou ting systems

mu st accelerate the evolu tion of the Internet from a best-effort service to a fun dam entally

reliable service. With the Internet em erging in its role as the new pu blic netw ork, users aredem and ing and expecting increased reliability and stability to sup por t mission-critical

app lications. The Internet cannot continu e to p rovide erratic best-effort service that generally

works but at other times fails. As they hav e come to expect with the Plain Old Telephone

Service (POTS), Internet users w ant to h ear d ial-tone an d receive quality service whenever

they wish to commun icate.

The Jun iper Netw orks M40 Internet backbone rou ter is the worlds most complete system for

Internet routing . It is the first system that has been specially built to combine Intern et scale,

Internet control, and u np aralleled forw arding performa nce. By integrating th e flexibility and

control of router software with the p acket-forward ing performance of a switch and p roviding

rock-solid stability during exceptional cond itions, the M40 system is designed to serve as the

found ation of the op tical Internet, while sup porting the Internets transition from a best-effort

to a fundamentally reliable communications system.

Impact of Rapid Growth on ISPs

The rapid growth of the Internet has had a profound impact on the way ISPs deploy and

maintain services to their customers. As ISPs struggle to expand their networks at phenomenal

rates, they are constantly challenged to overcome a nu mber of complex scaling issues:

s The challenge of enhancing reliability (7x24 service) during a time o f rapid ly increasing

traffic and customer exp ectations

s The need for traffic engineering tools to op timize the utilization of available band wid th

and maintain network reliability

s The challenge of maintaining pace with the rapid evolution of Internet technologynewprotocols, new routing hardware, new switching hardware, and fiber optic devices

s The logistics of find ing sufficient space and adequate p ower to d eploy equipmen t at a rate

commensurate w ith circuit availability and consumer d emand

s The challenge of providing the u nd erlying reliability required to offer differentiated

services to distingu ish themselves from their competition


3/18

Introduction


Any solution intended for the ISP market must add ress all these growth issues. If a new system

cannot enhance performance and contribute to the fundamental reliability of the network, the

solution is doomed to failure.

Delivering Solutions that Permit ISPs to Grow Their Networks

ISPs require solutions that scale in terms of both rou ting richness and packet-forwarding

performan ce (Figure 1). The dem and for a single system that ad dresses both these issues has

created frustration with the ISP commu nity, because the vend or commu nity has been un able to

deliver a satisfactory solution tha t meets bo th th ese requiremen ts. Formerly, if an ISP needed

software richness for enhan ced control in a complex netw ork design, it deployed rou ters and

accepted relatively slow forwarding performance.

Figure 1: ISP Dilemma

To overcome these limitations, several ISPs decided to use an overlay ap proach, designing

their networ ks with IP routers at the edges to provide software richness and ATM switches at

the core to deliver speed . During the transition to OC-12 based cores, ISPs examined theavailable tools, put them together in complex ways, and achieved the d esired result. How ever,

the resulting overlay network solutions create their own set of management problems:

s The physical topology does not match th e logical topology of the netw ork.

s The ATM cell tax results in inefficient use of prov isioned band wid th.

s The full mesh of PVCs creates n-squared scalability problems.

s The overlay approach requires coordinated managem ent of two distinct networks: the

ATM network and the overlay IP network.

Despite these limitations, many ISPs have been w illing to accept the challenges of dep loying

an overlay solution because it has been their only option to create a high-bandw idth n etwork.

How ever, it certainly w ould be preferable to have a specially bu ilt system th at simp lifies thecreation of high-bandw idth netw orks wh ile eliminating the cost and com plexity of the overlay

solution. The new class of Internet backbone routers seeks to prov ide this tool by allowing

network designers to get away from deploying redundant devices at different layers just to

meet the fund amen tal need for speed w ith control. Before discussing the features that mu st be

sup ported by the new gen eration of Internet backbone routers, lets take a mom ent to examine

the essential elemen ts of a routing system .

Packet-forwarding

performance

Software

richness

Internet growth

Switch

Router

M40


4/18

Introduction


Essential Elements of a Routing System

All routers must perform two fundamental tasks, routing and packet forwarding (Figure 2).

The routing process collects information about the network top ology and creates a forward ing

table. The packet-forw arding process copies a packet from an inpu t interface of the router to

the proper outpu t interface based on information contained in the forwarding table.

Figure 2: Routing and Packet-Forwarding Processes

Any routing system requires four essential elements to implement the routing and

packet-forward ing p rocesses: routing software, packet processing, a switch fabric, and line

cards (Figure 3). For any system that is designed to operate in th e core of the Internet, all four

elements mu st be equally powerful because a high-perform ance router can be only as strong as

its weakest elemen t.

Figure 3: Essential Elements of a Routing System

Routing software is the part of the system th at performs th e routing fun ction. It is responsible

for maintaining peer relationships, running the rou ting protocols, building the routing table,

and creating the forw arding table that is accessed by the packet-forward ing part of the system.Software also p rovides system control features includ ing traffic engineering, the u ser interface,

policy, and network m anagement.

Routing process

Packet-forwarding

process

Packets in Packets out

Routing

process

Packet-forwarding

processLine card

Line card

Line card

Line card

Packet

processing

Routing software

Switch fabric

Packet processing may be distributed to each line card, executed in a centralized fashion,

or performed by both the line cards and a centralized processor


5/18

Introduction


Regardless of a router s architecture, each packet entering the system requ ires a certain am oun t

of processing tha t is comp letely indep end ent of the packets length. The incoming

encapsulation mu st be removed, a longest-match route lookup n eeds to be p erformed, the

packet needs to be queued on the ou tput p ort, and the outgoing encapsulation mu st be

provid ed. The tasks of perform ing longest-match lookup s and the related packet processing at

a high rate are the most d ifficult challenges to overcome when d eveloping a high-perform ance

routing system.

The switch fabric provides the infrastructure for moving packets between router line cards.

Designers have been w orking w ith switch fabrics for a number of decades, and the issues are

well und erstood by th e vend or comm un ity. There even are a nu mber of off-the-shelf chip sets

available to router vend ors to build a sw itch fabric. These solutions might be crossbar switches,

banyan netw orks, Clos netw orks, perfect shuffle networks, and so forth.

A line card termina tes circuits of different physical med ia types, implemen ting the Layer 1 and

Layer 2 techn ologies su ch as DS-3, ATM, SONET, Frame Relay, and PPP. Issues involving th e

design and development of router line cards also are well und erstood by the vend or

commu nity. Line cards simply have to be built according to the p revailing stand ards that

define p hysical interface types, optical characteristics, electrical levels, and so forth.

Key Attributes for an Internet Backbone Router

The next generation of Internet backbone routers m ust be sp ecifically designed to deliver

Internet scale, Internet control, and un par alleled p erformance over an optical infrastructu re.

The key attributes supp orted by an y Internet backbone router mu st include:

s Stable and complete routing software that has been written by Internet experts and has

successfully passed extensive interop erability testing in large ISP networks

s Traffic engineering features that offer fund amen tally new an d soph isticated control to

sup port the efficient utilization of available resources in large ISP networks

s Packet processing capable of performing incoming decapsulation, route lookup, queuing,

and outgoing en capsu lation at wire speed, regardless of packet size or system

configuration

s A switch fabric that has been oversized to prov ide an effective aggregate band wid th of

40 Gbps (8xOC-48) to support th e transition to OC-48 based cores.

s A wid e variety of interface types cap able of delivering wire-rate performance

s A chassis capable of providing a p ort den sity of at least one slot per r ack-inch

s Mechanicals, serviceability, and m anagement th at make th e system very d eployable in the

core of a large ISP network

s An ability to maintain overa ll netw ork stability and adapt to a highly fluctuating

environment without impa cting other p arts of the network


6/18

M40 Internet Backbone Router Architecture



The fun damen tal architecture of the M40 systemcomplete separation betw een the routing

and packet-forwarding functionshas been implemented by designing a system that consists

of two indep endent components, a Routing Engine and a Packet Forward ing Engine. The goal

of this separation is to best ensure that h igh levels of route instability do no t impact the

performance of the Packet Forwarding Engine. Likewise, extremely large volum es of traffic do

not imp act the ability of the Routing Eng ine to maintain p eer relationsh ips and calculate

routing tables. The clean separation of these tw o functions perm its the delivery of both

sup erior forw arding p erformance and a highly reliable operating system (Figure 4).

Figure 4: M40 System Architecture

The remaind er of this section focuses on how the M40 Internet backbone rou ter provid es

leading-edge solutions for each of the key elements in a router s architecture: routing software,

ASIC-based packet processing an d looku p, switch fabric, and line cards. By prov iding a system

wh ere all four componen ts are equally pow erful, the M40 system d elivers a comp lete solution

that is un ique in its ability to operate successfully in the core of the Internet.

Routing Software: JUNOS Internet Software

Juniper N etworks developed the JUNOS software from the ground up to meet the exacting

requiremen t of ISPs. The software incorporates th e following features:

s A modular software architecture run ning in protected mem ory provides system reliability

and scalability.

s Indust rial-strength rou ting protocol imp lemen tationsBGP4, IS-IS, OSPF, rou te reflection,

confederations, commu nities, and so forthdeliver the features required by ISPs to

control and m anage their networks.

s An extremely flexible policy definition langu age simp lifies the man agemen t of routing

policy for tens of thousan ds of routes.

s Traffic engineering, using Multiprotocol Label Switching (MPLS) as the technology,

maximizes the efficient u se of scarce network resou rces and p rovides the founda tion for

differentiated serv ices, as well as evolving new services.

Routing

Engine

Packet

Forwarding

Engine

Router operating system

(Internet Control,

Internet Scale)

Wire speed

40-Mpps packet processor

Computer-scale ASICs

JUNOS

Routing

table

Line

cardLine

card

Packet processing

Forwarding

table

Switch fabric


7/18



s The user interface provides mu ltiple u ser access levels, configu ration change control,

sup port for ASCII files, and the ability to return to previou s versions of a configur ation. To

minimize the chan ce of software configur ation errors, it sup por ts the ability to make

mu ltiple configuration changes to a new configura tion in a single step.

s System security is prov ided by Secure Shell (SSH) access to the u ser interface, TCP MD5

auth entication for BGP sessions, architectural safeguard s aga inst d enial-of-service attacks,

and so forth.

Jun iper Netw orks under stand s the importan ce of routing software for ISPs to control and

man age their netw orks. Accordingly, the JUNOS routing software has alread y been tested and

qualified by the w orlds largest ISPs.

For a complete d iscussion of the features an d benefits of the JUNOS Internet softw are,

includ ing traffic engineering, refer to the Jun iper Netw orks wh ite pap er entitled Op timizing

Routing Software for Reliable Internet Grow th.

Packet Lookup: The Internet Processor ASIC

Packet processing is an area in wh ich Juniper N etworks has fund amentally advanced the state

of the art for Internet backbone rou ters. All route lookups are performed by a single,compu ter-scale ASIC that is significantly m ore complex than ASICs used in any other

commu nications d evices. Jun iper N etworks, staffed by man y of Silicon Valleys leading

experts in high-speed computing and ASIC design, is the industry pioneer in advancing ASIC

designs in communications devices to levels previously found only in leading-edge

computers.

The Internet Processor ASIC is the fun dam ental technical advan ce in the M40 system in term s

of performance, program mability, and the coherence between routing and forward ing. The

Internet Processor contains more th an 6.5 million transistors and over 1 million ga tes. As a

comparison, the new Intel Pentium II has 7.5 million transistors. It is the pow er of the Internet

Processor ASIC combined with the JUNOS software th at enables the M40 system to

successfully bridge the performance gap between routers and switches in the core of the

Internet.

Route Lookup

The Internet Processor ASIC performs longest-match looku ps a t a rate of 40 million rou te

lookup s per second, w hich is one hun dred times faster than m icroprocessor-based lookups

currently dep loyed in the Internet. A single Internet Processor ASIC provides more than

enough power to perform wire-speed lookups for an 8xOC-48 system.

In add ition, the Internet Processor ASIC can be configured to p erform longest-match lookup s

with p er-prefix accounting. Per-prefix accoun ting prov ides statistics describing the n um ber of

bytes and packets forward ed to each p refix in the IP forwarding table. ISPs can use these

statistics to obtain a p icture of how traffic is flowing across their netw ork.


8/18



Programmability

The Internet Processor is both a fully generic and a fully programm able lookup en gine. In the

initial release, it su pp orts IPv4 (includ ing IP mu lticast) and MPLS. Because of the Internet

Processors program mab ility, sup port can be extended to includ e other protocols such as IPv6

and Frame Relay, simply by programming the Packet Forwarding Engine, developing new

routing software for the Routing Engine, and communicating the new forwarding table to the

Internet Processor.

The programm ability of the Internet Processor allows the existing JUNO S softwa re to continue

to evolve and remain fully supp orted in hard ware. As routing continues to develop in a

nu mber of un pred ictable ways, the progr amm ability of the Internet Processor ASIC allows

Jun iper Netw orks to futu re-proof the Packet Forward ing Engine by allowing it to supp ort new

functionality in hardware.

Performance Insurance

A key attribu te facilitated by the Internet Processor ASIC is performan ce insurance.

Performance insuran ce safeguard s system stability by segregating, and thu s protecting, the

operation of the routing and forwarding fun ctions. Whenever a netw ork failure strikes, three

events occur:

s The routing p rocess experiences stress because of the amou nt of routing change. The

routing stress is manifest in the need for each router to m aintain peer relationship s,

transmit and process routing u pd ate messages, calculate a new shortest-path tree, apply

policy to complete the route selection p rocess, and mod ify the forward ing table used by

the Packet Forwarding Engine.

s The packet-forwarding process experiences stress because rou ters that are still connected

to active network links sudd enly can find themselves forced to operate at peak capacity or

at even greater capacity rather than the usu al background level.

s The links an d routers forced to carry the ad ditional traffic resulting from the failure

condition become critical to the stability of the netw ork as a wh ole. If these netw ork

elements are u nable to carry the increased traffic load and fail, a relatively simple local

failure can cascade across the entire service provider s netw ork.

The challenge facing service provid ers is that trad itional rou ter architectures fail to deliver the

performance insurance that p rovides rock-solid system stability during failure cond itions.

Jun iper Netw orks believes that the M40 system prov ides the best solution in the indu stry for

accomplishing this critical design objective, supp orting perform ance insuran ce by:

s Implementing an architecture that distinctly separates the functions performed by the

Routing Engine and the Packet Forw arding EngineThis design segregates each

compon ent of the M40 system so that the stress experienced by each part of the system

does not negatively imp act the performance of the other part of the system.

s Ensuring that the lookup performance of the Internet Processor ASIC is never

comprom isedThe Internet Processor ASIC is fully sized to p erform lookup s at a rate of

40 Mpp s regardless of how long the lookup o r how large the routing table. The 40 millionlookups per second is achieved w ith 80,000 routes an d 80,000 distinct destina tion

add resses as opposed to artificial benchmarks that d o not reflect the current state of the

Internet.

s Allowing the Packet Forwarding Engine to m aintain forwarding performance when there

are high rates of up dates to the forwarding table, by supp orting th e revolutionary concept

of atomic updates to its forwarding table.


9/18



Atomic Updates

The Internet Processor ASIC sup ports atom ic up dates to its centralized forward ing table

(Figure 5). In the M40 system, the rou ting table contains the rou tes learned from rou ting

protocol exchanges w ith neighbors and through static configuration. The forwarding table is

derived from the rou ting table and contains an index of IP prefixes (or MPLS labels) that are

actively u sed to associate a p refix (or MPLS label) with an outgoing interface. The Packet

Forward ing Engine uses the contents of the forw arding tab le, not the routing table, to make itsforwarding decisions.

Typ ically, mod ifying a specific route affects only a sm all portion of the forw arding tables d ata

structure. This means that the Routing Engine simply needs to u pd ate a portion of the binary

tree in free mem ory and switch a pointer, and the up date is instantan eous. Because forward ing

information does not have to be distributed to m ultiple line cards, the Internet Processor does

not requ ire the forward ing table to be locked as the table is mod ified. Any attemp t to perform a

route lookup gets either the old tree or the new tree, dep end ing on wh ether the location is read

before or after the pointer has been changed. The benefit of this design is that the forward ing

table remains consistent at all times. This mean s that d uring period s of route instability, the

Packet Forwarding Engine can simultaneously accept updates to its forwarding table while

continuing to make forward ing decisions at an extremely rapid r ate.

Figure 5: Atomic Updates

Other high-performance router architectures perform packet lookups on each individual line

card. This means that w hen a routing change occurs and the forwarding table must be

mod ified, the u pd ate mu st be distributed to each of the ind ividu al line cards. During the

up date and distribution process, the forwarding table must be locked to maintain route

consistency as the centralized table is mod ified. As a result, packets are required to queu e up

and wait until the forwarding table is updated and then un locked. Only after the forward ing

table is unlocked can packet forwarding resum e. Clearly, locking the forward ing table can have

a negative impact on system performance during exceptional conditions when a high rate of

routing chan ge is coup led with a d rama tic increase in traffic flowing throu gh the router.

Forwarding

table

RoutingEngine

Packet

ForwardingEngine Internet Processor

Update

Routing

table

New subtreeDeleted subtree

New pointerOld pointer

Forwarding table

(Binary tree data structure)


10/18



Traffic Visibility

With backbone rou ters forwarding tra ffic at 40-Mpps rates, an ISP needs to und erstand the

traffic trend s that are tak ing place in its networ k. For examp le, new app lications can affect

overall network performa nce or the volum e of traffic can shift in d ifferent parts of the netw ork.

The traffic sampling featu re of the Internet Processor allows an ISP to sam ple trafficsay 1 ou t

of 1,000 packets.

Traffic sampling occurs by send ing a n otification w ith a sp ecial flag to another par t of the

system, a process that occurs withou t impacting the lookup performance of the Internet

Processor. A notification is a Juniper N etworksdefined data structure that contains all the

information need ed to process a packet after it has been stored in shared m emory. Based on th e

information contained in the notification, the packet then can be retrieved from shared

memory an d forwarded to a user process executing in the Routing Engine without imp acting

the performance of the Packet Forwarding Engine.

Switch Fabric: Distributed Buffer Manager ASIC, I/O Manager ASIC, and Shared Memory

The M40 system prov ides a conservatively rated 40-Gbps switch fabric that is imp lemented as

a shared mem ory system. In ad dition to the Internet Processor ASIC, the sw itch fabric is

composed of the Distributed Buffer Manager ASICs, the I/ O Manager ASICs, and the shared

mem ory system (Figure 6). In the illustra tion, note that each p acket is fragm ented into 64-byte

blocks for efficient storage in the sha red m emory system, wh ile a notification d escribing the

head er of the packet is forwarded to the Internet Processor ASIC for route lookup .

Figure 6: M40 Packet Forwarding Engine

From a vendor s persp ective, the chief limitation of a shared mem ory interconnect is that it is

technically difficult to design and implem ent. However, this does not m ean that ven dor s

shou ld avoid designing switch fabrics based on this architecture. For a system of this size, that

is 8xOC-48, or even a few times larger, a shared mem ory architecture presents a solid app roach

that results in numerous system benefits.

Distributed BufferManager

Shared memory


Inputinterface

Outputinterface

(Packet)

I/OManager

I/OManager

Linecard

Linecard

(Blocks)

(Packet)

Route lookup

Internet

Processor(Notification) Notification

Packets

Blocks


11/18



Distributed Buffer Manager ASIC and Shared Memory

Each of the M40 systems two Distributed Buffer Manager ASICs contains m ore than

10.5 million tra nsistors and 1.7 million ga tes. Like the Internet Processor ASIC, the Distributed

Buffer Manager ASIC offers comp lete flexibility, because it can be p rogram med by Jun iper

Networks to parse Layer 3 headers and generate route lookup keys based on current or future

Internet requirements.

The Distributed Buffer Manager ASIC man ages the M40 Internet backbone rou ters shared

mem ory system. Each Flexible PIC Concentrator prov ides 128 megabytes of shared m emory

and four Physical Interface Concentrator (PIC) line card slots. The Distributed Buffer Manager

treats the mem ory on each Flexible PIC Concentrator as p art of the systems shared mem ory

pool.

The Distributed Buffer Manager ASICs, combined with a shared m emory architecture,

contribute to the performance and robustness of the M40 switch fabric:

s The Distributed Buffer Manager ASIC allows Jun iper Netw orks to dev elop a system that

offers the simp licity of single-stage buffering; that is, a single write and a single read from

shared m emory. Shared mem ory is totally nonblocking as compared to a crossbar switch

architecture, which requires mu ltiple inpu t queu es to prevent head -of-line blocking.

s The capacity of the M40 shared m emory interconn ect is oversized so th at it easily can

sup port eight full-du plex line-rate OC-48s. This design allows the M40 to defer all queuing

and drop algorithms to th e outpu t interface, so the M40 never has to drop packets on an

inpu t interface.

s In add ition to th e benefits of single-stage bu ffering to m aximize memory u tilization, each

Flexible PIC Concentrator p rovides 128 megabytes of shared p acket buffer memory, thu s

supp orting large bandw idthdelay designs. Large band widthdelay designs permit end

station TCP sessions to keep the pipe full of data and achieve better performance.

s The use of shared m emory w ith single-stage buffering permits the constru ction of a

low-latency system. How ever, it should be emp hasized that latency is predominately an

issue for LAN systems, not WAN system s, for wh ich the bandw idthdelay design is the

primary concern.s The integrity of data stored in the SDRAM shared m emory is guar anteed by the use of

error-correcting code (ECC) memory. ECC memory gu arantees that ra nd om bit errors do

not affect data integrity.

s Because the system architecture keeps the Routing Engine completely ind ependen t from

the switch fabric, the Routing Engine does not consume switch bandwidth.


12/18



I/O Manager ASIC

Each I/ O Manager ASIC contains more than 10 million transistors and 1 million gates. The I/ O

Manager ASIC is required to p erform different functions for incoming and outgoing packets.

Hence, its primary resp onsibilities in the p acket-forward ing process have been a llocated to

three independent modules operating within the ASIC: the Input Processor, the Memory

Access Interface, and th e Outp ut Processor:

s The Inpu t Processor acts on incoming packets. It is responsible for receiving packets from

the line card , decoding the Layer 2 head er, and performing the packet-to-block

segmentation.

s The Memory Access Interface acts on both incom ing and outgoing packets. It

commu nicates with the Distributed Buffer Manager ASIC and hand les buffer mem ory for

a specific Flexible PIC Concentrator. For incoming packets, it is responsible for servicing

write block requests mad e by the Distributed Buffer Manager ASIC. For outbou nd

packets, the I/ O Manager ASIC services read block requests mad e by the Distributed

Buffer Manager ASIC.

s The Outp ut Processor acts on outbou nd packets. It is responsible for receiving packet

notifications from the Distributed Buffer Manager ASIC, performing class-of-service

selection, output notification queuing, weighted round-robin queue service, and RandomEarly Discard (RED and WRED); maintaining statistical counters; performing

block-to-packet reassembly; and providing the outbound Layer 2 encapsulation.

Similar to both th e Internet Processor ASIC and the Distributed Buffer Manager ASIC, the I/ O

Manager ASIC offers maximu m flexibility because it also is highly p rogram mable. It can be

programmed by Juniper N etworks to provide sup port for Layer 2 decoding and encapsulation

schemes su ch as PPP, Frame Relay, and MPLS. As far as sup por ting class of service, the I/ O

Manager ASIC offers num erous op tions for assigning a n otification to a queue and servicing

the outp ut qu eues, as well as controlling drop profiles and the packet discard process. Finally,

the I/ O Manager p lays an integral role in supp orting the write-once, read-many facility

required for the efficient forw ard ing of IP m ulticast traffic.

Line Cards

The M40 system line cards are imp lemented with m edia-specific ASICs. For example, Jun iper

Netw orks has integrated full SONET/ SDH processing on a single highly integrated A SIC. On

other vendor s systems SONET/ SDH p rocessing typically is performed by a n um ber of

different comp onents, not a single, highly integra ted ASIC that performs a ll requ ired

functions.

The benefits delivered by the M40 system line card ASICs are increased p ort d ensity, higher

performance, lower pow er draw s, and enhan ced reliability. Sup er-POPs require a router that

coinciden tly supp orts a large num ber of different interface types and high p ort densities so

that the router can grow and evolve as ISP requiremen ts change. As a result, the M40 system

can function in a broad range of sup er-POP environments d uring an y stage of an ISPs

development.


13/18



The wide variety of line cards and the nu mber of slots enhance the configuration flexibility of

the M40 system in a sup er-POP environment. A fully pop ulated M 40 system prov ides

32 slotsan ind ustry-leading p ort dens ity of one slot per rack-inchoffering comp lete

mix-and-match flexibility for line card installation. Because the switch fabric has been

oversized, all line cards op erate at w ire rate for all packet sizes. The line card s available in 1998

include:

s OC-48 IP over SONET/ SDH



s OC-12 IP over ATM

s OC-3 IP over ATM

s DS-3 with an internal DSU

s Gigabit Ethernet

The line cards can be mixed and matched in each slot as desired. (The exception is the OC-48

line card, wh ich u ses four slots.) Table 1 illustrates the maximu m den sities for each line card

type in a fully popu lated M40 system.

Table 1: M40 Interface Densities

Interface Type Maximum Ports Per M40 System Maximum Ports Per 7-Foot Rack

OC-48 SONET/ SDH 8 16


OC-12 ATM 32 64


OC-3 ATM 64 128

DS3 128 256


14/18

How Packets Traverse the M40 Packet Forwarding Engine


How Packets Traverse the M40 Packet Forwarding Engine

Figure 7 illustrates the path a packet takes as it is processed by the Packet Forward ing Engine.

A packet arrives at the M40 system on the inp ut interface (Step 1). The m edia-specific ASIC at

that interface performs all the med ia-specific details, such as rem oving the payload from the

SON ET frame, HDLC framing, and checksum verification. Jun iper Netw orks has designed an d

developed three m edia-specific ASICs: one for SONET, a second for ATM, and a third for DS3

interfaces.

Figure 7: Packet Forwarding in the M40 System

From the med ia-specific stage, a serial stream of bytes is passed to the I/ O Manager ASIC(Step 2). The I/ O Manager d etermines wh ether the frame is IPv4 (includ ing IP multicast) or

MPLS and identifies the beginn ing of the Layer 3 packet. The I/ O Manager also sets flags in

the p ackets notification tha t might be used for differentiated serv ices. Finally, the I/ O

Manager chops the p acket into 64-byte blocks and p asses each of the blocks to the Distributed

Buffer Manager ASIC (Step 3). These blocks are sized for efficient storage and retrieval from

shared mem ory an d are un related to 53-byte ATM cells. The Distributed Buffer Manager ASIC

distributes the blocks evenly in a round -robin fashion across the shared mem ory (Step 4).

In para llel to distributing each of blocks to shared mem ory, the Distributed Buffer Manager

ASIC also extracts the rou te lookup key from the blocks it receives and constructs a packet

notification. Recall that a packet notification is a Juniper Netw orksdefined (and

programmable) data structure that contains all the information needed to process a packet

after the packet has been stored in shared mem ory. For a unicast IPv4 packet, the Distributed

Buffer Manager ASIC determines the incoming interface, destination IP ad d ress, source IP

add ress, and p rotocol value, as well as the source and d estination TCP/ UDP port num bers.

For an MPLS frame, the Distributed Buffer Manager extracts a route lookup key that con tains

the incoming interface and the value of the MPLS label. After collecting this information, the

Distributed Buffer Manager forwards the notification to the Internet Processor ASIC (Step 5) so

that it can m ake a forward ing decision for the packet.

I/OManager

mediaspecific


Shared memorymediaspecific

InternetProcessor

IP forwardingtable


1 2

3

5

4

6

7a

8

9 10

Inputinterface

Outputinterface

7b 8

I/OManager

Notifications

Packets

Blocks


15/18

The M40 Internet Backbone Router in an ISP Network


The Internet Processor ASIC perform s the rou te lookup . For an IPv4 packet, it searches the IP

forward ing table (Step 6) looking for the longest m atch for the d estination p refix. For an MPLS

frame, the Internet Processor performs an exact match looku p in the MPLS forwarding tab le.

After performing the looku p, the Internet Processor passes the n otification message containing

the forwarding decision to the Distributed Buffer Manager ASIC (Step 7a). The Distributed

Buffer Manager forward s the notification to the I/ O Manager ASIC for the ou tpu t interface

(Step 7b). For an IP mu lticast fram e, the Internet Processor forwards a notification to the I/ OManager for each outpu t port.

The I/ O Manager ASIC on the outp ut interface is respon sible for managing packet queues. The

packet itself is not queu ed, but instead the notification for the packet is queued , while the

actual packet remains stored as blocks in shared mem ory. In the specific case of an IP m ulticast

packet, the I/ O Manager for each outp ut interface independ ently queues a packet notification.

For every outpu t port there are four queu es, each of wh ich has some configured share of the

physical links band width. The I/ O Manager on the ou tput interface can take a num ber of

factors into accoun t when d eciding to qu eue a packet, includ ing the value of the IP precedence

bits, utilization of the inpu t interface, destination ad dress, and RED/ WRED algorithms.

When th e packet notification reaches the front of its queu e and is ready for transmission, the

I/ O Manager issues a request through the Distributed Buffer Manager to read the packetsblocks from sha red m emory (Step 8). The I/ O Manager reassembles the blocks into the packet

and forward s the frame to th e med ia-specific ASIC on the ou tpu t interface (Step 9).

The med ia-specific ASIC on the outp ut interface perform s the necessary med ia-specific

operations, such as PPP-over-SON ET scrambling and H DLC framing , places the bits in a

SONET frame, iden tifies the beginning of the payload in the SONET frame, and then serializes

the bits on the fiber (Step 10). At this p oint, the packet leaves the Packet Forwarding Engine for

the next hop along the p ath towards the destination.


In add ition to providing the raw performance and righ t features for an OC-48 Internetbackbone rou ter, the M40 system ad dresses other issues that ISPs demand : rock-solid

reliability under failure conditions and ea sy dep loyability in super-POP environments.

Rock-Solid Reliability Under Failure Conditions

The prim ary concern for any ISP is the stability of the netw ork as a w hole, not just the

reliability of an individu al system. Even though the M40 system is designed to be extremely

reliable from both a ha rdw are and a software p erspective, the basic objective for all ISPs is to

localize the imp act of any n etwork ou tage. ISPs do not w ant their routers behaving in ways

that allow a local failure to cascade across the netw ork, creating ad ditional outages.


16/18



To p rovide an app ropriate level of stability, ISPs traditionally design th eir networ ks

conservatively and try to an ticipate the impossible so that customers never become

par titioned , resulting in a complete loss of service. To do this, ISPs dep loy redu nd ant system s,

install numerous backup links, and ru n the network a round 50 percent capacity du ring normal

operating conditions. The ability of the netw ork to accommod ate some nu mber of failures

wh ile still being able to sw itch all traffic is a critical design objective. ISPs follow th is

conservative app roach for several reasons:

s If a rou ter or link fails, the rest of the n etwork still has su fficient capacity to carry the

add itional traffic load. During exceptional cond itions, the ma jority of the network can

continue to run at 50 percent capacity, but certain network elemen ts might need to operate

at close to 100 percent capacity as the traffic load increases.

s Conservative designs provide ISPs with a bit of wiggle room during p eriods of high

growth and rap id evolution. If an ISP has failed to prov ision adequ ate capacity in the

network topology, it might be willing to accept some add itional risk and let through pu t at

that point dr ift up to 60 or 70 percent until adequate transm ission capacity is available.

The challenge w ith this conservative app roach is that existing routers generally operate well

und er low load conditions, but they are not engineered to provide stable performance wh en

the traffic load jump s d rama tically to 100 percent. The M40 Internet backbone rou ter is un ique

in its ability to provide the rock-solid stab le performance that other systems lack du ring

periods of extreme stress:

s The M40 Internet backbone rou ter is fully sized with resp ect to both route processing and

packet forward ing. During exceptional cond itions, the Routing Engine continu es to

receive and transmit routing updates, perform route calculations, maintain peer

relationship s, react to interfaces going down , and so forth. Similarly, the Packet

Forward ing Engine continues to sw itch packets at a rate of 40 Mpps regard less of packet

size or load on the system .

s Complementing the architectural separation of the routing and packet-forward ing

processes, atom ic up da tes perm it the state of the Packet Forw arding Eng ine to concur w ith

the state of the Routing Engine without impacting forwarding performance. During

exceptional cond itions, atomic upda tes allow th e M40 system to avoid destab ilizing thelinks that still remain up , thus eliminating the p rimary reason for cascad ing failures.

s The traffic engineering featu res sup ported by the JUNOS Internet software allow ISPs to

man age around network failures. Lacking the p refailure transm ission capacity, the M40

system provid es tools that permit an ISP to determ ine the best method for d istributing the

current traffic load over available resources without creating congestion and further

destabilizing the network.


17/18



M40 System Deployment

The M40 system has been specifically designed to ad dresses the num erous challenges facing

ISPs in the su per-POP environm ent (Figure 8) as they transition their networ ks from existing

OC-3 and O C-12 based cores to OC-48 based cores. The M40 Internet backbone rou ter prov ides

indu stry-leading metrics in terms of:

sSizeEach M40 Internet backbone rou ter chassis is 35 in height, so tw o chassis can beinstalled in a single seven -foot-high rack.

s Power d raw The M40 system ad dresses the logistical issue of adequa te pow er

availability by draw ing less than 1 amp per rack-inch.

s Performance den sityThe M40 system p rovides forward ing performance of greater than

1 Mpps p er rack-inch, wh ich is ten times the performance currently available from other

systems targeted a t the Internet core.

s Slot densityA fully popu lated M40 system prov ides one slot per rack-inch, wh ich is

three to four times the slot density ava ilable from oth er systems targeted for sup er-POP

environments.

The M40 Internet backbone rou ter represents an en tirely new class of service provider system ,

with sp ecialized features that are familiar to those working in a telecommu nicationsenvironm ent. For examp le, a craft interface perm its a techn ician to m onitor the statu s of the

system, troubleshoot the system with help from a remote Network Operations Center (NOC),

and perform a nu mber of system functions.

Figure 8: Super-POP Environment

Router M40Switch

M40Switch

OC-48

OC-48

OC-48

OC-3, OC-12, or

Gigabit Ethernet

Legacy

DS0, DS1

Access

Access Intra-POP Core Backbone

Dense

DS3

Customer

Access

Router

Gigabit Ethernet

or OC-12

OC-3, OC-12, or

Gigabit Ethernet

Gigabit Ethernet

or OC-12


18/18

Conclusion


Conclusion

The M40 Internet backbone rou ter represents an en tirely new class of routing system that has

been sp ecifically d esigned to help ISPs negotiate the transition from O C-3 and OC-12 to OC-48

based backbones. The core of the Internet is constan tly developing along tw o dimen sions,

software richness and ban dw idth. The M40 system prov ides all the features that a next

generation routing system m ust supp ort:

s The Packet Forwarding Engine is oversized w ith packet processing and a switch fabric to

effortlessly sup port 8xOC-48 interfaces at full wire-speed. The M40 system provid es the

forward ing performance previously found only in switches, withou t sacrificing the

elements of network control.

s The Routing Engine executes indu strial-strength, full-featured rou ting p rotocol and traffic

engineering software designed and written by acknowledged ind ustry experts.

s The fun damen tal architecture of the M40 systemcomplete separation betw een the

routing and packet-forwarding functionshas been d esigned with the goal of providing

perform ance insuran ce to enhance the stability of large ISP networ ks du ring exceptiona l

conditions. The M40 system is un ique in its ability to contribu te to netw ork stability and

adap t to highly fluctuating environments without impacting other parts of the network.

s Programm able, comp uter-scale ASICs allows the JUNOS Internet software to continu e to

evolve while futu re-proofing the Packet Forwarding Engine by allowing it to sup port new

functionality w ithout hard ware changes.

s The variety and den sity of interfaces, as well as the m echanicals and ser viceability, make

the M40 Internet backbone rou ter eminen tly dep loyable in the core of large ISP netw orks.

s By p roviding a routing system w here all four fundam ental comp onents are equally

pow erful routing software, packet processing, the switch fabric, and line cardsthe

M40 system d elivers a comp lete solution th at is uniqu e in its ability to operate su ccessfully

in the core of the Internet.

Junip er Netw orks is a registered trad emarks of Jun iper Netw orks, Inc. All other trad emark s, service marks, registered trad emar ks, or registere

service marks mentioned in this document are the property of their respective owners.

Copyrigh t 1997, Jun iper N etwork s, Inc.

All rights reserved. Printed in USA.

backbone routers

Documents