design of a diversified router: model and system overview

Jon Turner, John DeHart, Fred [email protected], [email protected],

[email protected]://www.arl.wustl.edu/arl

Design of aDiversified Router:

Model and System Overview

2 - JST/JDD/FK - 04/19/23

Outline What is NOT covered in these slides JST’s original slides Schedule Model Traffic Types IP Addressing Components

»Switch»MetaLink Loopback Block»LC

Substrate Link Types Packet Formats LC Rx/Tx Design Implementation

Common Router Framework (CRF)»Functional Blocks for implementing a Router

3 - JST/JDD/FK - 04/19/23

Not Covered Control

»Installation»Configuration»Initialization»Monitoring»…

End Host»Substrate/Meta Model

Details about how PlanetLab works. These are very important topics but for now we will talk

about the Data Path in the Network.

4 - JST/JDD/FK - 04/19/23

Outline What is NOT covered in these slides JST’s original slides Schedule Model Traffic Types IP Addressing Components




5 - JST/JDD/FK - 04/19/23

System ArchitectureGeneral purpose blades.

» shared blades run Plab OS no change to current apps

» also support dedicated blades» use separate blade server to

preserve ATCA slots for NPsNP blades.

» support dedicated PEs control from Vserver on PE/GP

» shared PE options shared NP for fast path shared NP with plugins

10 GE fabric switch» VLANs used to isolate

metarouters» uplinks for connecting to

multiple chassesGood ratio of PEs to LC: 3:1

10 GE Switch

Line C

ard

Switch Blade

PE/G

P

PE/N

P

. . . . . . up to 10 1GEinterfaces

compute blade with disk

Radisys7010

Radisys 7010 with RTM

1 GE for control10 Gb/s for data

6 - JST/JDD/FK - 04/19/23

Sharing NPs Motivation

»MRs with limited IO bandwidth can’t make full use of NP blade

»extreme (but perhaps common) case – 25 MRs with 100 Mb/s IO

»useful to allow such MRs to share a PE Packet flow

» ingress LC terminates external tunnel and adds internal header

»MR fast path determines outgoing metalink and queues packet

»selected packets directed to Vserver on shared GP blade» injected packets placed on metalink queues in PE

Configuration»single NP blade hosts two shared NPs»each NP engineered for 5 Gb/s throughput»divide TCAM between NPs

LC

P L Q

P L Q

. . .

shared NP blade

S S

shared GP blade

. . .

LC. . .

7 - JST/JDD/FK - 04/19/23

DeMux Parse Lookup

...

HeaderFormat

Shared PE Processing

header includes MR+MI

use MR to find parser

codeand MR

control block

•inputs (in regs)•buffer pointer•MI•ctl blk pntr•output pntr

•output•16B lookup key

•code constraints•finite, acyclic flow graph•bounded path length•restricted memory accesspacket buffer, control block, output area

•thread-safe

•input: MR+lookup key•output:

•output MI•qid (supplied by substrate)

•lookup result•stats index

per metalink

queues with static rates

•inputs (in regs)•buffer pointer•output MI•ctl blk pntr•lookup result pntr

•output•buffer offset

Stats module not shown – inputs from DeMux, Lookup and QM. Potential for plugins using spare MEs

»one ME per MR, static SRAM regions,

8 - JST/JDD/FK - 04/19/23

Managing MR-Specific Data MR control software runs on separate PE. xScale provides interface through control daemon.

»enable/disable MR (input packets discarded when disabled)»read/write MR control block

block read/writes to from memory area using offsets

»read/write filters substrate fills in hidden fields (MR in key, qid in result)

»read statistics»read/write code segments

cryptographic hash used to verify compliance-checking only when MR is disabled

9 - JST/JDD/FK - 04/19/23

Line Card Functions Substrate functions only Ingress

»terminate tunnels (VLAN, plain IP, IP tunnel, MPLS)»map physical interface + tunnel id + MLI to MR+MI»map MR+MI to destination blade and qid»queue packets in queues with static rates (paced)»provision for xScale to respond to ARP requests

Egress»arriving packets include MR+MI+(next-hop-metanet-adr)»monitor rates for each MR/MI pair – raise alarm if threshold exceeded» if MI is designated as multipoint,

map next-hop-metanet-adr to MAC adr and write into blank spot in buffer if no matching entry, pass buffer pointer to ARP handler in xScale

xScale handles ARP, fields reply and enqueues packet when reply received if no reply, send exception notification MR exception interface

»place packet in outgoing per-MI queue

10 - JST/JDD/FK - 04/19/23

Possible Extensions Queueing in shared PE

»allow MR to define multiple queues per MI with queue weights, per queue space allocation and per MI space allocation

»substrate implements fair queueing scheduler WDRR or stratified round-robin packet discarded if queue using more than its space allocation and

more than its share of the shared space

»implement when users demand it or spare time available Limited Plugins

»replace Header Formatter with parallel block of MEs, one per MR»code restrictions (static and run-time checks)

access to restricted address ranges restricted use of bus bandwidth resources

instructions inserted to increment usage counters xScale (or perhaps dedicated ME) checks counters periodically MRs using excessive bus bandwidth are disabled

11 - JST/JDD/FK - 04/19/23

Outline JST’s original slides Schedule Model Traffic Types IP Addressing Components




12 - JST/JDD/FK - 04/19/23

Schedule Two time horizons:

» techX Project End of 2006

» Future Diversified Router Project Outstanding new grant request under which we do the rest of the work…

JDD’s Proposal: techX End of Year Target:» Data Path

Default IPv4 MR We should be able to use N instances of this to show multiple MRs running in a Substrate Router.

Legacy PlanetLab nodes in a Blade Server By Legacy we mean receiving/sending plain IP packets, no Meta/Substrate functionality present.

Basic LC Substrate functionality IP Tunnel Substrate Link termination Plain IP handling

MetaLink Loopback Block» Control?

Install an MR Configure an MR? Routing Protocols?

Can this be handled through static route configuration? Etc.?

» End Host: Use Plain IP?

13 - JST/JDD/FK - 04/19/23





14 - JST/JDD/FK - 04/19/23

Substrate/MetaNet Model

MR_A

MR_C

MR_A

MR_C

MR_A

MR_C

MR_B MR_B

The Substrate should provide a view to Meta Nets that their Meta Routers are directly connected via Physical Links»A Meta Interface emulates a Physical Interface»A Meta Link emulates a Physical Link

The Substrate should provide isolation between Meta Nets

MI MI

MIMI

MI MI

MI MI

MI MI

MIMI

MI MI

MI MI

Meta Net “A”

Meta Net “B”

Meta Net “C”

15 - JST/JDD/FK - 04/19/23

Substrate/MetaNet Model

MR_A

MR_C

MR_A

MR_C

MR_A

MR_C

Substrate Link

MR_B MR_B

MetaLink 1MetaLink 2MetaLink 3

SubstrateRouter X

SubstrateRouter Y

SubstrateRouter Z

Substrate Links connect Substrate Router Peers» A Substrate Link is terminated at a Physical Interface of a Substrate Router.

Meta Links connect Meta Router Peers» A Meta Link is terminated at a Meta Interface on a Meta Router

MI MI

MIMI

MI MI

MI MI

MI MI

MIMI

MI MI

MI MI

16 - JST/JDD/FK - 04/19/23

Mapping the Model onto Hardware

LC Loopback

Switch

DedicatedNP

DedicatedGP

SharedNP

SharedGP

Substrate

Meta

17 - JST/JDD/FK - 04/19/23

Mapping the Model onto Hardware Line Card

»Typically supports multiple Physical Interfaces Our Radisys blade when implementing a LC would have an RTM with 10 1-GE

physical interfaces.»Substrate Link Termination

Including IP Tunnel Substrate Link termination Mux/Demux of Meta Links into/out of Substrate Links. Pass through Meta Links

»Plain IP Switch

»Substrate Switch Meta Routers on shared blades do not have a view of the switch. All switching functions are provided by the Substrate Traffic from all MRs is isolated from other MRs, including dedicated blade MRs,

via the use of VLANs»Meta Switch

An MR on a dedicated blade has a view of a Meta-Switch including: Port its blade is on Portions of LCs that it has access to.

Loopback (next slide) NP: Network Processor Blade GP: General Purpose Processing Blade

18 - JST/JDD/FK - 04/19/23

Mapping the Model onto Hardware Line Card Switch Loopback

»Provides MRx:MIi connectivity to MRy:MIj within Substrate Router»More on this later when we provide details on the components…

NP: Network Processor Blade»Dedicated: entire blade is dedicated to one MR.

No Substrate functionality present.»Shared: blade is shared among several MRs.

Substrate functionality provides resource sharing and isolation GP: General Purpose Processing Blade

»May be realized by a blade in the ATCA chassis or in a blade server directly connected to switch blade which resides in ATCA chassis.

The switch blade has front panel Ethernet interfaces (uplinks) that could be connected to the blade server for this purpose.

»Dedicated: entire blade is dedicated to one MR. No Substrate functionality present.

»Shared: blade is shared among several MRs. Substrate functionality provides resource sharing and isolation.

19 - JST/JDD/FK - 04/19/23

Model: Layering and Abstractions Two Types of Traffic

»Legacy MAC layer Legacy Layer (i.e. non-Substrate, typically IPv4)

»Substrate MAC layer Substrate layer Meta layer

Figures below are LAYERS, not Packet formats:

Legacy

MAC

Meta

Substrate

MAC

20 - JST/JDD/FK - 04/19/23

Layer Attributes Attributes at the different layers

»Physical/MAC Type: (Ethernet, ATM, …) Wireless: (1/0) Type: (P2P, MA, …)

»Substrate MNid MRid MLI Link Type Link Model (P2P, Multi-Access) ARP support

»Meta MetaLink MI MnAddr

What is exposed to the MR?»Type: {Ethernet, Wireless, …}»P2P vs. Multi-Access»ARP Provided by Substrate or Not.»???

21 - JST/JDD/FK - 04/19/23





22 - JST/JDD/FK - 04/19/23

Traffic at a Substrate Router For now assume traffic is on wired Ethernet. Substrate:

»EtherType = Substrate»Each frame has a Substrate Header

MLI: Meta Link Identifier Identifies MR:MI to which this frame should be delivered

Len

»Line Card: sends Meta Frame to the Blade hosting the MR indicated by the MLI includes in the Meta Header the MR’s MI that this frame is for.

IP: Other:

23 - JST/JDD/FK - 04/19/23

Traffic at a Substrate Router Substrate: IP:

EtherType = IP» Substrate Tunnels:

IP Proto = Substrate Each frame has a Substrate Header immediately following IP Header

MLI: Meta Link Identifier– Identifies MR:MI to which this frame should be delivered

Len Line Card sends Meta Frame to Blade hosting MR indicated by MLI

Included in Meta Header is the MR’s MI that this frame is for.» Plain IP Control Protocols:

Contains no Substrate Header ICMP

Substrate on LC may need to process some ICMP messages. ICMP or a subset of ICMP messages sent to XScale on LC. Depending on processing in Substrate/XScale, ICMP messages may be forwarded on to MR(s).

BGP: Border Gateway Protocol (inter-domain routing protocol) Treated like Plain IP Data described below: Sent to default IP MR.

OSPF: Open Shortest Path First (intra-domain routing protocol) Treated like Plain IP Data described below: Sent to default IP MR.

IGMP: Internet Group Management Protocol, used for IPv4 Multicast Group membership Treated like Plain IP Data described below: Sent to default IP MR.

Other?» Plain IP Data:» Other?

Other:

24 - JST/JDD/FK - 04/19/23

Traffic at a Substrate Router Substrate: IP:

»Substrate Tunnels:»Plain IP Control Protocols:»Plain IP Data:

Contains no Substrate Header Line Card forms Meta Frame and sends it to default IP MR

A unique MI for the default IP MR is associated with each Physical Interface. Via the default IP MR, plain IP traffic may be destined for:

Other MRs on this Substrate Router via the Loopback Other MRs on other Substrate Routers via Meta Links carried on Substrate Links

– The Default IP MRs on peer Substrate Routers could exchange route information so they know how to get to other Substrate Routers.

Legacy (Pure IP) Planet Lab Nodes on this Substrate Router. Substrate Control Entities on XScale? Egress as Plain IP?

– Policy based option

»Other? Other:

»ARP:»Other?

25 - JST/JDD/FK - 04/19/23





26 - JST/JDD/FK - 04/19/23

IP Address Assignment

PlanetLabBlades

2

1

3

X.Y.Z.1

Sw

itch

U.V.W.1

R.S.T.1

Substrate Router(A.B.C.0 – A.B.C.255)

Consider a Substrate Router (SR) with the following characteristics:» 3 physical interfaces that connect to three

Internet Routers: Interface 1 X.Y.Z.1 Interface 2 U.V.W.1 Interface 3 R.S.T.1

» Blade server that contains 6 Legacy (Plain IP) PlanetLab Nodes

» A Default IPv4 MR» Block of IP Addresses A.B.C.0 – A.B.C.255

27 - JST/JDD/FK - 04/19/23


A.B.C.1

PlanetLabBlades

2

1

3

X.Y.Z.1A.B.C.128

A.B.C.129

A.B.C.130

A.B.C.2A.B.C.3A.B.C.4

Sw

itch

A.B.C.5A.B.C.6A.B.C.7U.V.W.1

R.S.T.1

X.Y.Z.2

U.V.W.2

R.S.T.2


Assign addresses to the SR as follows:» Each Physical Interface has a Substrate

IP Tunnel Address Interface 1 A.B.C.128 (A.B.C.0x80) Interface 2 A.B.C.129 (A.B.C.0x81) Interface 3 A.B.C.130 (A.B.C.0x82)

» Default IPv4 MR has 4 IP Addresses: To X.Y.Z.1 out Interface 1 : X.Y.Z.2 To U.V.W.1 out Interface 2 : U.V.W.2 To R.S.T.1 out Interface 3 : R.S.T.2 To PlanetLab Nodes: A.B.C.1

28 - JST/JDD/FK - 04/19/23


A.B.C.1

PlanetLabBlades

2

1

3

X.Y.Z.1A.B.C.128

A.B.C.129

A.B.C.130


Sw

itch


U.V.W.1

R.S.T.1

X.Y.Z.2

U.V.W.2

R.S.T.2

Route announcements:» Out Interface 1

A.B.C.128/32 A.B.C.0/25

» Out Interface 2 A.B.C.129/32 A.B.C.0/25

» Out Interface 3 A.B.C.130/32 A.B.C.0.25


29 - JST/JDD/FK - 04/19/23

BGP Notes RFC 4271: BGP-4 BGP is defined as an inter-Autonomous System (AS) routing

protocol. Autonomous System: A set of routers under a single technical

administration BGP Speaker: A router that implements BGP BGP Identifier: an IP Address assigned to a BGP Speaker. Same for

every local interface and BGP peer.»A router may also have separate IP Addresses for its interfaces.

UPDATE message used to exchange route information»NEXT_HOP attribute defines IP address of the router that SHOULD be

used as the next hop to the destinations listed in the UPDATE message. Would this potentially be used in ARP requests?

BGP uses TCP between peers. From RFC 4271, Section 3:

» In the context of this document, we assume that a BGP speaker advertises to its peers only those routes that it uses itself (in this context, a BGP speaker is said to “use” a BGP route if it is the most preferred BGP route and is used in forwarding).

30 - JST/JDD/FK - 04/19/23


»MetaLink Loopback Block»Switch»LC



31 - JST/JDD/FK - 04/19/23

Meta Link Loopback We would like to be able to route Plain IP pkts to an MR

»We can achieve this by using a default IPv4 MR»But we don’t want the IPv4 MR to have to know anything about other

MRs and their Meta-Interfaces That is a Substrate function

»So, we need a Substrate function to translate from MRx:MIi to MRy:MIj

We would also like to be able to route Plain IP pkts to Legacy PlanetLab Nodes residing on Blades in a Substrate Router»Legacy PlanetLab Nodes have no Substrate.»Again we can use the default IPv4 MR to achieve this.»But the packets cannot have a Substrate Header or Meta Header when

they arrive at the PlanetLab Nodes.»So, we need a Substrate function to strip these headers.

We would also like to be able to do similar things with IP Pkts in Substrate Frames on MetaLinks which terminate at the default IPv4 MR

We will implement a Meta Link Loopback block»Provides Internal Meta Link Loopbacks at the Substrate level.»Probably will reside on one or more Line Cards.

32 - JST/JDD/FK - 04/19/23

Meta Link Loopback

Loopback

DefIPv4MR

MR X

MR Y

PLab

LC B

Substrate Router

n

Physicalinterface ‘n’

on LC B

LC A

n

m

Physicalinterface ‘n’

on LC A

Switch

Physicalinterface ‘m’

on LC A

33 - JST/JDD/FK - 04/19/23

Meta Link Loopback: Plain IP to/from MR Y

Loopback

DefIPv4MR

MR X

MR Y

PLab

IP

IP

IP

Y

LC B

IPBn

IPYk

IPYkYk

YkY

a VLAN ‘a’ used.

Ab : MI b associated with MR A used

Substrate Router

n IPBn

LC A

IPAmIP

IP

n

IPAm

Plain IP

Plain IP m

Switch

IP

34 - JST/JDD/FK - 04/19/23

Meta Link Loopback: Plain IP to/from PLab

Loopback

DefIPv4MR

MR X

MR Y

PLab

IP

IP

LC B

IPBn a VLAN ‘a’ used.


Substrate Router

n IPBn

LC A

IPAmIP

IP

n

IPAm

Plain IP

Plain IP m

Switch

IP IPPi

P

IPPi

Plain IP

P Plain IP

35 - JST/JDD/FK - 04/19/23

Meta Link Loopback: General

Loopback

DefIPv4MR

MR X

MR Y

PLab

IPIPIPIP

IP

PXY

IP

LC B

IPBnIPPiIPXjIPYk

IPPiIPXjIPYk

XjYk

Xj

Yk

Plain IP

P

X

Y

a VLAN ‘a’ used.


Substrate Router

n IPBn

LC A

IPAmIP

IP

n

IPAm

Plain IP

Plain IP

Substrate

Plain IP m

IP

IP

IPAn

IPAn

Switch

36 - JST/JDD/FK - 04/19/23


»MetaLink Loopback Block»Switch»NP Blades»LC



37 - JST/JDD/FK - 04/19/23

Switch Switch Blade Specs:

»Promentum™ ATCA-2210 » http://www.radisys.com/products/ds-page.cfm?productdatasheetsid=1191»20-port 10GE fabric switch

14 10GE links to user slots 4 10GE links for external connections (up/cross links) on front panel

»24-port 1GE Base switch 14 1GE links to users lots 1GE link to redundant switch blade 1 10GE and 4 1GE links for external connections (up/cross links) on front panel

»Wire-speed L2 and L3 switching»4K IEEE 802.1Q VLANs»Etc…

Traversing the Switch:»Switching is based on Ethernet Destination Address» Isolation is based on VLAN.

One VLAN will be assigned to each MetaNet present on a Substrate Router. All switch traffic for a MetaNet will be required to use its assigned VLAN.

Frames from a MetaNet will only be transmitted to a port which is allowed to receive the specified VLAN.

38 - JST/JDD/FK - 04/19/23


»MetaLink Loopback Block»Switch»NP Blades»LC



39 - JST/JDD/FK - 04/19/23

NP Blades: Radisys 7010

40 - JST/JDD/FK - 04/19/23

NP Blades: Radisys 7010

SPI-4 Switch supports 6 Interfaces» Each Interface can support up to 16 SPI-4 Ports» Each Port on an Interface can be switched to any other Port on any other Interface

Radisys 7010 uses 4 of the 6 Interfaces 10G MAC Chip (IXF18105) supports and uses only 1 SPI-4 Port on its SPI-4 Interface

» Therefore, all traffic from 10G MAC Chip must go to one NPU, either NPUA or NPUB.» This should be fine for our LC in the future

One NPU will be for Ingress and one for Egress» For our NPE Blades, this will be a bit of pain.

We can direct all traffic to NPUA and then have a block right after RX that splits the traffic for NPUA and NPUB Traffic for NPUA continues on to the processing blocks on NPUA Traffic for NPUB would go directly to a TX block to be transmitted to NPUB via the SPI Switch. (Does not go to 10G MAC chip)

Still need to work out whether this will actually work or not.

10GbPer PortEthernetSwitchBlade

FIC

10GMAC

SPI-4Switch

NPUA

NPUB

/1

1/

10GbPer PortEthernetSwitchBlade

ATCABackplane

16//

16

16//

16

41 - JST/JDD/FK - 04/19/23

NP Blades: Radisys 7010 Radisys 7010

»Two IXP2850 NPs 1.4 GHz Core 700 MHz Xscale Each with:

3x256MB RDRAM, 533 MHz 3 Channels of QDR II SRAM, 8MB each, 200 MHz 16KB of Scratch Memory 16 Microengines

– 8KB instruction store– 640 32-bit words of local memory

Network Search Engine on 4th QDR channel Shared between NPs IDT75K72234 18Mb TCAM

SPI-4 Switch for interconnections

42 - JST/JDD/FK - 04/19/23

Line Cards : Radisys 7010 with RTM Radisys 7010 with RTM

»Same board as NP blade. Two IXP2850 NPs

One used for Ingress One used for Egress

»Rear Transition Module (RTM) Connects via ATCA Zone 3 10 1GE Physical Interfaces

Supports Fiber or Copper interfaces.

43 - JST/JDD/FK - 04/19/23

Radisys 7010 Blade

44 - JST/JDD/FK - 04/19/23





45 - JST/JDD/FK - 04/19/23

Substrate Links 2 Types of Substrate Links:

»Point-to-Point (P2P) Connects two peer Substrate Routers May be realized via:

Directly Connected (P2P-DC)– One physical link connects an Interface on one Substrate Router to an Interface on

another Substrate Router– Can use VLANs or not.

Tunnel (P2P-Tunnel)– Two Substrate Routers have a Tunnel defined that connects an interface on each

router.– Such a tunnel comprises exactly one Substrate Link.– Can use VLANs or not.

Multi-Access Network (P2P-VLAN0)– Substrate Routers with interfaces connected to a Multi-Access Network all use a

predefined VLAN (VLAN0).– VLAN0 is defined for each Multi-Access Network and my be different on each one.– Each of these Substrate Routers is pre-configured with the Ethernet addresses of

the other Substrate Routers connected to the Multi-Access Network.– By using the Ethernet address of peers, pairs of Substrate Routers have defined

Substrate Links.– With N Substrate Routers, there are at most (N*(N-1))/2 Substrate Links

»Multi-Access

46 - JST/JDD/FK - 04/19/23

Substrate Links 2 Types of Substrate Links:

»Point-to-Point (P2P)»Multi-Access

Supports Unicast, Multicast and Broadcast traffic from Meta Routers. Interfaces on Substrate Routers connected to a Multi-Access network

are NOT necessarily pre-configured with each others Ethernet addresses.

ARP may be used over the Multi-Access network to do the translation from MetaNet Address to Ethernet Address.

ARP protocol and tables are managed by XScale. Can use VLANs or not, but cannot use VLAN0.

47 - JST/JDD/FK - 04/19/23

Substrate Links Substrate Link Restrictions:

»P2P-DC does not co-exist on a physical interface with any of the other types of SL.

»P2P-Tunnel, P2P-VLAN0 and Multi-Access may all co-exist on the same physical interface.

»A physical interface may have: At most one P2P-DC SL

XOR One or more P2P-Tunnel SLs One or more P2P-VLAN0 SLs At most one Multi-Access SLs

48 - JST/JDD/FK - 04/19/23

IP Tunnel Substrate Links Each IP Tunnel Substrate Link is terminated at a single

Physical Interface on a Substrate Router. The only route TO this end of the tunnel is from the

outside into the associated physical interface.»Nothing inside the Substrate Router routes TO this end of the

Tunnel»From inside the Substrate Router packets are just sent INTO the

tunnel so they come out the other end. Included in the configuration of an IP Tunnel Substrate

Link is the Address (IP or MAC?) of the next/first hop for the route to the other end of the tunnel.»What happens if the routes in the Internet change and this

next/first hop is no longer the way to go? Physical Interface will receive ICMP No Route to Host or ICMP Redirect

messages. ICMP messages should be sent to the XScale on the Line Card. XScale should then nullify (mark as down, …) the IP Tunnel Substrate Link

and all MetaLinks/MetaInterfaces associated with that Substrate Link.

49 - JST/JDD/FK - 04/19/23



Substrate Link Types Packet Formats

Focus is on wired Ethernet LC Rx/Tx Design Implementation


50 - JST/JDD/FK - 04/19/23

LC

Point-to-Point Directly Connected (P2P-DC)

MR

MR

MRLC

MRMetaLinks

P2P-DC SL PacketWith VLANs

Type=802.1Q (2B)

MLI (2B)LEN (2B)

Meta Frame

TCI (2B)Type=Substrate (2B)

PAD (nB)

CRC (4B)

DstAddr (6B)SrcAddr (6B)

P2P-DC SL PacketWithout VLANs

MLI (2B)LEN (2B)

Meta Frame

Type=Substrate (2B)

PAD (nB)

CRC (4B)

DstAddr (6B)SrcAddr (6B)

P2P should not need the Type=Substrate since it is a point-to-point physical link and should only be carrying Substrate traffic.»Configuration/Policy/

Implementation option?

51 - JST/JDD/FK - 04/19/23

LCLC

Point-to-Point via Tunnel (P2P-Tunnel)

MR

MR

MR

MR

TunnelSubstrate Link

MetaLinks

P2P-Tunnel SL PacketWithout VLANs

P2P-Tunnel SL PacketWith VLANs

Type=IP (2B)

MLI (2B)LEN (2B)

PAD (nB)

CRC (4B)

Meta Frame

Dst Addr (4B)Src Addr (4B)

Ver/HLen/Tos/Len (4B)ID/Flags/FragOff (4B)

TTL (1B)Protocol=Substrate (1B)

Hdr Cksum (2B)

Type=802.1Q (2B)TCI ≠ VLAN0 (2B)

DstAddr (6B)

SrcAddr (6B)

Type=IP (2B)

MLI (2B)LEN (2B)

PAD (nB)

CRC (4B)

Meta Frame

Dst Addr (4B)Src Addr (4B)



Hdr Cksum (2B)

DstAddr (6B)

SrcAddr (6B)

This example shows a Substrate Link in an IP tunnel over Ethernet.

We could also envision other tunnel types:» MPLS» ATM» Sonet?» Etc…

Ether/MAC layer encap Tunnel Protocol

Tunnel layer encap Substrate

52 - JST/JDD/FK - 04/19/23

MPLS Tunnels We’ve looked at MPLS just enough to convince

ourselves that we can do something, but not enough to understand all the implications.»Some of the MPLS RFCs:

3031: Multiprotocol Label Switching Architecture 3032: MPLS Label Stack Encoding 3037: LDP Applicability

LDP: Label Distribution Protocol 4221: Multiprotocol Label Switching (MPLS) Management Overview

MPLS uses a 32-bit field which contains»Label (20b)»Experimental (3b)»S-Bit (1b): 1 indicates bottom of label stack»TTL (8b)

53 - JST/JDD/FK - 04/19/23

LC

LC

LC

LC

Point-to-Point via Multi-Access Network (P2P-VLAN0)

MR-1

M3

MR-4

MR-3

MR-2

MR-4

MR-1

MR-2

EthernetSwitch

MR-2

MR-2

For this example, on each LC pictured, one interface is being used.

54 - JST/JDD/FK - 04/19/23

P2P-VLAN0 SL Packet

(VLAN0)

Type=802.1Q (2B)

MLI (2B)LEN (2B)

Meta Frame

TCI = VLAN0 (2B)Type=Substrate (2B)

PAD (nB)

CRC (4B)

DstAddr (6B)

SrcAddr (6B)

P2P-VLAN0 Packet Format

VLAN=VLAN0 provides isolation from other traffic on the Multi-Access network.

55 - JST/JDD/FK - 04/19/23

Multi-Access

LC

LC

LC

LCMR-1

MR-2

MR-1

MR-2

MR-3

MR-2

MR-1

MR-3

EthernetSwitch

One MLI used per MetaNet on a Multi-Access Substrate Link.

We may want to use the MnID for the MLI for each.

56 - JST/JDD/FK - 04/19/23

Multi-Access Packet Formats

Multi-Access SL Packet

without VLANs

Type=Substrate(2B)MLI (2B)LEN (2B)

DstAddr (6B)

SrcAddr (6B)

Meta Frame

PAD (nB)

CRC (4B)

Multi-Access SL Packet with VLANs(VLAN≠0)

Type=802.1Q (2B)

MLI (2B)LEN (2B)

Meta Frame

TCI ≠ VLAN0 (2B)Type=Substrate (2B)

PAD (nB)

CRC (4B)

DstAddr (6B)

SrcAddr (6B)

57 - JST/JDD/FK - 04/19/23

ARP for IP Packet Format Ethernet Header

»Dst Addr»Src Addr»Type = 0x0806 (ARP)

Hard Type: Type of hardware address»1 Ethernet

Proto Type: Type of protocol address being mapped»E.g. for IP = 0x0800 (same as EtherType for IP)

Hard/Proto Size: size in bytes of address fields OP: Operation

»1: ARP Request»2: ARP Reply»3: RARP Request»4: RARP Reply

Sender Eth Address Sender IP Address Target Eth Address Target IP Address

Type=ARP(2B)HardType=Ether (2B)ProtoType=IP (2B)

DstAddr (6B)

SrcAddr (6B)

ARP (For IPv4)

HardSize (1B)ProtoSize (1B)

Op (2B)

Sender Eth Addr (6B)

Sender IP Addr (4B)

Target Eth Addr (6B)

Target IP Addr (4B)

58 - JST/JDD/FK - 04/19/23

ARP for MetaNet Multi-Access Packet Format

Ethernet Header»Dst Addr»Src Addr»Type = 0x0806 (ARP)

Hard Type: Type of hardware address»1 Ethernet

Proto Type: Type of protocol address being mapped

Hard/Proto Size: size in bytes of address fields OP: Operation

»1: ARP Request»2: ARP Reply»3: RARP Request»4: RARP Reply

Sender Eth Address Sender MetaNet Address (MNid:MnAddr) Target Eth Address Target MetaNet Address (MNid:MnAddr)

Type=ARP(2B)HardType=Ether (2B)

ProtoType=Substrate (2B)

DstAddr (6B)

SrcAddr (6B)

ARP (For MN)

Hard Size (1B)Proto Size (1B)

Op (2B)

Sender Eth Addr (6B)

Sender MN Addr (nB)

Target Eth Addr (6B)

Target MN Addr (nB)

59 - JST/JDD/FK - 04/19/23

Other Access Technologies ATM? Sonet? Wireless

»How would we characterize a wireless access network?»How would we provide substrate links on a wireless network?

Multi-Access network without any mechanism for isolation and differentiation of P2P and Multi-Access Links:»Two possibilities:

We do not support P2P on such a network We use one bit in the MLI (or one bit somewhere in some header) to

indicate P2P vs. Multi-Access.

60 - JST/JDD/FK - 04/19/23

Summary: Ethernet Pkt Formats – with VLANs

P2P-Tunnel SL PktWith VLANs

Type=802.1Q (2B)

MLI (2B)LEN (2B)

Meta Frame

TCI ≠ VLAN0 (2B)Type=IP (2B)

PAD (nB)

CRC (4B)

DstAddr (6B)

SrcAddr (6B)

Dst Addr (4B)

Src Addr (4B)



Hdr Cksum (2B)

P2P-VLAN0 SL Pkt

Type=802.1Q (2B)

MLI (2B)LEN (2B)

Meta Frame

TCI = VLAN0 (2B)Type=Substrate (2B)

PAD (nB)

CRC (4B)

DstAddr (6B)

SrcAddr (6B)

Multi-Access SLPkt with VLANs

Type=802.1Q (2B)

MLI (2B)LEN (2B)

Meta Frame

TCI ≠ VLAN0 (2B)Type=Substrate (2B)

PAD (nB)

CRC (4B)

DstAddr (6B)

SrcAddr (6B)

P2P-DC SL PktWith VLANs

Type=802.1Q (2B)

MLI (2B)LEN (2B)

Meta Frame

TCI (2B)Type=Substrate (2B)

PAD (nB)

CRC (4B)

DstAddr (6B)

SrcAddr (6B)

TunnelHdr (IPv4)

Meta

EthernetHdr

EthernetTrailer

Substr. Hdr

61 - JST/JDD/FK - 04/19/23

Summary: Ethernet Pkt Formats – without VLANs

P2P-Tunnel SL PktWithout VLANs

MLI (2B)LEN (2B)

Meta Frame

Type=IP (2B)

PAD (nB)

CRC (4B)

DstAddr (6B)

SrcAddr (6B)

Dst Addr (4B)

Src Addr (4B)



Hdr Cksum (2B)

P2P-VLAN0 SL Pkt

Multi-Access SLPkt without VLANs

MLI (2B)LEN (2B)

Meta Frame

Type=Substrate (2B)

PAD (nB)

CRC (4B)

DstAddr (6B)

SrcAddr (6B)

P2P-DC SL PktWithout VLANs

MLI (2B)LEN (2B)

Meta Frame

Type=Substrate (2B)

PAD (nB)

CRC (4B)

DstAddr (6B)

SrcAddr (6B)

TunnelHdr (IPv4)

Meta

EthernetHdr

EthernetTrailer

Substr. Hdr

62 - JST/JDD/FK - 04/19/23



Substrate Link Types Packet Formats LC Rx/Tx Design Implementation (Next set of slides)


63 - JST/JDD/FK - 04/19/23

Extra The next set of slides are for templates or extra

information if needed

64 - JST/JDD/FK - 04/19/23

Text Slide Template

65 - JST/JDD/FK - 04/19/23

Image Slide Template

66 - JST/JDD/FK - 04/19/23

Multicast Alternatives Provide support in CRF for Multicast

»Allow MRs to provide code to interpret Lookup Result. Define a default Unicast format for a Lookup Result:

– A pure Unicast MR could use this exclusively and would not need to provide any code to interpret Lookup Result.

Type bit: indicates Default format CRF Specific Data

– QID– Stats Index–

MR Related Data Used by CRF– MI–

MR Specific Data Opaque to CRF– Whatever the MR developer wants…

Define an MR-specific format: Type bit: indicates MR-Specific

– MR supplies a block of code to interpret result (next slide)

67 - JST/JDD/FK - 04/19/23

Core Components Functionality:

»Initialization»Table management

Route/Filter tables L2 Tables MR State Table (in Demux)

»MetaRouter Installation/Removal

68 - JST/JDD/FK - 04/19/23

Core Components (sample App)

69 - JST/JDD/FK - 04/19/23

Core Components

DeMux Parse LookupHeaderFormat

Rx TxQM

RX CoreDeMuxCore

ParseCore

LookupCore

Hdr FmtCore

QM Core

TX Core

MetaRouter Management Core

MR Lookup Tbl MgmtMR State Tbl Mgmt

MR Code Management

XScale

Micro-Engines

Init, etc.Tbl UpdatesCode Updates

MR L2 Tbl Mgmt

Portability Framework: Resource Manager

Portability Framework: Core Components Infrastructure

MR Specific CoreComponents ???

GPPE?

70 - JST/JDD/FK - 04/19/23

System Configuration

SwitchBlades (2)

IXPPE

Blade

IXPPE

Blade. . . GP

BladeGP

Blade. . .

NumLC NumPENumGP

NumLC + NumPE + NumGP + 2 (switch) <= 14» 14 = # of slots in a full ATCA Chassis

Good ratio of PEs to LCs is probably 3:1 Blade Server may take the place of the GP Blades

» Connected through Uplinks on Switch Blade(s)

. . .IXPLC

Blade

GE

GE

GE

GE

GE

GE

GE

GE

GE

GE

IXPLC

Blade

GE

GE

GE

GE

GE

GE

GE

GE

GE

GE

e.g.10 GE

InterfacesPer LC

BladeServer

71 - JST/JDD/FK - 04/19/23

MetaLinks Types of MetaLinks:

May be directional and asymmetric.

»Normal: Frames enter with a Substrate Header(MLI:LEN) and Meta Frame Frames are sent to a MetaRouter on a Processing Engine Frames leave with a Substrate Header(MLI:LEN) and Meta Frame

»Gateway: Frames either:

Enter an interface as Legacy and get delivered to an MR as Meta Frames – OR

Leave an MR as Meta Frames and get sent out an interface as Legacy

»Pass-Through: Frames enter with a Substrate Header and Meta Frame and leave

with a Substrate Header and Meta Frame without going to a MetaRouter.

i.e. they go LC to LC without visiting a Processing Engine. No need for Legacy Pass-Through

72 - JST/JDD/FK - 04/19/23

MPLS Tunnels We’ve looked at MPLS just enough to convince ourselves that we can do something,

but not enough to understand all the implications.» Some of the MPLS RFCs:

3031: Multiprotocol Label Switching Architecture 3032: MPLS Label Stack Encoding 3037: LDP Applicability

LDP: Label Distribution Protocol 4221: Multiprotocol Label Switching (MPLS) Management Overview

MPLS uses a 32-bit field which contains» Label (20b)» Experimental (3b)» S-Bit (1b): 1 indicates bottom of label stack» TTL (8b)

There are some special labels defined, for instance:» IPv4 NULL Label:

when at bottom of stack indicates that the encapsulated packet is IPv4 and should be routed as such. How would we use it?

» We could define a similar special label for Substrate.» Our Substrate Routers would need to participate in the MPLS label distribution protocols and

such.» The sending Substrate Router would push the special Substrate label onto the bottom of the

stack and then push the first routing label onto the stack.» Packets would arrive at the receiving Substrate router interface with this bottom of stack

special label that would tell us that the included packet was a substrate packet.» This special label could also include the MLI and hence be a part of the Substrate header.

73 - JST/JDD/FK - 04/19/23

OLD The rest of these are old slides that should be deleted

at some point.

design of a diversified router: model and system overview

Documents