John DeHartSPP Version 1RouterPlans and Design

SPP VersionsSPP Version 0:What we used for SIGCOMM PaperSPP Version 1:Bare minimum we would need to release something to PlanetLab UsersSPP Version 2:What we would REALLY like to release to PlanetLab users.

Upcoming Meetings and DatestechX Meetings:2/05/08: Status updateSPP V2SPP V1SPP V1 ControlNATSPP V1 next demoONL NPR2/12/08: ONL NP Router Distribution issues2/19/08: NAT Review2/26/08: SPP V1 Demo3/04/08: SPP V2 ReviewOther dates:2/22/08: Department Visitors day for doctoral candidatesClean up labs and machine roomsEtc.

Work Items (Updated 4/9/08)SPP V2DesignSPP V1DONE: Get Init Scripts working again without ControlRemove what is not needed in init scriptsDONE: UDP Tunnel Traffic Generation

3/4/08: Fred QuestionsAdd info on Substrate only NPE LookupsNPE Lookup Key: What is the 1 bit type field?This differentiates between:0: Substrate only lookup: when GPE returns a packet with no reclassify1: Normal lookupSlice ID: 15 bits, but we only use 12 bits, right?Yes, it should be 12 bits and there should be 3 reserved bits.Actually we are limiting it to 11 bits now.Slides have been updatedMove Exception bits in Lookup HF dataImplement LD bit in Lookup ResultWhat is the order of the lookup key ports ?Slides have been updatedLookup key fields, which come from Tunnel header and which come from encapsulated header.First word is all Substrate InfoRest is MN infoWhy dont we have tunnel rx IP Daddr and port number in key to fully map meta-interface.Yes, good point. This has been raised before and we keep side stepping it. We will use an Index in the key that selects 1 of 16 possible Rx IP DAddrs.Slides updated but not implemented yet.NPE Lookup Result: What is H Flag? HITWhat is D Flag? DROP12 exception bits? Yes, there are 12 exception bits but they are not part of the lookup result. They get carried over from the input ring from Parse to the output ring to HF.MAC Address still there, but we dont use it anymore right?Done (removed from slides)DONE: QParams: Have we added qlength in packets yet?GPE VLAN issues seem to require that a different set of IP Addresses be used for each different VLAN. This might require us to put the NPE Src IP Address for packets going to the GPE in the GPE Info table and ignore what is in the Per Scheduler IP Src address table .Can the data path do this?Looks like it should be easy since substrate encap uses the gpe info struct to figure out which scheduler is to be used so it can read the right src_ip struct. So the data is all there at the right place already.

QM: QLength in PacketsQParams: Have we added qlength in packets yet?No, I havent had time for that yet.QLength (Bytes) (32b)QParams in SRAMThreshold (Bytes) (32b)Quantum (32b)Unused (32b)QLength (Bytes) (32b)QParams in Local MemoryThreshold (Bytes) (32b)Quantum (32b)Unused (29b)TV1bHV1bLV1bFlags: Length Valid Head Valid Tail Valid Current:Proposed:QLength (Bytes) (32b)QParams in SRAMThreshold (Bytes) (32b)Quantum (32b)QLength (Bytes) (32b)QParams in Local MemoryThreshold (Bytes) (32b)Quantum (32b)QLength (Pkts) (28b)Flags(4b): Reserved Length Valid Head Valid Tail Valid

Rx IP DAddr in KeyWhy dont we have tunnel rx IP Daddr and port number in key to fully map meta-interface.Yes, good point. This has been raised before and we keep side stepping it. To fix it for real we need to make the key larger and take the performance hit.Actually, what we decided to do was to implement a small table (16 entries) of IP Addresses. The address that is matched by the incoming Rx IP DAddr will have its index used in the key as a 4-bit field.This is now reflected in the slides but has not yet been implemented in the code.

LD Bit in Lookup ResultImplement LD bit in Lookup Result

November HW TestFirst test in HW should happen in Nov. 2007Plan:Retry SPP V0 demo on Dev. Chassis with new boardsFinish all three projects in Simulation:LCI: Currently missing ICMP and NATLCE: Currently missing NATKE and Lookup are nearing completionHF MAC Address lookup fine tuningFlow stats: FS2 nearing completion, needs archive thread and testing testing testingInitialization scripts need workNPE:One more memory update needed for Substrate Encap (DRAM write)Working on initialization scripts.TestingTest all three in simulation including initialization scriptsConvert *.ind initialization scripts to cmd utility HW initialization scriptsReview *.ind and cmd scripts with Fred and control groupTCAM utility for SPP V1?Plan A: Use Jonathons test utilityPlan B: Packet generation for HW test? Use Traffic Generators?Test all in HW

November HW TestStatus (11/29/07):DONE: Packets going through all three projects:DONE: TCAM Utilities for all three projects seem to be workingNext StepsDONE: NPE config/init (JDD and MLW)Orchid (MLW)DONE: V1 Testing (JDD)This results in November HW Test milestoneHF MAC Address usage and Initialization cleanup (JDD)LCI: DONELCE: DONENPE: NextMove NPE to NPUA (JDD): DONEWe are still using the config we had to use because NPUA on the pre-production board was flakey.Orchid Integration (MLW and JDD) On hold.Performance testing (JDD and MLW)Expand lookup filters to exercise all SchedulersNeed access to traffic generators or hosts with Charlies UDP Tunnel driverFlow Stats testing (JDD, JM)NAT (DMZ)ICMP pkt handling (JDD) LCI KeyExtract and LCE KeyExtract need to extract ICMP ID when Protocol==ICMPControl Integration (JDD, FK, etc) In ProgressRLI MonitoringV2: To begin January 2008 (JDD, MLW)

Control Demo NotesFast Path changes needed:QID changes : DONEAll three projects. Change to:QM_ID (2b)Sched_ID (3b)QID (15b)32K Queues available with now implicit association with a QM or scheduler.Blocks Affected:QM: Format of Input, position of SchedID and QID LCI:Lookup: Format of output, change the format of TCAM result to report index in first word and results starting in second word so we can get a full 64 bit resultHF: Format of Input, Format of Output, position of VLAN in input dataPortSplitter: Format of Input, Format of Output, position of QM_ID in input dataNPE:Lookup: Format of outputHF : Format of Input, Format of OutputSubstrateEncap : Format of Input, Format of Output, position of QM_ID in input dataLCE:Lookup: Format of output , change the format of TCAM result to report index in first word and results starting in second word so we can get a full 64 bit resultHF: Format of Input, Format of Output, position of VLAN in input dataPortSplitter: Format of Input, Format of Output, position of QM_ID in input dataInitialization?How to test?

Control Demo NotesFast Path changes needed (continued):LC HF MAC tables need to be dynamically read: DONELCI Needs:Initialization of 1 SMAC Address (Fabric)Table of per scheduler DMAC addresses (1 per scheduler)LCE Needs:Table of per scheduler DMAC/SMAC addresses (1 pair per scheduler)To simplify things, lets have LCI and LCE use the same format table:Dst MAC (8 bytes)Src MAC (8 bytes)NPE Substrate Encap GPE info table entries need to be moved out of Slice data space and into a separate table of their own indexed by VLAN.DONEMAC Address coming from NPE to Egress is wrongFIXEDNPE Substrate Encap has hard coded DMac address.Needs to read the MAC address from memory when it reads the IP Src address.DONERemove the 8 bits of DMAC from Lookup resultDONE (no longer used)Max Buffer Limit (todo notes)Change JDD TCAM utilities to use DB IDs that Control uses.DONE

NotesFor NPE look at putting a limit on the number of outstanding buffers a Slice has at a time.Add a counter to the Substrate Decap VLAN/Slice table. When SD gets a packet, increment the counter for that SliceWhen a buffer is freed have the generic buf_free code decrement the counter for that slice.This will probably require recording the Slice ID in the buffer descriptor and having the buf_free code read the descriptor.Look at using all 10 external interfaces on LCEach interface that is used will be connected to different ports on the same router.Thus the SPP node does not have to worry about participating in routing protocols in V1.Use both fabric interfaces on GPEsV1 will use just one of them The one that is used will be associated with 1 external interface.The interfaces from different GPEs may or may not be associated with different external interfaces.There may be cases where GPEs share an IP AddressThere may be cases where GPEs have different IP AddressesWe need to support both casesCheck on how we handle fragmentationAdd Ring specs to block diagramSchedule for upcoming meetings:8/14: Charlies SIGCOMM talk and NAT8/21: Plugin Framework (Shakir)8/28: Flow Stats (JMM)

SPP V1 PlansSPP Version 1:1 5-Port NPE (still dont use NPUB)Support Multiple External IP AddressesSwitch Blade integration10GE Tx module integrationARP: Probably not needed in V1NAT:Flow Stats: Egress Traffic monitoringMR Code OptionsAnything new?Control Local ControlBooting NPUAdd/Remove SlicesMR ControlAdd/Remove RoutesNode ManagerGPEMultiple GPEsNATSSH ForwardingPLC integrationMain focus today will be on the LC:Block/ME designLookupsFlow statsARP

Cycle Budget (min eth packets)To hit 5 Gb rate:76B per min IPv4 packet (64 min Eth + 12B IFS)1.4Ghz clock rate5 Gb/sec * 1B/8b * packet/76B = 8.22 Mp/sec1.4Gcycle/sec * 1 sec/ 8.22 Mp = 170.3 cycles per packetcompute budget: 170 cycleslatency budget: (threads*170)8 threads: 1360 cyclesTo hit 10 Gb rate:76B per min IPv4 packet (64 min Eth + 12B IFS)1.4Ghz clock rate10 Gb/sec * 1B/8b * packet/76B = 16.44 Mp/sec1.4Gcycle/sec * 1 sec/ 16.44 Mp = 85.16 cycles per packetcompute budget: 85 cycleslatency budget: (threads*85)8 threads: 680 cycles

Cycle Budget (IPv4 MN packets)To hit 5 Gb rate:90B per min IPv4 packet (78 min IPv4MN + 12B IFS)1.4Ghz clock rate5 Gb/sec * 1B/8b * packet/90B = 6.94 Mp/sec1.4Gcycle/sec * 1 sec/ 6.94 Mp = 201.7 cycles per packetcompute budget: 201 cycleslatency budget: (threads*201)8 threads: 1608 cyclesTo support 6.94 M pkts/sec we can Read 28 Words and Write 28 Words per pkt per SRAM Bank(200M/6.94M) = 28.818To hit 10 Gb rate:90B per min IPv4 packet (78 min IPv4MN + 12B IFS)1.4Ghz clock rate10 Gb/sec * 1B/8b * packet/90B = 13.88 Mp/sec1.4Gcycle/sec * 1 sec/ 13.88 Mp = 100.86 cycles per packetcompute budget: 100 cycleslatency budget: (threads*100)8 threads: 800 cyclesTo support 13.88 M pkts/sec we can Read 14 Words and Write 14 Words per pkt per SRAM Bank(200M/13.88M) = 14.409

Cycle Budget (Average Pkts)To hit 5 Gb rate:218B per min IPv4 packet (200 avg IPv4MN + 12B IFS)1.4Ghz clock rate5 Gb/sec * 1B/8b * packet/218B = 2.87 Mp/sec1.4Gcycle/sec * 1 sec/ 2.87 Mp = 487.8 cycles per packetcompute budget: 487 cycleslatency budget: (threads*487)8 threads: 3896 cyclesTo support 2.87 M pkts/sec we can Read 69 Words and Write 69 Words per pkt per SRAM Bank(200M/2.87M) = 69.686To hit 10 Gb rate:218B per min IPv4 packet (200 avg IPv4MN + 12B IFS)1.4Ghz clock rate10 Gb/sec * 1B/8b * packet/218B = 5.74 Mp/sec1.4Gcycle/sec * 1 sec/ 5.74 Mp = 243.9 cycles per packetcompute budget: 243 cycleslatency budget: (threads*243)8 threads: 1944 cyclesTo support 5.74 M pkts/sec we can Read 34 Words and Write 34 Words per pkt per SRAM Bank(200M/5.74M) = 34.843

SPP V1 ARP NotesStatically configure the Ethernet Addr of next hop(s).Dont need ARP in V1.LC uses scheme similar to ONLLCE Lookup result contains Next Hop IP or NH Ethernet Addr.If NH Ethernet Addr is present than update packet and sendIf NH IP Addr present instead of NH Ethernet Addr then send to XScaleNeed to define shim/descriptor for LCE to XScalePhysical InterfaceNH IP AddressXScale will send ARP Broadcast on physical interfaceLCI receives Unicast ARP Response from RTM PortSends to XScale indicating which physical interface recvd on.XScale updates filter table If XScale has waiting packet, send to data path.LCI receives ARP Broadcast from RTM portSends to XScale indicating which physical interface recvd on.XScale processes and sends ARP Response if needed.ARP Entry Aging.

SPP V1 ARP InterfacesLCI to XScale InterfaceLCI just needs to detect EtherType Field of ARPShould be able to do this in Key Extract.Code already there to detect ARP and send to XScale.We may have to adjust for shim/descriptor to communicate additional info to XScaleLCE to XScale InterfaceNeeds to be Post Lookup and Pre QM.Needs to update the Shim/Descriptor to send info to the XScale.Hdr Format is probably the best place for this.XScale to LCE InterfaceShould be Queued to keep Port rate control sane.Does it need to be a separate scratch ring or can it go directly into the QM input ring(s)?

SPP V1 NAT NotesNAT Notes moved to separate file:SPP_V1_NAT_design.ppthttp://www.arl.wustl.edu/projects/techX/design/SPP/SPP_V1_NAT_design.ppt

SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)SWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2PortSplitterQM0QM1QM2QM3

SPP V1 LC Egress with 1x10Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMTCAMFlowStats1SRAM1SRAM2FlowStats2XScaleArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM31x10GTx11x10GTx2

SPP V1 LC Egress with 10x1Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMTCAM5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3

INGRESS Block Interfaces

SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)SWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2PortSplitterQM0QM1QM2QM3

Notes on Frame vs. Pkt LengthsRX reports Ethernet Frame LengthKE passes along IP Pkt length and Ethernet Hdr LengthHF uses Ethernet Hdr Length and Buffer Offset to find start of IP Pkt so it can put on new ethernet header.HF passes along Ethernet Frame LengthTX needs Ethernet Frame Length which it gets from buffer descriptor Buffer SizeQM Dequeue gets length from buffer descriptor Thus it will get Ethernet Frame Length just like TXQM Enqueue gets a length from input ring which must agree with what QM Dequeue gets from buffer descriptor.Thus: HF must pass Ethernet Frame length in output ring AND it must write it to buffer descriptor.QM Link rates should include IFS, etc.

SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)SWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2PortSplitterQM0QM1QM2QM3

SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)SWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2LookupKeyIP DAddr (32b)Buf Handle(24b)IP PktLength (16b)Intf(4b)UDP DPort (16b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags(8b)Protocol(8b)Type(8b)IP Hdr 2nd Word (32b)PortSplitterQM0QM1QM2QM3Rsv(4b)

SPP V1 LC IngressSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2XScaleNATScratchRingsPortSplitterQM0QM1QM2QM3Buf Handle(24b)IP PktLength (16b)TranslatedDPort/ID (16b)Stats Index (16b)Reserved(8b)VLAN (12b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

PerSchedQID(15b)Sch3bQM2bIP DAddr (32b)Buf Handle(24b)IP PktLength (16b)Intf(4b)TCP/UDP DPort Or ICMP ID (16b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags(8b)Protocol(8b)ICMPType (8b)Rsv(4b)Flags(8b)Rsvd2bN1bH1bI1bU1bT1bICMPNATHitUDPTCPIP Hdr Top 16 bitsOf 2nd Word (16b)OriginalDPort/ID (16b)IP Hdr Top 16 bitsOf 2nd Word (16b)TCP/UDPSPort (16b)Rsv1bTCP Flags6bIP SAddr (32b)IE1bICMP ERRIE1bICMP ERR

SPP V1 LC IngressSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2XScaleNATScratchRingsPortSplitterQM0QM1QM2QM3Reserved(8b)Buf Handle(24b)IP PktLength (16b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

IP Hdr Top 16 bitsOf 2nd Word (16b)Reserved(8b)OriginalDPort/ID (16b)Rsvd2bN1bH1bI1bU1bT1bIE1bICMP ERRIntf(4b)Buf Handle(24b)IP PktLength (16b)Eth HdrLen (8b)Flags (8b)

IP DAddr (32b)TCP/UDP DPort Or ICMP ID (16b)Protocol(8b)ICMPType (8b)Rsv(4b)IP_SAddr (32b)TCP/UDP SPort (16b)TCAM Hit Index (32b)IP Hdr 1st Word (32b)IP Hdr Top 16 bitsOf 2nd Word (16b)

SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)SWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2Buf Handle(24b)IP PktLength (16b)TranslatedDPort/ID (16b)Stats Index (16b)Reserved(8b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

IP Hdr 2nd Word (32b)PortSplitterQM0QM1QM2QM3Frame Length (16b)Buffer Handle(24b)Stats Index (16b)Reserved(8b)

PortSplitterQM0QM1QM2QM3SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)SWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2Frame Length (16b)Buffer Handle(24b)Stats Index (16b)Reserved(8b)Frame Length (16b)Buffer Handle(24b)Stats Index (16b)Reserved(8b)

PortSplitterQM0QM1QM2QM3SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)SWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2Frame Length (16b)Buffer Handle(24b)Stats Index (16b)Reserved(8b)

PortSplitterQM0QM1QM2QM3SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)SWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2

SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)SWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormat1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2Scr2NNPortSplitterQM0QM1QM2QM3

EGRESS Block Interfaces

Egress Buffer DescriptorBuffer_Next (32b)LW0Buffer_Size (16b)LW1LW2LW3LW4Reserved (32b)LW5LW6Packet_Next (32b)LW7Offset (16b)Packet_Size (16b)Reserved (8b)Free_list0000 (4b)Reserved (4b)Stats Index (16b)Reserved (16b)Reserved(8b)Reserved (4b)Reserved (4b)Reserved (16b)Reserved (16b)Reserved (4b)Reserved (4b)SliceID (VLAN)(12b)Reserved (4b)

SPP V1 LC Egress with 1x10Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatFlowStats11x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3SRAM1SRAM2FlowStats2TCAMXScaleArchive RecordsPortSplitterQM0QM1QM2QM3IN:F2DROP:12DROP:13DROP:14Rcv: 25ToHF:33ToXScale:43From:LK:34From:XS:44

SPP V1 LC Egress with 10x1Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMTCAM5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3

SPP V1 LC Egress with 10x1Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMTCAM5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3

5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3SPP V1 LC Egress with 10x1Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMTCAMLookupKey

5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleXScaleNAT PktreturnArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3SPP V1 LC Egress with 10x1Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMXScaleNAT MissScratch RingTCAMLookupResultHBuf Handle(24b)IP DAddr (32b)IP PktLength (16b)Reserved(8b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

IP Hdr Top 16 bitsOf 2nd Word (16b)Buf Handle(24b)IP_SAddr (32b)IP PktLength (16b)SrcMAC(8b)Eth HdrLen (8b)UDP SPort (16b)IP Proto (8b)Type(8b)IP Hdr 1st Word (32b)Reserved(8b)IP DAddr (32b)SliceID (12b)Rsv(4b)IP Hdr Top 16 bitsOf 2nd Word (16b)SliceID (12b)Rsv(4b)ICMPNATHitFlags(8b)

5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleXScaleNAT PktreturnArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3Proposed Change: SPP V1 LC EgressSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMXScaleNATScratchRingsTCAMHBuf Handle(24b)IP DAddr (32b)IP PktLength (16b)Reserved(8b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

IP Hdr Top 16 bitsOf 2nd Word (16b)Buf Handle(24b)IP_SAddr (32b)IP PktLength (16b)SrcMAC(8b)Eth HdrLen (8b)TCP/UDP SPort Or ICMP ID (16b)IP Proto (8b)ICMPType (8b)IP Hdr 1st Word (32b)Reserved(8b)IP DAddr (32b)IP Hdr Top 16 bitsOf 2nd Word (16b)Flags(8b)Rsvd2bN1bH1bI1bU1bT1bReserved (16b)OriginalSPort/ID (16b)IE1bICMP ERRTCP/UDP DPort (16b)Rsv1bTCP Flags6bIE1bICMP ERR

5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleXScaleNAT PktreturnArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3Proposed Change: SPP V1 LC EgressSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMXScaleTCAMBuf Handle(24b)IP DAddr (32b)IP PktLength (16b)Reserved(8b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

IP Hdr Top 16 bitsOf 2nd Word (16b)NATScratchRingsRsvd2bN1bH1bI1bU1bT1bOriginalSPort/ID (16b)IE1bICMP ERRBuf Handle(24b)IP PktLength (16b)Eth HdrLen (8b)Flags (8b)

IP_SAddr (32b)SrcMAC(8b)TCP/UDP SPort Or ICMP ID (16b)IP Proto (8b)ICMPType(8b)IP_DAddr (32b)TCP/UDP DPort (16b)TCAM Hit Index (32b)IP Hdr 1st Word (32b)IP Hdr Top 16 bitsOf 2nd Word (16b)

5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3NAT Changes: SPP V1 LC EgressSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMXScaleNATScratchRingsTCAMHBuf Handle(24b)IP DAddr (32b)IP PktLength (16b)Reserved(8b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

IP Hdr Top 16 bitsOf 2nd Word (16b)Buf Handle(24b)IP_SAddr (32b)IP PktLength (16b)SrcMAC(8b)Eth HdrLen (8b)TCP/UDP SPort Or ICMP ID (16b)IP Proto (8b)ICMPType (8b)IP Hdr 1st Word (32b)Reserved(8b)IP DAddr (32b)IP Hdr Top 16 bitsOf 2nd Word (16b)Flags(8b)Reserved (16b)Reserved (16b)

5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3NAT Change: SPP V1 LC EgressSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMXScaleTCAMBuf Handle(24b)IP DAddr (32b)IP PktLength (16b)Reserved(8b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

IP Hdr Top 16 bitsOf 2nd Word (16b)NATScratchRingsReserved (16b)

5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleXScaleNAT PktreturnArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3SPP V1 LC Egress with 10x1Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMXScaleNAT MissScratch RingTCAMNAT MISS!Buf Handle(24b)IP DAddr (32b)IP PktLength (16b)Reserved(8b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

IP Hdr Top 16 bitsOf 2nd Word (16b)SliceID (12b)Rsv(4b)Flags (8b)

5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3SPP V1 LC Egress with 10x1Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMTCAMEthernetFrame Length (16b)Buffer Handle(24b)Cntr Index (16b)Reserved(8b)Buf Handle(24b)IP DAddr (32b)IP PktLength (16b)Reserved(8b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

IP Hdr Top 16 bitsOf 2nd Word (16b)SliceID (12b)Rsv(4b)

5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3SPP V1 LC Egress with 10x1Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMTCAMEthernetFrame Length (16b)Buffer Handle(24b)Cntr Index (16b)Reserved(8b)EthernetFrame Length (16b)Buffer Handle(24b)Cntr Index (16b)Reserved(8b)

5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3SPP V1 LC Egress with 10x1Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMTCAMEthernetFrame Length (16b)Buffer Handle(24b)Cntr Index (16b)Reserved(8b)

5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3SPP V1 LC Egress with 10x1Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMTCAMFreelist

FlowStats11x10GTx11x10GTx2Stats(1 ME)SRAM3SRAM1SRAM2FlowStats2XScaleArchive RecordsPortSplitterQM0QM1QM2QM3SPP V1 LC Egress with 1x10Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMTCAMFreelist

5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3SPP V1 LC Egress with 10x1Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMTCAMFreelist

NPENext well look at the design for the NPE for SPP V1

NPE Buffer DescriptorBuffer_Next (32b)LW0Buffer_Size (16b)LW1LW2LW3LW4Reserved (32b)LW5LW6Packet_Next (32b)LW7Offset (16b)Packet_Size (16b)Reserved (8b)Free_list0000 (4b)Reserved (4b)Reserved (16b)Stats Index (16b)Reserved (16b)Reserved (16b)Reserved (16b)Reserved (4b)Reserved (4b)VLAN (12b)Reserved (4b)

SPP V1 NPE (MetaRouters)SWITCHMSFRx1RBUFRx2ParseLookupHdrFormat1x10GTx11x10GTx2MSFTBUFStats(1 ME)SRAM3TCAMSRAM1SRAM2SWITCHSubstrateDecapScr2NNPortSplitterQM0QM1QM2QM3

SPP V1 NPE (MetaRouters)SWITCHMSFRx1RBUFRx2ParseLookupHdrFormat1x10GTx11x10GTx2MSFTBUFStats(1 ME)SRAM3TCAMSRAM1SRAM2SWITCHSubstrateDecapScr2NNPortSplitterQM0QM1QM2QM3

SPP V1 NPE (MetaRouters)SWITCHMSFRx1RBUFRx2ParseLookupHdrFormat1x10GTx11x10GTx2MSFTBUFStats(1 ME)SRAM3TCAMSRAM1SRAM2SWITCHSubstrateDecapScr2NNPortSplitterQM0QM1QM2QM3Rx UDP DPort (16b)Buf Handle(32b)Slice ID(VLAN) (11b)MN Frm Offset (16b)MN Frm Length(16b)Rx IP SAddr (32b)Reserved (12b)Rx UDP SPort (16b)Code(4b)Slice Data Ptr (32b)Rx IP DAddr (32b)RSV1RxId(4b)

Scr2NNPortSplitterQM0QM1QM2QM3SPP V1 NPE (MetaRouters)SWITCHMSFRx1RBUFRx2ParseLookupHdrFormat1x10GTx11x10GTx2MSFTBUFStats(1 ME)SRAM3TCAMSRAM1SRAM2SWITCHSubstrateDecapLookup Key[111-80] DA (32b)Buf Handle(32b)IP Pkt Length (16b)IP Pkt Offset (16b)Lookup Key[ 79-48] SA (32b)Lookup Key[ 47-16] Ports (32b)Lookup KeyProto/TCP_Flags[15- 0] (16b)ExceptionBits (12b)Lookup Key[143-112] Type(1b)/RxID(4b)/Slice ID(11b)/Rx UDP DPort (16b)Rsv(4b)Slice Data Ptr (32b)Reserved (12b)Code(4b)Rx IP SAddr (32b)Rx UDP SPort (16b)Rx UDP DPort (16b)Buf Handle(32b)MN Frm Offset (16b)MN Frm Length(16b)Rx IP SAddr (32b)Reserved (12b)Rx UDP SPort (16b)Code(4b)Slice Data Ptr (32b)Rx IP DAddr (32b)Rx IP DAddr (32b)Slice ID(VLAN) (11b)RSv1RxId(4b)

Scr2NNPortSplitterQM0QM1QM2QM3SPP V1 NPE (MetaRouters)SWITCHMSFRx1RBUFRx2ParseLookupHdrFormat1x10GTx11x10GTx2MSFTBUFStats(1 ME)SRAM3TCAMSRAM1SRAM2SWITCHSubstrateDecapSlice Data Ptr (32b)Tx IP DAddr (32b)Buf Handle(32b)IP Pkt Length (16b)IP Pkt Offset (16b)Cntr Index (16b)RSVd(1b)D(1b)H(1b)Reserved(11b)LD(1b)Rx UDP DPort(16b)Tx UDP SPort(16b)Tx UDP DPort (16b)Rx IP SAddr (32b)Reserved (12b)Code(4b)Rx UDP SPort (16b)Exception Bits(12b)Lookup Key[111-80] DA (32b)Buf Handle(32b)IP Pkt Length (16b)IP Pkt Offset (16b)Lookup Key[ 79-48] SA (32b)Lookup Key[ 47-16] Ports (32b)Lookup KeyProto/TCP_Flags[15- 0] (16b)ExceptionBits (12b)Lookup Key[143-112] Type(1b)/RxID(4b)/Slice ID(11b)/Rx UDP DPort (16b)Rsv(4b)Slice Data Ptr (32b)Reserved (12b)Code(4b)Rx IP SAddr (32b)Rx UDP SPort (16b)Rx IP DAddr (32b)Rx IP DAddr (32b)Slice ID(VLAN) (11b)Rsvd(5b)RSVd(1b)

NPE Lookup Key and ResultTx IP DAddr (32b)Cntr Index (16b)DROp(1b)Rsvd(3b)Reserved(11b)LD(1b)Tx UDP SPort(16b)Tx UDP DPort (16b)Reserved(12b)Generic MN Key:IPv4 MN Key:Lookup Result as written in TCAM:Lookup Result as passed to HF(Note the movement of Exception bits):Tx IP DAddr (32b)Cntr Index (16b)RSVd(1b)DROp(1b)H(1b)LD(1b)Tx UDP SPort(16b)Tx UDP DPort (16b)Exception Bits(12b)Reserved(11b)RSVd(1b)TCP!TCPMove of exception bits notImplemented yet.DONE

NPE Substrate ONLY Lookup Key and ResultTx IP DAddr (32b)Cntr Index (16b)DROp(1b)Rsvd(3b)Reserved(11b)LD(1b)Tx UDP SPort(16b)Tx UDP DPort (16b)Reserved(12b)Rx UDP DPort (16b)(from Int. Hdr)T1(1b)Slice ID (VLAN)(11b)Rx IP DAIndex(4b)TX IP DAddr (from Int. Hdr) 111-80 (32b)0x00000000 79-48 (32b)0x0000 (16b)0x0000 (16b)Generic MN Key:Lookup Result as written in TCAM:Lookup Result as passed to HF(Note the movement of Exception bits):Tx IP DAddr (32b)Cntr Index (16b)RSVd(1b)DROp(1b)H(1b)LD(1b)Tx UDP SPort(16b)Tx UDP DPort (16b)Exception Bits(12b)Reserved(11b)RSVd(1b)TX UDP DPort (16b)(from Int. Hdr)MN Internal HeaderRx IP Da IndexIs calculated based onRX IP DaddrIn Int. HdrDo we want The Rx MISpecified in The substrateOnly key?Should Tx UDP Sport Be included also?STILL WORKING ON THISMove of exception bits notImplemented yet.DONEDONE

NPE Bypass Lookup Key and Result0x00000000 (32b)Cntr Index (16b)DROp(1b)Rsvd(3b)Reserved(11b)LD(1b)0x0000 (16b)0x0000 (16b)0x000 (12b)0x0000 (16b)Slice ID (VLAN)(11b)USER Output MI ID (32b)0x00000000 79-48 (32b)0x0000 (16b)0x0000 (16b)Key:Lookup Result as written in TCAM:Lookup Result as passed to HF(Note the movement of Exception bits):Tx IP DAddr (32b)Cntr Index (16b)RSVd(1b)DROp(1b)H(1b)LD(1b)Tx UDP SPort(16b)Tx UDP DPort (16b)Exception Bits(12b)Reserved(11b)RSVd(1b)MN Internal HeaderSTILL WORKING ON THIST1(1b)Rx IP DAIndex(4b)0x0000 (16b)Exception bits come fromInput ring.GPERx UDP Dport (anything needed?)USER Output MI IDHas to come from MN Internal Header

Scr2NNSubstrateEncapQM0QM1QM2QM3SPP V1 NPE (MetaRouters)SWITCHMSFRx1RBUFRx2ParseLookupHdrFormat1x10GTx11x10GTx2MSFTBUFStats(1 ME)SRAM3TCAMSRAM1SRAM2SWITCHSubstrateDecapTCAMTCAMSlice Data Ptr (32b)Tx IP DAddr (32b)Buf Handle(32b)Cntr Index (16b)RSVd(1b)D(1b)H(1b)Reserved(11b)LD(1b)Rx UDP DPort(16b)Slice ID(VLAN) (11b)Tx UDP SPort(16b)Tx UDP DPort (16b)Rx IP SAddr (32b)Reserved (12b)Code(4b)Rx UDP SPort (16b)Slice ID(VLAN) (11b)Cntr Index (16b)Tx IP DAddr (32b)Slice Data Ptr (32b)Reserved(12b)PerSchedQID(15b)Sch3bQM2bException Bits(12b)Rx IP DAddr (32b)Rsv(5b)Rsv(5b)RSVd(1b)

IPv4 Internal Header FormatFwdKey = [Tx UDP DPort + Tx UDP Sport + Tx IP DAddr]

PathCategoryTypefieldReasonOutgoing MN Internal HdrGPE->NPE[0]ReclassifyRx UDP DPort if set, otherwise Rx UDP Dport + FwdKey

NPE-> Egress LCFast pathNo MN Int HdrNPE->GPEException[2]No routeRx UDP DPort[3]Expired TTLRx UDP DPort[4]IP w/ optionsRx UDP DPort + FwdKey[5]Redirect due to Rx UDP DPort =Tx UDP SPortRx UDP DPort + FwdKeyControl[6]Local deliveryRx UDP DPort[7]InspectRx UDP DPortDebug[8]MonitorRx UDP DPort[9]Log due to error in pktsRx UDP DPort

IPv4 Internal Header FormatType (28b)Int. Hdr Length (2B)Rx UDP DPort (2B)Tx UDP SPort (2B)Tx UDP DPort (2B)Tx IP DAddr (4B)0000Rx UDP SPort (2B)Rx IP Saddr (4B)Reserved (2B)Rx IP Daddr (4B)MN Specific DataIPv4: Fwd KeyMN Specific DataIPv4: Fwd Key

Scr2NNSubstrate Encap

QM1QM2QM3QM4SPP V1 NPE (MetaRouters)SWITCHMSFRx1RBUFRx2ParseLookupHdrFormat1x10GTx11x10GTx2MSFTBUFStats(1 ME)SRAM3SRAM1SRAM2SWITCHSubstrateDecapTCAMTCAMEthernetFrame Length (16b)Buffer Handle(24b)Cntr Index (16b)Reerved(8b)Slice ID(VLAN) (11b)Cntr Index (16b)Tx IP DAddr (32b)Slice Data Ptr (32b)Reserved(12b)PerSchedQID(15b)Sch3bQM2bRsv(5b)

SubstrateEncapScr2NNQM0QM1QM2QM3SPP V1 NPE (MetaRouters)SWITCHMSFRx1RBUFRx2ParseLookupHdrFormat1x10GTx11x10GTx2MSFTBUFStats(1 ME)SRAM3TCAMSRAM1SRAM2SWITCHSubstrateDecapTCAMTCAMEthernetFrame Length (16b)Buffer Handle(24b)Cntr Index (16b)Reerved(8b)

Scr2NNPortSplitterQM0QM1QM2QM3SPP V1 NPE (MetaRouters)SWITCHMSFRx1RBUFRx2ParseLookupHdrFormat1x10GTx11x10GTx2MSFTBUFStats(1 ME)SRAM3TCAMSRAM1SRAM2SWITCHSubstrateDecap

Scr2NNPortSplitterQM0QM1QM2QM3SPP V1 NPE (MetaRouters)SWITCHMSFRx1RBUFRx2ParseLookupHdrFormat1x10GTx11x10GTx2MSFTBUFStats(1 ME)SRAM3TCAMSRAM1SRAM2SWITCHSubstrateDecap

TCAM Performance (Rates in M/sec)LC_Ingress/LC_EgressIPv4 MR

Lookup Size#LA-1 WordsCore SizeAssoc. DataSingle LA-1 Max RateMax Core RateAvg Shared Rate (Each of 2 LA-1s)32136325050256450251282512.536236325050256450251282512.56427232100100506450251282512.5723723267100506450251282512.512841443250100506450251282512.514451443240100406450251282512.51605288324050406450251282512.5

Changes for Key ExtractInput: No changesOutputMove Eth Hdr Len fieldAdd Type field to Lookup KeyIf ICMP extract TYPE field from ICMP pktOtherwise, set to 0.Add Src MAC to Lookup KeyExtract low 8 bits of Src MAC from ethernet headerAdd VLAN/IP_SAddr to Lookup KeyIf low 6 bits of Src MAC are all 1s then Src MAC is from NPEUse VLAN in the VLAN/IP_SAddr fieldVLAN goes in lower 12 bits, upper 20 bits are all 0sIf low 6 bits of Src MAC are NOT all 1s then it is from GPEUse IP_SAddr in the VLAN/IP_SAddr field

5x1GTx1(P0-P4)5x1GTx2(P5-P9)FlowStats1SRAM1SRAM2FlowStats2XScaleXScaleNAT PktreturnArchive RecordsPortSplitterQM0QM1QM2QM3Stats(1 ME)SRAM3SPP V1 LC Egress with 10x1Gb/s TxSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatMSFTBUFRTMXScaleNAT MissScratch RingTCAMLookupResultHBuf Handle(24b)IP DAddr (32b)IP PktLength (16b)Reserved(8b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

IP Hdr 2nd Word (32b)Buf Handle(24b)VLAN/IP_SAddr (32b)IP PktLength (16b)SrcMAC(8b)Eth HdrLen (8b)UDP SPort (16b)IP Proto (8b)Type(8b)IP Hdr 1st Word (32b)Reserved(8b)IP Hdr 2nd Word (32b)IP DAddr (32b)

Old SlidesThe following are the old Changes to .. slides from when we were going from V0 to V1They have gotten confusing now that we are going from V1 to V2

Changes for Key ExtractLookup Key Changes:Old Lookup Key (64b):SL Type (4b)Port (4b)IP DAddr (32b)IP Proto (8b)UDP DPort (16b)New Lookup Key (72b):Reserved (4b)Interface (4b)IP DAddr (32b)IP Proto (8b)UDP DPort (16b)ICMP Type (8b)Move Ethernet Hdr Length field

Changes for LookupLookup Key Changes:See KE notesLookup Result Changes:Add Translated DPort/IDMove MAC DAddrMove VlanRemove Port fieldQM ID and Scheduler IDQM_ID (2b)SchedID(3b)QID(15b)PerSchedQID(15b)QM_ID and SchedID are used to demux and get packet to correct QM and SchedulerThe QM/Scheduler uses just the PerSchedQID bits as the QID given to the SRAM Controller.Change in size of lookup resultMove Eth Hdr Len.Add Flags in 1st wordBit 0: HIT (Result is valid)Bit 1: NAT (NAT translation required, orig port != xlated Port)Bit 2: ICMP (Protocol == 1)

Changes for Header FormatLookup Result Changes:See Lookup notesProcess NAT MissIf (H==0) then NAT MissSend to XScaleIngress/Egress XScales will generate ICMP Error msg.Move Eth Hdr Len in input ringNo Interface field to pass alongPerform DPort/ID translationrecalculate IP Hdr checksum.Calculate incremental Transport (TCP and UDP) checksumCheck for arriving UDP checksum of 0 which implies that packet is not using the optional UDP checksum

Changes for Port SplitterNo Interface field in input ringQM(2b) determines which QM Scr ring to useQM will use the Sch(3b) to determine which scheduler to use.Port Splitter does not have to give a separate field anymore.The QM(2b), Sch(3b), QID(15b) should be left unchanged in the low 20 bitsThe QM will use the 15-bit QID field as the qid value given to the SRAM controller.

Changes for QMQM will extract the Sch(3b) to identify the scheduler (called the port_id in the code) instead of getting a separate port field.Association of a Scheduler with a physical interface:Dequeue currently reads the interface rate from SRAM periodically. We could extend this to have it also read the interface it is associated with at the same time it is reading the rate. This would also work for the LC_Ingress and NPE where we need to change the interface to 0 before sending it to TX. This could be accomplished by setting the interface read by Dequeue to 0 and then all the Dequeue engines (schedulers) would send to Interface 0.

Changes for Scr2NNNew Block based on Daves Port_ConcentratorQM now takes care of giving appropriate value for the Interface field

Changes for StatsDo we want to incorporate the improvements we made for the ONL Stats block?

Changes for Key ExtractInput: No changesOutputMove Eth Hdr Len fieldAdd Type field to Lookup KeyIf ICMP extract TYPE field from ICMP pktOtherwise, set to 0.Add Src MAC to Lookup KeyExtract low 8 bits of Src MAC from ethernet headerLow 8 bits of each blade MUST be unique across the chassisThese 8 bits will be the differentiating factor on lookups where a GPE is trying to use the same IP SAddr and Sport pair as an NPE is.In this case the GPE use will get NAT translated at the LC EgressAdd IP_SAddr to Lookup KeyAt one point in time we had this field being VLAN/IP_SAddr but this is not longer needed.The VLAN was going to be used when the traffic came from an NPE and the IP_SAddr when it came from a GPE.

Changes for LookupInput:Move Eth Hdr Len fieldAdd Type field to Lookup KeyAdd Src MAC to Lookup KeyAdd IP SAddr to Lookup KeyOutput:Re-organize outputMove IP DAddr to 5th word (Do we still need this?)Move Eth Hdr Len Add Flags in 1st wordBit 0: HIT (Result is valid)Bit 1: NAT (NAT translation required, orig port != xlated Port)Bit 2: ICMP (Protocol == 1)Add Translated Sport to Lookup Result

Changes for Header FormatInput:Re-organize input72 bit lookup resultMove IP DAddr to 5th wordMove Eth Hdr Len Add Flags in 1st wordAdd Translated Sport to Lookup ResultOutput to PortSplitter/QM:No changesOutput to XScale:NewFunction:Write Buffer descriptor including:Packet SizeBuffer SizeFreelistOffsetSliceID (VLAN)Stats IndexShould we also write the ethernet header length?Test HIT Flag to determine if NAT Hit or MissSend Miss to XScaleSend Hit to PortSplitter/QMPerform SPort/ID translationrecalculate IP Hdr checksum.Calculate incremental Transport (TCP and UDP) checksumCheck for arriving UDP checksum of 0 which implies that packet is not using the optional UDP checksum

Changes for Port SplitterNo Interface field in input ringQM(2b) determines which QM Scr ring to useSch(3b) needs to be copied up to the low 3 bits of the top byte to comply with QMs current input format.We will look into removing this requirement and see if it is easy to have the QM extract the scheduler bits itselfThe QM(2b), Sch(3b), QID(15b) should also be left unchanged in the low 20 bitsThe QM will give the 15-bit QID field to the SRAM Controller as the qid being used.

Changes for QMQM will extract the Sch(3b) to identify the scheduler (called the port_id in the code) instead of getting a separate port field.Association of a Scheduler with a physical interface:Dequeue currently reads the interface rate from SRAM periodically. We could extend this to have it also read the interface it is associated with at the same time it is reading the rate. This would also work for the LC_Ingress and NPE where we need to change the interface to 0 before sending it to TX. This could be accomplished by setting the interface read by Dequeue to 0 and then all the Dequeue engines (schedulers) would send to Interface 0.

Changes for FlowStatsNew BlockOutput for 10x1Gb/s Tx:To 1 of two scratch rings dependent on outgoing interfaceOutput for 1x10Gb/s Tx:To NN RingQM will now take care of setting the appropriate interface, FlowStats doesnt have to do anything special.

Changes for StatsDo we want to incorporate the improvements we made for the ONL Stats block?

Changes for LookupNo Port field in input dataBut Lookup doesnt look at the data anyway so no changes.

Changes for HFNo Port field in input dataUse QM/Sched bits to determine Src IP Address to use on the outgoing Tunnel Header.Src MAC Addr should be a constantDst MAC Addr should be configured:LCEGPESeparation of HF and Substrate Encap

Changes for Substrate EncapNew Block

Changes for QMUse Sched bits to determine which Scheduler to use.

NAT Changes: SPP V1 LC IngressSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2PortSplitterQM0QM1QM2QM3Buf Handle(24b)IP PktLength (16b)TranslatedDPort/ID (16b)Stats Index (16b)Reserved(8b)VLAN (12b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

PerSchedQID(15b)Sch3bQM2bIP DAddr (32b)Buf Handle(24b)IP PktLength (16b)Intf(4b)TCP/UDP DPort Or ICMP ID (16b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Reserved(8b)Protocol(8b)ICMPType (8b)Rsv(4b)Flags(8b)IP Hdr Top 16 bitsOf 2nd Word (16b)Reserved (16b)IP Hdr Top 16 bitsOf 2nd Word (16b)Reserved (16b)

NAT Changes: SPP V1 LC IngressSWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2XScaleNATScratchRingsPortSplitterQM0QM1QM2QM3Reserved(8b)Buf Handle(24b)IP PktLength (16b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

IP Hdr Top 16 bitsOf 2nd Word (16b)Reserved (16b)

SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)SWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2XScaleNAT MissScratch RingBuf Handle(24b)IP PktLength (16b)Reserved(8b)Eth HdrLen (8b)NAT MISS!Reserved(8b)PortSplitterQM0QM1QM2QM3Reserved(8b)Buf Handle(24b)IP PktLength (16b)TranslatedDPort/ID (16b)Stats Index (16b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

IP Hdr 2nd Word (32b)Flags (8b)

SPP V1 LC Ingress(1x10Gb/s and 10x1Gb/s)SWITCHMSFRx1RBUFRx2KeyExtractLookupHdrFormatScr2NN1x10GTx11x10GTx2MSFTBUFRTMStats(1 ME)SRAM3TCAMSRAM1SRAM2XScaleNAT MissScratch RingHLookupResultBuf Handle(24b)IP PktLength (16b)TranslatedDPort/ID (16b)Stats Index (16b)Reserved(8b)VLAN (12b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Flags (8b)

IP Hdr 2nd Word (32b)LookupKeyPortSplitterQM0QM1QM2QM3IP DAddr (32b)Buf Handle(24b)IP PktLength (16b)Intf(4b)UDP DPort (16b)Eth HdrLen (8b)IP Hdr 1st Word (32b)Reserved(8b)Protocol(8b)Type(8b)IP Hdr 2nd Word (32b)PerSchedQID(15b)Sch3bQM2bRsv(4b)ICMPNATHitFlags(8b)

***********************************************************************************************