Tiger4 OLT QOS introduction

By Jianguang Li

Agenda

• Tiger4 OLT ASIC Architecture• Tiger4 OLT data path • FE• Header t• TE• Header o• BM• ES• Flow control

Tiger4 OLT ASIC Architecture (1 die)

Tiger-4

ForwardEngine

CrossSwitch

Cascade10G Port

IngressAggregation

OutputScheduler

IBM EBM

Traffi cEngineer ing

D D R 3S D R A MC o n t r o l

10G/1G ONU Port

PacketReplication

1G OLTSERDES

MIIPort

ARM926CPU

EgressDispatch

ARM PeripheralManagement Access

MII

SFI (1G)

4xGE (UNI) Port

Uplink10G Port

XAUISERDES

XAUI/

4xGE

10G (EPON)SERDES

XFI (10G)

10G(EPON)SERDES

XFI (10G)

10GOLT Port

1G OLT Port

CS8124 chip 0 data path

FE: forward engine

• Upstream and downstream• PSB (upstream) and IPB(downstream)• Rx/TX 8021p mapping • Rx/TX DSCP/TC mapping• 8021p/ DSCP/TC TE remark• FE rate limit and policy control• VOQ• EPB

FE

Upstream and downstream

• Upstream: the packet from PON port(1G PON and 10G PON).

• Downstream: the packet from other port.

PSB and IPB table

• FlowID_Sel_BM• Top_802_1p_mode_sel and 802_1p_mark_control_BM • default_802_1p • DSCP_mode_sel and DSCP_mark_control_BM • default_DSCP• COS_mode_sel and COS_control_BM• default_COS• WRED_TD enable(IPB for both US and DS)• SPID and to CPU rate limit enable

FlowID_Sel_BM

B[3] : FlowID from S-Classification FIB; B[2] : FlowID from G-Classification FIB; B[1] : FlowID from VLAN Engine FIB; B[0] : FlowID from MC_FIB table for Non-Known UC packet. FlowID from EPSB table for Known UC packet. The priority is B[3] -> B[2] -> B[1] -> B[0] If all 4 bits are 0, upstream packets’ flow id = SPLLID,

downstream to PON known UC packets’ flow id = 512 + DPLLID, other packets’ flow id = default flow id.

Top_802_1p_mode_sel

• 00 (default): Outgoing top 8021p <= Rx_Top_8021p or IPB default 8021p(Untagged packets);

• 01(L2 Marking): Outgoing top 8021p <= IPB_8021p_Mark_BM;

• 10(L2 mapping): Outgoing top 8021p <= Map from Received top 8021p;

• 11(L3 to L2 mapping): Outgoing top 8021p <= Map from Outgoing DSCP.

802_1p_mark_control_BM

B[4] : 8021p from S-Classification FIB; B[3] : 8021p from G-Classification FIB; B[2] : 8021p from VLAN Engine FIB;B[1]: Reserved;B[0] represents 8021p from IPB default setting. The priority to select is: 1) S-Classify Engine 8021p; 2) G-

Classify Engine 8021p; 3) VLAN Mapping Engine 8021p; 4) IPB_Default_8021p

DSCP_mode_sel

00 (default): Final DSCP <= Rx_DSCP ; 01 (L3 Marking): Final DSCP <= IPB_DSCP_Mark_BM;10 (L3 mapping): Final DSCP <= Map from Rx DSCP; 11: Final DSCP <= from IPB default DSCP.

DSCP_mark_control_BM

B[3] : DSCP from S-Classification FIB; B[2] : DSCP from G-Classification FIB; B[1] : DSCP from VLAN Engine FIB;B[0] : DSCP from IPB default setting. The priority to select is: 1) S-Classify Engine DSCP; 2) G-

Classify Engine DSCP; 3) VLAN Mapping Engine DSCP; 4) IPB_Default_DSCP;

If all 4 bits are 0, TX DSCP = Rx DSCP

COS_mode_sel

000 : Final COS <= map from Rx_Top_8021p; 001: Final COS <= map from Rx_ DSCP (If non-IP, map

Rx_Top_8021p); 010:(Marking): Final COS <= IPB_COS_Select_BM; 011: From IPB.COS100: Final COS <= map from Tx_Top_8021p; 101: Final COS <= map from Tx_ DSCP (If non-IP, map

Tx_Top_8021p); 110,111: same as 0.

COS_control_BM

B[3] : COS from S-Classification FIB; B[2] : COS from G-Classification FIB; B[1] : COS from VLAN Engine FIB and B[0] : COS from IPB default setting. The priority to select is: 1) S-Classify Engine COS; 2) G-

Classify Engine COS; 3) VLAN Mapping Engine COS; 4) IPB _Default_COS;

Rx 8021p mapping

TX 8021p mapping

Rx DSCP/tc mapping

TX DSCP/TC mapping

8021p TE remarked

Dscp/tc TE remarked

MII project QOS

MII configuration

Downstream: configure uplink -> cascade don’t markUpstream: only pass one time FE, no problem.Local switch and p2p: has problem.

8 port project solution

8 ports problem

• Downstream: inner chip no problem, cross chip has problem.

• Upstream: no problem.• P2p and local switch: has problem.

FE rate limit control

• To CPU rate limit enable• Mirror rate limit enable(total, section 0~3)• Special packet control(rate limit enable and id)• Group A rate limit control(group A mode, group A half

mode)• Root_MC_ID control

Group A rate limit control

• total 1024 group A id• Total 8 modes 0. disable1. 0~511 US subs id, 512~1023 DS subs id 2. 0~511 SP_LLID, 512~1023 DP_LLID3. 0~511 ROOT_MC_RL_ID, 512~1023 US subs id 4. 0~511 ROOT_MC_RL_ID, 512~1023 DS subs id 5. 0~511 P2P_Group_ID[7:0] +256*UC, 512~1023 DS subs 6. Sub CLS engine7. VLAN engine

Root_MC id control

• PR group id *2 +cos[0]• Vlan group id *2 +cos[0]• 0~255 DS vlan group id based MC, 256~511 DS vlan group

id based BC and unknown MC• MC group id

FE policy control

• Default flow• Mark enable, include def_flow,MC_fib,epsb,dpllid,spllid.• Flow_b mode• Flow_id for 2k llid• Policy enable

Flow B mode

0: disable1: SPLLID[6,0]*8 + cos[2,0]2: SPLLID[7,0]*4 + cos[1,0]3: cos[2,0]*128 + SPLLID[6,0]

Flow_id for 2k llid

• 2k llid flow chart enable.• Cos[2:0] use as msb.• Lower 1024 llid use for Flow id and upper 1024 llid use

for flow B id.

VOQ: virtual output queue

• To PON known UC(include per llid replicated packet) VOQ = LLID*8 + COS(LLID = 0~255)

• Other known UC VOQ or per port replicated packet = 0x800+port*8 + COS

• Unknown packet(root MC VOQ), VOQ = 0x838 + COS

EPB: egress port behavior

• Bank id mode and bank id, bank0~bank3 are used for to PON port known UC, bank id = llid[1:0], bank4~bank7 are used for other packets, bank id = cos[1:0]

• Destination port selection, select IBM, EBM, management, etc.

VOQ remapping

Traffic Type Condition

NewVoQID

Upstream

SPID = PON (10G or 1G) & To_Aggr_Uplink

((KnowUC & DPID = Uplink|CAS)  |

(!KnowUC &  DPBM include Uplink|CAS & !Rplc_ind){6’h20, DPID[2:0],1’b1, SDID[1:0]}

Known Local switch packetSPID = PON (10G or 1G) & To_Aggr_Uplink = 0

(KnowUC & DPID = Uplink|CAS){6’h20, DPID[2:0],1’b0, DDID[1:0]}

Non Known Local switch packetSPID = PON (10G or 1G) & To_Aggr_Uplink = 0

(!KnowUC &  DPBM has Uplink|CAS & !Rplc_ind){6’h20, DPID[2:0],1’b0, SDID[1:0]}

Known pkt from Uplink/CAS to CAS/Uplink(SPID = !PON) & To_Aggr_Uplink = 0

( (KnowUC & DPID = Uplink|CAS) {6’h20, DPID[2:0],1’b1, DDID[1:0]}

Non Known pkt From Uplink/CAS to CAS/Uplink(SPID = !PON) & To_Aggr_Uplink = 0

(!KnowUC &  DPBM has Uplink|CAS & !Rplc_ind) {6’h20, DPID[2:0],1’b1, SDID[1:0]}

Bank id remap

Traffic Type ConditionNew BankID

Upstream

SPID = PON (10G or 1G) &   To_Aggr_Uplink

((KnowUC & DPID = Uplink|CAS) 

(!KnowUC &  DPBM has Uplink|CAS & !Rplc_ind)) {1’b1, SDID[1:0]}

Known pkt to Uplink/CASTo_Aggr_Uplink = 0

(KnowUC & DPID = Uplink|CAS) {1’b1, DDID[1:0]}

Non Known pkt to Uplink/CASTo_Aggr_Uplink = 0

(!KnowUC &  DPBM has Uplink|CAS & !Rplc_ind) {1’b1, SDID[1:0]}

Head T 1

oam_excl_rm [128] (1) : 0x1 Dp_llid [127:119] (9) : 0x0 Dp_llid_vld [118] (1) : 0x0 Voqid_cos [117:115] (3) : 0x0 special pkt type [114:110] (5) : 0x14

MATCH_KEEP-alive_OAM Flow_id_cir_en [109] (1) : 0x1 Dpid_is_cpu [108] (1) : 0x0 DPID [107:105] (3) : 0x0 Mc_pkt_ind [104] (1) : 0x0

Head t 2

pon_source_ind [103] (1) : 0x1 sp_cpu_rl_en [102] (1) : 0x0 spllid_cpu_rl_en [101] (1) : 0x0 sect_Mir_rl_en [100] (1) : 0x0 drop [99] (1) : 0x1 fe_te_dont_mark [98] (1) : 0x0 Pkt_color [97:96] (2) : 0x0 green flow_b_pl_en [95] (1) : 0x0 flow_b_id [94:85] (10) : 0x0 group_a_rl_en [84] (1) : 0x1

Head t 3

group_a_id [83:74] (10) : 0x0 Wred_TD_en [73] (1) : 0x1 Spid_rl_en [72] (1) : 0x1 Cpu_rl_en [71] (1) : 0x0 Bank_id [70:68] (3) : 0x0 Pkt_len [67:57] (11) : 0x44 Voq_id [56:45] (12) : 0x800 Mir_rl_en [44] (1) : 0x0 Mir_ind [43] (1) : 0x0 Mir_sect_id [42:41] (2) : 0x0

Head t 4

Pkt_type [40:39] (2) : 0x1 Mark_flag [38] (1) : 0x0 Mark_en [37] (1) : 0x1 cos [36:33] (4) : 0x0 spid [32:30] (3) : 0x0 spllid [29:21] (9) : 0x0 spec_pkt_rl_en [20] (1) : 0x0 spec_pkt_id [19:16] (4) : 0x0 Oam_excl_spid [15] (1) : 0x1 isp_plc_en [14] (1) : 0x0

Head t 5

isp_id [13:11] (3) : 0x0 Policing_en [10] (1) : 0x1 Flow_id [9:0] (10) : 0x0

TE

TE function

1. Rate limit2. Policy3. WRED4. Token bucket5. Counter

Rate limit

• Source port total rate limit• Source port packet type rate limit(BC, MC, UUC)• Source port special packet id rate limit(16 ID)• Source port to CPU rate limit• Total mirror and mirror section rate limit• LLID to CPU rate limit• Total to CPU rate limit• Group A rate limit(1024 id)• ONU special packet type rate limit

policy

• 1024 Flow id based policy• 1024 Flow B id based policy• CIR(commit input rate) and PIR(prommit input rate)• 8 ISP id policy• remark

WRED: Weighted Random Early Detection

• BM and TE• PWM flag• PRVT flag• MAX buf flag• DS/US, MC/UC, COS[2:0],MARK/UNMARK• 64 drop possibilities• Weight: Old(1-1/2^n) + New(1/2^n)

Token bucket

• Delta, add bytes per time• Time, 8000 clocks• CBS, the depth of token bucket, the maximum value of

token add.• TBC, token bucket counter, when write CBS to CSR, TBC

will be updated.

Token bucket

Head o 1

Bf_af_pr_ind [95] (1) : 0x0 Pkt_fwd_dest [94:92] (3) : 0x1 mirror_section_id [91:90] (2) : 0x1 mirror_pkt [89] (1) : 0x0 drop_pkt [83] (1) : 0x0 drop_src [88:84] (5) : 0x0 eth pkt type [82:81] (2) : 0x0 bank_id [80:78] (3) : 0x7 voqid [77:66](12) : 0x807 flow id h2b [65:64] (2) : 0x0

Head o 2

flow id l7b [63:57] (7) : 0x0 packet rpl ind [56] (1) : 0x0 know UC ind [55] (1) : 0x1 pspid [54:52] (3) : 0x4 spllid [51:43] (9) : 0x0 traffic type [42:41] (2) : 0x0 pr_group_id [40:33] (8) : 0x0 marked [32] (1) : 0x0 cos [31:29] (3) : 0x7 die_id [28:27] (2) : 0x2

Head o 3

dpid [26:20] (7) : 0x0 dpllid [19:11] (9) : 0x0 full_pkt_size [10: 0](11) : 0x40

BM: buffer management

• GBM: global buffer management(both IBM and EBM)• IBM: internal buffer management• EBM: external buffer management

BM diagram

Egress Scheduler

Wr_VEC FIFO

Memory Arbiter

Rd_VEC FIFO

Link List Manager Packet Write

Manager

Private Buffer Pool PoolPoooPoolPPool

Packet Read

Manager

Global Buffer Pool

FE INTF FIFO

Internal Data Path FIFO

1Kx128

96 128

128 Forwarding E

ngine T

raffic Engine

Data C

ongestion Control

10GPON Latn FIFO 0

BM

MAS-CSR

register

ddr3_Ctrl

IBM

Latn FIFO 1

Latn FIFO 2

Latn FIFO 3

Uplink

Cascade

MII0/MII1/CPU

128

Wr_Dat FIFO

128

256

RXD FIFO

RowCmd FIFO TXD FIFO Cmd2rd FIFO

DDR3 PTCL

DDR3 PUB/PHY

DDR3

256 256

128

EBM

0

.

.

.

.

.

.

7

B A N K

0

.

.

.

.

.

.

7

B A N K

0 .

.

.

.

.

.

7

B A N K

Latn FIFO 4

1GPON

Rtn-vec

1kx128

2kx128

1kx128

1kx128

2kx128 128

128

128

128

128

Netw

ork Interface in ON

U

mode

EB

M m

od

e

IBM

mode

10GPON, Uplink, Cascade (EBM output only)

Voq-based buf full flag Port-based buf full flag Bank-based buf usage CPU port buf usage

8COS pkt size update

Port-based H/L flow Bank-based H/L flow FIFO-based H/L flow

MC Latn FIFO 5

2kx128

128

GBM

• Manage FE->BM FIFO• Minimum packet size detect• Counter clear mode configuration• Err detect, counter and interrupt• Redirection • Uplink port pause control

IBM

EBM

• 8 Banks• Cell, 32 bytes, two packets cannot occupy same cell• Private buffer• Share buffer• Buffer size, 16kB• Flow control threshold• PWM• PRM• LLM

Tiger4 EBM configuration 1

Tiger4 EBM configuration 2

DDR bandwidth

Bandwidth = clock * 2 * (bit wide) *80%

Example: clock = 750Mhz, bit wide = 32

Bandwidth = 0.75*32*2*0.8 = 38.4 Gbps

ES: Egress schedule

Port schedule : bandwidth

10G PON port has 9/32 of the total BW;1G PON port has 1/32 of the total BWUplink port has 9/32 of the total BW; Cascade port has 3/32;Each of the 2 MAN port has 1/32 of the total BW;CPU port has 1/32 of the total BW;MC port has 7/32 of the total BW;

Port schedule: shaper

• Delta• Time• CBS

• Clock = 400Mhz

Port schedule: SP+DRR

• SP• DRR• SP+DRR

ONU schedule

• DRR• shaper

ONU queue scheduler

• Queue shaper• SP ， queue 7 has the highest priority• DRR• SP+DRR

Flow control

• RX flow control• TX flow control flow control based RX MAC threshold flow control base BM available port buffer available bank buffer available flow control based source port rate limit• QBB flow control RX is ES TX is based bank threshold

Thanks Jianguang