Scalable Edge Bridge FDBFor Datacenter Networks

July-2012

Agenda

Problem statement and related work

Protocol properties, concepts and operation

Proposal for data and control planes

Summary & discussion

2

A

B

C

OverlayNetwork

End-Station

Edge-Bridge

Problem Statement and Related Work

Problem Statement and Related Work

Problem statement Large # of VMs in datacenters (>1M) large address table in datacenter bridges Support for hot VM migration VM address must not change address table scaling

techniques based on address aggregation limit migration options– For example, IP stations can migrate within the same VLAN

Overlay networks solve address scaling problem in Core Bridges Core Bridge address table ~= # Edge Bridges << # of VMs in the network Lot’s of work on overlay protocols: SPB, PBB, VPLS, TRILL, VXLAN, NVGRE

How to scale the address table in Edge Bridges (EB)? VXLAN/NVGRE – specific solutions for IP overlay SPB/TRILL – none (July-2012)

Objective: provide a solution to address scaling in SPB Edge Bridges The solution must complement (not replace) overlay network protocols Preferably, one solution should fit many overlay network protocols, so it can be easily

adapted to work with other overlay protocols

4

A

B

C

Bridge FDB Scaling(BFS)

Protocol Concepts and Operation

Bridge FDB Scaling (BFS) Concepts

BFS defines a handshake between the EB and the End-Station(An End-Station may host 1 or more VMs) Capabilities exchange use control-plane Dynamic operation uses the data-plane

EB operation in a nutshell Learns addresses of local VMs & remote EBs (but not remote VMs) Uses data-plane signaling to informs the End-Station of the path in the overlay network Uses the path signaled by the End-Station to forward traffic to remote VMs over the overlay

network

End-Station operation in a nutshell Sends data traffic to EB with path indication Updates its path database (Path$) using the indications received from the EB

6

7

BFS Databases and Signaling

VM1

VM2

B

VM Port

A

B

C

A.1

A.2

C.1

B.1

B.2

C.2

DS

DS

B

P EB

1 A

2 B

3 CA.1$

VM Path

DS S.Path

Generated by VM

DS T.Path

DS

ServerEB

OverlayNetwork

EBServer

Rx byVM

Edge Bridge

End-Station

Path$

Overlay FDB

LocalFDB

EB Operation

Overlay FDB learning Control plane triggered as specified by the overlay protocol (e.g. IS-IS for SPB)

Address learning process (Local FDB) Data-plane learning

– Don’t learn on overlay ports– Learn on local ports

Forwarding packets received on local ports If packet has no T.Path indication

Lookup in local FDB using DAif found forward accordingly, don’t assign S.Path to traffic to local portselse flood to local and overlay ports

else // packet has T.Path indicationObtain the overlay path attributes using T.PathRemove T.Path, add ovelay tunnelSend to overlay

Forward packets received on overlay ports Lookup overlay FDB with the overlay header, obtain S.Path

Remove overlay header, assign S.PathLookup local FDB with DAif found, forward accordinglyelse flood to local ports 8

End-Station Operation

Forwarding packets received from VM Lookup Path$ with DA

If found, assign T.Path to the packet and forward to EBelse forward to EB w/o T.Path

Forward packets received from EB Use DA or 802.1Qbg/802.1BR indication to forward to the VM

Path$ update policy (packets received from EB) If packet has no S.Path, don’t update Path$

else // packet has S.Pathupdate Path$ if any of the following is met

DA indicates a VM hosted by this End-Station, ORDA=BC and L3-DA indicates a VM hosted by this End-Station

9

A

P EB

1 A

2 B

3 C

10

BFS Operation Example #1

VM1VM2 flooded Unicast forwarding

VM1

VM2

A

VM Port

C

VM Port

B

VM Port

A

B

C

A.1

A.2

C.1

B.1

B.2

C.2

21

DS

1 A.1

21

DS

BCA21

DS

BCA

Dataplane learning EB table size = # of local VMs + # of EBs in the network

C

P EB

1 A

2 B

3 C

B

P EB

1 A

2 B

3 C

A.1$

VM Path

B.1$

VM Path

21

DS

1

s.Path

21

DS

1

s.Path

21

DS

1

s.Path

21

DS

1

s.Path

1 1

Learn only in B.1

SPB Overlay

A

P EB

1 A

2 B

3 C

11

BFS Operation Example #2

VM2VM1 reply

VM1

VM2

A

VM Port

C

VM Port

B

VM Port

A

B

C

A.1

A.2

C.1

B.1

B.2

C.2

21

SD

1 A.1

BA

D S

21

Dataplane learning EB table size = # of local VMs + # of EBs in the network

C

P EB

1 A

2 B

3 C

B

P EB

1 A

2 B

3 C

A.1$

VM Path

B.1$

VM Path

11

DT.Path

2

S

1 1

12

DS.Path

2

S

21

SD

2 2

2 B.1

SPB Overlay

BFS Data and Control Planes(A Proposal)

13

BFS Data and Control Planes - A Proposal

Control protocol Capabilities negotiation between the End-Station and the Edge Bridge Modify 802.1Qaz (DCBx)

Data-plane protocol (2 options) Add Path-ID Tag (P-Tag)

– S-channel/E-Tag is outer– P-Tag is inner:

–16b source/target-path-id–Source/target depends on direction

Modify BPE E-Tag– End-StationEB

–Ingress-ECID – identical use to BPE–E-CID – target-path-id

– EBEnd-Station–Ingress-ECID

–Ingress-ECID < 4K local virtual port (identical to BPE)–Ingress-ECID =>4K source-path-id

–E-CID – identical use to BPE

DA (6B)

SA (6B)

S-Channel/E-Tag (8/4B)

P-Tag (4B) VLAN (4B)

Payload+

FCC

Summary

Summary of BFS Properties

Complements SPB towards scaling the EB FDB

A generic solution that can be considered for additional overlay protocols

Small Path$ in End-Station Holds active sessions only – comparable in size to the ARP$

Easy to implement Local scope: end-station to edge-bridge protocol Simple control-plane – only need to negotiate capabilities, no dynamic operation

– Extend DCBX 802.1Qaz

Simple extension of existing data-plane protocols– Extends 802.1BR/802.1Qbg with a P-Tag or modifies 802.1BR E-Tag

Easy to deploy Co-exists with 802.1Qbg/802.1BR protocols Support for incremental upgrade per EB granularity

15

Thank youContact: Carmi Arad, carmi@marvell.com