next generation l2 vpn -...

Next Generation L2 VPN

Ali Sajassi– Distinguished Engineer, Cisco

BRKMPL-2333

• EVPN Overview

• EVPN Technology Prime

• EVPN Startup Sequence

• EVPN Operation

• A Day in Life of a Packet

• EVPN-VPWS

• EVPN Deployment: DC Fabric Evolution with EVPN-VXLAN

• EVPN Deployment: DC Fabric and WAN Integration

Agenda

EVPN Overview

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

L2VPN Technologies Evolution

802.1D

VLAN Scale

Native L2 Bridging Technologies

802.1Q 802.1ad(QinQ)

802.1ah(MACinMAC)

802.1aq(SPB)

802.1Qbp(ECMP)

More VLAN

ScaleMAC Scale

Shortest

Path FwdECMP

• What about Inter Domain (WAN) Connectivity?

• What about IP or MPLS fabric?

• What about industry traction & multi-vendor interop?

• What about All-Active multi-homing?

• What about multi-pathing (not ECMP)?

IEEE 802.1Qbp

Large # of VLANs – 16 millions ✔

Large # of MACs – MAC-in-MAC ✔

Optimum Forwarding ✔

ECMP ✔

BRKMPL-2333 5


L2VPN Technologies Evolution

VPLS

PW Scale

L2 VPN Technologies

H-VPLS PBB-VPLS EVPN(RFC 7432)

MAC Scale

To Address

all major

shortcomings

• Inter Domain (WAN) Connectivity? Yes

• IP or MPLS fabric? Yes

• Industry traction & Multi-vendor interop? Yes

• All-Active multi-homing? Yes

• Multi-pathing (not ECMP)? Yes

PBB-EVPN

EVPN-VxLAN

EVPN-IRB

EVPN-VPWS

…

BRKMPL-2333 6


What’s the big deal about EVPN?EVPN is next generation all-in-one VPN solution

It not only does the job of many other VPN technologies but it does it better !!

E-LAN(MP2MP

L2VPN)

E-LINE(P2P

L2VPN)

E-TREE(P2MP

L2VPN)

L3VPN

EVPN

VPWS

PBB-

EVPN

EVPN

DC Fabric(IntraDC

Overlay)

IRB(L2/L3

Overlay)

DCI(InterDC)

EVPN-

IRB

EVPN-

Overlay

EVPN-

L3VPN

EVPN

ETREE

EVPN-

DCI

VPLS PW 4364VPLS-

ETREEVxLAN

TRILLVPLS,OTV

BRKMPL-2333 7


2006 2010 2011 2013 2015

- OPEN project was started at

Cisco

- OPEN = Optimum Ethernet

Network

- Introduced to IETF as

Routed-VPLS

- Merged with Juniper’s

MAC-VPN and was

introduced an EVPN

Following drafts were

introduced:

- EVPN

- PBB-EVPN

- EVPN-VPWS

- EVPN-Overlay

- EVPN-ETREE

Following drafts were

introduced

- EVPN IRB

- EVPN DCI

Enhancements

- Virtual ES

- Optimized ingress replication

- IGMP aggregation between

PODs

- mcast tunnels between DCs

- Inter-AS for IRB

- L3VPN multi-homing

BRKMPL-2333 8


EVPN in a Nut ShellMAC learning in control plane (via BGP)

IP or MPLS

PE1

CE1

PE2

PE3

CE3

PE4

C-MAC:

M1

Single active

multi-homingAll active multi-

homing

MAC Routing: Control plane (BGP)

advertise the learnt MACs from CE

Data Plane: IP or MPLS,

flexible

Optimum forwarding,

ECMP, Multi-pathing

Common L2/L3 VPN

Operational Mode

Flexible Policy Control

Consolidated VPN

service with x-EVPN

BRKMPL-2333 9

EVPN Technology Prime


Service Additional Capabilities

E-LAN • Provides All-Active multi-homing

• Prevents loop for both all-active & single-active even in transient

state

• Efficient utilization of network cross-sectional bandwidth (via

optimum forwarding, ECMP, multi-pathing on a per flow basis)

• Flexible policy control per MAC and per Site

EVPN Does it Better than VPLS !!

BRKMPL-2333 11


MAC1 MAC2CE1 CE2

• Packets originated from MAC2 arrives at both PE3 and PE4 which get forwarded subsequently to PE1. PE1 keeps flip/flopping between PE3 and PE4 for learning of MAC2!!

• VPLS cannot do proper load-balancing because doesn’t support Aliasing. When PE3 wants to forward a packet with destination address MAC1, it needs to send it to both PE1 and PE2 even though it only learned MAC1 from PE1.

PE1

PE2

PE3

PE4

MAC1 MAC2PE1

PE2

PE3

PE4

CE1 CE2

Flip/flopping !

Load balancing

BRKMPL-2333

VPLS cannot provide All-Active Multi-Homing Because:

MAC1 MAC2PE1

PE2

PE3

PE4

CE1 CE2

Echo !

• BUM packets forwarded from PE1 can get loopback to the originating CE

12


EVPN All-Active Multi-Homing Principles

MAC1 MAC2

MAC1 MAC2PE1

PE2

PE3

PE4

CE1 CE2

CE1 CE2

• ARP broadcast packet doesn’t get loopback to the originating CE device. Split-horizon

• Either PE3 or PE4 forward the broadcast frame to the far-end dual-homed device CE2. DF selection

• When PE1 & PE2 forward traffic for MAC1, there is no flip/fopping on PE3 because of MAC learning in control plane.

• When PE3 wants to forward a packet with destination address MAC1, it needs to send it to both PE1 and PE2 even though it only learned MAC1 from PE1. Load balancing via aliasing

PE1

PE2

PE3

PE4

MAC1 MAC2PE1

PE2

PE3

PE4

CE1 CE2

Echo !

Duplicate !

Load balancing

BRKMPL-2333 13


EVPN Efficient Cross-Sectional BW Utilizaiton

• EVPN provides per-flow load-balancing among egress PEs using BGP multi-pathing

• Per-flow load balancing between ingress and egress PEs are provided using IGP ECMP (ingress PE still needs to add entropy field in the packet).

P

E

P

E

P

E

P

P

P

P

Flow Based Multi-Pathing in the CoreVlan X -

F1Vlan X –

F2Vlan X –

F3Vlan X –

F4

P

E

P

E

P

E

Flow Based Load-balancing – PE to PE direction

Vlan X -

F1Vlan X

– F2

BRKMPL-2333 14


E-VPN Concepts

Ethernet Segment

• Represents a ‘site’

connected to one or more

PEs

• Uniquely identified by a 10-

byte global Ethernet

Segment Identifier (ESI)

• Could be a single device or

an entire network

Single-Homed Device (SHD)

Multi-Homed Device (MHD)

Single-Homed Network (SHN)

Multi-Homed Network (MHN)

BGP Routes

• E-VPN and PBB-EVPN

define a single new BGP

NLRI used to carry all E-

VPN routes

• NLRI has a new SAFI 70

(EVPN), AFI 25 (L2VPN)

• Routes serve control plane

purposes, including:

MAC address reachability

MAC mass withdrawal

Split-Horizon label adv.

Aliasing

Multicast endpoint discovery

Redundancy group discovery

Designated forwarder election

E-VPN Instance (EVI) & MAC-VRF

• EVI identifies a VPN in the

network

• Encompass one or more

bridge-domains, depending

on service interface type

Port-based

VLAN-based (shown above)

VLAN-bundling

VLAN aware bundling (NEW)

BGP Route Attributes

• New BGP extended

communities defined

• Expand information carried

in BGP routes, including:

MAC address moves

C-MAC flush notification

Redundancy mode

MAC / IP bindings of a GW

Split-horizon label encoding

PE

MAC-VRF

MAC-VRF

PE1

PE2

CE1

CE2

SHD

MHD

ESI1

ESI2

Route Types

[1] Ethernet Auto-Discovery (AD) Route

[2] MAC Advertisement Route

[3] Inclusive Multicast Route

[4] Ethernet Segment Route

[5] IP Prefix Route

Extended Communities

ESI MPLS Label

ES-Import

MAC Mobility

Default Gateway

BRKMPL-2333 15


EVPN Route Types & Benefits

Route Type Usage Benefits

Ethernet A-D Route

(Type 1)

• Aliasing

• Mass Withdraw of addresses

• SH/AA MH Indication

• Advertising Split-Horizon Label

• Loop avoidance – even

transient

• Fast convergence

• Efficient load balancing

• Per-site policy

MAC/IP Advertisement Route

(Type 2)

• Advertise MAC (and IP) reachability

• Advertise MAC/IP binding

• MAC mobility

• Per MAC policy

• ARP suppression

• Workload Mobility

Inclusive Multicast Route

(Type 3)

• Auto discovery of multicast tunnel

endpoints & mcast tunnel type

• Support multicast even

when core doesn’t

Ethernet Segment Route

(Type 4)

• Auto discovery of redundancy group • A/A and S/A MHD &

MHN support

IP Prefix Route

(Type 5)

• IP Prefix advertisement (not for IP

host advertisement)

• IP route aggregation

• Interop w/ L3VPNBRKMPL-2333 16


EVPN BGP Routes RFC7432

• EVPN defines a new BGP NLRI used to carry all EVPN routes

• BGP Capabilities Advertisement used to ensure that two speakers support EVPN NLRI (per RFC4760)

• AFI 25: L2VPN, SAFI 70: EVPN

Route Type

Length

Route type specific

1 byte

1 byte

[1] Ethernet Auto-Discovery (AD) Route

[2] MAC Advertisement Route

[3] Inclusive Multicast Route

[4] Ethernet Segment Route

[5] IP Prefix Route

EVPN NLRI

Variable

BRKMPL-2333 17


• EVPN Operation




EVPN Startup Sequence

Segment Auto-Discovery

ESI Auto-Sensing

Redundancy Group Membership Auto-Discovery

VPN Auto-Discovery

Multicast Tunnel Endpoint

Discovery

DF Election & VLAN Carving

ESI Label & MH type Discovery

BRKMPL-2333 19

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 20

ESI Auto-Sensing


ESI Auto-Sensing3

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

LACP PDU

exchangeCE LACP info:

LACP System ID (MAC) (6B)

e.g. 0011.0022.0033

LACP System Priority (2B)

e.g. 0000

LACP Port Key (2B)

e.g. 0018

ESI (10B) can be auto-generated1

from CE’s LACP information ->

concatenation of CE’s LACP

System Priority + Sys ID + Port Key

Example:

0000. 0011.0022.0033.0018

System

Priority

2 bytes 6 bytes 2 bytes

System MAC

AddressPort Key

BRKMPL-2333

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKMPL-2333

MPLS

PE1

CE1

PE2

PE3

PE4

PE 1 Eth Segment Route

RD = RD10

ESI = ESI1

ES-Import Route Target

e.g. 0011.0022.0033

MAC address portion

of ESI (6B)


RD = RD20

ESI = ESI1

ES-Import Route Target

e.g. 0011.0022.0033


ESI Auto-Sensing


RD – RD unique per

adv. PE


PE4

PE1000

Exchange of Ethernet

Segment Routes

21


MPLS

PE1

CE1

PE2

PE3

CE3

PE4

Ordered List of discovered PEs

starting from zero (lowest IP add)


ESI Auto-Sensing

PE Ordered List

Position PE

0 PE1

1 PE2

Modulo Operation

VID

VID mod N

(N = # of PEs)

(e.g. VID mod 2)

100 0

101 1

102 0

103 1

PE Ordered List

Position PE

0 PE1

1 PE2

Modulo Operation

VID (VID mod 2)

100 0

101 1

102 0

103 1

Result of modulo

operation is used to

determine DF and

BDF status

Example:

PE2 DF for VIDs 101, 103

PE2 BDF for VIDs 100, 102

Example:

PE1 DF for VIDs 100, 102

PE1 BDF for VIDs 101, 103




BRKMPL-2333 22



ESI Auto-Sensing



MPLS

PE1

CE1

PE2

PE3

PE4

PE1 Eth A-D per ES

RD = RD-1a

ESI1

Eth Tag = MAX-ET

Label = 0

ESI Label ext. com

L1

DF Election and VLAN Carving


ESI1

PE2 Eth A-D per ES

RD = RD-1b

ESI1

Eth Tag = MAX-ET

Label = 0

ESI Label ext. com

L2 23


VPN Auto-Discovery

Multicast Tunnel Endpoint

Discovery

PE 1 Inclusive Multicast Route

RD = RD-1a

PMSI Tunnel Attribute

Tunnel Type (e.g. Ing. Repl.)

Label (e.g. L1)

RT ext. community

RT-a

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

Tunnel Type – Ingress

Replication or P2MP LSP

Mcast MPLS Label – used to

transmit BUM traffic -

downstream assigned (ing.

repl.) or upstream assigned

(Aggregate Inclusive P2MP

LSP2)

PMSI - P-Multicast Service Interface

BUM – Broadcast / Unknown Unicast / Multicast


adv. PE per EVI

RT – RT associated with a

given EVI


RD = RD-2a



Label (e.g. L2)

RT ext. community

RT-a

Multicast Tunnel Endpoint Discovery

BRKMPL-2333 24


EVPN BGP route 0x4 - Ethernet Segment Route

• Usage:• Auto-discovery of multi-homed Ethernet Segments• Designated Forwarder election

• Tagged with ES-Import Extended Community

• PEs apply route filtering based on ES-Import community. Thus, Ethernet Segment route is imported only by the PEs that are multi-homed to the same Ethernet segment

RD

Ethernet Segment Identifier

8 bytes

10 bytes

Unique per Advertising PE

ESI of Ethernet Segment

Route Type specific encoding of E-VPN NLRI

IP Address Length

Originating Router’s IP add.

1 bytes

4 or 16 bytes

IP address length

IPv4 or IPv6 address

BRKMPL-2333 25


ES-Import RT Extended Community

Usage:

• Sent with Ethernet Segment route

• Limits the scope of Ethernet Segment routes distribution to PEs connected to the same multi-homed Segment

0x02

ES-Import6 bytes

MAC Address portion of the ESI

0x06

BRKMPL-2333 26


EVPN BGP Route 0x1 – Ethernet Auto-discovery Route

This route has two flavors:

Per-ES Ethernet A-D route

• Advertise the Split-Horizon Label associated with an Ethernet Segment

• For AA or SA MH indication

• Used for MAC Mass-Withdraw

Per-EVI Ethernet A-D route

• Advertise VPN label used for Aliasing or Backup-Path

RD


Ethernet Tag ID

8 bytes

10 bytes

3 bytes MPLS Label

Unique per Advertising PE per EVI


Set to VLAN or I-SID for VLAN-Aware

Bundling Service interface, otherwise 0

VPN (Aliasing) Label per (ESI,

Ethernet Tag)

Unique per Advertising PE


MUST be set to 0

MUST be set to MAX-ET

(0xFFFFFFFF)4 bytes

BRKMPL-2333 27


ESI Label extended community

Usage:

• Sent with Ethernet AD Route per ES

• Advertises the Split-Horizon Label for the Ethernet Segment

• Indicates the Redundancy Mode: Single Active vs. All-Active

0x01

Flags

Reserved

ESI MPLS Label

Bit 0: Redundancy Mode

(single active vs. all active)

Ethernet Segment Split-

Horizon Label

0x06

Set to 0

ESI-1

MAC1 MAC2PE1

PE2

PE3

PE4

Agg1 Agg2

ESI-2

PE1 advertises in BGP a split-horizon label associated

with the ESI-1 (in the Ethernet AD route)

Split-horizon label is only used for multi-destination

frames (unknown unicast, mcast, bcast)

When PE1 wants to forward a multi-destination frame, it

appends this SH label to the packet

PE2 uses this label to perform split-horizon filtering

for frames destined to ESI-1 - e.g., a frame originated by

a segment must not be received by the same segment

BRKMPL-2333 28


EVPN BGP route 0x3 – Inclusive Multicast

• Usage:

• Multicast tunnels used to transport Broadcast, Multicast and Unknown Unicast frames (BUM)

RD

Ethernet Tag ID

8 bytes

4 bytes


Set to VLAN or I-SID for VLAN-Aware Bundling Service interface, otherwise 0


IP Address Length

Originating Router’s IP add.

1 bytes

4 or 16 bytes

IP address length

IPv4 or IPv6 address

BRKMPL-2333 29


PMSI Tunnel Attribute – RFC6514

Flags

Tunnel Type

1 bytes

1 bytes

Flags based on RFC6514

Ingress Replication/mLDP etc.


MPLS Label

Tunnel Identifier

3 bytes

variable

Multicast MPLS Label

When the Tunnel Type is set to Ingress Replication, the Tunnel Identifier carries the unicast tunnel endpoint IP address of the local PE that is to be this PE's receiving endpoint address for the tunnel.

BRKMPL-2333 30


• EVPN Operation





MAC/IP Advertisement

Route (Type 2)

• Advertise MAC (and IP)

reachability


• MAC Mobility

• Per MAC (and IP)

policy

• ARP suppression


MAC Address Reachability

• PE1 & PE2 learns MAC1 from CE1 and advertises in BGP to all other PEs with ES field in the MAC/IP advertisement set to ESI1

• PE3 and PE4 learn that MAC1 sits behind ESI1 which in turn sits behind PE1 & PE2

• PE3 and PE4 now know for packets destined to CE1, they can load balanced between PE1 and PE2

BRKMPL-2333

MAC1

PE1

PE2

PE3

PE4CE1

CE3

CE4

RR

ESI1

32


ARP Broadcast Suppression

• CE1 sends out an ARP request for CE3’s IP3

• PE1 snoops the ARP packet and learns (MAC1, IP1). It adds MAC1 to its MAC-VRF, MAC1/IP1 binding to its ARP cache. It also advertises this binding to all other PEs in BGP and floods this initial ARP request.

• All other PEs learn of (MAC1, IP1). They add the MAC1 to their MAC-VRFs and add (MAC1, IP1) to their ARP cache.

• Now, when CE4 sends an ARP request for IP1, PE4 has the binding info and can provide an ARP response (e.g., ARP proxy).

MAC1, IP1

PE1

PE2

PE3

PE4

CE1 CE3

CE4

3. ARP Request (IP1)

4. ARP Reply (IP1)Act as ARP

proxy for IP1.

Challenge:

How to reduce ARP broadcasts over the

MPLS/IP network, especially in large

scale virtualized server deployments?

BRKMPL-2333



Route (Type 2)


reachability


• MAC Mobility


policy

• ARP suppression


MAC3, IP3

33


MAC Mobility

• At T0, PE1 learn the MAC1, and advertise to all other PEs

• At T1, MAC1 move to the PE3. PE1 is not aware of this

• PE3 learn the MAC1. It will overwrite the MAC route learnt from PE1

• PE3 will advertise MAC1 to all other PEs with sequence number +1

• All other PE will overwrite the MAC route

• Original PE1 will withdraw its old route

Challenge:

How to handle MAC move ?

MAC1, IP1

PE1

PE2

PE3

PE4

CE1 CE3

MAC1, IP1

BRKMPL-2333



Route (Type 2)


reachability


• MAC Mobility


policy

• ARP suppression


34


• PE advertises in EVPN Ethernet AS route with a split-horizon label (ESI MPLS Label) associated with each multi-homed Ethernet Segment

• Split-horizon label is only used for multi-destination frames (Unknown Unicast, Multicast & Broadcast)

• When an ingress PE floods multi-destination traffic, it encodes the Split-Horizon label identifying the source Ethernet Segment in the packet

• Egress PEs use this label to perform selective split-horizon filtering over the attachment circuit

Challenge:

How to prevent flooded traffic from

echoing back to a multi-homed

Ethernet Segment?

PE1

PE2

PE3

PE4

CE1 CE3

ESI-1 ESI-2

CE4

CE5Echo !

BRKMPL-2333

Split Horizon Filtering


Ethernet A-D Route

(Type 1)

• Advertising Split-Horizon

Label

• Aliasing

• Mass Withdraw of

addresses


• Loop avoidance –

even transient

• Efficient load


• balancing

• Per-site policy

35


• PEs advertise in BGP the ESIs of local multi-homed Ethernet Segments.

• All-Active Redundancy Mode indicated

• When PE learns MAC address on its AC, it advertises the MAC in BGP along with the ESI of the Ethernet Segment from which the MAC was learnt.

• Remote PEs can load-balance traffic to a given MAC address across all PEs advertising the same ESI.

Challenge:

How to load-balance traffic towards a multi-

homed device across multiple PEs when MAC

addresses are learnt by only a single PE?

MAC1

PE1

PE2

PE3

PE4

CE1 CE3

CE4ESI-1

I can

reach

ESI1(All-Active)

I can

reach

ESI1(All-Active)

MAC1

I can reach

MAC1 via ESI1

MAC1 ESI1 PE1

PE2

BRKMPL-2333

Aliasing


Ethernet A-D Route

(Type 1)


Label

• Aliasing


addresses



even transient

• Efficient load


• balancing

• Per-site policy

36


• PEs advertise two sets of information:

• MAC addresses along with the ESI from the address was learnt

• Connectivity to ESI(s)

• If a PE detects a failure impacting an Ethernet Segment, it withdraws the route for the associated ESI.

• Remote PEs remove failed PE from the path-list for all MAC addresses associated with an ESI.

• This effectively is a MAC ‘mass-withdraw’ function.

Challenge:

How to inform remote PEs of a failure

affecting many MAC addresses quickly while

the control-plane re-converges?

BRKMPL-2333

MAC Mass Withdraw


Ethernet A-D Route

(Type 1)


Label

• Aliasing


addresses



even transient

• Efficient load


• balancing

• Per-site policyMAC1

PE1

PE2

PE3

PE4

CE1 CE3

CE4ESI-1

MAC1,

MAC2,…

MACn

MAC1, MAC2, .. MACn ESI1

PE1

PE2

I lost ESI1

X

37


EVPN BGP Route 0x2 – MAC Advertisement

RD


Ethernet Tag ID

8 bytes

10 bytes

4 bytes

MAC Address Length

MAC Address

IP Address Length

IP Address

MPLS Label1

1 byte

6 bytes

1 byte

4 or 16

3 bytes


ESI of Ethernet Segment on which MAC Address was learnt. All 1s ESI for PBB-EVPN

Set to VLAN or I-SID for VLAN-Aware Bundling Service interface, otherwise 0

Allows for MAC Address ‘summarization’, i.e. hierarchical MAC Addresses. Typically set to 48

Could be C-MAC Address (EVPN) or B-MAC Address (PBB-EVPN)

To distinguish IPv4 vs. IPv6 addresses.

Used for ARP flood suppression or for Integrated Routing and Bridging (IRB).

MAC & IP Labels - downstream assigned

BRKMPL-2333

MPLS Label23 bytes

38


MAC Mobility extended community

• Used to tag the MAC Advertisement route

• EVPN: Indicates that a MAC address has moved from one PE to another

0x00

Reserved

Sequence Number4 bytesIndicates the count of MAC address mobility events

0x06

2 bytesSet to 0

BRKMPL-2333 39


EVPN BGP Route 0x1 – Ethernet Auto-discovery Route

This route has two flavors:

RD


Ethernet Tag ID

MPLS Label



Set to VLAN or I-SID for VLAN-Aware

Bundling Service interface, otherwise 0

VPN (Aliasing) Label per (ESI,

Ethernet Tag)

BRKMPL-2333 40

Per-EVI Ethernet A-D route

• Advertise VPN label used for Aliasing or Backup-Path

• Startup Sequence

• Operation




MPLS

PE1

CE1

PE2

PE3

CE3

PE4

Life of a PacketIngress Replication – Multi-destination Traffic Forwarding

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

VID 100

SMAC: M1

DMAC: F.F.F L3

L2 L5

L4

Mcast MPLS

Label assigned by

PE3 for incoming

BUM traffic on a

given EVI

PSN MPLS label

to reach PE3

ESI (split-horizon)

MPLS label

allocated by PE2

for segment ES1PE4 – non-DF for

given EVI drops

BUM traffic

PE2 – drops BUM

traffic originated

on ES1

PE1 receives broadcast

traffic from CE1. PE1

forwards it using ingress

replication – 3 copies

createdPE3 – as DF, it

forwards BUM

traffic towards

segment

During start-up

sequence, PE2 sent Per-

ESI Ethernet AD route

with ESI MPLS label

(split-horizon) (see

below)

PE 4 Inclusive Multicast

Route

RD = RD-4a


Tunnel Type = Ing. Repl.

Label = L4

RT ext. community

RT-aMcast MPLS Label – used to


downstream assigned (for

ingress replication)

During start-up

sequence, PE1, PE2,

PE3, PE4 sent Inclusive

Multicast route which

include Mcast label

PE 2 Eth A-D Route (Per-ESI)

RD = RD20

ESI = ESI1

ESI MPLS Label ext. comm.

Redund. Flag = All-Active

Label = L5

RT ext. community

RT-a, RT-b, RT-c, RT-d

ESI MPLS Label – used by

local PEs for split-horizon -



BRKMPL-2333 42


MPLS

PE1

CE1

PE2

PE3

CE3

PE4

Life of a Packet (cont.)Unicast Traffic Forwarding

PE1 MAC Route

RD = RD-1a

ESI = ESI1

MAC = M1

Label = L1

RT ext. community

RT-a

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

L1

MP2P VPN Label

assigned by PE1

for incoming traffic

for the target EVI

PSN MPLS label

to reach PE1

PE3 forwards

traffic destined to

M1 based on RIB

information (PE1)

PE3 RIB

VPN MAC ESI

RT-a M1 ES1

Path List

NH

PE1

VID 100

SMAC: M2

DMAC: M1VID 100

SMAC: M1

DMAC: F.F.F

MP2P VPN Label –

downstream allocated label

used by other PEs to send

traffic to advertised MAC

MAC advertised

by route

BRKMPL-2333 43


Life of a Packet (cont.)Unicast Forwarding and Aliasing

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

PE3, PE4 RIB

VPN MAC ESI

RT-a M1 ES1

Path List

NH

PE1

PE2

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

L2

Aliasing MPLS

Label assigned by

PE2 for (ES1, EVI)

pair

PSN MPLS label

to reach PE2

PE3 forwards

traffic on a flow

(flow 2) based on

RIB information

(towards PE2)

VID 100

SMAC: M4

DMAC: M1L1

MP2P VPN

Label

assigned by

PE1 for

incoming for

target EVI

PSN MPLS label

to reach PE1

PE3 forwards

traffic on a flow

(flow 1) based on

RIB information

(towards PE1)

VID 100

SMAC: M3

DMAC: M1

During start-up sequence,

PE2 sent Per-EVI Ethernet

AD route (see below)

PE1 MAC Route

RD = RD-1a

ESI = ESI1

MAC = M1

Label = L1

RT ext. community

RT-a

MP2P VPN Label –




MAC advertised

by route

PE 2 Eth A-D Route (Per-EVI)

RD = RD-2a

ESI = ESI1

Label = L2

RT ext. community

RT-a

Aliasing MPLS Label – used

by remote PEs to load-

balance among local PEs

During start-up

sequence, PE1 sent Per-

EVI Ethernet AD route

VID 100

SMAC: M1

DMAC: F.F.F

BRKMPL-2333 44


E-VPN Operational ScenariosMAC Mobility

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

VID 100

SMAC: M1

DMAC: M2

PE3 / PE4 RIB

VPN MAC ESI

RT-a M1 ES1

Path List

NH

PE1

PE1 advertises MAC

route for M1. Route

may include MAC

mobility community

1Host M1 moves

from CE1 to CE3’s

location

3

M1M1 M1

VID 100

SMAC: M1

DMAC: M2

PE3 / PE4 install

M1 route towards

PE1

2

PE3 MAC Route

RD = RD-3a

ESI = ESI2

MAC = M1

Label = L3

MAC Mobility ext.

community.

Seq. Num = 2

RT ext. community

RT-a

PE1 / PE2 RIB

VPN MAC ESI

RT-a M1 ES2

Path List

NH

PE3

PE1 MAC Route

RD = RD-1a

ESI = ESI1

MAC = M1

Label = L1

MAC Mobility ext.

community

Seq. Num = 1

RT ext. community

RT-a

After host sends traffic

at new location, PE2

now adv MAC route

for M1 incrementing

sequence # in MAC

mobility community

4

PE1 withdraws its

M1 route and

installs a new one

pointing to PE3

5

BRKMPL-2333 45


E-VPN Failure Scenarios / ConvergenceLink / Segment Failure – Active/Active per Flow

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

PE3, PE4 RIB

VPN MAC ESI

RT-a M1 ES1

Path List

NH

PE1

PE2

PE3, PE4 RIB

VPN MAC ESI

RT-a M1 ES1

Path List

NH

PE2

PE1 withdraws

individual MAC

advertisement routes

related to failed

segment

7

PE1 withdraws Ethernet

Segment Route

3

PE1 withdraws Per-ESI

Ethernet AD route for

failed segment

2

PE2 recalculates

DF/BDF. Becomes DF

for all EVIs on segment

5

Mass withdrawal - PE3

/ PE4 remove PE1 from

path list for all MAC

addresses of failed

segment (ES1)

4

PE1 detects failure

of one of its

attached segments

1

PE2 adv. M1 MAC route

after CE traffic is

hashed towards PE2

6

PE1

BRKMPL-2333 46


E-VPN Failure Scenarios / ConvergencePE Failure

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

PE3, PE4 RIB

VPN MAC ESI

RT-a M1 ES1

Path List

NH

PE1

PE2

PE3, PE4 RIB

VPN MAC ESI

RT-a M1 ES1

Path List

NH

PE2

BGP RR / PE2 detects

BGP session time-out

with PE1

2PE3 / PE4 invalidate

routes from PE1

3

PE1 experiences a

node failure (e.g.

power failure)

1BGP RR / PE3 detects

BGP session time-out

with PE1

2

PE2 reruns DF election.

Becomes DF for all

EVIs on segment

4

PE3 / PE4 will forward

M1 traffic towards PE2

6

BGP RR / PE4

detects BGP

session time-

out with PE1

2

PE2 adv. M1 MAC route

after CE traffic is

hashed towards PE2

5

PE1

BRKMPL-2333 47

EVPN-VPWS


Service Additional Capabilities

E-Line • All-active & single-active multi-homing support

• Both single-segment & multi-segment support

• Discovery & signaling via single protocol – BGP

EVPN-VPWS Does it Better than Legacy VPWS !!

BRKMPL-2333 49


EVPN BGP route type

Route type Usage EVPN EVPN VPWS

0x1 Ethernet Auto-Discovery

(A-D) Route

• MAC Mass-Withdraw

• Aliasing (load balancing)

• Split-Horizon

“Tagged with ESI Label Extended Community”

0x2 MAC Advertisement Route • Advertise MAC addresses

• Provide MAC / IP address bindings for ARP

broadcast suppression

“Tagged with MAC Mobility Extended

Community”

NOT used

0x3 Inclusive Multicast Route • Multicast tunnels used to transport

Broadcast, Multicast and Unknown Unicast

frames (BUM)

“Tagged with PMSI tunnel attribute” (P tunnel

type & ID) – RFC6514

NOT used

0x4 Ethernet Segment Route • Auto discovery of Multi-homed Ethernet

Segments, i.e. redundancy group discovery

• Designated Forwarder (DF) Election

“Tagged with ES-Import Extended Community”

BRKMPL-2333 50


EVPN BGP Extended Community

Attribute Usage Tagged BGP

route

EVPN EVPN VPWS

ESI label Extended

Community

• Split-Horizon for Ethernet

Segment.

• Indicate Redundancy Mode

(Single Active vs. All-Active)

Ethernet A-D

Route

ES-Import Extended

Community

• Limit the import scope of the

Ethernet Segment routes.

Ethernet

Segment Route

MAC Mobility Extended

Community

• E-VPN: Indicate that a MAC

address has moved from one

segment to another across PEs.

• PBB-EVPN: Signal C-MAC

address flush notification

MAC

Advertisement

Route

Not used

BRKMPL-2333 51


EVPN VPWS• Benefits of EVPN applied to point-to-point

services

• No signaling of PWs. Instead signals MP2P LSPs instead (ala L3VPN)

• All-active CE multi-homing (per-flow LB)

• Single-active CE multi-homing (per-service LB)

• Relies on a sub-set of EVPN routes to advertise Ethernet Segment and AC reachability

• PE discovery & signaling via a single protocol –BGP

• Per-EVI Ethernet Auto-Discovery route

• Handles double-sided provisioning with remote PE auto-discovery

• Under standardization: draft-ietf-bess-evpn-vpws

MPLS

PE1

CE1PE2

CE2

ES1 ES2

BGP Eth. Auto-

Discovery Route

EVPN NLRI

AC AC1 via PE1

Control-plane

attachment circuit

advertisement over the

Core

VPWS Service Config:

EVI = 100

Local AC ID = AC1

Remote AC ID = AC2


EVI = 100

Local AC ID = AC2

Remote AC ID = AC1

I have a P2P service that

needs to communicate

with the PE(s) that own

of AC = AC2

BRKMPL-2333 52


EVPN VPWS Operation – Single-homed

MPLS

PE1

CE1PE2

CE2

PE 1 Eth A-D Route

RD = RD-1a

ESI = ES1 (0)

Eth.Tag ID = AC1

Label (e.g. X)

RT ext. community

RT-a

PE 2 Eth A-D Route

RD = RD-2a

ESI = ES2 (0)

Eth.Tag ID = AC2

Label (e.g. Y)

RT ext. community

RT-a

PE1 RIB

VPN MAC ESI Eth.TAG

RT-a - 0 AC2

Path List

NH

PE2

ES1 ES2


given EVI

RD – RD unique per adv. PE

per EVI

MPLS Label – (downstream

assigned) used by remote

PEs to reach segment

ESI – 10 bytes ESI as specify

by EVPN Ethernet segment

IETF draft – zero for single-

homed


EVI = 100

Local AC ID = AC1

Remote AC ID = AC2


EVI = 100

Local AC ID = AC2

Remote AC ID = AC1

1

ES2 – Since CE2 is single

homed to PE2, ES2 = 0

Eth.Tag ID – 4-bytes local

AC-ID

PE2 RIB

VPN MAC ESI Eth.TAG

RT-a - 0 AC1

Path List

NH

PE1

2

3

4

5 6

BRKMPL-2333 53


EVPN VPWS Operation – Single-active

MPLS

PE1

CE1PE3

CE2

PE 1 Eth A-D Route

RD = RD-1a

ESI = ES1

Eth.Tag ID = AC1

Label (e.g. X)

RT ext. community

RT-a

PE 3 Eth A-D Route

RD = RD-2a

ESI = ES2 (0)

Eth.Tag ID = AC2

Label (e.g. Y)

RT ext. community

RT-a

PE1 & PE2 RIB

VPN MAC ESI Eth.TAG

RT-a - 0 AC2

Path List

NH

PE3

ES1

ES2


given EVI


per EVI






IETF draft


EVI = 100

Local AC ID = AC1

Remote AC ID = AC2


EVI = 100

Local AC ID = AC2

Remote AC ID = AC1

1




AC-ID

PE3 RIB

VPN MAC ESI Eth.TAG

RT-a - ES1

RT-a - ES1

RT-a - ES1 AC1

Path List

NH

PE1

PE2

PE1

2

3

4

5

6PE2

ES1VPWS Service Config:

EVI = 100

Local AC ID = AC1

Remote AC ID = AC2

Only one PE (PE1)

shows as next hop for

the remote ACSingle-Active == per-vlan

load-balancing CE-PEs

Two bundles on CE

device

BRKMPL-2333 54


EVPN VPWS Operation – All-active

MPLS

PE1

CE1PE3

CE2

PE 1 Eth A-D Route

RD = RD-1a

ESI = ES1

Eth.Tag ID = AC1

Label (e.g. X)

RT ext. community

RT-a

PE 3 Eth A-D Route

RD = RD-2a

ESI = ES2 (0)

Eth.Tag ID = AC2

Label (e.g. Y)

RT ext. community

RT-a

PE1 & PE2 RIB

VPN MAC ESI Eth.TAG

RT-a - 0 AC2

Path List

NH

PE3

ES1

ES2


given EVI


per EVI






IETF draft


EVI = 100

Local AC ID = AC1

Remote AC ID = AC2


EVI = 100

Local AC ID = AC2

Remote AC ID = AC1

1




AC-ID

2

3

4

5 PE2

ES1VPWS Service Config:

EVI = 100

Local AC ID = AC1

Remote AC ID = AC2PE3 RIB

VPN MAC ESI Eth.TAG

RT-a - ES1

RT-a - ES1

RT-a - ES1 AC1

Path List

NH

PE1

PE2

PE1,PE2

6

Both PEs (PE1/PE2)

shows as next hop for

the remote ACALL-Active == per-flow

load-baancing CE-PEs

Single bundle on CE

device

BRKMPL-2333 55

EVPN Deployment: DC Fabric Evolution w/ EVPN-VxLAN


L2 Fabric: Legacy VLAN, STP

• L2/L3 boundary: limited

mobility

• 4K VLAN: Limited scale

• Inefficient forwarding: STP

• Complex VLAN provisioning

• Vendor specific L2

enhancement

IP Fabric: VXLAN/EVPN, SDN

• Spine-leaf

• Virtual overlay across physical

boundary

• VXLAN: Ultra-high scale

• Efficient forwarding: L3 ECMPs

• EVPN control plane

• SDN enabled VXLAN and

service chaining provisioning

ASR9K)

The Evolution of the DC Fabric

L2 Fabric:

FP/Trill

IP Fabric:

VXLAN, DP Learning

BRKMPL-2333 57


DC Fabric – IP Underlay

Local LAN

Segment

Physical

Host

Local LAN

Segment

Physical

Host

Virtual Hosts

Local LAN

Segment

Virtual Switch

Edge Device

Edge Device

Edge Device

IP Interface

Edge device: could be

physical Leaf/ToR, or virtual

forwarder

BRKMPL-2333 58


DC Fabric – VXLAN Overlay

Local LAN

Segment

Physical

Host

Local LAN

Segment

Physical

Host

VTEP

VTEP

VTEP

VV

V

Encapsulation

Virtual Hosts

Local LAN

Segment

Virtual Switch

VTEP – VXLAN Tunnel End-Point

VNI/VNID – VXLAN Network Identifier

BRKMPL-2333 59


VXLAN Frame Format

MAC-in-IP Encapsulation

Un

de

rlay

Outer IP Header

Outer MAC Header

UDP Header

VXLAN Header

Original Layer-2 Frame Ove

rlay

14 Bytes

(4 Bytes Optional)

Ether Type

0x0800

VLAN ID

Tag

VLAN Type

0x8100

Src. MAC Address

Dest. MAC Address 48

48

16

16

16

20 Bytes

Dest. IP

Source IP

Header

Checksum

Protocol 0x11 (UDP)

IP Header

Misc. Data72

8

16

32

32

8 Bytes

Checksum 0x0000

UDP Length

VXLAN Port

Source

Port16

16

16

16

8 Bytes

Reserved

VNI

Reserved

VXLAN Flags

RRRRIRRR8

24

24

8

Src VTEP MAC Address

Next-Hop MAC Address

Src and Dst

addresses of

the VTEPs

VNI - allows

for 16M

possible

Segments

UDP 4789

Hash of the inner L2/L3/L4 headers of

the original frame.

Enables entropy for ECMP Load

balancing in the Network.

50 (

54)

Byte

s o

f O

ve

rhe

ad

BRKMPL-2333 60


VXLAN Evolution – BGP EVPN Control Plane

• Workload MAC / IP Addresses learnt by • Multi-Protocol BGP (MP-BGP) based

Control-Plane using EVPN NLRI

• Advertises Layer-2 & Layer-3 Address-to-VTEP Association

• Make Forwarding decisions at VTEPs for Layer-2 (MAC) and Layer-3 (IP), Integrated Route/Bridge (IRB)

• Reduces impact of ARP on the Network

• Standards Based• draft-ietf-bess-evpn-overlay

• draft-ietf-bess-evpn-inter-subnet-forwarding

• draft-ietf-bess-evpn-prefix-advertisement

RR RR

V2V1

V3

BGP Route-ReflectorRR

iBGP Adjacency

BRKMPL-2333 61


Use MP-BGP with EVPN Address Family on leaf nodes to distribute internal

host MAC/IP addresses, subnet routes and external reachability information

MP-BGP enhancements to carry up to 100s of thousands of routes with

reduced convergence time

BGP Update• Host-MAC• Host-IP• Internal IP Subnet• External Prefixes

MP-BGP for VXLAN EVPN Control PlaneEVPN Control Plane – Reachability DistributionEVPN route type 2: Host route, type 5: Subnet Route

LeafVTEPVTEPVTEPVTEP

Spine

BRKMPL-2333 62


MAC Host

IP

VNI VTEP

H-MAC-1 H-IP-

1

VNII-1 VTEP-1

EVPN Control Plane -- Host Advertisement

VTEP-2

VTEP-1

VTEP-3

H-MAC-1

H-IP-1

VLAN-1 /VNI-1

BGP Update:

H-MAC-1

H-IP-1

VTEP-1

VNI-1

BGP Update:

H-MAC-1

H-IP-1

VTEP-1

VNI-1

BGP Update:

H-MAC-1

H-IP-1

VTEP-1

VNI-1

Install host info to RIB/FIB:

H-MAC-1 MAC table

H-IP-1 VRF IP host table

Route

Reflector

Install host info to RIB/FIB:

H-MAC-1 MAC table

H-IP-1 VRF IP host table

Local learning of host info:

H-MAC-1 (MAC table)

H-IP-1 (VRF IP host table )

1

2

3

3

44

MAC Host

IP

VNI VTEP

H-MAC-1 H-IP-1 VNII-1 VTEP-1

MAC Host

IP

VNI VTEP

H-MAC-1 H-IP-1 VNII-1 VTEP-1

BRKMPL-2333 63


1. Host 1 moves to VTEP-3 from VTEP-1

VXLAN BGP Control Plane

Leaf

Spine

Host 1H-MAC-1

H-IP-1VLAN 10

VXLAN 5000

EVPN Control Plane --- Host Movement

VTEP-4VTEP-3VTEP-2VTEP-1

MAC IP VNI Next-Hop Encap Seq#

H-MAC-1 H-IP-1 5000 VTEP-3 VXLAN 1

2. VTEP-3 detects Host 1, sends MP-BGP update for Host 1 with its own VTEP address and a new seq #1

3. Other VTEPs learn about the new route of Host 1

NLRI:• Host H-MAC-1, H-IP-1 • NVE VTEP-3• VNI 5000

Ext. Community:• Encapsulation: VXLAN• Cost• Sequence number: 1

BRKMPL-2333

Host 1H-MAC-1

H-IP-1VLAN 10

VXLAN 5000

64


Inter-VXLAN Routing – EVPN IRB

Asymmetric

• Bridging & Routing on the ingress VTEP and bridging only on the egress VTEP

• Requires each VTEP to have all MAC addresses of their tenants in their ARP tables – can result in scale issue.

• Cisco follows Symmetric IRB

Symmetric

• Bridging & Routing on both the ingress and the egress VTEPs

• A VTEP only needs to maintain MACs for its directly attached endpoints. Optimal utilization of VTEP resources

Host 1

H-MAC-1

H-IP-1

VNI-A

VTEP-4VTEP-3VTEP-2VTEP-1

Host 2

H-MAC-2

H-IP-2

VNI-B

SVI A SVI B

IP Transport Network

Routing ?

BRKMPL-2333 65


MAC-VRF

MAC-VRF

66


Symmetric EVPN IRB (1)

• Routing on both ingress and egress

VTEPs

• Layer-3 VNI

• Tenant VPN indicator

• One per tenant VRF

• VTEP Router MAC

• Ingress VTEP routes packets onto the

Layer-3 VNI

• Egress VTEP routes packets to the

destination Layer-2 VNI

VTEP VTEP VTEP VTEP

Layer-3 VNI (VRF VNI)

Layer-2 VNI(Network VNI)

Layer-2 VNI(Network VNI)

BRKMPL-2333 67


Symmetric EVPN IRB (2)

Host 1

H-MAC-1

H-IP-1

VNI-A

VTEP-4Router MAC-4

VTEP-3VTEP-2VTEP-1

Router MAC-1

Host 2

H-MAC-2

H-IP-2

VNI-B

VNI

A

L3

VNI

VNI

B

L3

VNI

S-MAC: H-MAC-1D-MAC: GW-MAC

S-IP: H-IP-1D-IP: H-IP-2

Ingress VTEP

routes packets

from source VNI-A

to L3 VNI. D-MAC

in the inner header

is the egress

VTEP router MAC

1S-MAC: Router-MAC-1D-MAC: Router-MAC-4


S-IP: VTEP-1D-IP: VTEP-4

VNI: L3 VNI

S-MAC: MAC-VTEP4 local portD-MAC: H-MAC-2


Egress VTEP

routes packets

from L3 VNI to the

destination VNI-

B/VLAN.2

BRKMPL-2333 68


(Distributed) Anycast Gateway in with EVPN IRB

Host 1MAC1IP 1VLAN AVXLAN A

VTEPVTEPVTEPVTEP

SVI

GW IP

GW MAC

# VLAN to VNI mapping

vlan 200

vn-segment 5200

# Anycast Gateway MAC, identically configured on all VTEPs

fabric forwarding anycast-gateway-mac 0002.0002.0002

# Distributed IP Anycast Gateway (SVI)

# Gateway IP address needs to be identically configured on all

VTEPs

interface vlan 200

no shutdown

vrf member Tenant-A

ip address 20.0.0.1/24

fabric forwarding mode anycast-gateway

SVI

GW IP

GW MAC

SVI

GW IP

GW MAC

SVI

GW IP

GW MAC




The same anycast gateway virtual IP

address and MAC address are

configured on all VTEPs in the VNI.

BRKMPL-2333 69


EVPN IRB Anycast Gateway Deployment OptionsDistributed vs. Centralized Anycast Gateway

DC-1WAN

Leaf

VM1 VM2

DC-2

VM3 VM4

InternetBranch

IP-VPN

Internet

VPLS/PW/EVP

N

clien

t

DC Gateway

(L3 GW)

Leaf

(L2 only)

L3 EVPN

(per-VRF VNI)

IRB

IRB

IRBIRB

L2 only DC fabric L2/L3 DC fabric

Leaf

(IRB anycast

GW)

IRB IRB IRB IRB

IP-VPN

Internet

VPLS/PW/EVP

N

clien

t

DC Gateway

(IRB anycast

GW)

L2 EVPN

(per-BD VNI)

VPN

Client

Internet

Client

Centralized

Anycast GW

Distributed

Anycast GW

BRKMPL-2333 70

EVPN Deployment: DC Fabric and WAN Integration


The Solution Must be End-to-End

Leaf Leaf bLeaf bLeaf

DC

Spine

APIC

WAN/DCI

SDN-DC, VXLAN

overlay

SDN-WAN, MPLS/Segment Routing802.1q?

• Legacy 802.1q handoff

• It means multi-pathing is out the

door between DC & WAN

• VLANs and sub-interfaces

creation

• No policy level integration

• Small FIB/MAC table size on

border Leaf, create bottleneck

72


DC Gateway – Seamless DC and WAN Integration

Leaf Leaf bLeaf bLeaf

DC

Spine

APIC

WAN/DCI

Integration

Interworking

Scalable, Resilient, Optimized, End-to-End

• Common MP-BGP (EVPN AF) control plane

• VXLAN to MPLS data plane interworking

• SDN based auto provisioning: OpFlex,

APIC/VTS

• Integrated Policy control: WAN optimization

SDN-DC, VXLAN

overlay

SDN-WAN, MPLS/SR

• L3 Gateway

• L2 Gateway

• IRB with Anycast

Gateway

73


DC Gateway – L3 GatewayS-N Routing: all active

DC-1WAN

Leaf

Spine

DC

Gateway

VM1 VM2

DC-2

VM3 VM4

InternetBranch

VPN

Client

Internet

Client

IP-VPN

Internet

clien

t

Gateway

Leaf

(L3 anycast

GW)

L3 EVPN

(per-VRF VNI)

• EVPN/VXLAN to IP-VPN/MPLS interworking

• EVPN/VXLAN to global internet

• L3 EVPN between Leaf and GW (per-vrf VNI)

• Leaf is the L3 default gateway for VMs (with

EVPN IRB anycast GW) and does inter-vxlan

routing

L2/L3 DC fabric L2/L3 DC fabric

BRKMPL-2333 74


Gateway

DC Gateway – L2 GatewayL2 Stretch: E-W and S-Nall-active or single-active

DC-1WAN

Leaf

Spine

DC Gateway

VM1 VM2

DC-2

VM3 VM4

Branch

VPN

Client

VPLS/PW/EV

PN

clien

t

Gateway

Leaf

L2 EVPN

(per-BD VNI)

• L2 EVPN/VXLAN in the DC

• L2 DCI (E-W): EVPN/VXLAN with EVPN/VPLS

interworking

• L2 to client (S-N): EVPN/VXLAN with VPLS/PW

interworking

GatewayLeaf Leaf

L2 stretch: EVPN/VPLS (MPLS) L2 EVPN/VXLAN L2 EVPN/VXLAN

L2/L3 DC fabric L2/L3 DC fabric

BRKMPL-2333 75


DC Gateway – IRB Anycast GatewayIntegrated Routing and Bridging

DC-1WAN

Leaf

Spine

DC

Gateway

VM1 VM2

DC-2

VM3 VM4

InternetBranch IP-VPN

Internet

VPLS/PW/EVP

N

clien

t

Gateway

(L3 anycast GW)

Leaf

(L2 only)

L2 EVPN

(per-BD VNI)

• DC fabric is L2 only. All routing on DC gateway

• DC gateway is the L3 default gateway for VMs via

EVPN IRB anycast gateway

• Support both L2 and L3 for the same VNI at the

same time

Gateway GatewayLeaf Leaf

L2 stretch: EVPN/VPLS (MPLS) L2 EVPN/VXLAN L2 EVPN/VXLAN

IRB

IRB IRB IRB

L2 only DC fabric L2 only DC fabric

VPN

Client

Internet

Client

76


EVPN IRB Anycast Gateway Deployment OptionsDistributed vs. Centralized Anycast Gateway

DC-1WAN

Leaf

VM1 VM2

DC-2

VM3 VM4

InternetBranch

IP-VPN

Internet

VPLS/PW/EVP

N

clien

t

DC Gateway

(L3 GW)

Leaf

(L2 only)

L3 EVPN

(per-VRF VNI)

IRB

IRB

IRBIRB

L2 only DC fabric L2/L3 DC fabric

Leaf

(IRB anycast GW)

IRB IRB IRB IRB

IP-VPN

Internet

VPLS/PW/EVP

N

clien

t

DC Gateway

(IRB anycast GW)

L2 EVPN

(per-BD VNI)

VPN

Client

Internet

Client

Centralized

Anycast GW

Distributed

Anycast GW 77


IRB Anycast Gateway Deployment Option Comparison

Option 1

Distributed IRB Anycast Gateway

Option 2

Centralized IRB Anycast Gateway

DC Gateway Router • L2 or L3 gateway function

• Doesn’t require IRB function

• EVPN IRB for integrated L2 and L3

• IRB anycast gateway for VM’s default

gateway

Leaf • EVPN IRB for integrated L2 and L3

• IRB anycast gateway for VM’s

default gateway

• L2 EVPN peering across DC

• Cross-DC underlay IP routing is required

Pros • Optimized E-W inter-vxlan routing • Simple DC fabric design: L2 only

• Large ARP table on the DC gateway

router

Cons • EVPN IRB function across all Leaf

nodes

• Require both L2 and L3 function on

the leaf

• Sub-optimal E-W inter-vxlan routing

BRKMPL-2333 78


DC Gateway – the Policy Integration and Auto-Provisioning: Application-engineered Routing

DC-1WAN

Leaf

Spine

DC

Gateway

VM1 VM2

DC-2

VM3 VM4

High bandwidth flowLow latency flow

DC policy domain DC policy domainWAN policy domain

WAN Segment

RoutingVTS/APIC VTS/APIC

BRKMPL-2333

PBTS: steering packet to

SR-TE in the WAN

DC: classification

and marking

79


Example: Application has a preference for disjoint path in dual-plane

WAN networks

Segment

Routing

WAN

with WAE

ACI Fabric

Po

licy

WEBWEB

Customer has the requirement that traffic from

applications RED and BLUE should be

transported across disjoint paths in the WAN.

- Policy expressed on APIC and delivered by

SR-enabled +WAE WAN

BRKMPL-2333 80


Segment

Routing

WAN

with WAE

Example: Application requires the lowest latency path in the WAN

ACI Fabric

Po

licy

WEBWEB

WAN has cheap capacity via US with higher latency.

Scarce, expensive capacity via Russia, with lower

latency.

Customer identify the applications that require the

lowest possible latency path on APIC, integration

steers traffic on the path via Russia.

Tokyo

RussiaUS

Brussels

BRKMPL-2333 81

Summary


EVPN: Next Generation VPN

E-LAN(MP2MP

L2VPN)

E-LINE(P2P

L2VPN)

E-TREE(P2MP

L2VPN)

EVPN

VPWS

(PBB-)

EVPN

EVPN

DC Fabric(IntraDC

Overlay)

IRB(L2/L3

Overlay)

DCI(InterDC)

IP-VPN(L3VPN)

EVPN

DCI

EVPN-

IRB

EVPN-

Overlay

EVPN

ETREE

EVPN-

L3

VPLS PW 4364VPLS-

ETREE VPLS,OTV

BRKMPL-2333 83


References• [FabricPath]: FabricPath http://www.cisco.com/en/US/prod/switches/ps9441/fabric_path_promo.html

• [LISP]: Locator/ID Separation Protocol https://datatracker.ietf.org/wg/lisp/charter/

• [802.1Qbp] ECMP http://www.ieee802.org/1/files/public/docs2011/new-ashwood-sajassi-ecmp-par-0111-v04.pdf

• [EVPN]: BGP MPLS Based Ethernet VPN http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-04

• [TRILL]: Transparent Interconnection of Lots of Links https://datatracker.ietf.org/wg/trill/charter/http://tools.ietf.org/wg/trill/draft-ietf-trill-rbridge-protocol/

• [VL2]: VL2: A Scalable and Flexible Data Center Network http://ccr.sigcomm.org/online/?q=node/502

• [MOOSE]: Addressing the Scalability of Ethernet with MOOSE http://www.cl.cam.ac.uk/~mas90/MOOSE/MOOSE.pdf

• [PORTLAND]: PortLand: A Scalable Fault-Tolerant Layer 2 Data Center Network Fabric http://ccr.sigcomm.org/online/?q=node/503

• [SEATTLE]: Floodless in SEATTLE: A Scalable Ethernet Architecture for Large Enterprises http://www.cs.princeton.edu/~chkim/Research/SEATTLE/seattle.pdf

• [MONSOON]: Towards a Next Generation Data Center Architecture: Scalability and Commoditization http://research.microsoft.com/apps/pubs/default.aspx?id=79348

• [VLB]: Valiant Load Balancing in Backbone Networks http://www.stanford.edu/~ashishg/network-algorithms/rui.pdf

BRKMPL-2333 84

http://www.cisco.com/en/US/prod/switches/ps9441/fabric_path_promo.html

https://datatracker.ietf.org/wg/lisp/charter/

http://www.ieee802.org/1/files/public/docs2011/new-ashwood-sajassi-ecmp-par-0111-v04.pdf

http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-04

https://datatracker.ietf.org/wg/trill/charter/

http://tools.ietf.org/wg/trill/draft-ietf-trill-rbridge-protocol/

http://ccr.sigcomm.org/online/?q=node/502

http://www.cl.cam.ac.uk/~mas90/MOOSE/MOOSE.pdf

http://ccr.sigcomm.org/online/?q=node/503

http://www.cs.princeton.edu/~chkim/Research/SEATTLE/seattle.pdf

http://research.microsoft.com/apps/pubs/default.aspx?id=79348

http://www.stanford.edu/~ashishg/network-algorithms/rui.pdf


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

BRKMPL-2333 85

CiscoLive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/us

http://ciscolive.com/us


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKMPL-2333 86

Thank you

Please join us for the Service Provider Innovation Talk featuring:

Yvette Kanouff | Senior Vice President and General Manager, SP Business

Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

11:30 am - 12:30 pm, In the Oceanside A room

What to expect from this innovation talk

• Insights on market trends and forecasts

• Preview of key technologies and capabilities

• Innovative demonstrations of the latest and greatest products

• Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com

Appendix-A: PBB-EVPNStartup Sequence & A Day in Life of a Paket


PBB-EVPN Startup Sequence


ESI and B-MAC Auto-Sensing


VPN Auto-Discovery

Multicast Tunnel ID / Endpoint

Discovery

Backbone MAC (B-MAC) Reachability Advertisement

BRKMPL-2333 90


PBB-EVPN Startup Sequence (cont.)ESI and B-MAC Auto-Sensing



MPLS

PE1

CE1

PE2

PE3

CE3

PE4

LACP PDU

exchange

Source B-MAC used at PBB-EVPN PE on a

given ESI can be auto-generated* from CE’s

LACP information -> CE’s LACP System ID

MAC with U/L** (Universal / Locally

Administered) bit flipped

Example: 0211.0022.0033

CE LACP info:

LACP System ID (MAC) (6B)

e.g. 0011.0022.0033

LACP System Priority (2B)

e.g. 0000

LACP Port Key (2B)

e.g. 0018

ESI (10B) can be auto-generated*

from CE’s LACP information ->

concatenation of CE’s LACP

System Priority + Sys ID + Port Key

Example:

0000. 0011.0022.0033.0018

(*) ESI and B-MAC can also be manually configured

(**) U/L is second-least-significant bit of most significant byte

System

Priority

2 bytes 6 bytes 2 bytes

System MAC

AddressPort Key

B-MAC

B-MAC

B-MAC

B-MAC

BRKMPL-2333 91


PBB-EVPN Startup Sequence (cont.)BGP Ethernet Segment Route

MPLS

PE1

CE1

PE2

PE3

CE3

PE4


RD = RD10

ESI = ESI1

ES-Import ext. comm.

e.g. 0011.0022.0033

MAC address portion

of ESI (6B)


RD = RD20

ESI = ESI1

ES-Import ext. comm.

e.g. 0011.0022.0033





advertising PE

BRKMPL-2333 92


PBB-EVPN Startup SequenceDesignated Forwarder (DF) Election*

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

Ordered List of discovered PEs

starting from zero (lowest IP add)




PE Ordered List

Position PE

0 PE1

1 PE2

Modulo Operation

I-SID

I-SID mod N

(N = # of PEs)

(e.g. I-SID mod 2)

100 0

101 1

102 0

103 1

PE Ordered List

Position PE

0 PE1

1 PE2

Modulo Operation

I-SID (I-SID mod 2)

100 0

101 1

102 0

103 1

Exchange of Ethernet

Segment Routes

Result of modulo

operation is used to

determine DF and

BDF status

DF – Designated Forwarder

BDF – Backup Designated Forwarder

I-SID – PBB 24-bit Service Instance ID

Example:

PE2 DF for I-SIDs 101, 103

PE2 BDF for I-SIDs 100, 102

Example:

PE1 DF for I-SIDs 100, 102

PE1 BDF for I-SIDs 101, 103

(*) DF election with Service Carving shown (i.e. one DF per I-SID in the segment)BRKMPL-2333 93


PBB-EVPN Startup Sequence (cont.)BGP MAC Advertisement Route (B-MAC)




Backbone MAC (B-MAC) Reachability Advertisement

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

PE1 MAC Route

RD = RD-1a

ESI = MAX_ESI

MAC = B-M1

Label = L1

RT ext. community

RT-a

MP2P VPN Label –



traffic to advertised (MAC,EVI)B-MAC advertised

by route

PE3 / PE4 RIB

VPN MAC ESI

RT-a B-M1 n/a

Path List

NH

PE1

PE2

PE2 MAC Route

RD = RD-2a

ESI = 1

MAC = B-M1

Label = L2

RT ext. community

RT-a


advertising PE per EVI

ESI – reserved ESI

indicates advertised

MAC is a B-MAC

B-M1

B-M1

B-M2

B-M2

94


PBB-EVPN Startup SequenceBGP Inclusive Multicast Route

VPN Auto-Discovery

Multicast Tunnel ID / Endpoint

Discovery


RD = RD-1a



Label (e.g. L1)

RT ext. community

RT-a

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

Tunnel Type – Ingress

Replication or P2MP LSP



downstream assigned (ing.

repl.) or upstream assigned

(Aggregate Inclusive P2MP

LSP1)

PMSI - P-Multicast Service Interface

BUM – Broadcast / Unknown Unicast / Multicast


adv. PE per EVI


given EVI


RD = RD-2a



Label (e.g. L2)

RT ext. community

RT-a(1) Mcast MPLS label is not set for Inclusive Trees (P2MP LSP)

BRKMPL-2333 95


MPLS

PE1

CE1

PE2

PE3

CE3

PE4

Life of a PacketIngress Replication – Multi-destination Traffic Forwarding

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

VID 100

SMAC: M1

DMAC: F.F.F

Mcast MPLS

Label assigned by

PE3 for incoming

BUM traffic on a

given EVI

PSN MPLS label

to reach PE3

PE4 – non-DF for

given I-SID drops

BUM traffic PE2 – drops BUM

traffic originated

on same source

B-MAC (B-M1)



adds PBB encapsulation

and forwards it using

ingress replication – 3

copies created PE3 – as DF, it

forwards BUM

traffic towards

segment


Route

RD = RD-2a


Tunnel Type = Ing. Repl.

Label = L2

RT ext. community

RT-a






PE1, PE2, PE3, PE4 sent

Inclusive Multicast route

which include Mcast label

B-M1

B-M1

B-M2

B-M2

B-M1

B-M1

B-M2

B-M2

L2 PBB

L3 PBB

L4 PBB

PE3 MAC Table

I-SID xyz

C-MAC B-MAC

M1 B-M1

Data-plane based

MAC learning for

C-MAC / B-MAC

associationBRKMPL-2333


MPLS

PE1

CE1

PE2

PE3

CE3

PE4

Life of a PacketInclusive Trees – Multi-destination Traffic Forwarding

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

VID 100

SMAC: M1

DMAC: F.F.F

P2MP LSP sourced

at PE1

Inclusive (per-ISID)

trees

PE4 – non-DF for

given I-SID drops

BUM traffic PE2 – drops BUM

traffic originated

on same source

B-MAC (B-M1)



adds PBB encapsulation

and forwards it using an

Inclusive TreePE3 – as DF, it

forwards BUM

traffic towards

segment


Route

RD = RD-2a


Tunnel Type = P2MP

Label = 0

RT ext. community

RT-a

Mcast MPLS Label – not

assigned for Inclusive Trees


PE1, PE2, PE3, PE4 sent

Inclusive Multicast route

B-M1

B-M1

B-M2

B-M2

B-M1

B-M1

B-M2

B-M2

PE3 MAC Table

I-SID xyz

C-MAC B-MAC

M1 B-M1

Data-plane based

MAC learning for

C-MAC / B-MAC

association

PBB

PBB

PBB

BRKMPL-2333


MPLS

PE1

CE1

PE2

PE3

CE3

PE4

Life of a Packet (cont.)Unicast Traffic Forwarding

PE1 MAC Route

RD = RD-1a

ESI = 1

MAC = B-M1

Label = L1

RT ext. community

RT-a

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

MP2P VPN Label

assigned by PE1


for the target EVI

PSN MPLS label

to reach PE1

PE3 forwards traffic

destined to M1

using B-MAC B-M1

towards PE1

PE3 RIB

VPN MAC ESI

RT-a B-M1 n/a

Path List

NH

PE1

PE2

VID 100

SMAC: M2

DMAC: M1

VID 100

SMAC: M1

DMAC: F.F.F

MP2P VPN Label –




MAC advertised

by route

B-M1

B-M1

B-M2

B-M2

B-M1

B-M1

B-M2

B-M2

L1 PBB

PE3 MAC Table

I-SID xyz

C-MAC B-MAC

M1 B-M1

Data-plane based

MAC learning for

C-MAC / B-MAC

association


PE1 & PE2 advertised

MAC routes for B-MAC

(B-M1)

BRKMPL-2333 98


MPLS

PE1

CE1

PE2

PE3

CE3

PE4

Life of a Packet (cont.)Unicast Traffic Forwarding and Aliasing

PE1 MAC Route

RD = RD-1a

ESI = 1

MAC = B-M1

Label = L1

RT ext. community

RT-a

PE3 RIB

VPN MAC ESI

RT-a B-M1 n/a

Path List

NH

PE1

PE2

VID 100

SMAC: M1

DMAC: F.F.F

MP2P VPN Label –




MAC advertised

by route

B-M1

B-M1

B-M2

B-M2

PE3 MAC Table

I-SID xyz

C-MAC B-MAC

M1 B-M1


PE1 & PE2 advertised

MAC route for B-MAC (B-

M1)

PE2 MAC Route

RD = RD-2a

ESI = 1

MAC = B-M1

Label = L2

RT ext. community

RT-a

MPLS

PE1

CE1

PE2

PE3

CE3

PE4

MP2P VPN Label

assigned by PE2


for target EVI

PSN MPLS label

to reach PE2


on a flow (flow 2) to

M1 using B-MAC B-

M1 towards PE2

VID 100

SMAC: M4

DMAC: M1

MP2P VPN

Label

assigned by

PE1 for

incoming traffic

for target EVI

PSN MPLS label

to reach PE1


on a flow (flow 1) to

M1 using B-MAC B-

M1 towards PE1

VID 100

SMAC: M3

DMAC: M1

B-M1

B-M1

B-M2

B-M2

L2 PBB

L1 PBB

Data-plane based

MAC learning for C-

MAC / B-MAC

association

BRKMPL-2333 99


Life of a Packet (cont.)Active / Active Load Balancing from CE

MPLS

PE1

CE1

PE2

PE3

CE3

PE 1 / PE2 RIB

VPN MAC ESI

RT-a B-M3 0

Path List

NH

PE3

MPLS

PE1

CE1

PE2

PE3

CE3

VID 100

SMAC: M1

DMAC: M3

VID 100

SMAC: M2

DMAC: M3

MP2P VPN Label

assigned by PE3


for target EVI

PSN MPLS label

to reach PE3

PE1 forwards traffic to

M3 using B-MAC B-M3

towards PE3


M3 using B-MAC B-M3

towards PE3

PE3 MAC Route

RD = RD-3a

ESI = 0

MAC = B-M3

Label = L3

RT ext. community

RT-a

MP2P VPN Label –



traffic to advertised MACMAC advertised

by route

ESI == 0 used for

Single Home Device

B-M1

B-M1

B-M3

PE1 / PE2 MAC Table

I-SID xyz

C-MAC B-MAC

M3 B-M3

B-M1

B-M1

B-M3

L3 PBB

L3 PBB

BRKMPL-2333 100


MPLS

PE1

PE2

PE3

CE3

PE4

Life of a Packet (cont.)Active / Active Per-Service Load Balancing

MPLS

PE3

CE3

PE4

PE3, PE4 RIB

VPN MAC ESI

RT-a B-M1 n/a

RT-a B-M2 n/a

Path List

NH

PE1

PE2

MP2P VPN Label

assigned by PE1 for

incoming traffic for

target EVI


M1 using B-MAC B-M1

towards PE1

VID 100 I-SID 100

SMAC: M3

DMAC: M1

VID 100

SMAC: M1

DMAC: M3

During startup, PE2

advertises:

• Ethernet Segment route

• MAC Route for B-MAC B-

M2

PE2 elected DF for I-SID

200

CE1 configured with two

(2) separate bundles

towards PEs

PE1

CE1

PE2

During startup, PE1

advertises:

• Ethernet Segment route

• MAC Route for B-MAC B-

M1

PE1 elected DF for I-SID

100

PE1 MAC Route

RD = RD-1a

ESI = 1

MAC = B-M1

Label = L1

RT ext. community

RT-a

B-M1

B-M2

B-M3

B-M3 PE3 / PE4 MAC Table

I-SID 100

C-MAC B-MAC

M1 B-M1

VID 200

SMAC: M11

DMAC: M33

PE3 / PE4 MAC Table

I-SID 200

C-MAC B-MAC

M11 B-M2

L1 PBBCE1

B-M1

B-M2

B-M3

B-M3

VID 200 ISD 200

SMAC: M4

DMAC: M11

L2 PBB

BRKMPL-2333 101

next generation l2 vpn -...

Documents