• EVPN Overview
• EVPN Technology Prime
• EVPN Startup Sequence
• EVPN Operation
• A Day in Life of a Packet
• EVPN-VPWS
• EVPN Deployment: DC Fabric Evolution with EVPN-VXLAN
• EVPN Deployment: DC Fabric and WAN Integration
Agenda
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
L2VPN Technologies Evolution
802.1D
VLAN Scale
Native L2 Bridging Technologies
802.1Q 802.1ad(QinQ)
802.1ah(MACinMAC)
802.1aq(SPB)
802.1Qbp(ECMP)
More VLAN
ScaleMAC Scale
Shortest
Path FwdECMP
• What about Inter Domain (WAN) Connectivity?
• What about IP or MPLS fabric?
• What about industry traction & multi-vendor interop?
• What about All-Active multi-homing?
• What about multi-pathing (not ECMP)?
IEEE 802.1Qbp
Large # of VLANs – 16 millions ✔
Large # of MACs – MAC-in-MAC ✔
Optimum Forwarding ✔
ECMP ✔
BRKMPL-2333 5
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
L2VPN Technologies Evolution
VPLS
PW Scale
L2 VPN Technologies
H-VPLS PBB-VPLS EVPN(RFC 7432)
MAC Scale
To Address
all major
shortcomings
• Inter Domain (WAN) Connectivity? Yes
• IP or MPLS fabric? Yes
• Industry traction & Multi-vendor interop? Yes
• All-Active multi-homing? Yes
• Multi-pathing (not ECMP)? Yes
PBB-EVPN
EVPN-VxLAN
EVPN-IRB
EVPN-VPWS
…
BRKMPL-2333 6
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What’s the big deal about EVPN?EVPN is next generation all-in-one VPN solution
It not only does the job of many other VPN technologies but it does it better !!
E-LAN(MP2MP
L2VPN)
E-LINE(P2P
L2VPN)
E-TREE(P2MP
L2VPN)
L3VPN
EVPN
VPWS
PBB-
EVPN
EVPN
DC Fabric(IntraDC
Overlay)
IRB(L2/L3
Overlay)
DCI(InterDC)
EVPN-
IRB
EVPN-
Overlay
EVPN-
L3VPN
EVPN
ETREE
EVPN-
DCI
VPLS PW 4364VPLS-
ETREEVxLAN
TRILLVPLS,OTV
BRKMPL-2333 7
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
2006 2010 2011 2013 2015
- OPEN project was started at
Cisco
- OPEN = Optimum Ethernet
Network
- Introduced to IETF as
Routed-VPLS
- Merged with Juniper’s
MAC-VPN and was
introduced an EVPN
Following drafts were
introduced:
- EVPN
- PBB-EVPN
- EVPN-VPWS
- EVPN-Overlay
- EVPN-ETREE
Following drafts were
introduced
- EVPN IRB
- EVPN DCI
Enhancements
- Virtual ES
- Optimized ingress replication
- IGMP aggregation between
PODs
- mcast tunnels between DCs
- Inter-AS for IRB
- L3VPN multi-homing
BRKMPL-2333 8
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN in a Nut ShellMAC learning in control plane (via BGP)
IP or MPLS
PE1
CE1
PE2
PE3
CE3
PE4
C-MAC:
M1
Single active
multi-homingAll active multi-
homing
MAC Routing: Control plane (BGP)
advertise the learnt MACs from CE
Data Plane: IP or MPLS,
flexible
Optimum forwarding,
ECMP, Multi-pathing
Common L2/L3 VPN
Operational Mode
Flexible Policy Control
Consolidated VPN
service with x-EVPN
BRKMPL-2333 9
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Additional Capabilities
E-LAN • Provides All-Active multi-homing
• Prevents loop for both all-active & single-active even in transient
state
• Efficient utilization of network cross-sectional bandwidth (via
optimum forwarding, ECMP, multi-pathing on a per flow basis)
• Flexible policy control per MAC and per Site
EVPN Does it Better than VPLS !!
BRKMPL-2333 11
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
MAC1 MAC2CE1 CE2
• Packets originated from MAC2 arrives at both PE3 and PE4 which get forwarded subsequently to PE1. PE1 keeps flip/flopping between PE3 and PE4 for learning of MAC2!!
• VPLS cannot do proper load-balancing because doesn’t support Aliasing. When PE3 wants to forward a packet with destination address MAC1, it needs to send it to both PE1 and PE2 even though it only learned MAC1 from PE1.
PE1
PE2
PE3
PE4
MAC1 MAC2PE1
PE2
PE3
PE4
CE1 CE2
Flip/flopping !
Load balancing
BRKMPL-2333
VPLS cannot provide All-Active Multi-Homing Because:
MAC1 MAC2PE1
PE2
PE3
PE4
CE1 CE2
Echo !
• BUM packets forwarded from PE1 can get loopback to the originating CE
12
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN All-Active Multi-Homing Principles
MAC1 MAC2
MAC1 MAC2PE1
PE2
PE3
PE4
CE1 CE2
CE1 CE2
• ARP broadcast packet doesn’t get loopback to the originating CE device. Split-horizon
• Either PE3 or PE4 forward the broadcast frame to the far-end dual-homed device CE2. DF selection
• When PE1 & PE2 forward traffic for MAC1, there is no flip/fopping on PE3 because of MAC learning in control plane.
• When PE3 wants to forward a packet with destination address MAC1, it needs to send it to both PE1 and PE2 even though it only learned MAC1 from PE1. Load balancing via aliasing
PE1
PE2
PE3
PE4
MAC1 MAC2PE1
PE2
PE3
PE4
CE1 CE2
Echo !
Duplicate !
Load balancing
BRKMPL-2333 13
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN Efficient Cross-Sectional BW Utilizaiton
• EVPN provides per-flow load-balancing among egress PEs using BGP multi-pathing
• Per-flow load balancing between ingress and egress PEs are provided using IGP ECMP (ingress PE still needs to add entropy field in the packet).
P
E
P
E
P
E
P
P
P
P
Flow Based Multi-Pathing in the CoreVlan X -
F1Vlan X –
F2Vlan X –
F3Vlan X –
F4
P
E
P
E
P
E
Flow Based Load-balancing – PE to PE direction
Vlan X -
F1Vlan X
– F2
BRKMPL-2333 14
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
E-VPN Concepts
Ethernet Segment
• Represents a ‘site’
connected to one or more
PEs
• Uniquely identified by a 10-
byte global Ethernet
Segment Identifier (ESI)
• Could be a single device or
an entire network
Single-Homed Device (SHD)
Multi-Homed Device (MHD)
Single-Homed Network (SHN)
Multi-Homed Network (MHN)
BGP Routes
• E-VPN and PBB-EVPN
define a single new BGP
NLRI used to carry all E-
VPN routes
• NLRI has a new SAFI 70
(EVPN), AFI 25 (L2VPN)
• Routes serve control plane
purposes, including:
MAC address reachability
MAC mass withdrawal
Split-Horizon label adv.
Aliasing
Multicast endpoint discovery
Redundancy group discovery
Designated forwarder election
E-VPN Instance (EVI) & MAC-VRF
• EVI identifies a VPN in the
network
• Encompass one or more
bridge-domains, depending
on service interface type
Port-based
VLAN-based (shown above)
VLAN-bundling
VLAN aware bundling (NEW)
BGP Route Attributes
• New BGP extended
communities defined
• Expand information carried
in BGP routes, including:
MAC address moves
C-MAC flush notification
Redundancy mode
MAC / IP bindings of a GW
Split-horizon label encoding
PE
MAC-VRF
MAC-VRF
PE1
PE2
CE1
CE2
SHD
MHD
ESI1
ESI2
Route Types
[1] Ethernet Auto-Discovery (AD) Route
[2] MAC Advertisement Route
[3] Inclusive Multicast Route
[4] Ethernet Segment Route
[5] IP Prefix Route
Extended Communities
ESI MPLS Label
ES-Import
MAC Mobility
Default Gateway
BRKMPL-2333 15
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN Route Types & Benefits
Route Type Usage Benefits
Ethernet A-D Route
(Type 1)
• Aliasing
• Mass Withdraw of addresses
• SH/AA MH Indication
• Advertising Split-Horizon Label
• Loop avoidance – even
transient
• Fast convergence
• Efficient load balancing
• Per-site policy
MAC/IP Advertisement Route
(Type 2)
• Advertise MAC (and IP) reachability
• Advertise MAC/IP binding
• MAC mobility
• Per MAC policy
• ARP suppression
• Workload Mobility
Inclusive Multicast Route
(Type 3)
• Auto discovery of multicast tunnel
endpoints & mcast tunnel type
• Support multicast even
when core doesn’t
Ethernet Segment Route
(Type 4)
• Auto discovery of redundancy group • A/A and S/A MHD &
MHN support
IP Prefix Route
(Type 5)
• IP Prefix advertisement (not for IP
host advertisement)
• IP route aggregation
• Interop w/ L3VPNBRKMPL-2333 16
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN BGP Routes RFC7432
• EVPN defines a new BGP NLRI used to carry all EVPN routes
• BGP Capabilities Advertisement used to ensure that two speakers support EVPN NLRI (per RFC4760)
• AFI 25: L2VPN, SAFI 70: EVPN
Route Type
Length
Route type specific
1 byte
1 byte
[1] Ethernet Auto-Discovery (AD) Route
[2] MAC Advertisement Route
[3] Inclusive Multicast Route
[4] Ethernet Segment Route
[5] IP Prefix Route
EVPN NLRI
Variable
BRKMPL-2333 17
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN Startup Sequence
Segment Auto-Discovery
ESI Auto-Sensing
Redundancy Group Membership Auto-Discovery
VPN Auto-Discovery
Multicast Tunnel Endpoint
Discovery
DF Election & VLAN Carving
ESI Label & MH type Discovery
BRKMPL-2333 19
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
ESI Auto-Sensing
Segment Auto-Discovery
ESI Auto-Sensing3
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
LACP PDU
exchangeCE LACP info:
LACP System ID (MAC) (6B)
e.g. 0011.0022.0033
LACP System Priority (2B)
e.g. 0000
LACP Port Key (2B)
e.g. 0018
ESI (10B) can be auto-generated1
from CE’s LACP information ->
concatenation of CE’s LACP
System Priority + Sys ID + Port Key
Example:
0000. 0011.0022.0033.0018
System
Priority
2 bytes 6 bytes 2 bytes
System MAC
AddressPort Key
BRKMPL-2333
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKMPL-2333
MPLS
PE1
CE1
PE2
PE3
PE4
PE 1 Eth Segment Route
RD = RD10
ESI = ESI1
ES-Import Route Target
e.g. 0011.0022.0033
MAC address portion
of ESI (6B)
PE 2 Eth Segment Route
RD = RD20
ESI = ESI1
ES-Import Route Target
e.g. 0011.0022.0033
Segment Auto-Discovery
ESI Auto-Sensing
Redundancy Group Membership Auto-Discovery
RD – RD unique per
adv. PE
Redundancy Group Membership Auto-Discovery
PE4
PE1000
Exchange of Ethernet
Segment Routes
21
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
Ordered List of discovered PEs
starting from zero (lowest IP add)
Segment Auto-Discovery
ESI Auto-Sensing
PE Ordered List
Position PE
0 PE1
1 PE2
Modulo Operation
VID
VID mod N
(N = # of PEs)
(e.g. VID mod 2)
100 0
101 1
102 0
103 1
PE Ordered List
Position PE
0 PE1
1 PE2
Modulo Operation
VID (VID mod 2)
100 0
101 1
102 0
103 1
Result of modulo
operation is used to
determine DF and
BDF status
Example:
PE2 DF for VIDs 101, 103
PE2 BDF for VIDs 100, 102
Example:
PE1 DF for VIDs 100, 102
PE1 BDF for VIDs 101, 103
DF Election & VLAN Carving
Redundancy Group Membership Auto-Discovery
DF Election & VLAN Carving
BRKMPL-2333 22
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKMPL-2333
Segment Auto-Discovery
ESI Auto-Sensing
Redundancy Group Membership Auto-Discovery
ESI Label & MH type Discovery
MPLS
PE1
CE1
PE2
PE3
PE4
PE1 Eth A-D per ES
RD = RD-1a
ESI1
Eth Tag = MAX-ET
Label = 0
ESI Label ext. com
L1
DF Election and VLAN Carving
ESI Label & MH type Discovery
ESI1
PE2 Eth A-D per ES
RD = RD-1b
ESI1
Eth Tag = MAX-ET
Label = 0
ESI Label ext. com
L2 23
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VPN Auto-Discovery
Multicast Tunnel Endpoint
Discovery
PE 1 Inclusive Multicast Route
RD = RD-1a
PMSI Tunnel Attribute
Tunnel Type (e.g. Ing. Repl.)
Label (e.g. L1)
RT ext. community
RT-a
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
Tunnel Type – Ingress
Replication or P2MP LSP
Mcast MPLS Label – used to
transmit BUM traffic -
downstream assigned (ing.
repl.) or upstream assigned
(Aggregate Inclusive P2MP
LSP2)
PMSI - P-Multicast Service Interface
BUM – Broadcast / Unknown Unicast / Multicast
RD – RD unique per
adv. PE per EVI
RT – RT associated with a
given EVI
PE 2 Inclusive Multicast Route
RD = RD-2a
PMSI Tunnel Attribute
Tunnel Type (e.g. Ing. Repl.)
Label (e.g. L2)
RT ext. community
RT-a
Multicast Tunnel Endpoint Discovery
BRKMPL-2333 24
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN BGP route 0x4 - Ethernet Segment Route
• Usage:• Auto-discovery of multi-homed Ethernet Segments• Designated Forwarder election
• Tagged with ES-Import Extended Community
• PEs apply route filtering based on ES-Import community. Thus, Ethernet Segment route is imported only by the PEs that are multi-homed to the same Ethernet segment
RD
Ethernet Segment Identifier
8 bytes
10 bytes
Unique per Advertising PE
ESI of Ethernet Segment
Route Type specific encoding of E-VPN NLRI
IP Address Length
Originating Router’s IP add.
1 bytes
4 or 16 bytes
IP address length
IPv4 or IPv6 address
BRKMPL-2333 25
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
ES-Import RT Extended Community
Usage:
• Sent with Ethernet Segment route
• Limits the scope of Ethernet Segment routes distribution to PEs connected to the same multi-homed Segment
0x02
ES-Import6 bytes
MAC Address portion of the ESI
0x06
BRKMPL-2333 26
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN BGP Route 0x1 – Ethernet Auto-discovery Route
This route has two flavors:
Per-ES Ethernet A-D route
• Advertise the Split-Horizon Label associated with an Ethernet Segment
• For AA or SA MH indication
• Used for MAC Mass-Withdraw
Per-EVI Ethernet A-D route
• Advertise VPN label used for Aliasing or Backup-Path
RD
Ethernet Segment Identifier
Ethernet Tag ID
8 bytes
10 bytes
3 bytes MPLS Label
Unique per Advertising PE per EVI
ESI of Ethernet Segment
Set to VLAN or I-SID for VLAN-Aware
Bundling Service interface, otherwise 0
VPN (Aliasing) Label per (ESI,
Ethernet Tag)
Unique per Advertising PE
ESI of Ethernet Segment
MUST be set to 0
MUST be set to MAX-ET
(0xFFFFFFFF)4 bytes
BRKMPL-2333 27
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
ESI Label extended community
Usage:
• Sent with Ethernet AD Route per ES
• Advertises the Split-Horizon Label for the Ethernet Segment
• Indicates the Redundancy Mode: Single Active vs. All-Active
0x01
Flags
Reserved
ESI MPLS Label
Bit 0: Redundancy Mode
(single active vs. all active)
Ethernet Segment Split-
Horizon Label
0x06
Set to 0
ESI-1
MAC1 MAC2PE1
PE2
PE3
PE4
Agg1 Agg2
ESI-2
PE1 advertises in BGP a split-horizon label associated
with the ESI-1 (in the Ethernet AD route)
Split-horizon label is only used for multi-destination
frames (unknown unicast, mcast, bcast)
When PE1 wants to forward a multi-destination frame, it
appends this SH label to the packet
PE2 uses this label to perform split-horizon filtering
for frames destined to ESI-1 - e.g., a frame originated by
a segment must not be received by the same segment
BRKMPL-2333 28
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN BGP route 0x3 – Inclusive Multicast
• Usage:
• Multicast tunnels used to transport Broadcast, Multicast and Unknown Unicast frames (BUM)
RD
Ethernet Tag ID
8 bytes
4 bytes
Unique per Advertising PE per EVI
Set to VLAN or I-SID for VLAN-Aware Bundling Service interface, otherwise 0
Route Type specific encoding of E-VPN NLRI
IP Address Length
Originating Router’s IP add.
1 bytes
4 or 16 bytes
IP address length
IPv4 or IPv6 address
BRKMPL-2333 29
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
PMSI Tunnel Attribute – RFC6514
Flags
Tunnel Type
1 bytes
1 bytes
Flags based on RFC6514
Ingress Replication/mLDP etc.
Route Type specific encoding of E-VPN NLRI
MPLS Label
Tunnel Identifier
3 bytes
variable
Multicast MPLS Label
When the Tunnel Type is set to Ingress Replication, the Tunnel Identifier carries the unicast tunnel endpoint IP address of the local PE that is to be this PE's receiving endpoint address for the tunnel.
BRKMPL-2333 30
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Route Type Usage Benefits
MAC/IP Advertisement
Route (Type 2)
• Advertise MAC (and IP)
reachability
• Advertise MAC/IP binding
• MAC Mobility
• Per MAC (and IP)
policy
• ARP suppression
• Workload Mobility
MAC Address Reachability
• PE1 & PE2 learns MAC1 from CE1 and advertises in BGP to all other PEs with ES field in the MAC/IP advertisement set to ESI1
• PE3 and PE4 learn that MAC1 sits behind ESI1 which in turn sits behind PE1 & PE2
• PE3 and PE4 now know for packets destined to CE1, they can load balanced between PE1 and PE2
BRKMPL-2333
MAC1
PE1
PE2
PE3
PE4CE1
CE3
CE4
RR
ESI1
32
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
ARP Broadcast Suppression
• CE1 sends out an ARP request for CE3’s IP3
• PE1 snoops the ARP packet and learns (MAC1, IP1). It adds MAC1 to its MAC-VRF, MAC1/IP1 binding to its ARP cache. It also advertises this binding to all other PEs in BGP and floods this initial ARP request.
• All other PEs learn of (MAC1, IP1). They add the MAC1 to their MAC-VRFs and add (MAC1, IP1) to their ARP cache.
• Now, when CE4 sends an ARP request for IP1, PE4 has the binding info and can provide an ARP response (e.g., ARP proxy).
MAC1, IP1
PE1
PE2
PE3
PE4
CE1 CE3
CE4
3. ARP Request (IP1)
4. ARP Reply (IP1)Act as ARP
proxy for IP1.
Challenge:
How to reduce ARP broadcasts over the
MPLS/IP network, especially in large
scale virtualized server deployments?
BRKMPL-2333
Route Type Usage Benefits
MAC/IP Advertisement
Route (Type 2)
• Advertise MAC (and IP)
reachability
• Advertise MAC/IP binding
• MAC Mobility
• Per MAC (and IP)
policy
• ARP suppression
• Workload Mobility
MAC3, IP3
33
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
MAC Mobility
• At T0, PE1 learn the MAC1, and advertise to all other PEs
• At T1, MAC1 move to the PE3. PE1 is not aware of this
• PE3 learn the MAC1. It will overwrite the MAC route learnt from PE1
• PE3 will advertise MAC1 to all other PEs with sequence number +1
• All other PE will overwrite the MAC route
• Original PE1 will withdraw its old route
Challenge:
How to handle MAC move ?
MAC1, IP1
PE1
PE2
PE3
PE4
CE1 CE3
MAC1, IP1
BRKMPL-2333
Route Type Usage Benefits
MAC/IP Advertisement
Route (Type 2)
• Advertise MAC (and IP)
reachability
• Advertise MAC/IP binding
• MAC Mobility
• Per MAC (and IP)
policy
• ARP suppression
• Workload Mobility
34
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• PE advertises in EVPN Ethernet AS route with a split-horizon label (ESI MPLS Label) associated with each multi-homed Ethernet Segment
• Split-horizon label is only used for multi-destination frames (Unknown Unicast, Multicast & Broadcast)
• When an ingress PE floods multi-destination traffic, it encodes the Split-Horizon label identifying the source Ethernet Segment in the packet
• Egress PEs use this label to perform selective split-horizon filtering over the attachment circuit
Challenge:
How to prevent flooded traffic from
echoing back to a multi-homed
Ethernet Segment?
PE1
PE2
PE3
PE4
CE1 CE3
ESI-1 ESI-2
CE4
CE5Echo !
BRKMPL-2333
Split Horizon Filtering
Route Type Usage Benefits
Ethernet A-D Route
(Type 1)
• Advertising Split-Horizon
Label
• Aliasing
• Mass Withdraw of
addresses
• SH/AA MH Indication
• Loop avoidance –
even transient
• Efficient load
• Fast convergence
• balancing
• Per-site policy
35
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• PEs advertise in BGP the ESIs of local multi-homed Ethernet Segments.
• All-Active Redundancy Mode indicated
• When PE learns MAC address on its AC, it advertises the MAC in BGP along with the ESI of the Ethernet Segment from which the MAC was learnt.
• Remote PEs can load-balance traffic to a given MAC address across all PEs advertising the same ESI.
Challenge:
How to load-balance traffic towards a multi-
homed device across multiple PEs when MAC
addresses are learnt by only a single PE?
MAC1
PE1
PE2
PE3
PE4
CE1 CE3
CE4ESI-1
I can
reach
ESI1(All-Active)
I can
reach
ESI1(All-Active)
MAC1
I can reach
MAC1 via ESI1
MAC1 ESI1 PE1
PE2
BRKMPL-2333
Aliasing
Route Type Usage Benefits
Ethernet A-D Route
(Type 1)
• Advertising Split-Horizon
Label
• Aliasing
• Mass Withdraw of
addresses
• SH/AA MH Indication
• Loop avoidance –
even transient
• Efficient load
• Fast convergence
• balancing
• Per-site policy
36
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• PEs advertise two sets of information:
• MAC addresses along with the ESI from the address was learnt
• Connectivity to ESI(s)
• If a PE detects a failure impacting an Ethernet Segment, it withdraws the route for the associated ESI.
• Remote PEs remove failed PE from the path-list for all MAC addresses associated with an ESI.
• This effectively is a MAC ‘mass-withdraw’ function.
Challenge:
How to inform remote PEs of a failure
affecting many MAC addresses quickly while
the control-plane re-converges?
BRKMPL-2333
MAC Mass Withdraw
Route Type Usage Benefits
Ethernet A-D Route
(Type 1)
• Advertising Split-Horizon
Label
• Aliasing
• Mass Withdraw of
addresses
• SH/AA MH Indication
• Loop avoidance –
even transient
• Efficient load
• Fast convergence
• balancing
• Per-site policyMAC1
PE1
PE2
PE3
PE4
CE1 CE3
CE4ESI-1
MAC1,
MAC2,…
MACn
MAC1, MAC2, .. MACn ESI1
PE1
PE2
I lost ESI1
X
37
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN BGP Route 0x2 – MAC Advertisement
RD
Ethernet Segment Identifier
Ethernet Tag ID
8 bytes
10 bytes
4 bytes
MAC Address Length
MAC Address
IP Address Length
IP Address
MPLS Label1
1 byte
6 bytes
1 byte
4 or 16
3 bytes
Unique per Advertising PE per EVI
ESI of Ethernet Segment on which MAC Address was learnt. All 1s ESI for PBB-EVPN
Set to VLAN or I-SID for VLAN-Aware Bundling Service interface, otherwise 0
Allows for MAC Address ‘summarization’, i.e. hierarchical MAC Addresses. Typically set to 48
Could be C-MAC Address (EVPN) or B-MAC Address (PBB-EVPN)
To distinguish IPv4 vs. IPv6 addresses.
Used for ARP flood suppression or for Integrated Routing and Bridging (IRB).
MAC & IP Labels - downstream assigned
BRKMPL-2333
MPLS Label23 bytes
38
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
MAC Mobility extended community
• Used to tag the MAC Advertisement route
• EVPN: Indicates that a MAC address has moved from one PE to another
0x00
Reserved
Sequence Number4 bytesIndicates the count of MAC address mobility events
0x06
2 bytesSet to 0
BRKMPL-2333 39
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN BGP Route 0x1 – Ethernet Auto-discovery Route
This route has two flavors:
RD
Ethernet Segment Identifier
Ethernet Tag ID
MPLS Label
Unique per Advertising PE per EVI
ESI of Ethernet Segment
Set to VLAN or I-SID for VLAN-Aware
Bundling Service interface, otherwise 0
VPN (Aliasing) Label per (ESI,
Ethernet Tag)
BRKMPL-2333 40
Per-EVI Ethernet A-D route
• Advertise VPN label used for Aliasing or Backup-Path
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
Life of a PacketIngress Replication – Multi-destination Traffic Forwarding
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
VID 100
SMAC: M1
DMAC: F.F.F L3
L2 L5
L4
Mcast MPLS
Label assigned by
PE3 for incoming
BUM traffic on a
given EVI
PSN MPLS label
to reach PE3
ESI (split-horizon)
MPLS label
allocated by PE2
for segment ES1PE4 – non-DF for
given EVI drops
BUM traffic
PE2 – drops BUM
traffic originated
on ES1
PE1 receives broadcast
traffic from CE1. PE1
forwards it using ingress
replication – 3 copies
createdPE3 – as DF, it
forwards BUM
traffic towards
segment
During start-up
sequence, PE2 sent Per-
ESI Ethernet AD route
with ESI MPLS label
(split-horizon) (see
below)
PE 4 Inclusive Multicast
Route
RD = RD-4a
PMSI Tunnel Attribute
Tunnel Type = Ing. Repl.
Label = L4
RT ext. community
RT-aMcast MPLS Label – used to
transmit BUM traffic -
downstream assigned (for
ingress replication)
During start-up
sequence, PE1, PE2,
PE3, PE4 sent Inclusive
Multicast route which
include Mcast label
PE 2 Eth A-D Route (Per-ESI)
RD = RD20
ESI = ESI1
ESI MPLS Label ext. comm.
Redund. Flag = All-Active
Label = L5
RT ext. community
RT-a, RT-b, RT-c, RT-d
ESI MPLS Label – used by
local PEs for split-horizon -
downstream assigned (for
ingress replication)
BRKMPL-2333 42
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
Life of a Packet (cont.)Unicast Traffic Forwarding
PE1 MAC Route
RD = RD-1a
ESI = ESI1
MAC = M1
Label = L1
RT ext. community
RT-a
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
L1
MP2P VPN Label
assigned by PE1
for incoming traffic
for the target EVI
PSN MPLS label
to reach PE1
PE3 forwards
traffic destined to
M1 based on RIB
information (PE1)
PE3 RIB
VPN MAC ESI
RT-a M1 ES1
Path List
NH
PE1
VID 100
SMAC: M2
DMAC: M1VID 100
SMAC: M1
DMAC: F.F.F
MP2P VPN Label –
downstream allocated label
used by other PEs to send
traffic to advertised MAC
MAC advertised
by route
BRKMPL-2333 43
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Life of a Packet (cont.)Unicast Forwarding and Aliasing
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
PE3, PE4 RIB
VPN MAC ESI
RT-a M1 ES1
Path List
NH
PE1
PE2
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
L2
Aliasing MPLS
Label assigned by
PE2 for (ES1, EVI)
pair
PSN MPLS label
to reach PE2
PE3 forwards
traffic on a flow
(flow 2) based on
RIB information
(towards PE2)
VID 100
SMAC: M4
DMAC: M1L1
MP2P VPN
Label
assigned by
PE1 for
incoming for
target EVI
PSN MPLS label
to reach PE1
PE3 forwards
traffic on a flow
(flow 1) based on
RIB information
(towards PE1)
VID 100
SMAC: M3
DMAC: M1
During start-up sequence,
PE2 sent Per-EVI Ethernet
AD route (see below)
PE1 MAC Route
RD = RD-1a
ESI = ESI1
MAC = M1
Label = L1
RT ext. community
RT-a
MP2P VPN Label –
downstream allocated label
used by other PEs to send
traffic to advertised MAC
MAC advertised
by route
PE 2 Eth A-D Route (Per-EVI)
RD = RD-2a
ESI = ESI1
Label = L2
RT ext. community
RT-a
Aliasing MPLS Label – used
by remote PEs to load-
balance among local PEs
During start-up
sequence, PE1 sent Per-
EVI Ethernet AD route
VID 100
SMAC: M1
DMAC: F.F.F
BRKMPL-2333 44
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
E-VPN Operational ScenariosMAC Mobility
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
VID 100
SMAC: M1
DMAC: M2
PE3 / PE4 RIB
VPN MAC ESI
RT-a M1 ES1
Path List
NH
PE1
PE1 advertises MAC
route for M1. Route
may include MAC
mobility community
1Host M1 moves
from CE1 to CE3’s
location
3
M1M1 M1
VID 100
SMAC: M1
DMAC: M2
PE3 / PE4 install
M1 route towards
PE1
2
PE3 MAC Route
RD = RD-3a
ESI = ESI2
MAC = M1
Label = L3
MAC Mobility ext.
community.
Seq. Num = 2
RT ext. community
RT-a
PE1 / PE2 RIB
VPN MAC ESI
RT-a M1 ES2
Path List
NH
PE3
PE1 MAC Route
RD = RD-1a
ESI = ESI1
MAC = M1
Label = L1
MAC Mobility ext.
community
Seq. Num = 1
RT ext. community
RT-a
After host sends traffic
at new location, PE2
now adv MAC route
for M1 incrementing
sequence # in MAC
mobility community
4
PE1 withdraws its
M1 route and
installs a new one
pointing to PE3
5
BRKMPL-2333 45
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
E-VPN Failure Scenarios / ConvergenceLink / Segment Failure – Active/Active per Flow
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
PE3, PE4 RIB
VPN MAC ESI
RT-a M1 ES1
Path List
NH
PE1
PE2
PE3, PE4 RIB
VPN MAC ESI
RT-a M1 ES1
Path List
NH
PE2
PE1 withdraws
individual MAC
advertisement routes
related to failed
segment
7
PE1 withdraws Ethernet
Segment Route
3
PE1 withdraws Per-ESI
Ethernet AD route for
failed segment
2
PE2 recalculates
DF/BDF. Becomes DF
for all EVIs on segment
5
Mass withdrawal - PE3
/ PE4 remove PE1 from
path list for all MAC
addresses of failed
segment (ES1)
4
PE1 detects failure
of one of its
attached segments
1
PE2 adv. M1 MAC route
after CE traffic is
hashed towards PE2
6
PE1
BRKMPL-2333 46
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
E-VPN Failure Scenarios / ConvergencePE Failure
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
PE3, PE4 RIB
VPN MAC ESI
RT-a M1 ES1
Path List
NH
PE1
PE2
PE3, PE4 RIB
VPN MAC ESI
RT-a M1 ES1
Path List
NH
PE2
BGP RR / PE2 detects
BGP session time-out
with PE1
2PE3 / PE4 invalidate
routes from PE1
3
PE1 experiences a
node failure (e.g.
power failure)
1BGP RR / PE3 detects
BGP session time-out
with PE1
2
PE2 reruns DF election.
Becomes DF for all
EVIs on segment
4
PE3 / PE4 will forward
M1 traffic towards PE2
6
BGP RR / PE4
detects BGP
session time-
out with PE1
2
PE2 adv. M1 MAC route
after CE traffic is
hashed towards PE2
5
PE1
BRKMPL-2333 47
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Service Additional Capabilities
E-Line • All-active & single-active multi-homing support
• Both single-segment & multi-segment support
• Discovery & signaling via single protocol – BGP
EVPN-VPWS Does it Better than Legacy VPWS !!
BRKMPL-2333 49
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN BGP route type
Route type Usage EVPN EVPN VPWS
0x1 Ethernet Auto-Discovery
(A-D) Route
• MAC Mass-Withdraw
• Aliasing (load balancing)
• Split-Horizon
“Tagged with ESI Label Extended Community”
0x2 MAC Advertisement Route • Advertise MAC addresses
• Provide MAC / IP address bindings for ARP
broadcast suppression
“Tagged with MAC Mobility Extended
Community”
NOT used
0x3 Inclusive Multicast Route • Multicast tunnels used to transport
Broadcast, Multicast and Unknown Unicast
frames (BUM)
“Tagged with PMSI tunnel attribute” (P tunnel
type & ID) – RFC6514
NOT used
0x4 Ethernet Segment Route • Auto discovery of Multi-homed Ethernet
Segments, i.e. redundancy group discovery
• Designated Forwarder (DF) Election
“Tagged with ES-Import Extended Community”
BRKMPL-2333 50
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN BGP Extended Community
Attribute Usage Tagged BGP
route
EVPN EVPN VPWS
ESI label Extended
Community
• Split-Horizon for Ethernet
Segment.
• Indicate Redundancy Mode
(Single Active vs. All-Active)
Ethernet A-D
Route
ES-Import Extended
Community
• Limit the import scope of the
Ethernet Segment routes.
Ethernet
Segment Route
MAC Mobility Extended
Community
• E-VPN: Indicate that a MAC
address has moved from one
segment to another across PEs.
• PBB-EVPN: Signal C-MAC
address flush notification
MAC
Advertisement
Route
Not used
BRKMPL-2333 51
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN VPWS• Benefits of EVPN applied to point-to-point
services
• No signaling of PWs. Instead signals MP2P LSPs instead (ala L3VPN)
• All-active CE multi-homing (per-flow LB)
• Single-active CE multi-homing (per-service LB)
• Relies on a sub-set of EVPN routes to advertise Ethernet Segment and AC reachability
• PE discovery & signaling via a single protocol –BGP
• Per-EVI Ethernet Auto-Discovery route
• Handles double-sided provisioning with remote PE auto-discovery
• Under standardization: draft-ietf-bess-evpn-vpws
MPLS
PE1
CE1PE2
CE2
ES1 ES2
BGP Eth. Auto-
Discovery Route
EVPN NLRI
AC AC1 via PE1
Control-plane
attachment circuit
advertisement over the
Core
VPWS Service Config:
EVI = 100
Local AC ID = AC1
Remote AC ID = AC2
VPWS Service Config:
EVI = 100
Local AC ID = AC2
Remote AC ID = AC1
I have a P2P service that
needs to communicate
with the PE(s) that own
of AC = AC2
BRKMPL-2333 52
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN VPWS Operation – Single-homed
MPLS
PE1
CE1PE2
CE2
PE 1 Eth A-D Route
RD = RD-1a
ESI = ES1 (0)
Eth.Tag ID = AC1
Label (e.g. X)
RT ext. community
RT-a
PE 2 Eth A-D Route
RD = RD-2a
ESI = ES2 (0)
Eth.Tag ID = AC2
Label (e.g. Y)
RT ext. community
RT-a
PE1 RIB
VPN MAC ESI Eth.TAG
RT-a - 0 AC2
Path List
NH
PE2
ES1 ES2
RT – RT associated with a
given EVI
RD – RD unique per adv. PE
per EVI
MPLS Label – (downstream
assigned) used by remote
PEs to reach segment
ESI – 10 bytes ESI as specify
by EVPN Ethernet segment
IETF draft – zero for single-
homed
VPWS Service Config:
EVI = 100
Local AC ID = AC1
Remote AC ID = AC2
VPWS Service Config:
EVI = 100
Local AC ID = AC2
Remote AC ID = AC1
1
ES2 – Since CE2 is single
homed to PE2, ES2 = 0
Eth.Tag ID – 4-bytes local
AC-ID
PE2 RIB
VPN MAC ESI Eth.TAG
RT-a - 0 AC1
Path List
NH
PE1
2
3
4
5 6
BRKMPL-2333 53
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN VPWS Operation – Single-active
MPLS
PE1
CE1PE3
CE2
PE 1 Eth A-D Route
RD = RD-1a
ESI = ES1
Eth.Tag ID = AC1
Label (e.g. X)
RT ext. community
RT-a
PE 3 Eth A-D Route
RD = RD-2a
ESI = ES2 (0)
Eth.Tag ID = AC2
Label (e.g. Y)
RT ext. community
RT-a
PE1 & PE2 RIB
VPN MAC ESI Eth.TAG
RT-a - 0 AC2
Path List
NH
PE3
ES1
ES2
RT – RT associated with a
given EVI
RD – RD unique per adv. PE
per EVI
MPLS Label – (downstream
assigned) used by remote
PEs to reach segment
ESI – 10 bytes ESI as specify
by EVPN Ethernet segment
IETF draft
VPWS Service Config:
EVI = 100
Local AC ID = AC1
Remote AC ID = AC2
VPWS Service Config:
EVI = 100
Local AC ID = AC2
Remote AC ID = AC1
1
ES2 – Since CE2 is single
homed to PE2, ES2 = 0
Eth.Tag ID – 4-bytes local
AC-ID
PE3 RIB
VPN MAC ESI Eth.TAG
RT-a - ES1
RT-a - ES1
RT-a - ES1 AC1
Path List
NH
PE1
PE2
PE1
2
3
4
5
6PE2
ES1VPWS Service Config:
EVI = 100
Local AC ID = AC1
Remote AC ID = AC2
Only one PE (PE1)
shows as next hop for
the remote ACSingle-Active == per-vlan
load-balancing CE-PEs
Two bundles on CE
device
BRKMPL-2333 54
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN VPWS Operation – All-active
MPLS
PE1
CE1PE3
CE2
PE 1 Eth A-D Route
RD = RD-1a
ESI = ES1
Eth.Tag ID = AC1
Label (e.g. X)
RT ext. community
RT-a
PE 3 Eth A-D Route
RD = RD-2a
ESI = ES2 (0)
Eth.Tag ID = AC2
Label (e.g. Y)
RT ext. community
RT-a
PE1 & PE2 RIB
VPN MAC ESI Eth.TAG
RT-a - 0 AC2
Path List
NH
PE3
ES1
ES2
RT – RT associated with a
given EVI
RD – RD unique per adv. PE
per EVI
MPLS Label – (downstream
assigned) used by remote
PEs to reach segment
ESI – 10 bytes ESI as specify
by EVPN Ethernet segment
IETF draft
VPWS Service Config:
EVI = 100
Local AC ID = AC1
Remote AC ID = AC2
VPWS Service Config:
EVI = 100
Local AC ID = AC2
Remote AC ID = AC1
1
ES2 – Since CE2 is single
homed to PE2, ES2 = 0
Eth.Tag ID – 4-bytes local
AC-ID
2
3
4
5 PE2
ES1VPWS Service Config:
EVI = 100
Local AC ID = AC1
Remote AC ID = AC2PE3 RIB
VPN MAC ESI Eth.TAG
RT-a - ES1
RT-a - ES1
RT-a - ES1 AC1
Path List
NH
PE1
PE2
PE1,PE2
6
Both PEs (PE1/PE2)
shows as next hop for
the remote ACALL-Active == per-flow
load-baancing CE-PEs
Single bundle on CE
device
BRKMPL-2333 55
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
L2 Fabric: Legacy VLAN, STP
• L2/L3 boundary: limited
mobility
• 4K VLAN: Limited scale
• Inefficient forwarding: STP
• Complex VLAN provisioning
• Vendor specific L2
enhancement
IP Fabric: VXLAN/EVPN, SDN
• Spine-leaf
• Virtual overlay across physical
boundary
• VXLAN: Ultra-high scale
• Efficient forwarding: L3 ECMPs
• EVPN control plane
• SDN enabled VXLAN and
service chaining provisioning
ASR9K)
The Evolution of the DC Fabric
L2 Fabric:
FP/Trill
IP Fabric:
VXLAN, DP Learning
BRKMPL-2333 57
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Fabric – IP Underlay
Local LAN
Segment
Physical
Host
Local LAN
Segment
Physical
Host
Virtual Hosts
Local LAN
Segment
Virtual Switch
Edge Device
Edge Device
Edge Device
IP Interface
Edge device: could be
physical Leaf/ToR, or virtual
forwarder
BRKMPL-2333 58
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Fabric – VXLAN Overlay
Local LAN
Segment
Physical
Host
Local LAN
Segment
Physical
Host
VTEP
VTEP
VTEP
VV
V
Encapsulation
Virtual Hosts
Local LAN
Segment
Virtual Switch
VTEP – VXLAN Tunnel End-Point
VNI/VNID – VXLAN Network Identifier
BRKMPL-2333 59
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN Frame Format
MAC-in-IP Encapsulation
Un
de
rlay
Outer IP Header
Outer MAC Header
UDP Header
VXLAN Header
Original Layer-2 Frame Ove
rlay
14 Bytes
(4 Bytes Optional)
Ether Type
0x0800
VLAN ID
Tag
VLAN Type
0x8100
Src. MAC Address
Dest. MAC Address 48
48
16
16
16
20 Bytes
Dest. IP
Source IP
Header
Checksum
Protocol 0x11 (UDP)
IP Header
Misc. Data72
8
16
32
32
8 Bytes
Checksum 0x0000
UDP Length
VXLAN Port
Source
Port16
16
16
16
8 Bytes
Reserved
VNI
Reserved
VXLAN Flags
RRRRIRRR8
24
24
8
Src VTEP MAC Address
Next-Hop MAC Address
Src and Dst
addresses of
the VTEPs
VNI - allows
for 16M
possible
Segments
UDP 4789
Hash of the inner L2/L3/L4 headers of
the original frame.
Enables entropy for ECMP Load
balancing in the Network.
50 (
54)
Byte
s o
f O
ve
rhe
ad
BRKMPL-2333 60
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN Evolution – BGP EVPN Control Plane
• Workload MAC / IP Addresses learnt by • Multi-Protocol BGP (MP-BGP) based
Control-Plane using EVPN NLRI
• Advertises Layer-2 & Layer-3 Address-to-VTEP Association
• Make Forwarding decisions at VTEPs for Layer-2 (MAC) and Layer-3 (IP), Integrated Route/Bridge (IRB)
• Reduces impact of ARP on the Network
• Standards Based• draft-ietf-bess-evpn-overlay
• draft-ietf-bess-evpn-inter-subnet-forwarding
• draft-ietf-bess-evpn-prefix-advertisement
RR RR
V2V1
V3
BGP Route-ReflectorRR
iBGP Adjacency
BRKMPL-2333 61
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Use MP-BGP with EVPN Address Family on leaf nodes to distribute internal
host MAC/IP addresses, subnet routes and external reachability information
MP-BGP enhancements to carry up to 100s of thousands of routes with
reduced convergence time
BGP Update• Host-MAC• Host-IP• Internal IP Subnet• External Prefixes
MP-BGP for VXLAN EVPN Control PlaneEVPN Control Plane – Reachability DistributionEVPN route type 2: Host route, type 5: Subnet Route
LeafVTEPVTEPVTEPVTEP
Spine
BRKMPL-2333 62
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
MAC Host
IP
VNI VTEP
H-MAC-1 H-IP-
1
VNII-1 VTEP-1
EVPN Control Plane -- Host Advertisement
VTEP-2
VTEP-1
VTEP-3
H-MAC-1
H-IP-1
VLAN-1 /VNI-1
BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1
BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1
BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1
Install host info to RIB/FIB:
H-MAC-1 MAC table
H-IP-1 VRF IP host table
Route
Reflector
Install host info to RIB/FIB:
H-MAC-1 MAC table
H-IP-1 VRF IP host table
Local learning of host info:
H-MAC-1 (MAC table)
H-IP-1 (VRF IP host table )
1
2
3
3
44
MAC Host
IP
VNI VTEP
H-MAC-1 H-IP-1 VNII-1 VTEP-1
MAC Host
IP
VNI VTEP
H-MAC-1 H-IP-1 VNII-1 VTEP-1
BRKMPL-2333 63
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
1. Host 1 moves to VTEP-3 from VTEP-1
VXLAN BGP Control Plane
Leaf
Spine
Host 1H-MAC-1
H-IP-1VLAN 10
VXLAN 5000
EVPN Control Plane --- Host Movement
VTEP-4VTEP-3VTEP-2VTEP-1
MAC IP VNI Next-Hop Encap Seq#
H-MAC-1 H-IP-1 5000 VTEP-3 VXLAN 1
2. VTEP-3 detects Host 1, sends MP-BGP update for Host 1 with its own VTEP address and a new seq #1
3. Other VTEPs learn about the new route of Host 1
NLRI:• Host H-MAC-1, H-IP-1 • NVE VTEP-3• VNI 5000
Ext. Community:• Encapsulation: VXLAN• Cost• Sequence number: 1
BRKMPL-2333
Host 1H-MAC-1
H-IP-1VLAN 10
VXLAN 5000
64
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Inter-VXLAN Routing – EVPN IRB
Asymmetric
• Bridging & Routing on the ingress VTEP and bridging only on the egress VTEP
• Requires each VTEP to have all MAC addresses of their tenants in their ARP tables – can result in scale issue.
• Cisco follows Symmetric IRB
Symmetric
• Bridging & Routing on both the ingress and the egress VTEPs
• A VTEP only needs to maintain MACs for its directly attached endpoints. Optimal utilization of VTEP resources
Host 1
H-MAC-1
H-IP-1
VNI-A
VTEP-4VTEP-3VTEP-2VTEP-1
Host 2
H-MAC-2
H-IP-2
VNI-B
SVI A SVI B
IP Transport Network
Routing ?
BRKMPL-2333 65
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Symmetric EVPN IRB (1)
• Routing on both ingress and egress
VTEPs
• Layer-3 VNI
• Tenant VPN indicator
• One per tenant VRF
• VTEP Router MAC
• Ingress VTEP routes packets onto the
Layer-3 VNI
• Egress VTEP routes packets to the
destination Layer-2 VNI
VTEP VTEP VTEP VTEP
Layer-3 VNI (VRF VNI)
Layer-2 VNI(Network VNI)
Layer-2 VNI(Network VNI)
BRKMPL-2333 67
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Symmetric EVPN IRB (2)
Host 1
H-MAC-1
H-IP-1
VNI-A
VTEP-4Router MAC-4
VTEP-3VTEP-2VTEP-1
Router MAC-1
Host 2
H-MAC-2
H-IP-2
VNI-B
VNI
A
L3
VNI
VNI
B
L3
VNI
S-MAC: H-MAC-1D-MAC: GW-MAC
S-IP: H-IP-1D-IP: H-IP-2
Ingress VTEP
routes packets
from source VNI-A
to L3 VNI. D-MAC
in the inner header
is the egress
VTEP router MAC
1S-MAC: Router-MAC-1D-MAC: Router-MAC-4
S-IP: H-IP-1D-IP: H-IP-2
S-IP: VTEP-1D-IP: VTEP-4
VNI: L3 VNI
S-MAC: MAC-VTEP4 local portD-MAC: H-MAC-2
S-IP: H-IP-1D-IP: H-IP-2
Egress VTEP
routes packets
from L3 VNI to the
destination VNI-
B/VLAN.2
BRKMPL-2333 68
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
(Distributed) Anycast Gateway in with EVPN IRB
Host 1MAC1IP 1VLAN AVXLAN A
VTEPVTEPVTEPVTEP
SVI
GW IP
GW MAC
# VLAN to VNI mapping
vlan 200
vn-segment 5200
# Anycast Gateway MAC, identically configured on all VTEPs
fabric forwarding anycast-gateway-mac 0002.0002.0002
# Distributed IP Anycast Gateway (SVI)
# Gateway IP address needs to be identically configured on all
VTEPs
interface vlan 200
no shutdown
vrf member Tenant-A
ip address 20.0.0.1/24
fabric forwarding mode anycast-gateway
SVI
GW IP
GW MAC
SVI
GW IP
GW MAC
SVI
GW IP
GW MAC
Host 2MAC2IP 2VLAN AVXLAN A
Host 3MAC3IP 3VLAN AVXLAN A
Host 4MAC4IP 4VLAN AVXLAN A
The same anycast gateway virtual IP
address and MAC address are
configured on all VTEPs in the VNI.
BRKMPL-2333 69
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IRB Anycast Gateway Deployment OptionsDistributed vs. Centralized Anycast Gateway
DC-1WAN
Leaf
VM1 VM2
DC-2
VM3 VM4
InternetBranch
IP-VPN
Internet
VPLS/PW/EVP
N
clien
t
DC Gateway
(L3 GW)
Leaf
(L2 only)
L3 EVPN
(per-VRF VNI)
IRB
IRB
IRBIRB
L2 only DC fabric L2/L3 DC fabric
Leaf
(IRB anycast
GW)
IRB IRB IRB IRB
IP-VPN
Internet
VPLS/PW/EVP
N
clien
t
DC Gateway
(IRB anycast
GW)
L2 EVPN
(per-BD VNI)
VPN
Client
Internet
Client
Centralized
Anycast GW
Distributed
Anycast GW
BRKMPL-2333 70
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKMPL-2333
The Solution Must be End-to-End
Leaf Leaf bLeaf bLeaf
DC
Spine
APIC
WAN/DCI
SDN-DC, VXLAN
overlay
SDN-WAN, MPLS/Segment Routing802.1q?
• Legacy 802.1q handoff
• It means multi-pathing is out the
door between DC & WAN
• VLANs and sub-interfaces
creation
• No policy level integration
• Small FIB/MAC table size on
border Leaf, create bottleneck
72
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Gateway – Seamless DC and WAN Integration
Leaf Leaf bLeaf bLeaf
DC
Spine
APIC
WAN/DCI
Integration
Interworking
Scalable, Resilient, Optimized, End-to-End
• Common MP-BGP (EVPN AF) control plane
• VXLAN to MPLS data plane interworking
• SDN based auto provisioning: OpFlex,
APIC/VTS
• Integrated Policy control: WAN optimization
SDN-DC, VXLAN
overlay
SDN-WAN, MPLS/SR
• L3 Gateway
• L2 Gateway
• IRB with Anycast
Gateway
73
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Gateway – L3 GatewayS-N Routing: all active
DC-1WAN
Leaf
Spine
DC
Gateway
VM1 VM2
DC-2
VM3 VM4
InternetBranch
VPN
Client
Internet
Client
IP-VPN
Internet
clien
t
Gateway
Leaf
(L3 anycast
GW)
L3 EVPN
(per-VRF VNI)
• EVPN/VXLAN to IP-VPN/MPLS interworking
• EVPN/VXLAN to global internet
• L3 EVPN between Leaf and GW (per-vrf VNI)
• Leaf is the L3 default gateway for VMs (with
EVPN IRB anycast GW) and does inter-vxlan
routing
L2/L3 DC fabric L2/L3 DC fabric
BRKMPL-2333 74
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Gateway
DC Gateway – L2 GatewayL2 Stretch: E-W and S-Nall-active or single-active
DC-1WAN
Leaf
Spine
DC Gateway
VM1 VM2
DC-2
VM3 VM4
Branch
VPN
Client
VPLS/PW/EV
PN
clien
t
Gateway
Leaf
L2 EVPN
(per-BD VNI)
• L2 EVPN/VXLAN in the DC
• L2 DCI (E-W): EVPN/VXLAN with EVPN/VPLS
interworking
• L2 to client (S-N): EVPN/VXLAN with VPLS/PW
interworking
GatewayLeaf Leaf
L2 stretch: EVPN/VPLS (MPLS) L2 EVPN/VXLAN L2 EVPN/VXLAN
L2/L3 DC fabric L2/L3 DC fabric
BRKMPL-2333 75
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Gateway – IRB Anycast GatewayIntegrated Routing and Bridging
DC-1WAN
Leaf
Spine
DC
Gateway
VM1 VM2
DC-2
VM3 VM4
InternetBranch IP-VPN
Internet
VPLS/PW/EVP
N
clien
t
Gateway
(L3 anycast GW)
Leaf
(L2 only)
L2 EVPN
(per-BD VNI)
• DC fabric is L2 only. All routing on DC gateway
• DC gateway is the L3 default gateway for VMs via
EVPN IRB anycast gateway
• Support both L2 and L3 for the same VNI at the
same time
Gateway GatewayLeaf Leaf
L2 stretch: EVPN/VPLS (MPLS) L2 EVPN/VXLAN L2 EVPN/VXLAN
IRB
IRB IRB IRB
L2 only DC fabric L2 only DC fabric
VPN
Client
Internet
Client
76
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IRB Anycast Gateway Deployment OptionsDistributed vs. Centralized Anycast Gateway
DC-1WAN
Leaf
VM1 VM2
DC-2
VM3 VM4
InternetBranch
IP-VPN
Internet
VPLS/PW/EVP
N
clien
t
DC Gateway
(L3 GW)
Leaf
(L2 only)
L3 EVPN
(per-VRF VNI)
IRB
IRB
IRBIRB
L2 only DC fabric L2/L3 DC fabric
Leaf
(IRB anycast GW)
IRB IRB IRB IRB
IP-VPN
Internet
VPLS/PW/EVP
N
clien
t
DC Gateway
(IRB anycast GW)
L2 EVPN
(per-BD VNI)
VPN
Client
Internet
Client
Centralized
Anycast GW
Distributed
Anycast GW 77
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
IRB Anycast Gateway Deployment Option Comparison
Option 1
Distributed IRB Anycast Gateway
Option 2
Centralized IRB Anycast Gateway
DC Gateway Router • L2 or L3 gateway function
• Doesn’t require IRB function
• EVPN IRB for integrated L2 and L3
• IRB anycast gateway for VM’s default
gateway
Leaf • EVPN IRB for integrated L2 and L3
• IRB anycast gateway for VM’s
default gateway
• L2 EVPN peering across DC
• Cross-DC underlay IP routing is required
Pros • Optimized E-W inter-vxlan routing • Simple DC fabric design: L2 only
• Large ARP table on the DC gateway
router
Cons • EVPN IRB function across all Leaf
nodes
• Require both L2 and L3 function on
the leaf
• Sub-optimal E-W inter-vxlan routing
BRKMPL-2333 78
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Gateway – the Policy Integration and Auto-Provisioning: Application-engineered Routing
DC-1WAN
Leaf
Spine
DC
Gateway
VM1 VM2
DC-2
VM3 VM4
High bandwidth flowLow latency flow
DC policy domain DC policy domainWAN policy domain
WAN Segment
RoutingVTS/APIC VTS/APIC
BRKMPL-2333
PBTS: steering packet to
SR-TE in the WAN
DC: classification
and marking
79
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Example: Application has a preference for disjoint path in dual-plane
WAN networks
Segment
Routing
WAN
with WAE
ACI Fabric
Po
licy
WEBWEB
Customer has the requirement that traffic from
applications RED and BLUE should be
transported across disjoint paths in the WAN.
- Policy expressed on APIC and delivered by
SR-enabled +WAE WAN
BRKMPL-2333 80
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Segment
Routing
WAN
with WAE
Example: Application requires the lowest latency path in the WAN
ACI Fabric
Po
licy
WEBWEB
WAN has cheap capacity via US with higher latency.
Scarce, expensive capacity via Russia, with lower
latency.
Customer identify the applications that require the
lowest possible latency path on APIC, integration
steers traffic on the path via Russia.
Tokyo
RussiaUS
Brussels
BRKMPL-2333 81
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN: Next Generation VPN
E-LAN(MP2MP
L2VPN)
E-LINE(P2P
L2VPN)
E-TREE(P2MP
L2VPN)
EVPN
VPWS
(PBB-)
EVPN
EVPN
DC Fabric(IntraDC
Overlay)
IRB(L2/L3
Overlay)
DCI(InterDC)
IP-VPN(L3VPN)
EVPN
DCI
EVPN-
IRB
EVPN-
Overlay
EVPN
ETREE
EVPN-
L3
VPLS PW 4364VPLS-
ETREE VPLS,OTV
BRKMPL-2333 83
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
References• [FabricPath]: FabricPath http://www.cisco.com/en/US/prod/switches/ps9441/fabric_path_promo.html
• [LISP]: Locator/ID Separation Protocol https://datatracker.ietf.org/wg/lisp/charter/
• [802.1Qbp] ECMP http://www.ieee802.org/1/files/public/docs2011/new-ashwood-sajassi-ecmp-par-0111-v04.pdf
• [EVPN]: BGP MPLS Based Ethernet VPN http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-04
• [TRILL]: Transparent Interconnection of Lots of Links https://datatracker.ietf.org/wg/trill/charter/http://tools.ietf.org/wg/trill/draft-ietf-trill-rbridge-protocol/
• [VL2]: VL2: A Scalable and Flexible Data Center Network http://ccr.sigcomm.org/online/?q=node/502
• [MOOSE]: Addressing the Scalability of Ethernet with MOOSE http://www.cl.cam.ac.uk/~mas90/MOOSE/MOOSE.pdf
• [PORTLAND]: PortLand: A Scalable Fault-Tolerant Layer 2 Data Center Network Fabric http://ccr.sigcomm.org/online/?q=node/503
• [SEATTLE]: Floodless in SEATTLE: A Scalable Ethernet Architecture for Large Enterprises http://www.cs.princeton.edu/~chkim/Research/SEATTLE/seattle.pdf
• [MONSOON]: Towards a Next Generation Data Center Architecture: Scalability and Commoditization http://research.microsoft.com/apps/pubs/default.aspx?id=79348
• [VLB]: Valiant Load Balancing in Backbone Networks http://www.stanford.edu/~ashishg/network-algorithms/rui.pdf
BRKMPL-2333 84
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.
• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.
BRKMPL-2333 85
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
BRKMPL-2333 86
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services
Thursday, July 14th, 2016
11:30 am - 12:30 pm, In the Oceanside A room
What to expect from this innovation talk
• Insights on market trends and forecasts
• Preview of key technologies and capabilities
• Innovative demonstrations of the latest and greatest products
• Better understanding of how Cisco can help you succeed
Register to attend the session live now or
watch the broadcast on cisco.com
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
PBB-EVPN Startup Sequence
Segment Auto-Discovery
ESI and B-MAC Auto-Sensing
Redundancy Group Membership Auto-Discovery
VPN Auto-Discovery
Multicast Tunnel ID / Endpoint
Discovery
Backbone MAC (B-MAC) Reachability Advertisement
BRKMPL-2333 90
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
PBB-EVPN Startup Sequence (cont.)ESI and B-MAC Auto-Sensing
Segment Auto-Discovery
ESI and B-MAC Auto-Sensing
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
LACP PDU
exchange
Source B-MAC used at PBB-EVPN PE on a
given ESI can be auto-generated* from CE’s
LACP information -> CE’s LACP System ID
MAC with U/L** (Universal / Locally
Administered) bit flipped
Example: 0211.0022.0033
CE LACP info:
LACP System ID (MAC) (6B)
e.g. 0011.0022.0033
LACP System Priority (2B)
e.g. 0000
LACP Port Key (2B)
e.g. 0018
ESI (10B) can be auto-generated*
from CE’s LACP information ->
concatenation of CE’s LACP
System Priority + Sys ID + Port Key
Example:
0000. 0011.0022.0033.0018
(*) ESI and B-MAC can also be manually configured
(**) U/L is second-least-significant bit of most significant byte
System
Priority
2 bytes 6 bytes 2 bytes
System MAC
AddressPort Key
B-MAC
B-MAC
B-MAC
B-MAC
BRKMPL-2333 91
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
PBB-EVPN Startup Sequence (cont.)BGP Ethernet Segment Route
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
PE 1 Eth Segment Route
RD = RD10
ESI = ESI1
ES-Import ext. comm.
e.g. 0011.0022.0033
MAC address portion
of ESI (6B)
PE 2 Eth Segment Route
RD = RD20
ESI = ESI1
ES-Import ext. comm.
e.g. 0011.0022.0033
Segment Auto-Discovery
ESI and B-MAC Auto-Sensing
Redundancy Group Membership Auto-Discovery
RD – RD unique per
advertising PE
BRKMPL-2333 92
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
PBB-EVPN Startup SequenceDesignated Forwarder (DF) Election*
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
Ordered List of discovered PEs
starting from zero (lowest IP add)
Segment Auto-Discovery
ESI and B-MAC Auto-Sensing
Redundancy Group Membership Auto-Discovery
PE Ordered List
Position PE
0 PE1
1 PE2
Modulo Operation
I-SID
I-SID mod N
(N = # of PEs)
(e.g. I-SID mod 2)
100 0
101 1
102 0
103 1
PE Ordered List
Position PE
0 PE1
1 PE2
Modulo Operation
I-SID (I-SID mod 2)
100 0
101 1
102 0
103 1
Exchange of Ethernet
Segment Routes
Result of modulo
operation is used to
determine DF and
BDF status
DF – Designated Forwarder
BDF – Backup Designated Forwarder
I-SID – PBB 24-bit Service Instance ID
Example:
PE2 DF for I-SIDs 101, 103
PE2 BDF for I-SIDs 100, 102
Example:
PE1 DF for I-SIDs 100, 102
PE1 BDF for I-SIDs 101, 103
(*) DF election with Service Carving shown (i.e. one DF per I-SID in the segment)BRKMPL-2333 93
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKMPL-2333
PBB-EVPN Startup Sequence (cont.)BGP MAC Advertisement Route (B-MAC)
Segment Auto-Discovery
ESI and B-MAC Auto-Sensing
Redundancy Group Membership Auto-Discovery
Backbone MAC (B-MAC) Reachability Advertisement
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
PE1 MAC Route
RD = RD-1a
ESI = MAX_ESI
MAC = B-M1
Label = L1
RT ext. community
RT-a
MP2P VPN Label –
downstream allocated label
used by other PEs to send
traffic to advertised (MAC,EVI)B-MAC advertised
by route
PE3 / PE4 RIB
VPN MAC ESI
RT-a B-M1 n/a
Path List
NH
PE1
PE2
PE2 MAC Route
RD = RD-2a
ESI = 1
MAC = B-M1
Label = L2
RT ext. community
RT-a
RD – RD unique per
advertising PE per EVI
ESI – reserved ESI
indicates advertised
MAC is a B-MAC
B-M1
B-M1
B-M2
B-M2
94
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
PBB-EVPN Startup SequenceBGP Inclusive Multicast Route
VPN Auto-Discovery
Multicast Tunnel ID / Endpoint
Discovery
PE 1 Inclusive Multicast Route
RD = RD-1a
PMSI Tunnel Attribute
Tunnel Type (e.g. Ing. Repl.)
Label (e.g. L1)
RT ext. community
RT-a
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
Tunnel Type – Ingress
Replication or P2MP LSP
Mcast MPLS Label – used to
transmit BUM traffic -
downstream assigned (ing.
repl.) or upstream assigned
(Aggregate Inclusive P2MP
LSP1)
PMSI - P-Multicast Service Interface
BUM – Broadcast / Unknown Unicast / Multicast
RD – RD unique per
adv. PE per EVI
RT – RT associated with a
given EVI
PE 2 Inclusive Multicast Route
RD = RD-2a
PMSI Tunnel Attribute
Tunnel Type (e.g. Ing. Repl.)
Label (e.g. L2)
RT ext. community
RT-a(1) Mcast MPLS label is not set for Inclusive Trees (P2MP LSP)
BRKMPL-2333 95
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
Life of a PacketIngress Replication – Multi-destination Traffic Forwarding
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
VID 100
SMAC: M1
DMAC: F.F.F
Mcast MPLS
Label assigned by
PE3 for incoming
BUM traffic on a
given EVI
PSN MPLS label
to reach PE3
PE4 – non-DF for
given I-SID drops
BUM traffic PE2 – drops BUM
traffic originated
on same source
B-MAC (B-M1)
PE1 receives broadcast
traffic from CE1. PE1
adds PBB encapsulation
and forwards it using
ingress replication – 3
copies created PE3 – as DF, it
forwards BUM
traffic towards
segment
PE 2 Inclusive Multicast
Route
RD = RD-2a
PMSI Tunnel Attribute
Tunnel Type = Ing. Repl.
Label = L2
RT ext. community
RT-a
Mcast MPLS Label – used to
transmit BUM traffic -
downstream assigned (for
ingress replication)
During start-up sequence,
PE1, PE2, PE3, PE4 sent
Inclusive Multicast route
which include Mcast label
B-M1
B-M1
B-M2
B-M2
B-M1
B-M1
B-M2
B-M2
L2 PBB
L3 PBB
L4 PBB
PE3 MAC Table
I-SID xyz
C-MAC B-MAC
M1 B-M1
Data-plane based
MAC learning for
C-MAC / B-MAC
associationBRKMPL-2333
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
Life of a PacketInclusive Trees – Multi-destination Traffic Forwarding
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
VID 100
SMAC: M1
DMAC: F.F.F
P2MP LSP sourced
at PE1
Inclusive (per-ISID)
trees
PE4 – non-DF for
given I-SID drops
BUM traffic PE2 – drops BUM
traffic originated
on same source
B-MAC (B-M1)
PE1 receives broadcast
traffic from CE1. PE1
adds PBB encapsulation
and forwards it using an
Inclusive TreePE3 – as DF, it
forwards BUM
traffic towards
segment
PE 2 Inclusive Multicast
Route
RD = RD-2a
PMSI Tunnel Attribute
Tunnel Type = P2MP
Label = 0
RT ext. community
RT-a
Mcast MPLS Label – not
assigned for Inclusive Trees
During start-up sequence,
PE1, PE2, PE3, PE4 sent
Inclusive Multicast route
B-M1
B-M1
B-M2
B-M2
B-M1
B-M1
B-M2
B-M2
PE3 MAC Table
I-SID xyz
C-MAC B-MAC
M1 B-M1
Data-plane based
MAC learning for
C-MAC / B-MAC
association
PBB
PBB
PBB
BRKMPL-2333
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
Life of a Packet (cont.)Unicast Traffic Forwarding
PE1 MAC Route
RD = RD-1a
ESI = 1
MAC = B-M1
Label = L1
RT ext. community
RT-a
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
MP2P VPN Label
assigned by PE1
for incoming traffic
for the target EVI
PSN MPLS label
to reach PE1
PE3 forwards traffic
destined to M1
using B-MAC B-M1
towards PE1
PE3 RIB
VPN MAC ESI
RT-a B-M1 n/a
Path List
NH
PE1
PE2
VID 100
SMAC: M2
DMAC: M1
VID 100
SMAC: M1
DMAC: F.F.F
MP2P VPN Label –
downstream allocated label
used by other PEs to send
traffic to advertised MAC
MAC advertised
by route
B-M1
B-M1
B-M2
B-M2
B-M1
B-M1
B-M2
B-M2
L1 PBB
PE3 MAC Table
I-SID xyz
C-MAC B-MAC
M1 B-M1
Data-plane based
MAC learning for
C-MAC / B-MAC
association
During start-up sequence,
PE1 & PE2 advertised
MAC routes for B-MAC
(B-M1)
BRKMPL-2333 98
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
Life of a Packet (cont.)Unicast Traffic Forwarding and Aliasing
PE1 MAC Route
RD = RD-1a
ESI = 1
MAC = B-M1
Label = L1
RT ext. community
RT-a
PE3 RIB
VPN MAC ESI
RT-a B-M1 n/a
Path List
NH
PE1
PE2
VID 100
SMAC: M1
DMAC: F.F.F
MP2P VPN Label –
downstream allocated label
used by other PEs to send
traffic to advertised MAC
MAC advertised
by route
B-M1
B-M1
B-M2
B-M2
PE3 MAC Table
I-SID xyz
C-MAC B-MAC
M1 B-M1
During start-up sequence,
PE1 & PE2 advertised
MAC route for B-MAC (B-
M1)
PE2 MAC Route
RD = RD-2a
ESI = 1
MAC = B-M1
Label = L2
RT ext. community
RT-a
MPLS
PE1
CE1
PE2
PE3
CE3
PE4
MP2P VPN Label
assigned by PE2
for incoming traffic
for target EVI
PSN MPLS label
to reach PE2
PE3 forwards traffic
on a flow (flow 2) to
M1 using B-MAC B-
M1 towards PE2
VID 100
SMAC: M4
DMAC: M1
MP2P VPN
Label
assigned by
PE1 for
incoming traffic
for target EVI
PSN MPLS label
to reach PE1
PE3 forwards traffic
on a flow (flow 1) to
M1 using B-MAC B-
M1 towards PE1
VID 100
SMAC: M3
DMAC: M1
B-M1
B-M1
B-M2
B-M2
L2 PBB
L1 PBB
Data-plane based
MAC learning for C-
MAC / B-MAC
association
BRKMPL-2333 99
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Life of a Packet (cont.)Active / Active Load Balancing from CE
MPLS
PE1
CE1
PE2
PE3
CE3
PE 1 / PE2 RIB
VPN MAC ESI
RT-a B-M3 0
Path List
NH
PE3
MPLS
PE1
CE1
PE2
PE3
CE3
VID 100
SMAC: M1
DMAC: M3
VID 100
SMAC: M2
DMAC: M3
MP2P VPN Label
assigned by PE3
for incoming traffic
for target EVI
PSN MPLS label
to reach PE3
PE1 forwards traffic to
M3 using B-MAC B-M3
towards PE3
PE2 forwards traffic to
M3 using B-MAC B-M3
towards PE3
PE3 MAC Route
RD = RD-3a
ESI = 0
MAC = B-M3
Label = L3
RT ext. community
RT-a
MP2P VPN Label –
downstream allocated label
used by other PEs to send
traffic to advertised MACMAC advertised
by route
ESI == 0 used for
Single Home Device
B-M1
B-M1
B-M3
PE1 / PE2 MAC Table
I-SID xyz
C-MAC B-MAC
M3 B-M3
B-M1
B-M1
B-M3
L3 PBB
L3 PBB
BRKMPL-2333 100
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
MPLS
PE1
PE2
PE3
CE3
PE4
Life of a Packet (cont.)Active / Active Per-Service Load Balancing
MPLS
PE3
CE3
PE4
PE3, PE4 RIB
VPN MAC ESI
RT-a B-M1 n/a
RT-a B-M2 n/a
Path List
NH
PE1
PE2
MP2P VPN Label
assigned by PE1 for
incoming traffic for
target EVI
PE3 forwards traffic to
M1 using B-MAC B-M1
towards PE1
VID 100 I-SID 100
SMAC: M3
DMAC: M1
VID 100
SMAC: M1
DMAC: M3
During startup, PE2
advertises:
• Ethernet Segment route
• MAC Route for B-MAC B-
M2
PE2 elected DF for I-SID
200
CE1 configured with two
(2) separate bundles
towards PEs
PE1
CE1
PE2
During startup, PE1
advertises:
• Ethernet Segment route
• MAC Route for B-MAC B-
M1
PE1 elected DF for I-SID
100
PE1 MAC Route
RD = RD-1a
ESI = 1
MAC = B-M1
Label = L1
RT ext. community
RT-a
B-M1
B-M2
B-M3
B-M3 PE3 / PE4 MAC Table
I-SID 100
C-MAC B-MAC
M1 B-M1
VID 200
SMAC: M11
DMAC: M33
PE3 / PE4 MAC Table
I-SID 200
C-MAC B-MAC
M11 B-M2
L1 PBBCE1
B-M1
B-M2
B-M3
B-M3
VID 200 ISD 200
SMAC: M4
DMAC: M11
L2 PBB
BRKMPL-2333 101