BRKMPL 2333

Next Generation L2 VPN
Ali Sajassi Distinguished Engineer, Cisco

BRKMPL-2333
Agenda
EVPN Overview
EVPN Technology Prime
EVPN Startup Sequence
EVPN Operation
A Day in Life of a Packet
EVPN-VPWS
EVPN Deployment: DC Fabric Evolution
with EVPN-VXLAN
EVPN Deployment: DC Fabric and WAN
Integration
EVPN Overview
L2VPN Technologies Evolution
Native L2 Bridging Technologies
More VLAN Shortest

VLAN Scale MAC Scale ECMP
Scale Path Fwd
802.1D 802.1Q 802.1ad 802.1aq 802.1Qbp

802.1ah
(QinQ) (SPB) (ECMP)
(MACinMAC)
IEEE 802.1Qbp
What about Inter Domain (WAN) Connectivity?
Large # of VLANs 16 millions
What about IP or MPLS fabric?
Large # of MACs MAC-in-MAC
What about industry traction & multi-vendor interop?
Optimum Forwarding
What about All-Active multi-homing?
ECMP
What about multi-pathing (not ECMP)?
BRKMPL-2333 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
L2VPN Technologies Evolution
PBB-EVPN
L2 VPN Technologies
To Address
all major
PW Scale MAC Scale
shortcomings EVPN-VxLAN
VPLS H-VPLS PBB-VPLS EVPN

(RFC 7432)
EVPN-IRB
Inter Domain (WAN) Connectivity? Yes

IP or MPLS fabric? Yes
Industry traction & Multi-vendor interop? Yes EVPN-VPWS
All-Active multi-homing? Yes

Multi-pathing (not ECMP)? Yes

Whats the big deal about EVPN?
EVPN is next generation all-in-one VPN solution
It not only does the job of many other VPN technologies but it does it better !!
E-LAN E-LINE E-TREE DC Fabric IRB DCI
(MP2MP (P2P (P2MP L3VPN (IntraDC (L2/L3 (InterDC)
L2VPN) L2VPN) L2VPN) Overlay) Overlay)
VPLS- VxLAN
VPLS PW 4364 VPLS,OTV
ETREE TRILL
EVPN
PBB- EVPN EVPN EVPN- EVPN- EVPN- EVPN-

EVPN VPWS ETREE L3VPN Overlay IRB DCI
2006 2010 2011 2013 2015
Following drafts were Following drafts were Enhancements

- OPEN project was started at - Introduced to IETF as introduced: introduced - Virtual ES
Cisco Routed-VPLS - EVPN - EVPN IRB - Optimized ingress replication
- OPEN = Optimum Ethernet - Merged with Junipers - PBB-EVPN - EVPN DCI - IGMP aggregation between
Network MAC-VPN and was - EVPN-VPWS PODs
introduced an EVPN - EVPN-Overlay - mcast tunnels between DCs
- EVPN-ETREE - Inter-AS for IRB
- L3VPN multi-homing
EVPN in a Nut Shell
MAC learning in control plane (via BGP)
MAC Routing: Control plane (BGP)

advertise the learnt MACs from CE
PE1 PE3 Optimum forwarding,

Single active
All active multi- ECMP, Multi-pathing
multi-homing
homing
Common L2/L3 VPN
CE1 CE3
Operational Mode
IP or MPLS
C-MAC:
M1 Flexible Policy Control
Consolidated VPN
PE2 PE4 service with x-EVPN
Data Plane: IP or MPLS,

flexible
EVPN Does it Better than VPLS !!
Service Additional Capabilities

E-LAN Provides All-Active multi-homing
Prevents loop for both all-active & single-active even in transient
state
Efficient utilization of network cross-sectional bandwidth (via
optimum forwarding, ECMP, multi-pathing on a per flow basis)
Flexible policy control per MAC and per Site
VPLS cannot provide All-Active Multi-Homing
Because:
Flip/flopping !
Packets originated from MAC2 arrives
MAC1 CE1 CE2 MAC2
at both PE3 and PE4 which get PE1 PE3
forwarded subsequently to PE1. PE1
keeps flip/flopping between PE3 and
PE4 for learning of MAC2!!
PE2 PE4
MAC1 MAC2
BUM packets forwarded from PE1 CE1 PE1 PE3 CE2
can get loopback to the originating

Echo !
CE
PE2 PE4
VPLS cannot do proper load-balancing MAC1 CE1 PE1 PE3 CE2 MAC2
because doesnt support Aliasing.
When PE3 wants to forward a packet
with destination address MAC1, it Load balancing
needs to send it to both PE1 and PE2
even though it only learned MAC1 from
PE1. PE2 PE4
EVPN All-Active Multi-Homing Principles
MAC1 MAC2
ARP broadcast packet doesnt get CE1 PE1 PE3 CE2
loopback to the originating CE

Echo !
device. Split-horizon
PE2 PE4
Either PE3 or PE4 forward the MAC1 CE1 CE2 MAC2

PE1 PE3
broadcast frame to the far-end
dual-homed device CE2. DF
Duplicate !
selection
PE2 PE4
When PE1 & PE2 forward traffic for

MAC1, there is no flip/fopping on PE3 MAC1 CE1 PE1 PE3 CE2 MAC2
because of MAC learning in control
plane.
When PE3 wants to forward a packet Load balancing
with destination address MAC1, it

needs to send it to both PE1 and PE2 PE4
even though it only learned MAC1 from PE2
PE1. Load balancing via aliasing
EVPN Efficient Cross-Sectional BW Utilizaiton
Flow Based Load-balancing PE to PE direction

EVPN provides per-flow load- Vlan X -
P P
balancing among egress PEs F1 X
Vlan
F2
E E
using BGP multi-pathing
P
E
Vlan X -
Per-flow load balancing F1X Flow Based Multi-Pathing in the Core
Vlan
F2X
Vlan
between ingress and egress F3X
Vlan
P
E
P
E
P P
PEs are provided using IGP F4
ECMP (ingress PE still needs

to add entropy field in the P P
P
packet). E
E-VPN Concepts
E-VPN Instance BGP Route
Ethernet Segment BGP Routes
(EVI) & MAC-VRF Attributes
SHD Route Types Extended Communities

CE1 PE1
ESI1 [1] Ethernet Auto-Discovery (AD) Route ESI MPLS Label
MAC-VRF
[2] MAC Advertisement Route ES-Import
MHD CE2
[3] Inclusive Multicast Route MAC Mobility
MAC-VRF
ESI2 [4] Ethernet Segment Route Default Gateway
PE
PE2 [5] IP Prefix Route
EVI identifies a VPN in the Represents a site E-VPN and PBB-EVPN New BGP extended
network connected to one or more define a single new BGP communities defined
Encompass one or more PEs NLRI used to carry all E- Expand information carried
bridge-domains, depending Uniquely identified by a 10- VPN routes in BGP routes, including:
on service interface type byte global Ethernet NLRI has a new SAFI 70 MAC address moves
Port-based Segment Identifier (ESI) (EVPN), AFI 25 (L2VPN) C-MAC flush notification
VLAN-based (shown above) Could be a single device or Routes serve control plane Redundancy mode
VLAN-bundling an entire network purposes, including:
MAC / IP bindings of a GW
VLAN aware bundling (NEW) Single-Homed Device (SHD) MAC address reachability
Split-horizon label encoding
Multi-Homed Device (MHD) MAC mass withdrawal
Single-Homed Network (SHN) Split-Horizon label adv.
Multi-Homed Network (MHN) Aliasing
Multicast endpoint discovery
Redundancy group discovery
Designated forwarder election
EVPN Route Types & Benefits
Route Type Usage Benefits
Ethernet A-D Route Aliasing Loop avoidance even
(Type 1) Mass Withdraw of addresses transient
SH/AA MH Indication Fast convergence
Advertising Split-Horizon Label Efficient load balancing
Per-site policy
MAC/IP Advertisement Route Advertise MAC (and IP) reachability Per MAC policy
(Type 2) Advertise MAC/IP binding ARP suppression
MAC mobility Workload Mobility
Inclusive Multicast Route Auto discovery of multicast tunnel Support multicast even
(Type 3) endpoints & mcast tunnel type when core doesnt
Ethernet Segment Route Auto discovery of redundancy group A/A and S/A MHD &
(Type 4) MHN support
IP Prefix Route IP Prefix advertisement (not for IP IP route aggregation
(Type 5) host advertisement) Interop w/ L3VPN
EVPN BGP Routes RFC7432
EVPN defines a new BGP NLRI used to carry all EVPN routes
BGP Capabilities Advertisement used to ensure that two speakers support
EVPN NLRI (per RFC4760)
AFI 25: L2VPN, SAFI 70: EVPN [1] Ethernet Auto-Discovery (AD) Route
[2] MAC Advertisement Route
1 byte Route Type [3] Inclusive Multicast Route
[4] Ethernet Segment Route
1 byte Length [5] IP Prefix Route
Variable Route type specific
EVPN NLRI

EVPN Operation
Segment Auto-Discovery VPN Auto-Discovery
Multicast Tunnel Endpoint

ESI Auto-Sensing
Discovery
Redundancy Group Membership

Auto-Discovery
DF Election & VLAN Carving
ESI Label & MH type Discovery
ESI Auto-Sensing
ESI (10B) can be auto-generated1
from CEs LACP information ->
concatenation of CEs LACP
System Priority + Sys ID + Port Key System System MAC
Segment Auto-Discovery Example:
Priority Address
Port Key
0000. 0011.0022.0033.0018 2 bytes 6 bytes 2 bytes

PE1 PE3
ESI Auto-Sensing3 LACP PDU

exchange
CE LACP info:
LACP System ID (MAC) (6B)
e.g. 0011.0022.0033 CE1 CE3
LACP System Priority (2B)
e.g. 0000 MPLS
LACP Port Key (2B)
e.g. 0018
PE2 PE4
Redundancy Group Membership Auto-Discovery
PE 1 Eth Segment Route

RD = RD10 RD RD unique per
ESI = ESI1 adv. PE
Segment Auto-Discovery
MAC address portion ES-Import Route Target
of ESI (6B) e.g. 0011.0022.0033
PE1 PE3
ESI Auto-Sensing
PE4
CE1
MPLS
Auto-Discovery PE1000
PE2
Exchange ofPE4
Ethernet
PE 2 Eth Segment Route Segment Routes
RD = RD20
ESI = ESI1
ES-Import Route Target
e.g. 0011.0022.0033
Ordered List of discovered PEs
DF Election & VLAN Carving Modulo Operation
starting from zero (lowest IP add)
VID mod N PE Ordered List

VID (N = # of PEs)
(e.g. VID mod 2) Position PE
Result of modulo 0 PE1
100 0
Segment Auto-Discovery operation is used to
1 PE2
determine DF and 101 1
BDF status 102 0 Example:
103 1 PE1 DF for VIDs 100, 102
PE1 BDF for VIDs 101, 103
ESI Auto-Sensing PE1 PE3
Redundancy Group Membership CE1 CE3

Auto-Discovery
MPLS
DF Election & VLAN Carving PE Ordered List

Position PE
Modulo Operation 0 PE1
PE2 PE4
VID (VID mod 2) 1 PE2
100 0
Example:
101 1
PE2 DF for VIDs 101, 103
102 0 PE2 BDF for VIDs 100, 102
103 1 BRKMPL-2333 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
ESI Label & MH type Discovery
PE1 Eth A-D per ES
RD = RD-1a
ESI1
Eth Tag = MAX-ET
Label = 0
Segment Auto-Discovery ESI Label ext. com
L1
PE1 PE3
ESI Auto-Sensing
CE1
Redundancy Group Membership ESI1
MPLS
Auto-Discovery
DF Election and VLAN Carving PE2 Eth A-D per ES

RD = RD-1b PE2 PE4
ESI1
ESI Label & MH type Discovery Eth Tag = MAX-ET

Label = 0
ESI Label ext. com
L2
Multicast Tunnel Endpoint Discovery
PE 1 Inclusive Multicast Route
Tunnel Type Ingress RD = RD-1a
RD RD unique per
Replication or P2MP LSP PMSI Tunnel Attribute adv. PE per EVI
Mcast MPLS Label used to Tunnel Type (e.g. Ing. Repl.)

transmit BUM traffic - Label (e.g. L1)
downstream assigned (ing.
VPN Auto-Discovery repl.) or upstream assigned
RT ext. community
(Aggregate Inclusive P2MP RT-a
LSP2)
PE1 PE3
RT RT associated with a
Multicast Tunnel Endpoint given EVI
Discovery
CE1 CE3
MPLS

RD = RD-2a
PMSI Tunnel Attribute
Tunnel Type (e.g. Ing. Repl.) PE2 PE4
Label (e.g. L2)
RT ext. community
RT-a PMSI - P-Multicast Service Interface
BUM Broadcast / Unknown Unicast / Multicast
EVPN BGP route 0x4 - Ethernet Segment Route
Usage:
Auto-discovery of multi-homed Ethernet Segments
Designated Forwarder election
Tagged with ES-Import Extended Community

PEs apply route filtering based on ES-Import community. Thus, Ethernet Segment route is imported
only by the PEs that are multi-homed to the same Ethernet segment
Unique per Advertising PE

8 bytes RD
ESI of Ethernet Segment
10 bytes Ethernet Segment Identifier
IP address length
1 bytes IP Address Length
IPv4 or IPv6 address
4 or 16 bytes Originating Routers IP add.
Route Type specific encoding of E-VPN NLRI
ES-Import RT Extended Community
Usage:
Sent with Ethernet Segment route
Limits the scope of Ethernet Segment routes distribution to PEs connected to the same multi-homed
Segment
0x06
0x02 MAC Address portion of the ESI

6 bytes ES-Import
EVPN BGP Route 0x1 Ethernet Auto-discovery Route
This route has two flavors:
Per-ES Ethernet A-D route Per-EVI Ethernet A-D route
Advertise the Split-Horizon Label associated with an Advertise VPN label used for Aliasing or Backup-Path
Ethernet Segment
For AA or SA MH indication
Used for MAC Mass-Withdraw
Unique per Advertising PE per EVI

Unique per Advertising PE
8 bytes RD ESI of Ethernet Segment
ESI of Ethernet Segment
10 bytes Ethernet Segment Identifier Set to VLAN or I-SID for VLAN-Aware
MUST be set to MAX-ET Bundling Service interface, otherwise 0
4 bytes Ethernet Tag ID
(0xFFFFFFFF)
3 bytes MPLS Label VPN (Aliasing) Label per (ESI,
MUST be set to 0 Ethernet Tag)
ESI Label extended community ESI-2
MAC1 MAC2
Agg1 PE1 PE3 Agg2
Usage: ESI-1
Sent with Ethernet AD Route per ES

Advertises the Split-Horizon Label for the Ethernet Segment
PE4
PE2
Indicates the Redundancy Mode: Single Active vs. All-Active
0x06 PE1 advertises in BGP a split-horizon label associated

with the ESI-1 (in the Ethernet AD route)
0x01 Bit 0: Redundancy Mode Split-horizon label is only used for multi-destination
(single active vs. all active) frames (unknown unicast, mcast, bcast)
Flags
When PE1 wants to forward a multi-destination frame, it
Reserved Set to 0 appends this SH label to the packet
PE2 uses this label to perform split-horizon filtering
ESI MPLS Label Ethernet Segment Split- for frames destined to ESI-1 - e.g., a frame originated by
Horizon Label a segment must not be received by the same segment
EVPN BGP route 0x3 Inclusive Multicast
Usage:
Multicast tunnels used to transport Broadcast, Multicast and Unknown Unicast frames
(BUM)

8 bytes RD Set to VLAN or I-SID for VLAN-Aware
4 bytes Ethernet Tag ID Bundling Service interface, otherwise 0
1 bytes
IP address length
IP Address Length
IPv4 or IPv6 address
4 or 16 bytes Originating Routers IP add.
PMSI Tunnel Attribute RFC6514
Flags based on RFC6514

1 bytes Flags Ingress Replication/mLDP etc.
1 bytes Tunnel Type
Multicast MPLS Label
3 bytes MPLS Label
When the Tunnel Type is set to Ingress Replication, the Tunnel Identifier
variable Tunnel Identifier carries the unicast tunnel endpoint IP address of the local PE that is to be this
PE's receiving endpoint address for the tunnel.

EVPN Operation
MAC Address Reachability
MAC/IP Advertisement Advertise MAC (and IP) Per MAC (and IP)
Route (Type 2) reachability policy RR
Advertise MAC/IP binding ARP suppression
MAC Mobility Workload Mobility
PE1 PE3
CE3
ESI1 PE4 CE4
MAC1 CE1
PE2
PE1 & PE2 learns MAC1 from CE1 and advertises in BGP to all other PEs with ES field in the MAC/IP
advertisement set to ESI1
PE3 and PE4 learn that MAC1 sits behind ESI1 which in turn sits behind PE1 & PE2
PE3 and PE4 now know for packets destined to CE1, they can load balanced between PE1 and PE2
ARP Broadcast Suppression
Route Type Usage Benefits Challenge:
How to reduce ARP broadcasts over the
MAC/IP Advertisement Advertise MAC (and IP) Per MAC (and IP) MPLS/IP network, especially in large
Route (Type 2) reachability policy
Advertise MAC/IP binding ARP suppression scale virtualized server deployments?
MAC Mobility Workload Mobility
MAC3, IP3
PE1 PE3
MAC1, IP1
CE1 CE3
PE4 CE4
PE2 3. ARP Request (IP1)

4. ARP Reply (IP1)
Act as ARP
CE1 sends out an ARP request for CE3s IP3 proxy for IP1.
PE1 snoops the ARP packet and learns (MAC1, IP1). It adds MAC1 to its MAC-VRF, MAC1/IP1 binding to its ARP
cache. It also advertises this binding to all other PEs in BGP and floods this initial ARP request.
All other PEs learn of (MAC1, IP1). They add the MAC1 to their MAC-VRFs and add (MAC1, IP1) to their ARP cache.
Now, when CE4 sends an ARP request for IP1, PE4 has the binding info and can provide an ARP response (e.g., ARP
proxy).
MAC Mobility
Challenge:
Route Type Usage Benefits How to handle MAC move ?
MAC/IP Advertisement Advertise MAC (and IP) Per MAC (and IP)
Route (Type 2) reachability policy
Advertise MAC/IP binding ARP suppression
MAC Mobility Workload Mobility PE1 PE3
MAC1, IP1
CE1 CE3
PE4
MAC1, IP1
PE2
At T0, PE1 learn the MAC1, and advertise to all other PEs
At T1, MAC1 move to the PE3. PE1 is not aware of this
PE3 learn the MAC1. It will overwrite the MAC route learnt from PE1
PE3 will advertise MAC1 to all other PEs with sequence number +1
All other PE will overwrite the MAC route
Original PE1 will withdraw its old route
Split Horizon Filtering Challenge:
How to prevent flooded traffic from
echoing back to a multi-homed
Route Type Usage Benefits Ethernet Segment?
Ethernet A-D Route Advertising Split-Horizon Loop avoidance ESI-1 ESI-2
(Type 1) Label even transient
Aliasing Efficient load CE1 PE1 PE3 CE3
Mass Withdraw of Fast convergence
addresses balancing Echo !
CE5
SH/AA MH Indication Per-site policy
CE4 PE2 PE4
PE advertises in EVPN Ethernet AS route with a split-horizon label (ESI MPLS

Label) associated with each multi-homed Ethernet Segment
Split-horizon label is only used for multi-destination frames (Unknown Unicast,
Multicast & Broadcast)
When an ingress PE floods multi-destination traffic, it encodes the Split-Horizon
label identifying the source Ethernet Segment in the packet
Egress PEs use this label to perform selective split-horizon filtering over the
attachment circuit
Challenge:
Aliasing How to load-balance traffic towards a multi-
homed device across multiple PEs when MAC
addresses are learnt by only a single PE?
Ethernet A-D Route Advertising Split-Horizon Loop avoidance I can reach

(Type 1) Label even transient MAC1 via ESI1
Aliasing Efficient load I can
MAC1 ESI1 PE1
Mass Withdraw of Fast convergence reach

ESI1
addresses balancing (All-Active)
PE2
SH/AA MH Indication Per-site policy MAC1 PE1 PE3
MAC1 CE1
CE3
ESI-1 I can PE4 CE4

PE2
reach
PEs advertise in BGP the ESIs of local multi-homed Ethernet Segments. ESI1
(All-Active)
All-Active Redundancy Mode indicated
When PE learns MAC address on its AC, it advertises the MAC in BGP along
with the ESI of the Ethernet Segment from which the MAC was learnt.
Remote PEs can load-balance traffic to a given MAC address across all PEs
advertising the same ESI.
Challenge:
MAC Mass Withdraw How to inform remote PEs of a failure
affecting many MAC addresses quickly while
the control-plane re-converges?
Ethernet A-D Route Advertising Split-Horizon Loop avoidance

(Type 1) Label even transient
Aliasing Efficient load
MAC1, MAC2, .. MACn ESI1
X
Mass Withdraw of Fast convergence I lost ESI1 PE1
addresses balancing MAC1,
PE2
MAC2, PE1 PE3
SH/AA MH Indication Per-site policy MACn
MAC1
CE1 CE3
ESI-1 PE4 CE4

PE2
PEs advertise two sets of information:
MAC addresses along with the ESI from the address was learnt
Connectivity to ESI(s)
If a PE detects a failure impacting an Ethernet Segment, it withdraws the route for the
associated ESI.
Remote PEs remove failed PE from the path-list for all MAC addresses associated with an ESI.
This effectively is a MAC mass-withdraw function.
EVPN BGP Route 0x2 MAC Advertisement
ESI of Ethernet Segment on which MAC
Address was learnt. All 1s ESI for PBB-EVPN
8 bytes RD
Set to VLAN or I-SID for VLAN-Aware
10 bytes Ethernet Segment Identifier
Bundling Service interface, otherwise 0
4 bytes Ethernet Tag ID
Allows for MAC Address summarization, i.e.
1 byte MAC Address Length hierarchical MAC Addresses. Typically set to 48
6 bytes MAC Address
Could be C-MAC Address (EVPN) or B-MAC
1 byte IP Address Length Address (PBB-EVPN)
To distinguish IPv4 vs. IPv6 addresses.
4 or 16 IP Address
3 bytes Used for ARP flood suppression or for

MPLS Label1
Integrated Routing and Bridging (IRB).
3 bytes MPLS Label2
MAC & IP Labels - downstream assigned
MAC Mobility extended community
Used to tag the MAC Advertisement route
EVPN: Indicates that a MAC address has moved from one PE to another
0x06
0x00
Set to 0
2 bytes Reserved
Indicates the count of MAC address mobility
4 bytes Sequence Number events
EVPN BGP Route 0x1 Ethernet Auto-discovery Route
This route has two flavors:

Per-EVI Ethernet A-D route
Advertise VPN label used for Aliasing or Backup-Path

RD ESI of Ethernet Segment
Ethernet Segment Identifier Set to VLAN or I-SID for VLAN-Aware
Bundling Service interface, otherwise 0
Ethernet Tag ID
MPLS Label VPN (Aliasing) Label per (ESI,

Ethernet Tag)
Startup Sequence
Operation
Life of a Packet
Ingress Replication Multi-destination Traffic Forwarding
PE1 receives broadcast Mcast MPLS
During start-up traffic from CE1. PE1 PSN MPLS label Label assigned by
sequence, PE1, PE2, forwards it using ingress to reach PE3 PE3 for incoming
PE3, PE4 sent Inclusive replication 3 copies BUM traffic on a
Multicast route which created given EVI PE3 as DF, it
include Mcast label
During start-up forwards BUM
sequence, PE2 sent Per- PE1 PE3 PE1 PE3 traffic towards
ESI Ethernet AD route segment
with ESI MPLS label VID 100
(split-horizon) (see SMAC: M1
below) DMAC: F.F.F L3
CE1 CE3 CE1 CE3

L2 L5
MPLS MPLS
PE 4 Inclusive Multicast
PE 2 Eth A-D Route (Per-ESI)
Route
L4
RD = RD20
RD = RD-4a
ESI = ESI1
ESI MPLS Label ext. comm.
Tunnel Type = Ing. Repl.
Redund. Flag = All-Active PE2 PE4 PE2 ESI (split-horizon) PE4
Label = L4 MPLS label
Label = L5 allocated by PE2
ESI MPLS Label used by RT ext. community
PE2 drops BUM PE4 non-DF for
RT ext. community local PEs for split-horizon - for segment ES1
Mcast MPLS Label used to RT-a traffic originated given EVI drops
RT-a, RT-b, RT-c, RT-d downstream assigned (for on ES1 BUM traffic
transmit BUM traffic -
ingress replication)
downstream assigned (for
ingress replication) BRKMPL-2333 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Life of a Packet (cont.)
Unicast Traffic Forwarding
PE1 MAC Route
MP2P VPN Label PE3 forwards
downstream allocated label RD = RD-1a
MAC advertised PSN MPLS label MP2P VPN Label traffic destined to
used by other PEs to send ESI = ESI1 assigned by PE1 M1 based on RIB
by route to reach PE1
traffic to advertised MAC for incoming traffic information (PE1)
MAC = M1
for the target EVI
Label = L1
RT ext. community
PE1 PE3 PE1 PE3
RT-a VID 100
SMAC: M2
VID 100 DMAC: M1
SMAC: M1
DMAC: F.F.F
L1
CE1 CE3 CE1 CE3
MPLS MPLS
PE2 PE4 PE2 PE4
PE3 RIB Path List

VPN MAC ESI NH
RT-a M1 ES1 PE1
Unicast Forwarding and Aliasing PE3 forwards PE3 forwards
traffic on a flow traffic on a flow
PE1 MAC Route MP2P VPN
MP2P VPN Label (flow 1) based on (flow 2) based on
RD = RD-1a Label
downstream allocated label PSN MPLS label RIB information RIB information
MAC advertised assigned by
used by other PEs to send ESI = ESI1 to reach PE1 (towards PE1) (towards PE2)
by route PE1 for
traffic to advertised MAC
MAC = M1 incoming for
target EVI
Label = L1
During start-up
sequence, PE1 sent Per- RT ext. community VID 100
EVI Ethernet AD route PE1 PE3 PE1 PE3 SMAC: M3
RT-a
DMAC: M1
VID 100
VID 100 SMAC: M4
SMAC: M1 DMAC: M1
DMAC: F.F.F L1
CE1 CE3 CE1 CE3
During start-up sequence, MPLS MPLS

PE2 sent Per-EVI Ethernet L2
AD route (see below)
PE 2 Eth A-D Route (Per-EVI)

RD = RD-2a PE2 PE4 PE2 PE4
ESI = ESI1
Aliasing MPLS Label used PE3, PE4 RIB Path List
Label = L2 Aliasing MPLS
by remote PEs to load- VPN MAC ESI NH PSN MPLS label Label assigned by
RT ext. community balance among local PEs to reach PE2 PE2 for (ES1, EVI)
RT-a M1 ES1 PE1
RT-a pair
PE2
E-VPN Operational Scenarios
PE3 MAC Route
MAC Mobility PE1 MAC Route RD = RD-3a
RD = RD-1a ESI = ESI2
1 4
ESI = ESI1 3 MAC = M1
PE1 advertises MAC Host M1 moves After host sends traffic
route for M1. Route MAC = M1 Label = L3
from CE1 to CE3s at new location, PE2
may include MAC Label = L1 location MAC Mobility ext. now adv MAC route
mobility community community. for M1 incrementing
MAC Mobility ext.
community sequence # in MAC
Seq. Num = 2
mobility community
PE1 Seq. Num = 1 PE3 PE1 RT ext. community PE3
RT ext. community RT-a VID 100
VID 100 SMAC: M1
RT-a
SMAC: M1 DMAC: M2
DMAC: M2
CE1 CE3 CE1 CE3

M1
MPLS M1 MPLS M1
5
PE1 withdraws its
2 M1 route and
PE3 / PE4 install installs a new one
M1 route towards pointing to PE3
PE2 PE4 PE1 PE2 PE4
PE3 / PE4 RIB Path List PE1 / PE2 RIB Path List
VPN MAC ESI NH VPN MAC ESI NH
RT-a M1 ES1 PE1 RT-a M1 ES2 PE3
E-VPN Failure Scenarios / Convergence
Link / Segment Failure Active/Active per Flow
2 7
PE1 withdraws
PE1 withdraws Per-ESI individual MAC
Ethernet AD route for advertisement routes
1 failed segment related to failed
PE1 detects failure segment
of one of its
attached segments PE1 PE3 PE1 PE3
CE1 CE3 CE1 CE3
3 MPLS 4 MPLS
PE1 withdraws Ethernet Mass withdrawal - PE3
Segment Route / PE4 remove PE1 from
path list for all MAC
addresses of failed
segment (ES1)
5 PE2 PE4 6 PE2 PE4

PE2 recalculates PE3, PE4 RIB Path List PE3, PE4 RIB Path List
DF/BDF. Becomes DF PE2 adv. M1 MAC route
for all EVIs on segment VPN MAC ESI NH after CE traffic is VPN MAC ESI NH
hashed towards PE2
RT-a M1 ES1 PE1 RT-a M1 ES1 PE2
PE2
E-VPN Failure Scenarios / Convergence
PE Failure
2
BGP RR / PE3 detects
1 BGP session time-out
PE1 experiences a with PE1
node failure (e.g.
power failure) PE1 PE3 PE1 PE3
CE1 CE3 CE1 CE3
MPLS MPLS
6
2 3 PE3 / PE4 will forward
2 BGP RR / PE4 PE3 / PE4 invalidate M1 traffic towards PE2
BGP RR / PE2 detects detects BGP routes from PE1
BGP session time-out session time-
with PE1
PE2 out with PE1 PE4
5 PE2 PE4
PE2 adv. M1 MAC route
4 PE3, PE4 RIB Path List after CE traffic is PE3, PE4 RIB Path List
VPN MAC ESI NH hashed towards PE2 VPN MAC ESI NH
PE2 reruns DF election.
Becomes DF for all RT-a M1 ES1 PE1
PE1 RT-a M1 ES1 PE2
EVIs on segment
PE2
EVPN-VPWS
EVPN-VPWS Does it Better than Legacy VPWS !!
Service Additional Capabilities

E-Line All-active & single-active multi-homing support
Both single-segment & multi-segment support
Discovery & signaling via single protocol BGP
EVPN BGP route type
Route type Usage EVPN EVPN VPWS
0x1 Ethernet Auto-Discovery MAC Mass-Withdraw
(A-D) Route Aliasing (load balancing)
Split-Horizon
Tagged with ESI Label Extended Community
0x2 MAC Advertisement Route Advertise MAC addresses
Provide MAC / IP address bindings for ARP NOT used
broadcast suppression
Tagged with MAC Mobility Extended
Community
0x3 Inclusive Multicast Route Multicast tunnels used to transport
Broadcast, Multicast and Unknown Unicast NOT used
frames (BUM)
Tagged with PMSI tunnel attribute (P tunnel
type & ID) RFC6514
0x4 Ethernet Segment Route Auto discovery of Multi-homed Ethernet
Segments, i.e. redundancy group discovery
Designated Forwarder (DF) Election
Tagged with ES-Import Extended Community
EVPN BGP Extended Community
Attribute Usage Tagged BGP EVPN EVPN VPWS

route
ESI label Extended Split-Horizon for Ethernet Ethernet A-D

Community Segment. Route
Indicate Redundancy Mode
(Single Active vs. All-Active)
ES-Import Extended Limit the import scope of the Ethernet
Community Ethernet Segment routes. Segment Route
MAC Mobility Extended E-VPN: Indicate that a MAC MAC
Community address has moved from one Advertisement Not used
segment to another across PEs. Route
PBB-EVPN: Signal C-MAC
address flush notification
EVPN VPWS
Control-plane
attachment circuit
Benefits of EVPN applied to point-to-point advertisement over the
services Core
No signaling of PWs. Instead signals MP2P LSPs

instead (ala L3VPN) VPWS Service Config:
EVI = 100
All-active CE multi-homing (per-flow LB) Local AC ID = AC2
PE1 PE2 Remote AC ID = AC1
Single-active CE multi-homing (per-service LB)
CE1 CE2
Relies on a sub-set of EVPN routes to

ES1 MPLS ES2
advertise Ethernet Segment and AC
reachability VPWS Service Config:
PE discovery & signaling via a single protocol EVI = 100
Local AC ID = AC1
BGP
Remote AC ID = AC2
Per-EVI Ethernet Auto-Discovery route
Handles double-sided provisioning with remote PE
auto-discovery
I have a P2P service that BGP Eth. Auto-
needs to communicate Discovery Route
Under standardization: draft-ietf-bess-evpn- with the PE(s) that own EVPN NLRI
vpws of AC = AC2 AC AC1 via PE1
EVPN VPWS Operation Single-homed
ESI 10 bytes ESI as specify
by EVPN Ethernet segment
IETF draft zero for single- 3
homed RD RD unique per adv. PE
PE 1 Eth A-D Route
per EVI
RD = RD-1a
Eth.Tag ID 4-bytes local
ESI = ES1 (0)
AC-ID
Eth.Tag ID = AC1 RT RT associated with a
given EVI
Label (e.g. X)
MPLS Label (downstream
assigned) used by remote RT ext. community VPWS Service Config:
PEs to reach segment EVI = 100
RT-a
Local AC ID = AC2
PE1 PE2 Remote AC ID = AC1 ES2 Since CE2 is single
CE1 CE2 homed to PE2, ES2 = 0
1 ES1 MPLS ES2

2
VPWS Service Config:
EVI = 100
PE 2 Eth A-D Route
Local AC ID = AC1
Remote AC ID = AC2 RD = RD-2a
5 ESI = ES2 (0) 6
PE1 RIB Path List Path List
Eth.Tag ID = AC2 PE2 RIB
NH
VPN MAC ESI Eth.TAG Label (e.g. Y) VPN MAC ESI Eth.TAG
NH
PE2
RT ext. community PE1
RT-a - 0 AC2 4 RT-a - 0 AC1
RT-a
EVPN VPWS Operation Single-active
PE 1 Eth A-D Route RD RD unique per adv. PE
per EVI
by EVPN Ethernet segment RD = RD-1a
IETF draft
ESI = ES1
3
Eth.Tag ID = AC1
Eth.Tag ID 4-bytes local Label (e.g. X) RT RT associated with a
AC-ID given EVI
RT ext. community
RT-a
assigned) used by remote PE1 VPWS Service Config: Only one PE (PE1)
PEs to reach segment EVI = 100 shows as next hop for
Single-Active == per-vlan
Local AC ID = AC2 the remote AC
load-balancing CE-PEs
ES1 PE3 Remote AC ID = AC1
Two bundles on CE VPWS Service Config:
CE1 CE2
device EVI = 100
Local AC ID = AC1 ES2 Since CE2 is single
Remote AC ID = AC2 MPLS ES2
2 homed to PE2, ES2 = 0
1
VPWS Service Config: ES1
EVI = 100 PE 3 Eth A-D Route 6
Local AC ID = AC1 RD = RD-2a
PE2
5 Remote AC ID = AC2
ESI = ES2 (0)
PE3 RIB
Path List
PE1 & PE2 RIB Path List VPN MAC ESI Eth.TAG
Eth.Tag ID = AC2 NH
NH
VPN MAC ESI Eth.TAG Label (e.g. Y) RT-a - ES1 PE1
PE3
RT ext. community
RT-a - 0 AC2 4 RT-a - ES1 PE2
RT-a RT-a - ES1 AC1 PE1
EVPN VPWS Operation All-active
PE 1 Eth A-D Route RD RD unique per adv. PE
per EVI
by EVPN Ethernet segment RD = RD-1a
IETF draft
ESI = ES1
3
Eth.Tag ID = AC1
Eth.Tag ID 4-bytes local Label (e.g. X) RT RT associated with a
AC-ID given EVI
RT ext. community
RT-a
assigned) used by remote PE1 VPWS Service Config: Both PEs (PE1/PE2)
PEs to reach segment EVI = 100 shows as next hop for
ALL-Active == per-flow
Local AC ID = AC2 the remote AC
load-baancing CE-PEs
ES1 PE3 Remote AC ID = AC1
Single bundle on CE VPWS Service Config:
CE1 CE2
device EVI = 100
ES2 Since CE2 is single
Local AC ID = AC1
Remote AC ID = AC2 MPLS ES2
2 homed to PE2, ES2 = 0
1
VPWS Service Config: ES1
EVI = 100 PE 3 Eth A-D Route 6
Local AC ID = AC1 RD = RD-2a PE3 RIB
PE2
5 Remote AC ID = AC2
ESI = ES2 (0)
Path List
VPN MAC ESI Eth.TAG
PE1 & PE2 RIB Path List NH
Eth.Tag ID = AC2
NH RT-a - ES1 PE1
VPN MAC ESI Eth.TAG Label (e.g. Y)
PE3 RT-a - ES1 PE2
RT ext. community
RT-a - 0 AC2 4 RT-a - ES1 AC1 PE1,PE2
RT-a
EVPN Deployment:
DC Fabric Evolution w/ EVPN-VxLAN
The Evolution of the DC Fabric
IP Fabric: IP Fabric: VXLAN/EVPN, SDN

VXLAN, ASR9K)
DP Learning Spine-leaf
Virtual overlay across physical
L2 Fabric: boundary
FP/Trill VXLAN: Ultra-high scale
Efficient forwarding: L3 ECMPs
L2 Fabric: Legacy VLAN, STP EVPN control plane
L2/L3 boundary: limited SDN enabled VXLAN and
mobility service chaining provisioning
4K VLAN: Limited scale
Inefficient forwarding: STP
Complex VLAN provisioning
Vendor specific L2
enhancement
DC Fabric IP Underlay
Edge Device
Edge Device
Local LAN
Local LAN Segment
IP Interface
Segment
Physical
Host Physical
Edge Device
Local LAN Host
Segment
Virtual Switch
Virtual Hosts Edge device: could be

physical Leaf/ToR, or virtual
forwarder
DC Fabric VXLAN Overlay
VTEP
VTEP
V V
Local LAN
Local LAN Segment
Segment Encapsulation
Physical
Host VTEP
V Physical
Local LAN Host
Segment
Virtual Switch
VTEP VXLAN Tunnel End-Point
VNI/VNID VXLAN Network Identifier
Virtual Hosts
Next-Hop MAC Address
VXLAN Frame Format Dest. MAC Address 48
Src VTEP MAC Address
MAC-in-IP Encapsulation Src. MAC Address 48
VLAN Type 14 Bytes

16
0x8100 (4 Bytes Optional) IP Header
72
Misc. Data
VLAN ID
16
Outer MAC Header Tag
Protocol 0x11 (UDP) 8
Ether Type
16
Underlay
0x0800 Header
16 20 Bytes
Checksum
Source IP 32
50 (54) Bytes of Overhead
Outer IP Header Src and Dst

Source
16 Dest. IP 32 addresses of
Port
the VTEPs
UDP Header VXLAN Port 16 Hash of the inner L2/L3/L4 headers of
8 Bytes the original frame.
UDP Length 16 Enables entropy for ECMP Load
VXLAN Header UDP 4789 balancing in the Network.
Checksum 0x0000 16
Overlay
VXLAN Flags
VNI - allows
8
Original Layer-2 Frame RRRRIRRR for 16M
Reserved 24
possible
8 Bytes Segments
VNI 24
Reserved 8
VXLAN Evolution BGP EVPN Control Plane
Workload MAC / IP Addresses learnt by
Multi-Protocol BGP (MP-BGP) based
Control-Plane using EVPN NLRI
Advertises Layer-2 & Layer-3 Address-to- RR RR
VTEP Association
Make Forwarding decisions at VTEPs for
Layer-2 (MAC) and Layer-3 (IP),
Integrated Route/Bridge (IRB)
V1 V2
Reduces impact of ARP on the Network
Standards Based
draft-ietf-bess-evpn-overlay
draft-ietf-bess-evpn-inter-subnet-forwarding RR BGP Route-Reflector
draft-ietf-bess-evpn-prefix-advertisement V3 iBGP Adjacency
MP-BGP
EVPNforControl
VXLANPlane
EVPN
Control Plane Distribution
Reachability
EVPN route type 2: Host route, type 5: Subnet Route
BGP Update Spine

Host-MAC
Host-IP
Internal IP Subnet
External Prefixes
VTEP VTEP VTEP VTEP

Leaf
Use MP-BGP with EVPN Address Family on leaf nodes to distribute internal
host MAC/IP addresses, subnet routes and external reachability information
MP-BGP enhancements to carry up to 100s of thousands of routes with
reduced convergence time
EVPN Control Plane -- Host Advertisement
Install host info to RIB/FIB:
Install host info to RIB/FIB:
H-MAC-1 MAC table
BGP Update: H-MAC-1 MAC table
H-IP-1 VRF IP host table 4 BGP Update:
H-MAC-1 4 H-IP-1 VRF IP host table
H-MAC-1
MAC Host VNI VTEP H-IP-1
IP
H-IP-1 3 Route VTEP-1 MAC Host VNI VTEP
VTEP-1 3
Reflector VNI-1 IP
H-MAC-1 H-IP- VNII-1 VTEP-1 VNI-1
1 H-MAC-1 H-IP-1 VNII-1 VTEP-1
2
VTEP-2
VTEP-3 BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1
VTEP-1
1
MAC Host VNI VTEP

IP
H-MAC-1 H-IP-1 VNII-1 VTEP-1

Local learning of host info: H-MAC-1
H-IP-1
H-MAC-1 (MAC table)
VLAN-1 /VNI-1
H-IP-1 (VRF IP host table )
VXLAN BGP Control Plane
EVPN Control Plane --- Host Movement
NLRI: Spine
Host H-MAC-1, H-IP-1
NVE VTEP-3
VNI 5000
Ext. Community:
Encapsulation: VXLAN VTEP-1 VTEP-2 VTEP-3 VTEP-4 Leaf
Cost
Sequence number: 1 Host 1 Host 1
H-MAC-1 H-MAC-1
H-IP-1 H-IP-1
VLAN 10 VLAN 10
VXLAN 5000 VXLAN 5000
1. Host 1 moves to VTEP-3 from VTEP-1
2. VTEP-3 detects Host 1, sends MP-BGP update for Host 1 with its own VTEP address and a new seq #1
3. Other VTEPs learn about the new route of Host 1 MAC IP VNI Next-Hop Encap Seq#
H-MAC-1 H-IP-1 5000 VTEP-3 VXLAN 1
Inter-VXLAN Routing EVPN IRB
Routing ?
IP Transport Network
SVI A SVI B
VTEP-1 VTEP-2 VTEP-3 VTEP-4
Host 1 Host 2
H-MAC-1 H-MAC-2
H-IP-1 H-IP-2
VNI-A VNI-B
Asymmetric Symmetric
Bridging & Routing on the ingress VTEP Bridging & Routing on both the ingress and
and bridging only on the egress VTEP the egress VTEPs
Requires each VTEP to have all MAC A VTEP only needs to maintain MACs for
addresses of their tenants in their ARP its directly attached endpoints. Optimal
tables can result in scale issue. utilization of VTEP resources
Cisco follows Symmetric IRB

MAC- MAC-
VRF VRF
Symmetric EVPN IRB (1)
Routing on both ingress and egress
VTEPs
Layer-3 VNI
Layer-3 VNI (VRF VNI)
Tenant VPN indicator
One per tenant VRF
VTEP Router MAC Layer-2 VNI Layer-2 VNI
(Network VNI) (Network VNI)
Ingress VTEP routes packets onto the
Layer-3 VNI VTEP VTEP VTEP VTEP
Egress VTEP routes packets to the

destination Layer-2 VNI
Symmetric EVPN IRB (2)
Egress VTEP
routes packets
Ingress VTEP S-IP: VTEP-1 from L3 VNI to the
D-IP: VTEP-4 destination VNI-
routes packets VNI: L3 VNI
from source VNI-A 1 2 B/VLAN.
to L3 VNI. D-MAC S-MAC: Router-MAC-1
in the inner header D-MAC: Router-MAC-4
S-IP: H-IP-1
is the egress D-IP: H-IP-2 S-MAC: MAC-VTEP4 local port
VTEP router MAC
D-MAC: H-MAC-2
VNI L3 L3 VNI
S-IP: H-IP-1
A VNI VNI B
D-IP: H-IP-2
VTEP-1 VTEP-4
VTEP-2 VTEP-3
Router MAC-1 Router MAC-4
S-MAC: H-MAC-1
D-MAC: GW-MAC Host 1 Host 2
S-IP: H-IP-1 H-MAC-1 H-MAC-2
D-IP: H-IP-2 H-IP-1 H-IP-2
VNI-A VNI-B
(Distributed) Anycast Gateway in with EVPN IRB
# VLAN to VNI mapping

vlan 200
vn-segment 5200
# Anycast Gateway MAC, identically configured on all VTEPs

fabric forwarding anycast-gateway-mac 0002.0002.0002
# Distributed IP Anycast Gateway (SVI)

The same anycast gateway virtual IP # Gateway IP address needs to be identically configured on all
address and MAC address are VTEPs
configured on all VTEPs in the VNI. interface vlan 200
no shutdown
vrf member Tenant-A
ip address 20.0.0.1/24
fabric forwarding mode anycast-gateway
SVI SVI SVI SVI

GW IP GW IP GW IP GW IP
GW MAC GW MAC GW MAC GW MAC
VTEP VTEP VTEP VTEP
Host 1 Host 2 Host 3 Host 4

MAC1 MAC2 MAC3 MAC4
IP 1 IP 2 IP 3 IP 4
VLAN A VLAN A VLAN A VLAN A
VXLAN A VXLAN A VXLAN A VXLAN A
EVPN IRB Anycast Gateway Deployment Options
Distributed vs. Centralized Anycast Gateway
clien
t Internet VPN
clien
Client Client t
IP-VPN
Internet Internet IP-VPN
Centralized Branc
VPLS/PW/EVP h Internet
N
Anycast GW VPLS/PW/EVP
DC Gateway
IRB N
(IRB anycast DC Gateway
DC-1 IRB
GW) DC-2 (L3 GW)
WAN
L2 EVPN
(per-BD VNI) L3 EVPN
(per-VRF VNI)
Leaf
Leafonly) IRB
IRB IRB IRB IRB Leaf
(L2 IRB
(IRB anycast
VM1 VM2 L2 only DC fabric L2/L3 DC fabric VM3 VM4 GW)
Distributed
Anycast GW
EVPN Deployment:
DC Fabric and WAN Integration
The Solution Must be End-to-End
Legacy 802.1q handoff

APIC It means multi-pathing is out the
door between DC & WAN
VLANs and sub-interfaces
DC creation
No policy level integration
Small FIB/MAC table size on
Spine
border Leaf, create bottleneck
WAN/DCI
Leaf Leaf bLeaf bLeaf
802.1q?
SDN-DC, VXLAN SDN-WAN, MPLS/Segment Routing
overlay
DC Gateway Seamless DC and WAN Integration
L3 Gateway
L2 Gateway
IRB with Anycast
Gateway
Common MP-BGP (EVPN AF) control plane
VXLAN to MPLS data plane interworking
APIC SDN based auto provisioning: OpFlex,
APIC/VTS
DC Integrated Policy control: WAN optimization
Spine
WAN/DCI
Leaf Leaf bLeaf bLeaf
Integration
SDN-DC, VXLAN Interworking SDN-WAN, MPLS/SR
overlay
Scalable, Resilient, Optimized, End-to-End 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
DC Gateway L3 Gateway EVPN/VXLAN to IP-VPN/MPLS interworking
EVPN/VXLAN to global internet
S-N Routing: all active L3 EVPN between Leaf and GW (per-vrf VNI)
Leaf is the L3 default gateway for VMs (with
EVPN IRB anycast GW) and does inter-vxlan
routing
Internet VPN
clien
Client Client t
Internet
Branch IP-VPN
Internet
DC DC-1 Gateway
WAN DC-2
Gateway
Spine L3 EVPN
(per-VRF VNI)
Leaf
Leaf
(L3 anycast
VM1 VM2 L2/L3 DC fabric L2/L3 DC fabric GW)
VM3 VM4
DC Gateway L2 Gateway L2 EVPN/VXLAN in the DC
L2 Stretch: E-W and S-N L2 DCI (E-W): EVPN/VXLAN with EVPN/VPLS
all-active or single-active interworking
L2 to client (S-N): EVPN/VXLAN with VPLS/PW
interworking
clien
VPN
Client t
Branch
VPLS/PW/EV
PN
DC Gateway DC-1 Gateway

WAN DC-2
Spine
L2 EVPN
(per-BD VNI)
Leaf
Leaf
VM1 VM2
L2/L3 DC fabric L2/L3 DC fabric VM3 VM4
Leaf Gateway Gateway

Leaf
L2 EVPN/VXLAN L2 stretch: EVPN/VPLS (MPLS) L2 EVPN/VXLAN
DC Gateway IRB Anycast Gateway
Integrated Routing and Bridging DC fabric is L2 only. All routing on DC gateway
DC gateway is the L3 default gateway for VMs via
EVPN IRB anycast gateway
Support both L2 and L3 for the same VNI at the
same time
Internet VPN
clien
Client Client t
Internet IP-VPN
Branch
Internet
VPLS/PW/EVP
IRB N
DC DC-1 IRB IRB Gateway
IRB DC-2
Gateway WAN (L3 anycast GW)
Spine L2 EVPN
(per-BD VNI)
Leaf Leaf
VM1 VM2 L2 only DC fabric (L2 only)
L2 only DC fabric VM3 VM4
Leaf Gateway Gateway

Leaf 76
L2 EVPN/VXLAN L2 stretch: EVPN/VPLS (MPLS) L2 EVPN/VXLAN
2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IRB Anycast Gateway Deployment Options
Distributed vs. Centralized Anycast Gateway
clien
t Internet VPN
clien
Client Client t
IP-VPN
Internet IP-VPN
Internet Branc
VPLS/PW/EVP
Centralized Internet
h VPLS/PW/EVP
N Anycast GW
N
DC Gateway IRB
(IRB anycast GW) DC-1 DC Gateway
IRB DC-2
WAN (L3 GW)
L2 EVPN
(per-BD VNI) L3 EVPN
(per-VRF VNI)
Leaf IRB IRB IRB IRB

Leafonly)
(L2 IRB IRB Leaf
VM1 VM2 L2 only DC fabric L2/L3 DC fabric (IRB anycast GW)
VM3 VM4
Distributed
Anycast
2016 Cisco and/or its GW
affiliates. All rights reserved. Cisco Public 77
IRB Anycast Gateway Deployment Option
Comparison
Option 1 Option 2
Distributed IRB Anycast Gateway Centralized IRB Anycast Gateway
DC Gateway Router L2 or L3 gateway function EVPN IRB for integrated L2 and L3

Doesnt require IRB function IRB anycast gateway for VMs default
gateway
Leaf EVPN IRB for integrated L2 and L3 L2 EVPN peering across DC

IRB anycast gateway for VMs Cross-DC underlay IP routing is required
default gateway
Pros Optimized E-W inter-vxlan routing Simple DC fabric design: L2 only

Large ARP table on the DC gateway
router
Cons EVPN IRB function across all Leaf Sub-optimal E-W inter-vxlan routing
nodes
Require both L2 and L3 function on
the leaf
DC Gateway the Policy Integration and Auto-
Provisioning: Application-engineered Routing
DC: classification PBTS: steering packet to
and marking SR-TE in the WAN
DC policy domain WAN policy domain DC policy domain
WAN Segment
Routing
VTS/APIC VTS/APIC
DC DC-1
WAN DC-2
Gateway
Spine
Leaf
VM1 VM2
VM3 VM4
High bandwidth flow

Low latency flow 79
BRKMPL-2333 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Example: Application has a preference for disjoint path in dual-plane
WAN networks
ACI Fabric
Customer has the requirement that traffic from

WEB WEB applications RED and BLUE should be
Policy
transported across disjoint paths in the WAN.

Segment - Policy expressed on APIC and delivered by
Routing SR-enabled +WAE WAN
WAN
with WAE
Example: Application requires the lowest latency path in the WAN
ACI Fabric
WAN has cheap capacity via US with higher latency.

WEB WEB
Scarce, expensive capacity via Russia, with lower
Policy
latency.
Tokyo
Segment
Routing Russia Customer identify the applications that require the
US lowest possible latency path on APIC, integration
WAN steers traffic on the path via Russia.
with WAE
Brussels
Summary
EVPN: Next Generation VPN
E-LAN E-LINE E-TREE DC Fabric IRB DCI IP-VPN

(MP2MP (P2P (P2MP (IntraDC (L2/L3 (InterDC)
(L3VPN)
L2VPN) L2VPN) L2VPN) Overlay) Overlay)
VPLS-
VPLS PW VPLS,OTV 4364
ETREE
EVPN
(PBB-) EVPN EVPN EVPN- EVPN- EVPN EVPN-

EVPN VPWS ETREE Overlay IRB DCI L3
References
[FabricPath]: FabricPath http://www.cisco.com/en/US/prod/switches/ps9441/fabric_path_promo.html
[LISP]: Locator/ID Separation Protocol https://datatracker.ietf.org/wg/lisp/charter/
[802.1Qbp] ECMP http://www.ieee802.org/1/files/public/docs2011/new-ashwood-sajassi-ecmp-par-0111-v04.pdf
[EVPN]: BGP MPLS Based Ethernet VPN http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-04
[TRILL]: Transparent Interconnection of Lots of Links https://datatracker.ietf.org/wg/trill/charter/

http://tools.ietf.org/wg/trill/draft-ietf-trill-rbridge-protocol/
[VL2]: VL2: A Scalable and Flexible Data Center Network http://ccr.sigcomm.org/online/?q=node/502
[MOOSE]: Addressing the Scalability of Ethernet with MOOSE http://www.cl.cam.ac.uk/~mas90/MOOSE/MOOSE.pdf
[PORTLAND]: PortLand: A Scalable Fault-Tolerant Layer 2 Data Center Network Fabric http://ccr.sigcomm.org/online/?q=node/503
[SEATTLE]: Floodless in SEATTLE: A Scalable Ethernet Architecture for Large Enterprises

http://www.cs.princeton.edu/~chkim/Research/SEATTLE/seattle.pdf
[MONSOON]: Towards a Next Generation Data Center Architecture: Scalability and Commoditization
http://research.microsoft.com/apps/pubs/default.aspx?id=79348
[VLB]: Valiant Load Balancing in Backbone Networks http://www.stanford.edu/~ashishg/network-algorithms/rui.pdf
Complete Your Online Session Evaluation
Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
Complete your session surveys
through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.
Dont forget: Cisco Live sessions will be available

for viewing on-demand after the event at
CiscoLive.com/Online
Continue Your Education
Demos in the Cisco campus
Walk-in Self-Paced Labs
Lunch & Learn
Meet the Engineer 1:1 meetings
Related sessions
Thank you
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services
Thursday, July 14th, 2016

11:30 am - 12:30 pm, In the Oceanside A room
What to expect from this innovation talk

Insights on market trends and forecasts
Preview of key technologies and capabilities
Innovative demonstrations of the latest and greatest products
Better understanding of how Cisco can help you succeed
Register to attend the session live now or

watch the broadcast on cisco.com
Appendix-A: PBB-EVPN
Startup Sequence &
A Day in Life of a Paket
PBB-EVPN Startup Sequence
Segment Auto-Discovery VPN Auto-Discovery
Multicast Tunnel ID / Endpoint

ESI and B-MAC Auto-Sensing
Discovery

Auto-Discovery
Backbone MAC (B-MAC)

Reachability Advertisement
PBB-EVPN Startup Sequence (cont.)
ESI (10B) can be auto-generated*
from CEs LACP information ->
concatenation of CEs LACP
System Priority + Sys ID + Port Key System System MAC
Segment Auto-Discovery Port Key
Priority Address
Example:
0000. 0011.0022.0033.0018 2 bytes 6 bytes 2 bytes
PE1 PE3
ESI and B-MAC Auto-Sensing LACP PDU

B-MAC
exchange
B-MAC
CE LACP info: CE1 CE3

LACP System ID (MAC) (6B)
e.g. 0011.0022.0033 MPLS
LACP System Priority (2B)
e.g. 0000
LACP Port Key (2B) B-MAC
e.g. 0018
B-MAC
Source B-MAC used at PBB-EVPN PE on a PE2 PE4

given ESI can be auto-generated* from CEs
LACP information -> CEs LACP System ID
MAC with U/L** (Universal / Locally
Administered) bit flipped
(*) ESI and B-MAC can also be manually configured
Example: 0211.0022.0033 (**) U/L is second-least-significant bit of most significant byte
BGP Ethernet Segment Route
RD = RD10 RD RD unique per
ESI = ESI1 advertising PE
Segment Auto-Discovery
MAC address portion ES-Import ext. comm.
of ESI (6B) e.g. 0011.0022.0033
PE1 PE3
CE1 CE3
MPLS
Auto-Discovery
PE2 PE4
RD = RD20
ESI = ESI1
ES-Import ext. comm.
BRKMPL-2333 2016 Cisco and/or its e.g. 0011.0022.0033
affiliates. All rights reserved. Cisco Public 92
Ordered List of discovered PEs
starting from zero (lowest IP add)
PBB-EVPN Startup Sequence Modulo Operation
I-SID mod N PE Ordered List
Designated Forwarder (DF) Election* I-SID (N = # of PEs)
Position PE
(e.g. I-SID mod 2)
Result of modulo 0 PE1
100 0
operation is used to
1 PE2
determine DF and 101 1
Segment Auto-Discovery BDF status 102 0 Example:
103 1 PE1 DF for I-SIDs 100, 102
PE1 BDF for I-SIDs 101, 103
PE1 PE3
ESI and B-MAC Auto-Sensing Exchange of Ethernet

Segment Routes
CE1 CE3
MPLS
Auto-Discovery
Modulo Operation
PE Ordered List
I-SID (I-SID mod 2)
Position PE
100 0
0 PE1
101 1 PE2 PE4
1 PE2
102 0
103 1 Example:
PE2 DF for I-SIDs 101, 103 DF Designated Forwarder
PE2 BDF for I-SIDs 100, 102 BDF Backup Designated Forwarder
(*) DF election with Service Carving shown (i.e. one DF per I-SID in the segment) I-SID PBB 24-bit Service Instance ID
BGP MAC Advertisement Route (B-MAC) RD RD unique per
PE1 MAC Route advertising PE per EVI
MP2P VPN Label RD = RD-1a ESI reserved ESI

downstream allocated label ESI = MAX_ESI indicates advertised
Segment Auto-Discovery used by other PEs to send MAC is a B-MAC
traffic to advertised (MAC,EVI) MAC = B-M1
Label = L1 B-MAC advertised
by route
RT ext. community
PE1 PE3
RT-a
B-M1 B-M2
CE1 CE3
MPLS
Auto-Discovery
B-M2
B-M1
Backbone MAC (B-MAC) PE2 MAC Route
Reachability Advertisement RD = RD-2a

ESI = 1 PE2 PE4
MAC = B-M1 PE3 / PE4 RIB Path List

Label = L2 VPN MAC ESI NH
RT ext. community RT-a B-M1 n/a PE1
RT-a PE2
PBB-EVPN Startup Sequence PE 1 Inclusive Multicast Route
BGP Inclusive Multicast Route Tunnel Type Ingress RD = RD-1a
RD RD unique per
Replication or P2MP LSP PMSI Tunnel Attribute adv. PE per EVI
Mcast MPLS Label used to Tunnel Type (e.g. Ing. Repl.)

transmit BUM traffic - Label (e.g. L1)
downstream assigned (ing.
VPN Auto-Discovery repl.) or upstream assigned
RT ext. community
(Aggregate Inclusive P2MP RT-a
LSP1)
PE1 PE3
RT RT associated with a
Multicast Tunnel ID / Endpoint given EVI
Discovery
CE1 CE3
MPLS

RD = RD-2a
Tunnel Type (e.g. Ing. Repl.) PE2 PE4
Label (e.g. L2)
RT ext. community
RT-a PMSI - P-Multicast Service Interface
(1) Mcast MPLS label is not set for Inclusive Trees (P2MP LSP)
BUM Broadcast / Unknown Unicast / Multicast
Life of a Packet
Ingress Replication Multi-destination Traffic Forwarding
PE1 receives broadcast
traffic from CE1. PE1 Mcast MPLS
During start-up sequence, PSN MPLS label Label assigned by
PE1, PE2, PE3, PE4 sent adds PBB encapsulation to reach PE3
and forwards it using PE3 for incoming
Inclusive Multicast route BUM traffic on a
which include Mcast label ingress replication 3
copies created given EVI PE3 as DF, it
forwards BUM
PE1 PE3 PE1 PE3 traffic towards
VID 100 segment
SMAC: M1
DMAC: F.F.F
B-M1 B-M2 B-M1 B-M2
L3 PBB
CE1 CE3 CE1 CE3

L2 PBB
MPLS MPLS
L4 PBB
PE 2 Inclusive Multicast B-M2

B-M1
B-M2
B-M1
Route
RD = RD-2a
PE4 non-DF for
PMSI Tunnel Attribute PE2 PE4 PE2 PE4 given I-SID drops
PE2 drops BUM PE3 MAC Table BUM traffic
Tunnel Type = Ing. Repl. traffic originated I-SID xyz
Label = L2 Mcast MPLS Label used to on same source
transmit BUM traffic - B-MAC (B-M1) C-MAC B-MAC Data-plane based
RT ext. community downstream assigned (for MAC learning for
M1 B-M1
RT-a ingress replication) C-MAC / B-MAC
BRKMPL-2333 2016 Cisco and/or its affiliates. All rights reserved. association
Cisco Public 96
Life of a Packet
Inclusive Trees Multi-destination Traffic Forwarding
PE1 receives broadcast
P2MP LSP sourced
During start-up sequence, traffic from CE1. PE1
at PE1
PE1, PE2, PE3, PE4 sent adds PBB encapsulation
Inclusive (per-ISID)
Inclusive Multicast route and forwards it using an
trees
Inclusive Tree
PE3 as DF, it
forwards BUM
PE1 PE3 PE1 PE3 traffic towards
VID 100 segment
SMAC: M1
DMAC: F.F.F
B-M1 B-M2 B-M1 B-M2
PBB
CE1 CE3 CE1 CE3
MPLS MPLS
PBB PBB
PE 2 Inclusive Multicast B-M2
B-M1
B-M2
B-M1
Route
RD = RD-2a
PE4 non-DF for
PMSI Tunnel Attribute PE2 PE4 PE2 PE4 given I-SID drops
PE2 drops BUM PE3 MAC Table BUM traffic
Tunnel Type = P2MP traffic originated I-SID xyz
Label = 0 on same source
Mcast MPLS Label not B-MAC (B-M1) C-MAC B-MAC Data-plane based
RT ext. community assigned for Inclusive Trees MAC learning for
M1 B-M1
RT-a C-MAC / B-MAC
BRKMPL-2333 2016 Cisco and/or its affiliates. All rights reserved. association
Cisco Public 97
Unicast Traffic Forwarding
PE1 MAC Route
MP2P VPN Label
downstream allocated label RD = RD-1a PE3 forwards traffic
MAC advertised PSN MPLS label MP2P VPN Label destined to M1
used by other PEs to send ESI = 1 assigned by PE1
by route to reach PE1 using B-MAC B-M1
traffic to advertised MAC for incoming traffic
MAC = B-M1 towards PE1
for the target EVI
Label = L1
RT ext. community
PE1 PE3 PE1 PE3
VID 100 RT-a
VID 100
SMAC: M1
SMAC: M2
DMAC: F.F.F
B-M1 B-M1 DMAC: M1
B-M2 B-M2
CE1 CE3 CE1 L1 PBB CE3

PE1 & PE2 advertised
MAC routes for B-MAC
(B-M1)
B-M2 B-M2
B-M1 B-M1
PE2 PE4 PE2 PE4

PE3 RIB Path List PE3 MAC Table
VPN MAC ESI NH I-SID xyz Data-plane based
MAC learning for
RT-a B-M1 n/a PE1 C-MAC B-MAC
C-MAC / B-MAC
PE2 M1 B-M1 association
Unicast Traffic Forwarding and Aliasing PE3 forwards traffic PE3 forwards traffic
PE1 MAC Route MP2P VPN on a flow (flow 1) to on a flow (flow 2) to
MP2P VPN Label
Label M1 using B-MAC B- M1 using B-MAC B-
downstream allocated label RD = RD-1a PSN MPLS label
MAC advertised assigned by M1 towards PE1 M1 towards PE2
used by other PEs to send ESI = 1 to reach PE1
by route PE1 for
traffic to advertised MAC
MAC = B-M1 incoming traffic
for target EVI
Label = L1
RT ext. community VID 100
PE1 PE3 PE1 PE3 SMAC: M3
VID 100 RT-a
DMAC: M1
SMAC: M1 VID 100
DMAC: F.F.F SMAC: M4
B-M1 B-M2 B-M1 B-M2 DMAC: M1
L1 PBB
CE1 CE3 CE1 CE3

PE1 & PE2 advertised
MAC route for B-MAC (B- L2 PBB
M1)
B-M2 Data-plane based B-M1
B-M2
B-M1
PE2 MAC Route MAC learning for C-
MAC / B-MAC
RD = RD-2a association
PE2 PE4 PE2 PE4
ESI = 1
PE3 RIB Path List
MAC = B-M1 PE3 MAC Table MP2P VPN Label
VPN MAC ESI NH I-SID xyz PSN MPLS label assigned by PE2
Label = L2 to reach PE2
RT-a B-M1 n/a PE1 C-MAC B-MAC for incoming traffic
RT ext. community for target EVI
PE2 M1 B-M1
RT-a BRKMPL-2333 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
Active / Active Load Balancing from CE
PE3 MAC Route
ESI == 0 used for PE1 forwards traffic to
RD = RD-3a
MP2P VPN Label Single Home Device M3 using B-MAC B-M3 PSN MPLS label MP2P VPN Label
downstream allocated label ESI = 0 towards PE3 to reach PE3 assigned by PE3
used by other PEs to send MAC = B-M3 MAC advertised for incoming traffic
traffic to advertised MAC by route for target EVI
Label = L3
RT ext. community
PE1 PE1
RT-a
VID 100
SMAC: M1
B-M1 DMAC: M3 B-M1
PE3 PE3
L3 PBB
CE1 CE3 CE1 CE3
MPLS MPLS
B-M3 B-M3
B-M1 VID 100 B-M1 L3 PBB

SMAC: M2
DMAC: M3
PE2 PE2
PE 1 / PE2 RIB Path List PE1 / PE2 MAC Table PE2 forwards traffic to
I-SID xyz M3 using B-MAC B-M3
VPN MAC ESI NH
C-MAC B-MAC towards PE3
RT-a B-M3 0 PE3
M3 B-M3
Active / Active Per-Service Load Balancing
PE1 MAC Route
During startup, PE1 RD = RD-1a PE3 forwards traffic to
advertises: MP2P VPN Label M1 using B-MAC B-M1
Ethernet Segment route ESI = 1 assigned by PE1 for towards PE1
MAC Route for B-MAC B- MAC = B-M1 incoming traffic for
M1 target EVI
Label = L1
PE1 elected DF for I-SID
100 RT ext. community PE1 PE3
PE1 PE3 VID 100 I-SID 100
VID 100 RT-a SMAC: M3
SMAC: M1 DMAC: M1
DMAC: M3
B-M1 B-M3 B-M1 B-M3
CE1 configured with two CE1 CE3 CE1 L1 PBB VID 200 ISD 200
SMAC: M4
(2) separate bundles MPLS DMAC: M11
towards PEs MPLS
CE3
L2 PBB
VID 200 B-M3 PE3 / PE4 MAC Table B-M3
SMAC: M11 B-M2 I-SID 100 B-M2
DMAC: M33
C-MAC B-MAC
During startup, PE2 PE2 PE4
advertises: PE2 PE4 M1 B-M1
Ethernet Segment route PE3, PE4 RIB Path List
MAC Route for B-MAC B- PE3 / PE4 MAC Table
M2 VPN MAC ESI NH I-SID 200
PE2 elected DF for I-SID RT-a B-M1 n/a PE1 C-MAC B-MAC
200 M11 B-M2
RT-a B-M2 n/a PE2

BRKMPL 2333

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

BRKMPL 2333

Caricato da

Copyright:

Formati disponibili

Next Generation L2 VPN

Ali Sajassi Distinguished Engineer, Cisco

More VLAN Shortest

802.1D 802.1Q 802.1ad 802.1aq 802.1Qbp

VPLS H-VPLS PBB-VPLS EVPN

Inter Domain (WAN) Connectivity? Yes

All-Active multi-homing? Yes

PBB- EVPN EVPN EVPN- EVPN- EVPN- EVPN-

Following drafts were Following drafts were Enhancements

MAC Routing: Control plane (BGP)

PE1 PE3 Optimum forwarding,

Data Plane: IP or MPLS,

Service Additional Capabilities

can get loopback to the originating

loopback to the originating CE

Either PE3 or PE4 forward the MAC1 CE1 CE2 MAC2

When PE1 & PE2 forward traffic for

with destination address MAC1, it

Flow Based Load-balancing PE to PE direction

ECMP (ingress PE still needs

SHD Route Types Extended Communities

Variable Route type specific

EVPN Startup Sequence

Segment Auto-Discovery VPN Auto-Discovery

Multicast Tunnel Endpoint

Redundancy Group Membership

DF Election & VLAN Carving

ESI Label & MH type Discovery

0000. 0011.0022.0033.0018 2 bytes 6 bytes 2 bytes

ESI Auto-Sensing3 LACP PDU

PE 1 Eth Segment Route

VID mod N PE Ordered List

Redundancy Group Membership CE1 CE3

DF Election & VLAN Carving PE Ordered List

DF Election and VLAN Carving PE2 Eth A-D per ES

ESI Label & MH type Discovery Eth Tag = MAX-ET

Mcast MPLS Label used to Tunnel Type (e.g. Ing. Repl.)

PE 2 Inclusive Multicast Route

Tagged with ES-Import Extended Community

Unique per Advertising PE

Route Type specific encoding of E-VPN NLRI

0x02 MAC Address portion of the ESI

Per-ES Ethernet A-D route Per-EVI Ethernet A-D route

Used for MAC Mass-Withdraw

Unique per Advertising PE per EVI

Sent with Ethernet AD Route per ES

0x06 PE1 advertises in BGP a split-horizon label associated

Unique per Advertising PE per EVI

Route Type specific encoding of E-VPN NLRI

Flags based on RFC6514

EVPN Startup Sequence

PE2 3. ARP Request (IP1)

PE advertises in EVPN Ethernet AS route with a split-horizon label (ESI MPLS

Ethernet A-D Route Advertising Split-Horizon Loop avoidance I can reach

ESI-1 I can PE4 CE4

Ethernet A-D Route Advertising Split-Horizon Loop avoidance

ESI-1 PE4 CE4

3 bytes Used for ARP flood suppression or for

This route has two flavors:

Unique per Advertising PE per EVI

MPLS Label VPN (Aliasing) Label per (ESI,

CE1 CE3 CE1 CE3

PE2 PE4 PE2 PE4