Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
August 2016
Table of Contents
Introduction.....................................................................................................................................................................1
Technology Overview......................................................................................................................................................2
MACsec Frame Format.....................................................................................................................................................3
Ethernet WAN Transition for Carrier Services....................................................................................................................3
Layer 2 Adjacency with MACsec.......................................................................................................................................5
WAN MACsec and IPsec Comparison..............................................................................................................................6
Target Use Cases for MACsec.........................................................................................................................................8
WAN MACsec Deployment Models................................................................................................................................10
WAN MACsec Enhancements........................................................................................................................................12
Clear Tag Option to Enable Point-to-Multipoint Deployments........................................................................................ 12
Access Control Option for Smoother Migration.............................................................................................................. 12
Configurable EAPoL Destination Address....................................................................................................................... 13
Configurable Replay-Protection-Window Size............................................................................................................... 14
Configurable Key Lifetime and Hitless Key Rollover........................................................................................................ 14
Configurable 128/256-bit Encryption and Hitless Rollover............................................................................................. 15
CLI Syntax for Configuring Encryption Algorithm for Data Packets................................................................................. 15
CLI Syntax to Configure Encryption Algorithm for MKA Control Packets........................................................................ 15
Configuration Guidelines................................................................................................................................................16
Use Case 1: Point to Point E-Line Service..................................................................................................................... 17
Use Case 2: E-LAN Service (VPLS Service).................................................................................................................. 22
Use Case 3: MACsec + SGT Transport.......................................................................................................................... 26
Configurable MKA, Key Chain & MACsec Parameters....................................................................................................28
Performing Maintenance Tasks......................................................................................................................................30
Performing Maintenance Tasks (without Impacting Traffic)............................................................................................ 30
Performing Maintenance Tasks (Traffic Impacting)......................................................................................................... 32
Scalability......................................................................................................................................................................34
Monitoring and Troubleshooting....................................................................................................................................35
Monitoring MACsec Sessions: Sample Output............................................................................................................... 35
Monitoring MKA Sessions: Sample Output..................................................................................................................... 36
Network Design Considerations When Leveraging MACsec over Ethernet Service Offerings.........................................38
WAN MACsec Best Practices........................................................................................................................................42
Caveats and Restrictions...............................................................................................................................................43
Feature Limitations......................................................................................................................................................... 43
Glossary........................................................................................................................................................................44
Introduction
This white paper provides a reference point for Cisco sales staff, support teams, and organizations to use when
deploying media access control security (MACsec) in common use cases for WAN.
You can use WAN MACsec innovation from Cisco in order to optimize and simplify the overall deployment, func-
tionality, and operation of networks that require high-speed encryption beyond what IPsec can deliver, without
sacrificing performance and packet size agility. WAN MACsec provides a line-rate network encryption solution
over Layer 2 Ethernet transport services. MACsec is no longer just a LAN technology and can be leveraged
outside campus networks, whether it be over Metro Ethernet transport or Data Center Interconnect (DCI) links.
MACsec also secures WAN connections that are leveraging Ethernet as the link-layer media.
This paper provides an overview of MACsec beyond the campus LAN. MACsec can be a formidable encryption
solution for WAN and Metro Ethernet links. This document provides an overview of MACsec, compares MACsec
with current IP-based encryption solutions, and highlights key WAN/Metro use cases.
The goal of this white paper is to provide the details necessary to configure WAN MACsec, a line-rate network
encryption solution over Layer 2 Ethernet Transport Services in the Cisco ASR 1001-X platform. This document
also describes the best practices and recommendations for Cisco WAN MACsec deployments on ASR 1001-X
platforms. The solution has been thoroughly validated during the system test cycle, and some of the deployment
recommendations are included.
Technology Overview
Figure 1 Data encryption using MACsec for a WAN connection between routers
Encrypted Data
Encrypt
EVCs
Encrypted Data
L2 Service Provider
Network
Decrypt
6020F
MACsec
•• Encrypts all data except for the source and destination MAC addresses of an Ethernet packet.
•• Supports:
◦◦ The option to modify Extensible Authentication Protocol over LAN (EAPoL) Destination MAC and
EtherType to avoid MACsec key agreement (MKA) protocol data unit being consumed by intermediate
devices.
Encrypted
DMAC SMAC 802.1AE Header 802.1Q CMD ETYPE PAYLOAD ICV CRC
0x88E5
MACsec EtherType TCI/AN SL Packet Number SCI (optional)
6021F
MACsec Tag Format
As shown in the above figure, the frame format using MACsec leverages a tag format with a MACsec EtherType
(0x88E5), while allowing the Ethernet source/destination MAC addresses to be left in the clear for Ethernet frame
forwarding. Also note that MACsec encrypts all fields behind the source/destination MAC addresses, so unless
the ability to offset the encrypted field exists, fields such as MPLS labels and 802.1Q tags are encrypted and
not able to be used when the Ethernet frame traverses the underlying transport between encrypted stations. The
Cisco implementation of WAN MACsec described in this white paper allows the 802.1Q header to be left in the
clear, offering a vast amount of network design flexibility as it relates to MACsec frames traversing a public Ether-
net transport.
•• Simple troubleshooting.
MACsec, a hop-by-hop encryption technology (per link), offers several advantages over existing IPsec encryption
solutions in certain designs.
Enterprise WAN
A C E
Enterprise Enterprise
LAN LAN
Enterprise Enterprise
LAN LAN
B D F
6023F
802.1AE MACsec Secured Ethernet WAN Link
As shown in the above figure, the MACsec encryption process is applied per link, so as the Ethernet frame enters
the PHY layer encapsulation process, the MACsec encryption is applied as well.
The PHY layer processing for MACsec is performed as the last process prior to the frame being sent to the
egress port to be put on the physical connection to the neighbor device. Leveraging the encryption process late
in the frame-forwarding procedure offers several advantages, most notably:
•• It has no impact on Ethernet frame markings (802.1p for QoS, 802.1Q tag, Q-in-Q tags).
•• Certain implementations have the option to offset where MACsec encryption is applied, leaving chosen tags
in the clear, which can be advantageous to the Ethernet transport provider and the services offered based on
802.1Q and/or 802.1p bits (example: class of service).
In the case where an IP packet traverses multiple router hops (example: multiple P routers in an MPLS backbone)
to get from router A to router E, because the links are MACsec encrypted over the Ethernet link, each router hop
must encrypt/decrypt the frame. Some designers may believe this is a disadvantage in using MACsec, but it is
no different than networks where the source/destination of the packet traverses multiple router hops, because the
encryption process is done in the hardware PHY of the MAC ASIC. This is case with the ASR 1001-X —
there is no encryption processing penalty when you enable MACsec.
Remote Site 1
with ASR1K at edge
Remote Site 2
MACsec encryption between L2
with ASR1K at edge
adjacent CEs at remote site edges
CE
PE ASR1K
CE MPLS
ASR1K PE Cloud
EoMPLS/VPLS
Remote Site 3
with ASR1K at edge
Egress
PE CE
ASR1K
6024F
•• Simplifies operations.
These technologies are never one-size-fits-all, so it is up to the network designer to understand the holistic view
of the network so the proposed solution aligns with the business objectives and services the network aims to
offer. This is a key component when evaluating encryption technologies or any network solution and transport
overall.
•• Reducing the packet overhead (MACsec = 32 bytes) versus IPsec (GRE+Tunnel mode = ~74 bytes).
•• Encrypting (that is, hiding) the label and/or 802.1Q tag, leaving only the source/destination MAC address in
the clear between stations.
•• Targeting peer-model network designs (versus flat multipoint Ethernet network transport).
Considering these advantages, there are several key situations when MACsec could be considered over IPsec in
the design, including when:
•• High encryption rate is required.
•• The backbone routers are interconnected with either fiber/wavelength or Ethernet, in a point-to-point and/or
partial mesh topology design
•• Per-link encryption is suitable for the given design for core/edge router connectivity within the backbone
topology (that is, the core/edge routers leverage point-to-point Ethernet links for interconnection).
As the MACsec technology continues to evolve, consider the use cases that extend far beyond campus and
data center designs. MACsec opens up a whole new paradigm as it relates to secure WAN and Metro Ethernet
designs. The following WAN and Metro implementations require data encryption over untrusted networks and can
benefit from Layer 2 MACsec encryption:
•• Securing router core links (IP/MPLS, PE-P, P-P)—Secure high-speed backbone transport links (point-to-
point today), such as the following:
◦◦ Optical transport hand-off is client optics from a DWDM system, which are easy to tap into and intercept
•• Securing Metro Ethernet Services—Offer (as a service provider) or leverage secure 10/40/100GE Metro
Ethernet services, each link leveraging 802.1AE for encryption
•• Securing PE-CE link transport integration for MPLS IP VPN Services—Secure back-haul to an MPLS BGP
VPN service (L3 service)
•• Securing links over any Ethernet service—Applications include DCI, access to Cloud utilities, storage replica-
tion
•• Securing Over-the-Top Ethernet Links—Enterprise/Public Sector encrypts their Ethernet links on their own
CPE routers over-the-top of the SP Ethernet service (supported today on point-to-point Ethernet services
only)
As discussed in “WAN MACsec and IPsec Comparison,” one technology is not necessarily better than the other;
the network designer has options and can evaluate which solution best fits the network requirement given the
transport and design criteria that exist. MACsec clearly offers significant design options, given the popularity of
Ethernet and the ever-growing Ethernet services that service providers offer customers.
•• Branch connectivity
•• Aggregation deployment
•• Metro-E Ring/EoMPLS
•• Point-to-point, hub and spoke connectivity using Ethernet virtual private line (EVPL) service (VLAN-mode)
Typical use cases: migration from non-MACsec branches to MACsec branches, service multiplexing such as
Metro-E circuit and Internet or IPsec connection on the same physical interface.
•• Multipoint-to-multipoint, hub-and-spoke connectivity using Ethernet private LAN (EP-LAN) service (port-
mode)
•• Multipoint-to-multipoint, hub-and-spoke connectivity using Ethernet virtual private LAN (EVP-LAN) service
(VLAN-mode)
MACsec is commonly deployed in a Metro Ethernet WAN. The following four Metro Ethernet Service types are
supported as part of this solution as defined in the Metro Ethernet Forum (MEF).
Port-based
Service type services VLAN-based services
E-Line EPL EVPL
Point-to-point, UNI-to- EPL EVPL
UNI
E-LAN EP-LAN EVP-LAN
Multipoint-to-multipoint, EP-LAN EVP-LAN
UNIs-to-UNIs
Authenticated Authenticated
Encrypted
6026F
Eth 802.1Q 802.1AE ETYPE PAYLOAD ICV CRC
The macsec dot1q-in-clear 1 command controls the behavior of allowing 802.1Q tag in the clear. The default is
no macsec dot1q-in-clear (that is, no dot1q tag in the clear).
Tech Tip
With Cisco ASR1001-X, macsec dot1q-in-clear 1 can only be configured on physical interface, and
the setting is automatically inherited by the sub-interfaces.
•• must-secure does not allow transmit/receive of unencrypted packets from physical interface or sub inter-
faces and drops the packet except for MKA control protocol packets.
•• If a mix of MACsec and non-MACsec sub interfaces (example: sub interfaces with IPsec configured) co-
exists, then should-secure configuration is required.
The default is macsec access-control must-secure, which is effective as soon as MACsec CLI is configured
(that is, unencrypted packets are not allowed to be transmitted or received except for MKA control protocol pack-
ets).
Tech Tip
•• Broadcast
•• Default: (0180.C200.0003)
Tech Tip
You can configure the EAPoL destination address independently on either physical or sub interface
level. If it is configured on the physical interface, it is automatically inherited by the sub interfaces.
Explicit configuration on a sub-interface overrides the inherited value or policy for that sub-interface.
The replay protection window may be set to zero to enforce strict reception ordering and replay protection.
Tech Tip
You can configure a replay protection window independently on either physical or sub interface. If
you configure it on the physical interface, it is automatically inherited by the sub interfaces. Explicit
configuration on sub-interface overrides the inherited value or policy for that sub-interface.
The hex string must be 32 Hex characters or 64 characters depending on the Cryptographic algorithm selected
(example: AES-128-CMAC or AES-256-CMAC, respectively)
Keys roll over to the next key within the same key chain by configuring a second key (key 02) in the key chain and
configuring a lifetime for the first key. When the first key (key 01) lifetime expires, it automatically rolls over to the
next key in the list. If the same key is configured on both sides of the link at the same time, then the key rollover is
hitless (that is, the key rolls over without traffic interruption).
•• A key server uses the first cipher suite from the list.
•• All non-key servers must use the same cipher suite as the key server. The cipher suites can be configured in
any order in the non-key server. If the same cipher suite is not available in the list, the session establishment
fails.
•• In the absence of a MACsec cipher suite configuration, all the peers have a default list, which includes all the
cipher suites supported by the local hardware. For ASR1001-X, the cipher suite list is
GCM-AES-128, GCM-AES-256.
•• If a MACsec cipher suite is not configured by an admin, the default cipher suite of GCM-AES-128 (Galois/
Counter Mode of Advanced Encryption Standard cipher with 128-bit key) is used.
Cryptographic algorithm selection for MKA control protocol packets encryption is as follows:
•• The cryptographic algorithm to encrypt MKA control protocol packets is configured as part of key chain.
There can be only one cryptographic algorithm configured per key chain.
•• A key server uses the configured MKA cryptographic algorithm from the key chain used.
•• All non-key servers must use the same cryptographic algorithm as the key server.
If an admin does not configure an MKA cryptographic algorithm, a default cryptographic algorithm of AES-
CMAC-128 (Cipher-based message authentication code with 128-bit Advanced Encryption Standard) is used.
Configuration Guidelines
Solution deployment prerequisites:
•• Cisco ASR 1001-X with 15.S(01)S or later software.
Service provider network should provide MACsec Layer 2 control protocol transparency such as EAPoL.
Note: The WAN MACsec solution in ASR1001-X depends on MKA (as defined in 802.1X-2010) for advertis-
ing capabilities, exchange keying material, cipher suite negotiations, etc. MKA uses EAPoL protocol.
There is no special license needed for WAN MACsec, but it is available as part of regular crypto RTU license.
PE3
PE4
MACsec Encrypted Link
Traffic Generator
in CE1 VLAN
E-LINE (P2P)
6025F
ASR 1001-X CE2 E-LAN (P2MP)
Central
Remote Site Campus/DC
CE2 Carrier Ethernet CE1
Enterprise Service Enterprise
Network E-LINE (P2P) Network
6027F
• Sub-interface (802.1Q)
Ethernet service:
•• Point-to-point pseudo wire service (no MAC address lookup)
•• Port-mode
This guide uses the following conventions for Commands at a CLI or script prompt:
commands that you enter at the command-line Router# enable
interface (CLI).
Long commands that line wrap are underlined.
Commands to enter at a CLI prompt: Enter them as one command:
configure terminal police rate 10000 pps burst 10000
packets conform-action
Commands that specify a value for a variable:
ntp server 10.10.48.17 Noteworthy parts of system output (or of device
configuration files) are highlighted:
Commands with variables that you must define: interface Vlan64
class-map [highest class name] ip address 10.5.204.5 255.255.255.0
CE1 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
ip address 10.3.1.1 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
CE2 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
ip address 10.3.1.2 255.255.255.0
mka pre-shared-key key-chain KEY1
macsec
Network
Ethernet service:
•• Point-to-point PW service (no MAC address lookup)
CE1 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
macsec dot1q-in-clear 1
macsec replay-protection-window-size 1000
interface GigabitEthernet0/0/4.1
encapsulation dot1Q 10
ip address 10.3.1.1 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
interface GigabitEthernet0/0/4.2
encapsulation dot1Q 20
ip address 10.3.2.1 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
CE2 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
macsec dot1q-in-clear 1
macsec replay-protection-window-size 1000
interface GigabitEthernet0/0/4.1
encapsulation dot1Q 10
ip address 10.3.1.1 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
CE3 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
macsec dot1q-in-clear 1
macsec replay-protection-window-size 1000
interface GigabitEthernet0/0/4.1
encapsulation dot1Q 10
ip address 10.3.1.1 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
Enterprise
Network
6029F
CE4
Ethernet service
•• Point-to-point PW service (no MAC address lookup)
CE1 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
macsec dot1q-in-clear 1
macsec access-control should-secure
interface GigabitEthernet0/0/4.1
encapsulation dot1Q 10
ip address 10.3.1.1 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
interface GigabitEthernet0/0/4.2
encapsulation dot1Q 20
ip address 10.3.2.1 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
interface GigabitEthernet0/0/4.3
encapsulation dot1Q 30
ip address 10.3.3.1 255.255.255.0
CE2 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
macsec dot1q-in-clear 1
macsec access-control should-secure
interface GigabitEthernet0/0/4.1
encapsulation dot1Q 10
ip address 10.3.1.2 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
CE3 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
macsec dot1q-in-clear 1
interface GigabitEthernet0/0/4.1
encapsulation dot1Q 20
ip address 10.3.2.2 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
CE4 Configuration
interface GigabitEthernet0/0/4.1
encapsulation dot1Q 30
ip address 10.3.3.2 255.255.255.0
Enterprise
6031F
Network
Ethernet Service
•• Multipoint service (typically VPLS)
CE1 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
macsec dot1q-in-clear 1
macsec replay-protection-window-size 1000
eapol destination-address broadcast
interface GigabitEthernet0/0/4.1
encapsulation dot1Q 10
ip address 10.3.1.1 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
CE2/CE3 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
macsec dot1q-in-clear 1
interface GigabitEthernet0/0/4.1
encapsulation dot1Q 10
ip address 10.3.1.2 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
macsec replay-protection-window-size 1000
eapol destination-address broadcast
Enterprise
6032F
Network
10
CE1 P2MP VLAN
VLAN 10 EVCs
VLAN CE3
VLAN 20 10
0
VL
N2
AN
VLA
20
6033F
CE4 CE5
Ethernet Service
•• Multipoint service (typically VPLS)
CE1 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
macsec dot1q-in-clear 1
macsec replay-protection-window-size 1000
interface GigabitEthernet0/0/4.1
encapsulation dot1Q 10
ip address 10.3.1.1 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
interface GigabitEthernet0/0/4.2
encapsulation dot1Q 20
ip address 10.3.2.1 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
CE2/CE3 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
macsec dot1q-in-clear 1
macsec replay-protection-window-size 1000
interface GigabitEthernet0/0/4.1
encapsulation dot1Q 10
ip address 10.3.1.2 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
CE4/CE5 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
macsec dot1q-in-clear 1
macsec replay-protection-window-size 1000
interface GigabitEthernet0/0/4.2
encapsulation dot1Q 20
ip address 10.3.2.2 255.255.255.0
mka pre-shared-key key-chain KEY1 macsec
MKA Keying
Classification (802.1X-REV)
Central
Branch Site Campus/DC
CE Carrier Ethernet CE
Enterprise Service Enterprise
Network E-LINE (P2P) Network
• Physical
• Sub-interface (802.1Q)
CE1 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
ip address 10.3.1.1 255.255.255.0
cts manual
propagate sgt
mka pre-shared-key key-chain KEY1 macsec
CE2 Configuration
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface GigabitEthernet0/0/4
ip address 10.3.1.2 255.255.255.0
cts manual
propagate sgt
mka pre-shared-key key-chain KEY1 macsec
◦◦ 0 to 64
◦◦ Default: 0
•• macsec-cipher-suite
◦◦ macsec-cipher-suite gcm-aes-128
◦◦ macsec-cipher-suite gcm-aes-256
•• confidentiality-offset
◦◦ 0, 30, 50
◦◦ Default: 0
◦◦ Key ID
•• cryptographic-algorithm
◦◦ cryptographic-algorithm aes-128-cmac
◦◦ cryptographic-algorithm aes-256-cmac
•• keystring
◦◦ Hex Characters
◦◦ Default: NA
•• lifetime
◦◦ hh:mm:ss
◦◦ Default: unlimited
◦◦ 0-x
◦◦ Default: 64
•• macsec-access-control
◦◦ must-secure
◦◦ should-secure
◦◦ Default: must-secure
•• macsec-dot1q-in-clear
◦◦ 0, 1
◦◦ Default: 0
•• macsec
•• eapol destination-address
◦◦ bridge-group-address
◦◦ lldp-multicast-address
◦◦ broadcast
◦◦ Default: (01:80:C2:00:00:03)
From
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
To
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
lifetime local 10:30:00 Oct 30 2014 11:30:00 Oct 30 2014
key 02
key-string 11145678901234567890123456789012
From
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
interface TenGigabitEthernet0/0/0.10
mka pre-shared-key key-chain KEY1
To
key chain KEY1 macsec
key 01
key-string 12345678901234567890123456789012
key chain KEY2 macsec
key 01
key-string abcdef0987654321abcdef0987654321
interface TenGigabitEthernet0/0/0.10
mka pre-shared-key key-chain KEY2
Changing MACsec Cipher-Suite to Encrypt Data Traffic with 128/256 bit Encryption
mka policy POLICY1
macsec-cipher-suite gcm-aes-128
# alternative configuration using AES 256
# macsec-cipher-suite gcm-aes-256
interface GigabitEthernet0/0/1.10
mka policy POLICY1
Changing MKA Cipher-Suite to Encrypt Control Traffic with 128/256 bit Encryption
key chain KEY3 macsec
key 01
key-string abcdef0987654321abcdef0987654321
cryptographic-algorithm aes-128-cmac
# alternative configuration using AES 256
# cryptographic-algorithm aes-256-cmac
interface TenGigabitEthernet0/0/0.10
mka pre-shared-key key-chain KEY3
Scalability
Table 3 Number of supported MACsec peers per port in ASR 1001-X
1 Gb 10 Gb
Max # of SA 16 (receive)+16 (transmit) 64 (receive)+64 (transmit)
Max # of SC 8 (receive)+8 (transmit) 32 (receive)+32 (transmit)
Max # of point-to-point 8 32
Max # of sites in P2MP setup 8 32
4 (with 2 members in each 16 (with 2 members in
Max # of group CA group) each group)
Although MACsec is supported up to line-rate on each interface, the forwarding capability may be limited by the
maximum system forwarding capability.
Transmit SC:
SCI: 0022BDEF43830014
Transmitting: TRUE
Transmit SA:
Next PN: 1712
Receive SC:
Receiving: TRUE
Receive SA:
In Use: TRUE
Next PN: 1731
You can use the following show commands to monitor MKA session details:
•• show mka sessions
You can use the following debug commands to collect debug information:
•• debug mka [events | errors | packets]
6035F
PE4 PE3 PE4 PE3
In Figure 10, each PE router has a single physical attachment into the bridge domain. Each router creates a rout-
ing adjacency (an IGP or BGP) with every other router on the network, so applying N-1, each router establishes
three routing adjacencies. Now, consider 100 routers attached to the same bridge domain, and again, applying
N-1 routing adjacencies. Each PE now contains 99 IGP/BGP routing adjacencies. This flat network model poses
many significant challenges, including:
•• The N-1 maintenance of routing protocol adjacencies.
The lack of state for QoS on SLAs for bandwidth being sent per peer is because the bandwidth is shared at the
access links (example: 99 sites can send to 1, creating a major over-subscription factor for the access link).
Ethernet
Virtual Appears as Routers
Circuit are Peering Over
Service Secure Ethernet Wire
Ethernet Sub-interface
6036F
Ethernet Sub-interface
PE4 with 802.1Q Support PE3 PE4 with 802.1Q Support PE3
For core link interconnect, the recommended approach (Figure 11) is to leverage a point-to-point Ethernet ser-
vice that allows the deployment of a peering model and is supported with MACsec link encryption. This peering
approach offers several key advantages over the flat multipoint model, including:
•• Much better scaling from a routing peer perspective.
•• Deterministic QoS, because an SLA can be applied per Ethernet VC, quantifying the amount of bandwidth per
logical connection.
•• Deterministic traffic shaping per logical connection, eliminating the overrun of a particular site (based on how
the logical connections are sized).
As previously discussed, the Ethernet encryption concept offers advantages over IPsec and overlay networks, but
from an IP design perspective, there are scaling vulnerabilities lurking if the designer is not cautious.
Assuming the customer/agency is leveraging these various Ethernet services along with MACsec in order to gain
the benefits of a more simplistic deployment, a hybrid approach can be taken. The benefits of the hybrid ap-
proach include:
•• A hierarchical network design that allows massive scale for routing.
•• Use point-to-point Ethernet services in the core, while leveraging multipoint service at the access/aggrega-
tion back-haul.
Regional POP
Ethernet
Transport
6030F
In Figure 12, there is a mix of small and large sites, most regionally located, sharing the same flat Ethernet ser-
vice. This design poses many of the challenges mentioned above when leveraging a flat Ethernet service, and in
this example, there are 24 total routers sharing the Ethernet service, so each router contains 23 routing adjacen-
cies. In addition, there is no deterministic nature for how traffic is sent to a given site, and none of the routers on
the shared Ethernet domain will ever have the awareness of senders and receivers of traffic, so it is easy to over-
run the access circuit from CPE to Cloud.
One approach to overcoming these challenges, and building the foundation for a massively scalable network de-
sign, is to take a hierarchical approach.
Figure 12 Hybrid Ethernet E-LAN multipoint service/Ethernet point to point service
Regional POP
6022F
Ethernet “Multipoint” Service (E-LAN)
The hybrid model demonstrated in Figure 13 is leveraging the same routers and locations as in Figure 12, but the
hybrid model applies a hierarchical design while leveraging both point-to-point and multipoint Ethernet service
capabilities. The locations are split into regional PoPs, which dedicate a bridge domain and core router. Multi-
point Ethernet service is leveraged for the back-haul of remote-site/branch locations, and a point-to-point E-Line
service is used for core router interconnects. This approach allows deterministic QoS within the core and sim-
plicity of multipoint for back-haul, while limiting the number of routing adjacencies needing to be maintained on
each router. In Figure 12, in contrast, the peering model is N-1.
In the hybrid model, the maximum routing adjacency is seven (versus 23 in the flat model), but more importantly,
this design offers massive scalability and can leverage common core technologies such as MPLS traffic engi-
neering for bandwidth management, L2/L3 VPN services, and rapid convergence to name a few.
When you are leveraging Ethernet and MACsec, understanding the impact the Ethernet service can have on the
overall design is critical to choosing the appropriate transport service.
•• When configuring for the first time, ensure that you have out-of-band connectivity to the remote site in order
to avoid locking yourself out after enabling MACsec if the session fails to establish.
•• It is recommended that you configure access-control should-secure while enabling MACsec for the first
time and subsequently remove the CLI to change to default “Must secure” after the session establishment is
successful, unless it is needed for migration cases as mentioned in “Point-to-Point SA Configuration—Mix of
MACsec and Non-MACsec Spokes, VLAN-based E-Line Service (P2P).”
•• It is recommended that you configure an interface MTU adjusting for MACsec overhead, ~32 bytes. Although
MACsec encryption/decryption occurs at the PHY level and MTU is not an issue for the source or destination
router, it may be an issue for the intermediate SP router. Configuring an MTU value at the interface allows for
MTU negotiation that includes MACsec overhead.
•• mka policy, macsec replay-protection-window and eapol destination-address can be configured on the
main and/or sub-interface and the value is automatically inherited by the sub-interfaces when configured on
the main interface. Explicit configuration on the sub-interface overrides the inherited value or policy for that
sub-interface.
For port based sessions, pre-shared-key should be configured on main interface only.
For VLAN based sessions, pre-shared-key should be configured on sub-interface(s) only. If the pre-shared-
key is configured on the main interface, it will not be inherited to the sub-interface(s).
FEATURE LIMITATIONS
MACsec with EtherChannel (link bundling) is not supported.
•• MACsec configuration on EtherChannel is not supported.
Glossary
CA connectivity association
CAK connectivity association key
CE router customer edge router
CPE customer premise equipment
DCI data center interconnect
EAPoL Extensible Authentication Protocol over LAN
EP-LAN Ethernet private local-area network
EPL Ethernet private line
EVC Ethernet virtual circuit
EVP-LAN Ethernet virtual private local-area network
EVPL Ethernet virtual private line
GE gigabit Ethernet
GRE generic routing encapsulation
IP Internet protocol
IPsec Internet protocol security
L2TP Layer 2 tunneling protocol
LLDP link layer discovery protocol
MACsec media access control security
MAN metropolitan-area network
MEF Metro Ethernet Forum
MetroE Metro Ethernet
MKA MACsec key agreement
MPLS multiprotocol layer switching
MTU maximum transmission unit
NIST National Institute of Standards and Technology
P router provider router, sometimes referred to as P
P2MP point to multipoint
PE router provider edge router, sometimes referred to as PE
PoP point of presence for a service provider
QoS quality of service
SAK security association key
SA security association
SC secure channel
SGT security group tag
UDP user datagram protocol
VRF virtual routing and forwarding
WAN wide-area network
Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco Website at www.cisco.com/go/offices.
ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, “DESIGNS”) IN THIS MANUAL ARE PRESENTED “AS
IS,” WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT
SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION,
LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN
ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR
THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS
OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON
FACTORS NOT TESTED BY CISCO.
Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included
in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go
to this URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not
imply a partnership relationship between Cisco and any other company. (1110R)