Troubleshooting GETVPN Deployments

Troubleshooting GETVPN Deployments
BRKSEC-3051
Wen Zhang - Technical Leader, Services
2
Agenda
GETVPN Solution Overview

What Is GETVPN and Where Does It Fit?
Introduction to GETVPN
Technology Overview
GETVPN Deployment
Configuration and Deployment Considerations
Troubleshooting
Troubleshooting Tools and Techniques
Common Troubleshooting Scenarios
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Other Related Sessions
CiscoLive 2013
BRKSEC-2054 Deploying GET to Secure VPNs

BRKSEC-3013 Advanced IPSec with FlexVPN
BRKSEC-3052 Troubleshooting DMVPNs
BRKSEC-4054 Advanced Concepts of DMVPN
GETVPN Solution Overview
Cisco Group Encrypted Transport - GETVPN
What Is GETVPN?
Cisco GETVPN delivers a revolutionary solution for tunnel-less, any-
to-any and confidential branch communication
Large-scale any-to-any encrypted
communication
Any
Any --to
to --Any
Any Native routing without
Connectivity
Connectivity tunnel overlay
Optimal for QoS and Multicast
support - improves application
Cisco GET performance
VPN Transport agnostic - private
Scalable Real Time LAN/WAN, FR/ATM, IP, MPLS
Tunnel-Less VPN - A New Security Model
Before: IPSec P2P Tunnels After: Tunnel-Less VPN
WAN
Multicast
Scalabilityan issue (N^2 problem) Scalable architecture for any-to-any
Overlay routing connectivity and encryption
Any-to-any instant connectivity cant No overlaysnative routing
be done to scale Any-to-any instant connectivity
Limited QoS Enhanced QoS
Inefficient Multicast replication Efficient Multicast replication
VPN Technology Positioning
Data Center Core
IPSec Agg.
GM
GM
Internet
Edge KS KS
Remot Access SW WAN Edge

Clients
Internet/Shared GET
Network MPLS/Private
Encrypted
Network
EzVPN/FlexVP GM GM GM
N Spoke
DMVPN/FLexVPN DMVPN/FlexVP
N Spoke GETVPN GM GETVPN GM GETVPN GM
Spoke
VPN Technology Positioning (Cont.)
FlexVPN DMVPN GETVPN
Public Internet Public Internet

Infrastructure Network Private IP Transport
Transport Transport
Hub-Spoke and
Converged Site to Site Any-to-Any;
Network Style Spoke-to-Spoke; (Site-
and Remote Access (Site-to-Site)
to-Site)
Dynamic Routing or
Dynamic routing on Dynamic routing on IP
Routing IKEv2 Route
tunnels WAN
Distribution
Route Distribution Route Distribution Route Distribution
Failover Redundancy
Server Clustering Model Model + Stateful
Peer-to-Peer Peer-to-Peer
Encryption Style Group Protection
Protection Protection
Multicast replication at Multicast replication at Multicast replication in

IP Multicast
hub hub IP WAN network
Introduction to GETVPN
Group Encrypted Transport (GETVPN)
Uses three main components

Secure Group Keys
Header Preservation
Key Service
Is based on open standards with patented Cisco technology
Leverages existing IKE, IPSec, and multicast technologies
Takes advantage of the existing routing infrastructure
Group Security Functions
Routing Member
Key Server Key Server
Forwarding
Validate Group Members Replication
Manage Security Policy Routing
Create Group Keys
Distribute Policy/Keys
GM
Routing
Members
GM
GM
Group Member
Encryption Devices GM
Route Between Secure/ Unsecure Regions
Multicast Participation
Group Security Elements
Group Policy KS Cooperative
Key Servers Protocol
Key Encryption Key (KEK)
Traffic Encryption Key (TEK)

GM
Routing
Members
GM
GM
RFC3547:
Group Domain of GM
Interpretation (GDOI)
Basic GETVPN Architecture
Step 1: Group Members (GM) register via GDOI with the Key
Server (KS)
KS authenticates and authorizes the GM

KS pushes a set of IPSec SAs GM3 GM4
for the GM to use GM2
GM5
GM1
GM6
GM9
Key Server
GM8 GM7
Step 2: Data Plane Encryption

GM exchange encrypted traffic using the group keys
The traffic uses IPSec Tunnel Mode with Header Preservation
GM3 GM4
GM2
GM5
GM1
GM6
GM9
Key Server
GM8 GM7
Step 3: Periodic Rekey of Keys
KS pushes out replacement IPSec keys before current IPSec keys expire; this
is called a Rekey
GM3
GM4
GM2
GM5
GM1
GM6
GM9
Key Server
GM8
GM7
Header Preservation
IPSec Tunnel Mode vs. GETVPN
IP Packet IP Header IP Payload
IPSec New IP Header ESP IP Header IP Payload

Tunnel Mode
IPSec header inserted by VPN Gateway
New IP Address requires overlay routing
IP Header IP Payload
IP Packet
GETVPN Preserved Header ESP IP Header IP Payload
IP header preserved by VPN Gateway

Preserved IP Address uses original routing plane
GETVPN Data Path
Host1 GM 1 GM2 Host2
Encrypted/Authenticated Using Group SA
Original Src and Original IP

Dst Addresses ESP Data
Header
Encrypted
Rekey Methodology: Multicast Rekey
Rekey Message sent from key server to all group members

IP multicast message provides very efficient distribution
Rekeys resulting from configured KEK and TEK intervals Single rekey
packet sent to
or KS policy change multicast
enabled core
Key Server
GM2
GM1
Core replicates
the packets to
all GMs
GM3 GM4
Rekey Methodology: Unicast Rekey
Key Server maintains state of active group members

Group Member sends ACK in response to the rekey messages
Remove Group Member if the GM does not acknowledge
three rekeys
Key Server
GM2
GM1
GM3 GM4
Requirement for Time-Based Anti-Replay
Sequence number based anti-replay only works with

single sender
Need method to work for all senders using same IPSec SA
Key Server downloads relative pseudotime and window size to
all the GMs
GMs calculate pseudo-timestamp based on downloaded pseudotime and
sends out packet
Receiving GM verifies packet within window size
KS periodically refreshes GMs with pseudotime/window size - this means
clock does not need to be synchronized between GMs
Time-Based Anti-Replay
If Senders pseudotime falls in the below Receiver window,

packet accepted
Reject Accept Reject
Initial PTr - W PTr PTr + W

pseudotime
Anti-replay window
Packet 1 and Packet 2 have pseudotimeT0, providing loose anti-replay

protection (unlike counter-based)
T0 T10 T20
Packet1
Packet2
Cooperative Key Servers - HA
Single KS is a single point of failure
Two or more KSs known as COOP KSs manage a common set of keys and
security policies for GETVPN group members
Group members can register to any one of the available KSs
Cooperative KS1 Cooperative KS2
Subnet 1
Subnet 2
GM 1
GM 2
IP Network
Subnet 4 Subnet 3
GM 4 GM 3
GDOI Registration
Cooperative KS3
Cooperative Key Servers (Cont.)
One KS is elected as the Primary KS
Cooperative KSs periodically exchange and synchronize groups database,
policy and keys
Primary KS is responsible to generate and distribute group keys
Cooperative KS1 Cooperative KS2 (Secondary)

(Primary)
Subnet 1
Subnet 2
GM 1
GM 2
IP Network
Subnet 4 Subnet 3
GM 4
GM 3
Announcement Messages
Rekey Messages Cooperative KS3 (Secondary)
GETVPN Deployment Configuration
COOP Server Exportable RSA Keys
RSA Keys (generated only on KSs) are required for rekey

authentication
RSA public key distribution from Key Server to Group Member:

Public key generated in the RSA key pair, is sent to the GM at the
registration
The rekeys are signed by the private key of the KS and GM verifies the
signature in the re-key with the public key of the KS
Exporting RSA Key between Key Servers:
One of the key server in the redundancy group should generate the
exportable RSA keys and copy those keys to other
key servers
KS Configuration
Pre-shared Key crypto keyring gdoi1
pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123
!
ISAKMP Policy crypto isakmp policy 10
encr 3des
IPSec Transform authentication pre-share
!
IPSec Profile crypto ipsec transform-set 3DES-SHA esp-3des esp-sha-hmac
!
crypto ipsec profile gdoi1
Access-List used for set security-association lifetime seconds 7200
defining set transform-set 3DES-SHA
rekey (useful in !
multicast rekeys only) access-list 150 permit ip any host 225.1.1.1
!
access-list 160 deny eigrp any any
Access-list defining the access-list 160 deny pim any any
encryption policy access-list 160 deny udp any any eq isakmp
access-list 160 deny udp any any eq 848
access-list 160 permit ip any any
KS Configuration (Cont.)
GDOI Group ID crypto gdoi group getvpn1
identity number 101
Rekey Address mapping server local
(only for multicast rekeys) !rekey address ipv4 150 !
rekey lifetime seconds 14400
rekey retransmit 10 number 2
Rekey Properties rekey authentication mypubkey rsa getvpn1
rekey transport unicast
sa ipsec 1
Encryption ACL
profile gdoi1
match address ipv4 160
Source address for rekeys address ipv4 130.23.1.1
redundancy
local priority 10
COOP KS Config
peer address ipv4 130.1.2.1
!
GM Configuration
Pre-shared Key crypto keyring gdoi
pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123
!
ISAKMP Policy
crypto isakmp policy 10
encr 3des
authentication pre-share
group 2
GDOI Group !
crypto gdoi group getvpn1
KS Address identity number 101
server address ipv4 130.23.1.1
GDOI crypto map !
crypto map getvpn10 gdoi
set group getvpn1
Crypto map on the !
interface interface FastEthernet0/0
crypto map getvpn
GETVPN Platform Support
Platform Group Member Key Server

Software Yes Not supported
870 Yes Not supported
1821 Yes Not supported
1841/1900 Yes Yes
2800 (AIM/SSL)/2900 Yes Yes
3800 (AIM-II/AIM-III)/3900 Yes Yes
7200 NPEG1, VAM2+ Yes Yes
7200 NPEG2, VSA Yes Yes
Cisco ASR 1000 Yes Yes (since XE3.6)
Scalability and Performance
GETVPN Provides complete segregation of control and data plane

Key Server is responsible to maintain the control plane (key
management) and GM is responsible to handle the data plane (actual
user traffic)
KS and GM can NOT be configured on same IOS device
KS should be properly sized for number of branches (scale) in the
network
GM should be properly sized for traffic throughput at each branch
Deployment Best Practices
IKE/IPSec
Use specific pre-shared keys for all the GMs and KSs instead of using default key
KS
Always use COOP KSs
Set the huge buffer to 65535 and add 10 buffers to permanent buffer list
Configure periodic DPDs between the COOP KSs
Enable GM authorization
Policy
Aggregate the permit access-list entries to reduce the entries
Enable Time-Based Anti-Replay
Avoid re-encrypting traffic which is already encrypted (SSH, HTTPS)
Registration
Distribute GM registration to multiple KSs by arranging the KS order in configuration
Rekey Timers
Set TEK lifetime to 7200 Seconds
Set KEK lifetime to 86400 Seconds
GETVPN Troubleshooting
A problem well stated is a problem half solved
Charles F. Kettering
Troubleshooting GETVPN
Ultimately all problems manifest at the data plane -my user

application is not working over GETVPN!
But where really is the problem?
Control Plane
Events that lead up to SAs getting installed on the GMs
Data plane
Policy downloaded with SAs installed but traffic is not flowing
Troubleshooting GETVPN High Level Flow
Time Based
COOP Anti-Replay
IKE Fragmentation
MTU Issues
Registration
Transport Issues
Policy Download
Crypto
Rekey
policy/engine
Control Plane Data Plane
Troubleshooting Flow
GETVPN Control Plane
Common Control Plane Issues

GM registration issues
Policy download issues
COOP issues
Rekey failures
Understand the expected protocol flow and know how to check for them
Control Plane Troubleshooting Tools
GETVPN provides enhanced set of show commands for
functionality verification
IOS also provided wide variety of syslog messages to verify
proper GETVPN operations, and early insight into potential
problems
IPSec and GDOI related debugs can then be enabled for further
troubleshooting
GDOI conditional debugs 15.1(3)T
GDOI event trace 15.1(3)T
Show crypto gdoi (on KS)
Group Name : GET
Group Identity : 101
Registered GMs Group Members : 3
IPSec SA Direction : Both
Active Group Server : Local
COOP configuration Redundancy : Configured
Local Address : 130.23.1.1
Local Priority : 10
Local KS Status : Alive
Key Server Role Local KS Role : Primary
Group Rekey Lifetime : 1800 secs
Group Rekey
KEK lifetime remaining Remaining Lifetime : 88 secs
Rekey Retransmit Period : 10 secs
Rekey Retransmit Attempts: 3
Group Retransmit
Remaining Lifetime : 0 secs
IPSec SA Number : 1
IPSec SA Rekey Lifetime: 900 secs
Profile Name : gdoi1
Replay method : Count Based
Replay Window Size : 64
SA Rekey
TEK lifetime remaining Remaining Lifetime : 446 secs
ACL Configured : access-list 160
Show crypto gdoi ks member (on KS)
KS#show crypto gdoi ks members
Group Member Information :
Number of rekeys sent for group GET: 4
GMs IP address Group Member ID : 131.1.1.1

Group ID : 101
Group Name : getvpn1
KS GM is registered with Key Server ID : 130.2.1.1
Rekeys sent : 4
Rekey Acks Rcvd : 4
GM rekey history Sent seq num : 1 2 3 4

Rcvd seq num : 1 2 3 4
Show crypto gdoi (on GM)
GROUP INFORMATION KEKPOLICY:
Rekey Transport Type : Unicast
Group Name : GET Lifetime (secs) : 12295
Active KS
Group Identity : 101 Encrypt Algorithm : 3DES
Rekeys received : 270 Key Size : 192
IPSec SA Direction : Both Sig Hash Algorithm : HMAC_AUTH_SHA
Active Group Server : 134.50.0.1 Sig Key Length (bits) : 1024
Group Server list : 134.50.0.1
TEK POLICY:
FastEthernet0/0:
GM Reregisters in : 5187 secs
Rekey Received(hh:mm:ss) : 00:02:30 IPSec SA:
sadirection:outbound
Rekeys received spi: 0x7C45C74A(2084947786)
Cumulative : 270 transform: esp-aes esp-sha-hmac
After registration : 270 When was sa timing: remaining key lifetime
Rekey Acks sent : 270 last rekey (sec): (5246)
received Anti-Replay(Time Based) : 2 sec interval
ACL Downloaded From KS 134.50.0.1:
access-list deny eigrp any any Remaining
access-list deny tcp any any port = 179 IPSec SA
access-list deny udp any port = 848 Lifetime
any port = 848
access-list permit ip any any
GETVPN Control Plane Verification
Syslog Messages - KS
Rekey:
GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1
from address 101.1.1.1 with seq # 1
COOP:
GDOI-3-COOP_KS_UNREACH: Cooperative KS 10.0.9.1 Unreachable in group
G1
GDOI-5-COOP_KS_ELECTION: KS entering election mode in group G1
(Previous Primary = NONE)
GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.0.8.1 in group G1 transitioned to
Primary (Previous Primary = NONE)
GETVPN Control Plane Verification
Syslog Messages - GM
Registration:
CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group G1
using address 10.1.13.2
GDOI-5-GM_REKEY_TRANS_2_UNI: Group G1 transitioned to Unicast Rekey
GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for group
G1 using address 10.1.13.2
Rekey:
GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to
10.1.13.2 with seq # 3
Control Plane Debugging Challenges
Challenge
Networks are getting bigger and faster, traditional debugs may not scale
Solution
Use IPSec and GDOI conditional debugs to minimize the debugging
impact
Use the minimal level of debugs required
Challenge
Problems can be unpredictable with no identifiable trigger
Solution
Syslogs
GDOI Event Trace
GDOI Debug Level Granularity
All feature components can be debugged at 5 levels

Start with the highest level, enable additional levels as needed
GM1#debug crypto gdoi gm rekey ?
all-levels All levels
detail Detail level
error Error level
event Event level
packet Packet level
terse Terse level
Debug Level What you will get

Error Error Conditions
Terse Important messages to the user and protocol issues
Event State transitions and events such as send/receive rekeys
Detail Most detailed debug message information
Packet Dump of detailed packet information
All All of the above
GDOI Conditional Debugs
All IPSec and GDOI debugs can now be triggered with KS1 KS2
a conditional filter based on group or peer address

Use the unmatched flag to catch debugs with no MPLS/Private IP
context information GM1 GM500
To enable conditional debugs

1) Set the conditional filter
?
GM145
2) Enable relevant debugs of interest as usual

KS1# debug crypto gdoi condition peer add ipv4 10.1.20.2
% GDOI Debug Condition added.
KS1#
KS1# show crypto gdoi debug-condition
GDOI Conditional Filters:
Peer Address 10.1.20.2
Unmatched NOT set
KS1#debug crypto gdoi ks registration all-levels

GDOI Key Server Registration Debug level: (Packet, Detail, Event, Terse, Error)
Best practices when using the debug commands
Turn off console logging

Use NTP to sync up times on all devices
Enable msectimestamping debug and log messages
service timestamps debug datetime msec
service timestamps log datetime msec
Send the debugs to a syslog server
If no syslog server is available, use the logging buffer with an increased
buffer size
logging buffered 1000000 debugging
terminal exec prompt timestamp when using the show commands to
correlate show commands with the debug output
reload in x to prepare for the worst
GDOI Event Trace
Light weight event buffer to supplement syslogs

Always-on
Flexible output and display options
Event buffer
Continuous real time output
Output to file
Merged output from different feature components
Circular or one-shot buffer
Extensive exit path/error tracing capability
GDOI Event Trace - Example
GM1#show monitor event-trace gdoi?
all Show all the traces in current buffer
back Show trace from this far back in the past
clock Show trace from a specific clock time/date
coop GDOI COOP Event Traces
from-boot Show trace from this many seconds after booting
infra GDOI INFRA Event Traces
latest Show latest trace events since last display
merged Show entries in all event traces sorted by time
registration GDOI Registration event Traces
rekey GDOI Rekey event Traces
GM1#show monitor event-trace gdoi merged all

*May 25 20:20:57.706: Registration_events: GDOI_REG_EVENT: REGISTRATION_STARTED:
GM 10.1.20.2 to KS 10.1.11.2 for group G1
*May 25 20:21:08.970: Registration_events: GDOI_REG_EVENT: REGISTRATION_DONE: GM
10.1.13.2 to KS 10.1.11.2 for group G1
*May 26 00:45:52.878: Rekey_events: GDOI_REKEY_EVENT: REKEY_RCVD: From 10.1.11.2
to 10.1.13.2 with seq no 131 for the group G1
*May 26 00:45:52.878: Rekey_events: GDOI_REKEY_EVENT: ACK_SENT: From 10.1.11.2
to 10.1.13.2 with seq no 131 for the group G1
Troubleshooting Methodology
crypto gdoi group G1 crypto gdoi group G1
identity number 3333 identity number 3333
server local server local
rekey lifetime seconds 86400 rekey lifetime seconds 86400
rekey authenmypubkeyrsa get rekey authenmypubkeyrsa get
rekey transport unicast rekey transport unicast
sa ipsec 1 KS2 saipsec 1
KS1
profile gdoi-p profile gdoi-p
match address ipv4ENCPOL match address ipv4ENCPOL
replay counter window-size 64 replay time window-size 5
address ipv4 10.1.11.2 address ipv4 10.1.12.2
Ser 1/0: 10.1.11.2 Ser 1/0: 10.1.12.2
redundancy redundancy
local priority 10 local priority 2
peer address ipv4 10.1.12.2 MPLS/Private IP peer address ipv4 10.1.11.2
Ser 1/0: 10.1.20.2 Ser 1/0: 10.1.21.2

crypto gdoi group G1 crypto gdoi group G1
identity number 3333 identity number 3333
server address ipv4 10.1.11.2 server address ipv4 10.1.12.2
server address ipv4 10.1.12.2 GM1 server address ipv4 10.1.11.2
GM2
! !
crypto map gm_map 10 gdoi crypto map gm_map 10 gdoi
set group G1 Eth 0/0: 192.168.20.1/24 set group G1
Eth 0/0: 192.168.21.1/24
! !
interface Serial1/0 interface Serial1/0
crypto map gm_map crypto map gm_map
GETVPN Control Plane Setup Steps
COOP KS IKE Setup

COOP Election and Policy Creation
GM-KS IKE Setup

GM Authorization and Registration
GM Encryption Keys and Policy download
GM Data Encryption and Decryption
Periodic Key Renewal and Distribution (Rekeys)
GETVPN Common Issues Control Plane
IKE Setup
Encryption Policy
Key RenewalRekey
Control Plane Replay Check
Control Plane Packet Fragmentation Issue
IKE Setup Between KS and GM
First step in GM registration is IKE setup
On successful negotiation of the IKE process, GM proceeds with the
GDOI group registration
IKE SA is established at the time of registration eventually times out
as its no longer needed after registration
KS1# show crypto isakmp sa
IPv4 Crypto ISAKMP SA
Dst src state conn-id slot status
10.1.11.2 10.1.20.2 GDOI_IDLE 1013 0 ACTIVE
10.1.12.2 10.1.11.2 GDOI_IDLE 1004 0 ACTIVE
10.1.21.2 10.1.11.2 GDOI_REKEY 0 0 ACTIVE
Expires
after IKE
lifetime
GM1# show crypto isakmp sa
Dst src state conn-id slot status
10.1.11.2 10.1.20.2 GDOI_IDLE 1073 0 ACTIVE
10.1.20.2 10.1.11.2 GDOI_REKEY 1074 0 ACTIVE
IKE Setup IKE Failure
Symptoms
If a GM fails to register with the KS, it will continue to attempt

to register with the KS
*May 24 06:40:15.581: %CRYPTO-5-GM_REGSTER: Start registration to KS
10.1.11.2 for group G1 using address 10.1.20.2
GM1#
*May 24 06:41:25.581: %CRYPTO-5-GM_REGSTER: Start registration to KS
10.1.11.2 for group G1 using address 10.1.20.2
%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for

group G1 using address 10.1.20.2
KS1 KS2
Possible causes:
Network issues between the GM and KS MPLS/Private IP
IKE negotiation failure

KS policy issues GM1 GM2
Pre-Shared Key Mismatch
Troubleshooting
Verify routing information on KS and GM and try ping KS from the GM

After ruling out the connectivity issues, check the IKE SA on the GM
GM1#show crypto isakmp sa

Dst src state conn-id status
10.1.11.2 10.1.20.2 MM_KEY_EXCH 1038 ACTIVE
IKE SA not getting established; cant

get to GDOI_IDLE state
Verify the logs on the Key Server

KS1#
%CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 10.1.20.2 failed its
sanity check or is malformed
Pre-Shared Key Mismatch
Solution
Syslog pointing to a mismatched pre-shared key configuration

Can be verified using debug crypto isakmp
KS Config: crypto isakmp key cicso address 10.1.20.2
GM Config: crypto isakmp key cisco address 10.1.11.2
Correct the pre-shared key configuration
KS1(config)#no crypto isakmp key cicso address 10.1.20.2

KS1(config)#crypto isakmp key cisco add 10.1.20.2
KS1(config)#^Z
IKE Setup
Encryption Policy
Key RenewalRekey
GM Policy Download
As part of the registration process, KS pushes down the

encryption policies and keying material to the GM:
GM1#show crypto gdoi
<snip>
access-list deny eigrp any any
access-list deny ip 224.0.0.0 0.0.0.255 any
access-list deny ip any 224.0.0.0 0.0.0.255
access-list deny udp any port = 848 any port = 848
KEK POLICY:
Rekey Transport Type : Unicast
Lifetime (secs) : 2954
<snip>
TEK POLICY:
Serial1/0:
IPSec SA:
sa direction:inbound
spi: 0x2113F73B(554956603)
transform: esp-3desesp-sha-hmac
sa timing:remaining key lifetime (sec): (99)
Anti-Replay(Time Based) : 5 sec interval
<snip>
KS Policy Issues
Routing Control Plane Traffic Failure
In most environments, GETVPN runs on the CE devices and KS1 KS2
PE devices do not participate in GETVPN

MPLS/Private IP
Failure to deny control plane traffic (such as routing protocol) BGP
on the PE-CE link will cause routing protocol to go down as GM1

GM2
soon as GM successfully registers

To identify, look at the ACL downloaded at GM:
GM1#show crypto gdoi gm acl
Group Name: G1
access-list deny eigrp any any BGP is not denied in the ACL
access-list deny ip 224.0.0.0 0.0.0.255 any downloaded from the KS
access-list deny ip any 224.0.0.0 0.0.0.255
access-list deny udp any port = 848 any port = 848
ACL Configured Locally:
KS Policy Issues
Control Plane Traffic - Solution
If most of the CEs are running BGP with the PE routers, KS1 KS2
configure a global KS policy to deny BGP

MPLS/Private IP
KS1&2(config)# ip access-list extended ENCPOL BGP
KS1&2(config-ext-nacl)#1 deny tcp any any eq bgp
GM2
KS1&2(config-ext-nacl)#2 deny tcp any eq bgp any GM1
If only a handful of CEs are running BGP with the PE routers,

configure a local GM policy to deny BGP
GM1# GM1#show crypto gdoi gm acl
! Group Name: G1
access-list 150 deny tcp any any eq bgp ACL Downloaded From KS 10.1.11.2:
access-list 150 deny tcp any eq bgp any <snip>
! access-list permit ip any any
crypto map gm_map 10 gdoi ACL Configured Locally:
set group G1 Map Name: gm_map
match address 150 access-list 150 deny tcp any any port = 179
! access-list 150 deny tcp any port = 179 any
IKE Setup
Encryption Policy
Key Renewal - Rekey
GETVPN Rekeys
Once the GETVPN network is properly setup and is working, KS is

responsible for sending out rekey messages to all the GMs
KS can use unicast or multicast rekeys
Following syslog messages will appear in the log:
PRIMARY KS:
%GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1 from

address 10.1.11.2 with seq # 11
All the GMs:
%GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to

10.1.20.2 with seq # 11
Following the Rekey Flow
Rekey received by
IP?
Rekey Rekey Rekey verified by

Rekey sent?
delivered? received? IKE?
KS Network GM
Transport Rekey Processed
by GDOI?
Rekey
Acknowledged?
Missing RSA Key
Symptoms
When GM registers to the KS, the following KS1 KS2
message shows up in the syslog:

MPLS/Private IP
%GDOI-1-KS_NO_RSA_KEYS: RSA Key - get : Not found, Required for group G1 GM1 GM2
As a result KS will not send rekey messages, and GM

will re-register when the keys expire
%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1 may have

expired/been cleared, or didn't go through. Re-register to KS.
Missing RSA Key on the KS
Troubleshooting Steps
Check whether KS is sending out the rekeys or not:
KS1#show crypto gdoi ks rekey

Group G1 (Multicast)
Number of Rekeys sent : 0
Number of Rekeys retransmitted : 0
KEK rekey lifetime (sec) : 86400
Retransmit period : 10 No rekeys sent
Number of retransmissions : 2
IPSec SA 1 lifetime (sec) : 3600
Remaining lifetime (sec) : 166
Number of registrations after rekey : 22
KS needs RSA keys to sign the rekey messages; check logs for
clues and/or verify the RSA keys
Troubleshooting Steps (Cont.)
Verify RSA key configuration on the KS:

KS1#show running | section gdoi group
crypto gdoi group G1
identity number 3333
server local
rekey address ipv4 102
rekey lifetime seconds 86400
rekey authentication mypubkey rsa get
sa ipsec 1
profile gdoi-p
match address ipv4ENCPOL Labeled RSA key not present
no replay
address ipv4 10.1.11.2
Verify the RSA key pair name on the router:

KS1#show crypto key mypubkey rsa | include name
Key name: key1
Key name: key1.server
Solution
Generate the required RSA key pair

KS1(config)#crypto key generate rsa label get exportable modulus 1024
The name for the keys will be: getvpn-rsa-key
% The key modulus size is 1024 bits

% Generating 1024 bit RSA keys, keys will be exportable...[OK]
Verify rekey messages are now being sent on the KS

%GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1 from
address 10.1.11.2 with seq # 1
KS1#show crypto gdoi ks rekey

Rekeys are now sent
Group G1 (Unicast)
Number of Rekeys sent : 1
<SNIP>
Multicast Rekey Issues
Multicast Rekeys Failing - Symptom
GM is not getting the multicast rekey messages and therefore

continues to re-register with the KS
Rekey starts to work when switched from multicast rekey to
unicast rekey
Possible Causes
Packet delivery issue within the multicast routing infrastructure
End-to-end multicast routing enabled?
mVPN service provided by the MPLS core provider?
Multicast Rekey Failing
Troubleshooting
Check KS to verify multicast rekey messages are being sent KS1 KS2
Multicast
Network
%GDOI-5-KS_SEND_MCAST_REKEY: Sending Multicast Rekey for group G1
10.1.20.2 10.1.21.2
from address 10.1.11.2 to 226.1.1.1 with seq # 6
GM1 GM2
Make sure ICMP is excluded from the KS encryption

policy and is used as a tool to test multicast
KS1#ping 226.1.1.1
Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 226.1.1.1, timeout is 2 seconds:
Reply to request 0 from 10.1.21.2, 44 ms

No response from
GM1 (10.1.20.2)
Troubleshooting
Check the multicast forwarding path
WAN#show ip mroute 226.1.1.1

<snip>
(10.1.11.2, 226.1.1.1), 00:13:18/00:02:56, flags: T
Incoming interface: Serial0/0, RPFnbr 0.0.0.0
Outgoing interface list:
Serial3/0, Forward/Sparse-Dense, 00:13:18/00:00:00
Verify the OIL
Check the PIM neighbor

WAN#sh ip pim neighbor
PIM Neighbor Table
Neighbor Interface Uptime/Expires Ver DR
Address Prio/Mode
10.1.11.2 Serial0/0 01:03:54/00:01:16 v2 1 / S
10.1.21.2 Serial3/0 01:13:06/00:01:26 v2 1 / S
Solution
Enable PIM on the WAN router towards the GM
WAN(config)#int s2/0
WAN(config-if)#ip pim sparse-dense-mode
WAN(config-if)#end
%PIM-5-NBRCHG: neighbor 10.1.20.2 UP on interface

Serial2/0 (vrf default)
Check multicast routing path again

Re-test with multicast ping
Verify GM now receives the multicast rekey messages
Unicast Rekey Failing
Transient Network Issues
Due to transient changes in the network, unicast rekey packets might not
make it to the GM(s)
If the GMs does not receive the rekey, it will have to re-register
Symptoms:
Missing Following syslog on GM:
%GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to 10.1.21.2

with seq # 3
GM shows re-registration syslog:
%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1 may have expired/been

cleared, or didn't go through. Re-register to KS.
%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group G1 using
address 10.1.20.2
Unicast Rekey Failing
Troubleshooting and Solution
Verify whether the rekeys are not being sent, not being received
or not being processed
KS: GM:
show crypto gdoi ks members show crypto gdoi gm rekey
Group Member Information : Group G1 (Unicast)

Number of rekeys sent for group G1 : 380 Number of Rekeys received (cumulative) : 0
Number of Rekeys received after registration : 0
Group Member ID : 10.1.20.2 Number of Rekey Acks sent : 0
Group ID : 3333 Rekey (KEK) SA information :
Group Name : G1 dstsrcconn-id my-cookie his-cookie
Key Server ID : 10.1.11.2 New : 10.1.20.2 10.1.11.2 1098 44F7FC328302AC61
Rekeys sent : 1 Current : 10.1.20.2 10.1.11.2 1098 44F7FC328302AC61
Rekeys retries : 0 Previous: --- --- --- --- ---
Rekey Acks Rcvd : 0
Rekey Acks missed : 0
Always configure retransmissions to overcome transient issues

Unicast rekey dropped
rekey retransmit 30 number 3
Make sure UDP port 848 is not blocked in the data path
Rekey Fails Signature Validation
Primary KS fails, GM receives rekey from secondary KS, but

receives error:
*Apr 27 18:18:19.511: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode
failed with peer at 10.1.12.2
Syslog is not conclusive, lets see what we can get with some debugs
Signature validation failed!
GM1# debug crypto isakmp
Crypto ISAKMP debugging is on
GM1#
GM1# debug crypto gdoi
GDOI Generic Debug level: (Error, Terse)
*Apr 27 18:18:19.251: ISAKMP (0:1014): received packet from 10.1.12.2 dport 848
sport 848 Global (R) GDOI_REKEY
*Apr 27 18:18:19.251: GDOI:INFRA:(G1:0:1014:HW:0):Received Rekey Message!
*Apr 27 18:18:19.259: GDOI:INFRA:(G1:0:1014:HW:0):Signature Invalid! status = 13
*Apr 27 18:18:19.259: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode failed
with peer at 10.1.12.2
*Apr 27 18:18:19.259: ISAKMP: Receive GDOI rekey: Processing Failed. IKMP error = 6
Rekey Fails Signature Validation
Solution
Problem:
Secondary KS has its own RSA key pair instead of the
exported key pair from the primary KS1 KS2
To verify, compare the RSA key pairs

MPLS/Private IP
KS#show crypto key mypubkey rsa

GM1 GM2
Solution:
Generate exportable RSA key pair on the primary KS
KS1(config)#crypto key generate rsa modulus 1024 exportable label key1
Export RSA key pair to all secondary KSs

KS2(config)#crypto key import rsa key1 pem terminal <passphrase>
IKE Setup
Encryption Policy
Key RenewalRekey
Control Plane Replay Check Detection
Control Plane messages can carry time sensitive information and therefore
require replay protection
Rekey messages from KS to GM
COOP Announcement messages between KSs
Sequence number check to protect against replayed messages
Pseudotime check to protect against delayed messages with TBAR
enabled
Control Plane Replay check added in IOS version 12.4(15)T10, 12.4(22)T3,
12.4(24)T2, 15.0(1)M, and later
Code interoperability issue
Problem: customer upgraded IOS on a GM to 15.0(1)M for a bug fix,

and started to experience KEK rekey failures
The following errors are observed in the syslog
%GDOI-3-GDOI_REKEY_SEQ_FAILURE: Failed to process rekey seq # 1 in seq payload

for group G1, last seq # 11
%GDOI-3-GDOI_REKEY_FAILURE: Processing of REKEY payloads failed on GM 10.1.13.2
in the group G1, with peer at 10.1.11.2
%CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode failed with peer at
10.1.11.2
Code interoperability issue - solution
KS does not support control plane replay detection, and resets the rekey
sequence # for KEK rekey
GM interprets that as a replayed rekey message
Solution is to upgrade the KS to an IOS version that also supports the
control plane replay detection
New behavior KEK Rekey
*Apr 6 15:41:26.932: %GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from

10.1.11.2 to 10.1.13.2 with seq # 8
GM1#
*Apr 6 15:42:01.940: %GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from
10.1.11.2 to 10.1.13.2 with seq # 1
TEK Rekey with seq# reset
Control Plane Replay Check IOS Upgrade procedure
Recommended IOS releases

IOS: 15.2(4)M3
IOS-XE: 15.1(3)S4
IOS upgrade procedure

Step 1. Upgrade a secondary KS first, wait until COOP KS election is
completed
Step 2. Repeat step 1 for all secondary KS
Step 3. Upgrade primary KS
Step 4. Upgrade Group Members
IKE Setup
Encryption Policy
Key RenewalRekey
Control Plane Fragmentation Issues
COOP Announcement Packets
In a large network (1500+ GMs), COOP update packet becomes larger

than the default maximum buffer size
Default huge buffer size is 18024 bytes
Syslog message appears on the KSs:
%SYS-2-GETBUF: Bad getbuffer, bytes= 18872 -Process= "Crypto IKMP", ipl= 0, pid= 183
Tune buffers to increase huge buffers and add buffers to permanent list:
buffers huge permanent 10
buffers huge size 65535
Control Plane Fragmentation Issues (cont.)
COOP Announcement Packets
Large ANN messages are fragmented in transit between KSs

Can have up to 40+ IP fragments KS1 Frag
1
KS2
Frag2
One dropped fragment -> entire ANN dropped Frag3
How to identify? Frag4
%GDOI-3-COOP_KS_UNREACH: Cooperative KS 10.1.11.1 Unreachable in group G1.

%GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.1.12.2 in group G1 transitioned to FragN
Primary (Previous Primary = 10.1.11.2)
KS1#show ip traffic | section Frags

Frags: 10 reassembled, 3 timeouts, 0 couldn't reassemble
0 fragmented, 0 fragments, 0 couldn't fragment
Need to look at transit path features that may drop fragments,

Firewall, VFR, reassembly buffer size, etc.
Troubleshooting GETVPN Data Plane
Ultimately all problems manifest at the data plane -my user

application is not working over GETVPN!
But where really is the problem?
Control Plane
Events that lead up to SAs getting installed on the GMs
Data plane
Policy downloaded with SAs installed but traffic is not flowing
Generic IPSec Data Plane Troubleshooting
Need to have complete understanding of the forwarding path and how

to checkpoint it
Which device is the culprit, encrypting or decrypting router?
In which direction is the problem happening, ingress or egress?
Some syslogs may help reveal data plane drops
Data plane errors are typically rate limited
Common errors include replay, authentication failures
Heavily dependent upon show commands and counters to trace the
packet path
Sniffer capture of limited use due to encryption, however
ESP-NULL same crypto processing except packets not encrypted
DSCP coloring of packets to uniquely identify a flow
GETVPN Data Plane
IPSec tunnel mode just like IPSec classic so most IPSec troubleshooting
techniques still apply, however
Symmetrical encryption policy requirement
Unique challenges with Header Preservation
PMTUD
Time Based Anti-Replay
Extra encapsulation overhead Fragmentation boundary condition calculation
Timer Based Anti-Replay failure
Data Plane Troubleshooting Tools
Interface counters
Encryption/decryption counters
Netflow
IP Accounting
ACL
DSCP packet coloring
Embedded Packet Capture (EPC)
IPSec Data Plane Packet Flow Checkpoints
3 4
Traffic Direction 1 6
GM1
Private WAN
GM2
Client Server
2 5
Encrypting GM Decrypting GM
1. Ingress LAN interface 4. Ingress WAN interface
Input ACL Input ACL
Ingress Netflow Ingress Netflow
Embedded Packet Capture
2. Crypto engine Input IP precedence accounting
show crypto ipsec sa
5. Crypto engine
show crypto session detail show crypto ipsec sa
3. Egress WAN interface show crypto session detail
Egress Netflow 6. Egress WAN interface
Embedded Packet Capture Egress Netflow
Output IP precedence accounting
Importance of a Controlled Test
The case for ping x.x.x.x timeout 0

Separation from background traffic
Poor mans conditional filter
Packet coloring/marking
Tools to monitor based on DSCP/Precedence marking
ESP-NULL
IP characteristics for seemingly application issues
Ping works but TCP doesnt?
Why does IPSec care about TCP, or does it?
Encrypting GM Data Plane Flow
Verify clear traffic being received with Ingress Netflow

interface Ethernet0/0
ip address 192.168.13.1 255.255.255.0
ip flow ingress
!
GM1# show ip cache flow
<snip>
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Et0/0 192.168.13.2 Se1/0 192.168.14.2 06 E443 0017 11
Verify encryption operation performed

TCP port 23 = telnet
Lack of per-flow granularity
GM1# show crypto session detail
<snip>
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 4, origin: crypto map
Inbound: #pkts dec'ed 162 drop 0 life (KB/Sec) 0/146
Outbound: #pkts enc'ed 170 drop 0 life (KB/Sec) 0/146
Encrypting GM Data Plane Flow Cont.
Verify encrypted traffic existing GM with egress Netflow

interface Serial/0
ip address 10.1.13.2 255.255.255.252
ip flow egress
! Protocol 50 = ESP
GM1#show ip cache flow
<snip>
Et0/0 192.168.13.2 Se1/0* 192.168.14.2 32 EE5B 2BEF 170
GM1#show crypto ipsec sa

interface: Serial1/0 Active IPSec SA SPI
<snip>
current outbound spi: 0xEE5B2BEF(3998952431)
If per L4 flow granularity is desired, can use inbound precedence

coloring and egress precedence accounting
Decrypting GM Data Plane Flow
Verify encrypted traffic arriving on GM with Netflow
Protocol 50 = ESP
<snip>
Se1/0 192.168.13.2 Et0/0 192.168.14.2 32 EE5B 2BEF 170
Verify traffic decryption Inbound IPSec SA SPI

GM2#show crypto session detail
<snip>
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 10, origin: crypto map
Inbound: #pkts dec'ed 170 drop 0 life (KB/Sec) 0/150
Outbound: #pkts enc'ed 162 drop 0 life (KB/Sec) 0/150
Verify clear traffic forwarding post decryption

TCP port 23 = telnet
<snip>
Se1/0 192.168.13.2 Et0/0* 192.168.14.2 06 E6CC 0017 170
GETVPN Common Issues Data Plane
Asymmetrical Encryption Policy
Fragmentation/Path MTU
Other data plane issues common to IPSec
KS Policy Issues
Data Plane Traffic Failure
Encryption policies (what needs to be encrypted) are defined centrally at the KS

Symmetrical ACLs should be defined to either permit
or to deny traffic from getting encrypted
If the traffic is not being encrypted or being blocked, verify we have symmetrical ACL
MPLS/Private IP
GM1 GM2
Ethernet 0/0: Ethernet 0/0:
192.168.20.0/24 192.168.21.0/24
KS Access-list
ip access-list extended ENCPOL
permit ip 192.168.20.0 0.0.0.255 192.168.21.0 0.0.0.255
permit ip 192.168.21.0 0.0.0.255 192.168.20.0 0.0.0.255
Other data plane issues common to IPSec
Fragmentation Issues
PMTU Discovery
Large packets with the DF bit set may get black-holed in the GETVPN network
MTU 1500 MTU 1500
GM1
MTU 1000 GM2
1400B 1460B
ICMP 3/4
Server sends a large packet with the DF bit set in an attempt to

perform network PMTUD
PMTUD and GETVPN
Encrypting GM adds IPSec overhead and forwards it

Intermediate router drops the packet and sends back icmp3/4 to perform
PMTUD, two possibilities
This ICMP dropped by the encrypt GM because its not encrypted based on the
encryption policy
This ICMP gets forwarded to the end host but gets dropped due to
unauthenticated payload
Bottom line: PMTUD does not work with the current header preservation
implementation of GETVPN
PMTUD and GETVPN
Solution
Implement ip tcp adjust-mss to reduce the TCP packet segment size
Clear the DF bit in the encapsulating header
User Traffic
Encrypting GM
DF=0 DF=0 Data
interface Ethernet0/0 DF=1 Data

ip address 192.168.13.1 255.255.255.0
ip policy route-map clear-df-bit
!
route-map clear-df-bit permit 10
match ip address 111
set ip df 0
!
access-list 111 permit tcp any any
Other Data Plane Issues Common to IPSec
IPSec drop due to packet corruption
IPSec integrity check makes IPSec packets a lot more sensitive to packet
corruption in the network
Packet corruption symptoms
%CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=695
local=192.168.14.2 remote=192.168.13.2 spi=7C4E759F seqno=00000001
How to prove packets are corrupted in the network?

Enable EPC to capture packets into a circular buffer on both GMs
Use EEM (Embedded Event Manager) to
Synchronize and stop the capture on both routers when the RECVD_PKT_MAC_ERR message is
logged
Notify the network operator by email
Retrieve both captures to examine for packet corruption

GETVPN Troubleshooting Summary
Have a clear and concise problem description

Try to break the problem down to either control or data plane
Understand the expected protocol flow on the control plane and how
to check for them
Understand where/how to checkpoint the data plane
Syslog and event trace your friend
There is always TAC!
Complete Your Online Session Evaluation
Give us your feedback and
you could win fabulous prizes.
Winners announced daily.
Receive 20 Cisco Daily Challenge
points for each session evaluation
you complete.
Complete your session evaluation
online now through either the mobile
app or internet kiosk stations.
Maximize your Cisco Live experience with your
free Cisco Live 365 account. Download session
PDFs, view sessions on-demand and participate in
live activities throughout the year. Click the Enter
Cisco Live 365 button in your Cisco Live portal to
log in.
Appendix
GETVPN Scalability and Troubleshooting Tools

Key Server Scalability
Platform Crypto Card Max Number of GM Time to register to KS
7200/7201 VAM2+ 2000 15 sec *
3845 AIM-VPN/SSL-3 1000 15 sec *
3825 AIM-VPN/SSL-3 500 15 sec
7200/PKI VAM2+ 1000 20 sec **
* GM registration was distributed over two KSs to reduce the registration time
** GM registration was distributed over four KSs to reduce the registration time
GM Performance Attributes
(No Features)
PPS Mbps Max IMIX Latency(ms) Avg 100 pps Latency (ms)
871 Anti-Replay 3150 28 <10
1.18
No Anti-Replay 3232 28 <5
1841-onboard Anti-Replay 3506 33 <20
1.07
1841-aim/ssl Anti-Replay 8420 84 <10
0.68
0.47
0.34
0.33
0.25
(No Features)
PPS Mbps Max IMIX Latency(ms) Avg 100 pps Latency (ms)
3825-onboard Anti-Replay 35,505 283 <1
0.64
No Anti-Replay 35,500 283 <5
3825-aim/ssl Anti-Replay 44,170 199 <1
0.66
3845-onboard Anti-Replay 46,028 284 <5
0.76
3845-aim/ssl Anti-Replay 54,020 200 <1
0.81
7200-g1vam2+ Anti-Replay 60,592 266 <5
0.69
7200-g2vam2+ Anti-Replay 121,952 283 <5
0.17
7200-g2/vsa Anti-Replay
TBD
No Anti-Replay 160,000 980 TBD
ASR1000/FP5G Anti-Replay 440,000
TBD
No Anti-Replay 470,000 1,890 TBD
ASR1000/FP10G Anti-Replay 976,000 4,200
0.19
No Anti-Replay 1,011,000 4,220 <0.270
ASR1000/FP20G Anti-Replay 2,655,000 TBD
0.001
No Anti-Replay 2,685,000 8,530 <0.015
(No Features)
Frame Size ASR 1004 (10Gig) 7200 VSA 3845 AIM- ISRG2 ISRG2 ISRG2
VPN/SSL-3 3945 Onboard 2951 Onboard 1941
Crypto Crypto Onboard
Crypto
1400 Byte 4759 Mbps 925 Mbps 200 Mbps 820 Mbps 268 Mbps 154
Mbps
IMIX (90 Bytes 61%, 2289 Mbps 780 Mbps 177 Mbps 261Mbps 160 Mbps 64Mbp
594 bytes 24%, 1418 15%) s
GETVPN Verification
Common KS Syslog Messages
Syslog Messages Explanation
COOP_CONFIG_MISMATCH The configuration between the primary key server and secondary key server are
mismatched.
COOP_KS_ELECTION The local key server has entered the election process in a group.
COOP_KS_REACH The reachability between the configured cooperative key servers is restored.
COOP_KS_TRANS_TO_PRI The local key server transitioned to a primary role from being a secondary server
in a group.
COOP_KS_UNAUTH An authorized remote server tried to contact the local key server in a group.
Could be considered a hostile event.
COOP_KS_UNREACH The reachability between the configured cooperative key servers is lost. Could
be considered a hostile event.
KS_GM_REVOKED During rekey protocol, an unauthorized member tried to join a group. Could be
considered a hostile event.
KS_SEND_MCAST_REKEY Sending multicast rekey.
KS_SEND_UNICAST_REKEY Sending unicast rekey.
KS_UNAUTHORIZED During GDOI registration protocol, an unauthorized member tried to join a group.
Could be considered a hostile event.
UNAUTHORIZED_IPADDR The registration request was dropped because the requesting device was not
authorized to join the group.
GETVPN Verification
Common GM Syslog Messages
Syslog Messages Explanation
GM_CLEAR_REGISTER The clear crypto gdoi command has been executed by the local
group member.
GM_CM_ATTACH A crypto map has been attached for the local group member.
GM_CM_DETACH A crypto map has been detached for the local group member.
GM_RE_REGISTER IPSec SA created for one group may have been expired or
cleared. Need to reregister to the key server
GM_RECV_REKEY Rekey received
GM_REGS_COMPL Registration complete
GM_REKEY_TRANS_2_MULTI Group member has transitioned from using a unicast rekey
mechanism to using a multicast mechanism.
GM_REKEY_TRANS_2_UNI Group member has transitioned from using a multicast rekey
mechanism to using a unicast mechanism.
PSEUDO_TIME_LARGE A group member has received a pseudotime with a value that is largely
different from its own pseudotime.
REPLAY_FAILED A group member or key server has failed an anti-replay check.
Packet marking Techniques
IP TOS byte copied from inner header to the encapsulating

delivery header by default
How to mark
PBR
MQC
Local ping
How to monitor
IP precedence accounting
ACL counters
ToS/Precedence/DSCP Reference Chart
Least
7 6 5 4 3 2 1 0 Significant
Bit
IP Precedence Priority
DSCP ToS Byte
ToS
Hex - Decimal IP Precedence DSCP Binary
E0 224 7 Network Control 56 CS7 11100000
C0 192 6 Internetwork Control 48 CS6 11000000
B8 184 5 Critical 46 EF 10111000
A0 160 40 CS5 10100000
88 136 4 Flash Override 34 AF41 10001000
80 128 32 CS4 10000000
68 104 3 Flash 26 AF31 01101000
60 96 24 CS3 01100000
48 72 2 Immediate 18 AF21 01001000
40 64 16 CS2 01000000
20 32 1 Priority 8 CS1 00100000
00 0 0 Routine 0 Dflt 00000000
Packet marking - Examples
PBR
ip policy route-map mark
!
access-list 150 permit ip host 172.16.1.2 host 172.16.254.2
!
route-map mark permit 10
match ip address 150
set ip precedence flash-override
MQC IP flow in question marked with

precedence 4
class-map match-all my_flow
match access-group 150
!
policy-map marking
class my_flow
set ip precedence 4
!
service-policy input marking
Packet marking - Examples
Router Ping
Router#ping ip
Target IP address: 172.16.254.2
Repeat count [5]: 100
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]: 128
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 100, 100-byte ICMPEchos to 172.16.254.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Packet marking - Monitoring
IP Precedence Accounting
ip address 192.168.1.2 255.255.255.0
ip accounting precedence input
middle_router#show interface precedence

Ethernet0/0
Input
Precedence 4: 100 packets, 17400 bytes
Interface ACL
middle_router#sh access-list 144
Extended IP access list 144
10 permit ip any any precedence routine
20 permit ip any any precedence priority
30 permit ip any any precedence immediate
40 permit ip any any precedence flash
50 permit ip any any precedence flash-override (100 matches)
60 permit ip any any precedence critical
70 permit ip any any precedence internet (1 match)
80 permit ip any any precedence network
Using Packet Captures for Data Plane Issues
Packet captures can provide detailed packet information at the bits/bytes level
The new packet capture infrastructure introduced in 12.4(20)T makes this easy
to do
Ability to capture IPv4 and IPv6 packets in the CEF path
Configurable capture buffer and capture point parameters
Extensible output filtering and export capabilities
Support for various WAN encapsulation types
Using IOS Embedded Packet Captures
Key Configuration Steps
Create the capture buffer and capture point
Associate the capture point to the buffer
Start/stop the capture
Router#monitor capture buffer test-buffer

Router#monitor capture buffer test-buffer filter access-list 120
Filter Association succeeded
Router#
Router#monitor capture point ipcef test-capture serial 2/0 both
*Mar 26 20:33:10.896: %BUFCAP-6-CREATE: Capture Point test-capture created.
Router#monitor capture point associate test-capture test-buffer
Router#monitor capture point start test-capture
*Mar 26 20:34:03.108: %BUFCAP-6-ENABLE: Capture Point test-capture enabled.
Router#
Router#monitor capture point stop test-capture
*Mar 26 20:34:21.636: %BUFCAP-6-DISABLE: Capture Point test-capture disabled.
Using IOS Embedded Packet Captures
Now we have the packets captured, whats next?
Dump the packet on the router itself
Router# show monitor capture buffer test-buffer dump
15:34:07.228 EST Mar 26 2009 : IPv4 LES CEF : Se2/0 None
05CECE30: 0F00080045C0002C ....E@.,

05CECE40: 6D170000FE0649DD 02010102 01010114 m...~.I]........
05CECE50: 0017A3530FB6B9523EF1499C 60121020 ..#S.69R>qI.`..
05CECE60: 917A0000 02040218 00 .z.......
Dump the packet on the router itself

Or export it out and analyze it in Wireshark
Router# monitor capture buffer test-buffer export?
ftp: Location to dump buffer
http: Location to dump buffer
https: Location to dump buffer
rcp: Location to dump buffer
scp: Location to dump buffer
tftp: Location to dump buffer
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public
Use EEM and EPC to catch Packet Corruption
Peer1
event manager applet detect_bad_packet
event syslog pattern "RECVD_PKT_MAC_ERR"
action 1.0 cli command "enable"
action 2.0 cli command "monitor capture point stop test"
action 3.0 syslog msg "Packet corruption detected and capture stopped!"
action 4.0 snmp-trap intdata1 123456 strdata ""
Peer2
event manager applet detect_bad_packet

event snmp-notification oid 1.3.6.1.4.1.9.10.91.1.2.3.1.9. oid-val "123456" op
eq src-ip-address 20.1.1.1
action 1.0 cli command "enable"
action 2.0 cli command "monitor capture point stop test"
action 3.0 syslog msg "Packet corruption detected and capture stopped!"
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public

Troubleshooting GETVPN Deployments - BRKSEC-3051

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Troubleshooting GETVPN Deployments - BRKSEC-3051

Caricato da

Copyright:

Formati disponibili

Wen Zhang - Technical Leader, Services

GETVPN Solution Overview

BRKSEC-2054 Deploying GET to Secure VPNs

Remot Access SW WAN Edge

FlexVPN DMVPN GETVPN

Public Internet Public Internet

Multicast replication at Multicast replication at Multicast replication in

Uses three main components

Traffic Encryption Key (TEK)

KS authenticates and authorizes the GM

Step 2: Data Plane Encryption

Step 3: Periodic Rekey of Keys

IPSec New IP Header ESP IP Header IP Payload

GETVPN Preserved Header ESP IP Header IP Payload

IP header preserved by VPN Gateway

Encrypted/Authenticated Using Group SA

Original Src and Original IP

Rekey Message sent from key server to all group members

Key Server maintains state of active group members

Sequence number based anti-replay only works with

If Senders pseudotime falls in the below Receiver window,

Reject Accept Reject

Initial PTr - W PTr PTr + W

Packet 1 and Packet 2 have pseudotimeT0, providing loose anti-replay

Cooperative KS1 Cooperative KS2

Cooperative KS1 Cooperative KS2 (Secondary)

RSA Keys (generated only on KSs) are required for rekey

RSA public key distribution from Key Server to Group Member:

Platform Group Member Key Server

GETVPN Provides complete segregation of control and data plane

Ultimately all problems manifest at the data plane -my user

Control Plane Data Plane

Common Control Plane Issues

Group Member Information :

Number of rekeys sent for group GET: 4

GMs IP address Group Member ID : 131.1.1.1

GM rekey history Sent seq num : 1 2 3 4

All feature components can be debugged at 5 levels

Debug Level What you will get

a conditional filter based on group or peer address

To enable conditional debugs

2) Enable relevant debugs of interest as usual

KS1#debug crypto gdoi ks registration all-levels

Turn off console logging

Light weight event buffer to supplement syslogs

GM1#show monitor event-trace gdoi merged all

Ser 1/0: 10.1.20.2 Ser 1/0: 10.1.21.2

COOP KS IKE Setup

GM-KS IKE Setup

Periodic Key Renewal and Distribution (Rekeys)

Control Plane Replay Check

Control Plane Packet Fragmentation Issue

If a GM fails to register with the KS, it will continue to attempt

%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for

IKE negotiation failure

Verify routing information on KS and GM and try ping KS from the GM

GM1#show crypto isakmp sa

IPv6 Crypto ISAKMP SA

IKE SA not getting established; cant

Verify the logs on the Key Server

Syslog pointing to a mismatched pre-shared key configuration

KS Config: crypto isakmp key cicso address 10.1.20.2