CP-1005 - Top 10 ASA Firewall and VPN Troubleshooting Techniques Commands - 1 Hour PDF

Top 10 ASA Firewall and VPN
Troubleshooting Techniques
Omar Santos
os@cisco.com
Cisco PSIRT
Security Research and Operations
Agenda
Introduction of ASA Packet Flow
Top 5 Techniques when Troubleshooting

Firewall Problems
Top 5 Techniques when Troubleshooting

VPN Problems
Q&A
2014 Cisco and/or its affiliates. All rights reserved.
Understanding the
2013-2014 Cisco and/or its affiliates. All rights reserved.
Understanding Packet Flow

To effectively troubleshoot a connectivity problem, one must first understand the packet path
through the network.

Attempt to isolate the problem down to a single device.
Then perform a systematic walk of the packet path through the device to determine where the
problem could be
For problems relating to the Cisco ASA, always:
Determine the flow: Protocol, Source IP, Destination IP, Source Port, Destination Port
Determine the logical (named) interfaces through which the flow passes
TCP outside
172.16.164.216:5620 inside
192.168.1.150:50141, idle 0:00:00, bytes 0, flags saA
All firewall connectivity issues can be simplified to two

interfaces (ingress and egress) and the policies tied to both
4
Example Flow
With the Flow defined,

examination of configuration
issues boils down to just the two
Interfaces: Inside and Outside
TCP Flow
Source IP
: 10.1.1.9
Destination IP : 198.133.219.25
Source Port
: 11030
Destination Port
80
Interfaces
Source: Inside
Destination: Outside
Packet Flow
Servers
10.1.1.9
Eng
Accounting
Outside
198.133.219.25
Packet Processing: Ingress Interface

IPS or CX
Module
Yes
RX
Pkt
Ingress
Interface
Existing
Conn
No
NAT
Untranslate
Yes
ACL
Permit
No
Packet arrives on ingress interface
DROP
Yes
Stateful
Inspection
NAT IP
Header
No
DROP
Egress
Interface
L3
Route
No
No
DROP
DROP
Yes
L2
Addr
Yes
TX
Pkt
No
DROP
Input counters incremented by NIC and periodically retrieved by CPU
Software input queue (RX ring) is an indicator of packet load

Overrun counter indicates packet drops (usually packet bursts)
asa# show interface outside

Interface GigabitEthernet0/3 "outside", is up, line protocol is up
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
MAC address 0026.0b31.36d5, MTU 1500
IP address 148.167.254.24, subnet mask 255.255.255.128
54365986 packets input, 19026041545 bytes, 0 no buffer
Received 158602 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
[]
input queue (blocks free curr/low): hardware (255/230)
output queue (blocks free curr/low): hardware (254/65)
Packet Processing: Locate Connection

IPS or CX
Module
Yes
RX
Pkt
Ingress
Interface
Existing
Conn
No
NAT
Untranslate
ACL
Permit
No
DROP
Yes
Yes
Stateful
Inspection
No
DROP
NAT IP
Header
Egress
Interface
L3
Route
No
No
DROP
DROP
Yes
L2
Addr
Yes
TX
Pkt
No
DROP
Check first for existing connection in conn table

If conn entry exists, bypass ACL check and process in Fastpath
asa# show conn
TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO
If no existing connection
TCP SYN or UDP packet, pass to ACL and other policy checks in Session Manager
TCP non-SYN packet, drop and log

ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK
interface inside
on
Packet Processing: NAT Un-Translate

IPS or CX
Module
Yes
RX
Pkt
Ingress
Interface
Existing
Conn
No
NAT
Untranslate
ACL
Permit
No
DROP
Yes
Yes
Stateful
Inspection
NAT IP
Header
No
DROP
Egress
Interface
L3
Route
No
No
DROP
DROP
Yes
L2
Addr
Yes
TX
Pkt
No
DROP
Incoming packet is checked against NAT rules

Packet is un-translated first, before ACL check
In ASA 8.2 and below, incoming packet was subjected to ACL check prior to un-translation
NAT rules can determine the egress interface at this stage
Packet Processing: ACL Check

IPS or CX
Module
Yes
RX
Pkt
Ingress
Interface
Existing
Conn
No
NAT
Untranslate
ACL
Permit
Yes
No
DROP
Yes
Stateful
Inspection
NAT IP
Header
No
DROP
Egress
Interface
L3
Route
No
No
DROP
DROP
Yes
L2
Addr
Yes
TX
Pkt
No
DROP
First packet in flow is processed through ACL checks

ACLs are first configured match
First packet in flow matches ACE, incrementing hit count by one
asa# show access-list inside
access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)
Denied packets are dropped and logged

ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access-group "inside"
Packet Processing: Stateful Inspection

IPS or CX
Module
Yes
RX
Pkt
Ingress
Interface
Existing
Conn
No
NAT
Untranslate
ACL
Permit
No
DROP
Yes
Yes
Stateful
Inspection
No
DROP
NAT IP
Header
Egress
Interface
L3
Route
No
No
DROP
DROP
Yes
L2
Addr
Yes
TX
Pkt
No
DROP
Stateful inspection ensures protocol compliance at TCP/UDP/ICMP level

(Optional) Customisable application inspection up to Layer 7 (FTP, SIP, and so on)
Rewrite embedded IP addresses, open up ACL pinholes for secondary connections
Additional security checks are applied to the application payload

ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to 209.165.202.130 on
interface inside
ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port
before SETUP
10
Packet Processing: NAT IP Header

IPS or CX
Module
Yes
RX
Pkt
Ingress
Interface
Existing
Conn
No
NAT
Untranslate
ACL
Permit
No
DROP
Yes
Yes
Stateful
Inspection
NAT IP
Header
Egress
Interface
No
No
DROP
DROP
No
DROP
L3
Route
Yes
L2
Addr
Yes
TX
Pkt
No
DROP
Translate the source and destination IP addresses in the IP header

Translate the port if performing PAT
Update header checksums
(Optional) Following the above, pass packet to IPS or CX module
Real (pre-NAT) IP address information is supplied as meta data
11
Packet Processing: Egress Interface

IPS or CX
Module
Yes
RX
Pkt
Existing
Conn
Ingress
Interface
No
NAT
Untranslate
ACL
Permit
No
DROP
Yes
Yes
Stateful
Inspection
No
DROP
NAT IP
Header
Egress
Interface
L3
Route
No
No
DROP
DROP
Yes
L2
Addr
Yes
TX
Pkt
No
DROP
Packet is virtually forwarded to egress interface (not forwarded to the Ethernet NIC yet)
Egress interface is determined first by translation rules or existing conn entry, only THEN
the routing table

If NAT does not divert to the egress interface, the global routing table is consulted to
determine egress interface

Inside
DMZ
172.16.0.0/16
Outside
172.16.12.0/24
172.16.12.4
Packets received on outside and destined to

192.168.12.4 get routed to 172.16.12.4 on
inside based on NAT configuration.
nat (inside,outside) source static 172.16.0.0-net 192.168.0.0-net
nat (dmz,outside) source static 172.16.12.0-net 192.168.12.0-net
12
Packet Processing: L3 Route Lookup

IPS or CX
Module
Yes
RX
Pkt
Ingress
Interface
Existing
Conn
No
NAT
Untranslate
ACL
Permit
No
DROP
Yes
Yes
Stateful
Inspection
NAT IP
Header
No
DROP
Egress
Interface
L3
Route
No
No
DROP
DROP
Yes
L2
Addr
Yes
TX
Pkt
No
DROP
Once at egress interface, an interface route lookup is performed

Only routes pointing out the egress interface are eligible
Remember: NAT rule can forward the packet to the egress interface, even though the
routing table may point to a different interface

If the destination is not routable out of the identified egress interface, the packet is dropped
%ASA-6-110003: Routing failed to locate next hop for TCP from inside:192.168.103.220/59138
to dmz:172.15.124.76/23
13
Packet Processing: L2 Address Lookup

IPS or CX
Module
Yes
RX
Pkt
Ingress
Interface
Existing
Conn
No
NAT
Untranslate
ACL
Permit
No
DROP
Yes
Yes
Stateful
Inspection
NAT IP
Header
Egress
Interface
No
DROP
L3
Route
No
No
DROP
DROP
Yes
L2
Addr
Yes
TX
Pkt
No
DROP
Once a Layer 3 route has been found, and next hop IP address identified, Layer 2
resolution is performed
Layer 2 rewrite of MAC header
If Layer 2 resolution fails no syslog
show arp will not display an entry for the L3 next hop
debug arp will indicate if we are not receiving an ARP reply
arp-req: generating request for 10.1.2.33 at interface outside
arp-req: request for 10.1.2.33 still pending
14
Packet Processing: Transmit Packet

IPS or CX
Module
Yes
RX
Pkt
Ingress
Interface
Existing
Conn
No
NAT
Untranslate
ACL
Permit
No
DROP
Yes
Yes
Stateful
Inspection
No
DROP
NAT IP
Header
Egress
Interface
L3
Route
No
No
DROP
DROP
Yes
L2
Addr
Yes
TX
Pkt
No
DROP
Packet is transmitted on wire

Interface counters will increment on interface
Underrun counter indicates drops due to egress interface oversubscription

TX ring is full
asa# show interface outside

Interface GigabitEthernet0/1 "outside", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
MAC address 503d.e59d.90ab, MTU 1500
IP address 172.18.124.149, subnet mask 255.255.255.0
273399 packets output, 115316725 bytes, 80 underruns
input queue (blocks free curr/low): hardware (485/441)

output queue (blocks free curr/low): hardware (463/0)
15
*NOT RANKED BY IMPORTANCE
17
Uses of Syslogs
Primary mechanism for recording connections to and through the firewall
The best troubleshooting tool available
Archival Purposes
Live Debugging Purposes
Console
Syslog/FTP Server
Trap
SNMP Server
Syslog.
Flash
Local
Buffer
ASDM
18
ASA Syslog Level vs. Number of Messages

Log
Level
Number of Messages (SUM)

Description
Ver. 7.0
Ver. 7.2
Ver. 8.0
Ver. 8.1
Ver. 8.2
Ver. 8.3
Ver. 8.4
Ver. 9.1
Emergencies
Alerts
62 (62)
77 (77)
78 (78)
87 (87)
87 (87)
95 (95)
109 (109)
117 (117)
Critical
29 (91)
35 (112)
49 (127)
50 (137)
56 (143)
57 (152)
63 (172)
72 (189)
Errors
274 (365)
334 (446)
361 (488)
363 (500)
384 (527)
408 (560)
448 (620)
521 (710)
Warnings
179 (544)
267 (713)
280 (768)
281 (781)
315 (842)
324 (884)
357 (997)
420 (1130)
Notifications
161 (705)
206 (919)
216 (984)
218 (999)
237 (1079)
246 (1130)
265 (1242)
285 (1415)
Informational
234 (939)
302 (1221)
335 (1319)
337 (1336)
368 (1447)
377 (1507)
395 (1637)
430 (1845)
Debugging
217 (1156)
258 (1479)
266 (1585)
267 (1603)
269 (1716)
269 (1776)
276 (1913)
295 (2140)
19
Custom Syslog Levels

Assign any syslog message to any available level
Problem:
Levels
You want to record what exec commands are

being executed on the firewall; syslog ID 111009
records this information, but by default it is at
level 7 (debug)
ASA-7-111009: User johndoe executed cmd: show run
The problem is we dont want to log all 1775
other syslogs that are generated at debug level
0Emergency
1Alert
2Critical
3Errors
4Warnings
5Notifications
6Informational
asa(config)# logging message 111009 level 3
7Debugging
ASA-3-111009: User johndoe executed cmd: show run

20
TCP Connection Termination Reasons

If a TCP flow was built through the ASA, it will always log a teardown reason
TCP teardown message is logged at level 6 (informational) by default
If you are having problems abnormal connection termination, temporally increase your logging
level (or change the syslog level, and check the teardown reason
%ASA-6-302014: Teardown TCP connection 90 for outside:10.1.1.1/80 to inside:192.168.1.101/1107 duration 0:00:30 bytes 0
SYN Timeout
%ASA-6-302014: Teardown TCP connection 3681 for DMZ:172.16.171.125/21 to inside:192.168.1.110/24245 duration 0:01:03
bytes 12504 TCP Reset-O
21
21

Reason
Description
Conn-Timeout
Connection Ended Because It Was Idle Longer Than the Configured Idle Timeout
Deny Terminate
Flow Was Terminated by Application Inspection
Failover Primary Closed
The Standby Unit in a Failover Pair Deleted a Connection Because of a Message

Received from the Active Unit
FIN Timeout
Force Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout
Flow Closed by Inspection
Flow Was Terminated by Inspection Feature
Flow Terminated by IPS
Flow Was Terminated by IPS
Flow Reset by IPS
Flow Was Reset by IPS
Flow Terminated by
TCP Intercept
Flow Was Terminated by TCP Intercept
Invalid SYN
SYN Packet Not Valid
Idle Timeout
Connection Timed Out Because It Was Idle Longer than the Timeout Value
IPS Fail-Close
Flow Was Terminated Due to IPS Card Down
SYN Control
Back Channel Initiation from Wrong Side
22

Reason
Description
SYN Timeout
Force Termination After Twenty Seconds Awaiting

Three-Way Handshake Completion
TCP Bad Retransmission
Connection Terminated Because of Bad TCP Retransmission
TCP Fins
Normal Close Down Sequence
TCP Invalid SYN
Invalid TCP SYN Packet
TCP Reset-I
TCP Reset Was Sent From the Inside Host
TCP Reset-O
TCP Reset Was Sent From the Outside Host
TCP Segment Partial Overlap
Detected a Partially Overlapping Segment
TCP Unexpected Window Size Variation
Connection Terminated Due to a Variation in the

TCP Window Size
Tunnel Has Been Torn Down
Flow Terminated Because Tunnel Is Down
Unauth Deny
Connection Denied by URL Filtering Server
Unknown
Catch-All Error
Xlate Clear
User Executed the Clear Xlate Command
23
NetFlow Secure Event Logging (NSEL)

NetFlow v9 support added in ASA 8.1+
Provides a method to deliver binary logs at high speeds

Reduce processing overhead in printing logs
Combine multiple events into one NetFlow record
FlowSets Supported:
Flow Creation
Flow Teardown
Flow Denied
Flow Update in ASA 8.4(5)+ and 9.1(2)+

Remove redundant syslog messages
asa(config)# logging flow-export-syslogs disable
24
Case Study: Excessive Logging

logging
logging
logging
logging
logging
logging
logging
logging
enable
buffered debugging
console debugging
trap debugging
history debugging
host inside 192.168.1.10
host inside 192.168.1.11
host DMZ 192.168.2.121
snmp-server host inside 192.168.1.10

snmp-server host DMZ 192.168.2.121
flow-export destination inside 192.168.1.10

flow-export destination DMZ 192.168.2.121
4 logging destinations (buffer,

console, SNMP, and syslog)
3 syslog servers
3 SNMP servers
3 Netflow collectors
4 messages per PAT
connection (over 550 bytes)
%ASA-6-305011: Built dynamic TCP translation from inside:192.168.1.101/4675 to

1 connection:
outside:172.16.171.125/34605
%ASA-6-302013: Built outbound TCP connection 3367663 for outside:198.133.219.25/80
32 syslog messages
(198.133.219.25/80) to inside:192.168.1.101/4675 (172.16.171.125/34605)
26+ packets sent
%ASA-6-302014: Teardown TCP connection 3367663 for outside:198.133.219.25/80 to
100K connections/sec:
inside:192.168.1.101/4675 duration 0:00:00 bytes 1027 TCP FINs
2.8Gbps
%ASA-6-305012: Teardown dynamic TCP translation from inside:192.168.1.101/4675
to
outside:172.16.171.125/34605
duration
0:00:30
25
Case Study: Logging Optimization

Not logging to buffer
unless troubleshooting
Console logging is a
bottleneck (low rate)
Using minimum number of syslog

servers and Netflow collectors
logging enable
logging flow-export-syslogs disable
Do not duplicate syslogs

and Netflow data
logging list FAILOVER message 104003

Reduce severity
level for syslogs
logging trap errors

logging history FAILOVER
logging host inside 192.168.1.10
Send only certain

syslogs as SNMP traps
logging host DMZ 192.168.2.121

snmp-server host DMZ 192.168.2.121 poll
Not all SNMP servers

need to receive traps

flow-export destination DMZ 192.168.2.121
26
Logging Common Issues

logging flash-bufferwrap should only be used when logging to buffer at Level 1
logging history should only be used when you really have an SNMP server that you want to
receive all syslogs

logging console should only be enabled while actively troubleshooting on the console
logging standby should only be used if you want to receive double the syslogs
logging permit-hostdown should always be used with TCP syslogging
27
2
*NOT RANKED BY IMPORTANCE.
28
Xlate Table
show xlate displays information about NAT translations through the ASA
Second biggest memory consumer after conn table, no hardcoded size limit
You can limit the output to just the local or global IP
asa# show xlate local 10.2.1.2
5014 in use, 5772 most used
TCP PAT from inside:192.168.103.220/57762 to outside:10.2.1.2/43756 flags ri
idle 0:00:00 timeout 0:00:30
TCP PAT from inside:192.168.103.220/57761 to outside:10.2.1.2/54464 flags ri
idle 0:00:00
timeout
0:00:30
Depleted
NAT/PAT
pools
may cause connectivity issues
asa# show nat pool

TCP PAT pool outside, address 10.2.1.2, range 1-511, allocated 1
29
Detailed NAT Information

show nat displays information about the NAT table of the ASA
detail keyword will display object definitions

Watch the hit counts for policies that are not matching traffic
asa# show nat detail
Manual NAT Policies (Section 1)
1 (inside) to (outside) source static science-obj science-obj destination static vpn-obj vpn-obj
translate_hits = 0, untranslate_hits = 0
Source - Origin: 192.168.0.0/16, Translated: 192.168.0.0/16
Destination - Origin: 172.16.1.0/24, Translated: 172.16.1.0/24
Auto NAT Policies (Section 2)
1 (dmz) to (outside) source static webserver-obj 14.36.103.83
2 (inside) to (outside) source dynamic science-obj interface
Translate hits indicate

connections from real to
mapped
2014 Cisco and/orinterfaces
its affiliates. All rights reserved.
Untranslate hits indicate

connections from mapped to
real interfaces
Check specific
translation policies in
the applied order.
30
CONNECTION TABLE
3
31
Connection Table
asa# show conn detail
2 in use, 64511 most used
Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,
B - initial SYN from outside, b - TCP state-bypass or nailed,
C - CTIQBE media, c - cluster centralized,
D - DNS, d - dump, E - outside back connection, F - outside FIN, f - inside FIN,
G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data,
i - incomplete, J - GTP, j - GTP data, K - GTP t3-response
k - Skinny media, M - SMTP data, m - SIP media, n - GUP
O - outbound data, P - inside back connection, p - Phone-proxy TFTP connection,
q - SQL*Net data, R - outside acknowledged FIN,
R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN,
s - awaiting outside SYN, T - SIP, t - SIP transient, U - up,
V - VPN orphan, W - WAAS,
X - inspected by service module,
Narrow down the output with x - per session, Y - director stub flow, y - backup stub flow, Bidirectional byte count; use
NSEL to report each
show conn address <ip> Z - Scansafe redirection, z - forwarding stub flow
TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101,
flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127
UDP outside:172.18.124.1/123 dmz:10.1.1.9/123,
flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431
Conn flags indicate current

state
direction separately.
detail option adds uptime

and timeout information
32
Local Host Table

A local-host entry is created for every IP tracked by the ASA
It groups xlates, connections, and AAA information
Useful for monitoring connections terminating on servers or offending clients

asa# show local-host detail connection tcp 50
Interface dmz: 0 active, 0 maximum active, 0 denied
Interface inside: 1 active, 1 maximum active, 0 denied
local host: <192.168.103.220>,
TCP flow count/limit = 798/unlimited
TCP embryonic count to host = 0
TCP intercept watermark = unlimited
UDP flow count/limit = 0/unlimited
Conn:
TCP outside:172.18.124.76/80 inside:192.168.103.220/34078,
flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0
TCP outside:172.18.124.76/80 inside:192.168.103.220/34077,
flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0
(output
truncated)
Only display hosts that have

more than 50 active TCP
connections.
33
Connection Flags
Outbound Connection
inside
client
Inbound Connection
outside
ASA
inside
server
client
outside
ASA
server
34
PACKET CAPTURE
4
35
Packet Capture
Inside Capture
Outside Capture
In-line capability to record packets passing through ASA

Inside
Outside
Capture OUT
Two key steps in troubleshooting with captures

Capture IN
Apply capture under unique name to ingress and egress interfaces
Define the traffic that you want to capture, use pre-NAT on the wire information
Tcpdump-like format for displaying captured packets on the box
asa# capture OUT interface outside match ip any host 172.18.124.1
asa# capture IN interface inside match ip any host 172.18.124.1
asa# show capture IN
Unlike ACL, match covers

both directions of the flow
4 packets captured
1: 10:51:26.139046
2: 10:51:26.139503
3: 10:51:27.140739
4: 10:51:27.141182
4 packets shown
asa# no capture IN
802.1Q
802.1Q
802.1Q
802.1Q
vlan#10
vlan#10
vlan#10
vlan#10
P0
P0
P0
P0
172.18.254.46 > 172.18.124.1:

172.18.124.1 > 172.18.254.46:
172.18.254.46 > 172.18.124.1:
172.18.124.1 > 172.18.254.46:
icmp:
icmp:
icmp:
icmp:
echo
echo
echo
echo
request
reply
request
reply
Remember to remove the captures

when done with troubleshooting
36
Packet Capture
Capture buffer maintained in RAM (512KB by default, 30 MB max)
Stops capturing when full by default, circular option available

Default recorded packet length is 1518 bytes
May elevate CPU utilization on multiple-core ASA when applied
Copy captures off via TFTP or retrieve through HTTPS with your web browser
Do this before removing the capture with no capture
https://x.x.x.x/admin/capture/OUT/pcap/outsidecapture.pcap
Configured capture name
Save capture file under this name
Download binary PCAP to

open in your favorite packet
analyser (such as Wireshark)
37
Where Packets Are Captured in Packet Flow

IPS or CX
Module
Yes
RX
Pkt
Ingress
Interface
Existing
Conn
No
NAT
Untranslate
ACL
Permit
No
DROP
Ingress Packets
Captured
Yes
Yes
Stateful
Inspection
NAT IP
Header
Egress
Interface
No
DROP
L3
Route
No
No
DROP
DROP
Yes
L2
Addr
Yes
TX
Pkt
No
DROP
Egress Packets
Captured
Packets are captured at the first and last points they can be in the flow
Ingress packets are captured before most packet processing
Egress packets are captured after all processing
Transit packets show the destination MAC address rewritten
Self-sourced packets may show an empty MAC address (0000.0000.0000)
38
Accelerated Security Path (ASP)

Packets and flows dropped in the ASP will increment a counter
Frame drop counters are per packet
Flow drops are per flow

See command reference under show asp drop for full list of counters
asa# show asp drop
Frame drop:
Invalid encapsulation (invalid-encap)
Invalid tcp length (invalid-tcp-hdr-length)
Invalid udp length (invalid-udp-length)
No valid adjacency (no-adjacency)
No route to host (no-route)
Reverse-path verify failed (rpf-violated)
Flow is denied by access rule (acl-drop)
First TCP packet not SYN (tcp-not-syn)
10942
Bad TCP Checksum (bad-tcp-cksum)
10897
9382
10
5594
1009
15
25247101
36888
893
39
Capturing ASP Drops

Capture all frames dropped in the ASP
asa# capture drops type asp-drop all
Capture all frames with a specific drop reason

asa# capture drop type asp-drop ?
acl-drop
rule
all
bad-crypto
bad-ipsec-natt
bad-ipsec-prot
bad-ipsec-udp
bad-tcp-cksum
bad-tcp-flags
Flow is denied by configured

All packet drop reasons
Bad crypto return in packet
Bad IPSEC NATT packet
IPSEC not AH or ESP
Bad IPSEC UDP packet
Bad TCP checksum
Bad TCP flags
ASP flow drops are non-atomic and cannot be captured

asa# capture drops type asp-drop tcp-not-syn
40
PACKET TRACER
5
41
Packet Tracer
Unique capability to record the path of a specially tagged packet through ASA
Best way to understand the packet path in the specific software version
Inject a simulated packet to analyse the behaviour and validate configuration
Feature order
and name
asa# packet-tracer input inside tcp 192.168.1.101 23121 172.16.171.125 23 detailed

Phase: 1
Type: CAPTURE
Ingress interface
Subtype:
Result: ALLOW
Config:
Additional Information:
[]
Packet information as it
enters the ingress interface
Include detailed internal flow and
policy structure information
42
Sample Packet Tracer Output

asa# packet-tracer input outside tcp 172.18.124.66 1234 172.18.254.139 3389
Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
MAC Access list
Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
MAC Access list
2014
Phase: 3
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (outside,dmz) source dynamic any interface destination static interface Win7-vm service rdp-outside rdp-outside
NAT divert to egress interface dmz
Untranslate 172.18.254.139/3389 to 192.168.103.221/3389
Cisco and/or its affiliates. All.
rights reserved.
43
Sample Packet Tracer Output

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_in in interface outside
access-list outside_in extended permit tcp any any eq 3389
Phase: 8
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (outside,dmz) source dynamic any interface destination static interface Win7-vm service rdp-outside rdp-outside
Dynamic translate 172.18.124.66/1234 to 192.168.103.221/1234
Phase: 12
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
New flow created with id 16538274, packet dispatched to next module
44
Packet Tracer in ASDM

Launch from Tools >
Packet Tracer
Define simulated packet
Feature type and

resulting action
Direct link to edit policy
Associated
configuration
Final outcome (allowed or

dropped) and egress
interface information
45
Packet Tracer: Tracing Captured Packet

Enable packet tracer within an internal packet capture
asa# capture IN interface inside trace trace-count 20 match tcp any any eq
Trace inbound
packets only
Traced packet count per

capture (50 by default)
Find the packet that you want to trace in the capture

asa#
68
1:
2:
3:
4:
5:
show capture inside

packets captured
15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80:
15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746:
15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80:
15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80:
15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746:
...
S
S
.
P
.
ack
ack
ack
Ack
Select that packet to show the tracer results

asa# show capture inside trace packet-number 4
46
Packet Tracer Video
Embedded Event Manager

Troubleshooting tool added in 9.2(1), similar to IOS EEM
Powerful way to run CLI commands based on ASA events (syslogs) and save
the output
Trigger Event
Action
Syslog ID
Execute show commands
Console
Time based
Execute config commands
File on disk
Crash
Output Destination
None
Manual
48
48

Time-based events
Every midnight back up the ASA configuration to your tftp server
Every 3 hours gather the output of show memory detail and save it to the flash
Syslog based events

If the available 1550 byte blocks become depleted, gather show blocks pool 1550
dump and save to the disk
If the AAA server is marked down: ping tcp to the server on port 49, show aaa-server
to gather statistics, save to a file on disk, use SCH to email the file contents
Manual events
Gather the output of 10 different commands and save to a file
49
49

Goal: Backup the configuration when a user logs in, and again when they log off of a SSH
session
Determine the syslogs that should trigger the event
%ASA-6-605005: Login permitted from 14.36.103.220/54785 to 36net:14.36.103.88/ssh for user "cisco"
%ASA-5-611103: User logged out: Uname: cisco
Configure the event applet

event manager applet loginConfigBackup
event syslog id 605005
event syslog id 611103
action 1 cli command "show running-config"
output file rotate 50
!
Applet name
Trigger syslogs
Action Command
Output Destination
Files written to disk when a user logs in and then out

261
260
259
-rwx
-rwx
-rwx
161286
161331
161277
16:46:27 May 05 2014

16:46:14 May 05 2014
16:46:07 May 05 2014
eem-loginConfigBackup-0.log
50
50
VPN AUTHENTICATION DEBUGS
1
*NOT
RANKED BY IMPORTANCE.
52
Authentication Problems
debug webvpn <1-255>
Good Authentication
WebVPN: calling AAA with ewsContext (-925550560) and nh (927982512)!

WebVPN: started user authentication...
WebVPN: AAA status = (ACCEPT)
WebVPN: user: (user1) authenticated.
Bad Authentication
WebVPN: started user authentication...

webvpn_free_auth_struct: net_handle = 0xc839fc30
webvpn_allocate_auth_struct: net_handle = 0xc839fc30
webvpn_free_auth_struct: net_handle = 0xc839fc30
webvpn_auth.c:webvpn_aaa_callback[5107]
WebVPN: AAA status = (ERROR)
WebVPN: callback data is not valid!!
webvpn_remove_auth_handle: auth_handle = 5
53
53
RADIUS Authentication Problems

debug radius
RADIUS packet decode (authentication request)
-------------------------------------Raw packet data (length = 150).....
01 11 00 96 53 90 89 8e af bc 45 9a cb a8 c1 66
a7 54 fd f2 01 07 75 73 65 72 31 02 12 07 6f 5c
c4 03 ae cf cc bf df ec 1d 58 0f 31 38 05 06 00
00 70 00 1e 11 32 30 39 2e 31 36 35 2e 32 30 30
2e 32 32 35 1f 11 32 30 39 2e 31 36 35 2e 32 30
30 2e 32 32 36 3d 06 00 00 00 05 42 11 32 30 39
2e 31 36 35 2e 32 30 30 2e 32 32 36 04 06 0a 0a
0a fe 1a 24 00 00 00 09 01 1e 69 70 3a 73 6f 75
72 63 65 2d 69 70 3d 32 30 39 2e 31 36 35 2e 32
30 30 2e 32 32 36
Parsed packet data.....
Radius: Code = 1 (0x01)
Radius: Identifier = 17 (0x11)
Radius: Length = 150 (0x0096)
Radius: Vector: 5390898EAFBC459ACBA8C166A754FDF2
Radius: Type = 1 (0x01) User-Name
Radius: Value (String) =
75 73 65 72 31
Radius: Type = 2 (0x02) User-Password
send pkt 172.18.104.83/1645
RADIUS_SENT:server response timeout
RADIUS
RADIUS_DELETE
remove_req 0xcbeb5d00 session 0x14 id 17
|
|
|
|
|
|
|
|
|
|
....S.....E....f
.T....user1...o\
.........X.18...
.p...209.165.200
.225..209.165.20
0.226=.....B.209
.165.200.226....
...$......ip:sou
rce-ip=209.165.2
00.226
user1
Server not Responding

54
54
Domain Authentication Problem

debug ntdomain
Domain Controller Communication Problem
smb: negotiate phase failed: syserr = Network is down

Cifs_Connect_Server() returned FALSE, error_code = 18
ntdomain_process_ntinfo - state is NTDOMAIN_DELETE
INFO: Attempting Authentication test to IP address <172.18.85.123>
(timeout: 12 seconds)
ERROR: Authentication Server not responding: No error
Note: In this Example the Administrator Attempts to Authenticate to the Active Directory Server Using the TEST Utility Within ASDM
55
55
Additional Authentication Debugs

For Your
Reference
You can combine the debugs listed above with the debug webvpn
and debug aaa common when troubleshooting clientless
authentication problems.
56
56
Authentication Test Utility
Using the CLI:

test
aaa-server authentication NYGroup host 172.18.85.123 user domainuser password 123qweasd
57
57
58
Useful Show Commands

show vpn-sessiondb
asa# show vpn-sessiondb
--------------------------------------------------------------------------VPN Session Summary
--------------------------------------------------------------------------Active : Cumulative : Peak Concur : Inactive
---------------------------------------------AnyConnect Client
:
12 :
22 :
12 :
0
SSL/TLS/DTLS
:
12 :
22 :
12 :
0
--------------------------------------------------------------------------Total Active and Inactive
:
12
Total Cumulative :
22
Device Total VPN Capacity
:
25
Device Load
:
0%
----------------------------------------------------------------------------------------------------------------------------------------------------Tunnels Summary
--------------------------------------------------------------------------Active : Cumulative : Peak Concurrent
---------------------------------------------AnyConnect-Parent
:
12 :
22 :
12
SSL-Tunnel
:
12 :
22 :
12
DTLS-Tunnel
:
12 :
22 :
12
--------------------------------------------------------------------------Totals
:
12 :
6
59
show vpn-sessiondb additional options

asa# show vpn-sessiondb ?
exec mode commands/options:
anyconnect
detail
email-proxy
full
index
l2l
license-summary
ra-ikev1-ipsec
ratio
summary
vpn-lb
webvpn
|
<cr>
AnyConnect sessions
Show detailed output
Email-Proxy sessions
Output formatted for data management programs
Index of session
IPsec LAN-to-LAN sessions
Show VPN License summary
IKEv1 IPsec/L2TP-IPsec Remote Access sessions
Show VPN Session protocol or encryption ratios
Show VPN Session summary
VPN Load Balancing Mgmt sessions
WebVPN sessions
Output modifiers
60
debug webvpn
omar-asa# debug webvpn ?
<1-255>
anyconnect
webvpn anyconnect debugging
chunk
cifs
webvpn chunk debugging

webvpn cifs debugging
citrix
compression
webvpn citrix debugging

webvpn (anyconnect) compression debugging
cstp-auth
customization
webvpn cstp-auth debugging

webvpn customization debugging
failover
html
webvpn failover debugging

webvpn html debugging
javascript
kcd
webvpn javascript debugging

webvpn kcd debugging
listener
mus
webvpn listener debugging

webvpn MUS debugging
nfs
request
webvpn nfs debugging

webvpn request debugging
response
session
webvpn response debugging

webvpn session debugging
transformation
url
webvpn transformation debugging

webvpn url debugging
util
xml
webvpn util debugging

webvpn xml debugging
<cr>
61
SSL VPN TROUBLESHOOTING VIDEOS
63
DEBUG DAP TRACE

ASA(config)# debug dap trace
The DAP policy contains the following attributes:
------------------------------------------------1: action = continue
DAP_open: C9EEE930
DAP_add_CSD: csd_token = [4287F77A4F7347A553F4619C]
[ 0]: aaa.cisco.username = user2
[ 1]: aaa.cisco.tunnelgroup = DefaultWEBVPNGroup
dap_add_to_lua_tree:aaa["cisco"]["username"] = "user2";
dap_add_to_lua_tree:aaa["cisco"]["tunnelgroup"] = "DefaultWEBVPNGroup";
dap_clienttype_to_string(3) returns CLIENTLESS
dap_add_to_lua_tree:endpoint["application"]["clienttype"] = "CLIENTLESS";
dap_add_csd_data_to_lua:
endpoint.os.version = "Windows XP";
endpoint.os.servicepack = "2";
endpoint.location = "Default";
endpoint.protection = "secure desktop";
endpoint.fw["MSWindowsFW"] = {};
endpoint.fw["MSWindowsFW"].exists = "true;
64
64
Continuation of the debug dap

trace output
DEBUG DAP TRACE
endpoint.fw["MSWindowsFW"].description = "Microsoft Windows Firewall";

endpoint.fw["MSWindowsFW"].enabled = "true";
endpoint.av["McAfeeAV"] = {};
endpoint.av["McAfeeAV"].exists = "true";
endpoint.av["McAfeeAV"].description = "McAfee VirusScan Enterprise";
endpoint.av["McAfeeAV"].version = "7.0.0";
endpoint.av["McAfeeAV"].activescan = "true";
endpoint.av["McAfeeAV"].lastupdate = "132895";
endpoint.as["SpyBot"] = {};
endpoint.as["SpyBot"].exists = "true";
endpoint.as["SpyBot"].description = "Spybot - Search & Destroy 1.4";
endpoint.as["SpyBot"].version = "1.4";
endpoint.as["SpyBot"].activescan = "false";
endpoint.as["SpyBot"].lastupdate = "996895";
endpoint.enforce = "success";
Selected DAPs: McAfee-7,SpyBot
dap_request: memory usage = 19%

dap_process_selected_daps: selected 3 records
dap_aggregate_attr: rec_count = 3
DAP_close: C9EEE930
65
65
66
IPSec Debugs and Show Commands

omar-asa# debug crypto ?
ca
Set PKI debug levels
condition
Set IPSec/ISAKMP debug filters
engine
Set crypto engine debug levels
ike-common
Set IKE common debug levels
ikev1
Set IKEV1 debug levels
ikev2
Set IKEV2 debug levels
ipsec
Set IPSec debug levels
ss-api
Set Crypto Secure Socket API debug levels
vpnclient
Set EasyVPN client debug levels
67
IKEv2 Debug Commands

debugs specific for IKEv2
debug crypto ikev2 platform

Debugs ASA processing of IKEv2, not protocol specific exchanges.
This debug is useful for AAA and session management issues. Also to troubleshoot
the ASA cryptographic module performing encryption and decryption.
debug crypto ikev2 protocol
Debugs IKEv2 protocol specific exchanges.
debug crypto ikev2 timer
Debugs IKEv2 timer expiration. Useful when clients are complaining that their
connection is being timed-out too often.
Note: debug crypto ike-common can be used for both IKEv1 and IKEv2
68
68
show crypto ipsec sa

ciscoasa(config)# show crypto ipsec sa
interface: outside
Crypto map tag: def, local addr: 10.132.0.17
local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)

remote ident (addr/mask/prot/port): (172.20.0.21/255.255.255.255/0/0)
current_peer: 172.20.0.21
dynamic allocated peer ip: 10.135.1.5
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

#pkts decaps: 1145, #pkts decrypt: 1145, #pkts verify: 1145
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 2, #pre-frag failures: 1, #fragments created: 10
#PMTUs sent: 5, #PMTUs rcvd: 2, #decapstulated frags needing reassembly: 1
#send errors: 0, #recv errors: 0
69
show crypto ipsec sa (cont.)

local crypto endpt.: 10.132.0.17, remote crypto endpt.: 172.20.0.21
path mtu 1500, ipsec overhead 60, media mtu 1500
current outbound spi: DC15BF68
inbound esp sas:
spi: 0x1E8246FC (511854332)
transform: esp-3des esp-md5-hmac
in use settings ={L2L, Transport, Manual key, (OSPFv3), }
slot: 0, conn_id: 3, crypto-map: def
sa timing: remaining key lifetime (sec): 548
IV size: 8 bytes
replay detection support: Y
outbound esp sas:
spi: 0xDC15BF68 (3692412776)

70
show crypto ipsec sa (cont.)

in use settings ={L2L, Transport, Manual key, (OSPFv3), }
slot: 0, conn_id: 3, crypto-map: def
sa timing: remaining key lifetime (sec): 548
IV size: 8 bytes
replay detection support: Y
Crypto map tag: def, local addr: 10.132.0.17
local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
71
72
AnyConnect Diagnostics and Reporting Tool

useful for troubleshooting AnyConnect installation and connection problems
To Launch DART go to
the Status Overview
Tab and click on
Diagnostics
1
73
73
DART Wizard
Under Bundle Creation Option, select Default or Custom. The Default option includes the typical log files and
diagnostic information. DARTBundle.zip is saved to the local desktop. If you choose Custom, the DART wizard
allows you to specify where and what files want to include in the bundle.
74
74
DART Wizard
continued
75
75
DART Bundled Files

Advanced Detailed Logs for each Installed Module in AnyConnect
DART BUNDLE SUMMARY
Username:
Time:
unknown (user is offline, or username was not specified in Request)

Tue Apr 05 17:12:17 2011
OS:
OS username:
Upload URL:
DART Mode:
Bundle on client computer:
Win7 : WinNT 6.1.7600

omar
None (offline mode)
User-Initiated/Offline Mode
C:\Users\omar\Desktop\DARTBundle_0405_1353.zip
=============================================================================================================================================
Cisco AnyConnect Secure Mobility Client:
Files Included in Bundle:
ID
Filename
Description
Truncate? Final Size Orig. Size
---------------------------------------------------------------------------------------------------------------------------ac-install
update_pre3.0.txt
AnyConnect install logs. Includes web
No
10 bytes
10 bytes
and standalone install logs
ac-install
anyconnect-win-2.3.0254-web
No
322.35K
322.35K
-deploy-k9-install-22203701
062010.log
ac-install
update.txt
No
10 bytes
10 bytes
ac-install
VPNManifest.dat
No
181 bytes
181 bytes
ac-install
AnyConnectLocalPolicy.xml
No
589 bytes
589 bytes
ac-install
UpdateHistory_20110405_1244
No
705 bytes
705 bytes
00_log.txt
ac-logs
AnyConnect_pre3.0.txt
AnyConnect application logs
No
3.62M
3.62M
ac-logs
AnyConnect.txt
No
227.40K
227.40K
ac-logs
AnyConnect.evtx
No
1.06M
1.06M
ac-profile
CALO.xml
AnyConnect Profile
No
1.46K
1.46K
ac-profile
AnyConnectProfile.xsd
AnyConnect Profile
No
93.22K
93.22K
global-preferenc
preferences_global.xml
AnyConnect Global Preferences
No
546 bytes
546 bytes
es
user-preferences
preferences.xml
AnyConnect User Preferences
No
590 bytes
590 bytes
va-runtime
setupapi.app.log
Virtual Adapter runtime logs
No
320.88K
320.88K
va-runtime
setupapi.dev.log
Virtual Adapter runtime logs
No
9.70M
9.70M
----------------------------------------------------------------------------------------------------------------------------
MANY, MANY, MANY, MANY more

76
76
ANYCONNECT STATISTICS VIDEO
Thank you.

CP-1005 - Top 10 ASA Firewall and VPN Troubleshooting Techniques Commands - 1 Hour PDF

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

CP-1005 - Top 10 ASA Firewall and VPN Troubleshooting Techniques Commands - 1 Hour PDF

Caricato da

Copyright:

Formati disponibili

Top 10 ASA Firewall and VPN

Introduction of ASA Packet Flow

Top 5 Techniques when Troubleshooting

Top 5 Techniques when Troubleshooting

2014 Cisco and/or its affiliates. All rights reserved.

2013-2014 Cisco and/or its affiliates. All rights reserved.

Understanding Packet Flow

through the network.

192.168.1.150:50141, idle 0:00:00, bytes 0, flags saA

All firewall connectivity issues can be simplified to two

2014 Cisco and/or its affiliates. All rights reserved.

With the Flow defined,

2014 Cisco and/or its affiliates. All rights reserved.

Packet Processing: Ingress Interface

Packet arrives on ingress interface

Input counters incremented by NIC and periodically retrieved by CPU

Software input queue (RX ring) is an indicator of packet load

2014 Cisco and/or its affiliates. All rights reserved.

asa# show interface outside

Packet Processing: Locate Connection

Check first for existing connection in conn table

TCP non-SYN packet, drop and log

2014 Cisco and/or its affiliates. All rights reserved.

Packet Processing: NAT Un-Translate

Incoming packet is checked against NAT rules

2014 Cisco and/or its affiliates. All rights reserved.

Packet Processing: ACL Check

First packet in flow is processed through ACL checks

Denied packets are dropped and logged

2014 Cisco and/or its affiliates. All rights reserved.

Packet Processing: Stateful Inspection

Stateful inspection ensures protocol compliance at TCP/UDP/ICMP level

Rewrite embedded IP addresses, open up ACL pinholes for secondary connections

Additional security checks are applied to the application payload

2014 Cisco and/or its affiliates. All rights reserved.

Packet Processing: NAT IP Header

Translate the source and destination IP addresses in the IP header

(Optional) Following the above, pass packet to IPS or CX module

Real (pre-NAT) IP address information is supplied as meta data

2014 Cisco and/or its affiliates. All rights reserved.

Packet Processing: Egress Interface

the routing table

determine egress interface

2014 Cisco and/or its affiliates. All rights reserved.

Packets received on outside and destined to

Packet Processing: L3 Route Lookup

Once at egress interface, an interface route lookup is performed

routing table may point to a different interface

2014 Cisco and/or its affiliates. All rights reserved.

Packet Processing: L2 Address Lookup

2014 Cisco and/or its affiliates. All rights reserved.

Packet Processing: Transmit Packet

Packet is transmitted on wire

Underrun counter indicates drops due to egress interface oversubscription

2014 Cisco and/or its affiliates. All rights reserved.

asa# show interface outside

273399 packets output, 115316725 bytes, 80 underruns

input queue (blocks free curr/low): hardware (485/441)

*NOT RANKED BY IMPORTANCE

2014 Cisco and/or its affiliates. All rights reserved.

Live Debugging Purposes

2014 Cisco and/or its affiliates. All rights reserved.

ASA Syslog Level vs. Number of Messages