WAAS TShoot Guide

Cisco WAAS Troubleshooting Guide for Release 4.1.
3 and Later
Welcome to Cisco DocWiki. We encourage registered Cisco.com users to contribute to this wiki to collaborate on Cisco product documentation. You do not
need to log in to read the text. However, you must log in to edit the text. Select the "edit" tab to edit an article or select the "discussion" tab to submit questions
or comments about documentation content.
See Terms of Use and About DocWiki for more information about Cisco DocWiki.
Click here to return to the Cisco WAAS documentation on www.cisco.com.
Contents
1 Audience
2 Organization
3 Creating a PDF of the WAAS
Troubleshooting Wiki
4 Related Documentation
This guide provides a systematic approach to identifying and remedying problems that may arise as you use your WAAS system
over a period of time. This guide is not intended to replace configuration best practices or to be an all-inclusive guide for every
application. Rather, it is an attempt to provide you with the knowledge and skills necessary to correct the most common issues that
you may encounter.
This guide applies to WAAS version 4.1.3 and later releases.
Audience
This article is intended for trained network administrators who have experience with the configuration and maintenance of the
WAAS system.
Organization
This article consists of the following major sections:
Understanding the WAAS Architecture and Traffic Flow
Preliminary WAAS Troubleshooting
Troubleshooting Optimization
Troubleshooting Application Acceleration
Troubleshooting the CIFS AO
Troubleshooting the HTTP AO
Troubleshooting the EPM AO
Troubleshooting the MAPI AO
Troubleshooting the NFS AO
Contents
Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Troubleshooting the SSL AO
Troubleshooting the Video AO
Troubleshooting the Generic AO
Troubleshooting Overload Conditions
Troubleshooting WCCP
Troubleshooting AppNav
Troubleshooting Disk and Hardware Problems
Troubleshooting Serial Inline Clusters
Troubleshooting vWAAS
Troubleshooting WAAS Express
Troubleshooting NAM Integration
Creating a PDF of the WAAS Troubleshooting Wiki

Create a PDF of the Cisco WAAS Troubleshooting Guide that you can save on your computer and print.
Related Documentation
Release Notes for Cisco Wide Area Application Services
Cisco Wide Area Application Services Quick Configuration Guide
Cisco Wide Area Application Services Configuration Guide
Cisco Wide Area Application Services vWAAS Installation and Configuration Guide
Cisco Wide Area Application Services Command Reference
Cisco Wide Area Application Services Upgrade Guide
Cisco Wide Area Application Services API Reference
Cisco Wide Area Application Services Monitoring Guide
Using the Print Utilities to Troubleshoot and Fix Samba Driver Installation Problems
Cisco WAAS Installation and Configuration Guide for Windows on a Virtual Blade
Cisco WAAS Installation and Configuration Guide for ACNS on a Virtual Blade
Cisco WAAS on Service Modules for Cisco Access Routers
Cisco SRE Service Module Configuration and Installation Guide
Configuring Cisco WAAS Network Modules for Cisco Access Routers
This article describes the WAAS architecture and how data flows into, gets processed, and flows out of a WAAS device. It
provides a basic understanding of these concepts to assist you in troubleshooting the WAAS system.
Guide Contents
Main Article
Organization

Contents
1 Understanding the WAAS
Architecture
1.1 AOs
1.2 WoW and Virtual
Blades
1.3 Configuration
Management System
1.4 DRE with Scheduler
1.5 Storage
1.6 Network I/O
1.7 Interception & Flow
Management
1.7.1
Auto-Discovery
1.7.2 Policy
Engine
1.7.3
Filter-Bypass
2 WAAS Traffic Flow
Related Documentation
Understanding the WAAS Architecture

Having a basic understanding of the Wide Area Applications Services (WAAS) architecture and data flow can help to make
troubleshooting the WAAS system easier. This section describes the major functional areas of the WAAS system and how they
work together.
The WAAS system architecture is divided into a series of functional areas or services as shown in Figure 1.
Figure 1. WAAS Architecture
AOs
AOs (Application Optimizers, also known as Application Accelerators) are the application-specific software that optimizes certain
protocols at Layer 7 (beyond the generic Layer 4 optimizations). AOs may be viewed as the "applications" in the WAE system (in
an OS analogy). The generic AO acts as a catch-all for all traffic that has no protocol-specific AO and also functions as a delegate
if a protocol-specific AO decides to not apply optimization.
WoW and Virtual Blades

Windows server on WAAS (WoW) is Microsoft Windows Server running in a virtual blade. The virtualization feature of WAAS
allows you to configure one or more virtual blades, which are computer emulators that reside in a WAE or WAVE device. A
virtual blade allows you to allocate WAE system resources for use by additional operating systems that you install on the WAE
hardware. You can host third-party applications in the isolated environment provided by a virtual blade. For example, you could
configure a virtual blade in a WAE device to run Windows printing and domain lookup services.
Configuration Management System

The configuration management system (CMS) consists of the WAAS Central Manager and its database for storing WAAS device
configuration information. The CMS allows you to configure and manage WAE devices and device groups from a single Central
Manager GUI interface.
DRE with Scheduler

The DRE with scheduler (SO-DRE) is the key module in the Layer 4 optimization space, and is responsible for all data-reduction
techniques in the system, including data redundancy elimination (DRE) and persistent LZ compression. In addition to the
system-wide algorithms for data reduction that are implemented here, this component also includes a scheduling element that
allows the system to better control the order and pace of using DRE for different AOs.
Storage
The storage system manages the system disks and the logical RAID volumes on systems that have multiple disks. Disk storage is
used for system software, the DRE cache, the CIFS cache, and virtual blade storage.
Network I/O
The network input/output component is responsible for all aspects that are related to handling data communication coming into or
going out from a WAE, including WAE to WAE communication and WAE to client/server communication.
Interception & Flow Management

Interception and flow management consists of multiple submodules that, using policies configured by the user, intercept traffic,
automatically discover peers, and start optimization on a TCP connection. Some of the key submodules are Auto-Discovery,
Policy Engine, and Filter Bypass.
Auto-Discovery
Auto-discovery allows peer devices to discover each other dynamically and does not require that you preconfigure WAE pairs.
Auto-discovery is a multi-WAE end-to-end mechanism that defines a protocol between the WAEs that discovers a pair of peer
WAEs for a given connection.
WAE devices automatically discover one another during the TCP three-way-handshake that occurs when two nodes establish a
TCP connection. This discovery is accomplished by adding a small amount of data to the TCP options field (0x21) in the SYN,
SYN/ACK, and ACK messages. This TCP option allows WAE devices to understand which WAE is on the other end of the link
and allows the two to describe which optimization policies they would like to employ for the flow. If intermediate WAEs exist in
the network path, they simply pass through flows that are being optimized by other WAEs. At the end of the auto-discovery
process, the WAEs shift the sequence numbers in the TCP packets between the participating WAEs by increasing them to more
than 2 billion, to mark the optimized segment of the connection.
Policy Engine
The policy engine module determines if traffic needs to be optimized, which AO to direct it to, and the level of data-reduction
(DRE) if any, that should be applied to it. The policy engine classifies traffic beyond connection establishment (for example,
based on payload information) and changes the flow of a connection dynamically from unoptimized to optimized.
The elements of a policy include the following:
Application definition: A logical grouping of traffic to help report statistics on the type of traffic.
Traffic classifier: Access control list (ACL) that helps choose connections based on IP addresses, ports, and so on.
Policy Map: Binds the application and the classifier with an action, which specifies the type of optimization, if any, to be
applied. There are two kinds of policy maps:
Static policy map: Configured on the device through the CLI or GUI (or installed by default) and is persistent
unless removed.
Dynamic policy map: Automatically configured by the WAE and has a life span just long enough to accept a new
DRE with Scheduler

connection.
The following configuration example shows a policy engine application definition (Web) that includes a classifier (HTTP) and an
action (optimize full accelerate http):
wae(config)# policy-engine application map basic
wae(config-app-bsc)# name Web classifier HTTP action optimize full accelerate http set-dscp copy
Filter-Bypass
After interception, the filter-bypass module acts as the mediator between the policy engine and auto discovery. The filter-bypass
module tracks all optimized connections in a filtering table for the life of the connection. Additionally, it tracks pass-through
connections, but pass-through table entries are timed out after 3 seconds.
WAAS Traffic Flow

This section describes the packet flow in WAAS.
Figure 2 shows the filter-bypass flow establishment as a packet enters the system.
Figure 2. Filter-Bypass Flow Establishment
1. A SYN packet on a flow enters the system. This packet is routed to the filter-bypass module.
2. The filter-bypass module consults the policy engine on how the flow should be handled.
2a. The policy engine consults the configured and dynamically added policies, and based on the current operational status of the
AOs and SO-DRE, decides what the WAE could do for this flow: pass through, locally terminate, or optimize.
2b. The packet and the decision from the policy engine are then returned to the filter-bypass module.
3. The filter-bypass module acts on the policy engine decision in one of the following ways:
3a. Sends the packet out immediately (pass through).
3b. Sends the packet up for local termination by an AO.
3c. Sends the packet to the auto-discovery module for optimization.
If the filter-bypass module chooses option 3c, the packet is sent to the auto-discovery module. The auto-discovery module
Policy Engine

determines what optimizations can be done, based on the availability of a peer WAE and its enabled features. A peer WAE is
discovered through the use of TCP options added during the TCP handshake to the remote node. If the auto-discovery module
determines that a peer WAE is available, the connection is handed off for further processing once the TCP three-way-handshake
completes. If a peer WAE is discovered for the first time, the WAEs additionally negotiate about AO versions and capabilities.
This information is used to decide the AO level capabilities for the connection.
4. The connection is finally admitted into the system with specific L4 and L7 optimizations and handed off to the appropriate L4
(DRE) and L7 (AO) acceleration modules. For connections that are later discovered to be not optimize-able by protocol-specific
AOs (HTTP, MAPI, and so on), the connection is handled by the generic AO, with or without DRE optimization (as negotiated
during connection establishment).
This article introduces the basic concepts, methodology, and general troubleshooting guidelines for problems that may occur when
you configure and use your WAAS system.
Guide Contents
Main Article
Contents
1 Overview of the WAAS Troubleshooting Process
2 Verifying the WAAS Image
3 Enabling WAAS Logging
4 Running Diagnostics
5 Verifying the Physical Connectivity Between Peer WAAS Devices and to Application
Servers
6 Checking CPU Load
7 Gathering WAAS Troubleshooting Information
7.1 Rebooting the WAAS Device
7.2 Using show Commands
WAAS Traffic Flow

7.3 Generating a System Report
7.4 Capturing and Analyzing Packets
7.4.1 Using tcpdump
7.4.2 Using tethereal
8 Contacting Cisco Technical Support
Overview of the WAAS Troubleshooting Process

To troubleshoot your WAAS system, follow these general guidelines:
1. Maintain a consistent and recommended software version across all your WAAS devices. If versions must differ, the
Central Manager must be running the highest version. See the "Verifying the WAAS Image" section to determine the
version in use.
2. See the WAAS release notes for your software version for the latest features, operating considerations, caveats, and CLI
command changes.
3. Before you introduce configuration changes on the WAAS Central Manager, use the CMS backup feature to save your
configuration. If you run into problems with the new configuration, you can restore the previous configuration. See the
Backing Up and Restoring your WAAS System section in the Cisco Wide Area Application Services Configuration Guide.
Troubleshoot any problems with new configuration changes immediately after making them.
4. Verify that your configuration is correct for your network application. Make any required changes to the running-config
file, and then test the configuration. If it is satisfactory, save it to the startup-config file using the copy running-config
startup-config command.
5. Enable system message logging. See the "Enabling WAAS Logging" section.
6. Run the diagnostic tool to verify device functionality and connectivity. See the "Running Diagnostics" section.
7. Verify the physical connectivity between WAAS peers and to the application servers. See the "Verifying the Physical
Connectivity Between Peer WAAS Devices and to Application Servers" section.
8. Gather information that defines the specific symptoms. See the "Gathering WAAS Troubleshooting Information" section.
9. Refer to one of the other articles in this WAAS Troubleshooting Guide for information on troubleshooting specific
problems:
If the system appears to be having hardware or disk problems, see the article Troubleshooting Disk and Hardware
Problems.
If the system is having trouble receiving traffic, see the article Troubleshooting WCCP. This problem also could
be due to a firewall issue.
If the system is passing through traffic instead of optimizing it or is having problems optimizing specific kinds of
application traffic (HTTP, MAPI, SSL, and so on), see the articles Troubleshooting Optimization and
Troubleshooting Application Acceleration.
If the system is passing through more traffic than expected instead of optimizing it, see the article
Troubleshooting Overload Conditions.
10. After you have determined that your troubleshooting attempts have not resolved the problem, contact the Cisco Technical
Assistance Center (TAC) or your technical support representative. See the "Contacting Cisco Technical Support" section.
Contents
Verifying the WAAS Image

To display the version of the software image that is currently running in your WAAS device, enter the following command:
wae# show version
Cisco Wide Area Application Services Software (WAAS)
Copyright (c) 1999-2009 by Cisco Systems, Inc.
Cisco Wide Area Application Services Software Release 4.1.3a (build b25 May 23 2
009)
Version: oe7341-4.1.3a.25
<--------
Compiled 10:10:47 May 23 2009 by cnbuild

System was restarted on Wed May 27 14:45:28 2009.
The system has been up for 6 weeks, 2 hours, 35 minutes, 48 seconds.
This command provides other useful information, for example:

Device model (the numbers in the first part of the Version string encode the device model number; a WAE-7341 is shown
here.)
WAE uptime
To verify that there is no pending software upgrade (waiting for a device reboot), enter the following command:
wae# show version pending
No pending version
You should see the message "No pending version".
Enabling WAAS Logging

General system error logging to the disk file /local1/syslog.txt is enabled by default. You can check that logging is enabled by
entering the following command:
wae# show logging
Syslog to host is disabled.
Syslog to console is disabled
Priority for console logging is set to:
Syslog to disk is enabled
Priority for disk logging is set to:
Filename for disk logging is set to:
warning
<------------
notice
/local1/syslog.txt
Syslog facility is set to *

Syslog disk file recycle size is set to 10000000
To enable logging to the console, enter the following global configuration command:
wae(config)# logging console enable
NOTE: Setting the logging priority to a level lower than notice can be CPU intensive and can generate a large amount of output.
Use it judiciously and sparingly in a production environment.
The following directories are used by WAAS for log files:
Verifying the WAAS Image

/local1 ? Root directory for all log files and location of syslog.txt
/local1/logs ? Service log files (admin and transaction logs)
/local1/errorlog ? Service log files (debug logs)
/local1/errorlog/cifs ? CIFS internal log files
/local1/core_dir ? Process core dump files
You can use the following file system navigation commands to navigate and view the log files:
cd
pwd
dir
type-tail filename lines [| | follow]
find-pattern
Running Diagnostics
The WAAS Central Manager includes a built-in diagnostic tool that can help you troubleshoot many device problems, including
the following:
Network configuration
Interface configuration
Connectivity to hosts
WCCP configuration
Inline configuration
TFO configuration
WAFS configuration
We recommend that you run the diagnostic tool first before taking other troubleshooting actions. The tool reports on the status and
configuration of many system functions.
To run the diagnostic tool from the Central Manager, follow these steps:
1. From the WAAS Central Manager GUI navigation pane, choose My WAN > Manage Devices (or Manage Device
Groups).
2. Click the Edit icon next to the name of the device (or device group) for which you want to perform diagnostic tests.
3. In the navigation pane, choose Troubleshoot > Diagnostics Tests. The Diagnostic Tool window appears.
4. Check the check box next to each diagnostic test that you want to run, or check the top check box to run all tests.
5. Click Run.
6. View the test results in the lower part of the window. You may have to scroll the window to see all results.
For tests that fail, error messages describe the problem and provide recommended solutions. You can find error message
descriptions in the test command in the Cisco Wide Area Application Services Command Reference.
You can run the same diagnostic tests again and refresh the results by clicking the Refresh icon in the taskbar.
To print the results, click the Print icon in the taskbar.
To run the diagnostic tests from the CLI, use the test EXEC command.
Verifying the Physical Connectivity Between Peer WAAS Devices and

to Application Servers

To verify the physical connectivity of the peer WAAS device, follow these steps:
1. Check all cable connections on the switch or router that may impact the WAAS device.
2. Use the ping command to send an ICMP Echo request to the peer WAE.
wae# ping 10.1.1.2
PING 10.1.1.2 (10.1.1.2) 56(84) bytes of data.
64 bytes from 10.1.1.2: icmp_seq=1 ttl=37 time=83.9
ms
ms
ms
ms
ms
--- 10.1.1.2 ping statistics --5 packets transmitted, 5 received, 0% packet loss, time 3997ms
rtt min/avg/max/mdev = 79.274/80.538/83.904/1.793 ms
If a device is one hop away and you are unable to reach the device, then ping the intermediary gateway. If the gateway is not
reachable, enter the show ip routes command and check to make sure that the correct route is displayed. For example, enter:
wae# show ip routes
Destination
Gateway
---------------- ---------------10.10.10.1
0.0.0.0
10.43.62.4
0.0.0.0
10.43.62.0
0.0.0.0
10.10.10.0
0.0.0.0
0.0.0.0
10.43.62.1
Netmask
---------------255.255.255.255
255.255.255.255
255.255.255.192
255.255.255.0
0.0.0.0
If necessary, enter a static route for the gateway.

You can use a similar ping command to verify connectivity between the WAAS data center device and the application server
hosts.
Note that firewalls might block ICMP traffic and ICMP traffic does not follow the WCCP redirection path, so using the ping
command does not verify redirection or acceleration. As an alternative you could use a third party tool that performs a TCP-based
ping.
Checking CPU Load

To check the CPU load of a WAAS device, follow these steps:
1. From the WAAS Central Manager GUI navigation pane, choose My WAN > Manage Devices.
2. Click the Edit icon next to the name of the device on which you want to check the CPU load.
3. In the navigation pane, choose Monitor > Platform > CPU Statistics.
Figure 1. CPU Statistics
Verifying the Physical Connectivity Between Peer WAAS Devices andto Application Servers
11
You may want to adjust the time period of the chart, since the default is Last Hour. To adjust the time period, click the Settings
icon in the task bar and choose a different Time Frame such as Last Day or Last Week.
It is common for a WAAS device to show spikes or even longer durations of high CPU utilization during high user activity
periods. When the CPU remains at a high CPU level for significantly long durations, further troubleshooting or resizing of the
device may be indicated.
Gathering WAAS Troubleshooting Information

The following sections recommend ways to gather information that is relevant to the problem that is occurring and that is
necessary before contacting the Cisco Technical Assistance Center (TAC).
Rebooting the WAAS Device

Do not reboot the WAAS device unless it is absolutely necessary. Some information that is important to troubleshooting your
problem may not survive a reboot. Try to gather as much information as possible before rebooting.
Using show Commands

You can use several show commands in Exec mode to gather information specific to the symptoms you are observing in your
device. In most cases, you can gather the information you need to troubleshoot the device by entering the copy tech-support
command. This command runs many show commands that are useful for troubleshooting and gathers the output into a single file.
You can redirect the output of the copy tech-support command to a disk file, an FTP server, or a TFTP server. The command
syntax is as follows:
copy tech-support {disk filename | ftp {hostname | ip-address} remotedirectory remotefilename | tftp {hostname | ip-address}
remotefilename}
For example, to copy the output of the command to a disk file on the local system, specify the command as follows:
Checking CPU Load
12

wae# copy tech-support disk ts-report.txt
Other useful show commands include the following:

show alarms: Displays alarms.
show accelerator: Displays application accelerator status.
show license: Displays license status.
show statistics connection: Displays statistics for all TCP connections.
show statistics tfo: Displays TFO statistics.
show interface: Displays interface information and status. Verify that the speed and duplex match with the switch.
For WCCP deployments, use the following commands on the WAE:
show wccp gre
show wccp routers
show wccp wide-area-engine
show wccp flows
show egress-methods
For WCCP deployments, use the following commands on the router or switch (for each service group, where applicable):
show ip wccp
show ip wccp interfaces detail
show ip wccp service
show ip wccp service detail
For WCCP deployments, use the following commands on the router or switch when hashing is used:
show tcam counts
show mls stat
show mls netflow table detail
show mls netflow ip count
show mls netflow ip sw-installed count
show mls netflow ip sw-installed detail
show fm inteface interface_name
For WCCP deployments, use the following commands on the router or switch when masking is used:
show ip wccp service mask
show ip wccp service merge
show tcam interface interface_name acl {in | out} ip
show tcam interface interface_name acl {in | out} ip detail
Generating a System Report
A system report (sysreport) is a comprehensive report that you will need before you contact Cisco technical support. You can
generate a sysreport by running the copy sysreport command. The system report contains the output from many commands and
logs on the system, including show commands, network statistics, graphs, log contents, configuration settings, statistics, and so on.
It can take some time to generate a system report and it can be from 30 - 100 MB in size or larger. The system report contains
many more elements than are included in the copy tech-support command, and is generally needed when contacting Cisco
technical support.
Before generating a system report, use the test command to run the diagnostic tests so that this information is included in the
system report. When generating a system report on a Central Manager (or standby Central Manager), you should first make a
database backup by using the cms database backup command.
To generate a sysreport and store it to an FTP server, use this form of the command: copy sysreport ftp server-ip

remote-directory remote-file-name
For example:
wae# copy sysreport ftp 10.10.10.5 /reports wae1report
When generating a system report, do not use any command options that limit the report to a specific time period, as this could
cause information even within that time period not to be included.
Capturing and Analyzing Packets

Capturing packets (sometimes referred to as a "TCP dump") is a useful aid in troubleshooting connectivity problems with the
WAAS device or for monitoring suspicious activity. The WAAS device can track packet information for network traffic that
passes through it. The attributes of the packet are defined by an ACL. The WAAS device buffers the captured packets, and you
can copy the buffered contents to a file or to a remote server. You can also display the captured packet information on your
console or terminal.
Two packet capture utilities are available: tcpdump and tethereal. These commands require admin privileges.
By default, these commands capture only the first 64 bytes of each packet. We recommend that you use the -s 1600 option to
capture full packet data.
If you will be taking large traces, use tcpdump to create rolling packet captures in multiple files. (The -C option sets the
maximum size of each captured file in KB and the -M option sets the maximum number of log files to create.)
If you need to filter the packets captured, use tethereal with the -R read filter option. You can use tcpdump to create a large
packet capture, then use tethereal against the captured file to perform filtering.
Be careful when using tcpdump in a WCCP environment because tcpdump filters do not look within the GRE wrapper. You will
need to use tethereal if you need to do that.
With both commands, use the -i any option to capture all interfaces, or separate telnet sessions to capture on separate interfaces.
Use ^c (CTRL+c) to stop the packet capture.
There are several packet analysis tools that you can use to analyze packet capture files after you have captured them:
Wireshark: A free packet analysis tool with extensive capabilities (recommended over Ethereal).
Ethereal: Another free packet analysis tool with extensive capabilities.
Microsoft Netmon: Included with Windows server software.
Sniffer Pro
Using tcpdump
For the full tcpdump syntax, see tcpdump in the Cisco Wide Area Application Services Command Reference.
The most useful tcpdump options are as follows:
-i interface : The interface where you want to capture packets, for example:
lo : localhost
eth0 : GigabitEthernet 1/0
eth1 : GigabitEthernet 2/0
eth2 : InlinePort 1/1/wan
eth3 : InlinePort 1/1/lan
Generating a System Report
14

eth4 : InlinePort 1/0/wan
eth5 : InlinePort 1/0/lan
any : All available Ethernet ports. Be aware that the "any" interface cannot capture in promiscuous mode, so it
may miss some outgoing packets. For more information, see the Linux man page on tcpdump(8). Note: This
option is not available on WAAS version 4.1.5 and later.
bond0 : Logical interface that combines all physical interfaces.
-s snaplen: The maximum size that will be captured for each packet.
-w file: The name of the file where the captured packets will be written in their raw form.
-C count: The maximum size of the capture file, specified in thousands of bytes. If the -M option is also specified,
additional capture files are created.
-M num: The maximum number of log files created by rollover when the maximum file size is reached. This specifies
how many capture files to make before stopping the capture.
-D: Dumps the list of interfaces available for capturing.
The following example captures all packets to the file packets1.cap:
wae# tcpdump -i bond0 -s 1600 -w packets1.cap
Using tethereal
For the full tethereal syntax, see tethereal in the Cisco Wide Area Application Services Command Reference.
Useful tethereal options are as follows:
-R read_filter: Filtering can be very useful. Use the same filtering syntax as you would use with Ethereal or Wireshark, so
you can use one of those tools to help you compose a filter. tethereal is also useful for file conversion and filtering of a
packet capture file that has already been captured (for example, from tcpdump).
-F output_filetype: The default filetype is a libpcap file; however, the following options are available:
libpcap - libpcap (tcpdump, Ethereal, etc.)
rh6_1libpcap - RedHat Linux 6.1 libpcap (tcpdump)
suse6_3libpcap - SuSE Linux 6.3 libpcap (tcpdump)
modlibpcap - modified libpcap (tcpdump)
nokialibpcap - Nokia libpcap (tcpdump)
lanalyzer - Novell LANalyzer
ngsniffer - Network Associates Sniffer (DOS-based)
snoop - Sun snoop
netmon1 - Microsoft Network Monitor 1.x
netmon2 - Microsoft Network Monitor 2.x
ngwsniffer_1_1 - Network Associates Sniffer (Windows-based) 1.1
ngwsniffer_2_0 - Network Associates Sniffer (Windows-based) 2.00x
nettl - HP-UX nettl trace
visual - Visual Networks traffic capture
5views - Accellent 5Views capture
niobserverv9 - Network Instruments Observer version 9
The following examples show various options used for filtering and conversion:
To convert from one file format to another, use a command similar to the following:
wae# tethereal -r test-netmon.cap -F libpcap -w test-libpcap.cap
To use a read filter for the SYN flag, use a command similar to the following:
wae# tethereal -R "tcp.flags.syn eq 1"

To use a read filter for specific hosts (and look inside GRE packets), use a command similar to the following:
wae# tethereal -s 1600 -w dump1.cap ?R "ip.addr eq 2.43.183.254 and ip.addr eq 2.43.182.165"
Note: The tethereal command has some usage caveats that you should be aware of:
A filter defined using the -R option is ignored when it is combined with the -w option (writing to a file) in WAAS 4.1.1
and 4.1.3. To filter captured traffic and write to a disk file, use -f option to specify a capture filter. This issue is resolved in
version 4.1.5.
When using the -a option to print heavy traffic to the screen, it can take significantly longer than the autostop duration to
display the information on the screen. Wait for the command to finish. Displaying output to the console can take
significantly longer than through telnet or SSH, so console display is not recommended.
When using the -f option with the "host" or "not host" filter expression, the wrong traffic may be captured with WCCP
GRE encapsulated or VLAN traffic. With WCCP GRE traffic, tethereal sees only the outermost IP address, not the
original IP address inside the encapsulated packets. Add the "proto 47" keyword into the -f filter expression to capture the
correct traffic. Additionally, for VLAN traffic, add the "vlan" keyword into the -f filter expression to let the command
parse VLAN traffic correctly.
When using the -a filesize option together with the -R option, tethereal may stop unexpectedly and print the message
"Memory limit is reached" before reaching the specified autostop file size. In this case, the maximum memory limit for
the command was reached before the autostop file size limit.
Contacting Cisco Technical Support

If you are unable to resolve a problem after using the troubleshooting suggestions in the articles in this wiki, contact the Cisco
Technical Assistance Center (TAC) for assistance and further instructions. Before you call, have the following information ready
to help your TAC engineer assist you as quickly as possible:
Date that you received the WAAS hardware
Chassis serial number
Type of software and release number (if possible, enter the show version command)
Maintenance agreement or warranty information
A good problem description including:
What is the problem and what are the user visible symptoms?
Where and when it occurs
Error messages, alerts, and alarms seen
Steps to duplicate the problem
Brief explanation of the steps that you have already taken to isolate and resolve the problem
The diagnostic test output (see the "Running Diagnostics" section)
A Central Manager database backup (use the cms database backup command)
Information gathered in the "Gathering WAAS Troubleshooting Information" section.
Topology diagrams, including network/wiring diagrams and logical diagrams
Any other evidence of the problem such as packet captures, transaction logs, core files, WCCP show command output
from routers/switches and WAEs, and other log files.
You can reach TAC in one of these ways:
Create a service request online
Call the TAC at the telephone numbers on this page.
Contact the Cisco Small Business Support Center

This article describes how to troubleshoot basic optimization.
Guide Contents
Main Article
Contents
1 TFO
Troubleshooting
2 DRE
Troubleshooting
Basic WAAS optimizations include TCP flow optimization (TFO), data redundancy elimination (DRE), and persistent
Lempel-Ziv (LZ) compression.
TFO Troubleshooting
The number of TCP connections, their status, and disposition can give an indication of the health of the WAAS system in a
specific location. A healthy system will show a large number of connections, with a significantly large percentage of these closed
normally. The show statistics tfo detail command provides an indication of the volume, status, and disposition of connections
between a particular WAAS device and other devices in the network.
Contacting Cisco Technical Support
17

You can view global TFO statistics by using the show statistics tfo detail command as follows:
WAE# show statistics tfo detail
Total number of connections
: 2852
No. of active connections
: 3
No. of pending (to be accepted) connections
: 0
No. of bypass connections
: 711
No. of normal closed conns
: 2702
No. of reset connections
: 147
Socket write failure
: 0
Socket read failure
: 0
WAN socket close while waiting to write
: 0
AO socket close while waiting to write
: 2
WAN socket error close while waiting to read
: 0
AO socket error close while waiting to read
: 64
DRE decode failure
: 0
DRE encode failure
: 0
Connection init failure
: 0
WAN socket unexpected close while waiting to read : 32
Exceeded maximum number of supported connections : 0
Buffer allocation or manipulation failed
: 0
Peer received reset from end host
: 49
DRE connection state out of sync
: 0
Memory allocation failed for buffer heads
: 0
Unoptimized packet received on optimized side
: 0
Data buffer usages:
Used size:
0 B, B-size:
0 B, B-num: 0
Cloned size:
0 B, B-size:
0 B, B-num: 0
Buffer Control:
Encode size:
0 B, slow:
0, stop:
Decode size:
0 B, slow:
0, stop:
Scheduler:
Queue Size: IO:
0, Semi-IO:
0, Non-IO:
Total Jobs: IO:
1151608, Semi-IO:
5511278, Non-IO:
<-----Active connections
0
0
0
3690931
Policy Engine Statistics

------------------------Session timeouts: 0, Total timeouts: 0
Last keepalive received 00.5 Secs ago
Last registration occurred 15:00:17:46.0 Days:Hours:Mins:Secs ago
Hits:
7766, Update Released:
Active Connections:
3, Completed Connections:
Drops:
0
Rejected Connection Counts Due To: (Total: 0)
Not Registered
:
0, Keepalive Timeout
:
No License
:
0, Load Level
:
Connection Limit
:
0, Rate Limit
:
Minimum TFO
:
0, Resource Manager
:
Global Config
:
0, TFO Overload
:
Server-Side
:
0, DM Deny
:
No DM Accept
:
0
. . .
1088
7183
0
0
0
0
0
0
<-----Connection limit overload
The No. of active connections field reports the number of connections that are currently being optimized.
In the Policy Engine Statistics section of the output, the Rejected Connection Counts section show various reasons why
connections have been rejected. The Connection Limit counter reports the number of times that a connection has been rejected
because the maximum number of optimized connections has been exceeded. If you see a high number here, you should look into
overload conditions. See the article Troubleshooting Overload Conditions for more information.
Additionally, TFO optimization for connections that are pushed down from other AOs because they cannot optimize the traffic is
handled by the generic AO, which is covered in the article Troubleshooting the Generic AO.
TFO Troubleshooting
18

You can view TFO connection statistics by using the show statistics connection command. For details on using this command,
see the section "Checking the Optimized TCP Connections" in the Troubleshooting Overload Conditions article.
DRE Troubleshooting
When application acceleration is expected but not being observed, verify that the appropriate optimizations are being applied to
the traffic flow and that the DRE cache is reducing the size of the optimized traffic appropriately.
Policy engine maps for DRE and LZ optimization include the following:
DRE + LZ (full): policy-engine application map other optimize full
DRE only: policy-engine application map other optimize DRE yes compression none
LZ only: policy-engine application map other optimize DRE no compression LZ
TFO pass-through: policy-engine application map other pass-through
Various conditions could cause DRE and/or LZ not to be applied to a connection, even though it is configured:
Cache initialization is in progress
Disk I/O errors
Low memory
Data is not compressible or gain is too small
Data is encrypted such that it does not contain repeated byte sequences
Messages are too small to benefit from compression
Note: In all of the above conditions, the show statistics connection command will report Acceleration of "TDL" for connections
where this was the negotiated policy. Looking at the amount of DRE or LZ bypass traffic will tell you whether DRE or LZ
optimizations were actually applied. Use the show statistics connection conn-id command, as described later, and look at the
DRE encode numbers to see if the DRE or LZ ratio is near 0% and most of the traffic is bypassed. The first three conditions will
be reported by the "Encode bypass due to" field and the last three conditions result from the traffic data pattern and are accounted
for in the reported DRE and LZ ratios.
You can view the statistics for a specific connection to determine what basic optimizations were configured, negotiated with the
peer, and applied by using the show statistics connection conn-id command. First you will need to determine the connection ID
for a particular connection by using the show statistics connection command, as follows:
WAE#show stat conn
Current Active Optimized Flows:
Current Active Optimized TCP Plus Flows:
Current Active Optimized TCP Only Flows:
Current Active Optimized TCP Preposition Flows:
Current Active Auto-Discovery Flows:
Current Reserved Flows:
Current Active Pass-Through Flows:
Historical Flows:
1
0
1
0
0
10
0
375
D:DRE,L:LZ,T:TCP Optimization RR:Total Reduction Ratio

A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO
ConnID
343
Source IP:Port
10.10.10.10:3300
Dest IP:Port
PeerID Accel RR
10.10.100.100:80
00:14:5e:84:24:5f T
00.0%
<------
You will find the connection IDs for each connection listed at the end of the output. To view the statistics for a specific
connection, use the show statistics connection conn-id command, as follows:
WAE# sh stat connection conn-id 343

Connection Id:
343
Peer Id:
00:14:5e:84:24:5f
Connection Type:
EXTERNAL CLIENT
Start Time:
Tue Jul 14 16:00:30 2009
Source IP Address:
10.10.10.10
Source Port Number:
3300
Destination IP Address:
10.10.100.100
Destination Port Number: 80
Application Name:
Web
Classifier Name:
HTTP
Map Name:
basic
Directed Mode:
FALSE
Preposition Flow:
FALSE
Policy Details:
Configured:
TCP_OPTIMIZE + DRE + LZ
Derived:
Peer:
Negotiated:
Applied:
. . .
<-----Application name
<-----Classifier name
<-----Configured policy
<-----Policy negotiated with peer

<-----Applied policy
The Application Name and Classifier Name fields tell you the application and classifier applied to this connection.
The optimization policies are listed in the Policy Details section. If the Configured and Applied policies do not match, it means
that you configured one policy for this type of connection but a different policy was applied. This could result from the peer being
down, misconfigured, or overloaded. Check the peer WAE and its configuration.
The following section of output shows DRE encode/decode-related statistics including the number of messages, how many had
DRE applied, LZ applied, or bypassed DRE and LZ:
. . .
DRE: 353
Conn-ID: 353 10.10.10.10:3304 -- 10.10.100.100:139 Peer No: 0 Status: Active
-----------------------------------------------------------------------------Open at 07/14/2009 16:04:30, Still active
Encode:
Overall: msg:
178, in: 36520 B, out:
8142 B, ratio: 77.71%
<-----Overall compression
DRE: msg:
1, in:
356 B, out:
379 B, ratio:
0.00%
<-----DRE compression ratio
DRE Bypass: msg:
178, in: 36164 B
<-----DRE bypass
LZ: msg:
178, in: 37869 B, out:
8142 B, ratio: 78.50%
<-----LZ compression ratio
LZ Bypass: msg:
0, in:
0 B
<-----LZ bypass
Avg latency:
0.335 ms
Delayed msg:
0
<-----Avg latency
Encode th-put:
598 KB/s
<-----In 4.3.3 and earlier only
Message size distribution:
0-1K=0% 1K-5K=0% 5K-15K=0% 15K-25K=0% 25K-40K=0% >40K=0%
Decode:
Overall: msg:
14448, in:
5511 KB, out:
420 MB, ratio: 98.72%
<-----Overall compression
DRE: msg:
14372, in:
5344 KB, out:
419 MB, ratio: 98.76%
<-----DRE compression ratio
DRE Bypass: msg:
14548, in:
882 KB
<-----DRE bypass
LZ: msg:
14369, in:
4891 KB, out:
5691 KB, ratio: 14.07%
<-----LZ compression ratio
LZ Bypass: msg:
79, in:
620 KB
<-----LZ bypass
Avg latency:
4.291 ms
<-----Avg latency
Decode th-put:
6946 KB/s
Message size distribution:
0-1K=4% 1K-5K=12% 5K-15K=18% 15K-25K=9% 25K-40K=13% >40K=40%
<-----Output from here in 4.3.3 and
. . .
The following statistics are highlighted in the above example for both encoding and decoding:
Overall ratio - the overall compression ratio for the data including both DRE and LZ
DRE Troubleshooting
20

DRE ratio - the compression ratio due to DRE alone
DRE Bypass - the number of messages and bytes that bypassed DRE
LZ ratio - the compression ratio due to LZ alone
LZ Bypass - the number of messages and bytes that bypassed LZ
Avg latency - the average latency for the encode or decode operation
If you see a large amount of bypass traffic, the DRE compression ratio will be smaller than expected. It could be due to encrypted
traffic, small messages, or otherwise uncompressible data. Consider contacting TAC for further troubleshooting help.
If you see a large amount of LZ Bypass traffic, this could be due to a large amount of encrypted traffic, which is not generally
compressible.
The Average latency numbers can be useful for debugging a throughput issue. Depending on the platform, both the encode and
decode average latency are typically in the single digits of ms. If users experience low throughput and one or both of these
numbers are higher, it indicates an issue with encoding or decoding, generally on the side with the higher latency.
It may be useful to look at the DRE statistics data such as the oldest usable data, cache size, percent of cache used, hash table
RAM used, and so on by using the show statistics dre detail command, as follows:
WAE# sh stat dre detail
Cache:
Status: Usable, Oldest Data (age): 10h
Total usable disk size: 311295 MB, Used: 0.32%
Hash table RAM size:
1204 MB, Used: 0.00%
. . .
<-----Cache age
<-----Percent cache used
<-----Output from here is in 4.3.3 and earlier o
If you are not seeing significant DRE compression, it could be because the DRE cache is not populated with enough data. Check if
the cache age is short and less than 100 percent of the cache is used, which would indicate this situation. The compression ratio
should improve as the cache fills with more data. If 100% of the cache is used and the cache age is short, it indicates that the WAE
may be undersized and not able to handle the traffic volume.
If you are not seeing significant DRE compression, look at the Nack/R-tx counters in the following section of command output:
Connection details:
Chunks: encoded 398832, decoded 269475, anchor(forced) 43917(9407)
Total number of processed messges: 28229
num_used_block per msg: 0.053597
Ack: msg 18088, size 92509 B
Encode bypass due to:
remote cache initialization: messages: 1, size:
120 B
last partial chunk: chunks: 482, size: 97011 B
skipped frame header: messages: 5692, size:
703 KB
Nacks: total 0
R-tx: total 0
Encode LZ latency:
0.133 ms per msg
Decode LZ latency:
0.096 ms per msg
. . .

<-----Encode bypass reasons
<-----Nacks
<-----Retransmits
The Nacks and R-tx counters should generally be low relative to the traffic volume. For example, about 1 per 100 MB of original
(unoptimized) traffic. If you see significantly higher counts, it could indicate a DRE cache synchronization problem. Use the clear
cache dre command to clear the DRE cache on all devices, or contact TAC.
The encode bypass reasons counters report the number of bytes bypassed due to various reasons. This can help you determine
what is causing bypass traffic (other than an unoptimizable data pattern).

It is sometimes helpful to identify the connected and active peer WAEs and look at peer statistics, which you can do with the
show statistics peer dre command as follows:
WAE# sh stat peer dre
Current
Current
Current
Maximum
Maximum
Maximum
number
number
number
number
number
number
of
of
of
of
of
of
connected peers:
active peers:
degrade peers:
connected peers:
active peers:
degraded peers:
1
1
0
1
1
0
Active peer details:

Peer-No : 0
Context: 65027
Peer-ID : 00:14:5e:95:4a:b5
Hostname: wae7.example.com
<-----Peer hostname
-----------------------------------------------------------------------------Cache: Used disk: 544 MB, Age: 14d23h
Cache: Used disk: 544 MB
Peer version: 0.4
Ack-queue size:
38867 KB
Buffer surge control:
Delay:
avg-size
0 B, conn:
0, flush:
Agg-ft: avg-size
20902 B, conn:
388, flush:
remote low-buff: 0, received flush:
0
<-----Peer cache details in 4.3.3 and earlie

<-----Peer cache details in 4.4.1 and later
<----|
|<---In 4.3.3 and earlier only
0
|
0
|
<-----
Connections:
Total (cumulative): 3226861, Active: 597
Concurrent Connections (Last 2 min): max 593, avg 575
. . .
Other output from this command shows the encode and decode statistics similar to an individual connection.
This article describes a general troubleshooting methodology for all application accelerators.
Guide Contents
Main Article
DRE Troubleshooting
22

Contents
1 General Troubleshooting Methodology
for AOs
WAAS provides several application accelerators (also known as AOs) that accelerate various TCP protocols such as CIFS, HTTP,
NFS, EPM, MAPI, SSL, and live streaming video (RTSP). These application accelerators do not accelerate specific applications,
but rather they accelerate any application that uses the specified protocol.
This article describes a general troubleshooting methodology for all AOs.
General Troubleshooting Methodology for AOs
The following general troubleshooting methodology applies to all AOs:

1. Verify the AO configuration and operational state.
2. Verify the Application Traffic Policy configuration for the AO.
3. Check the global and AO-specific statistics.
4. Verify that connections are handled/optimized by the AO.
5. Debug and trace connection-specific AO statistics.
To verify the configuration and operational status of all AOs, use the show accelerator and show license commands as shown in
Figure 1.
Figure 1. Verifying All Accelerator Status
Contents
23
For specific information on troubleshooting individual AOs, see the following articles:
This article describes how to troubleshoot the CIFS AO.
General Troubleshooting Methodology for AOs
Guide Contents
Main Article
24

Contents
1 CIFS AO Troubleshooting
1.1 CIFS AO Logging
1.2 Windows Print Accelerator
Troubleshooting
CIFS AO Troubleshooting
The CIFS accelerator transparently optimizes CIFS traffic on ports 139 and 445.
You can verify the general AO configuration and status with the show accelerator and show license commands, as shown in
Figure 1. The Enterprise license is required for CIFS accelerator operation.
Figure 1. Verifying Accelerator Status
Contents
25
Next, verify the status that is specific to the CIFS AO by using the show accelerator cifs command, as shown in Figure 2. You
want to see that the CIFS AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State
is Enabled but the Operational State is Shutdown, it indicates a licensing problem.
Figure 2. Verifying CIFS Accelerator Status
Use the show running-config command to verify that the CIFS traffic policy is properly configured. You want to see accelerate
cifs for the WAFS application action and you want to see appropriate match conditions listed for the CIFS classifier, as follows:
WAE674# sh run | include CIFS
classifier CIFS
name WAFS classifier CIFS action optimize full accelerate cifs

WAE674# sh run | begin CIFS
...skipping
classifier CIFS
match dst port eq 139
exit
Use the show statistics connection optimized cifs command to check that the WAAS device is establishing optimized CIFS
connections. Verify that "TCDL" appears in the Accel column for a connection. A "C" indicates that the CIFS AO was used.
WAE674# sh stat conn opt cifs
Historical Flows:
3
3
0
1
0
0
100
D:DRE,L:LZ,T:TCP Optimization,
ConnID
1074
Source IP:Port
10.10.10.10:2704
Dest IP:Port
10.10.100.100:445
PeerID
00:14:5e:84:24:5f
Accel
TCDL
<------Look for "C"
If you see "TDL" in the Accel column, the connection was optimized by transport optimizations only and was not inspected by the
CIFS AO. This situation can happen if the CIFS AO is disabled, the Enterprise license is not configured, or if the maximum
connection limit is reached.
If you see a "G" instead of a "C" in the Accel column, then the connection was pushed down from the CIFS AO to the generic AO
and was optimized with transport optimizations only. This situation can happen if the connection requires SMB2 or a digital
signature and an error message is logged for it.
In version 4.1.3, the syslog has the following error message for digitally signed connections:
2009 Apr 25 13:42:08 wae java: %WAAS-CIFSAO-4-131230: (146708) Connection to test1.example.com will be handled by
generic optimization only, since test1.example.com requires digital signing.
In version 4.1.5 and later, check the CIFS internal error logs to see the reason why the connection was pushed down to the generic
AO. In the cifs_err.log, look for this message for SMB2 connections:
2009-06-29 10:15:04,996 WARN (actona.cifs.netbios.IPacketerHandlerOrigCifs:139) Thread-2 from host 10.56.64.205. Pushing down the connection.
Received SMBv2 packet
In the cifs_err.log, look for this message for digitally signed connections:
2009-10-29 05:37:54,541 WARN (actona.rxFlow.cifs.requests.NegotiateRequest:359) lightRxFlowPool-4 - Request ID: 1

Connection to 10.56.78.167 will be handled by generic optimization only, since 10.56.78.167 requires digital sign
To view similar information from the Central Manager, choose the WAE device, then choose Monitor > Optimization >
Connections Statistics.
Figure 3. Connection Statistics Report
CIFS AO Troubleshooting
27
You can view the CIFS connection statistics by using the show statistics connection optimized cifs detail command as follows:
WAE674# sh stat connection optimized cifs detail
Connection Id:
1801
Peer Id:
00:14:5e:84:24:5f
Connection Type:
EXTERNAL CLIENT
Start Time:
Thu Jun 25 06:15:58 2009
Source IP Address:
10.10.10.10
Source Port Number:
3707
10.10.100.100
Application Name:
WAFS
Classifier Name:
CIFS
Map Name:
basic
Directed Mode:
FALSE
Preposition Flow:
FALSE
Policy Details:
Configured:
Derived:
Peer:
Negotiated:
Applied:
Accelerator Details:
Configured:
CIFS
Derived:
CIFS
Applied:
CIFS
Hist:
None
Bytes Read:
Bytes Written:
<-----Should see WAFS

<-----Should see CIFS
<-----Should see CIFS configured

<-----Should see CIFS applied
Original
Optimized
-------------------- -------------------189314
10352510
91649704
28512
. . .
Connection details:
Chunks: encoded 3, decoded 49922, anchor(forced) 0(1)
Total number of processed messges: 1820
num_used_block per msg: 0.140659
Ack: msg 1609, size
7066 B
Encode bypass due to:
last partial chunk: chunks: 1, size:
142 B
skipped frame header: messages: 138, size: 27202 B
Nacks: total 0
R-tx: total 0
Encode LZ latency:
0.060 ms per msg
Decode LZ latency:
0.071 ms per msg
Aggregation encode: Retransmissions: 0
level 0: chunks:
3 hits:
0 miss:
<-----Packets lost between peers

3

level 1: chunks:
0 hits:
0 miss:
0
level 2: chunks:
0 hits:
0 miss:
0
level 3: chunks:
0 hits:
0 miss:
0
Aggregation decode: Collisions: 0
level 0: chunks:
174093 hits:
128716 miss:
0
level 1: chunks:
0 hits:
0 miss:
0
level 2: chunks:
0 hits:
0 miss:
0
level 3: chunks:
0 hits:
0 miss:
0
Aggregation stack memory usage: Sender:
452 B Receiver:
Noise filter: Chunks: 0, Bytes:
0 B
. . .
9119 B
If the Retransmissions counter increases, it means that packets are getting lost in the middle, between the two peer WAEs. This
situation will result in lower throughput. You should investigate possible causes for packet lose in the network between the two
peer WAEs.
You can view the CIFS request statistics by using the show statistics cifs requests command as follows:
Figure 4. Inspecting CIFS Request Statistics
CIFS AO Logging
The following log files are available for troubleshooting CIFS AO issues:
Transaction log files: /local1/logs/tfo/working.log (and /local1/logs/tfo/tfo_log_*.txt)
CIFS internal log file: /local1/errorlog/cifs/cifs_err.log
Debug log files: /local1/errorlog/cifsao-errorlog.current (and cifsao-errorlog.*)
For easier debugging, you should first set up an ACL to restrict packets to one host.
CIFS AO Logging
29

WAE674(config)# ip access-list extended 150 permit tcp host 10.10.10.10 any
WAE674(config)# ip access-list extended 150 permit tcp any host 10.10.10.10
To enable transaction logging, use the transaction-logs configuration command as follows:

wae(config)# transaction-logs flow enable
wae(config)# transaction-logs flow access-list 150
You can view the end of a transaction log file by using the type-tail command as follows:
wae# type-tail tfo_log_10.10.11.230_20090715_130000.txt

:EXTERNAL CLIENT :00.14.5e.84.24.5f :basic :WAFS :CIFS :F :(DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO)
(DRE,LZ,TFO) :<None> :(CIFS) (CIFS) (CIFS) :<None> :<None> :0 :180
Wed Jul 15 15:48:45 2009 :1725 :10.10.10.10 :2289 :10.10.100.100 :139 :OT :START :EXTERNAL CLIENT :00.14.5e.84.24.
:CIFS :F :(DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) :<None> :(CIFS) (CIFS) (CIFS) :<None> :
Wed Jul 15 15:48:55 2009 :1725 :10.10.10.10 :2289 :10.10.100.100 :139 :OT :END : EXTERNAL CLIENT :(CIFS) :0 :0 :15
To set up and enable debug logging of the CIFS AO, use the following commands.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a
production environment.
You can enable detailed logging to the disk as follows:
WAE674(config)# logging disk enable
WAE674(config)# logging disk priority detail
You can enable debug logging for connections in the ACL:

WAE674# debug connection access-list 150
The options for CIFS AO debugging are as follows:

WAE674# debug accelerator cifs ?
all
enable all CIFS accelerator debugs
shell
enable CIFS shell debugs
You can enable debug logging for CIFS connections and then display the end of the debug error log as follows:
WAE674# debug accelerator cifs all
WAE674# type-tail errorlog/cifsao-errorlog.current follow
Windows Print Accelerator Troubleshooting

The Windows print accelerator optimizes print traffic between clients and a Windows print server.
Troubleshooting the Windows print accelerator is similar to troubleshooting the CIFS AO. You can verify the general AO
configuration and status with the show accelerator and show license commands, as shown in Figure 1. The CIFS accelerator
must be enabled and the Enterprise license is required. Next, verify the status specific to the CIFS AO by using the show
accelerator cifs command.
Use the show statistics windows-print requests command and verify that the "Documents spooled" and "Pages spooled"
counters are incrementing, as follows:
WAE# sh stat windows-print requests
Statistics gathering period: hours: 6 minutes: 4 seconds: 2 ms: 484
Documents spooled: 29
Windows Print Accelerator Troubleshooting
<-----Should be incrementing
30

Pages spooled: 3168
Total commands: 61050
Remote commands: 849
ALL_COMMANDS total: 61050 remote: 849 async: 58719 avg local: 1.813ms avg remote: 177.466ms
. . .
This article describes how to troubleshoot the HTTP AO.

Guide Contents
Main Article
Contents
1 HTTP Accelerator Troubleshooting
1.1 Viewing HTTP Statistics
1.2 Viewing HTTPS
Statistics
1.3 Viewing the HTTP
Metadata Cache
1.4 Viewing the HTTPS
Metadata Cache
1.5 Metadata Cache
Cache-Control Behavior
1.6 Metadata Caching
Exceptions
2 HTTP AO Logging
Contents
31
HTTP Accelerator Troubleshooting

The HTTP accelerator optimizes HTTP and HTTPS (in version 4.3.1 and later) traffic using the following techniques:
TCP connection reuse across the WAN. Avoids a connection setup penalty for subsequent HTTP connections requested
by the same client. (Does not apply to HTTPS traffic.)
HTTP metadata caching. Certain HTTP responses are cached, along with their URLs and metadata information, so that
the edge WAE can respond locally to subsequent requests for the same URL. (Available only in version 4.2.1 and later.)
The three types of cached responses are as follows:
301 Permanently-Redirected
304 Not-Modified
401 Authorization-Required
HTTPS metadata caching. Certain HTTPS responses are cached, along with their URLs and metadata information, so
that the edge WAE can respond locally to subsequent requests for the same URL. (Available only in version 4.3.1 and
later.)
HTTP suppress server encoding. Removes the Accept-Encoding header from the HTTP and HTTPS requests,
preventing the server from sending compressed data towards the WAN. This allows the WAE to apply its own
compression, typically resulting in a better compression ratio. (Available only in version 4.2.1 and later.)
DRE hints. Provides specific hints to the DRE module to better compress the HTTP and HTTPS traffic based on the
additional knowledge on the HTTP protocol provided by parsing the layer 7 payload:
Skip header: Instructs the DRE module to not compress HTTP/HTTPS headers resulting in a better compression
of the object.
Flush: Instructs the DRE module to start compressing as soon as an HTTP/HTTPS transaction is fully processed.
Skip LZ: Instructs the DRE module to not apply LZ compression to all objects already compressed by the original
server, thus reducing the CPU overhead.
The HTTP metadata caching, suppress server encoding, and DRE hinting features can be configured separately. The TCP
connection reuse feature is always active when the HTTP AO is enabled and applies only to HTTP traffic.
You can verify the general AO configuration and status with the show accelerator and show license commands, as described in
the Troubleshooting Application Acceleration article. The Enterprise license is required for HTTP accelerator operation.
Next, verify the status that is specific to the HTTP AO by using the show accelerator http command, as shown in Figure 1. You
want to see that the HTTP AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State
is Enabled but the Operational State is Shutdown, it indicates a licensing problem. For each of the HTTP features the current mode
is shown (User/Default) along with the value (Enabled, Disabled or configured value). The Suppress Server Encoding and
Metadatacache items were added in version 4.2.1, and the DRE Hints and HTTPS Metadatacache items were added in version
4.3.1.
For HTTPS traffic to be optimized by both the SSL and HTTP AOs, ensure that one of these optional features is enabled: HTTPS
metadata caching, suppress-server-encoding or DRE hints.
Figure 1. Verifying the HTTP Accelerator Status
HTTP Accelerator Troubleshooting
32
Use the show running-config command to verify that the HTTP/HTTPS traffic policy is properly configured and which of the
features is enabled. You want to see accelerate http for the Web application action and you want to see appropriate match
conditions listed for the HTTP classifier, as follows:
WAE674# sh run | include HTTP
accelerator http suppress-server-encoding enable
accelerator http metadatacache https enable
accelerator http dre-hints enable
classifier HTTP
classifier HTTPS
name Web classifier HTTP action optimize full accelerate http
name Web classifier HTTPS action optimize DRE no compression none
WAE674# sh run | begin HTTP

...skipping
classifier HTTP
<----- in 4.2.1 and later

at least one of these must be en
<----- HTTP acceleration

<----- HTTPS static policy applies to t
matching any SSL accelerated-ser

WAE674# sh run | begin HTTPS
...skipping
classifier HTTPS
<----- add here any nondefault HTTPS po
Viewing HTTP Statistics

Use the show statistics accelerator http command to see the following statistics:
How much time is being saved by the HTTP AO. You can see the overall Time Saved by the entire HTTP AO or the Time
Saved by each of the features:
Time Saved by fast connection reuse
Time Saved by the three metadata caches
Number of cache hits/misses for the metadata caches
Number of times suppress server encoding is applied to HTTP requests
Number of times DRE hints are provided based on the content of the HTTP headers
Number of HTTP transactions (request+response) processed
Number of errors in the HTTP header processing
Number of cache revalidations
WAE674# sh stat accel http
HTTP:
Global Statistics
----------------Time Accelerator was started:
Time Statistics were Last Reset/Cleared:
Total Handled Connections:
Total Optimized Connections:
Total Connections Handed-off with Compression Policies Unchanged:
Total Dropped Connections:
Current Active Connections:
Current Pending Connections:
Maximum Active Connections:
Total Time Saved (ms):
Current Active Connections Free For Fast Connection Use:
Total Connections Handed-off:
Total Connections Handed-off with Compression Policies Disabled:
Total Connections Handed-off to SSL:
Total Connection Hand-off Failures:
Total Fast Connection Successes:
Total Fast Connection Failures:
Maximum Fast Connections on a Single Connection:
Total CONNECT Requests with Incomplete Message:
Percentage of Connection Time Saved:
Total Round Trip Time For All Connections (ms):
Total Fast Connections Initiated by Peer:
Total SYN Timeouts:
Total Time for Metadata Cache Miss (ms):
RTT saved by Redirect Metadata Cache (ms):
RTT saved by Authorization Redirect Metadata Cache (ms):
RTT saved by Content Refresh Check Metadata Cache (ms):
Total Time Saved by Fast Connection Use (ms):
Total Locally Served Redirect Responses:
Total Locally Served Unauthorized Responses:
Total Locally Served Conditional Responses:
Total Remotely Served Redirect Responses:
Total Remotely Served Unauthorized Responses:
Total Remotely Served Conditional Responses:
Total Requests with URL Longer than 255 Characters:
Tue Apr 6 06:04:06 2010

Tue Apr 6 06:04:06 2010
3743984
3743984
0
0
48
0
176
35584437
<-----Should
2
0
0
0
0
3617244
<-----Should
0
100
0
37
4922767377
0
0
2
<-----Output
5988
<-----Should
345
<-----Should
44987
<-----Should
456
453
<-----Should
56
<-----Should
4932
<-----Should
0
0
1
0
be incrementing
be incrementing
from here is in 4.2.1

be incrementing
be incrementing
be incrementing
be incrementing
be incrementing
be incrementing

Total
Total
Total
Total
Total
Total
Total
Total
Total
Total
Total
Requests with HTTP Pipelining:

Transactions Handled:
Server Compression Suppression:
Requests Requiring Server Content-Revalidation:
Responses not to be Cached:
Connections Expecting Authentication:
Connections with Unsupported HTTP Requests:
Connections with Unsupported HTTP Responses:
Hints Sent to DRE Layer to Flush Data:
Hints Sent to DRE Layer to Skip LZ:
Hints Sent to DRE Layer to Skip Header Information:
0
2
1
0
0
0
0
0
2
0
1
<-----Total number of HTTP transac

<-----Total number of Accept-Encod
If the Total Time Saved counter in the output above is not incrementing or is quite small, it indicates that the HTTP AO is not
providing much benefit. If the Total Time Saved by one of the three metadata caches is not incrementing or is quite small, it
indicates that the corresponding metadata cache is not providing much benefit.
The Total Server Compression Suppression counter indicates how many times the Accept-Encoding header has been removed, in
an attempt to provide a better compression by the WAE device. The Total Hints Sent to DRE Layer counters indicate how many
times each of the DRE hints (Flush Data, Skip LZ, Skip Header) has been issued to the DRE module, in an attempt to better
compress the data.
To view similar information from the Central Manager in version 4.2.1 and later, choose the WAE device, then choose Monitor >
Acceleration > HTTP Acceleration Report and choose the Details tab to see the following charts:
HTTP Response Time Savings (fast connection reuse, redirect, conditional, and unauthorized cached)
HTTP Optimization Count (number of times each of the above optimizations has been applied)
HTTP Optimization Techniques (for all HTTP optimizations, including metadata caches, connection reuse, DRE hints and
suppress-server-encoding)
To see debugging information on the HTTP header parsing and error conditions, use the show statistics accelerator http debug
command (in 4.3.1 and later) to determine the following:
Number of 301, 304 and 401 responses cached
Number of HTTP headers, version and methods
Reasons for HTTP responses not being cached
Total number of HTTP responses being cached
Reasons for HTTP requests not being served from the local cache
Use the show statistics connection optimized http command to check that the WAAS device is establishing optimized HTTP
connections. Verify that an "H" appears in the Accel column for HTTP connections, which indicates that the HTTP AO was used,
as follows:
WAE674# sh stat conn opt http
2
2
0
0
0
0
Historical Flows:
100
ConnID Source IP:Port
Dest IP:Port
PeerID
5929
10.10.10.10:3446
10.10.100.100:80
00:14:5e:84:24:5f
Accel
THDL
<-------Look for "H"
You can check connection statistics for closed connections by using the show statistics connection closed http command.
Viewing HTTP Statistics
35

Figure 2. Connection Statistics Report with HTTP
In the Connection Statistics report, the globe icon in the Applied Policy column shows that the HTTP AO was used for a
connection. (Place your cursor over an icon to see its meaning.)
You can view the HTTP connection statistics by using the show statistics connection optimized http detail command. Look for
the "Fast connections" counter in the output. A positive value for this counter means that the HTTP AO benefits clients by reusing
persistent connections, which reduces latency.
WAE674# show stat conn opt http detail
Connection Id:
1496
Peer Id:
00:14:5e:84:24:5f
Connection Type:
EXTERNAL CLIENT
Start Time:
Wed Jul 15 05:09:52 2009
Source IP Address:
10.10.10.10
Source Port Number:
1760
10.10.100.100
Application Name:
Web
Classifier Name:
HTTP
Map Name:
basic
Directed Mode:
FALSE
Preposition Flow:
FALSE
Policy Details:
Configured:
Derived:
Peer:
Negotiated:
Applied:
Configured:
HTTP
Derived:
HTTP
<-----Should see Web

<-----Should see HTTP
<-----Should see HTTP configured

Applied:
Hist:
<-----Should see HTTP applied
HTTP
None
Bytes Read:
Bytes Written:
. . .
HTTP : 1496
Original
Optimized
-------------------- -------------------266
139160
82686
128

05:09:52 2009
Total Bytes Read:
56367
Total Bytes Written:
56367
Total Bytes Buffered:
0
Total Internal Bytes Read:
Total Internal Bytes Written:
Bit Flags for I/O state:
Internal object pointer:
Wed Jul 15
Fast connections:
. . .
11
3269
3269
0
92
92
1040
2046823200
<-----Reused connections
Viewing HTTPS Statistics

(This section applies only to version 4.3.1 and later.)
Use the show statistics accelerator http https command to see the following statistics:
How much time is being saved by the HTTP AO for HTTPS traffic. You can see the overall Time Saved by the entire
HTTPS metadata cache or the Time Saved by each of the three metadata caches
Number of cache hits/misses for the metadata caches
Number of times suppress server encoding is applied to HTTPS requests
Number of times DRE hints are provided based on the content of the HTTPS headers
Number of HTTPS transactions (request+response) processed
Number of errors in the HTTPS header processing
Number of cache revalidations
WAE674# sh stat accel http https
HTTPS Statistics
----------------Total Optimized HTTPS Connections:
10
Total Handled HTTPS Connections:
10
Total Active HTTPS Connections:
2
Total Proxy-Connect HTTPS Connections:
0
Total Proxy-Connect HTTPS Insert Failures:
0
RTT saved by HTTPS Content Refresh Check Metadata Cache - (ms):
44
RTT saved by HTTPS Redirect Metadata Cache - (ms):
10
RTT saved by HTTPS Authorization Required Metadata Cache - (ms):
5
Total Locally Served HTTPS Conditional Responses:
44
Total Locally Served HTTPS Redirect Responses:
10
Total Locally Served HTTPS Unauthorized Responses:
5
Total Remotely Served HTTPS Conditional Responses:
32
Total Remotely Served HTTPS Redirect Responses:
2
Total Remotely Served HTTPS Unauthorized Responses:
1
Total Hints Sent to DRE Layer to Skip Header Information - HTTPS: 121
Total Hints Sent to DRE Layer to Flush Data - HTTPS:
121
<-----Should be
<-----Should be
<-----Should
<-----Should
<-----Should
<-----Should
<-----Should
<-----Should
be
be
be
be
be
be
incrementing
incrementing
incrementing
incrementing
incrementing
incrementing
incrementing
incrementing
37

Total
Total
Total
Total
Total
Total
Total
Total
Hints Sent to DRE Layer to Skip LZ - HTTPS:

Server Compression Suppression - HTTPS:
Time Saved from all HTTPS metadata cache hits:
Time HTTPS Cache Miss:
HTTPS Requests Requiring Server Content-Revalidation:
HTTPS Responses not to be Cached:
HTTPS Connections Bypassed due to URL Based Bypass List:
HTTPS Connections Bypassed due to IP Based Bypass List:
0
110
59
4
32
0
0
0
If the Total Time Saved counter in the output above is not incrementing or is quite small, it indicates that the HTTP AO is not
providing much benefit to the HTTPS traffic. If the Total Time Saved by one of the three metadata caches is not incrementing or
is quite small, it indicates that the corresponding metadata cache is not providing much benefit.
The Total Server Compression Suppression counter indicates how many times the Accept-Encoding header has been removed
from HTTPS requests, in an attempt to provide a better compression by the WAE device. The Total Hints Sent to DRE Layer
counters indicate how many times each of the DRE hints (Flush Data, Skip LZ, Skip Header) has been issued to the DRE module,
in an attempt to better compress the data.
To view similar information from the Central Manager in version 4.3.1 and later, choose the WAE device, then choose Monitor >
Acceleration > HTTPS Acceleration Report and choose the Details tab to see the following charts:
HTTPS Response Time Savings (redirect, conditional, and unauthorized cached)
HTTPS Optimization Count (number of times each of the above optimizations has been applied)
HTTPS Optimization Techniques (for all HTTPS optimizations, including metadata caches, DRE hints and
suppress-server-encoding)
To see debugging information on the HTTPS header parsing and error conditions, use the show statistics accelerator http debug
command to determine the following:
Number of 301, 304 and 401 responses cached
Number of HTTP headers, version and methods
Reasons for HTTP responses not being cached
Total number of HTTP responses being cached
Reasons for HTTP requests not being served from the local cache
Use the show statistics connection optimized http command to check that the WAAS device is establishing optimized HTTPS
connections. Verify that both an "H" and an "S" appear in the Accel column for HTTPS connections, which indicates that both the
HTTP and SSL AOs were used, as follows:
WAE674# sh stat conn opt http
2
2
0
0
0
0
Historical Flows:
100
ConnID
Source IP:Port
Dest IP:Port
PeerID
5929
10.10.10.10:3446
10.10.100.100:80
00:14:5e:84:24:5f
Accel
THSDL
<-------Look for "H" and "S"
You can check connection statistics for closed connections by using the show statistics connection closed http or show statistics
connection closed ssl commands.
38

Figure 3. Connection Statistics Report with HTTP and SSL
In the Connection Statistics report, the globe icon in the Applied Policy column shows that the HTTP AO was used for a
connection and the lock icon indicates that the SSL AO was applied. (Place your cursor over an icon to see its meaning.)
You can view the HTTPS connection statistics by using the show statistics connection optimized http detail and show statistics
connection optimized ssl detail commands.
WAE674# show stat conn opt http detail
Connection Id:
34
Peer Id:
00:14:5e:cd:9c:c9
Connection Type:
EXTERNAL CLIENT
Start Time:
Thu Oct 28 14:47:56 2010
Source IP Address:
10.3.2.1
Source Port Number:
40829
110.1.1.100
Application Name:
SSL
Classifier Name:
HTTPS
Map Name:
basic
Directed Mode:
FALSE
Preposition Flow:
FALSE
Policy Details:
Configured:
TCP_OPTIMIZE
Derived:
TCP_OPTIMIZE
Peer:
TCP_OPTIMIZE
Negotiated:
Applied:
Configured:
None
Derived:
None
Applied:
HTTP,SSL
Hist:
None
Bytes Read:
Bytes Written:
<-----Should see HTTP and SSL applied
Original
Optimized
-------------------- -------------------5162
21874
1977819
5108
Total Reduction Ratio: 98.639%
HTTP : 34
<-----Should see SSL

<-----Should see HTTPS

14:47:56 2010
Total Bytes Read:
1972570
1972570
. . .
Thu Oct 28
4620
4620
SSL : 34
14:47:56 2010
Total Bytes Read:
0
0
. . .
Hostname in HTTP CONNECT:
IP Address in HTTP CONNECT:
TCP Port in HTTP CONNECT:
Thu Oct 28
0
0
<------ the last three counters apply only to

Proxy Connect type of HTTPS connections
Viewing the HTTP Metadata Cache

To display the content of the three HTTP metadata caches (redirect, conditional, and unauthorized), use the show cache
http-metadatacache all command. Only the full URL and the expiration (in seconds) are displayed. You can also display the
content of each of the three caches separately by using the following commands:
show cache http-metadatacache redirect-response
show cache http-metadatacache conditional-response
show cache http-metadatacache unauthorized-response
The typical output of the above commands is as follows:
Redirect Cache
Active entries: 1, Max Entries: 1500
URL: www.abcnews.com/, Expiration (sec): 3206
Conditional Cache
URL: www.cisco.com/web/fw/i/quicklinks-rnd-corners.gif, Expiration (sec): 3594
URL: www.cisco.com/web/fw/i/hp-sprites.gif, Expiration (sec): 3594
URL: www.cisco.com/en/US/home/images/ba-actsGreen-logo.jpg, Expiration (sec): 3594
URL: www.cisco.com/en/US/home/images/fp-eos3.jpg, Expiration (sec): 3594
URL: www.cisco.com/en/US/home/images/fp-AP541n.jpg, Expiration (sec): 3594
URL: www.cisco.com/web/fw/c/home.min.css, Expiration (sec): 3592
Unauthorized Cache
URL: l.yimg.com/index.html, Expiration (sec): 86393
You can clear the content of the three caches with the clear cache http-metadatacache all command.
If you want to clear the content of each cache separately, you can use the following commands:
clear cache http-metadatacache redirect-response
clear cache http-metadatacache conditional-response
clear cache http-metadatacache unauthorized-response
If you want to specify a URL to be deleted you can use the following command:
Viewing the HTTP Metadata Cache
40

clear cache http-metadatacache {all|redirect|conditional|unauthorized} URL
Viewing the HTTPS Metadata Cache

To display the number of entries in the three HTTPS metadata caches (redirect, conditional, and unauthorized), use the show
cache http-metadatacache https command. Unlike the corresponding command for the HTTP metadata cache, the URL and the
expiration time are not displayed. You can also display the number of entries for each of the three caches separately by using the
following commands:
show cache http-metadatacache https redirect-response
show cache http-metadatacache https conditional-response
show cache http-metadatacache https unauthorized-response
The typical output of the above commands is as follows:
HTTPS Redirect Cache
Active HTTP entries: 0, Active HTTPS entries: 0 Max Entries: 3250
HTTPS Conditional Cache
HTTPS Unauthorized Cache
You can clear the content of the three caches with the clear cache http-metadatacache https command.
If you want to clear the content of each cache separately, you can use the following commands:
clear cache http-metadatacache https redirect-response
clear cache http-metadatacache https conditional-response
clear cache http-metadatacache https unauthorized-response
Metadata Cache Cache-Control Behavior

For HTTP and HTTPS (in version 4.3.1) 304 responses, the metadata cache honors all Cache-Control directives (Cache-Control:
no-cache, no-store, private, must-revalidate, proxy-revalidate, max-age=0, Pragma: no-cache). There is an option to disable such
Cache-Control checks, which means that all 304 responses with Cache-Control headers specifying no-cacheability are cached and
all requests with Cache-Control headers specifying no-cacheability can be served from the local cache.
Understand that disabling the cache control checks might increase the benefits of the metadata-cache, because some browsers or
web servers might have a default option to include one cache control header in all responses in order to force revalidation of the
object through the original server. This would make the metadata cache ineffective for 304 responses.
The option can be independently controlled for HTTP/S requests (cache lookups) and responses (cache insertions).
To disable cache control checks on HTTP/S 304 requests, use the following command:
WAE#accelerator http metadatacache request-ignore-no-cache enable
This command forces the metadatacache to disregard all Cache-Control directives in HTTP/S 304 requests. (The default [no] form
of this command forces the metadatacache to honor all Cache-Control directives in HTTP/S 304 requests.)
To disable cache control checks on HTTP/S 304 responses, use the following command:
WAE#accelerator http metadatacache response-ignore-no-cache enable
Viewing the HTTPS Metadata Cache
41

This command forces the metadatacache to disregard all Cache-Control directives in HTTP/S 304 responses. (The default [no]
form of this command forces the metadatacache to honor all Cache-Control directives in HTTP/S 304 responses.)
The metadata cache honors Cache-Control headers for 301 and 401 responses. If the response has any of the Cache-Control
headers (no-cache, no-store, private, must-revalidate, proxy-revalidate, max-age=0, Pragma: no-cache), it is not cached.
Metadata Caching Exceptions

There are certain exceptions to what is cached. The cache insertion or lookup does not occur when the HTTP AO encounters one
of the following conditions on the HTTP/S request/response being processed:
Non-RFC complaint requests and responses: malformed/invalid headers, repeated headers, missing headers, unexpected
body, unexpected chunked encoding
URL size is more than 255 characters
HTTP pipelined transactions
WebDav methods
HEAD method
301/401 responses with cookie headers
301 responses with a total header length of more than 768 bytes
401 responses with a total header length of more than 384 bytes
401 responses with a chunked body
401 responses with unsupported authentication method (supported methods include: Basic, NTLM, Negotiate, Kerberos,
Digest, Oauth)
Partial HTTP header (header split) available for processing
HTTP AO Logging
The following log files are available for troubleshooting HTTP AO issues:
Debug log files: /local1/errorlog/httpao-errorlog.current (and httpao-errorlog.*)


Wed Jul 15 13:37:00 2009 :1529 :10.10.10.10 :2004 :10.10.100.100 :80 :OT :END :EXTERNAL CLIENT :(HTTP) :0 :0 :107
Wed Jul 15 13:37:00 2009 :1529 :10.10.10.10 :1880 :10.10.100.100 :80 :SODRE :END :14357 :8406 :2181 :2761 :0
Wed Jul 15 13:38:19 2009 :1533 :10.10.10.10 :2008 :10.10.100.101 :135 :OT :START :EXTERNAL CLIENT :00.14.5e.84.24.
:Other :MS-EndPointMapper :F :(TFO) (TFO) (TFO) (TFO) (TFO) :<None> :(EPM) (EPM) (EPM) :<None> :<None> :0 :120
Wed Jul 15 13:38:19 2009 :1534 :10.10.10.10 :2009 :10.10.100.101 :1025 :OT :START :EXTERNAL CLIENT :00.14.5e.84.24
:uuide3514235-4b06-11d1-ab04-00c04fc2dcd2
To set up and enable debug logging of the HTTP AO, use the following commands.
Metadata Cache Cache-Control Behavior
42

You can enable detailed logging to the disk:
You can enable debug logging for connections in the ACL:

The options for HTTP AO debugging (on 4.2.1 and later) are as follows:
WAE674# debug accelerator http ?
all
enable
bypass-list
enable
cli
enable
conditional-response
enable
debugs
connection
enable
dre-hints
enable
metadatacache
enable
prefetch
enable
redirect-response
enable
debugs
shell
enable
suppress-server-encoding enable
transaction
enable
unauthorized-response
enable
all HTTP accelerator debugs

HTTP bypass-list debugs
HTTP CLI debugs
HTTP metadatacache conditional (304) response
HTTP
HTTP
HTTP
HTTP
HTTP
connection debugs
dre-hints debugs
metadatacache debugs
prefetch debugs
metadatacache redirect (301) response
HTTP
HTTP
HTTP
HTTP
shell debugs
suppress-server-encoding debugs
transaction debugs
auth-optimization debugs bugs
You can enable debug logging for HTTP connections and then display the end of the debug error log as follows:
WAE674# debug accelerator http connection
WAE674# type-tail errorlog/httpao-errorlog.current follow
This article describes how to troubleshoot the EPM AO.
HTTP AO Logging
Guide Contents
Main Article
43

Contents
1 EPM Accelerator
Troubleshooting
2 EPM AO Logging
EPM Accelerator Troubleshooting
The End Point Mapper (EPM) accelerator optimizes MS-RPC protocols that do not use predefined TCP ports. Clients contact the
EPM service on the server (TCP port 135) to negotiate a dynamic port that is based on the application UUID. The EPM AO listens
to the client communication and creates a dynamic policy entry to match the negotiated port. EPM is required to apply MAPI
specific optimizations or to provide accounting on any MS-RPC protocol.
the Troubleshooting Application Acceleration article. The Enterprise license is required for EPM accelerator operation.
Next, verify the status that is specific to the EPM AO by using the show accelerator epm command, as shown in Figure 1. You
want to see that the EPM AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State is
Enabled but the Operational State is Shutdown, it indicates a licensing problem.
Figure 1. Verifying the EPM Accelerator Status
Contents
44
Use the show running-config command to verify that the EPM traffic policy is properly configured. You want to see adaptor
EPM for the applications or UUIDs that are configured to use the EPM AO, as follows:
WAE674# sh run | begin EPM
...skipping
map adaptor EPM 1544f5e0-613c-11d1-93df-00c04fd7bd09
name Email-and-Messaging All action pass-through
exit
map adaptor EPM ms-sql-rpc
name SQL All action optimize full
exit
map adaptor EPM mapi
name Email-and-Messaging All action optimize full accelerate mapi
exit
map adaptor EPM ms-ad-replication
name Replication All action optimize full
exit
map adaptor EPM ms-frs
name Replication All action optimize full
exit
map adaptor EPM f5cc5a18-4264-101a-8c59-08002b2f8426
name Email-and-Messaging All action pass-through
Use the show policy-engine application dynamic command to verify the dynamic policy engine match conditions as follows:
WAE674# sh policy-engine application dynamic
Dynamic Match Freelist Information:
Allocated: 32768 In Use: 3 Max In Use: 4
Dynamic Match Type/Count Information:
None
0
Clean-Up
0
Host->Host
0
Host->Local
0
Local->Host
0
Local->Any
0
Any->Host
3
Any->Local
0
Any->Any
0
Individual Dynamic Match Information:
Allocations: 380

Number:
1
Type: Any->Host (6) User Id: EPM (3)
Src: ANY:ANY Dst: 10.10.100.101:1146
Map Name: uuida4f1db00-ca47-1067-b31f-00dd010662da
Flags: TIME_LMT REPLACE FLOW_CNT
Seconds: 1200 Remaining: - NA - DM Index: 32765
Hits: 54 Flows: 39 Cookie: 0x00000000
Number:
2
Src: ANY:ANY Dst: 10.10.100.101:1040
Map Name: uuid1544f5e0-613c-11d1-93df-00c04fd7bd09
Seconds: 1200 Remaining: 1163 DM Index: 32766
<----------------<----------------<----------------<----------------<----------------<-----------------
Use the show statistics connection optimized epm command to check that the WAAS device is establishing optimized EPM
connections. Verify that "TE" or "TDLE" appears in the Accel column for EPM connections, which indicates that the EPM AO
was used, as follows:
WAE674# sh stat conn opt epm
Historical Flows:
18
17
0
1
0
28
100
ConnID
Source IP:Port
Dest IP:Port
PeerID
Accel
2048
2049
10.10.10.10:3007
10.10.10.10:3009
10.10.100.101:135
10.10.100.101:135
00:14:5e:84:24:5f
00:14:5e:84:24:5f
TE
TE
<-----Look for "E"
You can check connection statistics for closed connections by using the show statistics connection closed epm command.
You can view the EPM connection specific statistics by using the show statistics connection optimized epm detail command as
follows:
WAE674# sh stat connection optimized epm detail
Connection Id:
1885
Peer Id:
00:14:5e:84:24:5f
Connection Type:
EXTERNAL CLIENT
Start Time:
Wed Jul 15 09:50:45 2009
Source IP Address:
10.10.10.10
Source Port Number:
2465
10.10.100.101
Application Name:
Other
Classifier Name:
MS-EndPointMapper
Map Name:
basic
Directed Mode:
FALSE
Preposition Flow:
FALSE
Policy Details:
Configured:
TCP_OPTIMIZE
Derived:
TCP_OPTIMIZE
Peer:
TCP_OPTIMIZE
EPM Accelerator Troubleshooting
<-----Should see MS-EndPointMapper
46

Negotiated:
Applied:
Configured:
Derived:
Applied:
Hist:
TCP_OPTIMIZE
TCP_OPTIMIZE
<-----Should see EPM configured
EPM
EPM
EPM
None
Bytes Read:
Bytes Written:
. . .
<-----Should see EPM applied
Original
Optimized
-------------------- -------------------5220
5076
5076
5220
EPM AO Logging
The following log files are available for troubleshooting EPM AO issues:
Debug log files: /local1/errorlog/epmao-errorlog.current (and epmao-errorlog.*)
For easier debugging, first set up an ACL to restrict packets to one host.

wae(config)# transaction logs flow enable

:uuide3514235-4b06-11d1-ab04-00c04fc2dcd2 :Replication :**Map Default** :F :(DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO
(DRE,LZ,TFO) :<None> :(None) (None) (None) :<None> :<None> :0 :169
Wed Jul 15 16:53:51 2009 :1798 :10.10.10.10 :2368 :10.10.100.101 :135 :OT :END :EXTERNAL CLIENT :(EPM) :228 :212 :
Wed Jul 15 16:53:51 2009 :1799 :10.10.10.10 :2369 :10.10.100.101 :1025 :OT :END:EXTERNAL CLIENT :(None) :596 :220
Wed Jul 15 16:53:51 2009 :1799 :10.10.10.10 :2369 :10.10.100.101 :1025 :SODRE :END :596 :220 :347 :429 :0
To set up and enable debug logging of the EPM AO, use the following commands.
You can enable debug logging for connections in the ACL as follows:
The options for EPM AO debugging are as follows:

WAE674# debug accelerator epm ?
EPM AO Logging
47

all
shell
enable all EPM accelerator debugs

enable EPM shell debugs
You can enable debug logging for EPM connections and then display the end of the debug error log as follows:
WAE674# debug accelerator epm connection
WAE674# type-tail errorlog/epmao-errorlog.current follow
This article describes how to troubleshoot the MAPI AO.
Guide Contents
Main Article
Contents
1 MAPI Accelerator
2 Encrypted MAPI Acceleration
2.1 Summary
2.2 Feature Information
2.3 Troubleshooting Methodology
2.3.1 Step 1 - Verify Encryption Service Identity configuration and key retrieval success
2.3.2 Step 2 - In 5.0.3 a new diagnostic command was introduced to check some of the required settings.
2.3.3 Step 3- Manually verify the WAE settings that are not checked by the diagnostic command above.
2.4 Data Analysis
2.5 Common Problems
2.5.1 Problem 1: The Encryption service identity configured on the Core WAE does not have the correct
permissions in AD.
2.5.2 Resolution 1: Consult the configuration guide and verify the object in AD has the correct
permissions. "Replicating Directory Changes" and "Replicating Directory Changes All" must both be set
to allow.
Contents
48

2.5.3 Problem 2: There is a time skew between the Core WAE and the KDC it attempts to retrieve the
key from
2.5.4 Resolution 2: Use ntpdate on all WAEs (especially the Core) to sync the clock with the KDC. Then
point to the enterprise NTP server (prefereably the same as the KDC).
2.5.5 Problem 3: The domain you defined for your Encryption service does not match the domain your
Exchange server is in.
2.5.6 Resolution 3: If your Core WAE services multiple Exchange servers in different domains you must
configure an Encryption service Identity for each domain the Exchange servers reside in.
2.5.7 Problem 4: If WANSecure fails your connections can drop to TG
2.5.8 Resolution 4: Remove peer cert verify configuration from both WAEs and restart the encrytpion
service on the Core WAE(s).
2.5.9 Problem 5: If NTLM is used by the Outlook client the connection will be pushed down to Generic
AO.
2.5.10 Resolution 5: The customer must enable / require Kerberos authentication in their Exchange
environment. NTLM is NOT supported (as of 5.1)
3 MAPI AO Logging
MAPI Accelerator
The MAPI accelerator optimizes Microsoft Outlook Exchange e-mail traffic. Exchange uses the EMSMDB protocol, which is
layered on MS-RPC, which in turn uses either TCP or HTTP (unsupported) as the low level transport.
The MAPI AO supports Microsoft Outlook 2000 through 2007 clients for both cached and noncached mode traffic. Secure
connections that use message authentication (signing) or encryption are not accelerated by the MAPI AO. Such connections and
connections from older clients are handed off to the generic AO for TFO optimizations. Additionally, Outlook Web Access
(OWA) and Exchange-Exchange connections are not supported.
Note: Microsoft Outlook 2007 has encryption enabled by default. You must disable encryption to benefit from the MAPI
application accelerator. In Outlook, choose Tools > E-mail Accounts, choose View or Change Existing E-mail Accounts, and
then click Next. Choose the Exchange account, and then click Change. Click More Settings, and then click the Security tab.
Uncheck the Encrypt data between Microsoft Office Outlook and Microsoft Exchange Server check box, as shown in Figure
1.
Alternatively, you can disable encryption for all users of an Exchange Server by using a Group Policy.
Figure 1. Disabling Encryption in Outlook 2007
MAPI Accelerator
49
In the following cases, the MAPI AO does not handle a connection:

Encrypted connection (handed off to the generic AO)
Unsupported client (handed off to the generic AO)
Unrecoverable parsing error. All TCP connections between the client and server service are disconnected. When the client
reconnects, all connections are handed off to the generic AO.
Client attempts to establish a new association group on the connection when the WAE is overloaded.
Client establishes a connection when the WAE is overloaded and MAPI reserved connection resources are not available.
The Outlook client and server interact in a session over a group of TCP connections called an association group. Within an
association group, object accesses can span across any connection and connections are dynamically created and released as
needed. A client can have more than one association group open at the same time to different servers or the same server. (Public
folders are deployed on different servers from the mail store.)
It is essential that all MAPI connections within an association group go through the same pair of WAEs in the branch and data
center. If some connections within an association group do not go through the MAPI AO on these WAEs, the MAPI AO would
not see the transactions performed on those connections and the connections are said to "escape" the association group. For this
reason, the MAPI AO should not be deployed on serially clustered inline WAEs that form a high availability group.
The symptoms of MAPI connections that escape their WAE association group are Outlook error symptoms such as duplicate
messages or Outlook stops responding.
During a TFO overload condition, new connections for an existing association group would be passed through and escape the
MAPI AO, so the MAPI AO reserves a number of connection resources in advance to minimize the impact of an overload
condition. For more details about reserved MAPI connections and their impact on device overload, see the section "MAPI
Application Accelerator Reserved Connections Impact on Overload" in the Troubleshooting Overload Conditions article.
Verify the general AO configuration and status with the show accelerator and show license commands, as described in the
MAPI Accelerator
50

Troubleshooting Application Acceleration article. The Enterprise license is required for MAPI accelerator operation and the EPM
application accelerator must be enabled.
Next, verify the status specific to the MAPI AO by using the show accelerator mapi command, as shown in Figure 2. You want
to see that the MAPI AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State is
Figure 2. Verifying the MAPI Accelerator Status
Use the show statistics accelerator epm command to verify that the EPM AO is functional. Check that the Total Handled
Connections, Total Requests Successfully Parsed, and Total Responses Successfully Parsed counters increase when a client is
started.
Use the show running-config command to verify that the MAPI and EPM traffic policies are properly configured. You want to
see accelerate mapi for the Email-and-Messaging application action and you want to see the MS-EndPortMapper classifier and
traffic policy defined, as follows:
WAE674# sh run | include mapi
map adaptor EPM mapi
name Email-and-Messaging All action optimize full accelerate mapi
WAE674# sh run | begin MS-EndPointMapper
...skipping
classifier MS-EndPointMapper
exit
WAE674# sh run | include MS-EndPointMapper
classifier MS-EndPortMapper
name Other classifier MS-EndPortMapper action optimize DRE no compression none accelerate MS-port-mapper

Use the show policy-engine application dynamic command to verify that dynamic match rules exist, as follows:
Look for a rule with User ID: EPM and Map Name: uuida4f1db00-ca47-1067-b31f-00dd010662da.
The Flows field indicates the total number of active connections to the Exchange service.
For each MAPI client you should see a separate entry with the User ID: MAPI.
Use the show statistics connection optimized mapi command to check that the WAAS device is establishing optimized MAPI
connections. Verify that "M" appears in the Accel column for MAPI connections, which indicates that the MAPI AO was used, as
follows:
WAE674# show stat conn opt mapi
Historical Flows:
2
1
1
0
0
12
0
161
<---------- Added in 4.1.5

ConnID
342
Source IP:Port
10.56.94.101:4506
Dest IP:Port
10.10.100.100:1456
PeerID
0:1a:64:d3:2f:b8
Accel RR
TMDL 61.0%
<-----Look for "M"
Note: In version 4.1.5, the Current Reserved Flows counter was added in the output. This counter refers to the number of reserved
MAPI connection resources on the WAE that are currently unused but set aside for future MAPI connections. For more details
about reserved MAPI connections and their impact on device overload, see the section "MAPI Application Accelerator Reserved
Connections Impact on Overload" in the Troubleshooting Overload Conditions article.
If you observe connections with "TGDL" in the Accel column, these connections were pushed down to the generic AO and
optimized with transport optimizations only. If these are connections that you expected to be handled by the MAPI AO, it may be
because they are encrypted MAPI connections. To check on the number of encrypted MAPI connections that have been requested,
use the show statistics accelerator mapi command as follows:
wae# sh stat accel mapi
MAPI:
Global Statistics
Number of Synch Get Buffer Requests:
Minimum Synch Get Buffer Size (bytes):
Maximum Synch Get Buffer Size (bytes):
Average Synch Get Buffer Size (bytes):
Number of Read Stream Requests:
Minimum Read Stream Buffer Size (bytes):
Maximum Read Stream Buffer Size (bytes):
Average Read Stream Buffer Size (bytes):
Minimum Accumulated Read Ahead Data Size (bytes):
MAPI Accelerator
Thu Nov
Thu Nov
8615
8614
0
1
20
0
512
1052
31680
31680
31680
3844
19
31744
14556
0
5 19:45:19 2009
5 19:45:19 2009
52

Maximum Accumulated Read Ahead Data Size (bytes):
Average Accumulated Read Ahead Data Size (bytes):
Local Response Count:
Average Local Response Time (usec):
Remote Response Count:
Average Remote Response Time (usec):
Current 2000 Accelerated Sessions:
Secured Connections:
Lower than 2000 Sessions:
Higher than 2007 Sessions:
1172480
594385
20827
250895
70486
277036
0
1
0
1
0
0
<-----Encrypted connections
You can find the IP addresses of clients requesting encrypted MAPI connections in the syslog by searching for messages like the
following:
2009 Jan 5 13:11:54 WAE512 mapi_ao: %WAAS-MAPIAO-3-132104: (929480) Encrypted connection. Client ip: 10.36.14.82
You can view the MAPI connection statistics by using the show statistics connection optimized mapi detail command as
follows:
WAE674# show stat conn opt mapi detail
Connection Id:
1830
Peer Id:
00:14:5e:84:24:5f
Connection Type:
EXTERNAL CLIENT
Start Time:
Thu Jun 25 06:32:27 2009
Source IP Address:
10.10.10.10
Source Port Number:
3774
10.10.100.101
Application Name:
Email-and-Messaging
<-----Should
Classifier Name:
**Map Default**
Map Name:
uuida4f1db00-ca47-1067-b31f-00dd010662da
<-----Should
Directed Mode:
FALSE
Preposition Flow:
FALSE
Policy Details:
Configured:
Derived:
Peer:
Negotiated:
Applied:
Configured:
MAPI
<-----Should
Derived:
MAPI
Applied:
MAPI
<-----Should
Hist:
None
Original
Optimized
-------------------- -------------------Bytes Read:
4612
1973
Bytes Written:
4086
2096
. . .
see Email-and-Messaging
see this UUID
see MAPI configured

see MAPI applied
Local and remote response counts and average response times are shown in this output:
. . .
MAPI : 1830
06:32:27 2009
Total Bytes Read:
Number of Synch Get Buffer Requests:
MAPI Accelerator
Thu Jun 25
46123985
40864046
0
53

Minimum Synch Get Buffer Size (bytes):
Maximum Synch Get Buffer Size (bytes):
Average Synch Get Buffer Size (bytes):
Number of Read Stream Requests:
Minimum Read Stream Buffer Size (bytes):
Maximum Read Stream Buffer Size (bytes):
Average Read Stream Buffer Size (bytes):
Minimum Accumulated Read Ahead Data Size (bytes):
Maximum Accumulated Read Ahead Data Size (bytes):
Average Accumulated Read Ahead Data Size (bytes):
Local Response Count:
Average Local Response Time (usec):
Remote Response Count:
Average Remote Response Time (usec):
. . .
0
0
0
0
0
0
0
0
0
0
0
0
19
89005
<---------<---------<---------<----------
Encrypted MAPI Acceleration

Summary
As of WAAS 5.0.1 the MAPI accelerator can now accelerate encrypted MAPI traffic. This feature will be enabled by default in
the 5.0.3 release. However, in order successfully accelerate encrypted MAPI traffic there are number of requirements in both the
WAAS and Microsoft AD environment. This guide will help you verify and troubleshoot eMAPI functionality.
Feature Information
eMAPI will be enabled by default in 5.0.3 and will require the following to successfully accelerate encrypted traffic.
1) CMS secure store must be initialized and open on all Core WAEs
2) The WAEs must be able to resolve the FQDN of the Exchange server(s) and Kerberos KDC (Active Directory Controller)
3) The WAE's clocks must be in sync with the KDC
4) SSL acclerator, WAN Secure, and eMAPI must be enabled on all WAEs in the path from Outlook to Exchange
5) The WAEs in the path must have the correct policy-map configuration
6) The Core WAE(s) must have one or more Encrypted Services Domain Identies configured (User or Machine account)
7) If a machine account is used this WAE must be joined to the AD domain.
8) Then with either the Machine or User account use case, those objects in Active directory need to be given specific permissions.
"Replicating Directory Changes" and "Replicating Directory Changes All" must both be set to allow.
The recommended way to do this is via a Universal Security group (e.g. assign the permissions to the group and then add the
WAAS devices and/or usernames specified in the Encryption service to this group). See the attached guide for screenshots of AD
configuration and WAAS CM GUI.
Troubleshooting Methodology
Encrypted MAPI Acceleration
54

Step 1 - Verify Encryption Service Identity configuration and key retrieval success
While the diagnostics command (step 2 below) verifies the existence of an Encryption service it does not verify whether key
retrieval will be successful. Hence we do not know by just running that diagnostic command if the proper permissions were given
to the object in Active Directory (either Machine or User account).
Summary of what needs to be done to configure and verify Encryption service will succeed key retrieval
User account :
1. create AD user
2. create AD group and set "Replicating Directory Changes" and "Replicating Directory Changes All" to ALLOW
3. add the user to the group created
4. define user account domain identity in encryption services
5. run get key diagnostic cli
windows-domain diagnostics encryption-service get-key <exchange server FQDN> <domain name>
Note that you should use the actual/real exchange server name configured on the server and not a NLB/VIP type FQDN which
might resolves to multiple exchange servers.
6. if key retrieval worked - done
Example of success:
pdi-7541-dc#windows-domain diagnostics encryption-service get-key pdidc-exchange1.pdidc.cisco.com pdidc.cisco.com
SPN pdidc-exchange1.pdidc.cisco.com, Domain name: pdidc.cisco.com
Key retrieval is in progress.
pdi-7541-dc#windows-domain diagnostics encryption-service get-key pdidc-exchange1.pdidc.cisco.com pdidc.cisco.com
SPN pdidc-exchange1.pdidc.cisco.com, Domain name: pdidc.cisco.com
Key for pdidc-exchange1.pdidc.cisco.com resides in memory key cache
Machine account
1. join core WAE device(s) to AD domain
2.create AD group and set "Replicating Directory Changes" and "Replicating Directory Changes All" to ALLOW
3. add machine account(s) to group created
4. configure encryption services to use machine account
5. Give sometime to get the Group Policy to be applied to the joined machine or force the application of group policy from the
AD. gpupdate /force.
Step 1 - Verify Encryption Service Identity configuration and key retrieval success
55

6. run get key diagnostic cli
windows-domain diagnostics encryption-service get-key <exchange server FQDN> <domain name>
Note that you should use the actual/real exchange server name configured on the server and not a NLB/VIP type FQDN which
might resolves to multiple exchange servers.
7. if key retrieval worked - done
For more details and screen shots on Encrytpion service and AD config see the attached guide.
Step 2 - In 5.0.3 a new diagnostic command was introduced to check some of the required settings.
?accelerator mapi verify encryption settings
1.CLI does various validity checks. It output is summary of ability to accelerate encrypted MAPI traffic as edge or core.
2.Checks the various components? status/config attributes for Encryption Service to work properly.
3.When a configuration issue is found it will output what is missing and the CLI or actions to fix it.
4.It give the summary out as Edge device and Core device. Device which can be both edge and core should have EMAPI
operational for both edge and core.
Below is a sample output from an incorrectly configured WAE:
Core#accelerator mapi verify encryption-settings
[EDGE:]
Verifying Mapi Accelerator State
-------------------------------Status: FAILED
Accelerator
Config State
Operational State
-------------------------------------mapi
Disabled
Shutdown
>>Mapi Accelerator should be Enabled
>>Mapi Accelerator should be in Running state
Verifying SSL Accelerator State
------------------------------Status: FAILED
>>Accelerator
Config State
Operational State
-------------------------------------ssl
Disabled
Shutdown
>>SSL Accelerator should be Enabled
>>SSL Accelerator should be in Running state
Verifying Wan-secure State
-------------------------Status: FAILED
>>Accelerator
Config State
Operational State
-------------------------------------wan-secure
Disabled
Shutdown
>>Wan-secure should be Enabled
>>Wan-secure should be in Running state
Verifying Mapi Wan-secure mode Setting
56

--------------------------------Status: FAILED
Accelerator Config Item
Mode
-------------------------WanSecure Mode
User
>>Mapi wan-secure setting should be auto/always
Verifying NTP State
-------------------Status: FAILED
>>NTP status should be enabled and configured
Value
-----Not Applicable
Summary [EDGE]:
===============
Device has to be properly configured for one or more components
[CORE:]
Verifying encryption-service State
---------------------------------Status: FAILED
Service
Config State
---------------------Encryption-service
Disabled
>>Encryption Service should be Enabled
>>Encryption Service status should be in
Operational State
----------------Shutdown
'Running' state
Verifying Encryption-service Identity Settings

---------------------------------------------Status: FAILED
>>No active Encryption-service Identity is configured.
>>Please configure an active Windows Domain Encryption Service Identity.
Summary [CORE]: Applicable only on CORE WAEs
============================================
Device has to be properly configured for one or more components
Below is the output from a Core WAE that is configured correctly:

Core#acc mapi verify encryption-settings [EDGE:]
Verifying Mapi Accelerator State
-------------------------------Status: OK
Verifying SSL Accelerator State
------------------------------Status: OK
Verifying Wan-secure State
-------------------------Status: OK
Verifying Mapi encryption Settings
---------------------------------Status: OK
Verifying Mapi Wan-secure mode Setting
--------------------------------Status: OK
Verifying NTP State
-------------------Status: OK
Summary [EDGE]:
===============
Device has proper configuration to accelerate encrypted traffic
[CORE:]
57

Verifying encryption-service State
---------------------------------Status: OK
Verifying Encryption-service Identity Settings
---------------------------------------------Status: OK
Summary [CORE]: Applicable only on CORE WAEs
============================================
Device has proper configuration to accelerate encrypted traffic
Step 3- Manually verify the WAE settings that are not checked by the diagnostic command above.
1) The above command while it chekcs for the existence of NTP configured, it does not actually verify the times are in sync
between the WAE and KDC. It is very important the times are in sync between the Core and KDC for key retrieval to be
successful.
If the manual check reveals they are out of sync a simple way to force the clock of the WAE to be in sync would be the ntpdate
command (ntpdate <KDC ip>). Then point the WAEs to the enterprise NTP server.
2) Verify dnslookup succeeds on all WAEs for the Exchange servers' FQDN and the KDCs' FQDN
3) Verify the class-map and policy-map is configured correctly on all WAEs in the path.
pdi-7541-dc#sh class-map type waas MAPI

Class-map type waas match-any MAPI
Match tcp destination epm mapi (0 flow-matches)
pdi-7541-dc#show policy-map type waas Policy-map type waas
WAAS-GLOBAL ( 6084690 total)
Class MAPI ( 0 flow-matches)
optimize full accelerate mapi application Email-and-Messaging
4) Verify the CMS secure store is open and initialized on all WAEs "show cms secure store"
Data Analysis
Besides analyzing the output of the diagnostic command and the manual show commands you may need to review the sysreport.
Specifically you'll want to review the mapiao-errorlog, sr-errorlog (core WAE only), and wsao-errorlog files.
Step 3- Manually verify the WAE settings that are not checked by the diagnostic command above.
58

There will be hints in each log depending on the scenario which will lead you to the reason connections drop to Generic AO.
As a reference here is sample output showing various working components
This output is from sr-errorlog and shows validation of Machine Account Encryption Service Identity
Note: This only confirms the Core WAE joined the domain and the machine account exists.
07/03/2012 19:12:07.278(Local)(6249 1.5) NTCE
07/03/2012 19:12:07.279(Local)(6249 1.5) NTCE
07/03/2012 19:12:07.279(Local)(6249 1.5) NTCE
07/03/2012 19:12:07.279(Local)(6249 1.5) NTCE
07/03/2012 19:12:07.279(Local)(6249 1.5) NTCE
07/03/2012 19:12:07.330(Local)(6249 1.5) NTCE
07/03/2012 19:12:07.330(Local)(6249 1.5) NTCE
ID_TAG :MacchineAcctWAAS
Name : pdi-7541-dc
Domain : PDIDC.CISCO.COM
Realm : PDIDC.CISCO.COM
CLI_GUID :
SITE_GUID :
CONF_GUID :
Status:ENABLED
Black_Listed:NO
AUTH_STATUS: SUCCESS
ACCT_TYPE:Machine [SRIdentityObject.cpp:85]
07/03/2012 19:12:07.331(Local)(6249 1.5) NTCE
07/03/2012 19:12:07.347(Local)(6249 1.5) NTCE
(278902)
(279018)
(279282)
(279306)
(279321)
(330581)
(330599)
Adding Identity MacchineAcctWAAS to map active list in SRMa

Adding identity(MacchineAcctWAAS) to Map [SRDiIdMgr.cpp:562
Activate Id: MacchineAcctWAAS [SRMain.cpp:260]
Identity MacchineAcctWAAS found in the Map [SRDiIdMgr.cpp:7
Authentication for ID: MacchineAcctWAAS [SRDiIdMgr.cpp:398]
Authentication success, tkt validity starttime 1341342727 e
(331685) DN Info found for domain PDIDC.CISCO.COM [SRIdentityObject.

(347680) Import cred successfull for pn: pdi-7541-dc@PDIDC.CISCO.COM
This output is from the Core sr-errorlog again and shows successful key retrieval from KDC.
10/23/2012 15:46:55.673(Local)(3780 1.2) NTCE (673766)
10/23/2012 15:46:55.673(Local)(3780 1.2) NTCE (673811)
Key retrieval is in Progress [SRServer.cpp:322]
10/23/2012 15:46:55.673(Local)(3780 0.0) NTCE (673818)
10/23/2012 15:46:55.673(Local)(3780 1.2) NTCE (673827)
10/23/2012 15:46:55.673(Local)(3780 1.2) NTCE (673834)
[SRDataServer.cpp:444]
10/23/2012 15:46:55.673(Local)(3780 0.0) NTCE (673922)
10/23/2012 15:46:55.673(Local)(3780 0.0) NTCE (673937)
10/23/2012 15:46:55.673(Local)(3780 0.0) NTCE (673950)
10/23/2012 15:46:55.674(Local)(3780 0.0) NTCE (674011)
PDI-7541-DC@PDIDC.CISCO.COM [GssCli.cpp:51]
10/23/2012 15:46:55.674(Local)(3780 0.0) NTCE (674020)
10/23/2012 15:46:55.674(Local)(3780 1.3) NTCE (674421)
10/23/2012 15:46:55.676(Local)(3780 1.3) NTCE (676280)
10/23/2012 15:46:55.676(Local)(3780 0.1) NTCE (676415)
10/23/2012 15:46:55.676(Local)(3780 0.0) NTCE (676607)
10/23/2012 15:46:55.677(Local)(3780 0.0) NTCE (677736)
10/23/2012 15:46:55.678(Local)(3780 0.1) NTCE (678001)
10/23/2012 15:46:55.679(Local)(3780 0.1) NTCE (679500)
10/23/2012 15:46:55.680(Local)(3780 0.1) NTCE (680011)
10/23/2012 15:46:55.680(Local)(3780 0.1) NTCE (680030)
10/23/2012 15:46:55.685(Local)(3780 0.1) NTCE (685502)
[SRKeyRetriever.cpp:269]
10/23/2012 15:46:55.685(Local)(3780 0.1) NTCE (685583)
[SRKeyMgr.cpp:296]
10/23/2012 15:46:55.685(Local)(3780 0.1) NTCE (685594)
Data Analysis
Key Not Found in cache, initiating retrieval for spn:exchan

Queued InitiateKeyRetrieval task [SRServer.cpp:264]10/23/20
Initiating key retrieval [SRServer.cpp:271]

initiating key retrieval in progress [SRDataServer.cpp:441]
Sending ack for result 2, item name /cfg/gl/sr/sr_get_key/p
Match found for DN: pdidc.cisco.com is ID:MacchineAcctWAAS

Identity MacchineAcctWAAS found in the Map [SRDiIdMgr.cpp:7
DN Info found for domain pdidc.cisco.com [SRIdentityObject.
DRS_SPN: E3514235-4B06-11D1-AB04-00C04FC2DCD2/e4c83c51-0b59
CREATED srkr obj(0x50aa00) for spn (exchangeMDB/pdidc-excha

Import cred successfull for pn: PDI-7541-DC@PDIDC.CISCO.COM
session(0x50aa00) Complete TGT stage of GSS Successful, Ini
SRKR: Success in posting connect to service <ip:0e:6e:03:a3
Connected to server. [IoOperation.cpp:389]
SRKR: Success in posting connect to service <ip:0e:6e:03:a3
Connected to server. [IoOperation.cpp:389]
Cleaning up credential cache for PDI-7541-DC@PDIDC.CISCO.CO
Parsing DRSBIND Response [AppApiDrsBind.cpp:222]
DRSBind Success, Status:00000000 [AppApiDrsBind.cpp:359]
session(0x50aa00) Successful in Key Retrieval from AD for S
Send Key response to the Client for spn: exchangeMDB/pdidc-
Deleting spn: exchangeMDB/pdidc-exchange1.pdidc.cisco.com e
59

This output is from the mapiao-errorlog file on the Edge WAE for a successful eMAPI connection
'''10/23/2012 17:56:23.080(Local)(8311 0.1) NTCE (80175) (fl=2433) Edge TCP connection initiated (-1409268656), Co
Flavor: 0 [EdgeTcpConnectionDceRpcLayer.cpp:43]
10/23/2012 17:56:23.080(Local)(8311 0.1) NTCE (80199) Edge TCP connection initiated (-1409268656), Conn: [14.110.3
[EdgeTcpConnectionDceRpcLayer.cpp:48]
10/23/2012 17:56:23.108(Local)(8311 0.0) NTCE (108825) (fl=2433) Bind Request from client with AGID 0x0, callId 2,
AuthType: SPNEGO AuthCtxId: 0 WsPlumb:1
[EdgeTcpConnectionDceRpcLayer.cpp:1277]'''
10/23/2012 17:56:23.109(Local)(8311 0.0) NTCE (109935) CheckAndDoAoshReplumbing perform replumbing wsPlumbState 1
10/23/2012 17:56:23.109(Local)(8311 0.0) NTCE (109949) (fl=2433) AOSH Replumbing was performed returned Status 0 [
10/23/2012 17:56:23.109(Local)(8311 0.0) NTCE (109956) CheckAndPlumb WanSecure(14) ret:= [1,0] WsPlumb:4 fd[client
10/23/2012 17:56:23.312(Local)(8311 0.1) NTCE (312687) (fl=2433) Connection multiplexing enabled by server on the
10/23/2012 17:56:23.312(Local)(8311 0.1) NTCE (312700) (fl=2433) Header signing enabled by server on the connectio
10/23/2012 17:56:23.312(Local)(8311 0.1) NTCE (312719) (fl=2433) OnNewConnection - Client IP 14.110.3.117 (0xe6e03
nAssociationGroup=0x11de4,conn_fd=26,
bWasConnectionFromReservedPool=0, bIsNewMapiSession=1 [ConnectionReservationManager.cpp:255]
'''10/23/2012 17:56:23.366(Local)(8311 0.1) NTCE (366789) (fl=2433) Received security context from core with auth
10/23/2012 17:56:23.367(Local)(8311 0.1) NTCE (367157) (fl=2433) Security Layer moved to ESTB state [FlowSecurityL
10/23/2012 17:56:23.368(Local)(8311 0.1) NTCE (368029) (fl=2433) Informational:: Send APC set to WS: asking for Ci
10/23/2012 17:56:23.368(Local)(8311 0.1) NTCE (368041) (fl=2433) Sec-Params [CtxId, AL, AT, ACT, DCT, [Hs, ConnMpl
[FlowIOBuffers.cpp:477]
10/23/2012 17:56:23.369(Local)(8311 0.0) NTCE (369128) (fl=2433) CEdgeTcpConnectionEmsMdbLayer::ConnectRequestComm
Product Minor:0, Build Major:6117,
Build Minor:5001 Client ip 14.110.3.117 Client port 58352 Dest ip 14.110.3.99 Dest port 27744 [EdgeTcpConnectionEm
10/23/2012 17:56:23.868(Local)(8311 0.1) ERRO (868390) (fl=2433) ContextHandle.IsNull() [EdgeTcpConnectionEmsMdbLa
10/23/2012 17:56:23.890(Local)(8311 0.0) NTCE (890891) (fl=2433) CEdgeTcpConnectionEmsMdbLayer::ConnectRequestComm
Product Minor:0, Build Major:6117,
Build Minor:5001 Client ip 14.110.3.117 Client port 58352 Dest ip 14.110.3.99 Dest port 27744 [EdgeTcpConnectionEm
Here is the corresponding Core WAE output from mapiao-errorlog for the same TCP conneciton
'''10/23/2012 17:56:54.092(Local)(6408 0.0) NTCE (92814) (fl=21) Core TCP connection initiated (11892640), Conn: [
lavor: 0 [CoreTcpConnectionDceRpcLayer.cpp:99]
10/23/2012 17:56:54.092(Local)(6408 0.0) NTCE (92832) Core TCP connection initiated (11892640), Conn: [14.110.3.11
[CoreTcpConnectionDceRpcLayer.cpp:104]'''
10/23/2012 17:56:54.175(Local)(6408 0.0) NTCE (175035) SrlibCache Cache eviction starting: static void srlib::CSrl
id*, aosh_work*) [SrlibCache.cpp:453]
10/23/2012 17:56:54.175(Local)(6408 0.0) NTCE (175068) last_cleanup_time (1344411860), evict_in_progress(1) handle
cpp:464]
10/23/2012 17:56:54.175(Local)(6408 0.0) NTCE (175121) SendNextCmd isDuringSend 0, WriteQueue sz 1, isDuringclose
10/23/2012 17:56:54.175(Local)(6408 0.0) NTCE (175132) SendNextCmd: Sending request: exchangeMDB/PDIDC-EXCHANGE1.p
[bClose 0] [SrlibClientTransport.cpp:168]
10/23/2012 17:56:54.185(Local)(6408 0.1) NTCE (185576) OnReadComplete len 4 status 0 isDuringRead 1, isDuringHeade
cpp:127]
10/23/2012 17:56:54.185(Local)(6408 0.1) NTCE (185587) Parse header, msg body len 152 [SrlibTransport.cpp:111]
10/23/2012 17:56:54.185(Local)(6408 0.1) NTCE (185592) ReadNextMsg isDuringRead 0, isDuringHeaderRead 1, isDuringc
10/23/2012 17:56:54.185(Local)(6408 0.1) NTCE (185623) OnReadComplete len 148 status 0 isDuringRead 1, isDuringHea
t.cpp:127]
'''10/23/2012 17:56:54.185(Local)(6408 0.1) NTCE (185688) Insert new KrbKey: exchangeMDB/PDIDC-EXCHANGE1.pdidc.cis
rlibCache.cpp:735]
'''10/23/2012 17:56:54.185(Local)(6408 0.1) NTCE (185747) ReadNextMsg isDuringRead 0, isDuringHeaderRead 0, isDuri
'''10/23/2012 17:56:54.261(Local)(6408 0.1) NTCE (261575) (fl=21) Successfully created memory keytab with name: ME
.com0nxrPblND [GssServer.cpp:468]
10/23/2012 17:56:54.261(Local)(6408 0.1) NTCE (261613) (fl=21) Successfully added entry in memory keytab. [GssServ
10/23/2012 17:56:54.261(Local)(6408 0.1) NTCE (261858) (fl=21) Successfully acquired credentials. [GssServer.cpp:1
Data Analysis
60
Common Problems
Below are some common reasons which results in eMAPI connection Hand-off to Generic AO (TG).
Problem 1: The Encryption service identity configured on the Core WAE does not have the correct
permissions in AD.
Output from sr-errolog on Core WAE
09/25/2012 18:47:54.147(Local)(9063 0.1) ERRO (147570) session(0x517fa0) Failed to Retrieve Key from AD for SPN:ex
'''09/25/2012 18:47:54.147(Local)(9063 0.1) ERRO (147592) Key retrieval failed with Status 16 [SRKeyMgr.cpp:157]
''''''09/25/2012 18:47:54.147(Local)(9063 0.1) ERRO (147623) Identity "WAASMacAct" has been blacklisted [SRDiIdMgr
''''''09/25/2012 18:47:54.147(Local)(9063 0.1) ERRO (147631) Key retrieval failed due to permission issue [SRKeyMg
'''09/25/2012 18:47:54.147(Local)(9063 0.1) ERRO (147636) Identity: WAASMacAct will be black listed. [SRKeyMgr.cpp
09/25/2012 18:47:54.147(Local)(9063 0.1) NTCE (147657) Calling KrbKeyResponse key handler in srlib [SRServer.cpp:
09/25/2012 18:47:54.147(Local)(9063 0.1) NTCE (147722) Queued send reponse buffer to client task [SrlibServerTrans
09/25/2012 18:47:54.147(Local)(9063 0.1) NTCE (147730) KrbKeyResponse, sent to client session object [SrlibServer.
09/25/2012 18:47:54.147(Local)(9063 0.0) NTCE (147733) SendNextCmd isDuringSend 0, WriteQueue size 1 isDuringClose
09/25/2012 18:47:54.147(Local)(9063 0.1) NTCE (147740) Send Key response to the Client
Resolution 1: Consult the configuration guide and verify the object in AD has the correct permissions.
"Replicating Directory Changes" and "Replicating Directory Changes All" must both be set to allow.
http://www.cisco.com/en/US/docs/app_ntwk_services/waas/waas/v511/configuration/guide/policy.html#wp1256547
Problem 2: There is a time skew between the Core WAE and the KDC it attempts to retrieve the key from
10/23/2012 01:31:33.507(Local)(1832 0.1) NTCE (507836) Initiating key retrieval [SRServer.cpp:271]

10/23/2012 01:31:33.507(Local)(1832 0.1) NTCE (507878) Match found for DN: pdidc.cisco.com is ID:MacchineAcctWAAS
10/23/2012 01:31:33.507(Local)(1832 0.1) NTCE (507888) Identity MacchineAcctWAAS found in the Map [SRDiIdMgr.cpp:7
10/23/2012 01:31:33.507(Local)(1832 0.1) NTCE (507901) DN Info found for domain pdidc.cisco.com [SRIdentityObject.
10/23/2012 01:31:33.507(Local)(1832 0.1) NTCE (507923) DRS_SPN: E3514235-4B06-11D1-AB04-00C04FC2DCD2/e4c83c51-0b59
PDI-7541-DC@PDIDC.CISCO.COM [GssCli.cpp:51]
10/23/2012 01:31:33.507(Local)(1832 0.1) NTCE (507933) CREATED srkr obj(0x2aaaac0008c0) for spn (exchangeMDB/pdidc
10/23/2012 01:31:33.508(Local)(1832 1.6) NTCE (508252) Import cred successfull for pn: PDI-7541-DC@PDIDC.CISCO.COM
10/23/2012 01:31:33.511(Local)(1832 1.6) ERRO (511151) CreateSecurityContext: gss_init_sec_context failed [majorSt
'''10/23/2012 01:31:33.511(Local)(1832 1.6) ERRO (511170) GSS_MAJOR ERROR:851968 msg_cnt:0, Miscellaneous failure
10/23/2012 01:31:33.511(Local)(1832 1.6) ERRO (511177) GSS_MINOR ERROR:2529624064 msg_cnt:0, Clock skew too great
10/23/2012 01:31:33.511(Local)(1832 1.6) ERRO (511182) gsskrb5_get_subkey failed: 851968,22, [GssCli.cpp:198]
10/23/2012 01:31:33.511(Local)(1832 1.6) ERRO (511188) session(0x2aaaac0008c0) Error: Invalid security ctx state,
[SRKeyRetriever.cpp:386]
10/23/2012 01:31:33.511(Local)(1832 1.6) ERRO (511193) session(0x2aaaac0008c0) Failed to Retrieve Key from AD for
[SRKeyRetriever.cpp:267]'''
10/23/2012 01:31:33.511(Local)(1832 0.0) ERRO (511213) Key retrieval failed with Status 1 [SRKeyMgr.cpp:157]

Resolution 2: Use ntpdate on all WAEs (especially the Core) to sync the clock with the KDC. Then point to the
enterprise NTP server (prefereably the same as the KDC).
Problem 3: The domain you defined for your Encryption service does not match the domain your Exchange
server is in.
10/23/2012
10/23/2012
10/23/2012
10/23/2012
10/23/2012
10/23/2012
10/23/2012
18:41:21.918(Local)(3780
18:41:21.918(Local)(3780
18:41:21.918(Local)(3780
18:41:21.918(Local)(3780
18:41:21.918(Local)(3780
18:41:21.918(Local)(3780
18:41:21.918(Local)(3780
1.5)
1.5)
0.0)
1.5)
0.0)
0.0)
0.0)
NTCE
NTCE
NTCE
NTCE
ERRO
NTCE
NTCE
(918788)
(918793)
(918790)
(918798)
(918813)
(918821)
(918832)
Key retrieval is in Progress [SRServer.cpp:322]

initiating key retrieval in progress [SRDataServer.cpp:441]
Initiating key retrieval [SRServer.cpp:271]
Sending ack for result 2, item name /cfg/gl/sr/sr_get_key/p
Failed to find Identity match for domain cisco.com [SRDiIdM
Failed to find identity match for domain [SRKeyMgr.cpp:120]
Send Key response to the Client for spn: exchangeMDB/pdidc-
Resolution 3: If your Core WAE services multiple Exchange servers in different domains you must configure
an Encryption service Identity for each domain the Exchange servers reside in.
Note, there is NO support for sub-domain include at this time. So if you have myexchange.sub-domain.domain.com , the
Encryption service Identity must be in sub-domain.domain.com; it CAN NOT be in the parent domain.
Problem 4: If WANSecure fails your connections can drop to TG

eMAPI connections can be handed over to Generic AO because WAN secure plumb failed. WAN Secure plumb failed because
cert verify failed. Peer cert verify will failed because the default self-signed peer cert is being used or the cert has legitimately
failed OCSP check.
Core WAE settings

crypto pki global-settings
ocsp url http://pdidc.cisco.com/ocsp
revocation-check ocsp-cert-url
exit
!
crypto ssl services host-service peering
peer-cert-verify
exit
!
WAN Secure:
Accelerator Config Item
-----------------------
Mode
----
Value
------
Resolution 2: Use ntpdate on all WAEs (especially the Core) to sync the clock with the KDC. Then point to the enterprise
62 NTP

SSL AO
Secure store
Peer SSL version
Peer cipher list
Peer cert
Peer cert verify
User
User
User
User
User
User
enabled
enabled
default
default
default
enabled
This will result in the following mapiao-errorlog and wsao-errorlog entries:

The hint here is the first highlighted line "disconnected more than four consecutive times"
Mapiao-errorlog on client side WAE:
'''10/08/2012 20:02:15.025(Local)(24333 0.0) NTCE (25621) (fl=267542) Client 10.16.1.201 disconnected more than fo
[EdgeTcpConnectionDceRpcLayer.cpp:1443]
'''10/08/2012 20:02:15.025(Local)(24333 0.0) NTCE (25634) (fl=267542) CEdgeIOBuffers:: StartHandOverProcessSingleC
[EdgeIOBuffers.cpp:826]
10/08/2012 20:02:15.025(Local)(24333 0.0) NTCE (25644) (fl=267542) CEdgeIOBuffers::CheckSendHandOverRequestToCoreA
fragment of call id 0, current call id is 2 [EdgeIOBuffers.cpp:324]
10/08/2012 20:02:15.048(Local)(24333 0.1) NTCE (48753) (fl=267542) Connection multiplexing enabled by server on th
10/08/2012 20:02:15.048(Local)(24333 0.1) NTCE (48771) (fl=267542) Header signing enabled by server on the connect
10/08/2012 20:02:15.048(Local)(24333 0.1) NTCE (48779) (fl=267542) CEdgeIOBuffers:: StartHandOverProcessSingleConn
Wsao-errorlog on client side WAE:
'''10/08/2012 20:04:34.430(Local)(5939 4.0) ERRO (430001) certificate verification failed 'self signed certificate
'''10/08/2012 20:04:34.430(Local)(5939 4.0) ERRO (430047) ssl_read failed: 'SSL_ERROR_SSL' [open_ssl.cpp:1217]
10/08/2012 20:04:34.430(Local)(5939 4.0) ERRO (430055) openssl errors: error:14090086: SSL routines: SSL3_GET_SERV
[open_ssl.cpp:1220]
Resolution 4: Remove peer cert verify configuration from both WAEs and restart the encrytpion service on the
Core WAE(s).
pdi-7541-dc(config)#crypto ssl services host-service peering
pdi-7541-dc(config-ssl-peering)#no peer-cert-verify
pdi-7541-dc(config)#no windows-domain encryption-service enable
pdi-7541-dc(config)#windows-domain encryption-service enable
Problem 5: If NTLM is used by the Outlook client the connection will be pushed down to Generic AO.
You will see the following in the mapiao-errorlog on the client side WAE:
'''waas-edge#find-patter match ntlm mapiao-errorlog.current

'''
09/21/2012 20:30:32.154(Local)(8930 0.1) NTCE (154827) (fl=83271) Bind Request from client with AGID 0x0, callId 1
PRIVACY '''AuthType:NTLM '''AuthCtxId: 153817840 WsPlumb: 2 [EdgeTcpConnectionDceRpcLayer.cpp:1277]
09/21/2012 20:30:32.154(Local)(8930 0.1) NTCE (154861) (fl=83271) '''Unsupported''' '''Auth Type :NTLM''' [EdgeTcp
(8930 0.0) NTCE (157628) (fl=83283) Bind Request from client with AGID 0x0, callId 2, to dest-ip 172.21. 12.96, Au
WsPlumb: 2 [EdgeTcpConnectionDceRpcLayer.cpp:1277]
Problem 4: If WANSecure fails your connections can drop to TG
63

Resolution 5: The customer must enable / require Kerberos authentication in their Exchange environment.
NTLM is NOT supported (as of 5.1)
Be aware there is a Microsoft tech brief that calls out a fall back to NTLM when a CAS is used.
The scenario where Kerberos does not function is specific to Exchange 2010, and is in the following scenario:
Multiple Exchange Client Access Servers (CAS) in an organization/domain.
These CAS servers are clustered together using any method - using Microsoft's built-in Client-array function, or a 3rd party load
balancer.
In the scenario above, Kerberos does not work - and clients will fall-back to NTLM by defult. I believe this is due to the fact that
clients have to AUTH to the CAS server vs. the Mailbox server, as they did in previous Exchange releases.
In Exchange 2010 RTM, there is no fix for this! Kerberos in the above scenario will never function pre-Exchange 2010-SP1.
In SP1, Kerberos can be enabled in these environments, but it's a manual process. See the article here:
http://technet.microsoft.com/en-us/library/ff808313.aspx
MAPI AO Logging
The following log files are available for troubleshooting MAPI AO issues:
Debug log files: /local1/errorlog/mapiao-errorlog.current (and mapiao-errorlog.*)

Wed Jul 15 19:12:35 2009 :2289 :10.10.10.10 :3740 :10.10.100.101
Wed Jul 15 19:12:35 2009 :2289 :10.10.10.10 :3740 :10.10.100.101
Wed Jul 15 19:12:35 2009 :2290 :10.10.10.10 :3738 :10.10.100.101
Wed Jul 15 19:12:35 2009 :2290 :10.10.10.10 :3738 :10.10.100.101
Wed Jul 15 19:12:35 2009 :2284 :10.10.10.10 :3739 :10.10.100.101
:1146
:1146
:1146
:1146
:1146
:OT :END :EXTERNAL CLIENT :(MAPI) :822 :634

:SODRE :END :730 :605 :556 :706 :0
:SODRE :END :4550 :15854 :6436 :2006 :0
To set up and enable debug logging of the MAPI AO, use the following commands.

The options for MAPI AO debugging are as follows:

WAE674# debug accelerator mapi ?
all enable all MAPI accelerator debugs
Common-flow enable MAPI Common flow debugs
DCERPC-layer enable MAPI DCERPC-layer flow debugs
EMSMDB-layer enable MAPI EMSMDB-layer flow debugs
IO enable MAPI IO debugs
ROP-layer enable MAPI ROP-layer debugs
ROP-parser enable MAPI ROP-parser debugs
RPC-parser enable MAPI RPC-parser debugs
shell enable MAPI shell debugs
Transport enable MAPI transport debugs
Utilities enable MAPI utilities debugs
You can enable debug logging for MAPI connections and then display the end of the debug error log as follows:
WAE674# debug accelerator mapi Common-flow
WAE674# type-tail errorlog/mapiao-errorlog.current follow
This article describes how to troubleshoot the NFS AO.
MAPI AO Logging
Guide Contents
Main Article
65

Contents
1 NFS Accelerator
Troubleshooting
2 NFS AO Logging
NFS Accelerator Troubleshooting
The NFS accelerator optimizes NFSv3 traffic. Other NFS versions are not optimized by the NFS AO.
the Troubleshooting Application Acceleration article. The Enterprise license is required for NFS accelerator operation.
Next, verify the status specific to the NFS AO by using the show accelerator nfs command, as shown in Figure 1. You want to
see that the NFS AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State is
Figure 1. Verifying the NFS Accelerator Status
Use the show running-config command to verify that the NFS traffic policy is properly configured. You want to see accelerate
nfs for the File-System application classifier NFS action and you want to see appropriate match conditions listed for the NFS
classifier, as follows:
WAE674# sh run | include NFS
Contents
66

name File-System classifier NFS action optimize full accelerate nfs
WAE674# sh run | begin NFS
...skipping
classifier NFS
exit
<-------------
<-------------
Use the show statistics connection optimized nfs command to check that the WAAS device is establishing optimized NFS
connections. Verify that "N" appears in the Accel column for NFS connections, which indicates that the NFS AO was used.
WAE674# sh stat conn opt nfs
C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO,
ConnID
582
Local IP:Port
10.56.94.101:33606
Remote IP:Port
10.56.94.80:2049
PeerID
0:1a:64:d3:2f:b8
Accelerator
NTDL
<-----Look for "N"
Use the show statistics accelerator nfs command to verify the following:
The NFS traffic is NFSv3. Look at the Total RPC Calls per NFS Version field. The output of that field is an array of 5
values, and you want to see mostly NFSv3 traffic, which is reported in the 4th counter. High numbers in other array
positions signify other NFS versions.
NFS traffic is not encrypted. Look at the Total RPC Calls per Authentication Flavor field. The output of that field is an
array of 4 values, and you want to see mostly unencrypted traffic, which corresponds to the first 3 counters. A high
number in the last counter signifies encrypted NFS traffic. Also check the Total RPC Calls with Unknown Authentication
Flavor field, where you want to see 0 or a small number because these connections are not optimized.
The NFS connection is asynchronous. Verify that the Percentage of Requests Served Locally field is nonzero.
WAE# sh statistics accelerator nfs
NFS:
Global Statistics
16:40:06 2009
16:40:06 2009
Total RPC Calls per Authentication Flavor:
298544
0
0
Total RPC Calls with Unknown Authentication Flavor:
Total RPC Calls per NFS Version:
0
0
298609
0
Total RPC Calls with Unknown NFS Version:
Total Requests:
Total Local Replies:
Percentage of Requests Served Locally:
Percentage of Requests Served Remotely:
Average Time to Generate Local READ Reply (ms):
Average Time to Generate Local WRITE Reply (ms):
Average Time to Generate Local GETATTR Reply (ms):
Average Time to Generate Local Reply (ms):
Average Time to Receive Remote Reply (ms):
Meta-Data Cache Access Count:
Meta-Data Cache Hit Count:
Fri Oct 23
Fri Oct 23
170
170
0
0
0
0
13
65
0
0
0
298609
191713
64
36
15
0
0
0
10
206017
191673
<----Should see 0 or few in last fi

<----Should see 0 or few
<----Should see 0 or few in first t

<----Should see 0 or few
<----Should be nonzero
67

Remaining Number Of Entries in Meta-Data Cache:
Meta-Data Cache Hit Ratio:
128926
93
You can view the NFS connection statistics by using the show statistics connection optimized nfs detail command as follows:
WAE674# show stat conn opt nfs detail
Connection Id:
1916
Peer Id:
00:14:5e:84:24:5f
Connection Type:
EXTERNAL CLIENT
Start Time:
Thu Jun 25 07:09:09 2009
Source IP Address:
10.10.10.20
Source Port Number:
928
10.10.100.102
Application Name:
File-System
<-----Should
Classifier Name:
NFS
<-----Should
Map Name:
basic
Directed Mode:
FALSE
Preposition Flow:
FALSE
Policy Details:
Configured:
Derived:
Peer:
Negotiated:
Applied:
Configured:
NFS
<-----Should
Derived:
NFS
Applied:
NFS
<-----Should
Hist:
None
Original
Optimized
-------------------- -------------------Bytes Read:
5120
4639
Bytes Written:
28136
1407
. . .
see File-System
see NFS
see NFS configured

see NFS applied
NFS : 1916
07:09:09 2009
Total Bytes Read:
28136
5120
Bit Flags for I/O state:
Histogram of Buffers Read From Local Endpoint:
1
0
0
0
Total NFS Requests:
Total Replies Served Locally:
Percentage of Requests Served Locally:
Percentage of Requests Served Remotely:
Average Time to Generate Local READ Reply (ms):
Average Time to Generate Local WRITE Reply (ms):
Average Time to Generate Local GETATTR Reply (ms):
Average Time to Generate Local Reply (ms):
Average Time to Receive Remote Reply (ms):
Total RPC Procedure Calls:
9
0
10
7
0
4
0
0
0
0
0
0
1
0
0
0
0
. . .
Total Unknown RPC Procedure Calls:
Total Write RPCs Using Stable-how Enumerated Values:
0
1
Thu Jun 25
5120
28136
19
31
32
4
12
88
0
0
0
0
103
0
0
0
1
0
0
0
68

Total WRITE RPCs with Invalid Stable-how Value:
Bytes Buffered for READ Purpose:
Start Time of Session:
07:09:09 2009
0
0
Thu Jun 25
Meta-Data Cache Access Count:

Meta-Data Cache Hit Count:
Remaining Number Of Entries in Meta-Data Cache:
Meta-Data Cache Hit Ratio:
Current number of entries in Meta-Data Cache:
. . .
9
4
1000
44
0
NFS AO Logging
The following log files are available for troubleshooting NFS AO issues:
Debug log files: /local1/errorlog/nfsao-errorlog.current (and nfsao-errorlog.*)

You can view the end of a transaction log file by using the type-tail command.
To set up and enable debug logging of the NFS AO, use the following commands.
The options for NFS AO debugging are as follows:

WAE674# debug accelerator nfs ?
all
enable all accelerator debugs
async-write
enable async write optimization debugs
attributes-cache enable attributes-cache optimization debugs
nfs-v3
enable NFSv3 layer debugs
read-ahead
enable read ahead optimization debugs
rpc
enable RPC layer debugs
shell
enable shell (infra) debugs
utils
enable utils debugs
You can enable debug logging for NFS connections and then display the end of the debug error log as follows:
NFS AO Logging
69

WAE674# debug accelerator nfs all
WAE674# type-tail errorlog/nfsao-errorlog.current follow
This article describes how to troubleshoot the SSL AO.
Guide Contents
Main Article
Contents
1 SSL Accelerator Overview
2 Troubleshooting the SSL AO
2.1 Troubleshooting HTTP AO to SSL AO Handoff
Connections
2.2 Troubleshooting Server Certificate Verification
2.3 Troubleshooting Client Certificate Verification
2.4 Troubleshooting Peer WAE Certificate Verification
2.5 Troubleshooting OCSP Revocation Checking
2.6 Troubleshooting DNS Configuration
2.7 Troubleshooting HTTP to SSL AO Chaining
2.8 SSL AO Logging
2.9 Troubleshooting Certificate Expiry Alarms on NME and
SRE Modules
Contents
70
SSL Accelerator Overview

The SSL accelerator (available in 4.1.3 and later) optimizes encrypted Secure Sockets Layer (SSL) and Transport Layer Security
(TLS) traffic. The SSL accelerator provides traffic encryption and decryption within WAAS to enable end-to-end traffic
optimization. The SSL accelerator also provides secure management of the encryption certificates and keys.
In a WAAS network, the data center WAE acts as a trusted intermediary node for SSL requests by the client. The private key and
server certificate are stored on the data center WAE. The data center WAE participates in the SSL handshake to derive the session
key, which it distributes securely in-band to the branch WAE, allowing the branch WAE to decrypt client traffic, optimize it,
reencrypt it, and send it over the WAN to the data center WAE. The data center WAE maintains a separate SSL session with the
origin server.
The following services are relevant for SSL/TLS optimization:
Accelerated Service ? A configuration entity that describes acceleration characteristics to be applied for a SSL server or
set of servers. Specifies the certificate and private key to be used while posing as a trusted intermediary, ciphers to be
used, SSL version allowed, and certificate verification settings.
Peering Service ? A configuration entity that describes acceleration characteristics to be applied for in-band SSL
connections between branch and data-center WAEs. This service is used for transferring session key information from
data-center to branch WAEs for optimizing SSL connections.
Central Manager Admin Service ? Not used directly by the SSL accelerator, but to be used by an administrator for the
configuration management of SSL accelerated services. Also used to upload certificates and private keys to be used in
SSL accelerated services.
Central Manager Management Service ? Not used directly by the SSL accelerator, but used for communication between
application accelerator devices and the Central Manager. This service is used for configuration management, secure store
encryption key retrieval, and device status updates.
The Central Manager secure store is essential for the SSL AO to operate because it stores secure encryption keys for all WAEs.
After each Central Manager reload, the administrator needs to reopen the secure store by providing the passphrase with the cms
secure-store open command. A WAE automatically retrieves its secure store encryption key from the Central Manager whenever
the WAE reboots, so no action is required on the WAE after a reload.
If clients are using an HTTP proxy solution, the initial connection is handled by the HTTP AO, which recognizes it as an SSL
tunnel request to port 443. The HTTP AO looks for a matching SSL accelerated service defined on the data center WAE and when
it finds a match, hands off the connection to the SSL AO. However, the traffic that the HTTP AO hands off to the SSL AO for an
HTTPS proxy gets reported as part of the web application statistics, not in the SSL application. If the HTTP AO does not find a
match, the connection is optimized as per static HTTPS (SSL) policy configuration.
The SSL AO can use self-signed certificates rather than CA-signed certificates, which can be helpful in deploying proof of
concept (POC) systems and in troubleshooting SSL issues. By using self-signed certificates, you can quickly deploy a WAAS
system without having to import the origin server certificates, and you can eliminate certificates as a potential source of problems.
You can configure a self-signed certificate in the Central Manager when creating an SSL Accelerated Service. However, when
you use a self-signed certificate, the client browser will display a security alert that the certificate is untrusted (because it is not
signed by a well-known CA). To avoid this security warning, install the certificate in the Trusted Root Certification Authorities
store on the client browser. (On Internet Explorer, on the security warning, click View Certificate, then on the Certificate dialog
click Install Certificate and complete the Certificate Import Wizard.)
Configuring the SSL Management Services is optional, and allows you to change the SSL version and cipher list used for Central
Manager communications to WAEs and to the browser (for administrative access). If you configure ciphers that are not supported
by your browser, you will lose the connection to the Central Manager. In this case, use the crypto ssl management-service
configuration command from the CLI to set the SSL management service settings back to the default.

the Troubleshooting Application Acceleration article. The Enterprise license is required for SSL accelerator operation.
Next, verify the status that is specific to the SSL AO on both the data center and branch WAEs by using the show accelerator ssl
command, as shown in Figure 1. You want to see that the SSL AO is Enabled, Running, and Registered, and that the connection
limit is displayed. If the Config State is Enabled but the Operational State is Shutdown, it indicates a licensing problem. If the
Operational State is Disabled, it may be because the WAE cannot retrieve the SSL keys from the Central Manager secure store,
either because the secure store is not open or the Central Manager is unreachable. Use the show cms info and ping commands to
confirm that the Central Manager is reachable.
Figure 1. Verifying the SSL Accelerator Status
If you see an Operational State of Gen Crypto Params, wait until the status becomes Running, which may take a few minutes
following a reboot. If you see a state of Retrieving Keys from CM for more than a few minutes, it could indicate that the CMS
service on the Central Manager is not running, that there is no network connectivity to the Central Manager, that the WAAS
versions on the WAE and Central Manager are incompatible, or that the Central Manager secure store is not open.
You can verify that the Central Manager secure store is initialized and open by using the show cms secure-store command as
follows:
cm# show cms secure-store
secure-store is initialized and open.
If the secure store is not initialized or open, you will see critical alarms such as mstore_key_failure and secure-store. You can
open the secure store with the cms secure-store open command or from the Central Manager, choose Admin > Secure Store.
Tip: Document the secure store password to avoid having to reset the secure store if you forget the password.
If there is a problem with the disk encryption on a WAE, that can also prevent the SSL AO from operating. Use the show disk
details command to verify that disk encryption is enabled and check if the CONTENT and SPOOL partitions are mounted. If
these partitions are mounted, it indicates that the disk encryption keys were successfully retrieved from the Central Manager and

encrypted data can be written and read from the disks. If the show disk details command shows "System is initializing," that
indicates the encryption keys have not yet been retrieved from the Central Manager and the disks have not yet been mounted. The
WAE will not provide acceleration services in this state. If the WAE is unable to retrieve disk encryption keys from the Central
Manager, it will raise an alarm.
You can verify that the SSL accelerated service is configured and its status is "Enabled" on the data center WAE (in the Central
Manager, choose the device, then choose Configure > Acceleration > SSL Accelerated Services ). A configured and enabled
accelerated service may be rendered inactive by the SSL accelerator due to the following conditions:
The certificate configured in the accelerated service been deleted from the WAE. Use the show running-config command
to determine the certificate being used in the accelerated service, then use the show crypto certificates and show crypto
certificate-details commands to confirm that the certificate is present secure store. If the certificate is missing, reimport
the certificate.
The accelerated service certificate has expired. Use the show crypto certificates and show crypto certificate-details
commands to check the certificate expiry date.
The accelerated service certificate has a valid date starting in the future. Use the show crypto certificates and show
crypto certificate-details commands and check the validity section of the command output. Also, ensure that the WAE
clock and timezone information is accurate.
You can verify that SSL connections have the correct policy applied, that is, they have full optimization with SSL acceleration, as
shown in Figure 2. In the Central Manager, choose the WAE device, then choose Monitor > Optimization > Connections
Statistics.
Figure 2. Verifying the Correct Policy on SSL Connections
Use the show running-config command to verify that the HTTPS traffic policy is properly configured. You want to see optimize
DRE no compression none for the SSL application action and you want to see appropriate match conditions listed for the HTTPS
classifier, as follows:
WAE674# sh run | include HTTPS
classifier HTTPS
name SSL classifier HTTPS action optimize DRE no compression none
WAE674# sh run | begin HTTPS
<-------------

...skipping
classifier HTTPS
exit
<-------------
An active accelerated service inserts dynamic policies corresponding to the server IP:port, server name:port, or server domain:port
configured within the accelerated service. These policies can be inspected using the show policy-engine application dynamic
command. The Dst field in each displayed policy indicates the server IP and port matching the accelerated service. For the
wildcard domain (for example, server-domain *.webex.com port 443), the Dst field will be 'Any:443'. For the server-name
configuration, forward DNS lookup is performed when the accelerated service is activated and all the IP addresses returned in the
DNS response will be inserted in the policy engine. This command is useful to catch situations where an accelerated service is
marked "inservice" but the accelerated service is rendered inactive because of some other error. For example, all accelerated
services are dependent on the peering service, and if the peering service is inactive because of a missing/deleted certificate, then
an accelerated service will also be marked as inactive although it appears to be "inservice" in the show running-config output. You
can verify that the SSL dynamic policy is active on the data center WAE by using the show policy-engine application dynamic
command. You can verify the peering service status by using the show crypto ssl services host-service peering command.
An SSL AO accelerated service configuration can have four types of server entries:
Static IP (server-ip)--available in version 4.1.3 and later
Catch All (server-ip any)--available in 4.1.7 and later
Hostname (server-name)--available in 4.2.1 and later
Wildcard domain (server-domain)--available in 4.2.1 and later
Once the connection is received by the SSL AO, it decides which accelerated service should be used for optimization. The static
IP configuration is given the highest preference, followed by server name, server domain, and then the server ip any. If none of the
configured and activated accelerated services match with the server IP for the connection, the connection is pushed down to the
generic AO. The cookie inserted into the policy engine by the SSL AO is used to determine which accelerated service and what
type of server entry is matched for a particular connection. This policy engine cookie is a 32-bit number and is meaningful only to
the SSL AO. The higher bits are used to indicate different server entry types and the lower bits indicate the accelerated service
index, as follows:
SSL Policy Engine Cookie Values

Cookie Value
Server Entry
Type
0x8xxxxxxx
Server IP
address
Static IP address configuration
0x4xxxxxxx
Server
hostname
Data center WAE performs a forward DNS lookup for the hostname and it adds the IP addresses
that are returned into the dynamic policy configuration. Refreshed every 10 minutes by default.
0x2FFFFFFF
Data center WAE performs a reverse DNS lookup on the destination host IP address to determine
Server
if it matches with the domain. If it matches, then SSL traffic is accelerated, and if it does not
domain name
match, the traffic is handled according to the static HTTPS policy.
0x1xxxxxxx
Server Any
Comments
All SSL connections are accelerated using this accelerated service configuration
Example 1: Accelerated Service with server-ip Configuration:

WAE(config)#crypto ssl services accelerated-service asvc-ip
WAE(config-ssl-accelerated)#description "Server IP acceleration"
WAE(config-ssl-accelerated)#server-cert-key server.p12
WAE(config-ssl-accelerated)#server-ip 171.70.150.5 port 443
WAE(config-ssl-accelerated)#inservice
74

The corresponding policy engine entry is added as follows:
WAE# sh policy-engine application dynamic
Allocations: 1751
< snip >

Number:
1
Type: Any->Host (6) User Id: SSL (4)
Src: ANY:ANY Dst: 171.70.150.5:443
Map Name: basic
Flags: SSL
Hits: 25 Flows: - NA - Cookie: 0x80000001
<---------------<----------------
<----------------
Example 2: Accelerated Service with server-name Configuration:

This configuration allows easy deployment for optimization of enterprise SSL applications. It is adaptable to DNS configuration
changes and reduces IT administrative tasks.
WAE(config)#crypto ssl services accelerated-service asvc-name
WAE(config-ssl-accelerated)#description "Server name acceleration"
WAE(config-ssl-accelerated)#server-name www.google.com port 443

Allocations: 1751
< snip >

Number:
1
Type: Any->Host (6) User Id: SSL
Src: ANY:ANY Dst: 74.125.19.104:443
Map Name: basic
Flags: SSL
DM Ref Index: - NA - DM Ref Cnt: 0
Number:
2
Src: ANY:ANY Dst: 74.125.19.147:443
Map Name: basic
Flags: SSL
Number:
3
Src: ANY:ANY Dst: 74.125.19.103:443
Map Name: basic
Flags: SSL
Number:
4
Src: ANY:ANY Dst: 74.125.19.99:443
Map Name: basic
Flags: SSL
(4)
<---------------<----------------
<---------------(4)
<---------------<----------------
<---------------(4)
<---------------<----------------
<---------------(4)
<---------------<----------------
75

<----------------
Example 3: Accelerated Service with server-domain Configuration:

This configuration allows WAAS devices to configure a single wildcard domain that avoids the need to know IP addresses for all
the servers. The data center WAE uses reverse DNS (rDNS) to match traffic belonging to the configured domain. Configuring a
wildcard domain avoids configuring multiple IP addresses, making the solution scalable and applicable for SaaS architecture.
WAE(config)#crypto ssl services accelerated-service asvc-domain
WAE(config-ssl-accelerated)#description "Server domain acceleration"
WAE(config-ssl-accelerated)#server-name *.webex.com port 443

Allocations: 1751
< snip >

Number:
1
Src: ANY:ANY Dst: ANY:443
Map Name: basic
Flags: SSL
Hits: 0 Flows: - NA - Cookie: 0x2FFFFFFF
<---------------<----------------
<----------------
Example 4: Accelerated Service with server-ip any Configuration:

This configuration provides a catch-all mechanism. When an accelerated service with server-ip any port 443 is made active, it
allows all connections on port 443 to be optimized by the SSL AO. This configuration can be used during POCs to optimize all
traffic on a particular port.
WAE(config)#crypto ssl services accelerated-service asvc-ipany
WAE(config-ssl-accelerated)#description "Server ipany acceleration"
WAE(config-ssl-accelerated)#server-ip any port 443

Allocations: 1751
< snip >

Number:
1
Src: ANY:ANY Dst: ANY:443
Map Name: basic
Flags: SSL
<---------------<----------------
<----------------
76

You can verify the ciphers being used with the show statistics crypto ssl ciphers commands, as shown in Figure 3.
Figure 3. Verifying Ciphers
You can verify that these ciphers match those configured on the origin server. Note: Ciphers that include DHE are not supported
by Microsoft IIS servers.
On an Apache server, you can verify the SSL version and cipher details in the httpd.conf file. These fields may also be in a
separate file (sslmod.conf) referenced from httpd.conf. Look for the SSLProtocol and SSLCipherSuite fields as follows:
SSLProtocol -all +TLSv1 +SSLv3
SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM
. . .
SSLCertificateFile /etc/httpd/ssl/server.crt
SSLCertificateKeyFile /etc/httpd/ssl/server.key
To verify the certificate issuer on an Apache server, use the openssl command to read the certificate as follows:
> openssl x509 -in cert.pem -noout -issuer -issuer_hash
issuer= / C=US/ST=California/L=San Jose/O=CISCO/CN=tools.cisco.com/emailAddress=webmaster@cisco.com be7cee67
In the browser, you can view a certificate and its details to determine the certificate chain, version, encryption key type, issuer
common name (CN), and subject/site CN. In Internet Explorer, click the padlock icon, click View Certificate, and then look at the
Details and Certification Path tabs for this information.
Most browsers require that client certificates be in the PKCS12 format rather than the X509 PEM format. To export the X509
PEM format to PKCS12 format, use the openssl command as follows on an Apache server:
77

> openssl pkcs12 -export -in cert.pem -inkey key.pem -out cred.p12
Enter Export Password:
Verifying - Enter Export Password:
If the private keys are encrypted, the passphrase is required for export. The export password is used again for importing
credentials to the WAAS device.
Use the show statistics accelerator ssl command to see the SSL AO statistics.
WAE7326# show statistics accelerator ssl
SSL:
Global Statistics
Mon Nov 10
15:28:47 2008
Mon Nov 10
15:28:47 2008
17
17
Total Connections Handed-off with Compression Policies Unchanged: 0
0
0
0
3
Total LAN Bytes Read:
25277124
Total Reads on LAN:
5798
Total LAN Bytes Written:
6398
Total Writes on LAN:
51
Total WAN Bytes Read:
43989
Total Reads on WAN:
2533
Total WAN Bytes Written:
10829055
Total Writes on WAN:
3072
. . .
<---------------<---------------<---------------<----------------
<---------------<---------------<---------------<---------------<---------------<---------------<---------------<----------------
Failed sessions and certificate verifications statistics can be useful for troubleshooting and are more easily retrieved by using the
following filter on the show statistics accelerator ssl command:
WAE# show statistics accelerator ssl | inc Failed
Total Failed Handshakes:
Total Failed Certificate Verifications:
Failed certificate verifications due to invalid certificates:
Failed Certificate Verifications based on OCSP Check:
Failed Certificate Verifications (non OCSP):
Total Failed Certificate Verifications due to Other Errors:
Total Failed OCSP Requests:
Total Failed OCSP Requests due to Other Errors:
Total Failed OCSP Requests due to Connection Errors:
Total Failed OCSP Requests due to Connection Timeouts:
Total Failed OCSP Requests due to Insufficient Resources:
47
28
28
0
28
0
0
0
0
0
0
DNS related statistics can be useful for troubleshooting server name and wildcard domain configuration. To retrieve these
statistics use the show statistics accelerator ssl command, as follows:
WAE# show statistics accelerator ssl
. . .
Number of forward DNS lookups issued:
Number of forward DNS lookups failed:
Number of flows with matching host names:
Number of reverse DNS lookups issued:
Number of reverse DNS lookups failed:
Number of reverse DNS lookups cancelled:
Number of flows with matching domain names:
Number of flows with matching any IP rule:
18
0
8
46
4
0
40
6
78

. . .
Pipe-through due to domain name mismatch:
. . .
SSL rehandshake related statistics can be useful for troubleshooting and can be retrieved using the following filter on the show
statistics accelerator ssl command:
WAE# show statistics accelerator ssl | inc renegotiation
Total renegotiations requested by server:
Total SSL renegotiations attempted:
Total number of failed renegotiations:
Flows dropped due to renegotiation timeout:
0
0
0
0
Use the show statistics connection optimized ssl command to check that the WAAS device is establishing optimized SSL
connections. Verify that "TDLS" appears in the Accel column for a connection. "S" indicates that the SSL AO was used as
follows:
WAE674# sh stat conn opt ssl
Historical Flows:
3
3
0
1
0
0
100
ConnID
342
Local IP:Port
10.56.94.101:3406
Remote IP:Port
10.10.100.100:443
PeerID
0:1a:64:d3:2f:b8
Accelerator
TDLS
<-----Look for "S"
You can check connection statistics for closed connections by using the show statistics connection closed ssl command.
If the connections are not getting optimized, check if WCCP/PBR is properly configured and working, and check for asymmetric
routing.
You can view the SSL connection statistics by using the show statistics connection optimized ssl detail command, where you
will see the dynamic policy that results from the configured SSL accelerated service. Note: The configured policy is TFO
optimization only, but full optimization is applied as a result of the configured SSL service.
WAE674# sh stat connection optimized ssl detail
Connection Id:
1633
Peer Id:
00:14:5e:84:24:5f
Connection Type:
EXTERNAL CLIENT
Start Time:
Wed Jul 15 06:35:48 2009
Source IP Address:
10.10.10.10
Source Port Number:
2199
10.10.100.100
Application Name:
SSL
Classifier Name:
HTTPS
Map Name:
basic
Directed Mode:
FALSE
Preposition Flow:
FALSE
Policy Details:
Configured:
TCP_OPTIMIZE
Derived:
Peer:
TCP_OPTIMIZE
Negotiated:
Applied:
<------TFO only is configured
<------Full optimization applied
79

Configured:
Derived:
Applied:
Hist:
Bytes Read:
Bytes Written:
. . .
None
None
SSL
None
<------SSL acceleration applied
Original
Optimized
-------------------- -------------------1318
584
208
1950
Later in this output, extended SSL session level details are shown as follows:
. . .
SSL : 1633
Total Bytes Read:
Memory address:
LAN bytes read:
Number of reads on LAN fd:
LAN bytes written out:
Number of writes on LAN fd:
WAN bytes read:
Number of reads on WAN fd:
WAN bytes written out:
Number of writes on WAN fd:
LAN handshake bytes read:
LAN handshake bytes written out:
WAN handshake bytes read:
WAN handshake bytes written out:
AO bytes read:
Number of reads on AO fd:
AO bytes written out:
Number of writes on AO fd:
DRE bytes read:
Number of reads on DRE fd:
DRE bytes written out:
Number of writes on DRE fd:
Number of renegotiations requested by server:
Number of SSL renegotiations performed:
Flow state:
LAN work items:
LAN conn state:
LAN SSL state:
WAN work items:
WAN conn state:
WAN SSL state:
W2W work items:
W2W conn state:
W2W SSL state:
AO work items:
AO conn state:
DRE work items:
DRE conn state:
Hostname in HTTP CONNECT:
IP Address in HTTP CONNECT:
TCP Port in HTTP CONNECT:
Tue Jul 10 18:23:20 2009

0
0
0
0
0x8117738
1318
4
208
2
584
23
1950
7
1318
208
542
1424
0
0
0
0
10
1
10
1
0
0
0x00080000
1
READ
SSLOK (0x3)
0
READ
SSLOK (0x3)
1
READ
SSLOK (0x3)
1
READ
1
READ
<-----Added in 4.1.5
<-----Added in 4.1.5
<-----Added in 4.1.5
80
Troubleshooting HTTP AO to SSL AO Handoff Connections

If a client must go through a proxy to reach an HTTPS server, the client?s request first goes as an HTTP CONNECT message to
the proxy (with the actual HTTPS server IP address embedded in the CONNECT message). At this point, the HTTP AO handles
this connection on the peer WAEs. The proxy creates a tunnel between the client and server port and relays subsequent data
between the client and that server IP address and port. The proxy responds back to the client with a ?200 OK? message and hands
off the connection to the SSL AO because the client intends to talk to the server over SSL. The client then initiates an SSL
handshake with the SSL server over the TCP connection (tunnel) that was set up by the proxy.
Check the following things when troubleshooting issues with handed-off connections:
Check the output of the show statistics accelerator http command to confirm that a connection was handled by the
HTTP AO and then handed off to the SSL AO. Look at the Total Handled Connections and Total Connections Handed-off
to SSL counters. If there are any issues, verify the following:
The HTTP AO is enabled and in the running state on the peer WAEs.
The SSL accelerated service is configured with the port used by the client in the CONNECT URL (or implied port
443 if HTTPS is being used). Often the proxy port is different from the CONNECT URL port and this proxy port
should not be configured in the SSL accelerated service. However, the proxy port should be included in the traffic
classifier that is mapped to the HTTP AO.
Check the output of the show statistics accelerator http command to confirm that this connection was handled and
optimized by the SSL AO. Look at the Total Handled Connections and Total Optimized Connections counters. If the
statistics counters are not correct, perform basic SSL troubleshooting as discussed in the previous section.
On the data center WAE, verify that the show statistics connection optimized detail command output shows the actual
SSL server?s hostname, IP address, and TCP port. If these fields are not set correctly, check the following:
Verify that the client browser proxy settings are correct.
Verify that the DNS server is configured on the data center WAE and is reachable. You can configure a DNS
server on the WAE with the ip name-server A.B.C.D command.
Troubleshooting Server Certificate Verification
Server certificate verification requires that you import the correct CA certificate to the data center WAE.
To troubleshoot server certificate verification follow these steps:
1. Inspect the server certificate and retrieve the Issuer name. This Issuer name within the server certificate must match the subject
name within the matching CA certificate. If you have PEM encoded certificates, you can use the following openssl command on a
server with openssl installed:
> openssl x509 ?in cert-file-name ?noout ?text
2. Ensure that the matching crypto pki ca configuration exists on the data center WAE by using show running-config command.
For a CA certificate to be used by the WAE in the verification process, a crypto pki ca configuration item is required for each CA
certificate imported. For example, if a CA certificate company1.ca is imported, then the following configuration must be made on
the data center WAE:
crypto pki ca company1

ca-certificate company1.ca
exit
Note: If a CA certificate is imported using the Central Manager GUI, the Central Manager automatically adds the above crypto
pki ca configuration to include the imported CA certificate. However, if the CA certificate is imported via the CLI, then you will
need to manually add the above configuration.

3. If the certificate being verified includes a certificate chain, then ensure that the certificate chain is coherent, and the topmost
issuer?s CA certificate is imported on the WAE. Use the openssl verify command to verify the certificate separately first.
4. If verification still fails, then examine the SSL accelerator debug log. Use the following commands to enable debug logging:
wae# config
wae(config)# logging disk priority debug
wae(config)# logging disk enable
wae(config)# exit
wae# undebug all
wae# debug accelerator ssl verify
wae# debug tfo connection all
5. Initiate a test connection and then examine the /local/local1/errorlog/sslao-errorlog.current log file. This file should indicate the
issuer name that was included in the server certificate. Ensure that this issuer name exactly matches the subject name of the CA
certificate.
If there are any other internal errors in the logs, it may be helpful to enable additional debug options.
6. Even if the Issuer name and Subject names match, the CA certificate may not be the correct one. In such cases, if the server
certificate is issued by a well-known CA, then a browser can be used to directly (without WAAS) reach the server. When the
browser sets up the connection, the certificate can be examined by clicking the Lock icon that appears on the bottom right of the
browser window or within the browser?s address bar. The certificate details may indicate the appropriate CA certificate matching
this server certificate. Check the Serial Number field within the CA certificate. This serial number should match the serial number
of the certificate that is being imported on the data center WAE.
7. If you have OCSP revocation checking enabled, disable it and check that certificate verification by itself works. For help
troubleshooting OCSP settings see the "Troubleshooting OCSP Revocation Checking" section.
Troubleshooting Client Certificate Verification

Verification of the client certificate may be enabled on the origin server and/or on the data center WAE. When WAAS is used to
accelerate SSL traffic, the client certificate received by the origin server is the certificate indicated in the machine-cert-key
specified in the crypto ssl services global-settings command on the data center WAE or the data center WAE machine self
signed certificate, if the machine-cert-key is not configured. As a result, if client certificate verification is failing on the origin
server, it may be because the data center WAE machine-certificate is not verifiable on the origin server.
If client certificate verification on the data center WAE is not working, it is likely because the CA certificate matching the client
certificate is not imported on the data center WAE. See the "Troubleshooting Server Certificate Verification" section for
instructions how to check if you have the correct CA certificate imported on the WAE.
Troubleshooting Peer WAE Certificate Verification

To troubleshoot peer certificate verification issues follow these steps:
1. Verify that the certificate being verified is a CA signed certificate. A self signed certificate by one WAE is not verifiable by
another WAE. WAEs by default are loaded with self signed certificates. A self signed certificate must be configured using the
crypto ssl services global-settings machine-cert-key command.
2. Verify that the correct CA certificate is loaded on the device that is verifying the certificate. For example, if peer-cert-verify is
configured on the data center WAE, then it is essential for the branch WAE certificate to be CA-signed and the same signing
CA?s certificate should be imported on the data center WAE. Do not forget to create a CA using the crypto pki ca command to
use the imported certificate, if you are importing the certificate manually through the CLI. When imported by the Central Manager
GUI, the Central Manager automatically creates a matching crypto pki ca configuration.
Troubleshooting Server Certificate Verification
82

3. If verification of the peer WAE still fails, check the debug logs as described in the "SSL AO Logging" section.
Troubleshooting OCSP Revocation Checking

If the system is having trouble making successful SSL connections with Online Certificate Status Protocol (OCSP) revocation
checking enabled, follow these troubleshooting steps:
1. Ensure that the OCSP responder service is running on the responder server.
2. Ensure good connectivity between the WAE and the responder. Use the ping and telnet commands (to the appropriate
port) from the WAE to check.
3. Confirm that the certificate being validated is indeed valid. The expiry date and correct responder URL are typically areas
where there are issues.
4. Verify that the certificate for OCSP responses is imported on the WAE. Responses from an OCSP responder are also
signed and the CA certificate matching the OCSP responses must reside on the WAE.
5. Check the show statistics accelerator ssl command output to check for OCSP statistics and check the counters
corresponding to OCSP failures.
6. If the OCSP HTTP connection is going through an HTTP proxy, try disabling the proxy to see if it helps. If it does help,
then check that the proxy configuration is not causing the connection failure. If the proxy configuration is fine, then there
may be some HTTP header peculiarity which may be causing some incompatibility with the proxy. Capture a packet trace
for further investigation.
7. If all else fails, you may have to capture a packet trace of the outgoing OCSP request for further debugging. You can use
the tcpdump or tethereal commands as described in the section "Capturing and Analyzing Packets" in the Preliminary
WAAS Troubleshooting article.
The URL used by the data center WAE to reach an OCSP responder is derived in one of two ways:
The static OCSP URL configured by the crypto pki global-settings configuration command
The OCSP URL specified in the certificate being checked
If the URL is derived from the certificate being checked, then it is essential to ensure that the URL is reachable. Enable the SSL
accelerator OCSP debug logs to determine the URL and then check for connectivity to the responder. See the next section for
details on using debug logs.
Troubleshooting DNS Configuration

If the system is having trouble optimizing SSL connections with server name and server domain configurations, follow these
troubleshooting steps:
1. Ensure that the DNS server configured on the WAE is reachable and can resolve names. Use the following command to check
the configured DNS server:
WAE# sh running-config | include name-server
ip name-server 2.53.4.3
Try to perform DNS or reverse DNS lookup on the WAE using the following commands:
WAE# dnslookup www.cisco.com
The specified host/domain name is unknown !
This response indicates the name cannot be resolved by the configured name servers.
Try ping/traceoute for the configured name servers to check their reachability and the round trip time.
WAE# ping 2.53.4.3
PING 2.53.4.3 (2.53.4.3) 56(84) bytes of data.
Troubleshooting Peer WAE Certificate Verification
83

--- 2.53.4.3 ping statistics --5 packets transmitted, 0 received, 100% packet loss, time 4008ms
WAE# traceroute 2.53.4.3
traceroute to 2.53.4.3 (2.53.4.3), 30 hops max, 38 byte packets
1 2.53.4.33 (2.53.4.33) 0.604 ms 0.288 ms 0.405 ms
2 * * *
3 * * *
4 * * *
5 * * *
2. If the DNS server is reachable and it can resolve names and still the SSL connections are not getting optimized, make sure the
accelerated service configuring the specified domain or hostname is active and there are no alarms for the SSL AO. Use following
commands:
WAE# show alarms
Critical Alarms:
---------------Alarm ID
--------------1 accl_svc_inactive
2 accl_svc_inactive
Module/Submodule
-------------------sslao/ASVC/asvc-host
sslao/ASVC/asvc-domain
Instance
--------------accl_svc_inactive
accl_svc_inactive
Major Alarms:
------------None
Minor Alarms:
------------None
The presence of the "accl_svc_inactive" alarm is an indication that there is some discrepancy in the accelerated service
configuration and there might be one or more accelerated services having overlapping configuration for server entries. Check the
accelerated service configuration and make sure the configuration is correct. Use the following command to verify the
configuration:
WAE# show crypto ssl accelerated service
Accelerated Service
Config State
Oper State
---------------------------------------asvc-ip
ACTIVE
ACTIVE
asvc-host
ACTIVE
INACTIVE
asvc-domain
ACTIVE
INACTIVE
Cookie
-----0
1
2
To check details about a particular accelerated service use the following command:
WAE# show crypto ssl accelerated service asvc-host
Name: asvc-host
Config state: ACTIVE, Oper state: INACTIVE, Cookie: 0x3, Error vector: 0x0
No server IP addresses are configured
The following server host names are configured:
lnxserv.shilpa.com port 443
Host 'lnxserv.shilpa.com' resolves to following IPs:
--none-No server domain names are configured
One reason that the operational state of the accelerated service might be INACTIVE is a DNS failure. For example, if there is a
server hostname in the accelerated service configuration and the WAE cannot resolve the server IP address, then it cannot
configure the appropriate dynamic policy.
3. If the statistics counter for ?Pipe-through due to non-matching domain name? is increasing, it is an indication that the SSL
connection is for a server that is configured for optimization. Check the policy engine entries using following command:
Troubleshooting DNS Configuration
84

WAE#sh policy-engine application dynamic
Number:
1
Src: ANY:ANY Dst: 2.53.4.2:443
Map Name: basic
Flags: TIME_LMT DENY
Hits: 1 Flows: - NA - Cookie: 0x2EEEEEEE
Check the connection status using the show statistics connection command. The first connection should show an Accelerator of
TSGDL and the subsequent connections, until the lifetime of the TIME_DENY policy entry, should be TDL.
4. If the DNS server is across the WAN with respect to the data center WAE, or if the reverse DNS response time is too long, then
some connections may be dropped. This depends on the client timeout and the rDNS response time. In this case, the counter for
?Number of reverse DNS lookups cancelled? increases and the connection is dropped. This situation is an indication that the DNS
server is not responsive or very slow and/or NSCD on WAAS is not working. The NSCD status can be checked using the show
alarms command. The probability of this happening is very low since in most deployments, the DNS server is expected to be on
the same LAN as the data center WAE.
Troubleshooting HTTP to SSL AO Chaining

NOTE: HTTP to SSL AO chaining was introduced in WAAS version 4.3.1. This section is not applicable to earlier WAAS
versions.
Chaining allows an AO to insert another AO at any time during the lifetime of a flow and both AOs can apply their AO-specific
optimization independently on the flow. AO chaining is different from the AO handoff feature provided by WAAS in pre-4.3.1
releases because with AO chaining the first AO continues to optimize the flow.
The SSL AO handles two types of connections:
Byte-0 SSL: The SSL AO receives the connection first and completes the SSL handshake. It parses the initial part of the
payload to check for an HTTP method. If the payload indicates HTTP, it inserts the HTTP AO; if not, it applies the
regular TSDL optimization.
Proxy CONNECT: The HTTP AO receives the connection first. It identifies the CONNECT header method in the client?s
request and inserts the SSL AO after the proxy confirms with a 200 OK message.
The SSL AO uses a lightweight HTTP parser that detects the following HTTP methods: GET, HEAD, POST, PUT, OPTIONS,
TRACE, COPY, LOCK, POLL, BCOPY, BMOVE, MKCOL, DELETE, SEARCH, UNLOCK, BDELETE, PROPFIND,
BPROPFIND, PROPPATCH, SUBSCRIBE, BPROPPATCH, UNSUBSCRIBE, and X__MS_ENUMATTS. You can use the
debug accelerator ssl parser command to debug issues related to the parser. You can use the show stat accel ssl payload
http/other command to view statistics of traffic classified based on the payload type.
Troubleshooting tips:
1. Make sure the HTTPS feature is enabled in the HTTP AO configuration as this is owned by the HTTP AO. For details,
see the Troubleshooting the HTTP AO article.
2. Check the connection state using the show stat connection command. If correctly optimized, it should show THSDL
indicating TCP, HTTP, SSL and DRE-LZ optimization. If any of these optimizations are missing, debug further on that
optimizer (SSL, HTTP, and so forth). For example, if the connection state shows THDL, it means SSL optimization was
not applied on the connection. Details on debugging issues related to the SSL AO follow.
3. Make sure the SSL AO is enabled and is in the running state (see the section "Troubleshooting the SSL AO").
4. Make sure there are no alarms by using the show alarms command.
5. If SSL traffic is not being optimized, make sure the server IP address, host-name, or domain-name and port number is
added as part of the accelerated service.
Troubleshooting HTTP to SSL AO Chaining
85

6. Make sure the accelerated service is in the ACTIVE state by using the show crypto ssl services accelerated-service
ASVC-name command (see the "Troubleshooting DNS Configuration" section).
7. Make sure the policy engine has an entry for this server and port by using the show policy-engine application dynamic
command.
8. If the destination server is using SSL on a non-default port (the default is 443), make sure this is reflected in the policy
engine configuration. The Central Manager relies on this information for reporting SSL traffic data.
9. Make sure the configured host-name resolves to a valid IP address by using the show crypto ssl services
accelerated-service ASVC-name command. If no IP address is found, check if the name server is configured correctly.
Also check the output of the dnslookup IP-address command.
wae# sh run no-policy
. . .
crypto ssl services accelerated-service sslc
version all
server-cert-key test.p12
server-ip 2.75.167.2 port 4433
server-ip any port 443
server-name mail.yahoo.com port 443
server-name mail.google.com port 443
inservice
wae# sh crypto ssl services accelerated-service sslc
Name: sslc
Config state: ACTIVE, Oper state: ACTIVE, Cookie: 0x0, Error vector: 0x0
The following server IP addresses are configured:
2.75.167.2 port 4433
any port 443
The following server host names are configured:
mail.yahoo.com port 443
Host 'mail.yahoo.com' resolves to following IPs:
66.163.169.186
mail.google.com port 443
Host 'mail.google.com' resolves to following IPs:
74.125.19.17
74.125.19.18
74.125.19.19
74.125.19.83
wae# dnslookup mail.yahoo.com
Official hostname: login.lga1.b.yahoo.com
address: 66.163.169.186
Aliases: mail.yahoo.com
Aliases: login.yahoo.com
Aliases: login-global.lgg1.b.yahoo.com
wae# dnslookup mail.google.com
Official hostname: googlemail.l.google.com
address: 74.125.19.83
address: 74.125.19.17
address: 74.125.19.19
address: 74.125.19.18
Aliases: mail.google.com
SSL AO Logging
The following log files are available for troubleshooting SSL AO issues:
SSL AO Logging
86

Debug log files: /local1/errorlog/sslao-errorlog.current (and sslao-errorlog.*)


:SSL :HTTPS :F :(TFO) (DRE,LZ,TFO) (TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) :<None> :(None) (None) (SSL) :<None> :<None>
Wed Jul 15 14:36:06 2009 :1633 :10.10.10.10 :2199 :10.10.100.100 :443 :SODRE :END :165 :15978764 :63429 :10339 :0
Wed Jul 15 14:36:06 2009 :1633 :10.10.10.10 :2199 :10.10.100.100 :443 :OT :END :EXTERNAL CLIENT :(SSL) :468 :16001
To set up and enable debug logging of the SSL AO, use the following commands.
The options for SSL AO debugging are as follows:

WAE674# debug accelerator
accelerated-svc
enable
alarm
enable
all
enable
am
enable
am-generic-svc
enable
bio
enable
ca
enable
ca-pool
enable
cipherlist
enable
client-to-server enable
dataserver
enable
flow-shutdown
enable
generic
enable
ocsp
enable
oom-manager
enable
openssl-internal enable
peering-svc
enable
session-cache
enable
shell
enable
sm-alert
enable
sm-generic
enable
sm-io
enable
sm-pipethrough
enable
SSL AO Logging
ssl ?
accelerated service debugs
SSL AO alarm debugs
all SSL accelerator debugs
auth manager debugs
am generic service debugs
bio layer debugs
cert auth module debugs
cert auth pool debugs
cipherlist debugs
client-to-server datapath debugs
dataserver debugs
flow shutdown debugs
generic debugs
ocsp debugs
oom-manager debugs
opnessl internal debugs
peering service debugs
session cache debugs
SSL shell debugs
session manager alert debugs
session manager generic debugs
session manager i/o debugs
sm pipethrough debugs
87

synchronization
verify
waas-to-waas
enable synchronization debugs

enable certificate verification debugs
enable waas-to-waas datapath debugs
You can enable debug logging for SSL connections and then display the end of the debug error log as follows:
WAE674# debug accelerator ssl all
WAE674# debug connection all
Enabling debug messages for all connections.
Are you sure you want to do this? (y/n) [n]y
WAE674# type-tail errorlog/sslao-errorlog.current follow
Troubleshooting Certificate Expiry Alarms on NME and SRE Modules

The SSL AO generates alarms when the self-signed machine certificate has expired (or is within 30 days of expiration) and a
custom global machine certificate is not configured on the WAAS device. The WAAS software generates factory self-signed
certificates with an expiration date of 5 years from the first startup of the WAAS device.
The clock in all WAAS NME and SRE modules is set to January 1, 2006 during first startup, even though the NME or SRE
module is more recent. This causes the self-signed certificate to expire on January 1, 2011 and the device generates certificate
expiration alarms.
If you are not using the default factory certificate as the global certificate, and instead are using a custom certificate for the SSL
AO, you will not experience this unexpected expiration and you can update the custom certificate whenever it expires. Also, if you
have updated the NME or SME module with a new software image and have synchronized the clock to a more recent date, you
may not experience this issue.
The symptom of certificate expiration is one of the following alarms (shown here in the output of the show alarms command):
Major Alarms:
------------Alarm ID
--------------1 cert_near_expiration
Module/Submodule
-------------------sslao/SGS/gsetting
Instance
--------------cert_near_expiration
Module/Submodule
-------------------sslao/SGS/gsetting
Instance
--------------cert_expired
or
Alarm ID
--------------1 cert_expired
The Central Manager GUI reports the following alarm: "Certificate__waas-self__.p12 is near expiration it is configured as
machine cert in global settings"
You can use one of the following solutions to resolve this problem:
Configure a different certificate for global settings:
SRE# crypto generate self-signed-cert waas-self.p12 rsa modulus 1024
SRE# config
SRE(config)# crypto ssl services global-settings machine-cert-key waas-self.p12
Update the self-signed factory certificate with a later expiration date. This solution requires a script that you can obtain by
contacting Cisco TAC.
NOTE: This issue is fixed by the resolution of caveat CSCte05426, released in WAAS software versions 4.1.7b, 4.2.3c, and 4.3.3.
The certification expiration date is changed to 2037.
Troubleshooting Certificate Expiry Alarms on NME and SRE Modules
88
This article describes how to troubleshoot the video AO.
Guide Contents
Main Article
Contents
1 Video Accelerator
Troubleshooting
2 Video AO Logging
Video Accelerator Troubleshooting
The Video accelerator optimizes Windows Media live streams that are requested over RTSP. Requests for RTSP-UDP streams are
denied by WAAS and the player will automatically request an RTSP-TCP stream. Incoming stream splitting allows multiple
clients to watch live video over a single stream on the WAN.
the Troubleshooting Application Acceleration article. The Video and Enterprise licenses are required for Video accelerator
operation.
Next, verify the status that is specific to the video AO by using the show accelerator video command, as shown in Figure 1. You
want to see that the video AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State
is Enabled but the Operational State is Shutdown, it indicates a licensing problem.
Contents
89

Figure 1. Verifying the Video Accelerator Status
Use the show statistics accelerator video command to see the Video AO statistics. The following output shows that one
incoming video stream from the WAN was split to 10 clients, which removed 9 video streams from the WAN.
wae# sh stat acc video
Time elapsed since "clear statistics": 1days 0hr 50min 30sec
Video Connections
==================================================================
Connections handled
num
%
-----------------------------------------------------------------Total handled
3330
100.00
Windows-media live accelerated
3329
99.97
Un-accelerated pipethru
1
0.03
Un-accelerated dropped due to config
0
0.00
Error dropped connections
0
0.00
Windows-media active sessions
current
max
-----------------------------------------------------------------Outgoing (client) sessions
10
10
Incoming (server) sessions
1
10
<-----10 client sessions split from

<-----1 incoming stream
Windows-media byte savings

==================================================================
% Bytes saved
Incoming(server) bytes
Outgoing(client) bytes
56.01
2.07 GB
4.71 GB
To examine the reasons why the video AO is not accelerating video connections, use the show statistics accelerator video detail
command. In the example below, the video is not a live broadcast stream but is a video-on-demand (VoD), which is not

accelerated.
wae# sh stat acc video detail
< snip >
Unaccelerated Connections
num
%
-----------------------------------------------------------------Total Unaccelerated
1
100.00
Unsupported player
0
0.00
Unsupported transport
0
0.00
Unsupported protocol
0
0.00
Windows-media VoD
1
100.00
Max stream bitrate overload
0
0.00
Max aggregate bitrate overload
0
0.00
Max concurrent sessions overload
0
0.00
Other
0
0.00
<----------- VoD, not live
Error dropped connections

num
%
-----------------------------------------------------------------Total errors
0
0.00
Client timeouts
0
0.00
Server timeouts
0
0.00
Client stream errors
0
0.00
Server stream errors
0
0.00
Other errors
0
0.00
If videos are not being accelerated as expected, it is often because they are not marked with the live broadcast cache-control
header, x-wms-stream-type="broadcast". VoD streams lack this header. Figure 2 shows where to find the cache-control header in
the Windows Media Server response to the player, using Wireshark.
Figure 2. Windows Media Cache-Control Header
91
The URLs for video streams are case sensitive to the video AO, so if a video stream is not being optimized or not playing,
carefully check the URL case and verify that the video is still played. Also verify that the video can be played directly from the
video server, without using WAAS in the network path, to ensure that the video is playable.
Use the show statistics connection optimized video command to check that the WAAS device is establishing optimized video
connections. Verify that "V" appears in the Accel column for video connections, which indicates that the video AO was used as
follows:
WAE# sh stat conn opt video
Historical Flows:
500
500
0
0
0
15
0
302

ConnID
1603
1604
Source IP:Port
2.75.13.3:1442
2.75.13.3:1443
Dest IP:Port
PeerID Accel RR
2.75.11.3:554 00:1a:64:64:b1:ec TV
00.0%
2.75.11.3:554 00:1a:64:64:b1:ec TV
100.0%
<-----Look for "V"
92

1605
2.75.13.3:1444
2.75.11.3:554 00:1a:64:64:b1:ec TV
100.0%
You can see in the connections above that DRE and LZ optimizations are not used with video, but the primary server connection
is TFO optimized. All subsequent connections for the same video stream show a reduction of 100% because they are completely
removed from the WAN and instead are split from the primary stream at the branch WAE.
Figure 3. Connection Statistics Report with Video
The show statistics connection optimized video windows-media command is useful to show the status of all inbound video
streams, including the requesting URL. The show statistics connection optimized video detail command is useful to list all the
inbound and outbound video streams being handled by the video AO.
Video AO Logging
The following log files are available for troubleshooting video AO issues:
Debug log files: /local1/errorlog/videoao-errorlog.current (and videoao-errorlog.*)
Debug log files for the WM module: /local1/errorlog/wmt_errorlog.current (and wmt_errorlog.*)
wae(config)# transaction-logs accelerator video windows-media enable
You can view the end of a transaction log file by using the type-tail command.
To set up and enable debug logging of the video AO, enable detailed logging to the disk:

The options for video AO debugging are as follows:
WAE674# debug accelerator video ?
all
enable all video accelerator debugs.
gateway
enable gateway debugs
shell
enable Video shell debugs
windows-media enable windows-media debugs
You can enable debug logging for video connections and then display the end of the debug error log as follows:
WAE674# debug accelerator video all
WAE674# type-tail errorlog/videoao-errorlog.current follow
This article describes how to troubleshoot the Generic AO.
Guide Contents
Main Article
Contents
1 Generic Accelerator
Troubleshooting
2 Generic AO Logging
Contents
94
Generic Accelerator Troubleshooting

The Generic accelerator optimizes traffic that is pushed down from the other AOs because they cannot optimize the traffic. The
generic AO performs TFO optimization only. (DRE and LZ compression optimizations are performed by the SO-DRE
component)
The generic AO receives connections under the following conditions:
Failure case: An AO determines that it cannot handle the connection after sensing that the data is incomprehensible to it.
For example, if the CIFS AO senses encrypted data or unauthenticated content, it will not be able to handle it and will
push down the connection to the generic AO.
Multiple protocol handling: For instance, the video AO could accept all connections that are related to multiple protocols
like WMT, RTSP, and so on. However, the video AO currently provides only RTSP optimization, so it will not handle the
connections that are related to other protocols and it will push down these connections to the generic AO.
Common scenarios in which connections are pushed down to the generic AO include the following conditions where there is
connectivity that the AO does not understand or cannot optimize:
Unauthenticated CIFS
SMB-signed CIFS
Encrypted MAPI
Non-RTSP video
One way to check if the generic AO is being used is to look at statistics from the other AOs. For example, the CIFS AO reports
connections that are pushed down to the generic AO as follows:
WAE674# sh stat accelerator cifs detail
CIFS:
Global Statistics
11:55:09 2009
04:16:35 2009
Number of local reply generating requests:
Number of remote reply generating requests:
The Average time to generate a local reply (msec):
Average time to receive remote reply (ms):
Tue Jul 14
Thu Jul 16
32
1
24
0
0
0
4
3388
415
25
2147
<-----Pushed down to generic
You can also check connection statistics to see what optimizations are being applied to connections. In the show statistics
connection output, a "G" indicates the connection was handled by the generic AO as follows:
WAE674# sh stat connection
2
2
0
0
5
0

Historical Flows:
100
ConnID
Source IP:Port
Dest IP:Port
PeerID
Accel
3722
3924
10.10.10.10:2162
10.10.10.10:2464
10.10.100.100:445
10.10.100.101:445
00:14:5e:84:24:5f
00:14:5e:84:24:5f
TCDL
TGDL
<-----Look for "G"
If you take a closer look at the connection above, you will see that CIFS was configured but the generic AO was applied as
follows:
WAE674# sh stat connection conn-id 3924
Connection Id:
3924
Peer Id:
00:14:5e:84:24:5f
Connection Type:
EXTERNAL CLIENT
Start Time:
Thu Jul 16 06:10:44 2009
Source IP Address:
10.10.10.10
Source Port Number:
2464
10.10.100.101
Application Name:
WAFS
Classifier Name:
CIFS
Map Name:
basic
Directed Mode:
FALSE
Preposition Flow:
FALSE
Policy Details:
Configured:
Derived:
Peer:
Negotiated:
Applied:
Configured:
CIFS
Derived:
CIFS
Applied:
GENERICAO
Hist:
CIFS
<-----CIFS configured
<-----Generic applied
Connections Statistics. Connections handled by the generic AO look as follows:
Figure 1. Connection Statistics Report with Generic
You can use the show statistics accelerator generic detail command to see more details about connections being handled by the

generic AO as follows:
WAE# sh stat accelerator generic detail
Generic:
------Time elapsed since "clear statistics": 1days 18hr 25min 20sec
Time Accelerator was started:
11:55:02 2009
11:55:02 2009
. . .
Tue Jul 14
Tue Jul 14
366
366
0
0
1
0
2
Global Generic AO connection statistics

=======================================
Total number of connections handled:
Total number of active connections:
Total number of bytes transferred from client:
Total number of bytes transferred from server:
366
1
12055
12492
Global Generic AO connection error statistics

=============================================
Source connection closed:
Destination connection closed:
Source connection aborted:
Destination connection aborted:
Source connection error:
Destination connection error:
Out of memory:
Kernel Queue abort error:
730
0
0
0
0
0
0
0
<------------
If you are seeing a large Total number of connections handled, some kind of configuration or communication error might be
causing a large number of connections to be pushed down.
Generic AO Logging
The following log files are available for troubleshooting generic AO issues:
Debug log files: /local1/errorlog/genericao-errorlog.current (and genericao-errorlog.*)

Generic AO Logging
97

To set up and enable debug logging of the generic AO, use the following commands.
The options for generic AO debugging are as follows:

WAE674# debug accelerator generic ?
all
enable all GENERIC accelerator debugs
connection enable GENERIC accelerator connection debugs
misc
enable GENERIC accelerator miscellaneous debugs
shell
enable GENERIC accelerator shell debugs
stats
enable GENERIC accelerator stats debugs
You can enable debug logging for generic AO connections and then display the end of the debug error log as follows:
WAE674# debug accelerator generic connection
WAE674# type-tail errorlog/genericao-errorlog.current follow
This article describes how to troubleshoot overload conditions.
Guide Contents
Main Article
Generic AO Logging
98

Contents
1 Overview
2 How to Monitor TFO Flows and Overload Conditions
2.1 Checking the TCP Connection Limit
2.2 Checking the Optimized TCP Connections
3 MAPI Application Accelerator Reserved Connections Impact on
Overload
4 Solutions for Overload Conditions
Overview
The Cisco WAAS network would have been designed to optimize a certain number of TCP connections, based on customer
requirements. Depending on the model of the WAE, there could be additional connection limitations for the SSL and CIFS
application accelerators. When either the overall connection limit or a specific application accelerator connection limit is
exceeded, the device is overloaded. In this situation, more traffic is entering the device than it can handle and so traffic may not be
optimized as expected (overloaded traffic is passed through unoptimized).
How to Monitor TFO Flows and Overload Conditions
When a WAAS accelerator device is overloaded, you typically see the following Central Manager alarm: Entering overload state
due to Max connections (nnn). The number nnn is the number of times the WAE has become overloaded since the last reboot.
The device also logs a syslog error message similar to the following: Sysmon: %WAAS-SYSMON-3-445015: Fault detected: The
TFO accelerator is overloaded (connection limit)
You can use various show commands at the CLI to determine the number of allowed and actual connections and gather more
information.
Checking the TCP Connection Limit
The first useful command is show tfo detail, which can tell you how many optimized TFO connections that the device can handle,
as follows:
wae-7341# show tfo detail

Policy Engine Config Item
------------------------State
Default Action
Connection Limit
Effective Limit
Keepalive timeout
Contents
Value
----Registered
Use Policy
12000
11988
3.0 seconds
<-----Maximum number of TFO optimized connections
99

The Connection Limit value tells you that this WAAS device can support 12000 TFO optimized connections.
The Effective Limit may be lower than the Connection Limit if the MAPI AO has reserved some connections. The reserved
connections are subtracted from the Connection Limit to get the Effective Limit.
Checking the Optimized TCP Connections

To understand the TCP flows on the device, you can use the show statistics connection command (in version 4.1.1, use the show
statistics connection all command). This command displays the currently handled TFO/DRE/LZ flows, pass-through flows, and
flows that are being handled by a specific application accelerator. An example of this command follows:
wae# show statistics connection
Historical Flows:
5
5
0
0
0
12
0
143
<---------- Added in 4.1.5
ConnID
Source IP:Port
Dest IP:Port
PeerID
Accel
92917
92918
92921
94458
36883
10.86.232.131:41197
10.86.232.131:41198
10.86.232.131:41216
10.86.232.131:45354
10.86.232.136:1857
70.70.7.11:3268
70.70.7.11:3268
70.70.7.11:3268
70.70.7.11:1026
10.86.232.131:1026
00:1a:64:69:19:fc
00:1a:64:69:19:fc
00:1a:64:69:19:fc
00:1a:64:69:19:fc
00:1a:64:69:19:fc
TDL
TDL
TDL
TDL
TDL
From the first line in the output (Current Active Optimized Flows), you can see that the device currently has five active optimized
flows. From the second counter (Current Active Optimized TCP Plus Flows), you can see that they are all being handled with
TFO/DRE/LZ optimization (TFO Plus means that DRE and/or LZ optimization is being used in addition to TFO). The third
counter (Current Active Optimized TCP Only Flows) shows flows that are optimized by TFO only.
Another useful counter is the Current Active Auto-discovery Flows, which displays flows that have not been fully set up to
become optimized flows or pass-through flows. To become fully set up, the connection must see the SYN, SYN ACK, ACK
handshake, which is useful to note when dealing with an overload condition. The Current Active Pass-Through Flows counter
shows connections that the device has determined to be pass-through or where the device did not see the SYN, SYN ACK, ACK
setup. These flows will not be counted as optimized flows. For pass-through flows, a device should be able to handle up to 10
times the number of optimized flows for which it is rated.
The Current Reserved Flows counter shows the number of connections reserved for the MAPI accelerator. For more details about
reserved MAPI connections and their impact on device overload, see the section MAPI Application Accelerator Reserved
Connections Impact on Overload.
The sum of the following three counters tells you how close the WAE device is to its connection limit:
Current Active Optimized Flows
Current Active Auto-Discovery Flows
Current Reserved Flows (available only in 4.1.5 and later)
If this sum is equal to or greater than the connection limit, the device is in an overload condition.
Checking the TCP Connection Limit
100

Details about the five optimized flows are displayed in the table below the counters.
Another command that you can use to see the number of TFO flows currently on a device is the show statistics tfo detail
command. Two of the most useful counters in the output are "No. of active connections" and under the Policy Engine Statistics the
"Active connections", as follows:
wae# show statistics tfo detail
Total number of connections
: 22915
No. of active connections
: 3
No. of pending (to be accepted) connections
: 0
No. of bypass connections
: 113
No. of normal closed conns
: 19124
No. of reset connections
: 3788
Socket write failure
: 2520
Socket read failure
: 0
WAN socket close while waiting to write
: 1
AO socket close while waiting to write
: 86
WAN socket error close while waiting to read
: 0
AO socket error close while waiting to read
: 80
DRE decode failure
: 0
DRE encode failure
: 0
Connection init failure
: 0
WAN socket unexpected close while waiting to read : 1048
Exceeded maximum number of supported connections : 0
Buffer allocation or manipulation failed
: 0
Peer received reset from end host
: 53
DRE connection state out of sync
: 0
Memory allocation failed for buffer heads
: 0
Unoptimized packet received on optimized side
: 0
Data buffer usages:
Used size:
0 B, B-size:
0 B, B-num: 0
Cloned size:
54584 B, B-size:
73472 B, B-num: 111
Buffer Control:
Encode size:
0 B, slow:
0, stop:
Decode size:
0 B, slow:
0, stop:
AckQ Control:
Total:
0, Current:
0
Scheduler:
Queue Size: IO:
0, Semi-IO:
0, Non-IO:
Total Jobs: IO:
219110, Semi-IO:
186629, Non-IO:
<-----Current optimized connections
0
0
Policy Engine Statistics

------------------------Session timeouts: 0, Total timeouts: 0
Last keepalive received 00.0 Secs ago
Last registration occurred 8:03:54:38.7 Days:Hours:Mins:Secs ago
Hits:
52125, Update Released:
Active Connections:
3, Completed Connections:
Drops:
0
Rejected Connection Counts Due To: (Total: 12)
Not Registered
:
12, Keepalive Timeout
:
No License
:
0, Load Level
:
Connection Limit
:
0, Rate Limit
:
Minimum TFO
:
0, Resource Manager
:
Global Config
:
0, Server-Side
:
DM Deny
:
0, No DM Accept
:
Auto-Discovery Statistics
------------------------Total Connections queued for accept:
Connections queuing failures:
Socket pairs queued for accept:
Socket pairs queuing failures:
Checking the Optimized TCP Connections
0
49227
17945
37257
0
0
0
0
0
0
<-----Active Connections
<-----Connection Limit
22907
0
0
0
101

AO discovery successful:
AO discovery failure:
0
0
In some cases, the two counters will differ and the reason is that the ?No. of active connections? displays all the current flows that
being optimized by TFO, TFO/DRE, TFO/DRE/LZ, and TFO/DRE/LZ and an application accelerator. The ?Active Connections?
under the policy engine statistics includes all the flows in the state above plus the connections that are optimized only by TFO and
an application accelerator. This situation means that a TCP flow has come in and matched an application accelerator classifier but
the SYN, SYN ACK, ACK handshake has not been completed.
In many TFO overload cases, if the problem is still occurring, you can look at these commands and determine if the number of
optimized flows is around the rated number of optimized TCP connections for the hardware. If it is, then you can view the flow
details and see what is using up all the flows to determine if this traffic is legitimate and overloading the device or is a virus,
security scanner, or something else occurring on the network.
The "Connection Limit" counter under the policy engine statistics reports the number of connections rejected and passed through
because the WAE has exceeded its rated number of optimized TCP connections. If this counter is high, it means the WAE is
frequently getting more connections than it can handle.
If the number of optimized connections is not close to the rated number of optimized TCP connections and you are still getting an
overload alarm, then you should look at the Current active auto-discovery flows from the show statistics connection command or
the ?Active Connections? under Policy Engine Statistics from the show statistics tfo detail command. In some cases, the number
of optimized connections can be very low but the Active Connections under the Policy Engine Statistics are roughly equal to the
rated number of optimized flows for the hardware. This situation means that there are many flows that match a classifier but they
are not fully established. When a TCP SYN matches a classifier, it will reserve an optimized connection. This connection will not
appear in the optimized TCP connections count until the TCP handshake is finished and optimization starts. If the device
determines that the flow should not be optimized, it will be removed from the count of active connections under the Policy Engine
Statistics.
To further troubleshoot cases where TFO overload is occurring and the Policy Engine Statistics Active Connections seem to be
using up all the optimized TCP connections on the device, use the show statistics accelerator detail command. In the output of
this command, look at the Active Connections under the Policy Engine Statistics for each application accelerator to determine
which application accelerator is receiving these connections that are not fully established. Next, look at what state these flows
might be in by using the show statistics filtering command, which provides you with the number of filtering tuples on the device,
as follows:
wae# show statistics filtering
Number of filtering tuples:
Number of filtering tuple collisions:
Packets dropped due to filtering tuple collisions:
Number of transparent packets locally delivered:
Number of transparent packets dropped:
Packets dropped due to ttl expiry:
Packets dropped due to bad route:
Syn packets dropped with our own id in the options:
Syn-Ack packets dropped with our own id in the options:
Internal client syn packets dropped:
Syn packets received and dropped on estab. conn:
Syn-Ack packets received and dropped on estab. conn:
Syn packets dropped due to peer connection alive:
Syn-Ack packets dropped due to peer connection alive:
Packets recvd on in progress conn. and not handled:
Packets dropped due to peer connection alive:
Packets dropped due to invalid TCP flags:
Packets dropped by FB packet input notifier:
Packets dropped by FB packet output notifier:
Number of errors by FB tuple create notifier:
Number of errors by FB tuple delete notifier:
18
0
0
965106
0
0
10
0
0
0
0
0
525
0
0
1614
0
0
0
0
0

Dropped WCCP GRE packets due to invalid WCCP service:
Dropped WCCP L2 packets due to invalid WCCP service:
Number of deleted tuple refresh events:
Number of times valid tuples found on refresh list:
0
0
0
0
The number of filtering tuples is the number of flows on the device that are optimized, in pass-through, in FIN WAIT state, in
setup state, and so on. Each established flow appears as two tuples, one for each side of the flow, so the number that you see in
this output may be much larger than the number of flows that you are seeing in the other commands.
To get more information on the flows in the filtering list, you can use the show filtering list command as follows:
wae# show filtering list
E:
s:
B:
T:
Established, S: Syn, A: Ack, F: Fin, R: Reset

sent, r: received, O: Options, P: Passthrough
Bypass, L: Last Ack, W: Time Wait, D: Done
Timedout, C: Closed
Local-IP:Port
10.86.232.82:23
10.86.232.131:58775
70.70.7.11:3268
70.70.7.11:3268
10.86.232.131:57920
10.86.232.82:23
10.86.232.131:58787
70.70.7.11:1026
10.86.232.131:48698
10.86.232.131:58774
70.70.7.11:389
10.86.232.131:58728
10.86.232.131:58784
70.70.7.11:1026
70.70.7.11:1026
10.86.232.131:58790
70.70.7.11:3268
Remote-IP:Port
10.86.232.134:41784
70.70.7.11:3268
10.86.232.131:58775
10.86.232.131:57920
70.70.7.11:3268
161.44.67.102:4752
70.70.7.11:1026
10.86.232.131:58787
70.70.7.11:1026
70.70.7.11:389
10.86.232.131:58774
70.70.7.11:1026
70.70.7.11:1026
10.86.232.131:58784
10.86.232.131:48698
70.70.7.11:3268
10.86.232.131:58790
Tuple(Mate)
0xbc1ae980(0x0
)
0x570b2900(0x570b2b80)
0x570b2b80(0x570b2900)
0x570b2d80(0x570b2800)
0x570b2800(0x570b2d80)
0xbc1aee00(0x0
)
0x570b2080(0x570b2e80)
0x570b2e80(0x570b2080)
0x570b2f00(0x570b2880)
0x570b2300(0x570b2180)
0x570b2180(0x570b2300)
0x570b2380(0x570b2a00)
0x570b2e00(0x570b2980)
0x570b2980(0x570b2e00)
0x570b2880(0x570b2f00)
0x570b2100(0x570b2c80)
0x570b2c80(0x570b2100)
State
E
EW
EDL
E
E
E
EW
EDL
PE
EW
EDL
E
EW
EDL
PE
EW
EDL
If the show statistics accelerator all command shows you which application accelerator is using up all the optimized TFO
connections, you can filter on that port or traffic. For example, if you want to filter on port 80 traffic, use the show filtering list |
I :80 command.
Look at the legend in the State column. In the case where the flows are in the SYN state, you may see a lot of flows with a state of
S. If the WAE has sent back the SYN ACK with options set you may see the state SAsO. This indication may help you determine
the state of the flow and from there, you can determine if there is a routing problem, virus, or a problem with the WAE not
releasing connections. You may need traces to determine exactly what is happening to the flows but the commands above should
give you an idea of what to look for.
MAPI Application Accelerator Reserved Connections Impact on

Overload
Often, a TFO overload can be caused by the MAPI application accelerator reserved connections, so it is helpful to understand the
process of how the MAPI application accelerator reserves connections.
The MAPI application accelerator reserves TFO connections to ensure that it will have enough connections available to it to
accelerate all current and future connections that the clients will make to the Exchange servers. It is normal for a MAPI client to
make multiple connections. If a client makes the initial connection through the MAPI application accelerator, but the subsequent
connections fail in the MAPI application accelerator, there is a risk that the client's connection might fail.
MAPI Application Accelerator Reserved Connections Impact on Overload
103

In order to avoid these potential connection failures, the MAPI application accelerator reserves connection resources as follows:
Before any client connections begin, it reserves 10 connections for itself, as a buffer for anticipated new connections.
For each client connection to the server, it reserves three TFO connections for that client-server pair and one of the three
is used as an active connection for this first connection. If the same client makes a second or third connection to the same
server, those are handled out of the reserved connection pool. If a client only ever makes a single connection to the server,
those two reserved connections will be unused and remain in the reserved pool. If the client makes a connection to a
different server, three new connections are again reserved for that client-server pair.
All of these reserved connections are designed to improve performance and to reduce the possibility of a client connection failing
because of the inability to make additional connections through the MAPI application accelerator.
Overload occurs when Current Active Optimized Flows + Current Active Auto-Discovery Flows + Current Reserved Flows is
greater than the device's fixed Connection Limit. In general, new connections would then be passed through. But some new MAPI
connections may still be optimized. When the device is at the overload point, if a client makes an additional request to a MAPI
server it already has connected to, then reserved connections are used. But if there are not enough reserved connections (for
example, if a client makes a fourth connection to the same MAPI server and the WAE is already in overload) then an escaped
connection condition might occur, and this could lead to erroneous behavior such as a client receiving many duplicate copies of
the same single mail message.
If the system did not forward the connection to the MAPI application accelerator, you should see "PT Rjct Resources" or "PT in
progress", depending on whether there is activity on the connection. If the connection was forwarded to the MAPI application
accelerator and then the reservation failed, the connection will be marked with a "G" for the Accelerator, instead of an "M" (in the
show statistics connection optimized mapi command output). For an example of this command, see the article Troubleshooting
the MAPI AO.
If you are experiencing frequent overload conditions, it is important to understand how the Outlook clients are making
connections (how many connections to how many Exchange servers). With Outlook running on a client, hold the Ctrl key while
you right-click on the Outlook icon in the system tray on the task bar. Choose Connection Status to display the list of servers to
which the Outlook client has connected. From that you can see how many connections the client is making and to how many
different Exchange servers. If the client is making connections to several different servers, it would be helpful to investigate ways
to consolidate mail so a user only opens MAPI connections to a single Exchange server, and makes use of multiple connections to
that server.
It is also useful to investigate whether there are any other applications that might be making MAPI connections.
Solutions for Overload Conditions

Examine optimized connections to see if they are legitimate. In many cases, a Denial of Service (DoS) attack encountered in the
network may be causing the WAE to attempt to optimize connections. If so, employ a DoS protection mechanism in the network
to proactively close the connections.
In cases where the connections are legitimate, the WAE deployed in the location is undersized and may need to be upgraded, or an
additional WAE can be deployed to increase scalability within that site.
This article describes how to troubleshoot WCCP issues.
Guide Contents

Main Article
Contents
1 Troubleshooting WCCP on the Router
1.1 Troubleshooting WCCP on the Catalyst 6500 Series Switches and the ISR and 3700
Series Routers
1.2 Troubleshooting WCCP on the ASR 1000 Series Routers
2 Troubleshooting WCCP on the WAE
3 Troubleshooting Configurable Service IDs and Variable Timeouts in Version 4.4.1
The following symptoms indicate possible WCCP issues:

The WAE is not receiving traffic (could be due to WCCP misconfiguration)
End users cannot reach their server applications (could be due to blackholing of traffic)
Network slowness when WCCP is enabled (could be due to router dropping packets or high router CPU usage)
Overly high router CPU usage (could be due to redirection in software instead of hardware)
WCCP issues can result from problems with the router (or redirecting device) or from the WAE device. It is necessary to look at
the WCCP configuration both on the router and on the WAE device. First we will look at the WCCP configuration on the router,
then we will check the WCCP configuration on the WAE.
Contents
105
Troubleshooting WCCP on the Router

This section covers troubleshooting on the following devices:
Catalyst 6500 Series Switches and the ISR and 3700 Series Routers
ASR 1000 Series Routers
Troubleshooting WCCP on the Catalyst 6500 Series Switches and the ISR and 3700
Series Routers
Begin troubleshooting by verifying WCCPv2 interception on the switch or router by using the show ip wccp IOS command as
follows:
Router# show ip wccp
Global WCCP information:
Router information:
Router Identifier:
Protocol Version:
10.88.81.242
2.0
Service Identifier: 61
Number of Service Group Clients:
Number of Service Group Routers:
Total Packets s/w Redirected:
Process:
Fast:
CEF:
Service mode:
Service access-list:
Total Packets Dropped Closed:
Redirect access-list:
Total Packets Denied Redirect:
Total Packets Unassigned:
Group access-list:
Total Messages Denied to Group:
Total Authentication failures:
Total Bypassed Packets Received:
--More--
1
1
68755
2
0
68753
Open
-none0
-none0
0
-none0
0
0
<-----Client = WAE
<-----Increments for software-based redirection
<----<----<-----
<-----Match service group but not redirect list
<-----Packets have incorrect service group password
On platforms that use software-based redirection, verify that the Total Packets s/w Redirected counters are incrementing in the
above command output. On platforms that use hardware-based redirection, these counters should not be incrementing much. If
you are seeing these counters increment significantly on hardware-based platforms, WCCP could be misconfigured on the router
(WCCP GRE is processed in software by default), or the router could be falling back to software redirection due to hardware
resources issues such as running out of TCAM resources. More investigation is required if you see these counters incrementing on
a hardware-based platform, which could lead to high CPU usage.
The Total Packets Denied Redirect counter increments for packets that match the service group but do not match the redirect list.
The Total Authentication failures counter increments for packets that are received with the incorrect service group password.
On routers where WCCP redirection is performed in the software, continue by verifying WCCPv2 interception on the router by
using the show ip wccp 61 detail IOS command as follows:
Router# show ip wccp 61 detail
WCCP Client information:
WCCP Client ID:
Protocol Version:
State:
Initial Hash Info:
10.88.81.4
2.0
Usable
00000000000000000000000000000000
Troubleshooting WCCP on the Router
<-----Should be Usable
106

Assigned Hash Info:
Hash Allotment:
Packets s/w Redirected:
Connect Time:
Bypassed Packets
Process:
Fast:
CEF:
00000000000000000000000000000000
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
256 (100.00%)
2452
01:19:46
<-----Buckets handled by this WAE

<-----Time WAE has been in service group
0
0
0
Verify that the WAE state in the service group 61 is Usable. Verify that hash buckets are assigned to the WAE in the Hash
Allotment field. The percentage tells you how many of the total hash buckets are handled by this WAE. The amount of time that
the WAE has been in the service group is reported in the Connect Time field. The hash assignment method should be used with
software-based redirection.
You can determine which WAE in the farm will handle a particular request by using the show ip wccp service hash dst-ip src-ip
dst-port src-port hidden IOS command on the router as follows:
Router# show ip wccp 61 hash 0.0.0.0 10.88.81.10 0 0
WCCP hash information for:
Primary Hash:
Src IP: 10.88.81.10
Bucket:
9
WCCP Client: 10.88.81.12
<-----Target WAE
On routers where WCCP redirection is performed in the hardware, continue by verifying WCCPv2 interception on the router by
using the show ip wccp 61 detail IOS command as follows:
Cat6k# sh ip wccp 61 detail
WCCP Client ID:
10.88.80.135
Protocol Version:
2.0
State:
Usable
Redirection:
L2
Packet Return:
GRE
Packets Redirected:
0
Connect Time:
1d18h
Assignment:
MASK
Mask SrcAddr
DstAddr
SrcPort DstPort
---- ------------------- ------0000: 0x00001741 0x00000000 0x0000 0x0000
Value
----0000:
0001:
0002:
0003:
SrcAddr
------0x00000000
0x00000001
0x00000040
0x00000041
DstAddr
------0x00000000
0x00000000
0x00000000
0x00000000
SrcPort
------0x0000
0x0000
0x0000
0x0000
DstPort
------0x0000
0x0000
0x0000
0x0000
<-----Use generic GRE for hardware-based platforms
<-----Use Mask for hardware-based redirection
<-----Default mask
CE-IP
----0x0A585087
0x0A585087
0x0A585087
0x0A585087
(10.88.80.135)
(10.88.80.135)
(10.88.80.135)
(10.88.80.135)
You want to see the Mask assignment method for routers that are capable of hardware redirection.
In order to save TCAM resources on the router, consider altering the default WCCP mask to suit your network environment.
Consider these recommendations:
Use the smallest number of mask bits possible when using WCCP redirect ACL. A smaller number of mask bits when
used in conjunction with Redirect ACL results in lower TCAM utilization. If there are 1-2 WCCP clients in a cluster, use
one bit. If there are 3-4 WCCP clients, use 2 bits. If there are 5-8 WCCP clients, then use 3 bits and so on.
We do not recommend using the WAAS default mask (0x1741). For data center deployments, the goal is to load balance
the branch sites into the data center rather than clients or hosts. The right mask minimizes data center WAE peering and
Troubleshooting WCCP on the Catalyst 6500 Series Switches and the ISR and 3700Series Routers
107

hence scales storage. For example, use 0x100 to 0x7F00 for retail data centers that have /24 branch networks. For large
enterprises with a /16 per business, use 0x10000 to 0x7F0000 to load balance the businesses into the enterprise data
center. In the branch office, the goal is to balance the clients that obtain their IP addresses via DHCP. DHCP generally
issues client IP addresses incrementing from the lowest IP address in the subnet. To best balance DHCP assigned IP
addresses with mask, use 0x1 to 0x7F to only consider the lowest order bits of the client IP address to achieve the best
distribution.
The TCAM resources consumed by a WCCP redirect access-list is a product of the content of that ACL multiplied against the
configured WCCP bit mask. Therefore, there is contention between the number of WCCP buckets (which are created based on the
mask) and the number of entries in the redirect ACL. For example, a mask of 0xF (4 bits) and a 200 line redirect permit ACL may
result in 3200 (2^4 x 200) TCAM entries. Reducing the mask to 0x7 (3 bits) reduces the TCAM usage by 50% (2^3 x 200 =
1600).
Catalyst 6500 series and Cisco 7600 series platforms are capable of handling WCCP redirection in both the software and
hardware. If packets are inadvertently being redirected in software, when you expect hardware redirection, it could result in overly
high router CPU use.
You can inspect the TCAM information to determine if redirection is being handled in the software or the hardware. Use the show
tcam IOS command as follows:
Cat6k# show tcam interface vlan 900 acl in ip
* Global Defaults not shared
Entries from Bank 0
Entries from Bank 1

permit
punt
tcp host 10.88.80.135 any

ip any any (8 matches)
<-----Packets handled in software
"Punt" matches represent requests not handled in the hardware. This situation could be caused by the following errors:
Hash assignment instead of mask
Outbound redirection instead of inbound
Redirect exclude in
Unknown WAE MAC address
Using a loopback address for the generic GRE tunnel destination
In the following example, the policy-route entries show that the router is doing full hardware redirection:
Cat6k# show tcam interface vlan 900 acl in ip
* Global Defaults not shared
Entries from Bank 0
Entries from Bank 1

permit
policy-route
policy-route
policy-route
policy-route
policy-route
tcp
tcp
tcp
tcp
tcp
tcp
host 10.88.80.135 any

any 0.0.0.0 255.255.232.190 (60 matches)
any 0.0.0.1 255.255.232.190 (8 matches)
any 0.0.0.64 255.255.232.190 (16 matches)
any 0.0.0.65 255.255.232.190 (19 matches)
any 0.0.1.0 255.255.232.190
<-----These entries show hardware redirection

policy-route
policy-route
policy-route
policy-route
policy-route
policy-route
policy-route
policy-route
tcp
tcp
tcp
tcp
tcp
tcp
tcp
tcp
any
any
any
any
any
any
any
any
0.0.1.1 255.255.232.190
0.0.1.64 255.255.232.190
0.0.1.65 255.255.232.190
0.0.2.0 255.255.232.190
0.0.2.1 255.255.232.190
0.0.2.64 255.255.232.190
0.0.2.65 255.255.232.190 (75 matches)
0.0.3.0 255.255.232.190 (222195 matches)
The Here I Am (HIA) from the WAE must enter the same interface that the WAE MAC is known through. We recommend that
you use a loopback interface and not a directly connected interface in the WAE router list.
Troubleshooting WCCP on the ASR 1000 Series Routers

The commands for troubleshooting WCCP on the Cisco ASR 1000 Series routers are different from the other routers. This section
shows commands that you can use to get WCCP information on the ASR 1000.
To display route processor WCCP information, use the show platform software wccp rp active commands as follows:
ASR1000# sh platform software wccp rp active
Dynamic service 61
Priority: 34, Number of clients: 1
Assign Method: Mask, Fwd Method: GRE, Ret Method: GRE
L4 proto: 6, Use Source Port: No, Is closed: No
Dynamic service 62
Priority: 34, Number of clients: 1
Assign Method: Mask, Fwd Method: GRE, Ret Method: GRE
L4 proto: 6, Use Source Port: No, Is closed: No
<-----Number of WAE clients

<-----Assignment, forwarding, and return methods
<----<-----
The following example shows additional commands that you can use to examine forwarding processor information:
ASR1000# sh platform software wccp fp active ?
<0-255>
service ID
cache-info Show cache-engine info
interface
Show interface info
statistics Show messaging statistics
web-cache
Web-cache type
|
Output modifiers
<cr>
To display redirected packet statistics for each interface, use the show platform software wccp interface counters command as
follows:
ASR1000# sh platform software wccp interface counters
Interface GigabitEthernet0/1/2
Input Redirect Packets = 391
Output Redirect Packets = 0
Interface GigabitEthernet0/1/3
Input Redirect Packets = 1800
Output Redirect Packets = 0
Use the show platform software wccp web-cache counters command to display WCCP cache information as follows:
ASR1000# sh platform software wccp web-cache counters
Service Group (0, 0) counters
unassigned_count = 0
dropped_closed_count = 0
bypass_count = 0
bypass_failed_count = 0
denied_count = 0
redirect_count = 0
Troubleshooting WCCP on the ASR 1000 Series Routers
109

To display low level details, use the following commands:
show platform so interface F0 brief
show platform software wccp f0 interface
debug platform software wccp configuration
For more information, see the white paper "Deploying and Troubleshooting Web Cache Control Protocol Version 2 on Cisco ASR
1000 Series Aggregation Services Routers"
Troubleshooting WCCP on the WAE

Begin troubleshooting on the WAE by using the show wccp services command. You want to see both services 61 and 62
configured, as follows:
WAE-612# show wccp services
Services configured on this File Engine
TCP Promiscuous 61
TCP Promiscuous 62
Next check the WCCP status by using the show wccp status command. You want to see that WCCP version 2 is enabled and
active as follows:
WAE-612# show wccp status
WCCP version 2 is enabled and currently active
Look at the WCCP farm information by using the show wccp wide-area-engine command. This command shows the number of
WAEs in the farm, their IP addresses, which one is the lead WAE, routers that can see the WAEs, and other information, as
follows:
WAE612# show wccp wide-area-engine
Wide Area Engine List for Service: TCP Promiscuous 61
Number of WAE's in the Cache farm: 3
Last Received Assignment Key IP address: 10.43.140.162
Last Received Assignment Key Change Number: 17
Last WAE Change Number: 16
Assignment Made Flag = FALSE
IP address = 10.43.140.162
Lead WAE = YES
Routers seeing this Wide Area Engine(3)
10.43.140.161
10.43.140.166
10.43.140.168
<-----All WAEs in farm should have same Key IP
Weight = 0
IP address = 10.43.140.163
Lead WAE = NO
10.43.140.161
10.43.140.166
10.43.140.168
Weight = 0
IP address = 10.43.140.164
Lead WAE = NO
10.43.140.161
10.43.140.166
10.43.140.168
Weight = 0
. . .
Look at the router information by using the show wccp routers command. Verify that there is bidirectional communication with
WCCP-enabled routers and all routers show the same KeyIP and KeyCN (change number), as follows:

WAE-612# show wccp routers
Router Information for Service: TCP Promiscuous 61
Routers Seeing this Wide Area Engine(1)
Router Id
Sent To
Recv ID
KeyIP
KeyCN
10.43.140.161
10.43.140.161
00203A21
10.43.140.162 17
10.43.140.166
10.43.140.166
00203A23
10.43.140.162 17
10.43.140.168
10.43.140.165
00203A2D
10.43.140.162 17
Routers not Seeing this Wide Area Engine
-NONERouters Notified of from other WAE's
-NONEMulticast Addresses Configured
-NONE. . .
MCN
52
53
25
<-----Verify routers have same KeyIP
In cases where the WAE is not Layer 2-adjacent to the router, or a loopback address is used, either static routes or a default
gateway is required to support WCCP.
To examine the hash bucket distribution in the service group, use the show wccp flows tcp-promiscuous command as follows:
wae# sh wccp flows tcp-promiscuous
Flow counts for service: TCP Promiscuous 61
Bucket
Flow Counts
0- 11:
0
0
0
0
0
0
12- 23:
0
0
0
0
0
0
24- 35:
0
0
0
0
0
0
36- 47:
0
0
0
0
0
0
48- 59:
0
0
0
0
0
0
60- 71:
0
0
0
0
0
0
72- 83:
0
0
0
0
0
0
84- 95:
0
0
0
0
0
0
96-107:
0
0
0
0
0
0
108-119:
0
0
0
0
0
0
120-131:
0
0
0
0
0
0
132-143:
0
0
0
0
0
0
144-155:
0
0
0
0
0
0
156-167:
0
0
0
0
0
0
168-179:
0
0
0
0
0
0
180-191:
0
0
0
0
0
0
192-203:
0
0
0
0
0
0
204-215:
0
0
0
0
0
0
216-227:
0
0
0
0
0
0
228-239:
0
0
0
0
0
0
240-251:
0
0
0
0
0
0
252-255:
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Alternatively, you can use the summary version of the command to see similar information, as well as bypass flow information:
wae# sh wccp flows tcp-promiscuous summary
Flow summary for service: TCP Promiscuous 61
Total Buckets
OURS = 256
0- 59:
60-119:
120-179:
180-239:
240-255:
BYP
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
OOOOOOOOOO
= 0
0- 59: .......... .......... .......... .......... .......... ..........

60-119: .......... .......... .......... .......... .......... ..........
111

120-179: .......... .......... .......... .......... .......... ..........
180-239: .......... .......... .......... .......... .......... ..........
240-255: .......... ......
AWAY = 0
0- 59:
60-119:
120-179:
180-239:
240-255:
. . .
..........
..........
..........
..........
..........
..........
..........
..........
..........
......
..........
..........
..........
..........
..........
..........
..........
..........
..........
..........
..........
..........
..........
..........
..........
..........
Use the show wccp gre command to display GRE packet statistics as follows:
WAE-612# show wccp gre
Transparent GRE packets received:
5531561
Transparent non-GRE packets received:
0
Transparent non-GRE non-WCCP packets received: 0
Total packets accepted:
5051
Invalid packets received:
0
Packets received with invalid service:
0
Packets received on a disabled service:
0
Packets received too small:
0
Packets dropped due to zero TTL:
0
Packets dropped due to bad buckets:
0
Packets dropped due to no redirect address:
0
Packets dropped due to loopback redirect:
0
Pass-through pkts dropped on assignment update:0
Connections bypassed due to load:
0
Packets sent back to router:
0
GRE packets sent to router (not bypass)
0
Packets sent to another WAE:
0
GRE fragments redirected:
0
GRE encapsulated fragments received:
0
Packets failed encapsulated reassembly:
0
Packets failed GRE encapsulation:
0
--More--
<-----Increments for WCCP GRE redirection

<-----Increments for WCCP L2 redirection
<-----Increments for ACE or PBR redirection
<-----Accepted for optimization; peer WAE found
<-----Handled with WCCP negotiated return egress
If WCCP redirection is working, either of the first two counters should be incrementing.
The Transparent non-GRE packets received counter increments for packets that are redirected using the WCCP Layer 2 redirect
forwarding method.
The Transparent non-GRE non-WCCP packets received counter increments for packets that are redirected by a non-WCCP
interception method (such as ACE or PBR).
The Total packets accepted counter indicates packets that are accepted for optimization because auto-discovery found a peer
WAE.
The GRE packets sent to router (not bypass) counter indicates packets that were handled using the WCCP negotiated return egress
method.
The packets sent to another WAE counter indicates that flow protection is occurring when another WAE is added to the service
group and begins handling a bucket assignment that was previously being handled by another WAE.
Verify that the egress methods that are being used are the expected ones by using the show egress-methods command as follows:
WAE674# show egress-methods
112

Intercept method : WCCP
TCP Promiscuous 61 :
WCCP negotiated return method : WCCP GRE
Destination
----------any
Egress Method
Configured
---------------------WCCP Negotiated Return
Egress Method
Used
------------WCCP GRE
<-----Verify these are expected
Destination
----------any
Egress Method
Configured
---------------------WCCP Negotiated Return
Egress Method
Used
------------WCCP GRE
<-----Verify these are expected
Egress method mismatches can occur under the following conditions:

The negotiated return egress method is configured, but WCCP negotiates the Layer 2 return method and only GRE return
is supported by WAAS.
The generic GRE egress method is configured, but the interception method is Layer 2 and only WCCP GRE is supported
as the interception method when generic GRE egress is configured.
In either of these cases, a minor alarm is raised and is cleared when the mismatch is resolved by changing the egress method or the
WCCP configuration. Until the alarm is cleared, the default IP forwarding egress method is used.
The following example shows the command output when a mismatch exists:
WAE612# show egress-methods
Intercept method : WCCP
Destination
----------any
Egress Method
Configured
---------------------Generic GRE
Egress Method
Used
------------IP Forwarding
WARNING:
WCCP has negotiated WCCP L2 as the intercept method for

which generic GRE is not supported as an egress method
in this release. This device uses IP forwarding as the
egress method instead of the configured generic GRE
egress method.
WCCP negotiated return method : WCCP
Egress Method
Destination
Configured
----------- ---------------------any
Generic GRE
WARNING:
GRE
Egress Method
Used
------------IP Forwarding
WCCP has negotiated WCCP L2 as the intercept method for

which generic GRE is not supported as an egress method
in this release. This device uses IP forwarding as the
egress method instead of the configured generic GRE
egress method.
<-----Mismatch
<-----Warning if mismatch occurs
<-----Mismatch
<-----Warning if mismatch occurs
For Catalyst 6500 Sup720 or Sup32 routers, we recommend using the generic GRE egress method, which is processed in
hardware. Additionally, we recommend using one multipoint tunnel for ease of configuration, instead of one point-to-point tunnel
for each WAE. For tunnel configuration details, refer to the section Configuring a GRE Tunnel Interface on a Router in the Cisco
113

Wide Area Application Services Configuration Guide.
To view the GRE tunnel statistics for each intercepting router, use the show statistics generic-gre command as follows:
WAE# sh stat generic
Tunnel Destination:
Tunnel Peer Status:
Tunnel Reference Count:
Packets dropped due to failed encapsulation:
Packets dropped due to no route found:
Packets sent:
Packets sent to tunnel interface that is down:
Packets fragmented:
10.10.14.16
N/A
2
0
0
0
0
0
Failure to ensure that egress packets from a WAE are not reintercepted can lead to a redirection loop. If a WAE detects its own ID
returned in the TCP options field, a redirection loop has occurred and results in the following syslog message:
%WAAS-SYS-3-900000: 137.34.79.11:1192 - 137.34.77.196:139 - opt_syn_rcv: Routing Loop detected - Packet has our ow
You can search the syslog.txt file for instances of this error by using the find command as follows:
WAE-612# find match ?Routing Loop? syslog.txt
This error also shows up in the TFO flow statistics available in the show statistics filtering command as follows:
WAE-612# show statistics filtering
. . .
. . .
<-----Indicates a redirection loop
If you are doing outbound redirection on the router, as traffic leaves the router it will get redirected back to the WAE, which will
reroute the packet out the router, causing a routing loop. If the data center WAE and servers are on different VLANs and the
branch WAE and the clients are on different VLANS, you can avoid a routing loop by using the following router configuration on
the WAE VLAN:
ip wccp redirect exclude in
If the WAE shares the same VLAN with its adjacent clients or servers, you can avoid routing loops by using the negotiated return
method, or generic GRE return for platforms where WCCP redirection is performed in the hardware. When using generic GRE
return, the WAE uses a GRE tunnel to return traffic to the router.
Troubleshooting Configurable Service IDs and Variable Timeouts in

Version 4.4.1
NOTE: The WCCP configurable service IDs and variable failure detection timeout features were introduced in WAAS version
4.4.1. This section is not applicable to earlier WAAS versions.
All WAEs in a WCCP farm must use the same pair of WCCP service IDs (the default is 61 and 62), and these IDs must match all
routers that are supporting the farm. A WAE with different WCCP service IDs than those configured on the routers is not allowed
to join the farm and the existing "Router Unreachable" alarm is raised. Likewise, all WAEs in a farm must use the same value for
the failure detection timeout. A WAE raises an alarm if you configure it with a mismatching value.
If you see an alarm that a WAE is not able to join a WCCP farm, check that the WCCP service IDs configured on the WAE and
the routers in the farm match. On the WAEs, use the show wccp wide-area-engine command to check the configured service IDs.
On the routers, you can use the show ip wccp IOS command.
Troubleshooting Configurable Service IDs and Variable Timeouts in Version 4.4.1
114

To check if the WAE has connectivity to the router, use the show wccp services detail and show wccp router detail commands.
Additionally, you can enable WCCP debug output on the WAE by using the debug ip wccp event or debug ip wccp packet
commands.
If you see a "Router Unusable" minor alarm for a WAE, it could mean that the variable failure detection timeout value set on the
WAE is not supported by the router. Use the show alarm minor detail command to check if the reason for the alarm is "Timer
interval mismatch with router":
WAE# show alarm minor detail
Minor Alarms:
------------Alarm ID
--------------1 rtr_unusable
Module/Submodule
-------------------WCCP/svc051/rtr2.192.9.161
Instance
---------------
Jan 11 23:18:41.885 UTC, Communication Alarm, #000005, 17000:17003

WCCP router 2.192.9.161 unusable for service id: 51 reason: Timer interval
mismatch with router
<-----Check reason
<-----
On the WAE, check the configured failure detection timeout as follows:

WAE# show wccp services detail
Service Details for TCP Promiscuous 61 Service
Service Enabled
: Yes
Service Priority
: 34
Service Protocol
: 6
Application
: Unknown
Service Flags (in Hex)
: 501
Service Ports
:
0
0
:
0
0
Security Enabled for Service
: No
Multicast Enabled for Service
: No
Weight for this Web-CE
: 1
Negotiated forwarding method
: GRE
Negotiated assignment method
: HASH
Negotiated return method
: GRE
Negotiated HIA interval
: 2 second(s)
Negotiated failure-detection timeout : 30 second(s)
. . .
0
0
0
0
<-----Failure detection timeout configured
On the router, check if the IOS version supports variable failure detection timeout. If so, you can check the configured setting by
using the show ip wccp xx detail command, where xx is the WCCP service ID. There are three possible results:
WAE is using default failure detection timeout of 30 seconds and router is configured the same or does not support
variable timeout: The router output shows no details about the timeout setting. This configuration operates fine.
WAE is using non-default failure detection timeout of 9 or 15 seconds and router does not support variable timeout: State
field shows "NOT Usable" and the WAE cannot use the router. Change the WAE failure detection timeout to the default
value of 30 seconds by using the wccp tcp failure-detection 30 global configuration command.
WAE is using non-default failure detection timeout of 9 or 15 seconds and router supports variable timeout: Client
timeout field shows the configured failure detection timeout, which matches the WAE. This configuration operates fine.
If the WCCP farm is unstable due to link flapping, it could be because the WCCP failure detection timeout is too low.
This article describes how to troubleshoot an AppNav deployment.
Guide Contents
Main Article
Contents
1 AppNav Troubleshooting
1.1 In-Path (Inline) Interception
1.2 Off-Path (WCCP) Interception
1.2.1 Configuring and Verifying WCCP
Interception on the Router
1.2.2 Additional Information
1.3 Network Connectivity Troubleshooting
1.3.1 Passing Through Specific Traffic
1.3.2 Disabling an Inline ANC
1.3.3 Disabling an Off-Path ANC
1.4 AppNav Cluster Troubleshooting
1.4.1 AppNav Alarms
1.4.2 Central Manager Monitoring
1.4.3 AppNav CLI Commands for Monitoring
Cluster and Device Status
1.4.4 AppNav CLI Commands for Monitoring
Flow Distribution Statistics
1.4.5 AppNav CLI Commands for Debugging
Connections
1.4.6 Connection Tracing
1.4.7 AppNav Debug Logging
1.5 AppNav Packet Capture
Contents
116
AppNav Troubleshooting
Cisco WAAS AppNav simplifies network integration of WAN optimization and greatly reduces dependency on the intercepting
switch or router by using AppNav Controllers (ANCs) to distribute traffic among WAAS Nodes (WNs) for optimization using a
powerful class and policy mechanism. You can use WAAS nodes (WNs) to optimize traffic based on sites and/or applications.
This article describes how to troubleshoot AppNav.
NOTE: The AppNav feature was introduced in WAAS version 5.0.1. This section is not applicable to earlier WAAS versions.
In-Path (Inline) Interception

In inline mode, ANCs are positioned in the path of network traffic where they intercept packets and distribute them to WNs.

The interface configuration for an inline deployment assigns the interception and distribution roles to separate interfaces on the
Cisco AppNav Controller Interface Module. A bridge-group interface is required for interception and consists of two or more
physical or port-channel interfaces or one of each. The bridge-group interface does not have fail to wire capability; that is, it fails
open and traffic is not mechanically bridged after a device failure or loss of power. AppNav uses clustering to provide high
availability if the AppNav Controller Interface Module, the link path, or connectivity to the AppNav Controller Interface Module
is lost or there is a power failure.
Note: Bridge interfaces do not block bridge protocol data unit (BPDU) packets and in the case of redundant interfaces that create
loops, one of the interfaces is blocked by the Spanning Tree Protocol.
Troubleshooting inline interception consists of these steps:
Verify correct inline placement of the ANC by checking the network design. If necessary, use basic tools like ping and
traceroute, or Layer 7 tools or applications to confirm that the network traffic path is as expected. Check the physical
cabling of the ANC.
Verify that the ANC is set to inline interception mode.
Verify that the bridge-group interface is configured correctly.
The last two steps can be performed either in Central Manager or at the command line, though the Central Manager is the
preferred method and is described first.
In the Central Manager, choose Devices > AppNavController, then choose Configure > Interception > Interception
Configuration. Verify that the Interception Method is set to Inline.
In the same window, verify that a bridge interface is configured. If a bridge interface is needed, click Create Bridge to create it.
You can assign up to two member interfaces to the bridge group. You can use the VLAN Calculator to define the VLAN entries
based on include or exclude operations. Note that the bridge interface is not assigned an IP address.
Use the Alarm panel or the show alarm exec command to check if any bridge related alarms are raised on the device. A
bridge_down alarm indicates that one or more member interfaces in the bridge are down.
From the CLI, follow these steps to configure inline operation:
1. Set the interception method to inline:
wave# config
wave(config)# interception-method inline
2. Create the bridge-group interface:

wave(config)# bridge 1 protocol interception
3. (Optional) Specify the list of VLANs to intercept, if needed:

wave(config)# bridge 1 intercept vlan-id all
4. Add two logical/physical interfaces to the bridge-group interface:

wave(config)# interface GigabitEthernet 1/0
wave(config-if)# bridge-group 1
wave(config-if)# exit
wave(config)# interface GigabitEthernet 1/1
wave(config-if)# bridge-group 1
wave(config-if)# exit

You can use the show bridge exec command to verify the bridge interface operational status and see statistics for the bridge.
wave# show bridge 1
lsp: Link State Propagation
flow sync: AppNav Controller is in the process of flow sync
Member Interfaces:
GigabitEthernet 1/0
GigabitEthernet 1/1
Link state propagation: Enabled
VLAN interception:
intercept vlan-id all
<<< VLANs to intercept
Interception Statistics:
GigabitEthernet 1/0
Operation State
:
Down
Input Packets Forwarded/Bridged
:
16188
Input Packets Redirected
:
5068
Input Packets Punted
:
1208
Input Packets Dropped
:
0
Output Packets Forwarded/Bridged :
7843
Output Packets Injected
:
301
Output Packets Dropped
:
2
GigabitEthernet 1/1
Down(lsp)
<<< Down due to LSP
7845
0
605
0
21256
301
0
In the example above, the Gig 1/0 interface is down and the Gig 1/1 interface is also down due to link state propagation (LSP).
You might also see Down(flow sync), which means that the ANC is joining a cluster and synchronizing flow information with
other ANCs in the cluster. It keeps the interception path (bridge interface) shut for about two minutes until all ANCs are
synchronized so that existing flows can be distributed correctly.
The lower part of the output shows traffic statistics for the member interfaces.
Off-Path (WCCP) Interception

In WCCP mode, WCCP routers are positioned in the path of network traffic where they intercept packets and redirect them to
ANCs, which are located off-path. Since AppNav handles the interception processing, the intelligent flow distribution, and load
consideration between WAAS accelerators, the WCCP configuration on the routers is simplified significantly.
119
In the interface configuration for an off-path deployment, the interception and distribution roles can share the same interfaces on
the Cisco AppNav Controller Interface Module, but it is not required.
Troubleshooting off-path interception consists of these steps:
Verify correct placement of the WCCP routers to ensure they are in the path of traffic going to and from the optimized
hosts. You can use the show run or show wccp commands to verify that these are the same routers that are configured for
WCCP. If necessary, use basic tools like ping and traceroute, or Layer 7 tools or applications to confirm that all traffic
needing optimization passes through the WCCP routers.
Verify the WCCP configuration on the WAAS ANCs, using either the Central Manager (preferred) or the CLI.
Verify the WCCP configuration on the redirecting routers, using the router CLI.
To verify the WCCP configuration on the ANCs, in the Central Manager, choose Devices > AppNavController, then choose
Configure > Interception > Interception Configuration.
Verify that the Interception Method is set to WCCP.
Verify that the Enable WCCP Service check box is checked.
Verify that the Use Default Gateway as WCCP Router check box is checked or that the WCCP router IP addresses are
listed in the WCCP Router field.
Verify that the other settings such as the load balancing mask and redirect method are configured properly for your
deployment.
Check for any WCCP related alarms on the ANCs that are part of the router WCCP farm. On the Central Manager, click the
Alarms panel at the bottom of the screen or use the show alarm command on each device to view alarms. Correct any alarm
conditions by changing the configuration on the ANC or router, as needed.
From the CLI, follow these steps to configure WCCP operation:
1. Set the interception method to wccp.
120

wave# config
wave(config)# interception-method wccp
2. Configure the WCCP router list, which contains the IP addresses of the routers participating in the WCCP farm.
wave(config)# wccp router-list 1 10.10.10.21 10.10.10.22
3. Configure the WCCP service ID. A single service ID is preferred for AppNav, though two service IDs are supported.
wave(config)# wccp tcp-promiscuous 61
4. Associate the configured router list with the WCCP service.

wave(config-wccp-service)# router-list-num 1
5. Configure the WCCP assignment method (only the mask method is supported on an ANC). If you do not specify the
dst-ip-mask or src-ip-mask options, the default source IP mask is set to f and the destination IP mask is set to 0.
wave(config-wccp-service)# assignment-method mask
6. Configure the WCCP redirect method (the egress and return methods are set automatically to match the redirect method and are
not configurable for an ANC). You can choose L2 (the default) or GRE. L2 requires that the ANC has a Layer 2 connection with
the router and the router is also configured for Layer 2 redirection.
wave(config-wccp-service)# redirect-method gre
7. Enable the WCCP service.

wave(config-wccp-service)# enable
Verify WCCP interception on each ANC by using the show running-config command. The two examples below show the
running config output for L2 redirect and GRE redirect.
Show running-config wccp (for L2 redirect):
wave# sh run wccp
wccp router-list 1 10.10.10.21 10.10.10.22
wccp tcp-promiscuous service-pair 61
router-list-num 1
enable
exit
<<< L2 redirect is default so is not shown in running config
Show running-config wccp (for GRE):

wave# sh run wccp
wccp router-list 1 10.10.10.21 10.10.10.22
wccp tcp-promiscuous service-pair 61
router-list-num 1
redirect-method gre
enable
exit
<<< GRE redirect method is configured
Verify the WCCP status on each ANC by using the show wccp status command.
wave# show wccp routers
WCCP Interception :
121

<<< Shows Disabled if WCCP is not configured
<<< Shows Disabled if WCCP is not enabled
Configured State : Enabled

Operational State : Enabled
Services Enabled on this WAE:
TCP Promiscuous 61
<<< Shows NONE if no service groups are configured
Verify the routers that have responded to keep-alive messages in the WCCP farm by using the show wccp routers command.
wave# show wccp routers
Router Information for Service Id: 61
Routers Seeing this Wide Area Engine(2)
Router Id
Sent To
192.168.1.1
10.10.10.21
192.168.1.2
10.10.10.22
Routers not Seeing this Wide Area Engine
-NONERouters Notified of from other WAE's
-NONE-
<<< List of routers seen by this ANC
<<< List of routers not seen by this ANC
<<< List of routers notified of but not configured in router lis
Verify each ANCs' view of the other ANCs in the WCCP farm and the routers reachable by each of them by using the show wccp
clients command.
wave# show wccp clients
Wide Area Engine List for Service: 61
Number of WAE's in the Cache farm: 2
IP address = 10.10.10.31
Lead WAE = NO
192.168.1.1
192.168.1.2
Weight = 0
<<< Number of ANCs in the farm

<<< Entry for each ANC in the farm
IP address = 10.10.10.32
Lead WAE = YES
192.168.1.1
192.168.1.2
<<< List of routers seeing this ANC
Weight = 0
<<< YES indicates ANC is serving as the lead

<<< List of routers seeing this ANC
Verify that packets are being received by each ANC from the routers in the farm by using the show statistics wccp command.
Statistics for traffic that is received from, passed through, and sent to each router are shown. Cumulative statistics for all routers in
the farm are shown at the bottom. A similar command is show wccp statistics. Note that "OE" refers to the ANC devices here.
wave# sh statistics wccp
WCCP Stats for Router
Packets Received from Router
Bytes Received from Router
Packets Transmitted to Router
Bytes Transmitted to Router
Pass-thru Packets sent to Router
Pass-thru Bytes sent to Router
Redirect Packets sent to OE
Redirect Bytes sent to OE
:
:
:
:
:
:
:
:
:
10.10.10.21
1101954
103682392
1751072
2518114618
0
0
1101954
103682392
WCCP Stats for Router

Packets Received from Router
Bytes Received from Router
Packets Transmitted to Router
Bytes Transmitted to Router
Pass-thru Packets sent to Router
Pass-thru Bytes sent to Router
Redirect Packets sent to OE
:
:
:
:
:
:
:
:
:
10.10.10.22
75264
10732204
405193
597227459
0
0
75264
10732204
Cummulative WCCP Stats:

Total Packets Received from all Routers : 1177218
122

Total
Total
Total
Total
Total
Total
Total
Bytes Received from all Routers : 114414596

Packets Transmitted to all Routers : 2156265
Bytes Transmitted to all Routers : 3115342077
Pass-thru Packets sent to all Routers : 0
Pass-thru Bytes sent to all Routers : 0
Redirect Packets sent to OE : 1177218
: 114414596
Configuring and Verifying WCCP Interception on the Router

To configure WCCP interception on each router in the WCCP farm, follow these steps.
1. Configure the WCCP service on the router by using the ip wccp router command.
Core-Router1 configure terminal
Core-Router1(config)# ip wccp 61
2. Configure WCCP interception on the router LAN and WAN interfaces. You can configure the same service ID on both
interfaces if you are using a single service ID on the ANCs.
Core-Router1(config)# interface GigabitEthernet0/0
Core-Router1(config-subif)# ip address 10.20.1.1 255.255.255.0
Core-Router1(config-subif)# ip wccp 61 redirect in
Core-Router1(config-subif)# ip router isis inline_wccp_pod
Core-Router1(config-subif)# exit
Core-Router1(config)# interface GigabitEthernet0/1
Core-Router1(config-subif)# ip wccp 61 redirect in
Core-Router1(config-subif)# ip router isis inline_wccp_pod
Core-Router1(config-subif)# glbp 701 ip 10.19.1.254
Core-Router1(config-subif)# duplex auto
Core-Router1(config-subif)# speed auto
Core-Router1(config-subif)# media-type rj45
Core-Router1(config-subif)# exit
3. (Optional) Configure a tunnel interface if you are using generic GRE egress (only if you chose GRE for the ANC WCCP
redirect method).
Core-Router1(config)# interface Tunnel1
Core-Router1(config-subif)# no ip redirects
Core-Router1(config-subif)# tunnel source GigabitEthernet0/0.3702
Core-Router1(config-subif)# tunnel mode gre multipoint
Verify the WCCP configuration on each router in the farm by using the show wccp command.
Core-Router1 sh ip wccp 61 detail
WCCP Client ID:
10.10.10.31
Protocol Version:
2.00
State:
Usable
Redirection:
GRE
Packet Return:
GRE
Assignment:
MASK
Connect Time:
00:31:27
Redirected Packets:
Process:
0
CEF:
0
GRE Bypassed Packets:
Process:
0
Configuring and Verifying WCCP Interception on the Router
<<< ANC IP address
<<< Negotiated WCCP parameters

<<<
<<<
123

CEF:
Mask Allotment:
Assigned masks/values:
0
16 of 16 (100.00%)
1/16
Mask SrcAddr
DstAddr
SrcPort DstPort
---- ------------------- ------0000: 0x0000000F 0x00000000 0x0000 0x0000
Value
----0000:
0001:
0002:
0003:
0004:
0005:
0006:
0007:
0008:
0009:
0010:
0011:
0012:
0013:
0014:
0015:
SrcAddr
------0x00000000
0x00000001
0x00000002
0x00000003
0x00000004
0x00000005
0x00000006
0x00000007
0x00000008
0x00000009
0x0000000A
0x0000000B
0x0000000C
0x0000000D
0x0000000E
0x0000000F
DstAddr
------0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
SrcPort
------0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
DstPort
------0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
0x0000
<<< Configured mask
<<< Mask assignments
Additional Information
For additional information, see these documents:
WCCP Network Integration with Cisco Catalyst 6500: Best Practice Recommendations for Successful Deployments
Cisco Wide Area Application Services Web Cache Communication Protocol Redirection: Cisco Router Platform Support
Configuring Advanced WCCP Features on Routers, from the Cisco Wide Area Application Services Configuration Guide
Configuring WCCP on WAEs, from the Cisco Wide Area Application Services Configuration Guide
Network Connectivity Troubleshooting

When troubleshooting WAAS, it may be helpful to determine how the network is behaving with WAAS disabled. This is helpful
when traffic is not only failing to be optimized, but failing to get through at all. In these cases, it may turn out that the problem is
not WAAS related. Even in cases where traffic is getting through, this technique may help determine which WAAS devices
require troubleshooting.
Before testing Layer 3 connectivity, verify that the AppNav Controller Interface Module is connected to proper switch ports. If the
connected switch supports and has Cisco Discovery Protocol (CDP) enabled, run the command show cdp neighbors detail to
verify proper connectivity to the network switch.
Disabling WAAS may not be applicable in all cases. If some traffic is being optimized and some is not, it may be unacceptable to
disable WAAS, thereby disrupting the traffic that is being optimized successfully. In such a case, the interception ACL or the
AppNav policy can be used to pass through the specific type of traffic that is experiencing problems. For details, see the section
Passing Through Specific Traffic.
To disable WAAS, different steps are performed for inline mode than for off-path mode:
Inline mode requires putting the interception bridge in the pass-through state. For details, see the section Disabling an
Inline ANC.
Off-path mode requires disabling the WCCP protocol. For details, see the section Disabling an Off-Path ANC.
Additional Information
124

In AppNav environments, only the ANCs need to be disabled. WNs need not be disabled, since they do not participate in
interception.
Once WAAS is disabled, check network connectivity using standard methods.
Check Layer 3 connectivity using tools like ping and traceroute.
Check application behavior to determine upper layer connectivity
If the network is experiencing the same connectivity problems that it was with WAAS enabled, the problem is most likely
non-WAAS related.
If the network is working fine with WAAS disabled, but had connection problems with WAAS enabled, then there are
probably one or more WAAS devices requiring attention. The next step is to isolate the problem to specific WAAS
devices.
If the network has connectivity with and without WAAS enabled, but there is no optimization, then there are probably one
or more WAAS devices requiring attention. The next step is to isolate the problem to specific WAAS devices.
To check network behavior with WAAS enabled, follow these steps:
1. Reenable the WAAS functionality on the WAAS ANCs and, if applicable, the WCCP routers.
2. If you have determined that there is a WAAS-related problem, enable each AppNav cluster, and/or ANC individually, to isolate
it as a potential cause of the observed problem.
3. As each ANC is enabled, perform the same basic network connectivity tests as in earlier steps and note whether this specific
ANC seems to be operating correctly. Do not be concerned with individual WNs at this stage. The goal at this stage is to
determine which clusters, and which specific ANCs, are experiencing desired or undesired behavior.
4. As each ANC is enabled and tested, disable it again so that the next one can be enabled. Enabling and testing each ANC in turn
allows you to determine which ones require further troubleshooting.
This troubleshooting technique is most applicable in situations where the WAAS configuration appears to be not only failing to
optimize, but also causing problems with normal network connectivity.
Passing Through Specific Traffic
You can pass through specific traffic either by using an interception ACL or by configuring the AppNav policy for pass through.
Create an ACL that denies the specific traffic to be passed through and permits everything else. In this example, we want
to pass through HTTP traffic (dest port 80). Set the ANC interception access list to the defined ACL. Connections
destined for port 80 are passed through. You can use the show statistics pass-through type appnav command to verify
that pass-through is happening by checking that the PT Intercept ACL counters are incrementing.
anc# config
anc(config)# ip access-list extended pt_http
anc(config-ext-nacl)# deny tcp any any eq 80
anc(config-ext-nacl)# permit ip any any
anc(config-ext-nacl)# exit
anc(config)# interception appnav-controller access-list pt_http
Configure the ANC policy to pass through traffic matching specific classes.
class-map type appnav HTTP
match tcp dest port 80
policy-map type appnav my_policy
.
.
Network Connectivity Troubleshooting
125

.
class HTTP
pass-through
Disabling an Inline ANC

There are several ways to disable an inline ANC by putting it in pass-through state:
Set the interception bridge VLAN list to none. In the Central Manager, choose an ANC device, then choose Configure >
Interception > Interception Configuration. Select the bridge interface and click the Edit taskbar icon. Set the VLANs
field to the value "none".
Disable the service context containing the ANC. In the Central Manager, choose a cluster, then click the AppNav
Controllers tab, select an ANC, and click the Disable taskbar icon.
Apply an interception ACL with "deny ALL" criteria. This method is preferred. (The first two methods disrupt existing
optimized connections.) Define an ACL with the deny ALL criteria. In the Central Manager, choose an ANC device, then
choose Configure > Interception > Interception Access List, and choose the deny ALL access list in the AppNav
Controller Interception Access List drop-down list.
To disable interception with an ACL from the CLI, use the following commands:
anc# config
anc(config)# ip access-list standard deny
anc(config-std-nacl)# deny any
anc(config-std-nacl)# exit
anc(config)# interception appnav-controller access-list deny
Putting an ANC in pass-through state:

Disables WAAS interception, not the interfaces.
Disables all WAAS optimization.
Causes all traffic to pass through unaffected.
Disabling an Off-Path ANC
To disable an ANC that is running in off-path mode, disable the WCCP protocol for the ANC. You can do this action on the ANC
or on the redirecting router or both. On the ANC, you can disable or delete the WCCP services, or you can remove the
interception method or change it from WCCP to another method.
To disable WCCP interception, in the Central Manager, choose an ANC device, then choose Configure > Interception >
Interception Configuration. Uncheck the Enable WCCP Service check box or click the Remove Settings taskbar icon to remove
WCCP interception settings completely (they will be lost).
To disable WCCP interception from the CLI, use the following commands:
anc# config
anc(config)# wccp tcp-promiscuous service-pair 61
anc(config-wccp-service)# no enable
In some cases, there may be multiple ANCs receiving redirected traffic from the same router. For convenience, you may choose to
disable WCCP at the router, rather than the ANCs. The advantage is that you can remove multiple ANCs from a WCCP farm in a
single step. The disadvantage is that you cannot do this from the WAAS Central Manager.
To disable WCCP at the router, use the following syntax:
RTR1(config)# no ip wccp 61
RTR1(config)# no ip wccp 62
Passing Through Specific Traffic
<<< Only needed if you are using two WCCP service IDs
126

To reenable WCCP at the router, use the following syntax:
RTR1(config)# ip wccp 61
RTR1(config)# ip wccp 62
<<< Only needed if you are using two WCCP service IDs
At each WCCP router, verify that the ANCs you chose to disable are not showing up as WCCP clients. The following output is
displayed when the WCCP services have been deleted on the router.
RTR1# show ip wccp 61
The WCCP service specified is not active.
AppNav Cluster Troubleshooting

To troubleshoot an AppNav Cluster, you can use the following tools:
AppNav Alarms
Central Manager Monitoring
AppNav CLI Commands for Monitoring Cluster and Device Status
AppNav CLI Commands for Monitoring Flow Distribution Statistics
Connection Tracing
AppNav Debug Logging
AppNav Alarms
The Cluster Membership Manager (CMM) raises the following the alarms due to error conditions:
Degraded Cluster (Critical)--Partial visibility among ANCs. ANC will pass through new connections.
Convergence Failed (Critical)--ANC failed to converge on a stable view of ANCs and WNs. ANC will pass through new
connections.
ANC Join Failed (Critical)--ANC failed to join an existing cluster due to potential degradation of the cluster with the
ANC in it.
ANC Mixed Farm (Minor)--ANCs in the cluster are running different but compatible versions of the cluster protocol.
ANC Unreachable (Major)--A configured ANC is unreachable.
WN Unreachable (Major)--A configured WN is unreachable. This WN is not used for traffic redirection.
WN Excluded (Major)--A configured WN is reachable but excluded because one or more other ANCs cannot see it. This
WN is not used for traffic redirection (new connections).
You can see alarms in the Central Manager Alarms panel or by using the show alarms EXEC command on a device.
Note: The CMM is an internal AppNav component that manages the grouping of ANCs and WNs into an AppNav cluster
associated with a service context.
You can use the Central Manager to verify, monitor, and troubleshoot AppNav clusters. The Central Manager has a global view of
all registered WAAS devices in your network and can quickly help you locate most AppNav issues.
From the Central Manager menu, choose AppNav Clusters > cluster-name. The cluster home window displays the cluster
topology (including WCCP and gateway routers), overall cluster status, device status, device group status, and link status.
First, verify that the overall cluster status is operational.
Disabling an Off-Path ANC
127
Note that the ANC and WN icons shown in this diagram have the same device name because they reside on the same device. On
an ANC that is also optimizing traffic as a WN, these two functions are shown as separate icons on the topology diagram.
An orange triangle warning indicator is shown on any device for which the Central Manager may not have current information
because the device has not responded within the last 30 seconds (the device could be offline or unreachable).
You can get a detailed 360 degree status view of any ANC or WN device by hovering your cursor over the device icon. The first
tab displays alarms on the device. You should resolve any alarms that are inhibiting proper cluster operation.
Click the Interception tab to verify the device interception method on each ANC.
File:Waast-an-interceptiontab.png
If interception is down, the status appears as follows:
Click the Cluster Control tab to see the IP address and status of each device in the cluster that this ANC can see. Each ANC in the
cluster should have the same list of devices. If not, it indicates a configuration or network issue.
File:Waast-an-clustercontrol.png
If all ANCs cannot see each other, the cluster is nonoperational and all traffic is passed through due to the cluster's inability to
synchronize flows.
If all ANCs are connected but have different views of the WNs, the cluster is in a degraded state. Traffic is still distributed, but
only to the WNs that are seen by all ANCs.
Any WNs not seen by all ANCs are excluded.
Click the Interfaces tab to verify the state of the physical and logical interfaces on the ANC.
129
Look at the 360 degree view on each WN in the cluster and verify the green status of all accelerators in the Optimization tab. A
yellow status for an accelerator means that the accelerator is running but is unable to service new connections, for example
because it is overloaded or because its license has been removed. A red status indicates that the accelerator is not running. If any
accelerators are yellow or red, you must separately troubleshoot those accelerators. If the Enterprise license is missing, the
description reads System license has been revoked. Install the Enterprise license in the Admin > History > License Management
device page.
130
A split cluster results from connectivity issues between ANCs in the cluster. If the Central Manager can communicate with all
ANCs, it can detect a split cluster, however, if it cannot communicate with some ANCs, it cannot detect the split. The
"Management status is offline" alarm is raised if the Central Manager loses connectivity with any device and the device is shown
as offline in the Central Manager.
It is best to separate the management interfaces from the data interfaces to maintain management connectivity even if a data link is
down.
131
In a split cluster, each subcluster of ANCs independently distributes flows to the WNGs that it can see, but since flows between
the subclusters are not coordinated, it can cause reset connections and the overall cluster performance is degraded.
Check the Cluster Control tab of each ANC to see if one or more ANCs are unreachable. The "Service controller is unreachable"
alarm is raised if two ANCs that once could communicate with each other lose connectivity between themselves but this situation
is not the only cause of a split cluster so it is best to check the Cluster Control tab of each ANC.
132
If an ANC has a gray status light, it might be disabled. Check that all ANCs are enabled by clicking the AppNav Controllers tab
below the topology diagram. If an ANC is not enabled, its Enabled status is No. You can click the Enable taskbar icon to enable
an ANC.
Check the AppNav policy on each ANC that has anything other than a green status light. If you hover your cursor over the status
light on a device, a tool tip tells you the status or problem, if one is detected.
133
To check the defined policies, from the Central Manager menu, choose Configure > AppNav Policies and then click the Manage
button.
There should generally be a single policy assigned to all ANCs in the cluster. The default policy is named appnav_default. Select
the radio button next to a policy and click the Edit taskbar icon. The AppNav Policy pane shows you the ANCs to which the
selected policy is applied. If all ANCs are not shown with a checkmark, click the checkbox next to each unchecked ANC to assign
134

the policy to it. Click OK to save the changes.
After verifying the policy assignments, you can verify the policy rules in the AppNav Policies page that remains displayed. Select
any policy rule and click the Edit taskbar icon to change its definition.
An ANC could have a yellow or red status light if one or more policies are overloaded. Check the Overloaded Policies tab of the
360 degree device view to see a list of monitored policies that are overloaded.
135
If an ANC is joining the cluster, it is shown with a yellow status light and joining status.
136

The Interception tab of the 360 degree device view shows that the interception path is down due to the joining state. Interception is
held down until the ANC has synchronized its flow tables with the other ANCs and is ready to accept traffic. This process
typically takes no more than two minutes.
If you remove an ANC from a cluster, it is still shown for a few minutes in the topology diagram and as alive in the Cluster
Control tab, until all ANCs agree on the new cluster topology. It does not receive any new flows in this state.
137

Several CLI commands are useful for troubleshooting on an ANC:
show run service-insertion
show service-insertion service-context
show service-insertion appnav-controller-group
show service-insertion service-node-group all
show service-insertion appnav-controller ip-address
show service-insertion service-node [ip-address]
show service-insertion service-node-group group-name
Use these commands on a WN:
show run service-insertion
show service-insertion service-node
You can use the show service-insertion service-context command on an ANC to see the service context status and stable view of
the devices in the cluster:
ANC# show service-insertion service-context
Service Context
Service Policy
Cluster protocol ICIMP version
Cluster protocol DMP version
Time Service Context was enabled
Current FSM state
Time FSM entered current state
Last FSM state
Time FSM entered last state
Joining state
Time joining state entered
Cluster Operational State
Interception Readiness State
Device Interception State
Stable AC View:
10.1.1.1
Stable SN View:
10.1.1.1
Current AC View:
10.1.1.1
Current SN View:
10.1.1.1
:
:
:
:
:
:
:
:
:
:
:
:
:
:
test
appnav_default
1.1
1.1
Wed Jul 11 02:05:23
Operational
Wed Jul 11 02:05:55
Converging
Wed Jul 11 02:05:45
Not Configured
Wed Jul 11 02:05:23
Operational
Ready
Not Shutdown
<<< Active AppNav policy
2012
<<< Service context status
2012
2012
2012
<<< Status of this ANC
<<< Interception is not shut down by C

<<< Stable view of converged ANCs
10.1.1.2
<<< Stable view of converged WNs
10.1.1.2
10.1.1.2
10.1.1.2
10.1.1.3
If the Device Interception State field (above) shows Shutdown, it means that the CMM has shut down interception due to this
ANC not being ready to receive traffic flows. For example, the ANC could still be in the joining process and the cluster has not
yet synchronized flows.
The Stable View fields (above) list the IP addresses of the ANCs and WNs seen by this ANC device in its last converged view of
the cluster. This is the view used for distribution operations. The Current View fields list the devices advertised by this ANC in its
heartbeat messages.
You can use the show service-insertion appnav-controller-group command on an ANC to see the status of each ANC in the
ANC group:
ANC# show service-insertion appnav-controller-group
All AppNav Controller Groups in Service Context

Service Context
Service Context configured state
: test
: Enabled
AppNav Controller Group : scg

Member AppNav Controller count : 2
Members:
10.1.1.1
10.1.1.2
AppNav Controller
:
AppNav Controller ID
:
Current status of AppNav Controller
:
Time current status was reached
:
Joining status of AppNav Controller
:
Secondary IP address
:
:
Cluster protocol Incarnation Number
:
Cluster protocol Last Sent Sequence Number
:
Cluster protocol Last Received Sequence Number:
10.1.1.1
1
Alive
Wed Jul 11 02:05:23 2012
Joined
10.1.1.1
1.1
2
0
0
<<< Joining means ANC is still joining

<<< Source IP used in cluster protocol p
<<< ANC and WN devices advertised by th
Current AC View of AppNav Controller:

10.1.1.1
10.1.1.2
Current SN View of AppNav Controller:
10.1.1.1
10.1.1.2
AppNav Controller
:
AppNav Controller ID
:
Current status of AppNav Controller
:
:
Joining status of AppNav Controller
:
Secondary IP address
:
:
Cluster protocol Incarnation Number
:
Cluster protocol Last Sent Sequence Number
:
Cluster protocol Last Received Sequence Number:
Current AC View of AppNav Controller:
10.1.1.1
10.1.1.2
Current SN View of AppNav Controller:
10.1.1.1
10.1.1.2
<<< Status of this ANC
10.1.1.2 (local)
1
Alive
Wed Jul 11 02:05:23 2012
Joined
10.1.1.2
1.1
2
0
0
<<< local indicates this is the local A
<<< ANC and WN devices advertised by th
10.1.1.3
For a list of possible ANC statuses and joining statuses, see the show service-insertion command in the Cisco Wide Area
Application Services Command Reference.
You can use the show service-insertion service-node command on an ANC to see the status of a particular WN in the cluster:
ANC# show service-insertion service-node 10.1.1.2
Service Node:
: 20.1.1.2
Service Node belongs to SNG
: sng2
Service Context
: test
Service Context configured state
: Enabled
Service Node ID
:
Current status of Service Node
:
:
:
Cluster protocol incarnation number
:
Cluster protocol last sent sequence number
:
Cluster protocol last received sequence number:
AO state
-------AO
-tfo
State
----GREEN
1
Alive
Sun May 6 11:58:11 2011
1.1
1
1692060441
1441393061
For
--3d 22h 11m 17s
<<< WN is visible
<<< Overall/TFO state reported by WN
139

epm
cifs
mapi
http
video
nfs
ssl
ica
GREEN
GREEN
GREEN
RED
RED
GREEN
YELLOW
GREEN
3d 22h
3d 22h
3d 22h
3d 22h
11d 2h
3d 22h
3d 22h
3d 22h
<<< AO states reported by WN
11m 17s
11m 17s
11m 17s
14m 3s
2m 54s
11m 17s
11m 17s
11m 17s
You can use the show service-insertion service-node-group command on an ANC to see the status of a particular WNG in the
cluster:
ANC# show service-insertion service-node-group sng2
Service Node Group name
: sng2
Service Context
: scxt1
Member Service Node count
: 1
Members:
10.1.1.1
10.1.1.2
Service Node:
:
Service Node belongs to SNG
:
:
:
:
:
:
AO state
-------AO
-tfo
epm
cifs
mapi
http
video
nfs
ssl
ica
State
----GREEN
GREEN
GREEN
GREEN
RED
RED
GREEN
YELLOW
GREEN
For
--3d 22h
3d 22h
3d 22h
3d 22h
3d 22h
11d 2h
3d 22h
3d 22h
3d 22h
10.1.1.1
sng2
Excluded
Sun Nov 6 11:58:11 2011
1.1
1
1692061851
1441394001
12m 52s
12m 52s
12m 52s
12m 52s
15m 38s
4m 29s
12m 52s
12m 52s
12m 52s
Service Node:
:
Service Node belongs to WNG
:
:
:
:
:
:
AO state
-------AO
-tfo
epm
cifs
mapi
http
video
nfs
ssl
ica
State
----GREEN
GREEN
GREEN
GREEN
RED
RED
GREEN
YELLOW
GREEN
For
--3d 22h
3d 22h
3d 22h
3d 22h
3d 22h
11d 2h
3d 22h
3d 22h
3d 22h
<<< WN status
10.1.1.2
sng2
Alive
Sun Nov 6 11:58:11 2011
1.1
1
1692061851
1441394001
<<< WN status
12m 52s
12m 52s
12m 52s
12m 52s
15m 38s
4m 29s
12m 52s
12m 52s
12m 52s
140

<<< AO status for entire WNG
SNG Availability per AO

----------------------AO
Available
---------tfo
Yes
epm
Yes
cifs
Yes
mapi
Yes
http
No
video
No
nfs
Yes
ssl
No
ica
Yes
Since
----3d 22h
3d 22h
3d 22h
3d 22h
3d 22h
11d 2h
3d 22h
11d 2h
3d 22h
12m 52s
12m 52s
12m 52s
12m 52s
15m 38s
4m 29s
12m 52s
4m 29s
12m 52s
The first WN in the example above has a status of Excluded, which means that the WN is visible to the ANC but it is excluded
from the cluster because one or more other ANCs cannot see it.
The SNG Availability per AO table shows if each AO is able to service new connections. An AO is available if at least one WN in
the WNG has a GREEN status for the AO.
You can use the show service-insertion service-node command on a WN to see the status of the WN:
WAE# show service-insertion service-node
: 1.1
Service started at
: Wed Jul 11 02:05:45 2012
Current FSM state
: Operational
Time FSM entered current state
: Wed Jul 11 02:05:45 2012
Last FSM state
: Admin Disabled
Time FSM entered last state
: Mon Jul 2 17:19:15 2012
Shutdown max wait time:
Configured
: 120
Operational
: 120
Last 8 AppNav Controllers
-------------------------AC IP
My IP
e Last Heard
-------------------Reported state
-------------Accl
---TFO (System)
EPM
CIFS
MAPI
HTTP
VIDEO
NFS
SSL
ICA
<<< WN is responding to health probes
DMP Version
Incarnation
Sequence
Tim
-----------
-----------
--------
---
<<< TFO and AO reported states

State
----GREEN
GREEN
GREEN
GREEN
GREEN
GREEN
GREEN
RED
GREEN
For
--43d
43d
43d
43d
43d
43d
43d
43d
43d
Reason
-----7h
7h
7h
7h
7h
7h
7h
7h
7h
45m
44m
44m
44m
44m
44m
44m
44m
44m
8s
40s
41s
43s
45s
41s
44s
21s
40s
Monitored state of Accelerators

------------------------------TFO (System)
Current State: GREEN
Time in current state: 43d 7h 45m 8s
EPM
CIFS
<<< TFO and AO actual states
141

MAPI
HTTP
VIDEO
NFS
SSL
Current State: RED
Reason:
AO is not configured
ICA
The monitored state of an accelerator is its actual state but the reported state can differ because it is the lower of the system state
or the accelerator state.
For more information about troubleshooting optimization on a WN, see the Troubleshooting Optimization and Troubleshooting
Application Acceleration articles.
Several CLI commands are useful for troubleshooting policies and flow distribution on an ANC:
show policy-map type appnav policymap-name -- Shows the policy rules and hit counts for each class in the policy map.
show class-map type appnav class-name -- Shows the match criteria and hit counts for each match condition in the class
map.
show policy-sub-class type appnav level1-class-name level2-class-name -- Shows the match criteria and hit counts for
each match condition in a class map in a nested AppNav policy map.
show statistics class-map type appnav class-name -- Shows traffic interception and distribution statistics for a class
map.
show statistics policy-sub-class type appnav level1-class-name level2-class-name -- Shows traffic interception and
distribution statistics for a class map in a nested AppNav policy map.
show statistics pass-through type appnav -- Shows AppNav traffic statistics for each pass-through reason.
show appnav-controller flow-distribution -- Shows how a specific hypothetical flow would be classified and distributed
by an ANC, based on the defined policy and dynamic load conditions. This command can be useful to verify how a
particular flow will be handled on an ANC and what class it belongs to.
Use these commands on a WN to troubleshoot flow distribution:
show statistics service-insertion service-node ip-address -- Shows statistics for accelerators and traffic distributed to the
WN.
show statistics service-insertion service-node-group name group-name -- Shows statistics for accelerators and traffic
distributed to the WNG.
You can use the show statistics class-map type appnav class-name command on an ANC to troubleshoot flow distribution, for
example to determine why traffic might be slow for a particular class. This could be an application class map such as HTTP or, if
all traffic to a branch seems slow, it could be a branch affinity class map. Here is an example for the HTTP class:
142

ANC# show statistics class-map type appnav HTTP
Class Map
From Network to SN
------------------HTTP
Redirected Client->Server:
Bytes
3478104
Packets
42861
Redirected Server->Client:
Bytes
1154109763
Packets
790497
Connections
----------Intercepted by ANC
Passed through by ANC
Redirected by ANC
Accepted by SN
Passed through by SN (on-Syn)
Passed through by SN (post-Syn)
Passthrough Reasons
------------------Collected by ANC:
PT Flow Learn Failure
PT Cluster Degraded
PT SNG Overload
PT AppNav Policy
PT Unknown
From SN to Network
------------------
11588180
102853
9842597
60070
<<<
<<<
<<<
<<<
<<<
<<<
4
0
4
4
0
0
Packets
-------
Indicated by SN:
PT No Peer
...
Are connections being intercepted?

Passed-through connections
Are connections being distributed to W
Connections accepted by WNs
Connections might be passed through by
Connections might be passed through by
<<< Why is ANC passing through connections
Bytes
----0
0
0
0
0
0
0
0
0
0
<<<
<<<
<<<
<<<
<<<
Asymmetric connection; interception pr

ANCs cannot communicate
All WNs in the WNG are overloaded
Connection policy is pass-through
Unknown passthrough
<<< Why are WNs passing through connection

<<< List of WN pass-through reasons
The WN pass-through reasons in the Indicated by SN section increment only if pass-through offload is configured on a WN.
Otherwise, the ANC does not know that the WN is passing through a connection and does not count it.
If the Connections: Intercepted by ANC counter is not incrementing, there is an interception problem. You can use the WAAS
TcpTraceroute utility to troubleshoot the placement of the ANC in the network, find asymmetric paths, and determine the policy
applied to a connection. For details, see the section Connection Tracing.
AppNav CLI Commands for Debugging Connections
To debug an individual connection or set of connections on an ANC, you can use the show statistics appnav-controller
connection command to display the active connection list.
anc# show statistics appnav-controller connection
Collecting Records. Please wait...
Optimized Flows:
------------------------Client
Server
SN-IP
AC Owned
2.30.5.10:38111
2.30.5.10:38068
2.30.5.10:59861
2.30.5.10:59860
2.30.5.10:43992
2.30.5.10:59859
2.30.5.10:59858
2.30.5.10:59857
2.30.5.10:59856
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
2.30.1.10:5004
2.30.1.10:5003
2.30.1.10:445
2.30.1.10:445
2.30.1.10:5001
2.30.1.10:445
2.30.1.10:445
2.30.1.10:445
2.30.1.10:445
2.30.1.21
2.30.1.21
2.30.1.21
2.30.1.21
2.30.1.5
2.30.1.21
2.30.1.21
2.30.1.21
2.30.1.21
Passthrough Flows:
AppNav CLI Commands for Debugging Connections
143

------------------------Client
Server
2.30.5.10:41911
2.30.1.10:5002
Passthrough Reason
PT Flowswitch Policy
You can filter the list by specifying the client or server IP address and/or port options and you can show detailed statistics about
connections by specifying the detail keyword.
anc# show statistics appnav-controller connection server-ip 2.30.1.10 detail
Collecting Records. Please wait...
Optimized Flows
-------------------------Client: 2.30.5.10:55330
Server: 2.30.1.10:5001
AppNav Controller Owned: Yes
Service Node IP:2.30.1.5
Classifier Name: se_policy:p5001
Flow association: 2T:No,3T:No
Application-ID: 0
Peer-ID: 00:14:5e:84:41:31
<<<
<<<
<<<
<<<
<<<
<<<
This ANC is seeing activity on this connection

Connection is distributed to this SN
Name of matched class map
Connection is associated with dynamic app or session (MAPI and ICA only)?
AO that is optimizing the connection
ID of the optimizing peer
Client: 2.30.5.10:55331
Server: 2.30.1.10:5001
AppNav Controller Owned: Yes
Service Node IP:2.30.1.5
Classifier Name: se_policy:p5001
Flow association: 2T:No,3T:No
Application-ID: 0
Peer-ID: 00:14:5e:84:41:31
...
You can specify the summary option to display the number of active distributed and pass-through connections.
anc# show statistics appnav-controller connection summary
Number of optimized flows
= 2
Number of pass-through flows = 17
Connection Tracing
To assist in troubleshooting AppNav flows, you can use the Connection Trace tool in the Central Manager. This tool shows the
following information for a particular connection:
If the connection was passed through or distributed to a WNG
Pass-through reason, if applicable
The WNG and WN to which the connection was distributed
Accelerator monitored for the connection
Class-map applied
To use the Connection Trace tool, follow these steps:
1. From the Central Manager menu, choose AppNav Clusters > cluster-name, then choose Monitor > Tools > Connection
Trace.
2. Choose the ANC, the peer WAAS device, and specify connection match criteria.
3. Click Trace to display matching connections.
The WAAS TCP Traceroute is another tool not specific to AppNav that can help you troubleshoot network and connection issues,
including asymmetric paths. You can use it to find a list of WAAS nodes between the client and server, and the configured and
Connection Tracing
144

applied optimization policies for a connection. From the Central Manager, you can choose any device in your WAAS network
from which to run the traceroute. To use the WAAS Central Manager TCP Traceroute tool, follow these steps:
1. From the WAAS Central Manager menu, choose Monitor > Troubleshoot > WAAS Tcptraceroute. Alternatively, you can
choose a device first and then choose this menu item to run the traceroute from that device.
2. From the WAAS Node drop-down list, choose a WAAS device from which to run the traceroute. (This item does not appear if
you are in the device context.)
3. In the Destination IP and Destination Port fields, enter the IP address and port of the destination to which you want to run the
traceroute
4. Click Run TCPTraceroute to display the results.
WAAS nodes in the traced path are displayed in the table below the fields. You can also run this utility from the CLI with the
waas-tcptrace command.
The following log file is available for troubleshooting AppNav cluster manager issues:
Debug log files: /local1/errorlog/cmm-errorlog.current (and cmm-errorlog.*)
To set up and enable debug logging of the AppNav cluster manager, use the following commands.
You can enable detailed logging to the disk:
WAE(config)# logging disk enable
WAE(config)# logging disk priority detail
The options for cluster manager debugging (on 5.0.1 and later) are as follows:
WAE# debug cmm ?
all
enable
cli
enable
events
enable
ipc
enable
misc
enable
packets enable
shell
enable
timers
enable
all
CMM
CMM
CMM
CMM
CMM
CMM
CMM
CMM debugs
cli debugs
state machine events debugs
ipc messages debugs
misc debugs
packet debugs
infra debugs
state machine timers debugs
You can enable debug logging for the cluster manager and then display the end of the debug error log as follows:
WAE# debug cmm all
WAE# type-tail errorlog/cmm-errorlog.current follow
You can also enable debug logging for the flow distribution manager (FDM) or the flow distribution agent (FDA) with these
commands:
WAE# debug fdm all
WAE# debug fda all
145

The FDM determines where to distribute flows based on the policy and dynamic load conditions of the WNs. The FDA collects
WN load information. The following log files are available for troubleshooting FDM and FDA issues:
Debug log files: /local1/errorlog/fdm-errorlog.current (and fdm-errorlog.*)
Debug log files: /local1/errorlog/fda-errorlog.current (and fda-errorlog.*)
AppNav Packet Capture

A new packet-capture command is introduced to allow capturing data packets on interfaces on the Cisco AppNav Controller
Interface Module. This command can also capture packets on other interfaces, and can decode packet capture files. The
packet-capture command is preferred over the deprecated commands tcpdump and tethereal, which cannot capture packets on
the Cisco AppNav Controller Interface Module. See the Cisco Wide Area Application Services Command Reference for details on
command syntax.
Note: Either packet capture or debug capture can be active, but not both simultaneously.
Data packets sent between ANCs and WNs are encapsulated, as shown in the following diagram.
If you capture packets at points 1 or 4 in the diagram, they are unencapsulated. If you capture packets at points 2 or 3, they are
encapsulated.
Here is sample output for an encapsulated packet capture:
anc# packet-capture appnav-controller interface GigabitEthernet 1/0 access-list all
Packet-Capture: Setting virtual memory/file size limit to 419430400
Running as user "admin" and group "root". This could be dangerous.
Capturing on eth14
0.000000
2.58.2.11 -> 2.1.6.122
TCP https > 2869 [ACK] Seq=1 Ack=1 Win=65535 Len=0
4.606723
2.58.2.175 -> 2.43.64.21
TELNET Telnet Data ...
...
37.679587
2.58.2.40 -> 2.58.2.35
GRE Encapsulated 0x8921 (unknown)
37.679786
2.58.2.35 -> 2.58.2.40
GRE Encapsulated 0x8921 (unknown)
Here is sample output for an unencapsulated packet capture:

anc# packet-capture appnav-controller access-list all non-encapsulated
Capturing on eth14
0.751567
2.58.2.175 -> 2.43.64.21

1.118363
1.868756
...
2.58.2.175 -> 2.43.64.21

2.58.2.175 -> 2.43.64.21

Packet capture guidelines:

A packet-capture ACL is always applied to inner IP packet for WCCP-GRE and SIA encapsulated packets.
Packet capture is done on all ANC interfaces if the ANC interface for the packet capture is not provided.
Here is sample output for a packet capture on a WN interface:
anc# packet-capture interface GigabitEthernet 0/0 access-list 10
Capturing on eth0
0.000000
2.1.8.4 -> 2.64.0.6
0.000049
2.64.0.6 -> 2.1.8.4
0.198908
2.1.8.4 -> 2.64.0.6
TCP 18449 > telnet [ACK] Seq=2 Ack=2 Win=3967 Len=0
0.234129
2.1.8.4 -> 2.64.0.6
0.234209
2.64.0.6 -> 2.1.8.4
Here is an example of decoding a packet capture file:

anc# packet-capture decode /local1/se_flow_add.cap
Running as user "admin" and group "root". This could be dangerous. 1
100.1.1.2 -> 100.1.1.1
GRE Encapsulated SWIRE 2
0.127376
100.1.1.2 -> 100.1.1.1
GRE Encapsulated SWIRE
0.000000
You can specify a src-ip/dst-ip/src-port/dst-port for filtering the packets:

anc# packet-capture decode source-ip 2.64.0.33 /local1/hari_pod_se_flow.cap
Running as user "admin" and group "root". This could
3
0.002161
2.64.0.33 -> 2.64.0.17
TCP 5001 >
Win=5792 Len=0 MSS=1460 TSV=326296092 TSER=326296080
4
0.002360
2.64.0.33 -> 2.64.0.17
TCP 5001 >
Win=5792 Len=0 MSS=1406 TSV=326296092 TSER=326296080
be dangerous.
33165 [SYN, ACK] Seq=0 Ack=1
WS=4
33165 [SYN, ACK] Seq=0 Ack=1
WS=4
This article describes how to troubleshoot disk, RAID, and hardware issues.
Guide Contents
Main Article
AppNav Packet Capture
147

Contents
1 Checking the Disk Health
2 Disk Failures
3 RAID-5 Rebuilding and Synchronization
4 Firmware Upgrade for WAE-7341/7371/674
5 Boot Sequence Issue on WAE-7341/7371/674
6 Serial Port Disabled Issue on WAE-7341/7371/674
7 Boot Status Display
8 Replacing Disks on the WAE-612 with WAAS Versions 4.0.11 and Earlier
Releases
8.1 Disk01 Fails
8.2 Disk00 Fails and Disk01 has Problematic Status and is Marked Bad
8.3 Disk00 Fails and Disk01 is Not Marked Bad
Checking the Disk Health
You can check the disk health from the Central Manager or from the command line. On a Central Manager, choose the device to
check, and then choose Monitor > Disks to get a report on the disk status. For more details, see the Disks Report section in the
Cisco Wide Area Application Services Configuration Guide.
From the command line, you can use the show disks details command as follows:
WAE674# show disks details
RAID Physical disk information:

disk00: Online
disk01: Rebuilding
disk02: Online
J8WM2DTC
J8WMPV9C
J8WMYG6C
286102 MB
286102 MB
286102 MB
RAID Logical drive information:

Drive 1:
RAID-5 Critical
Enabled
(read-cache) Enabled (write-back)
Contents
<-------replaced disk is rebuilding
<-------RAID logical drive is rebuilding
148

Mounted file systems:
MOUNT POINT
TYPE
/sw
internal
/swstore
internal
/state
internal
/local/local1
SYSFS
.../local1/spool PRINTSPOOL
/obj1
CONTENT
/dre1
CONTENT
/ackq1
internal
/plz1
internal
DEVICE
/dev/sda1
/dev/sda2
/dev/sda3
/dev/sda6
/dev/data1/spool
/dev/data1/obj
/dev/data1/dre
/dev/data1/ackq
/dev/data1/plz
SIZE
991MB
991MB
7935MB
22318MB
991MB
248221MB
248221MB
991MB
2975MB
INUSE
892MB
733MB
176MB
139MB
32MB
130MB
130MB
32MB
64MB
FREE USE%
99MB 90%
258MB 73%
7759MB
2%
22179MB
0%
959MB
3%
248091MB
0%
248091MB
0%
959MB
3%
2911MB
2%
Disk encryption feature is disabled.
It is also useful to check the Predictive Failure Analysis (PFA) flag for RAID-5 disks by using the show disks tech-support
command. You will find the PFA flag at the end of the output. If the PFA flag is set to Yes, it indicates a predicted drive failure
and you should replace the disk. A critical alarm is also raised on the WAE.
Disk Failures
Disk failures are automatically detected by the system. Failed disks are automatically removed from service.
You can also shut down a disk for scheduled replacement by using the following commands:
For a RAID-5 system:
WAE674# disk disk-name disk01 replace
Controllers found: 1
Command completed successfully.
After you replace a disk on a RAID-5 system, the system rebuilds the logical RAID drive automatically.
For a RAID-1 system:
WAE7326# config
WAE7326(config)# disk disk-name disk01 shutdown
Device maybe busy while going offline ... please wait!
mdadm: set /dev/sdb1 faulty in /dev/md0
mdadm: set /dev/sdb2 faulty in /dev/md1
. . .
After you replace the disk on a RAID-1 system, use the following command to reenable the disk:
WAE7326# config
WAE7326(config)# no disk disk-name disk01 shutdown
RAID-5 Rebuilding and Synchronization

On a RAID-5 system, a RAID rebuild occurs when a hard disk is replaced, and a RAID synchronization occurs when WAAS is
installed onto a system by CD or when you run the disk recreate-raid EXEC command. During a RAID rebuilding or
synchronization process, which is managed by the RAID firmware, the hard disk LEDs blink constantly as the drives are set up
with the RAID configuration. The RAID array rebuilding or synchronization process can take up to 6 hours to complete on a
WAE-7371 with six 300-GB hard disks. Unfortunately, there is no indication of the time remaining.
Checking the Disk Health
149

CAUTION: Do not power cycle or remove a disk from the system when any of the drive LEDs are blinking because the disk can
be damaged.
If you do remove a disk during the RAID build process, reinsert the disk and wait up to 6 hours for the RAID build process to
complete.
There are slight differences in the RAID rebuild and synchronization, as follows:
Rebuild: Occurs after hard disk replacement. A disk and RAID failure alarm is raised on the device. The hard disk LEDs
blink rapidly and the LED on the replaced hard disk remains amber until the rebuild process is complete. The show disks
detail command shows the RAID physical disk that was replaced in the "Rebuilding" state and the RAID-5 logical disk in
the "Critical" state.
Synchronization: Occurs after system installation from the CD or RAID recreation. A RAID failure alarm is raised on the
device. The hard disk LEDs blink rapidly until the rebuild process is complete. The show disks detail command shows all
the RAID physical disks in the "Online" state and the RAID-5 logical disk in the "Impacted" state.
Firmware Upgrade for WAE-7341/7371/674

Ensure that your WAE-7341/7371/674 appliance has the recommended RAID controller firmware, 5.2-0 (15418). You can check
the RAID controller firmware with the show disks tech-support command as follows:
wae# show disks tech-support
Controllers found: 1
---------------------------------------------------------------------Controller information
---------------------------------------------------------------------Controller Status : Okay
Channel description : SAS/SATA
Controller Model : IBM ServeRAID 8k
Controller Serial Number : 40453F0
Physical Slot : 0
Installed memory : 256 MB
Copyback : Disabled
Data scrubbing : Disabled
Defunct disk drive count : 0
Logical drives/Offline/Critical : 1/0/0
--------------------------------------------------Controller Version Information
--------------------------------------------------BIOS : 5.2-0 (15418)
Firmware : 5.2-0 (15418)
<-----Firmware version
Driver : 1.1-5 (2449)
Boot Flash : 5.1-0 (15418)
--------------------------------------------------. . .
If your RAID controller firmware needs to be updated, obtain the recommended version from the Cisco software download
website (registered customers only) and upgrade the firmware as described in the documentation that accompanies the firmware.
Boot Sequence Issue on WAE-7341/7371/674

WAE-7341/7371/674 appliances are designed to boot from the internal compact flash storage device, not the hard disk. If the
WAE BIOS is inadvertently changed to boot from the hard disk, the WAE will fail to boot.
If you encounter this situation, change the BIOS back to boot from the compact flash to allow proper booting. For details on how
to change the startup sequence, see the chapter Using the Configuration/Setup Utility Program in the Cisco Wide Area Application

Engine 7341, 7371, and 674 Hardware Installation Guide. You can choose the option Load Default Settings to restore the
correct default settings, which include booting from the internal compact flash storage device.
Serial Port Disabled Issue on WAE-7341/7371/674

Sometimes after multiple power cycles during the device boot, the serial port becomes disabled.
If you encounter this situation, you should reenable the serial port. For details, see the chapter Using the Configuration/Setup
Utility Programin the Cisco Wide Area Application Engine 7341, 7371, and 674 Hardware Installation Guide. You can choose the
option Load Default Settings to restore the correct default settings, which include enabling the serial port.
Boot Status Display

To monitor the boot process on Cisco WAE and WAVE appliances, connect to the serial console port on the appliance as directed
in the Hardware Installation Guide.
Cisco WAE and WAVE appliances have video connectors that should not be used in normal operation. The video output is for
troubleshooting purposes only during the BIOS boot and stops displaying output as soon as the serial port becomes active.
If you are monitoring the video output, it may appear that the device has stopped booting when the output stops, but it is normal
for the video output to stop while the device continues booting.
Replacing Disks on the WAE-612 with WAAS Versions 4.0.11 and

Earlier Releases
If you are running WAAS version 4.0.11 or an earlier release on a WAE-612 device and a disk fails, the replacement procedure
varies, depending on the failure symptoms and the WAAS version that is in use. See the following sections, depending on the
failure symptoms:
Disk01 Fails
Disk00 Fails and Disk01 has Problematic Status and is Marked Bad
Disk00 Fails and Disk01 is Not Marked Bad
If you are running WAAS version 4.0.13 or a later release, see the Performing Disk Maintenance for RAID-1 Systems section in
the Cisco Wide Area Application Services Configuration Guide for the hot-swap disk replacement procedure.
NOTE: On a WAE-612 that is running any WAAS version from 4.0.13 through 4.0.19, which supports the hot-swap replacement
of drives, a problem may occur while replacing the drives while the unit is running. Occasionally, after a drive hot-swap
procedure, the WAE-612 may stop operating and require a reboot. To avoid this problem, upgrade your WAAS software to
version 4.0.19 or a later release.
Disk01 Fails
If the disk only in slot 01 (right slot) fails and disk00 is good, use the following procedures to replace the disk, depending on the
WAAS version on the device.
WAAS version 4.0.5 and earlier releases
1. Mark disk01 as bad.
2. Upgrade the WAAS software to WAAS 4.0.7.
3. Shut down the WAE and replace disk01.
Boot Sequence Issue on WAE-7341/7371/674
151

4. Start up the WAE.
5. Mark disk01 as good and reload the WAE.
WAAS versions 4.0.7 through 4.0.11
1. Mark disk01 as bad.
Disk00 Fails and Disk01 has Problematic Status and is Marked Bad
If disk00 fails and disk01 shows a status of Problematic, with an asterisk (*) next to the status (the asterisk means the disk is
marked bad), it means that disk00 has failed but disk01 is misclassified as bad and its partition table has been removed. In this
situation, all data will be lost after the disk replacement.
Use the following procedures to replace the disk, depending on the WAAS version on the device.
1. Mark disk00 as bad (if it is not already done).
4. Mark disk00 as good (disk01 should still show as bad) and reload the WAE.
5. Reinstall a later version of the WAAS software (for example, by using the copy ftp install command or other method).
You should see RAID rebuilds from disk00 to disk01.
1. Mark disk00 as bad (if it is not already done).
4. Mark both disk00 and disk01 as good and reload the WAE.
5. Reinstall a later version of the WAAS software (for example, by using the copy ftp install command or other method).

If disk00 fails and there is no asterisk (*) next to the status of disk01 (an asterisk means that the disk is marked bad), it means that
disk00 has failed and the partition table of disk01 is intact. The status of disk01 may show as Problematic or as something else. In
this situation, data will not be lost after disk replacement.
Use the following procedures to replace the disk, depending on the WAAS version on the device.
1. Upgrade the WAAS software to WAAS 4.0.7 or a later release.
2. Mark disk00 as good.
3. Shut down the WAE and remove disk00.
4. Move disk01 (right disk) to the disk00 position on the left side.
5. Insert a replacement disk into the disk01 slot.
Disk01 Fails
152

1. Mark disk00 as good.
2. Shut down the WAE and remove disk00.
3. Move disk01 (right disk) to the disk00 position on the left side.
4. Insert a replacement disk into the disk01 slot.
This article describes how to troubleshoot Serial Inline Cluster issues.
Guide Contents
Main Article
Contents
1 Checking for Connectivity Between the Serial
Peers
2 Verifying that the Serial Peers are Configured
Correctly
3 Verifying that a Serial Inline Cluster is
Operational
4 Detecting Serial Peer Configuration Mismatch
5 Troubleshooting MAPI Acceleration
5.1 Check EPM and MAPI Dynamic
Policies
153

5.2 Check Filtering and Auto-discovery
Statistics
5.3 Enabling Debug Logging
6 Troubleshooting Interception Access Lists
6.1 Connections Are Not Optimized
6.2 Connections Are Not Being Bypassed
as Expected
6.3 Enabling Debug Logging
NOTE: Serial inline clustering between non-optimizing peers and interception ACLs were introduced in WAAS version 4.2.1.
This section is not applicable to earlier WAAS versions.
Checking for Connectivity Between the Serial Peers

To see which devices are connected to the inline interfaces, use the show cdp neighbors command, as follows:
WAE#show cdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater
Device ID
Local Intrfce
Holdtme
Capability Platform Port ID
BBSw-R32-R62
Inline 1/1/lan
154
S I
WS-C3750G-Gig 3/0/17
BBSw-R32-R62
Inline 1/0/lan
154
S I
WS-C3750G-Gig 2/0/18
BBSw-R32-R62
Gig 1/0
126
S I
WS-C3750G-Gig 2/0/22
PLT-32-08-7301
Inline 1/1/wan
148
R
7301
Gig 0/2
PLT-32-08-7301
Inline 1/0/wan
147
R
7301
Gig 0/1
WAE-32-08-7341
Inline 1/1/wan
145
T H
OE7341
Inline 1/1/w
WAE-32-08-7341
Inline 1/0/wan
145
T H
OE7341
Inline 1/0/w
If the serial peers are separated by one or more switches, the peer will not show up in the output above.
Verifying that the Serial Peers are Configured Correctly

To verify that the serial peers are configured correctly, use the show peer optimization command, as follows:
WAE#show peer optimization
Configured Non-optimizing Peers:
Peer Device Id: 00:1a:64:c2:40:8c
Run this command on both the peers and make sure that each device shows up on the other correctly.
Use the show device-id command to check the device ID, as follows:
WAE#show device-id
System Device ID is: 00:21:5e:57:e9:d4
Verifying that a Serial Inline Cluster is Operational

Given the following topology example:
BR-WAE -----------WAN----------- DC-WAE2 -- DC-WAE1
or
BR-WAE1 -- BR-WAE2 -----------WAN----------- DC-WAE2 -- DC-WAE1

Normally, optimization should take place between the outermost WAEs, that is, BR-WAE and DC-WAE1, or BR-WAE1 and
DC-WAE1. To ensure this, verify the device IDs on the connections by using the show statistics connection command. The
PeerID on BR-WAE should indicate that it is optimizing with DC-WAE1 and the PeerID on DC-WAE1 should indicate that it is
optimizing with BR-WAE.
BR-WAE#show statistics connection
Historical Flows:
7552
7563
0
0
12891
100
3053
429

ConnID
786432
786435
786438
786440
Source IP:Port
190.190.3.175:19268
190.190.5.115:19283
199.199.3.0:58436
190.190.2.231:19312
Dest IP:Port
155.155.7.208:80
155.155.0.144:80
155.155.9.15:443
155.155.0.112:80
PeerID
00:21:5e:52:25:5c
00:21:5e:52:25:5c
00:21:5e:52:25:5c
00:21:5e:52:25:5c
Accel
THDL
THDL
TSDL
THDL
RR
00.0%
86.0%
00.0%
86.0%
The PeerID in the above output should match that of DC-WAE1.

All connections on DC-WAE2 should be in the "PT Intermediate" state.
If DC-WAE1 fails or goes into overload, new connections should be optimized between BR-WAE1 and DC-WAE2. You can
verify this by using the show statistics connection optimized command on DC-WAE2. Optimized connections should be seen on
DC-WAE2, with the peer ID of BR-WAE1 as the peer device.
If BR-WAE1 fails or goes into overload, there should not be optimization between DC-WAE2 and DC-WAE1. All connections
should be in the "PT Non-optimizing Peer" state on DC-WAE1 and "PT No Peer" on DC-WAE2. An example of the expected
show statistics connection command output follows:
DC-WAE1# sh stat conn
Historical Flows:
Local IP:Port
2.74.2.162:37116
2.74.2.18:80
Remote IP:Port
2.74.2.18:80
2.74.2.162:37116
0
0
0
0
0
100
1
1
Peer ID
ConnType
00:21:5e:27:ae:14 PT Non-optimizing Peer
00:21:5e:27:ae:14 PT Non-optimizing Peer
DC-WAE2# sh stat conn

Verifying that a Serial Inline Cluster is Operational
0
0
0
155

Historical Flows:
Local IP:Port
2.74.2.162:37116
2.74.2.18:80
Remote IP:Port
2.74.2.18:80
2.74.2.162:37116
0
0
100
1
1
Peer ID
N/A
N/A
ConnType
PT No Peer
PT No Peer
You can also use the Central Manager Connection Statistics report (Device > Monitor > Optimization > Connections Statistics)
to display device connection statistics in a table, as shown in Figure 1. The Peer IDs are indicated by device name.
Figure 1. Central Manager Device Connection Statistics Report
Detecting Serial Peer Configuration Mismatch

Serial peers must be configured so that each is designated as a non-optimizing peer with the other. If device A is configured as a
peer of B, but B is not configured as a peer of A, that is a mismatch. To discover a mismatch, you can use the Central Manager
My WAN > Configure > Peer Settings page, which reports on the status of all serial peers, as shown in Figure 2. All correctly
configured serial peers have a green check mark in the Mutual Pair column. Any devices without a green check mark are
incorrectly configured with a serial peer that is not also configured with the device as its serial peer.
Figure 2. Central Manager Peer Settings
To detect a serial peer configuration mismatch, you can also look for syslog messages such as the following:
%WAAS-SYS-4-900000: AD: Serial Mode configuration mismatch with peer_id=00:21:5e:27:a8:80
Detecting Serial Peer Configuration Mismatch
156

This error indicates that the serial peer configuration is not symmetrical on both of the peer devices.
Troubleshooting MAPI Acceleration

General MAPI AO troubleshooting is covered in the section "MAPI Accelerator" in the Troubleshooting Application Acceleration
article.
The following issues can occur with MAPI acceleration on serial inline clusters:
Outlook connection to the Exchange server is disconnected and restored
Outlook connection to the Exchange server is disconnected and stays that way
Outlook has problems establishing connections with the Exchange server
Outlook connection to the Exchange server is not optimized by WAAS (either it is in pass-through or no MAPI AO
optimization is done)
MAPI escaped connections because of the EPM policy timeout in the DC WAE
Check EPM and MAPI Dynamic Policies

Use the show policy-engine application dynamic command to check the EPM and MAPI dynamic policies, as follows:
WAE34#show policy-engine application dynamic
Allocations: 14

None
0
Clean-Up
0
Host->Host
0
Host->Local
0
Local->Host
0
Local->Any
0
Any->Host
3
Any->Local
0
Any->Any
0
Number:
1
Src: ANY:ANY Dst: 10.56.45.68:1067
Map Name: uuid1544f5e0-613c-11d1-93df-00c04fd7bd09
DM Ref Index: -None- DM Ref Cnt: 0
Number:
2
Src: ANY:ANY Dst: 10.56.45.68:1025
Map Name: uuidf5cc5a18-4264-101a-8c59-08002b2f8426
Number:
3
Src: ANY:ANY Dst: 10.56.45.68:1163
<------ EPM Policy
<------ EPM Policy

WAE33#show policy-engine application dynamic
Allocated: 32768 In Use: 2 Max In Use: 5 Allocations: 12
None
0
Clean-Up
0
Host->Host
1
Host->Local
0
Local->Host
0
Local->Any
0
Any->Host
1
Any->Local
0
Any->Any
0
Number:
1
Type: Host->Host (2) User Id: MAPI (5)
Src: 10.56.45.246:ANY Dst: 10.56.45.68:1163
Flags: REPLACE FLOW_CNT RSRVD_POOL REF_SRC_ANY_DM
DM Ref Index: 32767 DM Ref Cnt: 0
<------ MAPI Policy
Number:
2
Src: ANY:ANY Dst: 10.56.45.68:1163
Check Filtering and Auto-discovery Statistics

Check the output of the following commands to see if the relevant MAPI counters are incremented.
WAE#show stat auto-discovery
Auto discovery structure:
Allocation Failure:
Allocation Success:
Deallocations:
Timed Out:
.
.
.
Auto discovery Miscellaneous:
RST received:
SYNs found with our device id:
SYN retransmit count resets:
SYN-ACK sequence number resets (syncookies):
SYN-ACKs found with our device id:
SYN-ACKs found with mirrored options:
Connections taken over for MAPI optimization:
WAE#show stat filtering
Number of filtering tuples:
Number of filtering tuple collisions:
Packets dropped due to filtering tuple collisions:
Number of transparent packets locally delivered:
Number of transparent packets dropped:
Packets dropped due to ttl expiry:
Packets dropped due to bad route:
In ternal client syn packets dropped:
Check EPM and MAPI Dynamic Policies
0
12886550
12872245
1065677
87134
0
0
0
0
0
0
<----- MAPI & Serial Inline cluster statistic
44892
402
3
287133100
0
0
589
0
0
158

Syn packets received and dropped on estab. conn:
Syn-Ack packets received and dropped on estab. conn:
Syn packets dropped due to peer connection alive:
Syn-Ack packets dropped due to peer connection alive:
Packets recvd on in progress conn. and not handled:
Packets dropped due to peer connection alive:
Packets dropped due to invalid TCP flags:
Packets dropped by FB packet input notifier:
Packets dropped by FB packet output notifier:
Number of errors by FB tuple create notifier:
Number of errors by FB tuple delete notifier:
Dropped WCCP GRE packets due to invalid WCCP service:
Dropped WCCP L2 packets due to invalid WCCP service:
Number of deleted tuple refresh events:
Number of times valid tuples found on refresh list:
SYN packets sent with non-opt option due to MAPI:
Internal Server conn. not optimized due to Serial Peer:
Duplicate packets to synq dropped:
1
22016
0
4
0
1806742
0
0
0
0
0
0
0
0
0
0
<----- MAPI & Serial Inline Cluster statistic
0
8
Enabling Debug Logging

If looking at the dynamic policies and the filtering and auto-discovery statistics does not help, then enable debug logging so that a
technical support engineer can troubleshoot what is happening with MAPI accelerated connections in a serial inline cluster.
Enable debugging by running the following commands:
WAE#debug
WAE#debug
WAE#debug
WAE#debug
policy-engine connection
auto-discovery connection
filtering connection
connection acl
As always, disk logging needs to be enabled and the logging level for disk has to be set to debug.
Troubleshooting Interception Access Lists

This section describes how to troubleshoot the following problems related to interception ACLS:
Connections are not optimized
Connections are not being bypassed as expected
Connections Are Not Optimized

If connections are not optimized as expected, it could be due to the following causes.
1. The interface may be down. If it is an inline interface, all traffic will be bypassed in hardware. Use the following command to
check the interface status:
WAE#show interface inlinegroup 1/0
Interface is in intercept operating mode.
Standard NIC mode is off.
<------ Interface must be in intercepting mode
2. If the interface is up, check the state of the connections, and if they are in pass-through, check the reason using the following
command:
Check Filtering and Auto-discovery Statistics
159

WAE#show stat connection pass-through
Current Active Optimized TCP Preposition
Historical Flows:
Local IP:Port
Remote IP:Port
155.155.14.9:21
199.199.1.200:28624
155.155.13.92:21
199.199.1.147:26564
9004
9008
0
0
10294
100
2994
443
Flows:
Peer ID
N/A
N/A
ConnType
PT App Cfg
PT App Cfg
<----- Pass-through reason
3. If the reason appears as "PT Interception ACL", then it is due to the interception ACL denying the SYN packets.
You can look at the following output to drill down into the ACL to see which condition matched:
WAE#show ip access-list
Space available:
49 access lists
499 access list conditions
Standard IP access list test
1 permit any (1296 matches)
(implicit deny any: 0 matches)
total invocations: 1296
Interface access list references:
None Configured
Application access list references:
INTERCEPTION
Standard
Any IP Protocol
test
Connections Are Not Being Bypassed as Expected

If connections are not being bypassed as expected, make sure that the interception ACL configuration took effect using the
following command:
WAE#show ip access-list
Space available:
49 access lists
499 access list conditions
Standard IP access list test
1 permit any (1296 matches)
(implicit deny any: 0 matches)
total invocations: 1296
Interface access list references:
None Configured
Application access list references:
INTERCEPTION
Standard
Any IP Protocol
test
Check the hit counts from the above output to see if they are incrementing as expected.

If everything appears correct by using the commands above but there is still an issue, enable the following debug logging and look
for the policy-engine decision on the SYN packet of interest.
WAE#debug policy-engine connection
As always, disk logging needs to be enabled and the logging level for disk has to be set to debug.
Connections Are Not Optimized
160

This article describes how to troubleshoot vWAAS.
Guide Contents
Main Article
Contents
1 Identifying a vWAAS Device
2 Troubleshooting vWAAS Device
Registration
3 Verifying vWAAS Virtual Interfaces
4 Troubleshooting vWAAS Networking
5 Troubleshooting VPATH Interception
6 Troubleshooting Undersized Alarm
Virtual WAAS (vWAAS) implements a virtual WAAS appliance in VMware ESXi on a host server such as Cisco UCS.
NOTE: vWAAS was introduced in WAAS version 4.3.1. This section is not applicable to earlier WAAS versions.
Identifying a vWAAS Device
You can identify a vWAAS device from the Manage Devices page of the WAAS Central Manager. The device type appears as
OE-VWAAS for all types of vWAAS devices. The show version and show hardware CLI commands also show the device
Version as OE-VWAAS.
Figure 1. vWAAS Device Type
161
The model of the vWAAS device is determined from the number of CPUs and Maximum TCP Connections shown in the Device
Dashboard window when you select the device from the Manage Devices page. These two fields are displayed only for vWAAS
devices.
Figure 2. vWAAS Capabilities
The models are as follows:

vWAAS-750: 2 CPUs, 750 maximum TCP connections
Identifying a vWAAS Device
162

vCM-100N: 2 CPUs, 100 maximum nodes
vCM-2000N: 4 CPUs, 2000 maximum nodes
For vCM devices, you can use the show hardware command to determine the number of CPUs, which tells you which model of
vCM is installed.
Note: The vWAAS device shows 2 disks installed. The first, disk00, is 4 GB and emulates the flash storage in a physical WAAS
device. The second, disk 01, emulates the hard disk in a physical WAAS device and varies in size depending on the vWAAS
model.
The show tfo detail command also displays the maximum TCP connection limit:
vWAAS# show tfo detail
Policy Engine Config Item
------------------------State
Default Action
Connection Limit
Effective Limit
Keepalive timeout
Value
----Registered
Use Policy
750
750
3.0 seconds
<------ Max TCP connection limit
Troubleshooting vWAAS Device Registration

You must register each vWAAS device with the WAAS Central Manager for normal operation. If a vWAAS device is not
registered with the Central Manager, it shows the Not registered alarm:
vWAAS# show alarms
Critical Alarms:
---------------None
Major Alarms:
------------Alarm ID
--------------1 notregistered
. . .
Module/Submodule
-------------------vwaas/model
Instance
--------------<------Not registered
To register the vWAAS device with the Central Manager, use the cms enable global configuration command on the vWAAS
device:
vWAAS# config
vWAAS(config)# cms enable
Registering WAAS Application Engine...
Sending device registration request to
Please wait, initializing CMS tables
Successfully initialized CMS tables
. . .
management services enabled
Central Manager with address 2.75.16.100
You can verify the registration with the show cms info command:
vWAAS# show cms info
Device registration information :
Device Id
Device registered as
1730
= WAAS Application Engine
ala

Current WAAS Central Manager
= 2.75.16.100
Registered with WAAS Central Manager = 2.75.16.100
Status
= Online
Time of last config-sync
= Thu Aug 19 18:38:13 2010
CMS services information :
Service cms_ce is running
<----- Successful
registration
<----- CMS service is running
vWAAS device registration and deregistration is logged in the system message log with a line that begins with "vWAAS:". You
can view the system message log in the Central Manager by choosing Admin > Logs > System Messages.
Figure 3. vWAAS Registration Syslog Message
Verifying vWAAS Virtual Interfaces

Two virtual interfaces are available on vWAAS devices.
In the Central Manager device > Configure > Network > Network Interfaces page, the vWAAS interface type appears as Virtual
(Port Channel, Standby, Inline, and GigabitEthernet are not applicable), which is similar to the GigabitEthernet . Some of the
GigabitEthernet interface options, such as Port Channel, autosense, speed, mode, and standby, do not apply to virtual interfaces.
You can also see the virtual interfaces with the show running-config command:
VWAAS# show running-config interface
primary-interface Virtual 1/0
!
!
!
interface Virtual 1/0
ip address 10.104.227.25 255.255.255.128
exit
interface Virtual 2/0
shutdown
exit
Verifying vWAAS Virtual Interfaces
164

Additional details are available with the show interface virtual 1/0 or show interface virtual 2/0 commands.
To make interface configuration changes, you can use the Central Manager Network Interfaces page or the interface, ip, and
primary-interface configuration commands, as follows:
vWAAS# config
vWAAS(config)# interface virtual 1/0
vWAAS(config-if)# ip addr 10.10.10.15 255.255.255.0
vWAAS(config-if)# end
vWAAS# config
vWAAS(config)# ip default-gateway 10.10.10.1
vWAAS(config)# primary-interface virtual 1/0
vWAAS(config)# end
Troubleshooting vWAAS Networking

If you see no connections on the vWAAS device, check the vWAAS networking configuration in the vSphere Client. Is the
vWAAS device connected to the correct vSwitch?
Using the vSphere Client, you can trace vWAAS network connectivity from the device page. Identify which network label the
network adapter is connected to, determine the virtual switch that this network is connected to, and determine the physical NIC
that is a member of this virtual switch. Verify that the configuration is correct.
Also make sure the virtual switch VLAN settings are correctly configured to reach the network.
Verify the configured IP address, netmask, default gateway, and primary interface on the vWAAS device. For details, see the
previous section, "Verifying vWAAS Virtual Interfaces".
From the vWAAS device, ping the default gateway and Central Manager to make sure they are reachable.
Troubleshooting VPATH Interception

A vWAAS device can use VPATH or WCCP interception methods, but not both. To check if VPATH interception is enabled
from the Central Manager, choose the vWAAS device, then choose Configure > Interception > VPATH. If the Enable VPATH
box is checked, then it is enabled. WCCP must be disabled before VPATH can be enabled.
You can use the vn-service vpath global configuration command to enable or disable VPATH interception.
From the vWAAS device CLI, you can view VPATH status and statistics with the show statistics vn-service vpath command:
vWAAS# show statistics vn-service vpath
VPATH Statistics
*****************
Packet Statistics
----------------VPATH Enabled
VPATH Packet received
Optimized TCP Packets VPATH returned
WAAS Bypassed VPATH packets returned
VPATH encapsulated IP pkts(excluding TCP) returned
VPATH encapsulated Non-IP packets returned
VPATH Fragments received
VPATH Fragments returned
VPATH Packets returned when VPATH not configured
Non-VPATH Packets received
Error Statistics
-----------------
Troubleshooting vWAAS Networking
=
=
=
=
=
=
=
=
=
=
YES
4783472
918762
15537
0
26
0
0
0
810022
<-----Should be YES
165

VPATH intercepted packets dropped
VPATH Packet CRC failures
VPATH packets with unsupported Version
VPATH packets with wrong request type
=
=
=
=
0
0
0
0
To determine if VPATH is sending ARP requests, use the tcpdump arp command.
To display VPATH MAC address information for TCP flows, use the show statistics connection egress-methods command:
vWAAS# show statistics connection egress-methods
----------------------------------------------------------------------|
|
TUPLE
|
MATE
|
----------------------------------------------------------------------Local-IP:Port
10.104.227.25:443
10.104.227.28:36052
Remote-IP:Port
10.104.227.28:36052
10.104.227.25:443
Directed Mode
No
No
Egress method
IP Forwarding
IP Forwarding
VPATH mode
Yes
Yes
WCCP Service|Bucket
Tuple Flags
NON-WCCP|L2|
NON-WCCP|L2|
Intercepting Device (ID):
ID IP address
ID MAC address
ID IP address updates
0
0
ID MAC address updates 0
0
Egress Tunnel Dst
VPATH MAC Address
00:02:3D:83:B5:03
00:02:3D:83:B5:03
Memory address
0xffff8101078b1b80
0xffff8101078b1b80
. . .
<-----VPATH connection
<-----VPATH MAC address
Troubleshooting Undersized Alarm

If the proper memory and hard disk resources are not allocated to the vWAAS device, the following alarm is shown:
vWAAS# show alarms
Critical Alarms:
---------------None
Major Alarms:
------------Alarm ID
--------------1 undersized
. . .
Module/Submodule
-------------------vwaas/model
Instance
--------------memory
<-----Undersized alarm
You should never see this alarm if you are using valid OVA files to deploy vWAAS. If you see this alarm, delete the vWAAS VM
and redeploy it using a valid OVA file.
This article describes how to troubleshoot WAAS Express operation.
Troubleshooting VPATH Interception
Guide Contents
Main Article
166

Contents
1 Verifying WAAS Express Image Version
2 Verifying WAAS Express License
3 Verifying WAAS Enabled Interfaces
4 Verifying WAAS Optimized Connections
5 Verifying WAAS Optimized Data
6 Verifying WAAS Express Alarms
7 Verifying WAAS Express Peers
8 Offline Alarms
9 Verifying WAAS Express HTTPS Configuration
10 WAAS-Express - WAE - WAAS CM Compatibility
10.1 WAAS-Express Version 1.0,1.5
10.1.1 Known Issues
10.2 WAAS-Express Version 2.0.0
10.2.1 Known Issues
11 Unexpected WAAS-Express License Expiration
12 WAAS-Express and WAAS CM interaction issues
12.1 Symptom: WAAS-Express fail to register with the WAAS CM
12.1.1 Possible Cause #1: Connectivity issue
12.2 Symptom: WAAS CM shows WAAS-Express goes off-line after successful registration
12.2.1 Possible Cause #1: WAAS-Express device certificate changes
12.2.2 Possible Cause #2: Incorrect certificates or trustpoints are used
12.2.3 Possible Cause #3: Device authentication problem
12.2.4 Debug Information
12.3 Symtom: Mismtach Statistic between WAAS CM and WAAS-Express
12.3.1 Possible Cause #1: Clocks are not synchronized
13 Connections are not getting optimized
13.1 Symptom: Connections are getting pass-through
13.1.1 What could cause asymetric routing or dropped packets in the network
13.1.2 Information to be provided to development team:
14 Connections are not getting the desired optimization level
14.1 Symtom: Established connections do not get the desired or configured policy to use CIFS, SSL, or
Troubleshooting Undersized Alarm
167

HTTP-Express AO
14.2 Symtom: Expected connection optimization is THDL, but established connection has TDL
14.3 Symtom: Expected connection optimization is TCDL, but established connection has TDL
14.4 Symtom: Expected connection optimization is TSDL, but established connection has TDL
14.5 Expected connection optimization is TSHDL, but established connection has only TSDL or THDL
15 Symptom: Unexpected Connection Reset
15.1 Steps to troubleshoot
15.2 Information to be provided to the development team:
16 Router crash/tracebacks
16.1 Information to be provided to the development team:
17 Slow connection/degraded performance
17.1 Step to troubleshoot
18 Hung connections
18.1 Step to troubleshoot and collect information
19 SSL-Express Accelerator issues:
19.1 Having issues with SSL-Express Accelerator enable or disable
20 Moving WAAS-Express device between Device-Groups on CM
21 Other useful information
21.1 Statistics mismatch on WAAS-Express and WCM/WAE:
21.1.1 Information in addition to debugs and show commands, that needs to be provided to the
development team:
21.2 Troubleshooting router crash
21.3 Capturing packets on router
WAAS Express is WAAS functionality built into IOS running on a device such as a router. The WAAS Central Manager can
manage a WAAS Express device along with other WAAS devices in the WAAS network. This article describes how to
troubleshoot WAAS Express device operation.
Note: WAAS Express Central Manager support was introduced in WAAS version 4.3.1. This section is not applicable to
earlier WAAS versions.
Verifying WAAS Express Image Version

To verify the WAAS Express image version use the show waas status command on the WAAS Express router. To view the
WAAS Express image version from the WAAS Central Manager, choose My WAN > Manage Devices.
waas-express# show waas status
IOS Version: 15.1(20101018:232707)
WAAS Express Version: 1.1.0
. . .
<----- IOS version

<----- WAAS Express version
Verifying WAAS Express License

The WAAS Express license comes in two varieties: evaluation license (valid for 12 years) and permanent license. Use the show
waas status command on the WAAS Express device to display the license information.
IOS Version: 15.1(20101018:232707)
. . .
Contents
168

WAAS Feature License
License Type:
Evaluation total period:
Evaluation period left:
Evaluation
625 weeks 0
622 weeks 6
<----- Indicates an evaluation license

day
days
Verifying WAAS Enabled Interfaces

Use the show waas status command on the WAAS Express device to list the set of interfaces on which WAAS is enabled. This
command also displays the kind of optimization supported by the device. Some of the WAAS Express router models do not
support DRE.
IOS Version: 15.1(20101018:232707)
WAAS Enabled Interface
Policy Map
GigabitEthernet0/1
waas_global
<----- Interfaces on
GigabitEthernet0/2
waas_global
Virtual-TokenRing1
waas_global
Virtual-TokenRing2
waas_global
GigabitEthernet0/0
waas_global
Virtual-TokenRing10
waas_global
License Type:
Evaluation
625 weeks 0 day
622 weeks 6 days
DRE Status
: Enabled
<----LZ Status
: Enabled + Entropy
Maximum Flows
: 50
<----Total Active connections
: 0
<----Total optimized connections
: 0
<-----
which optimization is enabled
Indicates DRE is supported

Number of optimized connections supported
Total number of connections active
Total number of optimized connections
Note: WAAS should be enabled on WAN interfaces only. If connections, to be optimized, are routed over multiple WAN
interfaces, then, WAAS should be applied on all those WAN interfaces.
Note: If WAAS is enabled on a logical or virtual interface it need not be implemented on the corresponding physical
interface.
Verifying WAAS Optimized Connections

On the WAAS Express device, use the show waas connection command to list the set of optimized connections. Pass-through
connections are not included.
waas-express# show waas
ConnID
Source IP:Port
1999
64.103.255.217
1910
64.103.255.217
1865
64.103.255.217
status
:59211
:56860
:59206
Dest IP:Port
192.168.4.2
192.168.4.2
192.168.4.2
:1742
:61693
:23253
PeerID
0021.5e57.a768
0021.5e57.a768
0021.5e57.a768
Accel
TLD
TLD
TLD
<----- TFO, LZ and DRE are appl
To view similar information from the Central Manager, choose the WAAS Express device, then choose Monitor > Optimization
> Connections Statistics to see the Connections Summary Table.
Figure 1. Connections Summary Table
Verifying WAAS Express License
169
Verifying WAAS Optimized Data

On the WAAS Express device, use the show waas statistics application command to list the optimized data classified into each
application. The WAAS Express device does not show pass-through data. This data is used to generate the TCP related charts in
the WAAS Central Manager.
waas-express# show waas statistics application
Number of applications :
1
Application:
waas-default
TCP Data Volumes
Connection Type
Inbound
Opt TCP Plus
53001765483
Orig TCP Plus
0
Opt TCP Only
1165
Orig TCP Only
60
Internal Client
0
Internal Server
0
TCP Connection Counts
Connection Type
Active
Opt TCP Plus
50
Opt TCP Only
0
Internal Client
0
Internal Server
0
Outbound
41674120
87948683030
863
0
0
0
Completed
126
71
0
0
Pass Through Connection Counts

Connection Type
Completed
PT Asymmetric
0
PT Capabilities
0
PT Intermediate
0
PT_Other
0
Connection Reset:
0
Cleared connections 0
Verifying WAAS Express Alarms

On the WAAS Express device, use the show waas alarms command to list the alarms that are present in the device and their
status.
waas-express# show waas alarms
WAAS status:
enabled
Alarms

Connection limit exceeded:
Too many peers discovered:
WAAS license expired:
WAAS license revoked:
WAAS license deleted:
High CPU:
on
off
off
off
off
off
<----- on indicates this alarm is active. off indicates inactive
To view alarms for all devices from the Central Manager, choose My WAN > Alerts. In addition to the alarms listed above, an
alarm is raised if the clocks of the WAAS Express and WAAS Central Manager devices are not synchronized.
Verifying WAAS Express Peers

On the WAAS Express device, use the show waas statistics peer command to list the peer devices of the WAAS Express device.
waas-express# show waas statistics peer
Number of Peers :
1
Peer:
0021.5e57.a768
TCP Data Volumes
Connection Type
Inbound
Opt TCP Plus
597068158
Orig TCP Plus
0
Opt TCP Only
0
Orig TCP Only
0
Internal Client
0
Internal Server
0
TCP Connection Counts
Connection Type
Active
Opt TCP Plus
50
Opt TCP Only
0
Internal Client
0
Internal Server
0
Outbound
5212151
6867128187
0
0
0
0
Completed
0
0
0
0
Pass Through Connection Counts

Connection Type
Completed
PT Asymmetric
0
PT Capabilities
0
PT Intermediate
0
PT_Other
0
Connection Reset:
0
Cleared connections 0
Router#show waas statistics aoim
Total number of peer syncs:
Current number of peer syncs in progress:
Number of peers:
Number of local application optimizations (AO):
Number of AO discovery successful:
Number of AO discovery failure:
1
0
1
3
1
0
Local AO statistics
Local AO:
TFO
Total number of incompatible connections: 0
Version:
0.11
Registered:
Yes
Local AO:
HTTP
Version:
1.1
Registered:
Yes
Local AO:
SSL
Version:
1.0
Registered:
Yes
Verifying WAAS Express Alarms
171

Peer AOIM Statistics
Number of Peers :
1
Peer:
Peer IP:
Peer Expiry Time:
Peer Compatible:
Peer active connections:
Peer Aoim Version:
Peer sync in progress:
Peer valid:
Peer Software Version:
Peer AOs:
Peer AO:
Compatible:
Version:
Peer AO:
Compatible:
Version:
Peer AO:
Compatible:
Version:
0027.0d79.c215
20.0.0.2
00:00:02
Yes
0
1.0
No
Yes
4.4.3(b4)
<--- Peer ID
<--- Peer IP
TFO
Yes
0.20
HTTP
Yes
1.4
SSL
Yes
1.0
Router#show waas statistics dre peer

DRE Status:
Enabled
Current number of connected peers

Current number of active peers
0
1
Peer-ID
Hostname
IP reported from peer
Peer version
0027.0d79.c215
waasx1-b-wae.cisco.com
20.0.0.2
4.4.3(b4)
Cache:
Cache in storage
Age
0 B
00:00:00
AckQ:
AckQ in storage
0 B
WaitQ:
WaitQ in storage
WaitQ size
0 B
0 B
Sync-clock:
Local-head
Local-tail
Remote-head
Curr-sync-clock
0 ms
0 ms
18609143000 ms
24215235228 ms
Encode Statistics
DRE msgs:
R-tx total:
R-tx chunk-miss:
R-tx collision:
Bytes in:
Bytes out:
Bypass bytes:
Compression gain:
Decode Statistics
DRE msgs:
Bytes in:
Bytes out:
Bypass bytes:
Compression gain:
Verifying WAAS Express Peers
<--- Peer ID
<--- Peer hostname
<--- Peer IP
1
0
0
0
0
0
178
0%
4
299
277
51
0%
172

Nacks generated:
To view similar information from the Central Manager, choose Monitor > Topology.
Offline Alarms
The WAAS Express device may go to an offline state in the Central Manager because of the following issues:
Central Manager does not have WAAS Express device credentials.
Credentials are not configured for this WAAS Express device in the Central Manager. The WAAS Central Manager needs
the WAAS Express username and password to communicate with the WAAS Express device. You can configure
credentials in the Central Manager by choosing My WAN (or a WAAS Express device or device group) > Admin >
WAAS Express Credentials.
Authentication failed while communicating with WAAS Express device.
The Central Manager is not able to communicate with the WAAS Express because wrong credentials are configured. You
can configure credentials in the Central Manager by choosing My WAN (or a WAAS Express device or device group) >
Admin > WAAS Express Credentials.
SSL Handshake failed while communicating with WAAS Express devcie.
The WAAS Express device certificate is changed and the same certificate is not imported for this device in the Central
Manager. To reimport the WAAS Express device certificate, choose the WAAS Express device, then choose Admin >
Certificate.
No route to WAAS Express device.
The Central Manager is not able to reach the WAAS Express Device. Configure the correct WAAS Express management
IP address by choosing the WAAS Express device, then choosing DeviceName > Activation.
Connection is refused by WAAS Express device.
The HTTPS server port configured on the WAAS Express device is not the same as the port shown in the Central
Manager DeviceName > Activation page. Configure the correct WAAS Express HTTPS server port in this page.
WAAS support is not available on WAAS Express device.
The WAAS Express device is downgraded to an IOS image version with no WAAS support. Install an IOS image with
WAAS support.
Connection timed out while communicating with WAAS Express device.
The WAAS Express device is taking more than 30 seconds to respond to the Central Manager. It could be because the
WAAS Express device is overloaded or the network is slow.
License is expired on WAAS Express device.
The Evaluation license on the WAAS Express device is expired. Install a Permanent license by using the WAAS Express
license install command.
SSL connection closed incorrectly while communicating with WAAS Express device.
Offline Alarms
173

The WAAS Express device and Central Manager are using the cipher rc4-128-md5 for SSL communication. Sometimes
the Central Manager fails to decrypt the SSL data sent by the WAAS Express. Configure the ciphers 3des-ede-cbc-sha,
des-cbc-sha, and rc4-128 by using the WAAS Express command ip http secure-ciphersuite 3des-ede-cbc-sha
des-cbc-sha rc4-128-sha.
Failed to check the status of WAAS Express device.
The Central Manager is not receiving configuration status from the WAAS Express device. Contact Cisco TAC for
assistance troubleshooting.
Management Status is offline.
If you see this error message, contact Cisco TAC for assistance troubleshooting.
Verifying WAAS Express HTTPS Configuration

To verify the HTTPS server configuration on the WAAS Express device, use the show ip http server secure status command.
waas-express# show ip http server secure status
HTTP
HTTP
HTTP
HTTP
HTTP
HTTP
secure
secure
secure
secure
secure
secure
server
server
server
server
server
server
status: Enabled
port: 443
ciphersuite: 3des-ede-cbc-sha des-cbc-sha rc4-128-sha
client authentication: Disabled
trustpoint: local
active session modules: ALL
WAAS-Express - WAE - WAAS CM Compatibility

WAAS-Express Version 1.0,1.5
This version of WAAS-Express supports the transport optimization which includes TFO, LZ, and DRE.
WAAS-Express version 1.0 is introduced in IOS software release 15.1(3)T1
WAAS-Express version 1.5 is introduced in IOS software release 15.1(4)M. In addition to optimization, this release adds support
for embedded monitoring capability called Performance Agent (PA). For more information on PA, please see PA page on CCO
Recommended WAAS-Express IOS image: 15.1(3)T1
Recommended WAE version: >= 4.3.1
Recommended WCM version: 4.4.5a
Known Issues
IOS version WAE version WAAS CM version
15.1(3)T1
5.0.1
4.4.5a
Known Issues
Connections originating on data-center side will not be optimized:
CSCtz82646
WAAS-Express Version 2.0.0
This version of WAAS-Express, in addition to supporting transport optimization, also support selected applicaiton optimization,
specifically HTTP Express, SSL Express, and CIFS Express AO.
Recommended WAAS-Express IOS image: 15.2(4)M1

Recommended WAE version: 5.0.1
Recommended WCM version: 5.0.1
Verifying WAAS Express HTTPS Configuration
174

Known Issues
IOS
version
WAE
version
WAAS CM
version
Known Issues
15.2(4)M1
< 4.4.3c
< 5.0.1
HTTP-Express Accelerator requires 4.4.3c or later. Connections will not have http
optimization, however, will have TDL.
15.2(4)M1
< 5.0.1
< 4.4.5a
Missing classifier name in connection statistics seen on WCM.
< 5.0.1
CSCub21189: Policy-map changes not properly sync'ed with WAAS-Express device

CSCtw50988: SMB: connection reset while downloading a file
CSCtr07216: Transaction with invalid hdr not handled correctly in WAAS-X <->
WAE case
CSCua49764: Https created WExp certificate - WExp went to offline after upgrade
< 5.0.1
CSCub21189: Policy-map changes not properly sync'ed with WAAS-Express device

WAE case
CSCua49764: Https created WExp certificate - WExp went to offline after upgrade
< 5.0.1
CSCtx82427: IOS-WAAS: SSL connection reset at end of transfer (EOT)

CSCtz08485: Incompatible HTTP-AO detection failure
(%WAAS-3-WAAS_LZ_CONN_ABORT)
CSCtu19564: Crash observed in dt21 with Waas+VPN+ZBFW+NAT+NETFLOW
CSCtz85134: WAAS Express SSL-Express changes self-signed trustpoint after
reload
CSCua22313: HTTPS page dont get displayed with IE6 conn optim by WAAS
Express 2.0
CSCty04359: Manually created WExp certificate - after upgrade Wexp went to
offline
WAE case
15.2(4)M1
15.2(3)T1
15.2(3)T
< 5.0.1
< 5.0.1
< 5.0.1
Unexpected WAAS-Express License Expiration

The WAAS-Express license is active in show license. However, WAAS-Express license is expired in show waas status.
This is potentially a known bug, CSCtw86624. Verify this by issuing following show commands. WAAS CM thinks that
license is expired and shows the device as offline. However, the connections should be optimized, since based on the
license, the feature is active.
Solution: Upgrade to a recommended WAAS-Express Version 2 image - 15.2(4)M1 or install a permanent license.
Router#sh license | beg WAAS_Express
Index 12 Feature: WAAS_Express
Period left: Life time
License Type: RightToUse
License State: Active, In Use <---- License is Active
License Count: Non-Counted
License Priority: Low
Router#show waas status
IOS Version: 15.2(2.9)T
WAAS Enabled Interface
GigabitEthernet0/1
Known Issues
Policy Map
waas_global
175

License Type:
Evaluation
0 seconds <---- License is expired.
0 seconds
WAAS-Express and WAAS CM interaction issues

For a step-by-step detailed WAAS-Express registration process, please check the following document: WAAS Express
Deployment Guide
Symptom: WAAS-Express fail to register with the WAAS CM

Possible Cause #1: Connectivity issue
Can the WAAA-Express router reaches WAAS CM?
Troubleshoot steps: Verify that WAAS CM is ping?able from the router. In addition, if WAAS-Express router is behind
NAT and/or firewall, a static NAT entry and/or firewall permit rule are required to allow WAAS CM to connect to
WAAS-Express HTTPS server. To manage WAAS-Express devices behind NAT/Firewall, WAAS CM allows user to
manually change/specify address of WAAS-Express device for WAAS CM to use. User can change the address from the
device activation page.
Solution: Check route and network topology to make sure WAAS CM is reachable from the router and vice versa, please
enable the following debugs on WAAS-Express device.
If required, check following debugs to figure out if SSL handshake during registration is failing:
debug ip http all
debug ssl openssl
debug ssl openssl
debug ssl openssl
debug ssl openssl
errors
ext
msg
states
Note: The above ssl debugs are verbose.

Did the certificate change upon router reload?
Verify this by comparing the WAAS-Express router certificate expiration date stored on the WAAS CM. Navigate to this
page from the WAAS-Express device page, Admin->Certificate. Compare the certificate information with the output of
show crypto pki certificate output on the WAAS-Express router. If there is any mismatch, it is very likely the certificate
ia re-generated.
Solution: Upgrade to 15.2(3)T1 or 15.2(4)M1 and later
Symptom: WAAS CM shows WAAS-Express goes off-line after successful registration

Possible Cause #1: WAAS-Express device certificate changes
Verify this by comparing the WAAS-Express router certificate expiration date stored on the WAAS CM. Navigate to this
page from the WAAS-Express device page, Admin->Certificate. Compare the certificate information with the output of
show crypto pki certificate output on the WAAS-Express router. If there is any mismatch, it is very likely the certificate
ia re-generated.
Unexpected WAAS-Express License Expiration
176

Issue show run | include crypto pki trustpoint. Non-persistent trustpoint naming is in the format of
TP-self-signed-xxxxxxxxxx.
router#show run | include crypto pki trustpoint
crypto pki trustpoint TP-self-signed-4046801426 <-- Indicate this is non-persistent trustpoint
Solution: Follow this link to create persistent trustpoint.

There are serveral instances where the certificate could be re-generated but the main reason is trustpoing is created as
non-persistent. If you enable SSL Express AO with 15.2(3)T, you could also potentially hit CSCtz85134.
Solution: Upgrade to 15.2(4)M1 and re-create persistent trustpoint. Delete the certificate from WAAS CM and
re-register.
Was this an upgrade from 15.1(3)T to 15.2(3)T?
In 15.2(3)T, there is a mandatory config within the crypto pki trustpoint, which requires rsa-keypair to be configured. If
this config does not present before upgrade, this could potentially cause the router not be able to detect the trustpoint. This
will cause HTTPS connectivity to fail. This problem is documented in CSCty04359.
Solution: Remove the trustpoint and re-create. Delete the certificate from WAAS CM and re-register.
Possible Cause #2: Incorrect certificates or trustpoints are used
Does the router have multiple trustpoints configured?
During WAAS CM registration, WAAS-Express router selects the trustpoint which it uses for sending certificate to
WAAS CM. This may be different trustpoint from what the local HTTPS server on the WAAS-Express router uses.
Solution: Verify that the same trustpoing is configured in ip http secure-trustpoint <trustpoint_name> and ip http-client
secure-trustpoint <trustpoint_name>
Possible Cause #3: Device authentication problem
Is authentication failing?
Verify that you can login to the WAAS-Express router, by directing your browser to WAAS-Express router using HTTPS
and attempt the authentication manually.
Solution: Verify that manual authentication is successful.
Debug Information
If you believe you are running into certificate related issuses, please provide below information to support team.
Router#show crypto pki trustpoints status

State:
Keys generated ............. Yes (General Purpose, non-exportable) <--- check if this shows ?No? for the self-sign
Issuing CA authenticated ....... Yes <--- check if this shows ?No? for the self-signed certificate
Certificate request(s) ..... Yes <--- check if this shows ?No? for the self-signed certificate
Router#show crypto pki trustpoints status
Trustpoint TP-self-signed-2330253483:
Issuing CA certificate configured:
Subject Name:
cn=IOS-Self-Signed-Certificate-2330253483
Possible Cause #1: WAAS-Express device certificate changes
177

Fingerprint MD5: 3F5E9EB4 6BD680FE 8A1C1664 0939ADCB <--- Check fingerprints before and after upgrade
Fingerprint SHA1: DFF10AF4 83A90CAD 71528B3C CCD4EF0C E338E501
Router General Purpose certificate configured:
Subject Name:
cn=IOS-Self-Signed-Certificate-2330253483
Fingerprint MD5: 3F5E9EB4 6BD680FE 8A1C1664 0939ADCB
Fingerprint SHA1: DFF10AF4 83A90CAD 71528B3C CCD4EF0C E338E501
State:
Keys generated ............. Yes (General Purpose, non-exportable)
Issuing CA authenticated ....... Yes
Certificate request(s) ..... Yes
Router#show crypto pki certificates
?
Validity Date:
start date: 20:16:14 UTC May 26 2011 <--- Check whether these dates are valid
end
date: 20:16:14 UTC May 24 2016
?
Provide outputs for following commands:
show
show
show
show
show
show
show
crypto pki certificates storage

crypto pki trustpoints
crypto key storage
crypto key pubkey-chain rsa
crypto key mypubkey all
crypto key mypubkey rsa
ip http server all
Symtom: Mismtach Statistic between WAAS CM and WAAS-Express

Possible Cause #1: Clocks are not synchronized
WAAS CM and WAAS-Express clock need to be in sync and hence configuring NTP server to sync clocks is highly
recommended.
Are clock-mismatch messages seen on WAAS CM?
Verify that the router clock is the same as WAAS CM clock in UTC format. Remove any timezone and
summertime configuration and compare the UTC time between WAAS CM and WAAS-Express router.
Known DDTSs: CSCtz32667, CSCtz97973, CSCtk74707, CSCtl24210. Identify if your problem resembles any
of these DDTS and follow the workaround suggested in the DDTS.
Solution: Configure NTP and verify that all devices' clock are synchronized. Follow the workaround in the
DDTS mentioned above, or upgrade to the latest 15.2(4)M1 or later.
Connections are not getting optimized

Symptom: Connections are getting pass-through
Validate pass-through statistics/reason using show waas statistics pass-through. Look for the reason of why connections are
getting pass-through.
Router#show waas statistics pass-through
Pass Through Statistics:
Overall:
0
No Peer:
0
Rejected due to Capabilities:
0
Rejected due to Resources:
0
Debug Information
178

Interface Application config:
Interface Global config:
Assymmetric setup:
Peer sync was in progress:
IOS WAAS is intermediate router:
Internal error:
Other end is in black list:
AD version mismatch:
Incompatable AO:
Connection limit exceeded:
AOIM peertable full:
AOIM multiple sync request passthrough:
Others:
0
0
0
0
0
0
0
0
0
0
0
0
0
<---- Traffic classified for pass-through?

<---- Asymmetric route in the setup?
<---- Incompatible peer?
Check auto-discovery statistics (and/or use auto-discovery debugs).

Use the following command to check the reason '''show waas statistics auto-discovery'''
Enable following debugs for more information:
debug
debug
debug
debug
debug
waas
waas
waas
waas
waas
infra error
infra events
auto-discovery error
auto-discovery event
auto-discovery op <---- Verbose debug
If the counter for Interface Application Configincrements, it is likely your policy is configured to pass-through this
particulate connection. Check your WAAS policy on both WAAS-Express and its peer.
Solution: Check and validate your optimization policy. Use below debug to discover if traffic is marked as pass-through
in the policy.
show policy-map type waas interface
debug waas infra events
If the counter for Interface Global Config increments, this could be caused by asymetrical routing in your network. This
is the case where WAAS-Express or its peer does not see both directions of the TCP traffic. This could be caused by true
asymetrical routing in the network, or could be caused by some packets are getting dropped by devices in the traffic path
(ACL, firewall, etc.)
Solution: Check for asymetric routing of dropped packets in the network.See what could cause asymetric routing or
dropped packets in the network below.
Connections could also be pass-through if the peers are not compatible with each other. This may happen if you run the
non-compatible version between WAAS-Express and WAE. Check the compatibility table above for recommedned
software releases.
Solution #1: Check if the peer is incompatible using show waas statistics aoim
Solution #2: If you believe you have asymetrical routing scenario in your network, check the following.
What could cause asymetric routing or dropped packets in the network
Multiple WAN links in either the WAAS-Express router or the peer. Note that WAAS-Express it not supported on
active/active or active/standby routers because both traffic leaving and entering the WAN need to be on the same
WAAS-Express router. If there are multiple WAN links, make sure all the WAN links have config waas enable. Make
sure that all the WAN links and routers on the peer routers have config to redirect traffic to WAAS.
Symptom: Connections are getting pass-through
179

Control packets (SYN, SYN-ACK, ACK) are not tagged with WAAS option.This could happen if the traffic is not
redirected to WAAS on the peer side. Check your WCCP ACL.
Information to be provided to development team:
Network topology
IOS version
Configuration
Following debugs and show commands:
debug
debug
debug
debug
debug
waas
waas
waas
waas
waas
auto-discovery event
auto-discovery operation
infra error
infra event
show waas statistics auto-disc

show waas statistics pass
show waas statistics aoim
Note: Pass-through connections are not counted in the per-platform connection limit. WAAS-Express does not track
pass-through connections, hence there are no statistics related to pass-through flows. There, however, are counters that
indicate how many flows were put into pass-through and why.
Connections are not getting the desired optimization level

This is usually caused by misconfiguration. HTTP-Express Accelerator and CIFS-Express Accelerator are disabled by default
in WAAS-Express Version 2 image.Check that the Express Accelerator is enabled globally.
Symtom: Established connections do not get the desired or configured policy to use
CIFS, SSL, or HTTP-Express AO
Verify that CIFS, SSL, or HTTP-Express AO is enabled globally
router#show waas status
IOS Version: 15.2(4)M1
WAAS Enabled Interface Policy Map
FastEthernet8 waas_global
License Type: EvalRightToUse
Evaluation total period: 8 weeks 4 days
Evaluation period left: 7 weeks 4 days
DRE Status : Enabled
LZ Status : Enabled + Entropy
CIFS-Express AO Status : Disabled
SSL-Express AO Status : Enabled
HTTP-Express AO Status : Disabled <---- HTTP Express AO is disabled by default
Maximum Flows : 75
Total Active connections : 4
Total optimized connections : 4
What could cause asymetric routing or dropped packets in the network
180
Symtom: Expected connection optimization is THDL, but established connection has

TDL
This typically is caused by mis-configuration of the policy.
Note: HTTP-Express AO is not enabled by default.
Solution #1: Check if the core WAAS device is compatible. This check can be done using show waas statistics aoim
Solution #2: Check if HTTP-Express Accelerator is getting negotiated during auto-discovery using auto-discovery
debugs. This may be because the accelerator is disabled globally (note that HTTP accelerator is not enabled by default), or
HTTP class is missing ?accelerate http? in the action.
class HTTP
optimize tfo dre lz application Web accelerate http-express
Check Configured, Derived and Applied Accelerator fields under show waas connection detail
Router#show waas connection detail
...
Negotiated Policy:
Configured Accelerator:
Derived Accelerator:
Applied Accelerator:
Hist. Accelerator:
Bytes Read Orig:
...
TFO, LZ, DRE

HTTP-Express
HTTP-Express
HTTP-Express
None
174
Check handoff statistics/reason in show waas statistics accelerator http-express [https|debug]
Symtom: Expected connection optimization is TCDL, but established connection has

TDL
This may be because the accelerator is disabled, or CIFS/WAFS class is missing accelerate cifs in the action.
Note: CIFS-Express AO is disabled by default.
class CIFS
optimize tfo dre lz application CIFS accelerate cifs-express
Check handoff statistics/reason in show waas statistics accelerator cifs-express

Router#show waas statistics accelerator cifs-express
CIFS-Express AO Statistics
...
Unsupported dialects / CIFS version:
Currently active unsupported dialects / CIFS version:
Unsupported due to signing:
...
0
0
0
Symtom: Expected connection optimization is TSDL, but established connection has

TDL
In case of SSL-Express Accelerator, the core WAE SSL-AO may not be up and running. Check: Cisco Wide Area
Application Services SSL Application Optimizer Deployment Guide
The connection may also be getting pipe?ed. This can checked using show waas statistics accelerator ssl

Router#show waas statistics accelerator ssl
SSL-Express:
Global Statistics
...
Pipe through due to C2S cipher mismatch:
Pipe through due to C2S version mismatch:
Pipe through due to W2W cipher mismatch:
Pipe through due to W2W version mismatch:
Pipe through due to detection of non-SSL traffic:
Pipe through due to unknown reasons:
Total pipe through connections:
...
16:31:37 UTC Jul 26 2012

0
0
0
0
0
0
0
Expected connection optimization is TSHDL, but established connection has only

TSDL or THDL
SSL-Express Accelerator introduces HTTP-Express Accelerator in the path. Make sure both SSL-Express and HTTP-Express
Accelerator are enabled globally.
The connection got pipe-through?ed and shows up as TG. As shown above, check reason in show waas statistics
accelerator ssl
If the connection shows up as TSDL could be due to one of the following
HTTP-Express Accelerator is disabled.
HTTP-Express Accelerator is not compatible with the HTTP AO on core WAAS device.
At least 3 optimization features of HTTP-Express Accelerator are not enabled.
The first data packet does not contain HTTP content.
If the connection shows up as THDLcould be due to one of the following
SSL-Express Accelerator is not up and running on edge device.
SSL AO is not up and running on core device.
SSL-AO was not negotiated in AOIM.
For proxy, HTTP CONNECT request is to a port other than 443.
The 3-way DATA-INSPECT handshake where both edge and core devices notify each other regarding addition of
SSL-AO to the optimization for this connection fails.
Post DATA-INSPECT handshake, the 3-way TFO handshake where both edge and core devices agree to add
SSL-AO to the optimization for this connection fails.
Provide following show command outputs for debugging:
show
show
show
show
show
show
show
show
show
show
waas
waas
waas
waas
waas
waas
waas
waas
waas
waas
status
alarms
accelerator detail
accelerator http
accelerator smb
accelerator ssl
statistic global
statistic auto-discovery
statistic aoim
statistic pass-through
Symptom: Unexpected Connection Reset
Typically, there will also be error message which indicates the type of error along with the flow that is getting reset. For example,
Aug 18 03:02:52.861: %WAAS-3-WAAS_TFO_DEC_FRAME_FAILED: IOS-WAAS failed to decode TFO frame for connection 100.2.0
Symtom: Expected connection optimization is TSDL, but established connection hasTDL
182

Steps to troubleshoot
Turn on error debugs, depending on the module, debug waas <module_name> error.
Check End-Reason in show waas connection detail
Check show waas statistics error for possible reasons.
Is a core-dump generated on core WAE when connection resets are seen?
Malformed TCP headers sent by WAAS-Express resulted in core-dumps on WAE.
DDTSs capturing this issue: CSCto59459, CSCua61097. Search for these DDTS and check whether the issue
seen is similar to the one outlined by them.
If this is an SSL-Express Accelerator connection is the reset being caused by W2W handshake failure?
Information to be provided to the development team:
Debug logs Show command logs show-tech show-running config Network topology Client and server details, along with the
application (and version, e.g. IE6) being used for connection.
debug
debug
debug
debug
debug
debug
debug
debug
debug
waas
waas
waas
waas
waas
waas
waas
waas
waas
infra error
aoim error
tfo error
lz error
dre error
accelerator ssl error
accelerator http error
accelerator cifs error
Router crash/tracebacks
Router crashes and tracebacks may have been seen during testing. Search of previous cases and DDTSs for similar known issues.
In addition we also need to isolate what feature is resulting in the crash. If an IOS feature other than ios-waas or layer4-forwarding
is resulting in a crash/traceback, then that particular feature development team/ router TAC should be contacted accordingly.
Do a topic search at topic.cisco.com
Check previous customer cases for similar/known issues.
Information to be provided to the development team:
show tech or if not possibleshow running-config output
Exact IOS version.
Exact steps to reproduce the problem.
Decodes of traceback, or crashinfo in the case of crash.
Topology of the network
Any relevant information that will help with the reproduction of the problem internally.
Slow connection/degraded performance

Degraded performance may be caused by various reasons: the nature of the traffic, the load on the router, network topology or
packet drops in the network. For dealing with slow connections, we need to determine relative degradation with respect to
pass-through or non-optimized connections.

Step to troubleshoot
What is the optimization action for the connection?
Check Accel field in show waas connection. Is it TDL, THDL, TSDL, etc?
If a particular Accelerator is being used, does turning it off recover from the poor performance?
If there is upload traffic, try disable uplink DRE in the WAAS-Express parameter-map.
If the connection is put in TFO-only mode, is there a degradation seen with respect to pass-through mode?
What is the load on the router, check cpu utilization using: show proc cpu history
Check whether CPU throttling messages are seen in the log. When the CPU is too high, WAAS-Express slows
down the optimization in order to protect the CPU from being too overloaded
Check output of interface statistics to determine if there are packet drops.
Check if there are any ACLs that are dropping packets. A good debug to find which feature drops any packets is debug ip
cef drop.
Check if any device in the middle is dropping packets.
WAEs turn on ECN by default, and send packets with ECT bit set. Old devices may not like packets with ECT bit
set and hence can drop these packets leading to retransmissions and hence degraded performance. In a particular
customer case, a device (with an old IOS image) in the middle was dropping packets that had ECT bit set in the
TCP header.
ECN can be turned off on core WAE by using the following command in config mode: no tcp ecn enable
Does the setup have WAAS-Express enabled on multiple WAN links? If so, is the load-sharing being used a supported
option?
Per-packet load-sharing is not a supported option.
Per-destination load-sharing is a supported option. There should be no performance impact seen with this
load-sharing.
Asymmetric routing in the network, causing packet drops and retransmissions.
If the router does not see all packets of a particular flow, this may lead to slow/hung connections.
Slow connection with uplink-dre
Re-transmissions due to NACKs: Check show waas statistics dre. Check the R-tx .. fields
ACK-queue full: Check show waas statistics dre. Check the AckQ full and AckQ high fields
Connection slowed after enabling CIFS-Express/SSL-Express/HTTP-Express Accelerators.
Unsupported version/dialect.
Low compression ratio.
Check statistics under show waas connection detail, show waas statistic lz, show waas statistic dre
Check for connection handoff/pipe-through.
Note: Per-packet load-sharing is not a supported deployment. This is not a default load sharing mode.
Hung connections
There are no known issues with hung connections, please provide the following information to the development team to help RCA
the problem.
Step to troubleshoot and collect information
Search the flow in WAAS-Express connection table using show waas connection.
Router#show waas connection
ConnID
Source IP:Port
3336
192.168.22.99 :37797
Router#
Dest IP:Port
192.168.42.99
:80
PeerID
0016.9d39.20bd
Accel
THDL
Display the detail about the connection

Step to troubleshoot
184

Router#show waas connection client-port 37797 detail
connection ID:
Peer Id:
Connection Type:
Start Time:
Source IP Address:
Source Port Number:
Destination Port Number:
Application Name:
Classifier Name:
Peer Policy:
Configured Policy:
Negotiated Policy:
Configured Accelerator:
Derived Accelerator:
Applied Accelerator:
Hist. Accelerator:
Bytes Read Orig:
Bytes Written Orig:
Bytes Read Opt:
Bytes Written Opt:
Auto-discovery information:
---<snip>---
3336
0016.9d39.20bd
External
19:45:34 UTC Dec 21 2011
192.168.22.99
37797
<------ Unique port number required for next step
192.168.42.99
80
Web
HTTP
TFO, LZ, DRE
TFO, LZ, DRE
TFO, LZ, DRE
HTTP-Express
HTTP-Express
HTTP-Express
None
43056412
25
162
43359878
Find an equivalent flow in L4F table using show l4f flows.

Router#show l4f flows | include 37797
F4DF6EA0 Proxy TCP
192.168.22.99:37797
Router#
192.168.42.99:80
From the first column, collect the L4F flow id and use the information to get the detail L4F connection information.
Router#show l4f flow detail F4DF6EA0
Flow Address : F4DF6EA0
Index
: 11
Idle Time
: 0.004
Family
: IPv4
Protocol
: TCP
VRF ID
: 0
Address1
: 192.168.22.99:37797
Address2
: 192.168.42.99:80
State
: L4F_STATE_PROXYING
Flags
: 0x00012000
App Context
: 0x41D4728C
CEF pak
: 0x0
Endpoint1 FD 1073748479
State
: EP-ESTAB
Flags
: 0x00000001
Client
: L4F_FEATURE_WAAS
Association
: OUTPUT
CEF Fwd State : 0xC20D2C74
Proc Fwd State: 0xC1E36EA8
TCB Address
: 0xC01F0D9C
<------ Address required for next step
Endpoint2 FD 1073748480
State
: EP-ESTAB
Flags
: 0x00000001
Client
: L4F_FEATURE_WAAS
Association
: INPUT
CEF Fwd State : 0xC20D2248
Proc Fwd State: 0xC1E36F20
TCB Address
: 0x4002AB6C <------ Address required for next step
185

The output of show l4f flow detail <flow_id> show the two TCP TCBs. Use the TCB information in show tcp tdb
<tcb_info>
Router#show tcp tcb 0xC01F0D9C
Connection state is ESTAB, I/O status: 1, unread input bytes: 31504
Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 255
Local host: 192.168.42.99, Local port: 80
Foreign host: 192.168.22.99, Foreign port: 37797
Connection tableid (VRF): 0
Maximum output segment queue size: 50
Enqueued packets for retransmit: 0, input: 22
mis-ordered: 0 (0 bytes)
Event Timers (current time is 0x85115B0):

Timer
Starts
Wakeups
Next
Retrans
2
0
0x0
TimeWait
0
0
0x0
AckHold
10192
0
0x0
SendWnd
0
0
0x0
KeepAlive
20129
0
0x851FFF4
GiveUp
2
0
0x0
PmtuAger
0
0
0x0
DeadWait
0
0
0x0
Linger
0
0
0x0
ProcessQ
1
1
0x0
iss:
irs:
688070906
684581592
sndwnd:
rcvwnd:
6144
1263
snduna:
rcvnxt:
scale:
scale:
688070932
713368125
9
7
sndnxt:
maxrcvwnd:
delrcvwnd:
688070932
32767
0
SRTT: 6687 ms, RTTO: 59312 ms, RTV: 52625 ms, KRTT: 0 ms
minRTT: 0 ms, maxRTT: 2857348 ms, ACK hold: 200 ms
Status Flags: passive open, Timestamp echo present
Option Flags: keepalive running, SACK option permitted, non-blocking reads
non-blocking writes, win-scale, 0x200000, 0x1000000, 0x10000000
0x20000000
IP Precedence value : 0
Datagrams (max data segment is 1432 bytes):

Rcvd: 20129 (out of order: 0), with data: 20127, total data bytes: 28786532
Sent: 30017 (retransmit: 0, fastretransmit: 0, partialack: 0, Second Congestion: 0), with data: 1, total data byte
Packets received in fast path: 53559, fast processed: 2, slow path: 21294
fast lock acquisition failures: 7, slow path: 0
Router#
Router#show tcp tcb 0x4002AB6C
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 255
Local host: 192.168.22.99, Local port: 37797
Foreign host: 192.168.42.99, Foreign port: 80
Connection tableid (VRF): 0
Maximum output segment queue size: 50
Enqueued packets for retransmit: 50, input: 0
mis-ordered: 0 (0 bytes)
Event Timers (current time is 0x8519A48):

Timer
Starts
Wakeups
Next
Retrans
27124
0
0x8519D3B
TimeWait
0
0
0x0
AckHold
2
0
0x0
SendWnd
0
0
0x0
KeepAlive
28560
0
0x85284A4
GiveUp
27121
0
0x8545964
PmtuAger
0
0
0x0
186

DeadWait
Linger
ProcessQ
0
0
19975
iss: 2832065240
irs: 2835554554
0
0
19975
0x0
0x0
0x0
snduna: 2867154917
rcvnxt: 2835554717
sndwnd: 261120 scale:

7
rcvwnd: 65535 scale:
7
bic_last_max_cwnd:
8388480
sndnxt: 2867205953
maxrcvwnd:
delrcvwnd:
65535
0
SRTT: 1000 ms, RTTO: 1003 ms, RTV: 3 ms, KRTT: 0 ms

minRTT: 80 ms, maxRTT: 1000 ms, ACK hold: 200 ms
Status Flags: active open
Option Flags: keepalive running, SACK option permitted,
Timestamp option used, non-blocking reads, non-blocking writes
win-scale, 0x200000, 0x1000000, 0x10000000, 0x20000000
IP Precedence value : 0
Datagrams (max data segment is 1432 bytes):

Rcvd: 28560 (out of order: 0), with data: 2, total data bytes: 162
Sent: 28672 (retransmit: 0, fastretransmit: 28, partialack: 3, Second Congestion: 0), with data: 28671, total data
Packets received in fast path: 21244, fast processed: 21240, slow path: 29668
fast lock acquisition failures: 21374, slow path: 0
Router#
The following command output can be useful in debugging the WAAS-Express AO.
show
show
show
show
show
waas
waas
waas
waas
waas
statistics
statistics
statistics
statistics
statistics
errors
accelerator
accelerator
accelerator
accelerator
http-express
cifs-express
ssl-express
ssl-express debug
The following is a service-internal command (for debugging only)

show waas connection conn-id [id] debug
show waas statistics accelerator http-express debug
show waas statistics accelerator ssl-express debug
Hung connections can be cleared using the following command.

clear waas connection conn-id [id]
Router(config-if)#no waas enable forced
SSL-Express Accelerator issues:

Having issues with SSL-Express Accelerator enable or disable
Check if security license is enabled
Router#show waas status | include SSL-Express AO Status
SSL-Express AO Status
: Unavailable (security license not enabled)
Router#show license detail securityk9
Index: 1
Feature: securityk9
License Type: RightToUse
?
Version: 1.0
Check if you have an NPE image (this image does not support SSL-Express Accelerator)
SSL-Express Accelerator issues:
187

Router#show waas status | include SSL-Express AO Status
SSL-Express AO Status
: Unsupported
Router#show license detail securityk9
% Error: No license for securityk9 found - License feature not found
Enable ssl, aoim and infra debugs during enable/disable operation and provide debug logs.
Connection getting reset because of W2W handshake failure
Check SSL-Express Accelerator error statistics using show waas statistics errors | i SSL-Express
Check certificates:
Router#show running-config all | include waas-ssl-trustpoint

Router#show crypto pki trustpoints <trustpoint-name> status
WAAS#show crypto certificates

WAAS#show crypto certificate-detail WORD
Check alarms:
Router#show waas alarms
...
WAAS SSL-Express CA enrolled trustpoint deleted:
WAAS SSL-Express router certificate deleted:
...
off
off
Check configuration on edge and core devices. Check they are in-sync with respect to cipher-list, SSL version, and
certificate verification and revocation checks.
If self-signed certificates are being used, revocation-check and certificate verification should be disabled.
Turn on debug waas accelerator ssl error
Connection getting pipe-through?ed because of C2S Unsupported cipher
Check SSL-Express Accelerator error statistics using show waas statistics errors | i SSL-Express
Turn on debug waas accelerator ssl
Check cipher-list configured in the accelerated-svc on core WAAS device.
No SSL optimization (Pipe-through)
Check SSL-Express status on WAAS Express device: show waas accelerator ssl-express
Check SSL AO status on peer WAAS device: show accelerator ssl
Check SSL-Express statistics: show waas statistics accelerator ssl-express | i Pipe
Unable to access HTTPS page from internet
Since server is in internet, it?s private key and certificate can?t be installed on core WAAS device. Even after
accepting warning for certificate in the browser some objects on page may not show-up.
These objects may be served from CDN (content-delivery network). This issue is not unique to WAAS-Express.
That is, it should happen when connection is optimized between two WAAS devices as well.
Users will need to add exception to the browser to ignore certificate from CDN URL.
CDN URL can be found in page source.
Show commands used for further debugging and RCA:
show
show
show
show
waas
waas
waas
waas
statistics
statistics
statistics
statistics
accelerator
accelerator
accelerator
accelerator
ssl
ssl debug
ssl ciphers
ssl peering
Having issues with SSL-Express Accelerator enable or disable
188
Moving WAAS-Express device between Device-Groups on CM

If a WAAS-Express device is moved between device-groups on the WCM, it is sometimes seen that the policy definitions under
the new device-group do not take effect. When a device is unassigned from a device-group, it gets the policies from the backup
policy set of what the device last owned.
Use the following steps when moving the device between device-groups:
* Go to the Policy Definitions page of that device and select the new device-group and click on Submit.
OR
* Go to device-group-1 -> Assign Devices page and unassign the device from this DG.
* Go to device-group-2 -> Assign Devices page and assign the device to this DG.
* Go to device-group-2 -> Policy Definitions page and click on 'Force DG settings' button.
Other useful information

Statistics mismatch on WAAS-Express and WCM/WAE:
There are no known issues in this area. Please collect the logs using following procedure and provide them to the development
team.
*
*
*
*
*
*
Disable waas on Waas-Express device

Clear statistics on WAAS-Express and core WAE
Enable waas on Waas-Express device
Let traffic run, disable waas on Waas-Express device
Collect statistics
Present screen-shots and show command outputs.
Information in addition to debugs and show commands, that needs to be provided to the development team:
show
show
show
show
show
show
show
show
tech-support
ip interface
ip virtual-reassembly
ip route
ip cef detail
ip cef internal
ip cef switching statistics
process cpu history
Troubleshooting router crash

http://www.cisco.com/en/US/products/hw/iad/ps397/products_tech_note09186a00800b4447.shtml
Capturing packets on router

To debug connection problems, you may need to capture packets on the WAAS Express device.
For details on IOS packet capture, see the document: IP Traffic Export.
Example to configure packet capture:
ip traffic-export profile waas_wan mode capture bidirectional
interface Serial0/0/0
ip virtual-reassembly out

encapsulation frame-relay
ip traffic-export apply waas_wan size 20000000
frame-relay map ip 10.0.0.2 557 broadcast
no frame-relay inverse-arp
frame-relay local-dlci 557
Use following commands to start, stop, copy and clear the buffer:
traffic-export
traffic-export
traffic-export
traffic-export
int
int
int
int
s0/0/0
s0/0/0
s0/0/0
s0/0/0
start
stop
copy ftp://username:password@192.168.1.116//tftpboot/ngwo.pcap
clear
This article describes how to troubleshoot the integration of Network Analysis Module (NAM) into the WAAS Central Manager.
Guide Contents
Main Article
Contents
1
Connectivity
Issues
2 Rendering
Issues
3 Chart Data
Issues
The Cisco Network Analysis Module (NAM) is a standalone network analysis product that you can access through the WAAS
Central Manager if you have a NAM server installed in your network. This article describes how to troubleshoot the integration of
NAM into the WAAS Central Manager.
Capturing packets on router
190

NOTE: Support for NAM integration into the WAAS Central Manager was introduced in WAAS version 4.4.1. This section is
not applicable to earlier WAAS versions.
Connectivity Issues
If you are unable to connect to NAM from the WAAS Central Manager, do the following:
In the Central Manager GUI, choose Configure > Network Analysis Module (Beta) > Basics > Setup and ensure that
NAM server address and credentials are entered correctly, then click Test Connectivity/Credentials. All of the address,
user, and password fields should show a green check mark to indicate success. Any fields that show a red X are
configured incorrectly.
Ensure that the NAM HTTP or HTTPS server is enabled.
Ensure that the NAM server IP address and port are reachable both from the Central Manager and from the client
computer on which you are running the browser to access the Central Manager. You should be able to use the same IP
address and port to access the NAM server from both machines, regardless of whether NAT is involved.
Check your browser settings. In Internet Explorer, you may need to revert to the default settings in the Security and
Privacy tabs of Tools > Internet Options, then change the Privacy setting to Low.
If you are using the NAM HTTPS server, the NAM self-signed certificate may not be installed in the browser. Manually
add the NAM self-signed certificate into your browser. Alternatively, you can launch the NAM user interface in a separate
browser tab or window and accept and install the certificate, which allows you to view the integrated charts.
If you are unable to directly telnet into the NAM server, use the exsession on command on the NAM console
Rendering Issues
If the NAM page does not render properly or some action buttons on NAM do not work, it is likely due to an incompatible
browser version. The NAM server has the following browser requirements:
Internet Explorer 8 or later, or Firefox 3.6 or later
Java version 6 update 22 or later
Adobe Flash 10.0.45.2 or later (install the latest version from Adobe)
JavaScript must be enabled
If you get the message, "Navigation to the webpage was canceled" when you click on a NAM report, log out of the Central
Manager and log in again. When the browser asks if you want to view only the content delivered securely, click "No" to show all
content. The Central Manager data is based on HTTPS and NAM access uses HTTP by default, so the browser must be able to
show both secure (HTTPS) and unsecure (HTTP) content.
Chart Data Issues

Charts Show Unexpected Results
If the charts show unexpected results, this is likely caused by lack of clock synchronization. If the clock on the client machine
running the browser is not synchronized, you may see the following error on the page: Client or NAM time incorrect. Ensure that
the clocks on the client machine running the browser, the NAM server, and the WAAS Central Manager are synchronized. One
way to do this is by configuring an NTP server on all three machines.
Transaction Time Over Time Charts Show Zig-Zag Pattern
This chart pattern is common in POC scenarios where there is usually only one flow traversing the WAAS network. NAM's
default granularity for response time metrics is five minutes. If the flow duration is greater than five minutes and less than ten
minutes, and this is done continuously in a loop, it still results in a zig-zag pattern because the FA flows are reported by WAEs to

NAM only after they end.
Throughput > Network Charts Show Unusual Router Interface Names
The router interfaces may be displayed as if2, if15, and so on, instead of actual interface names such as GigaE 0/0, Tunnel0, and
so on. To see the correct router interface names, configure the netflow data source with SNMP credentials/community strings. In
the Central Manager GUI, choose Configure > Network Analysis Module (Beta) > Advanced > Data Sources. Select the
Netflow data source and edit it and provide SNMP credentials/community string.
Charts Show Higher Volume Than Expected
Charts may show higher than expected volume due to double counting. Filter to a specific data source to avoid double counting.
Data Sources Do Not Show on NAM
If a data source does not show on NAM, it could be due to network or firewall issues. Ensure Netflow is enabled on UDP port
3000 and flow-monitor on TCP port 7878, and they are not blocked by a firewall.
Flow Agent Connection Issues Persist
If Flow Agent connection issues persist, disable the flow monitor on the WAEs, restart NAM, and reenable the flow monitor on
the WAEs. You can control the flow monitor setting from the Central Manager GUI on the Configure > Monitoring > Flow
Monitor page.
Top Talkers Data Discrepancy
You may see a discrepancy in byte count between Top Talkers Detail and Top N Applications in Top Talkers Summary due to a
difference in the way these two reports count traffic:
Top Talkers Detail counts payload bytes only
Top Talkers Overview counts packet header and payload bytes
Higher Than Expected Throughput on Top Application or Application Throughput Charts
If netflow is enabled on both LAN and WAN interfaces on the same router, when a netflow data source is selected, the
Throughput > Top applications and Application charts show aggregated throughput, which is the sum of the LAN and WAN
throughput. To get accurate throughput data, either change the data source to WAAS flow agent, PA, or change the netflow export
to limit results to only the WAN interface.
No Passthrough Data Shown on the Performance Analysis > Application page
Make sure that the WAAS data sources are configured with the correct segments selected. The recommended configuration is to
have the branch WAE configured with Client and Pass-through segments, and the server side WAE configured with Server WAN
and Server segments.
Why Is There a Difference in Statistics When Different Data Sources Are Selected?
The netflow data source reflects data through the netflow export device (router/switch) and the FA data source reflects data
intercepted by WAAS.
Chart Data Issues
192

WAAS TShoot Guide

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

WAAS TShoot Guide

Caricato da

Copyright:

Formati disponibili

Cisco WAAS Troubleshooting Guide for Release 4.1.

Click here to return to the Cisco WAAS documentation on www.cisco.com.

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Creating a PDF of the WAAS Troubleshooting Wiki

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Understanding the WAAS Architecture

WoW and Virtual Blades

Configuration Management System

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

DRE with Scheduler

Interception & Flow Management

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

WAAS Traffic Flow

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

WAAS Traffic Flow

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Overview of the WAAS Troubleshooting Process

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Verifying the WAAS Image

Compiled 10:10:47 May 23 2009 by cnbuild

This command provides other useful information, for example:

You should see the message "No pending version".

Enabling WAAS Logging

Syslog facility is set to *

Verifying the WAAS Image

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Verifying the Physical Connectivity Between Peer WAAS Devices and

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

If necessary, enter a static route for the gateway.

Checking CPU Load

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Gathering WAAS Troubleshooting Information

Rebooting the WAAS Device

Using show Commands

Checking CPU Load

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Other useful show commands include the following:

Generating a System Report

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Capturing and Analyzing Packets

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

wae# tethereal -r test-netmon.cap -F libpcap -w test-libpcap.cap

wae# tethereal -R "tcp.flags.syn eq 1"

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Contacting Cisco Technical Support

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Contacting Cisco Technical Support

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Policy Engine Statistics

<-----Connection limit overload

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

D:DRE,L:LZ,T:TCP Optimization RR:Total Reduction Ratio

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

<-----Policy negotiated with peer

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

<-----In 4.3.3 and earlier only

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

Active peer details:

<-----Peer cache details in 4.3.3 and earlie

Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later

This article describes a general troubleshooting methodology for all AOs.

General Troubleshooting Methodology for AOs

The following general troubleshooting methodology applies to all AOs: