CIS 188 CCNP TSHOOT

(Troubleshooting)
Ch. 7 Troubleshooting Performance
Issues Part 1
Rick Graziani
Cabrillo College
graziani@cabrillo.edu
Fall 2012

Materials
Book:
Troubleshooting and Maintaining
Cisco IP Networks (TSHOOT)
Foundation Learning Guide:
Foundation learning for the CCNP
TSHOOT 642-832
By Amir Ranjbar
Book
ISBN-10: 1-58705-876-6
ISBN-13: 978-1-58705-876-9
eBook
ISBN-10: 1-58714-170-1
ISBN-13: 978-1-58714-170-6

Topics
Part 1: Troubleshooting Network Application Services
NetFlow accounting
IP SLAs (Service Level Agreements)
NBAR (Network-Based Application Recognition) packet
inspection
SLB (Server Load Balancing)
QoS (Quality of Service) and AutoQoS
Part 2: Routers and Switches
Performance Issues on Switches
Performance Issues on Routers

Troubleshooting Network Application

Services

Cisco Application Networking Services, or ANS:

Comprehensive portfolio of application networking solutions and
technologies that enable successful and secure delivery of applications
within data centers to local, remote, and branch-office users.

 Main categories of applications services

Our focus is on network infrastructure services aimed at optimizing
application traffic.

 4-step Application Optimization Cycle

Used to incrementally increases your understanding of network
applications

Step 1. Baseline application traffic:

Establish a reference from which service quality and application delivery
effectiveness can be measured.
You must understand:
basic traffic
application flows
network performance
Tools
NetFlow
NBAR
IP SLAs

Step 2. Optimize the network:

Apply optimization or control techniques to enhance application
performance.
Apply tools to reduce congestion and prioritize traffic:
QoS
AutoQoS
Server Load Balancing (SLB)
Wide Area Application Services (WAAS)
Application Control Engine (ACE),
Note: WAAS, and ACE are out of the scope for this book

Step 3. Measure, adjust, and verify:

Assess the effectiveness of the optimization tools such as a QoS reducing
configuration.
Includes:
Continuously monitoring and collecting information about the network
and application behavior
Comparing the behavior before and after the optimization and QoS
initiatives
Tools:
NetFlow
IP SLAs

Step 4. Deploy new applications:

Deploy new applications and updates to existing applications to meet
changing business needs.
New baselines need to be established.
And the application optimization cycle must start all over again.
Tools:
NBAR

10

 This section will have a brief discussion on the following Cisco IOS
application optimization tools:
NetFlow accounting
IP SLAs
NBAR packet inspection
Server load balancing
QoS
AutoQoS
NetFlow, IP SLAs, and NBAR packet inspection, provide a good starting
point for creating a network performance baseline.

11

Tools and Approach

Tools
NetFlow accounting
IP SLAs
NBAR packet inspection
Approach

Overview
Issues
Troubleshooting Examples

12

NetFlow

NetFlow efficiently provides a vital set of services for IP applications,

including:
Network traffic accounting
Usage-based network billing
Network planning
Security denial of service monitoring
Overall network monitoring
Designed by Cisco, NetFlow is now in its ninth version, which is now on the
IETF standards track to become an industry-wide standard.

13

 A flow is a unidirectional stream of packets, between a given source and a

destination, that have several components in common.
Source IP address
Destination IP address
Source Port
Destination Port
Protocol (Layer 3 or 4)
Input interface
Type of Service (ToS) Value (Differentiated Services Code Point, or DSCP)

14

 NetFlow is enabled per-interface basis using the command

Router(configif)#iproutecacheflow(physicalinterface)
Router(configif)#ipflowingress(subinterface)
If you want to export the NetFlow cache entries to an external collector, you must
define:
NetFlow version
Destination IP address and port number of the collector
Other commands are available to define:
Cache timeouts
Cache size
How you want to aggregate the data

15

 Display the NetFlow

statistics with the
show ip cache flow
See main flow
parameters, such as:
source IP address
destination IP
addresses
ports
Also see volume and
performance
information such as:
total number of
flows
flows per second
bytes per second
packets per
second
16

 Network Management: How to Configure NetFlow on Cisco

Routers
http://www.youtube.com/watch?v=KujLCfW2V8w
17

 Network Management: Harness the Power of NetFlow on Your

Network
http://www.youtube.com/watch?v=FgyKPgt2nBg&feature=related

18

 Network Management Software: Orion NetFlow Traffic Analyzer

Product Tour
http://www.youtube.com/watch?v=810DQbwNLjI&feature=relmfu
19

Common NetFLow
Issues

NetFlow is very efficient in exporting aggregated information to a NetFlow

Analyzer however
Might need to be tuned so that it does not create performance
degradation in the NetFlow-enabled device by consuming too much
memory or CPU cycles.
You may have to set limits:
The number of entries in the cache
Tune the aging timers of NetFlow (how long a flow remains in the cache
before being timed out and exported).
20

 Common NetFlow export culprits

A destination IP address has not been configured.
A source interface has not been configured.
Optional but the NetFlow Collector might be configured to expect
NetFlow information from a specific interface.
A source interface has been configured, but it is not in the up state.
The subnet of the destination is unreachable
21

Using Cisco IOS IP SLAs to Control Path Selection

Cisco IOS IP SLAs send simulated data across the network and measures
performance between multiple network locations or across multiple network
paths.
The information collected includes data about:
response time
one-way latency
jitter (interpacket delay variance)
packet loss
voice quality scoring
network resource availability
application performance
server response time

22

Cisco IP SLA

IP SLA, feature of Cisco IOS software allows you to configure a router to

send synthetic traffic to:
A host computer
Router that has been configured to respond (Responder)

23

 IP SLA is very useful for:

performance measurement
monitoring
network baselining.
You can tie the results of the IP SLA operations to other features of your
router and trigger action based on the results of the probe.

24

 To implement IP SLA network performance measurement, you need to

perform the following tasks:
Enable the IP SLA responder, if required.
Configure the required IP SLA operation type.
Configure any options available for the specified operation type.
Configure threshold conditions, if required.
Schedule the operation to run, and then let the operation run for a
period of time to gather statistics.
Display and interpret the results of the operation using the Cisco IOS
CLI or a network management system (NMS), with Simple Network
Management Protocol (SNMP).

25

 Depending on the type of probe you setup, you may or may not need to
configure an IP SLA Responder.
For example, if you are setting up a simple echo probe to a IP host, you do
not need a responder.
An IP SLA Responder allows for more detailed information to be retrieved.

26

 IP SLA Responder is a component embedded in the destination Cisco

routing device.
Allows the system to anticipate and respond to IP SLA request packets
Provides a large advantage with accurate measurements without the need
for dedicated probes and additional statistics not available from standard
ICMP-based measurements.
See information regarding IP SLAs with Responder Time Stamps
IP SLA Source (Cisco device) uses an IP SLA Control Protocol to
communicate with the IP SLA Responder.
Tells the responder which port it should listen to and respond.
Responder will enable the specified UDP or TCP port for a specific
duration.

27

Example: Network
Availability
Router(config)#iproute0.0.0.00.0.0.0fa0/0
Router(config)#iproute0.0.0.00.0.0.0fa0/15

fa0/0
fa0/1
172.16.1.1

Customer A is multihoming to two ISPs.

Customer A is not using BGP with the ISPs; but using static default routes.
Two default static routes with different administrative distances are
configured
Link to ISP-1 is the primary link
Link to ISP-2 is the backup link
The static default route with the lower administrative distance will be
preferred and injected into the routing table.
However, if there is a problem within the ISP-1 domain but its interface to
Customer A is still up, all traffic from Customer A will still go to that ISP
The traffic may then get lost within the ISP.

28

fa0/0
fa0/1
172.16.1.1

The solution to this issue is the Cisco IOS IP SLAs functionality

Configure the SLAs to:
Continuously check the reachability of a specific destination such as:
Provider edge [PE] router interface
ISP's DNS server
Any other specific destination: 10.1.1.1 and 172.16.1.1
Conditionally announce the default route only if the connectivity is
verified.
29

R1(config)#ipsla11
R1(configrtr)#typeechoprotocolipIcmpEcho10.1.1.1sourceinterfacefa0/0
R1(configrtr)#frequency10

Probe

R1(config)#ipslascheduleschedule11lifeforeverstarttimenow
R1(config)#track1rtr11reachability
R1(config)#iproute0.0.0.00.0.0.0fa0/02track1

Tracking
Object
Status of
Tracking Object

172.16.1.1
Defining the Probe
ip sla: defines probe 11
type echo: specifies that the ICMP echoes are sent:
To destination 10.1.1.1 to check connectivity
With the source interface of FastEthernet0/0
frequency 10: schedules the connectivity test to repeat every 10 seconds.
ip sla monitor schedule 11 life forever start-time now: defines the start
time of now and it will continue forever

30

R1(config)#ipsla11
R1(configrtr)#typeechoprotocolipIcmpEcho10.1.1.1sourceinterfacefa0/0
R1(configrtr)#frequency10

Probe

R1(config)#ipslascheduleschedule11lifeforeverstarttimenow
R1(config)#track1rtr11reachability
R1(config)#iproute0.0.0.00.0.0.0fa0/02track1

Tracking
Object
Status of
Tracking Object

172.16.1.1
Defining the Tracking Object
track 1 rtr 11 reachability: Specifies that:
Object 1 is tracked (next step)
Linked to probe 11 (defined in the first step) so that the reachability of
the 10.1.1.1 is tracked.

31

R1(config)#ipsla11
R1(configrtr)#typeechoprotocolipIcmpEcho10.1.1.1sourceinterfacefa0/0
R1(configrtr)#frequency10

Probe

R1(config)#ipslascheduleschedule11lifeforeverstarttimenow
Tracking
Object

R1(config)#track1ipsla11reachability
R1(config)#iproute0.0.0.00.0.0.0fa0/02track1

Status of
Tracking Object

AD=2

172.16.1.1

Defining an action based on the status of the tracking object

ip route 0.0.0.0 0.0.0.0 fa0/0 2 track 1: Conditionally announces the default
route, out fa0/0, with an administrative distance 2 if the result of tracking
object 1 is true if the probe is successful.
To summarize: If 10.1.1.1 is reachable, a static default route out Fa0/0 with an
administrative distance of 2, is installed in the routing table.
32

R1(config)#ipsla22
R1(configrtr)#typeechoprotocolipIcmpEcho172.16.1.1sourceinterfacefa0/1
R1(configrtr)#frequency10

Probe

R1(config)#ipslaschedule22lifeforeverstarttimenow
R1(config)#track2ipsla22reachability
R1(config)#iproute0.0.0.00.0.0.0fa0/13track2

Tracking
Object
Status of
Tracking Object

172.16.1.1
Defining the Probe
ip sla: defines probe 22
type echo: specifies that the ICMP echoes are sent:
To destination 172.16.1.1 to check connectivity,
With the source interface of FastEthernet0/1
frequency 10: schedules the connectivity test to repeat every 10 seconds.
ip sla monitor schedule 22 life forever start-time now: defines the start
time of now and it will continue forever

33

R1(config)#ipslamonitor22
R1(configrtr)#typeechoprotocolipIcmpEcho172.16.1.1sourceinterfacefa0/1
R1(configrtr)#frequency10

Probe

R1(config)#ipslamonitorschedule22lifeforeverstarttimenow
R1(config)#track2ipsla22reachability
R1(config)#iproute0.0.0.00.0.0.0fa0/13track2

Tracking
Object
Status of
Tracking Object

172.16.1.1
Defining the Tracking Object
track 1 rtr 22 reachability: Specifies that:
Object 2 is tracked (next step)
Linked to probe 22 (defined in the first step) so that the reachability of
the 172.16.1.1 is tracked.

34

R1(config)#ipsla22
R1(configrtr)#typeechoprotocolipIcmpEcho172.16.1.1sourceinterfacefa0/1
R1(configrtr)#frequency10

Probe

R1(config)#ipslaschedule22lifeforeverstarttimenow
Tracking
Object

R1(config)#track2ipsla22reachability
R1(config)#iproute0.0.0.00.0.0.0fa0/13track2

Status of
Tracking Object

AD=2
AD=3

172.16.1.1

Defining an action based on the status of the tracking object

ip route 0.0.0.0 0.0.0.0 fa 0/1 3 track 2: Conditionally announces the default
route, exit fa0/1, with an administrative distance 3 if the result of tracking
object 1 is true if the probe is successful.
To summarize: If 172.16.1.1 is reachable, a static default route exit fa0/1 with an
administrative distance of 3 is offered to the routing table.
Because this default route has a higher AD of 3, if the path via R2 is available,
this path will be the backup path.

35

R1(config)#ipsla11
R1(configrtr)#typeechoprotocolipIcmpEcho10.1.1.1sourceinterfacefa0/0
R1(configrtr)#frequency10

Probe

R1(config)#ipslaschedule11lifeforeverstarttimenow
Tracking
Object

R1(config)#track1ipsla11reachability
R1(config)#iproute0.0.0.00.0.0.0fa0/02track1

Status of
Tracking Object

R1(config)#ipsla22
R1(configrtr)#typeechoprotocolipIcmpEcho172.16.1.1sourceinterfacefa0/1
R1(configrtr)#frequency10

Probe

R1(config)#ipslaschedule22lifeforeverstarttimenow
Tracking
Object

R1(config)#track2ipsla22reachability
R1(config)#iproute0.0.0.00.0.0.0fa0/13track2

If 10.1.1.1 is reachable, a static default

route via R2 with an administrative
distance of 2, is installed in the routing
table
If 172.16.1.1 is reachable, a static default
route via R3 with an administrative
distance of 3 is available to the routing
table as a backup path.

Status of
Tracking Object

AD=2
AD=3

172.16.1.1

36

ip sla 1
type icmp-echo 10.1.100.1
ip sla schedule 1 life forever start-time now
!
ip sla 2
type icmp-echo 10.1.1.1
ip sla schedule 2 life forever start-time now
!
ip sla 3
type icmp-echo 10.1.100.253
ip sla schedule 3 life forever start-time now
!
track 1 ip sla 1 reachability
delay up 10
! Delays state change
!
track 2 ip sla 2 reachability
delay up 20
!
track 3 ip sla 3 reachability
delay up 30
!
track 10 list boolean or (At least one UP for
condition to be TRUE)
object 1
object 2
!
track 30 list boolean and (ALL UP for condition
to be TRUE)
object 1
object 2
object 3

event manager applet PERFECT-STORM

event track 10 state down
action 1.0 cli command "enable"
action 1.1 cli command "conf t"
action 1.2 cli command "interface g0/1.100"
action 1.3 cli command "shutdown"
action 1.4 syslog msg "SRV1 out through R3 WAN"
!
event manager applet PERFECT-STORM-OVER
event track 30 state up
action 1.0 cli command "enable"
action 1.1 cli command "conf t"
action 1.2 cli command "interface g0/1.100"
action 1.3 cli command "no shutdown"
action 1.4 syslog msg "SRV1 out through either
WAY"
!

37

Common IP SLA Issues

Sender

Sender

Receiver

Probes will cause a burden if overscheduled

If multiple senders overwhelm one receiver, or if the device is already a
bottleneck and its CPU utilization is high.
Senders generally suffer more from the over-scheduling and frequency of
probes.
Probe scheduling can be problematic if the clock on the device is out of
sync
Reason synchronizing through Network Time Protocol (NTP) is highly
recommended

39

 Cisco Internetwork Performance Monitor (IPM)

Several Cisco network management applications use IP SLAs One example
is the Cisco Internetwork Performance Monitor (IPM) in CiscoWorks2000
RWAN bundle.

40

 Solarwinds presents: What Is IP SLA?

http://www.youtube.com/watch?v=Wp8diS6bRtU&feature=relmfu

41

 Intro to Cisco IP SLA Operations - SolarWinds Video

http://www.youtube.com/watch?v=x-fQr24kFKg

42

 Network Performance Monitoring: Using IP SLA Monitor with

Orion NPM
http://www.youtube.com/watch?
v=YKXoexOVsaE&feature=related

43

NBAR

Network-Based Application Recognition (NBAR) is another

important tool for baselining and traffic classification purposes.
If an application is recognized and classified by NBAR, the network
can invoke services for that specific application.
Classifying packets
Then applying QoS to the classified traffic
Identify malicious or unwanted traffic
Block or filter it

44

 Leveraging Cisco Tools: Using NBAR to Manage QoS Policies

in Your Environment
http://www.youtube.com/watch?v=HA6YtxPgw1I

45

 There is a long list of applications identified by NBAR.

Traditionally routers do not recognize applications by simply inspecting the
layer 3 and layer 4 headers.
NBAR performs deep packet inspection up to the application layer for traffic
classification.
Configuring NBAR is beyond the scope of this class but for more info:
Configuring Network-Based Application Recognition
http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6558/ps6
616/prod_case_study09186a00800ad0ca.html
www.cisco.com/en/US/docs/ios/12_2/qos/configuration/guide/qcfnbar.pd
f

46

Router(config)#interfacefa6/0
Router(configif)#ipnbarprotocoldiscovery

Simplest use of NBAR is baselining through protocol discovery.

To gather information about the applications known to NBAR that are
transiting an interface use the interface configuration command ip nbar
protocol-discovery
NBAR can also be used to:
Plan your QoS deployment
Simply to understand the type of traffic traveling around your network
47

Router(config)#interfacefa6/0
Router(configif)#ipnbarprotocoldiscovery
Router#showipnbarprotocoldiscovery

Using NBAR protocol discovery is simple:

after your enable it on an interface
show ip nbar protocol-discovery command to look at application
statistics at any point during your analysis.

48

 NBAR depends on Cisco Express Forwarding (CEF) so it doesn't cause huge

performance degradation on the routers.
NBAR can only be used to classify packets of known applications
In order to be able to match more protocols and applications that the base
IOS NBAR feature supports, you need to upload Packet Description
Language Modules (PDLMs).
Currently available PDLMs support Citrix ICA (Independent Computing
Architecture) and peer-to-peer file-sharing applications such as KaZaa2,
Gnutella, BitTorrent, BitDonkey2000, and WinMX.

49

NetFlow and NBAR: Main Objectives and

Benefits
Main Objective

Main Benefit
NetFlow

Flow Characterization

Which users utilize the network

What types of traffic
When is the network utilized
Where does the traffic go

Network Usage

IP accounting and Billing Technology

Capacity Planning, Traffic Engineering,

Peering

Traffic & routing information analysis

Data Export

Persistent Network Usage Record

NBAR

Identify & classify traffic based on

payload attributes & protocol
characteristics

Optimize application performance via

QoS
Validation or reclassification of ToS
marking based on packet inspection

NetFlow and NBAR Differentiation

Link Layer
Header

Interface
TOS

IP
Header

TCP/UDP
Header

Data
Packet

NetFlow

Protocol
Source
IP Address
Destination
IP Address

NetFlow
Monitors data in Layers 2 thru 4
Determines applications by port
Utilizes a 7-tuple for flow

Source
Port
Destination
Port

Deep Packet
(Payload)
Inspection

NetFlow and NBAR both

leverage Layer 3 and 4 Header
Information

NBAR

NBAR

Examines data from Layers 3

through 7
Uses Layers 3 & 4 plus packet
inspection for classification
Stateful inspection of dynamic-port
traffic

52

NetFlow and NBAR useful for Security

Flow information is useful against attacks
NetFlow Mitigates Attacks
Identify the attack
Count the Flows
Inactive flows signal a worm
attack
Classify the attack
Small size flows to same
destination
What is being attacked and
origination of attack

NetFlow Security partners Arbor

Networks and Mazu, Adlex
Cisco IT prevented SQL slammer at
Cisco by watching flows
per port

Signature-based detection
Not historically a main focus
for NBAR
Real-time loadable PDLMs could
provide rapid-update mechanism
for new signatures
Not staffed to react against
malicious applications

NBAR can detect worms based on

payload signatures
Nimbda
Code Red
Slammer

Cisco PSIRT provided customers

with NBAR solution to combat
Code Red & Nimbda

Summary of Benefits
NetFlow

NBAR

Internet Access Monitoring

Protocol distribution
Where traffic is going/ coming

User Monitoring
Application Monitoring
Accounting and Billing
DDOS Monitoring
Peering Arrangements
Network Planning
Traffic Engineering

Deep & Stateful Packet

Inspection
Protocol & Application
Discovery
Standard protocols
Corporate applications
(Citrix, ...)
Undesired traffic
(peer-to-peer, worms, )

Real-time PDLM Signature

Update

SLB

Cisco IOS SLB feature is a Cisco IOS-based solution that provides server
load balancing.
Allows you to define a virtual server that represents a cluster of real servers,
known as a server farm.
SLB load-balances the connection to a chosen real server based on the
configured load-balance algorithm or predictor.
Clients initiate their connections to a virtual IP address
55

Router(config)#ipslbserverfarmSERVERS
Router(configslbsfarm)#real10.10.1.3
Router(configslbsfarmreal)#inservice
Router(configslbsfarm)#real10.10.1.5
Router(configslbsfarmreal)#inservice

Router(config)#ipslbvserverWEBSITE
Router(configslbvserver)#virtual172.17.63.215tcpwww
Router(configslbvserver)#serverfarmSERVERS
Router(configslbvserver)#inservice

If you need more/less capacity, you simply add/remove servers.

Clients will still point to the VIP, what happens inside the server farm is
transparent to them.

56

QoS and AutoQoS

QoS - each traffic class is treated differently by the network, with different
methods of:
congestion management
congestion avoidance
policing
other policies
Cisco AutoQoS is an automation tool for deploying QoS policies.
57

Router(configif)#autodiscoveryqos

In the first phase, information is gathered and traffic is baselined to define

traffic classes and volumes; this is called autodiscovery.
auto discovery qos
You must let discovery run for a period of time that is appropriate for your
baselining or monitoring needs:
Three days to two weeks is the usual range.
The router will collect information on traffic metrics, using NBAR to classify
and identify the traffic at the application layer.
58

Router(config)# interface type mod/num

Router(config-if)# auto qos

In the second phase you enter the interface auto qos command.
Cisco AutoQoS provides the user a simple, intelligent Command Line
Interface (CLI) for enabling campus LAN and WAN QoS on Cisco switches
and routers.
The network administrator does not need to possess extensive knowledge of
the underlying network technology (PPP, Frame Relay, ATM), required QoS
service policies, or link efficiency mechanisms needed to ensure voice quality
and reduce latency, jitter, and packet drops.
For Cisco AutoQoS to work certain requirements must be met:
Cisco Express Forwarding (CEF) must be enable on the interface
The interface must have an IP address configured
For serial interfaces configure the appropriate bandwidth
On point-to-point serial interfaces, both sides must be configured
AutoQoS

59

Common AutoQoS Issues

CEF disabled
Cisco Express Forwarding must be enabled.
Wrong bandwidth:
The bandwidth on a serial interface is not autosensed.
Cisco AutoQoS uses the configured interface bandwidth to enable or
disable certain QoS features such as compression and fragmentation.
If the configured bandwidths are different Cisco AutoQoS may enable
certain features on one side while disabling them on the other side of
the same link.
This can cause Layer 2 issues and bring the interface down.
60

 Modifying the Cisco AutoQoS configuration after the feature has been
enabled can be problematic.
If you disable AutoQoS on an interface using the no auto qos
command, all Cisco AutoQoS generated commands are removed.
Any commands that were modified are not removed.
If you don't remove those commands manually, they may cause errors
or performance problems.
AutoQoS does not work on an interface if it already has a policy applied to
it.
BEFORE you apply AutoQoS, check and make sure the interface has:
An IP address
Proper bandwidth configured
CEF enabled
No QoS policies already applied

61

Troubleshooting Examples

62

NetFlow Troubleshooting Example

NetFlow is used for traffic metering and baselining and a NetFlow Collector
server with the IP address 10.1.1.10 is used to collect and aggregate
NetFlow data.

63

Lo x.x.x.x
10.1.1.10
Port 9991

The reported problem is that the NetFlow Collector is not receiving data
from router R1, a NetFlow-enabled routers.
We start this troubleshooting exercise by:
Testing connectivity between R1 and the NetFlow Collector
Checking the NetFlow-specific configuration
Using ping, we confirm IP connectivity between R1 and NetFlow Collector,
Through some fact gathering we discover that the NetFlow Collector:
IP address is 10.1.1.10
NetFlow port number is 9991
Configured to receive NetFlow data from R1 on R1s loopback interface
as the source
64

10.1.1.10 Port 9991

Check if R1 is actually exporting NetFlow and if these are any flows to export
R1 is collecting plenty of data.

65

10.1.1.10 Port 9991

Next, we check if R1 is exporting the NetFlow data to the correct server

Problems:
The NetFlow Collectors address is 10.1.152.1 instead of 10.1.1.10
The source interface is FastEthernet0/0 instead of Loopback0

66

10.1.1.10 Port 9991

Make configuration corrections:

Change the NetFlow Collectors address from 10.1.152.1 to 10.1.1.10
Change the source interface from FastEthernet0/0 to Loopback0
67

10.1.1.10 Port 9991

Verify corrections.
NetFlow Collector is now receiving the NetFlow exports; the problem is
solved.

68

AutoQoS Troubleshooting Example

Service
Provider
s0/0/0

s0/0/0

The connection between routers R1 and R2 is down but the service provider
maintains that the backbone service is fully operational.

76

s0/0/0

s0/0/0

Service
Provider

Check the status of the ip interfaces on R1, using the show ip interfaces
brief command
Serial 0/0/0 is UP, but its line protocol is DOWN

77

s0/0/0

s0/0/0

R1#showinterfaceserial0/0/0
Serial0/0/0isup,lineprotocolisdown
Hardwareis
MTU1500bytes,BW115Kbit,DLY20000usec,rely255/255,
load1/255
EncapsulationHDLC,loopbacknotset
Fullduplexenabled
<outputomitted>

Service
Provider

Using the show interfaces command, you'll notice that serial 0/0/0 is
configured for HDLC (High-Level Data Link Control) encapsulation.
The documentation, however, indicates that this serial interface must be
configured for Point-to-Point Protocol (PPP) encapsulation.

78

s0/0/0

s0/0/0

Service
Provider

On R1, you fix the encapsulation problem by changing the encapsulation to

ppp.
Interface serial 0/0/0s line protocol status changes to UP, and a ping from
R1 to R2 (to ensure end-to-end connectivity) is successful.
It seems strange that the encapsulation on serial 0/0/0 was changed from
PPP to HDLC.

79

s0/0/0

s0/0/0

Service
Provider

R1(config)#interfaceserial0/0/0
R1(configif)#noautoqosvoip

Through some investigation and fact gathering we learn that the problem
was reported after someone tried to enable AutoQoS on this interface.
The tech removed AutoQoS, but the circuit remained down.
What the tech didn't know is that when AutoQoS was removed, the interface
encapsulation was changed back to serial interface default encapsulation,
which is HDLC.
All that was needed was to set the encapsulation to PPP, as we just did and
the interface came back up but...

80

s0/0/0

s0/0/0

Service
Provider

However, the issue that remains is that we need to make use of AutoQoS on
this interface.
AutoQoS (enterprise) has two phases.
First phase we need to enable "auto discovery" on the interface for a while
(minimum of 3 days is recommended).
Second phase, you actually enable AutoQoS, which will generate
templates and policies based on the discovery enabled in phase one.
Notice that the interface went down soon after you enter the auto qos
command on serial 0/0/0.
It seems that AutoQoS attempted to set the multilink feature on serial 0/0/0
and that failed.

81

s0/0/0

s0/0/0

Service
Provider

AutoQoS attempts the multilink feature for

fragmentation purposes, on low bandwidth
interfaces.
We need to find out what is the configured
bandwidth for the serial 0/0/0.
R1s running configuration reveals that the
bandwidth on serial 0/0/0 is set to 200
(kbps) rather than 2000.
This tells you why AutoQoS considered this
interface a slow bandwidth interface and
attempted the multilink feature on it.
PPP multilink allows multiple (usually
slower asynchronous) links to act as one
link.
PPP multilink must be configured on both
ends or the link will be down.

82

s0/0/0

s0/0/0

Service
Provider

The next logical step is to:

Remove AutoQoS first.
When you remove the AutoQoS feature, an error occurs and a message
indicates that all policies have not been removed.
Fix the bandwidth configuration on R1s serial 0/0/0 interface
Reapply the AutoQoS feature
When you retry the AutoQoS feature you receive yet another error message that indicates
because there is a policy on the interface, AutoQoS cannot be applied.

83

s0/0/0

s0/0/0

2000

Service
Provider

R1(config)#interfaceserial0/0/0
R1(configif)#noservicepolicyinputTEST
R1(configif)#noservicepolicyoutputTEST
R1(configif)#encapsulationppp
R1(configif)#autoqosvoip
R1#ping172.16.1.2
!!!!!

The running configuration shows a service-policy called TEST is applied to serial

0/0/0 interface for both inbound and outbound traffic.
You must:
remove those lines
reset encapsulation back to PPP
then re-apply AutoQoS
This time AutoQoS succeeds and the interface stays up.
Test the connection between R1 and R2 with a ping to ensure success

84

CIS 188 CCNP TSHOOT

(Troubleshooting)
Ch. 7 Troubleshooting Performance
Issues Part 1
Rick Graziani
Cabrillo College
graziani@cabrillo.edu