BRKRST 3067

Troubleshooting Catalyst 4K and 6K
BRKRST-3067
www.ciscolivevirtual.com
Agenda
Troubleshooting Catalyst 4500
Architecture
Interface Packet Drops
Catalyst 4500-E Series Switches
High CPU
IOS and IOS-XE Crashes
IOS-XE Licenses
Catalyst 4500 Series Switches
BRKRST-3067
2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Architecture Overview
Intelligent Supervisors
Shared Packet
Memory
Supervisor Engine 7-E, 6-E, 6L-E, V-10GE, V, IV,

II-Plus-10GE, II-Plus, II-Plus
CPU
TCAMs1
Switching ASICs
Packet
Processor
Packet Processor
NFL2
Forwarding
Engine
Forwarding Engine
Supervisor
Specialised Hardware
Line Card
TCAMs for ACLs, QoS, L3 forwarding
Stub ASICs
NetFlow (NFL) for statistics gathering
Transparent Line Cards

Wire-rate, oversubscribed, PoE
10/100, 10/100/1000, GE, 10GE
BRKRST-3067
Front Panel Ports
Supervisor and Blocking Line Card Block

Diagram
Cisco Public
Next Generation Supervisors

Supervisor 6L-E
Supervisor 6-E
Supervisor 7-E
BRKRST-3067
Cisco Public
Agenda
Architecture
High CPU
IOS-XE Licenses
BRKRST-3067
Cisco Public
Troubleshooting Interface Drops

Packet Processing
Engine
24Gbps
Packet drops in transmit direction
46xx
3G
E Series Linecard
TX queue is full
Stub ASICs
Oversubscription
Pause frames received
Packet drops in receive direction
Front Panel Ports
Minimal buffer on receiving queue
48Gbps
Packet Processing
Engine
47xx
6G
E Series Linecard
Stub ASICs
BRKRST-3067
Cisco Public
Front
Panel Ports
24Gbps

TX Queue is full
SUP6-E# show interfaces g2/47 counters detail | begin Drops
Port
Tx-Drops-Queue-1 Tx-Drops-Queue-2 Tx-Drops-Queue-3 Tx-Drops-Queue-4
Gi2/47
0
0
0
0
Port
Gi2/47
0
0
0
37748571
SUP6-E# show interfaces g2/47 counters detail | begin Drops
Port
Gi2/47
Port
Gi2/47
0
0
0
37874327
Default queue 8 when QoS is disabled
Next steps
Consider implementing a QoS policy to share traffic amongst queues

Increment default output queue-limit of 40 packets to allow for more
buffer space
BRKRST-3067
Cisco Public

Peer is sending Pause Frames
SUP6-E# show interfaces g2/47 counters detail | begin RxPause
Port
Rx-No-Pkt-Buff
RxPauseFrames
TxPauseFrames
Gi2/47
0
130
0
PauseFramesDrop
0
SUP6-E#show platform software interface g2/47 all | inc Busy

Switch Phyport Gi2/7 BusyStatus : ON, PauseStatus : OFF
BusyBitActivityCount
: 474
Ignore Subport Busy
: False
BusyBit is activated when received
pause frames exceed threshold.
%C4K_HWPORTMAN-4-BLOCKEDTXQUEUE: Blocked transmit queue HwTxQId7 on Switch

Phyport Gi2/47, count=116
Error log displayed when BusyBit is set
BRKRST-3067
Cisco Public

Peer is sending Pause Frames Continued
Next steps
Disable flow control receive on the local interface
Disable flow control send on the peer
SUP6-E# show interfaces g2/47 flowcontrol
Port
Send FlowControl Receive FlowControl
admin
oper
admin
oper
--------- -------- -------- -------- -------Gi2/47
on
off
off
off
BRKRST-3067
RxPause TxPause
------- ------130
0
Cisco Public

Minimal buffer on receiving stub ASIC (applicable to linecards with oversubscription ratio)
Sup6-E# show
0 input
Sup6-E# show
0 input
interface
errors, 0
interface
errors, 0
gi1/13
CRC, 0
gi1/13
CRC, 0
| include overrun
frame, 86432 overrun, 0 ignored
| include overrun
frame, 206658 overrun, 0 ignored
Sup6-E# show interface gi1/13 counter all | begin Rx-No

Port
Rx-No-Pkt-Buff
RxPauseFrames
TxPauseFrames
Gi1/13
206658
0
0
PauseFramesDrop
0
Sup6-E# show platform software interface g1/13 stub stat | in Overrun

OverrunPackets
: 206658
(look for Rx Stats)
Next steps
Enable flow control send on the local interface
Enable flow control receive on the peer

BRKRST-3067
Cisco Public
10
Agenda
Architecture
High CPU
IOS-XE Licenses
BRKRST-3067
Cisco Public
11
Troubleshooting High CPU

Functions of the 4500 CPU
Sends and receives control plane traffic
Shared Packet
Memory
STP, CDP, PAgP, VTP, DTP, routing

protocols
Supervisor 6-E
CPU
TCAMs
Program dynamic entries into hardware

ACLs, CEF entries
Packet
Processor
Manage access to the switch
Forwarding
Engine
Tx Queue
Memory
Telnet, SSH
Line Card
Stub ASICs
Manage system components

Fan tray, power supply, PoE
Front Panel Ports
BRKRST-3067
Cisco Public
12

Software-forwarded data traffic
Packets copied to CPU but originally switched in

hardware
Host mac address learning
Packets punted to CPU for processing

Routing updates, BPDUS, flood of traffic
Packets sent to the CPU for forwarding

AppleTalk, IPX
BRKRST-3067
Cisco Public
13

Usage on IOS threads
Dual core
SUP7-E# show processes cpu sorted detail

Core 0: CPU utilization for five seconds: 4%; one minute: 2%; five minutes: 2%
Process
Core 1: CPU utilization for five seconds: 6%; one minute: 3%; five minutes: 2%
breakdown
PID
T C TID
Runtime(ms) Invoked uSecs 5Sec
1Min
5Min
TTY
Process
(%)
(%)
(%)
9433
L
2946931
1131416 0
6.70605 A 4.39062 4.12207 0
iosd
9433
L 1 11383 984896
4669930 0
7.11
A 3.22
3.00
0
iosd
9433
L 1 9433
1961205
6644042 0
6.22
A 5.44
5.11
0
iosd
9433
L 1 11386 829
18630
0
0.00
A 0.00
0.00
0
iosd
71
I
55575
8787502 0
1.11
R 1.00
1.00
0
Cat4k Mgmt HiPri
52
I
4221576
2152734 0
0.33
R 0.33
0.33
0
IDB Work
72
I
1033445
1988579 0
0.33
R 0.33
0.33
0
Cat4k Mgmt LoPri
89
I
4
132
0
0.00
R 0.00
0.00
0
Exec
SUP6-E#sh proc cpu sorted

CPU utilization for five seconds: 6%/0%; one minute: 6%; five minutes: 6%
PID Runtime(ms)
Invoked
uSecs
5Sec
1Min
5Min TTY Process
51
12294972
8770348
1401 3.75% 3.78% 3.79%
0 Cat4k Mgmt LoPri
50
2556152 16464011
155 2.47% 2.49% 2.47%
0 Cat4k Mgmt HiPri
92
0
38032
0 0.07% 0.00% 0.00%
0 Ethchnl
111
20
142
140 0.07% 0.00% 0.00%
0 Exec
38
2044
380316
5 0.07% 0.06% 0.07%
0 IDB Work
99
44600
106342
419 0.07% 0.04% 0.05%
0 CDP Protocol
BRKRST-3067
Cisco Public
14

Usage on platform dependent threads
These processes use CPU under the Cat4K Mgmt HiPri and Cat4K Mgmt
LoPri
HiPri: within CPU target
LoPri: exceeds CPU target
SUP6-E#show platform health

%CPU
%CPU
RunTimeMax
Priority Average %CPU Total
Target Actual Target Actual
Fg
Bg 5Sec Min Hour CPU
RkiosObflMan
0.50
0.00
4
0 100 500
0
0
0 0:15
GalChassisVp-review
3.00
0.09
10
33 100 500
0
0
0 5:34
S2w-JobEventSchedule 10.00
0.00
10
0 100 500
0
0
0 0:00
Stub-JobEventSchedul 10.00
0.50
10
5 100 500
0
0
0 13:58
Lj-poll
1.00
0.01
2
0 100 500
0
0
0 1:18
StatValueMan Update
1.00
0.01
1
0 100 500
0
0
0 2:18
Pim-review
0.10
0.00
1
0 100 500
0
0
0 0:18
Ebm-host-review
1.00
0.00
8
0 100 500
0
0
0 0:05
Ebm-host-util-review
1.00
0.00
10
0 100 500
0
0
0 0:00
Ebm-port-review
0.10
0.00
1
0 100 500
0
0
0 0:01
Protocol-aging-revie
0.20
0.00
2
0 100 500
0
0
0 0:00
EbmHostRedundancyMan
2.00
0.00
20
0 100 500
0
0
0 0:00
Acl-Flattener
1.00
0.00
10
0 100 500
0
0
0 0:00
IrmFibThrottler Thro
2.00
0.00
7
0 100 500
0
0
0 0:26
BRKRST-3067
Cisco Public
15

Traffic in the CPU queue
Events that trigger packets to be sent to the CPU queue
SUP7-E# show platform cpu packet statistics
Packets Dropped In Processing Overall
Total
5 sec avg 1 min avg 5 min avg 1 hour avg
-------------------- --------- --------- --------- ---------36802848
0
0
0
0
Packets Dropped In Processing by CPU event
Event
Total
5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ---------Sa Miss
36778332
0
0
0
0
Input Acl Fwd
19
0
0
0
0
Mac flap?
Input ACl Copy
24497
0
0
0
0
Next steps
Identify the event that is dropping packets so that we know what type
of packet to look for in further CPU debugging
BRKRST-3067
Cisco Public
16
Troubleshooting High CPU - Tools

In-Built CPU sniffer
Packets that are punted to CPU are sent to a circular buffer

Buffer retains 1024 packets at a time and does not occupy CPU cycles
SUP6-E# debug platform packet all buffer
platform packet debugging is on
SUP6-E# show platform cpu packet buffered
Total Received Packets Buffered: 1024
Ingress interface of packet
------------------------------------Index 0:
3 days 23:23:18:54927 - RxVlan: 1006, RxPort: Gi1/1
Priority: Normal, Tag: No Tag, Event: 11, Flags: 0x40, Size: 64
Eth: Src 00:00:0B:00:00:00 Dst 00:22:90:E0:D6:FF Type/Len 0x0800
Ip: ver:IpVersion4 len:24 tos:0 totLen:46 id:0 fragOffset:0 ttl:64 proto:tcp
src: 10.10.10.100 dst: 172.16.100.100 hasIpOptions firstFragment lastFragment
Remaining data:
0: 0x0 0x64 0x0 0x64 0x0 0x0 0x0 0x0 0x0 0x0
Source/destination address
10: 0x0 0x0 0x50 0x0 0x0 0x0 0x8A 0x37 0x0 0x0
20: 0x0 0x1 0xB5 0x77 0x6A 0x7E
BRKRST-3067
Cisco Public
17
Troubleshooting High CPU - Tools

SPAN Capture for CPU
Monitor CPU bound traffic both in send and receive direction
SUP6-E# show running-config | include monitor
monitor session 1 source cpu rx
monitor session 1 destination interface Gi1/48
SUP6-E# show monitor session 1
Session 1
--------Type
: Local Session
Source Ports
:
RX Only
: CPU
Destination Ports
: Gi1/48
Encapsulation
: Native
Ingress
: Disabled
Learning
: Disabled
Connect laptop with Wireshark
Next steps
Is there a pattern
and is this legitimate traffic?
Cisco Public
BRKRST-3067
18
Agenda
Architecture
High CPU
IOS-XE Licenses
BRKRST-3067
Cisco Public
19
IOS Process Crash
Reasons for a crash
Bus error, forced by software, address errors, watchdog timeouts

All crashes generate a crashinfo
Present in crashinfo or platform crashdump
------------------ show platform crashdump -----------------Last powerfail: 03/09/2006 02:10:24
Verify time of outage
Current time: 02/19/2007 16:09:59
Last reload status: 00008800 038D0000
Last crash: 02/19/2007 15:44:40
Build: 12.2(31)SG ENTSERVICES
Tracebacks that need to be decoded
buildversion addr: 12288034
pc=112088A4 lr=11208824 msr=20029030 vector=00000300
cr=20000042 ctr=11207418 xer=00000000
r0=8000FBCE r1=13FFE680 r2=0000C000 r3=13FFE6C8 r4=13FFE620 r5=00000002
r6=00000000 r7=000000FF r8=FFFFFFFF r9=00000000 r10=00000002 r11=00000008
BRKRST-3067
Cisco Public
20
IOS-XE Process Crash

IOS-XE vs. IOS
IOS-XE kernel is Linux
Runs several different processes
IOSd, FFM, HA, licensing
IOS runs as one process in a single unprotected memory space
IOS-XE crash infrastructure collects crashinfo files for any failing process
BRKRST-3067
Cisco Public
21
IOS-XE Process Crash

Files generated upon a crash: system kernel info, crashinfo file and core file
By default, crashinfo files are stored in the crashinfo directory(135MB parition of
bootflash)
Generation of core files requires configuration
Enables generation of
process core dump file
Sup7-E(config)# exception coredump

SUP7-E#show exception information
Exception configuration information
Coredump file - disabled,compressed
Maximum number of files
Core - 10 file(s)
Process crashinfo - 10 file(s)
Configured storage devices
1 - crashinfo:
2 - not assigned
3 - not assigned
Dump protocol - not configured
BRKRST-3067
Default 10 crashinfo files

and 10 core files saved
Ehnaced crash dump is

supported on Sup6-E from
15.0(2)SG, core dump from
15.1(1)SG
(not supported)
Cisco Public
22
Agenda
Architecture
High CPU
IOS-XE Licenses
BRKRST-3067
Cisco Public
23
IOS-XE Licensing (CSL)

Feature Set License
Feature Sets
Entservices
Entservices
Ipbase
Lanbase
Ipbase
Lanbase
IPbase
Lanbase
BRKRST-3067
Entservices
Ipbase
Lanbase
Lanbase
Cisco Public
24
IOS-XE Licensing (CSL)

Permanent license is node-locked
License is for a chassis UDI (Universal Device Identifier), but stored on
Supervisor bootflash
License synced to hot standby supervisor
No Product Activation Key (PAK) is generated for customer
BRKRST-3067
Cisco Public
25
License Show Commands

Sup7e# show license all
License Store: Primary License Storage1
StoreIndex: 0 Feature: internal_service Version: 1.0 License Type: Evaluation
Evaluation period left: 23 hours 59 minutes
License State: Active, Not in
2
Use, EULA accepted
StoreIndex: 2 Feature: entservices Version: 1.0

License Type: Permanent
Permanent
License State: Active, In Use
node-locked
license
License Store: Dynamic License Storage
License Type: Evaluation
Evaluation total period: 8 weeks 4 days
Evaluation period left: 4 weeks
3 days
License State: Inactive
Temporary License
BRKRST-3067
Cisco Public
26
Troubleshooting License Installation

License must be installed and operational
The system will boot up with default Lanbase license level if no licenses
are installed
Sup7e# dir bootflash:
44268 Jan 4 2011 21:46:41 7slot_ent_FOX1418GEW0_20110103155106655.lic
Sup7e#license install bootflash:7slot_ent_FOX1418GEW0_20110103155106655.lic

Installing licenses from "bootflash:7slot_ent_FOX1418GEW0_20110103155106655.lic
Installing...Feature:entservices...Successful:Supported
1/1 licenses were successfully installed
0/1 licenses were existing licenses
0/1 licenses were failed to install
BRKRST-3067
Cisco Public
27

Is the license operational?
Sup7e#show license all
License Store: Primary License Storage
License Type: Permanent
License State: Active, Not in Use
License Count: Non-Counted
License Priority: Medium
Reboot required to make the license operational

Sup7e# show version
Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500eUNIVERSAL-M), Version 03.01.00.SG RELEASE SOFTWARE (fc4)
License Information for 'WS-X45-SUP7-E'
License Level: lanbase
Type: Default. No valid license found
Next reboot license Level: entservices
BRKRST-3067
Cisco Public
28

License Bootlevel: HA Upgrade
Standby supervisor will always boot to the same license level as that
of Active supervisor
If running SSO and Active and Standby are running different license
levels, need to manually set level.
Sup7e(config)#license boot level entservices
BRKRST-3067
Cisco Public
29

Series Switches
Agenda
Sup720 Architecture
Sup720 Layer 2 and Layer 3 Unicast Troubleshooting
High CPU utilisation

Virtual Switch System (VSS) Troubleshooting
Sup2T Architecture
Sup2T Layer 3 Unicast Troubleshooting
BRKRST-3067
Cisco Public
31
Supervisor 720/PFC3 Architecture

Layer2 Control-plane
E.g., LACP, BPDU and
hardware programming
Layer3 Control-plane
E.g., OSPF, BGP, SNMP
Replication engine
E.g., Multicast, SPAN
L3/L4 forwarding
SFP /
SFP GETX
MSFC 3
Flash
DRAM
Flash
DRAM
ACE
Counter
RP
CPU
1 Gbps
SP
CPU
1 Gbps
Port ASIC
L3/4 Engine
MET
Switch Fabric
20 Gbps
16 x 20G Traces
Traces # 1 to 16
Integrated Switch
Fabric
BRKRST-3067
QoS
Adj
FIB ACL
NetFlow TCAM TCAM TCAM
TCAM
EOBC
Fabric Interface
and
Replication Engine
L2 forwarding
L2 Engine
PFC3
DBUS
RBUS
L2 CAM (64K)
16 Gbps Bus
Cisco Public
32
Agenda
Sup720 Architecture

Sup2T Architecture
BRKRST-3067
Cisco Public
33
L2 Packet Flow Troubleshooting

Check the L2 forwarding engine counters
Verify the channels used in the flow
Port
Port
ASIC
ASIC
Fabric
Interface &
MET
Replication
Engine
Port
ASIC
Layer 2 Layer 2
Engine
Engine
L3/4
DFC3 Engine
Port
ASIC
Fabric
Interface &
Replication
Engine
4 x 1x10GE port ASIC
WS-X6704
Module 8
MET
Switch Fabric
MET
Fabric
Interface &
Replication
Engine
Port
ASIC
Port
ASIC
Layer 2 Layer 2
Engine
Engine
L3/4
DFC3 Engine
Fabric
Interface &
Replication
Engine
Port
Port
ASIC ASIC
WS-X6748
Module 7
MET
4 x 12xGE port ASIC

BRKRST-3067
Cisco Public
34
Layer 2 Learning and Forwarding

Layer 2 forwarding is based on {VLAN, MAC} pairs
Entries are stored based on result of hash done on MAC and VLAN
MAC learning is done per PFC or DFC
Each PFC/DFC maintains separate L2 CAM table
PFC and DFCs age entries independently
Refreshing of entries based on seeing traffic from specific host
New learns on one forwarding engine communicated to other
engines via MAC-Sync process
MAC table size:
64K entries on PFC3A / 3B / 3BXL and DFC3A / 3B / 3BXL
96K entries on PFC3C / 3CXL and DFC3C / 3CXL
BRKRST-3067
Cisco Public
35
L2 Unicast Traffic Network Configuration
BRKRST-3067
ARPA
Cisco Public
Te8/3
0011.bced.e400
Te8/3
Type
Po1
Cat6K
Vlan700
Host1
7.0.1.1
Gig7/4
Hardware Addr
Vlan700
Te8/1
ARPA
Gig8/1
000b.fca2.fe0a
Gig7/3
Age (min)
Type
Gig8/2
Cat6K#show ip arp 7.0.1.2

Protocol Address
Interface
Internet 7.0.1.2
Hardware Addr
Gig7/5
Age (min)
Gig8/3
Cat6K#show ip arp 7.0.1.1

Protocol Address
Interface
Internet 7.0.1.1
Host2
7.0.1.2
Te8/1
Verify that ARP entry is present for both hosts
R2
Po2
Po1
R1
36
L2 Unicast Traffic
Te8/3
Te8/3
Host2
7.0.1.2
Te8/1
Verify the interface that both mac-addresses are learnt on
Te8/1
R2
Cat6K#show mac-address-table address 000b.fca2.fe0a vlan 700

Legend: * - primary entry; age - seconds since last seen
vlan
mac address
type
learn
age
ports
------+----------------+--------+-----+----------+-------------------
dynamic
Yes
170
Po2
Cat6K#show mac-address-table address 0011.bced.e400 vlan 700

vlan
mac address
type
learn
age
BRKRST-3067
dynamic
Yes
170
Po2
ports
------+----------------+--------+-----+----------+------------------Module 8[FE 1]:

* 700 0011.bced.e400
dynamic Yes
170
Po1
Module 8[FE 2]:
* 700 0011.bced.e400
Gig7/4
Module 7[FE 2]:

* 700 000b.fca2.fe0a
Cat6K
Two Forwarding Engines
Gig8/1
Po2
Gig7/3
50
Gig8/2
Yes
Gig7/5
dynamic
Gig8/3
Module 7[FE 1]:

* 700 000b.fca2.fe0a
Po1
Po1
Cisco Public
Host1
7.0.1.1
Po1
R1
37
L2 Unicast Traffic
BRKRST-3067
Cisco Public
Te8/1
Te8/3
Te8/3
Cat6K#show mac-address-table address 000b.fca2.fe0a vlan 700 all

vlan
mac address
type
learn
age
ports
------+----------------+--------+-----+----------+------------------Module 1:
700 000b.fca2.fe0a
dynamic Yes
170
Po2
Active Supervisor:
700 000b.fca2.fe0a
dynamic Yes
170
Po2
Standby Supervisor:
700 000b.fca2.fe0a
dynamic Yes
170
Po2
Module 7[FE 1]:
* 700 000b.fca2.fe0a
dynamic Yes
50
Po2
Module 7[FE 2]:
* 700 000b.fca2.fe0a
dynamic Yes
170
Po2
Module 8[FE 1]:
700 000b.fca2.fe0a
dynamic Yes
170
Po2
Module 8[FE 2]:
700 000b.fca2.fe0a
dynamic Yes
170
Po2
Po1
Host1
7.0.1.1
Gig7/3
Gig7/4
Gig8/2
Gig8/1
Cat6K
Gig7/5
Primary
Entry
Host2
7.0.1.2
Gig8/3
Verify that there is an entry for all fowarding engines (FE)
Te8/1
R2
Po2
Po1
R1
38
Te8/3
Host2
7.0.1.2
Te8/3
EtherChannel Load-Balancing Configuration:
R2
Te8/1
R1#show etherchannel load-balance module 1
Check load balancing configuration.

Use ingress Module number in
command in case per-module loadbalancing is configured (SXH images
and later)
Te8/1
L2 Unicast Traffic
dst-ip
mpls label-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Destination MAC address
Mode is dst-ip. Only use dest IP as

argument. Prior to 12.2(33)SXH, use test
etherchannel load-balance (same
arguments) on the SP
IPv4: Destination IP address
IPv6: Destination IP address
Cat6K
BRKRST-3067
Gig7/3
Gig7/4
Gig8/2
Gig8/1
Would select Gi8/1 of Po1
Gig7/5
Computed RBH: 0x1
Gig8/3
MPLS: Label or IP
R1#show etherchannel load-balance hash-result interface po1 ip 7.0.1.2
Po1
Po2
Link selected is Gi8/1 in Po1 of R1 for traffic to

7.0.1.2 leaving R1
Cisco Public
Host1
7.0.1.1
Po1
R1
39

Check L3/L4 tables
Confirm HW and SW forwarding tables are synchronised
Port
Port
ASIC
ASIC
Fabric
Interface &
MET
Replication
Engine
Layer 2 Layer 2
Engine
Engine
L3/4
DFC3 Engine
Port
Port
ASIC
ASIC
Fabric
Interface &
Replication
Engine
4 x 1x10GE port ASIC
WS-X6704
Module 8
MET
Switch Fabric
MET
Fabric
Interface &
Replication
Engine
Port
ASIC
BRKRST-3067
Port
ASIC
Layer 2 Layer 2
Engine
Engine
L3/4
DFC3 Engine
Fabric
Interface &
Replication
Engine
Port
ASIC
Port
ASIC
Cisco Public
WS-X6748
Module 7
MET
4 x 12xGE port ASIC

40
FIB/Adjacency Tables
L3 FIB Table Programming Flow
show ip route (RIB)
Verify Layer 3
IOS Routing Table (RP)
Verify Layer 2 rewrite
show ip arp
IOS ARP Cache Table (RP)
show ip cef
remote
command
module <mod>
show ip cef
IOS FIB Table (RP)
IOS Adjacency Table (RP)
IOS FIB Table (SP/DFC)
IOS Adjacency Table (SP/DFC)
MLS FIB Table (SP/DFC)
MLS Adjacency Table (SP/DFC)
show mls cef lookup <ip

address> <mod>
BRKRST-3067
show mls cef adjacency entry

Cisco Public
show ip cef
adjacency
remote
command
module <mod>
show adjacency
detail
41
L3 Unicast Traffic
Cat6K
Cisco Public
Gig7/3
Gig7/4
Gig8/1
Next hop used for HW based CEF

(HW forwarding path). Note: 0 is
used for both src and dest L4 port
numbers as test flow was ICMP echo
Gig8/2
Next hop used for SW based

CEF (SW forwarding data path)
R1#show mls cef exact-route 8.0.1.1 0 9.0.1.2 0

Interface: Vl705, Next Hop: 7.5.1.2, Vlan: 705, Destination Mac:
0050.f0f8.7400
BRKRST-3067
Te8/3
Equal Cost Routes to

the destination prefix
R1#show ip cef exact-route 8.0.1.1 9.0.1.2

8.0.1.1
-> 9.0.1.2
: Vlan701 (next hop 7.1.1.2)
Check which link between

R1 and Cat6K is chosen.
Te8/3
SW
Gig7/5
Vlan702
Vlan705
Vlan704
Vlan703
Vlan701
Te8/1
3328, type internal
Te8/1
Host2
9.0.1.2
Gig8/3
R1#show ip route 9.0.1.0 | include via

Known via "eigrp 700", distance 90, metric
Redistributing via eigrp 700
* 7.2.1.2, from 7.2.1.2, 00:21:58 ago, via
7.5.1.2, from 7.5.1.2, 00:21:58 ago, via
7.4.1.2, from 7.4.1.2, 00:21:58 ago, via
7.3.1.2, from 7.3.1.2, 00:21:58 ago, via
7.1.1.2, from 7.1.1.2, 00:21:58 ago, via
R2
Po2
HW
Host1
8.0.1.1
Po1
R1
42
L3 Unicast Traffic
Te8/3
Te8/3
Cat6K#show ip route 9.0.1.0 | i via

Known via "eigrp 700", distance 90, metric 3072, type internal
* 7.7.1.2, from 7.7.1.2, 00:07:33 ago, via TenGigabitEthernet8/3
7.6.1.2, from 7.6.1.2, 00:07:33 ago, via TenGigabitEthernet8/1
Te8/1
Host2
9.0.1.2
Te8/1
R2
Cat6K
BRKRST-3067
Cisco Public
Host1
8.0.1.1
Gig7/5
Gig7/3
Gig7/4
Gig8/2
Gig8/1
L3 Interface map internally to

a 1-port VLAN
Gig8/3
Cat6K#show vlan internal usage

[snip]
1090 TenGigabitEthernet8/3
1091 TenGigabitEthernet8/1
[snip]
R1
43
L3 Unicast Traffic
SW
Host2
9.0.1.2
8.0.1.1 -> 9.0.1.2 => IP adj out of TenGigabitEthernet8/1, addr 7.6.1.2

Cat6K#show
Te8/3
R2
Te8/3
Use show ip cef exact-route command to

find which of the two possible next hops is
nexthop 7.6.1.2 TenGigabitEthernet8/1
being used for SW switched flows.
Arguments are source and destination
IPs
Cat6K#show ip cef exact-route 8.0.1.1 9.0.1.2
Te8/1
9.0.0.0/8
Te8/1
Cat6K#show ip cef 9.0.1.2
ip cef adjacency TenGigabitEthernet 8/1 7.6.1.2
7.6.1.2/32
Cat6K
Show all prefixes using that Adjacency.
attached to TenGigabitEthernet8/1
9.0.0.0/8
Cat6K#show
Te8/1
, 000f.f8e4.d000 (Hash: 007F)
Te8/3
, 000f.f8e4.d000 (Hash: 7F80)
Use mls cef exact-route on

ingress module to display next
hop for HW switched flows.
mls cef exact-route 8.0.1.1 0 9.0.1.2 0 module 7
Gig7/4
108749 9.0.0.0/8
Adjacency
Gig8/1
Prefix
Gig7/3
Index
HW
Gig8/2
Codes: decap - Decapsulation, + - Push Label
Use mls cef lookup on ingress module to

display next hops for HW switched flows
Gig7/5
Cat6K#show mls cef lookup 9.0.1.2 mod 7
Gig8/3
Interface: Te8/3, Next Hop: 7.7.1.2, Vlan: 1090, Destination Mac: 000f.f8e4.d000
BRKRST-3067
Cisco Public
Host1
8.0.1.1
R1
44
L3 Unicast Traffic
Cat6K#show adjacency ten 8/3 7.7.1.2 detail
Protocol Interface
Address
IP
7.7.1.2(17)
TenGigabitEthernet8/3
Aggregate HW adjacency
statistics (SW collects it from all
DFC/PFCs for all prefixes linked
to this adjacency)
2001 packets, 228114 bytes

epoch 0
Rewrite information
(Dmac|Smac|0800): verify it is
conform with next hop rewrite info
sourced in sev-epoch 774
Encap length 14
000FF8E4D0000050F0F874000800
ARP
To get HW adjacency statistic

for this prefix on this module
Cat6K#show mls cef lookup 9.0.1.2 detail mod 7

Codes: M - mask entry, V - value entry, A - adjacency index, P - priority bit
D - full don't switch, m - load balancing modnumber, B - BGP Bucket sel

V0 - Vlan 0,C0 - don't comp bit 0,V1 - Vlan 1,C1 - don't comp bit 1
RVTEN - RPF Vlan table enable, RVTSEL - RPF Vlan table select
Format: IPV4_DA - (8 | xtag vpn pi cr recirc tos prefix)
Format: IPV4_SA - (9 | xtag vpn pi cr recirc prefix)
M(108749 ): E | 1 FFF
0 0 0 0
255.0.0.0
V(108749 ): 8 | 1 0
0 0 0 0
9.0.0.0
BRKRST-3067
Start adjacency pointer is 294933, 14 + 1 =

15 adjacencies linked to the prefix
(A:294933 ,P:1,D:0,m:14,B:0)
Cisco Public
45
L3 Unicast Traffic
Cat6K#show mls cef adjacency entry 294933 to 294947 mod 7
Index: 294933
smac: 0050.f0f8.7400, dmac: 000f.f8e4.d000

mtu: 9234, vlan: 1091, dindex: 0x0, l3rw_vld: 1
packets: 0, bytes: 0
Index: 294947
smac: 0050.f0f8.7400, dmac: 000f.f8e4.d000
15 HW adjacencies linked to this

prefix: which one is really used ?
mtu: 9234, vlan: 1090, dindex: 0x0, l3rw_vld: 1

Cat6K#show mls cef adjacency entry 294933 to 294947 mod 7 | i packets
294933
294934
294935
294936
294937
294938
294939
Based on the packet counts, we see that the 4th

adjacency (entry 294936) is being used.
Verify that the rewrite information is correct for

the adjacency.
Cat6K#show mls cef adjacency entry 294936 detail mod 7

Index: 294936
smac: 0050.f0f8.7400, dmac: 000f.f8e4.d000
Note counter is cleared here; counter is

cleared when adjacency is read.
BRKRST-3067
Cisco Public
46
Agenda
Sup720 Architecture

Sup2T Architecture
BRKRST-3067
Cisco Public
47

Components Involved
RP: show ibc
RP: show process cpu
Uplink ports
MSFC 3
Flash
RP: show ip traffic

RP: show interfaces
DRAM
RP
CPU
Flash
SP
CPU
DRAM
1 Gbps
Inband
C
Port ASIC
C
1 Gbps
Inband
Sup720
SP: show ibc
SP: show process cpu
BRKRST-3067
Cisco Public
48
High CPU Utilisation

Identify if it is process driven or interrupts
Total CPU usage (Process + Interrupt)
CPU usage due to Interrupt
DUT# show process cpu

PID Runtime(ms) Invoked uSecs
5Sec
1Min
5Min TTY Process
2
720
88
8181
9.12% 1.11% 0.23% 18 Virtual Exec
Next Steps
Process: recurring events, control plane process etc.
Interrupts: incorrect switching path, system exceeding hardware
resources
BRKRST-3067
Cisco Public
49
High CPU Utilisation - Process

Process: ARP Input
Caused by ARP flooding
Static route configured with next-hop interface instead of IP
Incrementing at very high
rate
VLAN SVI is Virtual and

counter gives only the
amount/rate of processswitched or control-pane
traffic
BRKRST-3067
Cat6K#show ip traffic | begin ARP

ARP statistics:
Rcvd: 6512 requests, 2092 replies, 0 reverse, 0 other
Sent: 258 requests, 707 replies (0 proxy), 0 reverse
Drop due to input queue full: 20
<snip>
Cat6K#show interfaces | include line protocol|rate
Vlan501 is up, line protocol is up
5 minute input rate 23013521 bits/sec, 2535 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
Cisco Public
Look for abnormal input rate
50

Process: IP Input
Caused by traffic that needs to be process switched or sent to the CPU
Common Causes and Next Steps
Find the source host and isolate. Consider per-port broadcast storm-control
Broadcast Storm
Traffic with IP-Options enabled
Find the source host and isolate.

Consider HW Rate-limiters
Traffic to which ICMP redirect or Unreachable required e.g., TTL=1, ACL Deny
Traffic that needs further CPU processing e.g., ACL Logging
Consider Optimised ACL
Logging (OAL)
BRKRST-3067
Cisco Public
Under VLAN SVIs

configure:
no ip unreachables
no ip redirects
no ip proxy-arp
Consider HW Ratelimiters
51

Process: Exec and Virtual Exec
Responsible for tty lines

(console, auxiliary)
Cat6K#show process cpu | include CPU|Virtual |Exec

PID Runtime(ms)
Invoked
uSecs
5Sec
1Min
5Min TTY Process
3
272
194
1402 29.00% 2.12% 1.89%
0 Exec
54
180
1443
124
0.00% 0.00% 0.00%
1 Virtual Exec
High CPU when too many

messages sent to console/vty
Responsible for vty lines

(telnet, SSH)
Next steps
Check if any debug is enabled via show debug. Issue undebug all if it is
not needed
Disable logging by no logging console or no logging terminal
BRKRST-3067
Cisco Public
52
High CPU Utilisation - Traffic

Analyse input buffer
Find the interface that's holding

most of the buffers
Use when an input queue is oversubscribed

Cat6K#show buffers assigned
Header
DataArea Pool Rcnt Size Link
46FDBC14 8029784 Small
1
77
36
46FE0010 802CBC4 Small
1
77
36
. . .
Enc
1
1
Flags
200
200
Input
Vl100
Vl100
Output
None
None
Cat6K#show buffers input-interface vlan 100 dump

Buffer information for RxQ3 buffer at 0x378B3BC
data_area 0x7C05EF0, refcount 1, next 0x0, flags 0x200
linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
if_input 0x46C7C68 (Vlan100), if_output 0x0 (None)
inputtime 2d03h (elapsed 00:00:01.024)
outputtime 00:00:00.000 (elapsed never), oqnumber 65535
datagramstart 0x7C05F36, datagramsize 62, maximum size 2196
mac_start 0x7C05F36, addr_start 0x7C05F36, info_start 0x0
network_start 0x7C05F44, transport_start 0x7C05F58, caller_pc 0x6C1564
source: 137.34.219.3, destination: 224.0.0.2, id: 0x0000, ttl: 1,
TOS: 192 prot: 17, source port 1985, destination port 1985
BRKRST-3067
Cisco Public
Packet details
53

Debug netdr capture
Capture packets being received and sent by RP to buffer space
Does not introduce extra CPU processing
Cat6K#debug netdr cap ?
acl
and-filter
continuous
destination-ip-address
dstindex
ethertype
interface
or-filter
rx
source-ip-address
srcindex
tx
vlan
<cr>
BRKRST-3067
(11) Capture packets matching an acl

(3) Apply filters in an and function: all must match
(1) Capture packets continuously: cyclic overwrite
(10) Capture all packets matching ip dst address
(7) Capture all packets matching destination index
(8) Capture all packets matching ethertype
(4) Capture packets related to this interface
(3) Apply filters in an or function: only one must
match
(2) Capture incoming packets only
(9) Capture all packets matching ip src address
(6) Capture all packets matching source index
(2) Capture outgoing packets only
(5) Capture packets matching this vlan number
Cisco Public
54

Netdr capture output
Cat6K#show netdr cap
A total of 10 packets have been captured
The capture buffer wrapped 0 times
Total capture capacity: 4096 packets
Vlan
------- dump of incoming inband packet ------interface Vl10, routine draco2_process_rx_packet_inline

dbus info: src_vlan 0xA(10), src_indx 0xC0(192), len 0x76(118)
bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
08020401 000A0000 00C00000 76080000 00010428 0E000040 00000000 03800000
mistral hdr: req_token 0x0(0), src_index 0xC0(192), rx_offset 0x76(118)
Ethertype
requeue 0, obl_pkt 0, vlan 0xA(10)
destmac 00.23.04.18.F8.80, srcmac 00.13.7F.8B.84.C1, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 100, identifier 13202
df 0, mf 0, fo 0, ttl 255, src 10.10.10.1, dst 10.10.10.3
icmp type 8, code 0
------- dump of incoming inband packet ------interface Vl10, routine draco2_process_rx_packet_inline
dbus info: src_vlan 0xA(10), src_indx 0xC0(192), len 0x76(118)
bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
BRKRST-3067
Cisco Public
Source/Destination
55
Agenda
Sup720 Architecture


Sup2T Architecture
BRKRST-3067
Cisco Public
56
Virtual Switch System (VSS)

Virtual Switching System consists of two Cisco Catalyst 6500 Series
switches connected through a special etherchannel called a Virtual
Switch Link (VSL) to become one logical entity.
Virtual Switch Domain
Virtual Switch Link (VSL)
Switch 1
BRKRST-3067
Switch 2
Cisco Public
One Logical Switch
57
VSS Control Plane

VSS Specific Protocols
VSL Protocol (VSLP) runs between active and standby switch over
the VSL, and has two components: LMP and RRP
Link Management Protocol (LMP): Runs over each individual link
in VSL
Bi-directionality; keepalives; connectivity (peer-to-peer)
Role Resolution Protocol (RRP): Runs on each side (each peer)

of the VSL port channel
HW/SW compatibility; role negotiation; control-link selection
BRKRST-3067
Cisco Public
58
VSS Control Plane

VSS Configuration Check
Cat6K#show switch virtual role
Switch Switch Status Preempt
Priority Role
Session ID
Number
Oper(Conf) Oper(Conf)
Local Remote
-----------------------------------------------------------------LOCAL
1
UP
TRUE (Y*)
200(200) ACTIVE
0
0
REMOTE
2
UP
TRUE (Y*)
100(100) STANDBY 2977
3643
Standby configured preempt timer(switch 2): 5 minutes
Active configured preempt timer(switch 1): 5 minutes
In dual-active recovery mode: No
Cat6K#show switch virtual link port-channel
Flags: D - down
P - bundled in port-channel
Interfaces are identified by
. . .
[switch#]/[mod#]/[port#]
Group Port-channel Protocol
Ports
------+-------------+-----------+------------------256
Po256(RU)
Te1/3/3(P)
Te1/3/4(P)
Te1/5/4(P)
255
Po255(RU)
Te2/2/3(P)
Te2/2/6(P)
Te2/5/4(P)
BRKRST-3067
Cisco Public
Switch 1 is active,
switch 2 is standby,
both are up
Te1/3/6(P)
Te2/2/8(P)
Switch 1
side of the
VSL
Switch 2
side of the
VSL
59
VSS Control Plane

VSS LMP Check
Cat6K#show switch virtual link
Status and uptime of the VSL
VSL Status : UP
VSL Uptime : 18 hours, 12 minutes
VSL SCP Ping : Pass
Control link carries EOBC
VSL ICC Ping : Pass
and IBC control messages
VSL Control Link : Te1/5/4
(SCP and ICC/IPC)
Cat6K#show switch virtual link port
LMP summary
Link info:
Configured: 4
Operational: 4
Peer Peer
Peer
Peer
Timer(s)running
Interface Flag State
Flag MAC
Switch Interface (Time remaining)
-------------------------------------------------------------------------------Te1/5/4
vfs operational vfs 0011.bc75.4400 2
Te2/5/4
T4(220ms)
T5(175s)
Te1/3/3
Te2/2/6
T4(220ms)
T5(175s)
Te1/3/4
Te2/2/8
T4(220ms)
Check LMP state and
T5(175s)
Flags (vf) of the links in the
VSL bundle
Te1/3/6
Te2/2/3
T4(768ms)
T5(175s)
Flags: v - Valid flag set
f - Bi-directional flag set s - Negotiation flag set
BRKRST-3067
Cisco Public
60
VSS Control Plane

VSS LMP Check
Indicates problem sending

or receiving LMP packets
from peer
Cat6k#show vslp lmp counters

Instance #1:
LMP counters
Tx
Rx
Interface
OK
Fail
Bidir
Uni
Fail
Bad
-------------------------------------------------------------------Te1/5/4
12649
0
12675
1
0
0
Te1/3/3
12000
0
12024
0
0
0
Te1/3/4
11999
0
12024
0
0
0
Te1/3/6
12001
0
12025
0
0
0
Indicates
problem
sending LMP
packet to the
VSL peer
Packets received from VSL

peer without our info, proving
the link is unidirectional at that
moment
Rx error details
Interface My info
My info
Bad MAC
Bad switch Domain id
Peer info
mismatch
absent
Address
id
mismatch
mismatch
------------------------------------------------------------------------------Te1/5/4
0
1
0
0
0
0
Te1/3/3
0
0
0
0
0
0
These errors usually indicate a
Te1/3/4
0
0
0
0
0
0
misconfiguration on one of the peers
Te1/3/6
0
0
0
0
0
0
1 unidirectional packet when
first link in VSL came up
BRKRST-3067
Cisco Public
61
VSS Control Plane

VSS RRP Check
Cat6K#show switch virtual role detail
Switch Switch Status Preempt
Priority Role
Session ID
Number
Local Remote
Check 1 is active, 2 is standby
-----------------------------------------------------------------LOCAL
1
UP
TRUE (Y*)
200(200) ACTIVE
0
0
REMOTE
2
UP
TRUE (Y*)
100(100) STANDBY 2977
3643
Standby configured preempt timer(switch 2): 5 minutes
RRP Counters:
-------------------------------------------------------------------Inst. Peer Direction Req
Acc
Est
Rsugg
Racc
State machine info on RRP protocol;
---------------------------------------------------------------------current state is role resolved
1
1
Tx
0
2
0
2
6
1
1
Rx
2
0
2
0
6
RRP FSM info
---------------------------------------------------------------------sm(vslp_rrp RRP SM information for Instance 1, Peer 1), running yes, state role_res
Last transition recorded: (req)-> hold (srt_exp)-> hold (est)-> role_neg (srt_exp)-> role_neg
(est)-> role_neg (racc)-> role_res (srt
Switch is not in dual active recovery mode
_. . .
BRKRST-3067
Cisco Public
62
VSS L2/L3 Forwarding Path
(VLAN 701 705)
Gig5/9
Gig4/16
Gig2/2
Gig5/2
Gig2/4
Physical
BRKRST-3067
Host1
8.0.1.1
R1
Cisco Public
Host1
8.0.1.1
Ten1/1
Ten1/4
Ten2/2/7
Cat6K
Gig2/6/12
Gig1/9/36
Gig2/9/15
Gig1/6/2
Gig1/5/1
Logical
VSL
Gig5/9
Gig4/16
Gig2/2
Gig5/2
Gig2/4
Host2
9.0.1.2
Host2
9.0.1.2
Ten1/3/2
R2
R2
Po2
Po1
R1
63
VSS Data Plane Forwarding Path

VSS Data Plane Design: Minimise Load on VSL
Multi-chassis Ether Channel (MEC):
VSL
Channel hash is modified on VSS so

that local links in the MEC are
preferred over links on remote switch
MEC
Equal Cost Multi Path (ECMP):

Adjacency table is modified on VSS to
prefer next hops attached to local
switch
BRKRST-3067
Cisco Public
64
VSS L2/L3 Forwarding

VSS Data Plane Troubleshooting L2 MEC
VSS specific commands

augmented with switch id
Verify the load-balancing algorithm used

Cat6K#show etherchannel load-balance switch 2 module 2
EtherChannel Load-Balancing Configuration:
src-dst-ip enhanced
mpls label-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address
MPLS: Label or IP
Important: Only use parameters consistent

with the configured load-balancing algorithm.
Command uses all the specified arguments to
calculate the hash.
Identify the physical path for flow from host 2 host 1

Cat6K#show etherchannel load-balance hash-result interface Port-channel 2 switch 1 ip
9.0.1.2
vlan 705 8.0.1.1
Computed RBH: 0x6

Would select Gi1/6/2 of Po2
Cat6K#show etherchannel load-balance hash-result interface Port-channel 2 switch 2 ip
9.0.1.2 vlan 705 8.0.1.1

Computed RBH: 0x6
Would select Gi2/9/15 of Po2
BRKRST-3067
Cisco Public
65
VSS L2/L3 Forwarding
Routing table shows two equal cost paths to 9.0.0.0/8

Cat6K#show ip route 9.0.0.0 | i via
Known via "eigrp 101", distance 90, metric 3072, type internal
7.7.1.2, from 7.7.1.2, 1d00h ago, via TenGigabitEthernet2/2/7
* 7.6.1.2, from 7.6.1.2, 1d00h ago, via TenGigabitEthernet1/3/2
Looking at the HW table shows next hop directly attached to local switch is preferred
Cat6K#show mls cef lookup 9.0.1.0 switch 1 mod 3
Packet coming in on switch 1 module 3, for 9.0.0.0/8
prefers next hop attached to local switch id 1
Index Prefix
Adjacency
108775 9.0.0.0/8
Te1/3/2
, 000f.35ed.7c00
Cat6K#show mls cef lookup 9.0.1.0 switch 2 mod 2
Packet coming in on switch 2 module 2, for 9.0.0.0/8
prefers next hop attached to local switch id 2
Index Prefix
Adjacency
108775 9.0.0.0/8
Te2/2/7
, 000f.35ed.7c00
Cat6K#show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 1 mod 3
Interface: Te1/3/2, Next Hop: 7.6.1.2, Vlan: 4064, Destination Mac: 000f.35ed.7c00
Cat6K#show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 2 mod 2
Interface: Te2/2/7, Next Hop: 7.7.1.2, Vlan: 4056, Destination Mac: 000f.35ed.7c00
BRKRST-3067
Cisco Public
66
VSS Dual-Active Detection

VSL Failure
Both switches assume the active role
Im active
Im active
Leads to two independent routers with

identical configurations
VSL
Three methods for detection and recovery

Active
VSLP Fast Hello
Virtual Switch Domain
Standby
Enhanced Port Aggregation Protocol (ePAgP)
Bi-directional Forwarding Detection (BFD)
BRKRST-3067
Cisco Public
67

VSS Dual Active Detection Methods
VSLP Fast Hello: (available in 12.2(33)SXI onwards)
Uses a dedicated heartbeat link between the VSL peers to send periodic hellos
Received hellos contain remote VSL peers current state and role
Enhanced PAgP
Uses new Type-Length-Value (TLV) field in PAgP packets. contains the ID of
the active switch
Requires Multi-chassis EtherChannel (MEC) switch support ePAgP
BFD
Uses a dedicated connection between VSL peers
Requires IP addresses in different subnet on the interfaces in the BFD link
BRKRST-3067
Cisco Public
68

VSS Dual Active Detection Setup
Logical
VSL
Physical
Po1
Po2
Po1
Ten1/4
Ten2/2/7
R1
R1
BRKRST-3067
Gig1/6/1
Gig5/9
Gig4/16
Gig2/2
Gig5/2
Gig2/4
Gig5/9
Gig4/16
Gig2/2
Gig5/2
Gig2/4
Po2
Port-Channel 2 is the only portchannel trusted for ePAgP dualactive detection
Cat6K
Gig2/6/12
Gig1/9/36
Gig2/9/15
Gig1/6/2
Gig1/5/1
Gig1/6/1
Gig2/9/1
Fast Hello
Ten1/1
R2
Ten1/3/2
R2
Cisco Public
69

VSS Dual Active Detection Troubleshooting
Cat6K#show startup-config | begin switch virtual
switch virtual domain 1
BFD Dual Active detection is explicitly disabled
switch mode virtual
...
Enhanced PAgP messages are only sent on
no dual-active detection bfd
channels configured in trust mode
dual-active detection pagp trust channel-group 2
dual-active exclude interface GigabitEthernet1/5/3
dual-active exclude interface GigabitEthernet2/5/3
!
...
Interfaces can be excluded from recovery
interface GigabitEthernet1/6/1
mode. They will not go down when the
switch goes into recovery mode
no switchport
no ip address
dual-active fast-hello
...
interface GigabitEthernet2/9/1
no switchport
no ip address
dual-active fast-hello
BRKRST-3067
Cisco Public
70

Cat6K#show switch virtual dual-active summary
Shows that both enhanced PAgP
and fast hello methods of dual
active detection are enabled
Pagp dual-active detection enabled: Yes

Fast-hello dual-active detection enabled: Yes
Interfaces excluded from shutdown in recovery mode:
Gi1/5/3
Gi2/5/3
Port channel 2 is trusted for Enhanced

PAGP dual active detection at least 1
trusted port channel needed !!
Cat6K#show switch virtual dual-active pagp

PAgP dual-active detection enabled: Yes
PAgP dual-active version: 1.1
Channel group 2 dual-active detect capability w/nbrs

Dual-Active trusted group: Yes
Dual-Active
Partner
Partner
Partner
Port
Detect Capable
Name
Port
Version
Gi1/5/1
Yes
R1
Gi2/4
1.1
Gi1/6/2
Yes
R1
Gi5/2
1.1
Check that the neighbour runs a

SW version that supports
Enhanced PAGP if not, no
dual active detection !!
BRKRST-3067
Cisco Public
71

Recovery Mode: all non-VSL and all non-excluded ports will be

internally shutdown
VSL
SW1
SW2
Active
Standby
Gig1/6/1
Gig2/9/1
Fast Hello
Po2
I lost my VSL.
Let me take over as Active!
ePAgP
Gig5/9
Gig4/16
Gig2/2
Gig5/2
Gig2/4
Looks like my peer is still

there and becoming
Active. Let me go to
Recovery Mode!
Po1
R1
BRKRST-3067
Cisco Public
72

Trigger dual active situation by bringing down the VSL
Enhanced PAGP
detected both switch ids
were active at the same
time
*Apr 1 12:40:22.885 CET: %PAGP_DUAL_ACTIVE-SW1_SP-1-RECOVERY: PAgP running on Gi1/5/1

triggered dual-active recovery: active id 0011.bc75.4400 received, expected 0011.5d54.6800
*Apr 1 12:40:22.945 CET: %DUAL_ACTIVE-SW1_SP-1-DETECTION: Dual-active condition detected: all
non-VSL and non-excluded interfaces have been shut down
Original active switch (switch 1 in
example) goes into recovery mode
On switch in recovery mode, do show ip interface brief | inc up to

check the interfaces that are up.
Apr 1 12:40:20.096 CET: %VSLP-SW2_SPSTBY-2-VSL_DOWN: All VSL links went down while switch is
in Standby role
*Apr 1 12:40:20.096 CET: %DUAL_ACTIVE-SW2_SPSTBY-1-VSL_DOWN: VSL is down switchover, or
possible dual-active situation has occurred
*Apr 1 12:40:20.100 CET: %PFREDUN-SW2_SPSTBY-6-ACTIVE: Initializing as Virtual Switch ACTIVE
processor
Original Standby switch goes into Active mode
BRKRST-3067
Cisco Public
73

Switch
Switch Status
Preempt
Priority
Role
Number
Session ID
Local
On switch id 1: originally active, now

Remote in recovery mode
-----------------------------------------------------------------LOCAL
UP
TRUE (Y*)
200(200)
ACTIVE
Does not see switch 2

(as VSL is still down)

In dual-active recovery mode: Yes
Triggered by: PAgP detection
Mechanism that detected dual active was

Enhanced PAgP, via link 1/5/1
Triggered on interface: Gi1/5/1

<snip>
Switch
Switch Status
Preempt
Priority
Role
Number
Session ID
Local
Remote
-----------------------------------------------------------------LOCAL
UP
TRUE (Y*)
100(100)
ACTIVE

BRKRST-3067
Cisco Public
Switch id 2 is now the real active

switch, and doesnt see switch id 1 as
long as the VSL is down
74

Bringing up the VSL
*Apr 1 12:49:29.513 CET: %DUAL_ACTIVE-1-VSL_RECOVERED: VSL has
situation: Reloading switch 1
recovered during dual-active
*Apr 1 12:49:29.513 CET: %VS_GENERIC-5-VS_CONFIG_DIRTY: Configuration

reload request until configuration is saved
has changed. Ignored
*Apr 1 12:49:32.781 CET: %LINK-SW2_SP-3-UPDOWN: Interface TenGigabitEthernet2/5/4 changed

state to up
*Apr 1 12:49:49.128 CET: %VSLP-SW2_SP-5-VSL_UP:
MAC=0011.5d54.6800 over Te2/2/6
*Apr
Ready for Role Resolution with Switch=1,
1 12:49:50.320 CET: Initializing as Virtual Switch ACTIVE processor
*Apr 1 12:49:52.140 CET: %VSLP-SW2_SP-5-RRP_MSG: Peer Switch with unsaved configurations needs
to be reloaded. Please save relevant configurations on the peer switch and reload it.
Configuration deemed DIRTY since configuration mode was entered and

configuration was not written. Do NOT change configuration when the switch
is in Recovery mode.
Configuration must be saved MANUALLY and reloaded to bring the switch
back up into the VSS
BRKRST-3067
Cisco Public
75
Agenda
Sup720 Architecture

Sup2T Architecture
BRKRST-3067
Cisco Public
76
Supervisor 2T/PFC4 Architecture

Fabric interface
/ Replication
engine to
interface with
fabric & bus,
and for
multicast/SPAN
replication
MSFC5 Complex contains

single (combined SP/RP)
CMP CPU
NetFlow
Hash &
Data
tables for
stats &
features
1GE /
10GE
Uplinks
MSFC 5
Central
Management
Processor
FIB contains IPv4 /

IPv6 prefixes and
MPLS entries
CL2 TCAM
ADJ
contains
contains
QoS ACL
Rewrite
entries
info
NetFlow
2 x 1 Gbps
LIF Map
contains port
mappings
CL1
ADJ
FIB
CL2
TCAM TCAM TCAM TCAM
Port ASIC
CPU
DRAM
CL1 TCAM
contains
Security &
Feature ACL
entries
LIF
MAP
L3/4 Engine
RPF
Table
Flash
MET
Switch Fabric
26 x 40G Traces
Switch Fabric
Traces # 1 to 26
BRKRST-3067
40 Gbps
EOBC
Fabric Interface
&
Replication Engine
DBUS
RBUS
LIF Table
LIF Stats
L2 Engine
L2 CAM (128K)
PF
C4
Logical Interface Statistics

ACE
Counter
Cisco Public
Logical
Interface (LIF)
Table contains
entries
Reverse Path
Forwarding info
Hardware
for ACL
TCAM
counters
L2 CAM
contains
MAC
entries
77
Sup720 vs. Sup2T Overview

Sup720
Sup2T
Route Processor (RP)
Control-Plane
Data-Plane
BRKRST-3067
Switch Processor (SP)
Dual Core
Processor
MSFC 3
MSFC 5
Policy Feature Card
Policy Feature Card
PFC3
PFC4
Cisco Public
78
Sup2T Overview
Integrated with Control Management Processor (CMP)
USB-based console and flash support
Key improvement: Single-Cycle Input & Output Processing
Has superior Control-Plane protection.
Supports all Sup720 features, including VSS
BRKRST-3067
Cisco Public
79
Agenda
Sup720 Architecture

Sup2T Architecture
BRKRST-3067
Cisco Public
80

Verify IP route
R2
.2
Gig3/25
.5
.6
Gig3/26
Sup2T
Sup2T#show ip route 192.168.200.9

Routing entry for 192.168.200.8/30
Known via "static", distance 1, metric 0
Routing Descriptor Blocks:
192.168.100.5
Route metric is 0, traffic share count is 1
* 192.168.100.1
BRKRST-3067
Gig3/1
192.168.100.4/30
.1
192.168.100.0/30
Host2 Gig3/2
192.168.200.9
Sup2T#show ip route 192.168.200.5

Routing entry for 192.168.200.4/30
Known via "static", distance 1, metric 0
Routing Descriptor Blocks:
192.168.25.5
* 192.168.25.1
.1
Gig3/1
Cisco Public
Host1
192.168.200.5
Gig1/26
192.168.25.4/30
.2
192.168.25.0/30
Gig1/25
.6
.5
Gig3/2
R1
81

SW
192.168.200.8/30
R2
nexthop 192.168.100.5 GigabitEthernet3/26
.1
Sup2T#show ip cef exact-route 192.168.200.5 192.168.200.9
.2
192.168.200.5 -> 192.168.200.9 => IP adj out of GigabitEthernet3/25, addr

192.168.100.1
192.168.100.0/30
Host2 Gig3/2
192.168.200.9
Gig3/1
192.168.100.4/30
Sup2T#show ip cef 192.168.200.9
Gig3/25
.5
.6
Gig3/26
Sup2T#show ip cef adjacency GigabitEthernet 3/25 192.168.100.1

192.168.100.1/32
Sup2T
attached to GigabitEthernet3/25
192.168.200.8/30
No more MLS. Use

platform hardware.
Index Prefix
219265 192.168.200.8/30
Adjacency
Gi3/25
Gi3/26
.2
,001d.e656.cc00 (Hash: 007F)

,001d.e656.cc00 (Hash: 7F80)
.1
Sup2T#show platform hardware cef exact-route 192.169.200.5 0
Gig3/1
192.168.200.9 0
Interface: Gi3/26, Next Hop: 192.168.100.5, ifnum: 0x7E, Destination Mac:
001d.e656.cc00
LIF: 0x2000411B
BRKRST-3067
Cisco Public
Host1
192.168.200.5
Gig1/26
192.168.25.4/30
Gig1/25
HW
192.168.25.0/30
Sup2T#show platform hardware cef lookup 192.168.200.9
.6
.5
Gig3/2
R1
82
R2
SW
.2
Cisco Public
.6
Gig3/26
.2
Host1
192.168.200.5
Gig1/26
192.168.25.4/30
Gig1/25
Gig3/1
.5
Sup2T
.1
BRKRST-3067
Gig3/1
Gig3/25
192.168.25.0/30
Sup2T#show adjacency Gig3/25 192.168.100.1 detail

Protocol Interface
Address
IP
GigabitEthernet3/25
192.168.100.1(13)
2001 packets, 11682 bytes
epoch 2
sourced in sev-epoch 29
Encap length 14
001DE656CC00001DE65828000800
L2 destination address byte offset 0
L2 destination address byte length 6
Link-type after encap: ip
Rewrite information
ARP
(Dmac|Smac|0800): verify it is
conform with next hop rewrite info
.1
192.168.100.0/30
Host2 Gig3/2
192.168.200.9
192.168.100.4/30
L3 FIB counters and tables
.6
.5
Gig3/2
R1
83
R2
HW
.2
Gig3/25
Codes: M - mask entry, V - value entry, A - adjacency index, NR- no_route bit
LS - load sharing count, RI - router_ip bit, DF: default bit
CP - copy_to_cpu bit, AS: dest_AS_number, DGTv - dgt_valid bit
DGT: dgt/others value
Format:IPV4 (valid class vpn
prefix)
M(54794 ):
1
F
3FFF 255.255.255.255
V(54794 ):
1
0
0
192.168.100.5
(A:376833, LS:0, NR:0, RI:0, DF:0 CP:0 DGTv:1,
DGT:0)
.2
.1
Gig3/1
Cisco Public
.6
Gig3/26
Gig1/25
Start adjacency pointer is 376833
BRKRST-3067
.5
Sup2T
192.168.25.0/30
Sup2T#show platform hardware cef ip 192.168.100.5 detail
Gig3/1
192.168.100.4/30
.1
192.168.100.0/30
Host2 Gig3/2
192.168.200.9
Host1
192.168.200.5
Gig1/26
192.168.25.4/30
.6
.5
Gig3/2
R1
84
R2
.2
Gig3/1
192.168.100.4/30
.1
Sup2T#show platform hardware cef adjacencies entry 376833 detail

Index: 376833 -- Valid entry (valid = 1)
Adjacency fields:
___________________________________________________
|adj_stats = EN | fwd_stats = EN | trig = 0
|_________________|__________________|______________
|l3_enable = ON (classify as Layer3) | age = 0
|_________________|__________________|______________
|format = IP
| rdt = ON
| ignr_emut = 0
|_________________|__________________|______________
|vpn = 0x3FFF
| elif = 0x411B
| ri = 3
|_________________|__________________|______________
|top_sel = 0
| zone_enf = OFF
| fltr_en = OFF
|_________________|__________________|______________
|frr_te = OFF
| idx_sel = 0
| tnl_encap = 0
|_________________|__________________|______________
|rw_hint = 0
| ttl_control = 4 |
|_________________|__________________|______________
192.168.100.0/30
Host2 Gig3/2
192.168.200.9
Gig3/25
.5
.6
Gig3/26
Sup2T
.2
.1
192.168.25.0/30
Gig1/25
Gig3/1
Gig1/26
192.168.25.4/30
.6
.5
Gig3/2
. . . Continued . . .
BRKRST-3067
Cisco Public
Host1
192.168.200.5
R1
85

BRKRST-3067
Cisco Public
192.168.100.4/30
.2
Gig3/1
Gig3/25
.5
.6
Gig3/26
Sup2T
Gig1/25
.2
Rewrite MAC info
.1
Gig3/1
Host1
192.168.200.5
Gig1/26
192.168.25.4/30
RIT fields: The entry has a Layer2 Format

_________________________________________________________
|decr_ttl = YES
| pipe_ttl = 0
| utos = 0
|_________________|__________________|____________________
|l2_fwd = 0
| rmac = 0
| ccc = L3_REWRITE
|_________________|__________________|____________________
|rm_null_lbl = YES| rm_last_lbl = YES| pv = 0
|_________________|__________________|____________________
|add_shim_hdr= NO | rec_findex = N/A | rec_shim_op = N/A
|_________________|__________________|____________________
|rec_dti_type = N/A
| rec_data = N/A
|____________________________________|____________________
|modify_smac = YES| modify_dmac = YES| egress_mcast = NO
|____________________________________|____________________
|ip_to_mac = NO
|_________________________________________________________
|dest_mac = 001d.e656.cc00
| src_mac = 001d.e658.2800
|___________________________|_____________________________
|
Statistics
Statistics: Packets = 1138
Bytes
= 569016
.1
192.168.100.0/30
Host2 Gig3/2
192.168.200.9
Continued . . .
192.168.25.0/30
. .
R2
.6
.5
Gig3/2
R1
86
Session Summary
Understanding of the architecture of Sup6-E Sup7-E, Sup-720 and
Sup-2T
Input and output buffers of 4500 line cards
Tools to troubleshoot high CPU
Understanding of the IOS-XE architecture
Verifying L2 and L3 forwarding path through Sup-720 and Sup-2T
Troubleshooting all that is VSS
..Ensuring packets get to their destination as efficiently as possible
BRKRST-3067
Cisco Public
87
Q&A
Complete Your Online Session

Evaluation
Complete your session evaluation:
Directly from your mobile device by visiting
www.ciscoliveaustralia.com/mobile and login
by entering your username and password
Visit one of the Cisco Live internet
stations located throughout the venue
Open a browser on your own computer
to access the Cisco Live onsite portal
BRKRST-3067
Dont forget to activate your Cisco Live

Virtual account for access to all session
materials, communities, and on-demand and
live activities throughout the year. Activate your
account at any internet station or visit
www.ciscolivevirtual.com.
Cisco Public
89
BRKRST-3067
Cisco Public
90
Appendix
Catalyst 4500
Command Summaries
Chassis, Supervisor, Line Card comparison
HA Redundancy
ISSU
Unicast Forwarding
Catalyst 6500
GOLD/ EEM Health Monitoring
BRKRST-3067
Cisco Public
91
Troubleshooting Power over Ethernet

Command Summary
Troubleshooting Steps
Commands
Verify PoE line card is online
show module
Verify power supplies are capable of supplying inline power
show power detail
Verify inline power available and operational
show power detail
Verify the inline power status of the port
show power inline <interface> [detail]
Verify port configuration and allocated power in agreement
show running-config interface <interface>
Verify PoE line card supports enough power per port, per slot
Appendix table, line card datasheets
Verify phone is not drawing more power than it should
show power inline police <interface>
Verify power negotiation is successful
debug interface <interface>

debug ilpower powerman
undebug all
undebug interface <interface>
Verify whether problem persists after changing ports, stub ASICs, and/or line
cards
show power inline <interface>
BRKRST-3067
Cisco Public
92

Command Summary
Commands
Check CPU usage on IOS threads
show process cpu detailed process iosd [sorted]
Check CPU usage on platform dependent threads
show platform health
Check traffic on the CPU queue
show platform cpu packet statistics
SPAN the traffic send to the CPU queue
monitor session 1 source cpu rx

monitor session 1 destination interface Gi1/48
SPAN the traffic send to the CPU queue using internal

inband capture tool
debug platform packet all buffer

show platform cpu packet buffered
BRKRST-3067
Cisco Public
93
Troubleshooting Licensing
Command Summary
Commands
Display all licenses
show license all
Currently used license
show license in-use
Detailed license information
show license detail <feature name>
Displays evaluation license
show license evaluation
Displays expiring license
show license expiring
Shows all the license files
show license file
Shows license handles checkout

activity
show license handle
Shows all permanent licenses
show license permanent
Displays license statistics
show license statistics
Displays license status
show license status
Brief summary of license(s)
show license summary
BRKRST-3067
Cisco Public
94
Troubleshooting QoS
Command Summary
Commands
Check QoS configuration
show running-config
Check classification/marking/policing on interface
show policy-map interface <>
Check freelist availability
show platform hardware interface all | include FreeListCount
Check QoS TCAM resource
show platform hardware acl statistics utilization brief
Check policer hardware resource
show platform hardware qos policer utilization
Check interface hardware tx-queue
show platform hardware interface <> tx-queue
Check various drops on interface
show interface <> count all
BRKRST-3067
Cisco Public
95
Troubleshooting Flexible NetFlow

Command Summary
Display the flow label information
Commands
show platform hardware flow label
show platform hardware flow table label <value>
Display flow classification information
show platform hardware flow classification
Display the flow parameter table
show platform hardware flow classification
Display the flow parameter table
show platform hardware flow parameters
Netflow engine label mapping information
show platform software flow mapping
State of various FNF platform modules
show platform software flow state
BRKRST-3067
Cisco Public
96
Appendix
Catalyst 4500
Command Summaries
HA Redundancy
ISSU
Unicast Forwarding
Catalyst 6500
BRKRST-3067
Cisco Public
97
Catalyst 4500 +E, -E and classic chassis

Modular
4503-E
4507R+E
4510R+E
4506-E
24 Gbps per slot

-E Chassis support 12.2(31)SGA6
onward
-E/+E chassis, E-Series Sup and
46xx line card
48 Gbps per slot

+E Chassis support 12.2(53)SG4 onward
Sup7-E, 47xx line card IOS-XE 3.2.nSG
4507R+E, 4510R+E, 4503-E, 4506-E
6 Gbps per slot

E-Series and Classic supervisors
Classic line cards
See the appendix for supervisor, line card, and chassis product and compatibility details.
BRKRST-3067
Cisco Public
98
Catalyst 4900M and 4900 Series

Fixed Configuration
Catalyst 4948E
176 Gbps capacity1

4 Gigabit Ethernet SFP/SFP+ uplinks
10/100/1000
Ethernet downlinks
320 Gbps capacity
Catalyst 4900M
Catalyst 4948-10GE
Catalyst 4948
BRKRST-3067
Fixed 10 Gigabit Ethernet (10GE) ports
10/100/1000, GE, and 10GE half-cards
136 Gbps capacity

10 Gigabit Ethernet uplinks
10/100/1000 Ethernet downlinks
96 Gbps capacity
Gigabit Ethernet SFP uplinks

10/100/1000 Ethernet downlinks
Cisco Public
99
Supervisor Comparison
Layer 2/3/4
Services
Supervisor 7-E
Supervisor 6-E
Supervisor 6L-E
Bandwidth3 &
Throughput
Full L2/3/4,
enhanced routing1,
IPv6
848 Gbps
Full L2/3/4,
enhanced routing1,
IPv6
320 Gbps
Basic Layer 2/3/4
280 Gbps
Chassis Support
225 mpps
Supervisor V10GE
Full L2/3/4,
enhanced routing,
IPv62
136 Gbps
Supervisor V
See Sup V-10GE
96 Gbps
Uplinks
NetFlow
-E/+E Chassis
E-series and
classic
4x10GE
Yes
All E-Series and

Classic
E-Series &
Classic
4 x GE or 2 x 10
GE (TwinGig)
No
All E-Series and

Classic
E-Series &
Classic
2 x 10 GE, 4 x GE,
10 GE and 2 x GE,
or 2 x GE and
10GE
No
See Sup 6-E
Classic
4 x GE and 2 x
10GE
Yes
See Sup 6-E
Classic
2 x GE
Optional
All except 4510R-E

& 4510R
Classic
2 x GE
Optional
See Sup IV
Classic
4 x GE and 2 x
10GE
No
See Sup IV
Classic
2 x GE
No
4503-E 4503
Classic
8 x GE & 12
10/100/1000 PoE
No
250 mpps
250 mpps
Line Cards
102 mpps
72 Mpps
Supervisor IV
See Sup V-10GE
64 Gbps
48 mpps
Supervisor IIPlus-10GE
Basic Layer 2/3/4
Supervisor IIPlus
Basic Layer 2/3/4
Supervisor IIPlus-TS
Basic Layer 2/3/4
1. EIGRP,
BRKRST-3067
108 Gbps
81 mpps
64 Gbps
48 mpps
64 Gbps
48 mpps
OSPF, BGP, IS-IS.

2. IPv6
2012 Cisco
and/orIn
its software.
affiliates. All rights reserved.
3. Bi-directional.
Cisco Public
100
Supervisor Comparison, Continued

Unicast
Routes
Security ACL & QoS

entries
Input & Output Policers
NAC & DHCP

snooping entries
Supervisor 7-E
256,000
128,000
16,000 input/output, user

configurable
12,000
Supervisor 6-E
256,000
128,000

configurable
12,000
Supervisor 6L-E
57,000
32,000

configurable
3072
Supervisor V10GE
128,000
64,000
8,000 each direction
6,000
Supervisor V
128,000
64,000
3,000
Supervisor IV
128,000
64,000
3,000
Supervisor IIPlus-10GE
32,000
32,000
512 each direction
3,000
Supervisor II-Plus
32,000
32,000
512 each direction
3,000
Supervisor IIPlus-TS
32,000
32,000
512 each direction
3,000
BRKRST-3067
Cisco Public
101
Line Card Comparison

Type
+E series
Line Card
BRKRST-3067
Speed
Port Type
Wire-rate or
Oversubscribed?
WS-4748-UPOE+E
48
10/100/1000
RJ45
Wire-rate
WS-4748-RJ45V+E
48
10/100/1000
RJ45
Wire-rate
12
10GBASE-X
SFP+
2.5:1 oversubscribed
WS-X4648-RJ45-E
48
10/100/1000
RJ45
2:1 oversubscribed
WS-X4640-CSFP-E
80
1000BASE-X
CSFP
4:1 oversubscribed
WS-X4624-SFP-E
24
1000BASE-X
SFP
Wire-rate
WS-X4612-SFP-E
12
1000BASE-X
SFP
Wire-rate
WS-X4712-SFP+E
E-Series
Ports
Cisco Public
102
Line Card Comparison

Type
E-Series
Classic FE
over Fibre
Classic FE
over Copper
Classic FE
PoE
BRKRST-3067
Line Card
Ports
Speed
Port Type
Wire-rate or
Oversubscribed?
WS-X4648-RJ45V-E
48
10/100/1000
RJ45
2:1 oversubscribed
WS-X4648-RJ45V+E
48
10/100/1000
RJ45
2:1 oversubscribed
WS-X4606-X2-E
10GBASE-X
X2 or SFP w/
TwinGig
2.5:1 (X2) and Wire-rate (SFP)
WS-X4248-FE-SFP
48
1000BASE-X
SFP
Wire-rate
WS-X4124-FX-MT
24
100BASE-FX
MM MT-RJ
Wire-rate
WS-X4148-FX-MT
48
100BASE-FX
MM MT-RJ
Wire-rate
WS-X4148-FE-BD-LC
48
100BASE-BX10-D
SMF Single LC
Wire-rate
WS-X4124-RJ45
24
10/100
RJ45
Wire-rate
WS-X4148-RJ
48
10/100
RJ45
Wire-rate
WS-X4148-RJ21
48
10/100
RJ45
Wire-rate
WS-X4224-RJ45V
24
10/100
RJ45
Wire-rate
WS-X4248-RJ45V
48
10/100
RJ45
Wire-rate
WS-X4248-RJ21V
48
10/100
RJ21
Wire-rate
Cisco Public
103
Line Card Comparison, Continued

Type
Classic GE
over Fibre
Classic GE
over Copper
Classic GE
over Copper
PoE
BRKRST-3067
Line Card
Ports
Speed
Port Type
Wire-rate or
Oversubscribed?
WS-X4302-GB
1000BASE-X
GBIC
Wire-rate
WS-X4306-GB
1000BASE-X
GBIC
Wire-rate
WS-X4418-GB
18
1000BASE-X
GBIC
2 ports Wire-rate
4:1 oversubscribed
WS-X4448-GB-LX
48
1000BASE-LX
SFP
8:1 oversubscribed
WS-X4448-GB-SFP
48
1000BASE-X
SFP
8:1 oversubscribed
WS-X4506-GB-T
6+6
10/100/1000 and
1000BASE-X
RJ45 with PoE and

SFP
Wire-rate
WS-X4424-GB-RJ45
24
10/100/1000
RJ-45
4:1 oversubscribed
WS-X4448-GB-RJ45
48
10/100/1000
RJ-45
8:1 oversubscribed
WS-X4548-GB-RJ45
48
10/100/1000
RJ-45
8:1 oversubscribed
WS-X4524-GB-RJ45V
24
10/100/1000
RJ-45
4:1 oversubscribed
WS-X4548-GB-RJ45V
48
10/100/1000
RJ-45
8:1 oversubscribed
Cisco Public
16 ports
104
Fixed Configuration Switches

Catalyst 4948E
Catalyst 4900M
Catalyst 4948-10GE
Catalyst 4948
Bandwidth1
176 Gbps
320 Gbps
136 Gbps
96 Gbps
Throughput
131 mpps (IPv4) and 110

mpps (IPv6)
250 mpps (IPv4) and 125

mpps (IPv6)
102 mpps
72 mpps
IPv6
In hardware
In hardware
In software
In software
Height
1 RU
2 RU
1 RU
1RU
Max 10/100/1000 ports
48
40
48
48
Max 10GE ports
24
Max GE SFP ports
32 with TwinGig
Shared packet memory
17.5MB
16MB
16MB
16MB
VLANs
4096
4096
2048
2048
Multicast entries
32,000 (IPv4) and 16,000

(IPv6)
56,000 (IPv4) and 28,000

(IPv6)
28,000 (L3) and 16,000 (L2)
28,000 (L3) and 16,000

(L2)
Security & QoS entries
32,000 (input) + 32,000

(output)
128,000
32,000
32,000
MAC addresses
55,000
55,000
55,000
32,000
1. Bi-directional.
BRKRST-3067
Cisco Public
105
Link Error Counters

Error
Description
Mitigation
CrcAlign-Err
Counts frames that do not end in an even number of

octets and have a bad CRC. Indicates a physical layer
issue.
Swap out cable with known good cable. Test with known good switch
port. Test with known good NIC card or other peer port. Check for
duplex mismatch.
Symbol-Err
Counts how many times the port receives an invalid

symbol.
port. Test with known good NIC card or other peer port.
Undersize
Counts frames that are less than 64 bytes.
Sniff peer device to determine if it is sending undersize frames.
Oversize
Counts frames that are greater than the ports

configured or default MTU.
Verify configured MTU on switch port and its peer.
Fragments
Counts frames that are less than 64 bytes with a bad

CRC.
port. Test with known good NIC card or other peer port.
Single-Col
Counts how many times a collision occurs before the

port transmitted a frame successfully.
Normal for half-duplex ports. Abnormal for full-duplex ports. Check for
duplex mismatch. Check if link is over-utilized.
Multi-Col
Counts how many times multiple collisions occur before

the port transmits a frame successfully.
Normal for half-duplex ports. Abnormal for full-duplex ports. Check for
duplex mismatch. Check if link is over-utilized.
Late-Col
Counts how many times the port detects the collision

after the time it takes to send the frame (i.e., 5.12
microseconds for a 64-byte frame on a 100 Mbps link).
Check for duplex mismatch.
BRKRST-3067
Cisco Public
106
Decoding Module Status

Show Module
Error
Description
Mitigation
Faulty
Line card failed online diagnostics or line

card bring-up sequence
May be a hardware issue. Call TAC.
Authfail
Line card authentication failed
Report issue to TAC.
Offline
Line card is not fully booted
Occurs when switch brings the line card online. This should be a transient state.
In Reset
Line card is powered down
Due to no hw-module module <module> power configuration.
PwrOver
Module is consuming more than 50 W

above administratively allocated inline
power
Determine if connected devices are receiving the right amount of inline power according to
show power inline. If correct, disconnect phones one at a time, noting the inline power
utilization change. If one or more devices trigger a change greater than what is listed in show
power inline, those devices may be faulty.
PwrMax or
PwrFault
Module is consuming more than 50 W

above the modules limit
Determine if connected devices are receiving the right amount of inline power according to
show power inline. If correct, disconnect phones one at a time, noting the inline power
utilization change. If one or more devices trigger a change greater than what is listed in show
power inline, those devices may be faulty.
PwrDeny
Insufficient power to bring module online
Configure dual power supplies in combined mode (power redundancy combined), or

install power supplies with higher capacity.
BRKRST-3067
Cisco Public
107
Appendix
Catalyst 4500
Command Summaries
HA Redundancy
ISSU
Unicast Forwarding
Catalyst 6500
BRKRST-3067
Cisco Public
108
Leveraging Dual Supervisors

RPR
SSO
Standby supervisor fully boots after

switchover
Standby supervisor boots fully
L2 and L3 tables rebuilt after switchover
L2 tables, configs synchronised
Switchover traffic loss: 1-2 min.
Switchover L2 traffic loss: 200 ms
12.2(12c)EW onward
L3 tables dynamically rebuilt
12.2(20)EWA onward
ISSU
NSF
Requires SSO
Requires SSO
L3 forwarding continues during

switchover
Enables in-service IOS upgrades
NSF-aware: 12.2(20)EWA onward

NSF-capable: 12.2(31)SG onward
BRKRST-3067
Images are NSF-capable

12.2(31)SGA onward
Cisco Public
109
Troubleshooting Supervisor Redundancy

Verify Ethernet Out-of-Band Channel (EOBC) Counters
Catalyst-4510R# show controllers
Dagobah MAC: EOBC port
MAC address: 02:00:00:00:01:00
Restarted: 1
Rx packets: 36273
Rx packets reported by hardware: 36273
Rx error count: 0
Rx engine restarted: 0
Tx packets: 36273
Tx packets reported by hardware: 36273
Currently queued Tx packets: 0
High watermark for Tx queue: 10
Tx error count: 0
Dropped Tx packets: 5
Internal error count: 0
Interrupt count: 35055
Link events: 1
Count of Phy polls: 35327
Count of postprocessing: 35042
Phy status: 100Mb Half
BRKRST-3067
Continuously incrementing?
Initiate a switchover to reset
EOBC MAC
EOBC speed and duplex for Classic

supervisors
Cisco Public
110
Troubleshooting SSO
Sanity Check
Must Match
Should Match
Supervisor type
Rommon version
Supervisor Hw revision
Memory1
Boot variables
Optical modules
IOS version2
May Match
Configuration register
value
Catalyst-4507# show module
Chassis Type : WS-C4507R

Power consumed by backplane : 40 Watts
Mod Ports Card Type
Model
Serial No.
---+-----+--------------------------------------+------------------+----------1
2
Mod
Sup V-10GE 10GE (X2), 1000BaseX (SFP)
WS-X4516-10GE
JAB09160071
WS-X4516-10GE
JAE1008W6KF
Redundancy role
Operating mode
Redundancy status
----+-------------------+-------------------+---------------------------------1
Standby Supervisor
SSO
Standby hot
Active Supervisor
SSO
Active
1. Supervisors II-Plus and II-Plus-10GE can be configured with 256 or 512 MB. Use show version to verify capacity.
2. Different IOS versions accepted (and expected!) during software upgrade or downgrade using ISSU (12.2(37)SG and beyond).
BRKRST-3067
Cisco Public
111
Troubleshooting SSO, Continued

Standby Offline
Catalyst-4507#show module
Chassis Type : WS-C4507R
Power consumed by backplane : 40 Watts
Mod Ports Card Type
Model
Serial No.
---+-----+--------------------------------------+------------------+----------1
2
WS-X4516-10GE
JAB09160071
Supervisor
M MAC addresses
Hw
Fw
Sw
Supervisor present,
Status
but in ROMMON
--+--------------------------------+---+------------+----------------+--------1 000c.8523.4940 to 000c.8523.4945 2.0 12.2(25r)EW
12.2(37)SG
Ok
2 Unknown
Unknown
Other
Mod
Redundancy role
Unknown
Operating mode
Redundancy status
----+-------------------+-------------------+---------------------------------1
Active Supervisor
SSO
Active
Standby Supervisor
SSO
Disabled
Catalyst-4507#
BRKRST-3067
Cisco Public
112

Digging Deeper
Catalyst-4507# show redundancy
Redundant System Information :
-----------------------------Available system uptime = 1 day, 23 hours, 2 minutes
Increments due to:
Switchovers system experienced = 0
1. Manual reload via CLI

2. Supervisor removal
3.
Hardware or software crash
4. Sup-to-sup keepalive timeout
If Simplex, check if standby supervisor is
present and in ROMMON
Standby failures = 1
Last switchover reason = none1
Hardware Mode = Duplex
Configured Redundancy Mode = Stateful Switchover
Operating Redundancy Mode = Stateful Switchover
Maintenance Mode = Disabled
If mismatched, super-visor
type, IOS version, or
memory do not match
If Down, check if standby supervisor is
present and in ROMMON
Communications = Up
1. Issue show redundancy switchover history for more information.
Output continued on next slide

BRKRST-3067
Cisco Public
113

Digging Deeper
Current Processor Information :
------------------------------Active Location = slot 2
Current Software state = ACTIVE
Uptime in current state = 1 day, 23 hours, 2 minutes
Image Version = Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500ENTSERVICES-M), Version 12.2(40)SG, RELEASE SOFTWARE (fc2)
DISABLED: check if standby

supervisor is present, in ROMMON
---------------------------STANDBY COLD: check
for supervisor type, IOS
Standby Location = slot 1
version, and memory
Current Software state = STANDBY HOT
mismatch
Peer Processor Information :
Match!
Uptime in current state = 1 hour, 9 minutes

Image Version = Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500ENTSERVICES-M), Version 12.2(40)SG, RELEASE SOFTWARE (fc2)
BRKRST-3067
Cisco Public
114

Logging into Standby
Verify running-config is synchronised to standby supervisor

Verify hardware tables are synchronised
Verify software tables are synchronised
Check standby supervisors local log
Catalyst-4507# show redundancy | include Standby Location
Standby Location = slot 1
Catalyst-4507# session module 1
Connecting to standby virtual console
Type "exit" or "quit" to end this session
Catalyst-4507-standby-console# ^e
Standby console enabled.
BRKRST-3067
Cisco Public
115
SSO System Messages

Over 30 system messages to track redundancy states, behaviour,
and alerts
Example: losing standby connectivity and IOS version mismatch
1 %C4K_REDUNDANCY-3-COMMUNICATION: Communication with the peer Supervisor
The peer Supervisor has been lost
2 %C4K_REDUNDANCY-3-SIMPLEX_MODE:
%C4K_REDUNDANCY-6-DUPLEX_MODE: The peer Supervisor has been detected
3 %C4K_IOSMODPORTMAN-6-MODULEONLINE:
has been lost
Module 2 (WS-X4516-10GE S/N: JAE1008W6KF Hw: 3.0) is online
%C4K_REDUNDANCY-2-IOS_VERSION_CHECK_FAIL: IOS version mismatch. Active supervisor version is 12.2(37)SG. Standby supervisor version is
12.2(31)SG. Redundancy feature may not work as expected.
4 %C4K_REDUNDANCY-6-MODE:
ACTIVE supervisor initializing for rpr mode
%C4K_REDUNDANCY-3-COMMUNICATION: Communication with the peer Supervisor has been established
5
6
1
Standby supervisor was removed, reloaded, or

crashed
One operational supervisor in the switch
Active and standby supervisor IOS version do not match. NOTE:

version check will pass for IOS images that support ISSU.
Fall back to RPR mode even though SSO is configured
Standby supervisor detected, online
Ready to automatically synchronise VLAN database, calendar,

configuration register, and boot variables.
BRKRST-3067
Cisco Public
116
Non-Stop Forwarding Overview

Continuous L3 Forwarding During Switchovers
NSF-capable router
NSF-capable router
NSF-capable router
RP Switchover!
Route
updates
Route
updates
Route
updates
Supervisor
Switchover!
Catalyst 4510R
NSF-aware switch
L3 traffic flowing
Catalyst 4510R
NSF-aware switch
L3 traffic flowing
Catalyst 4510R
NSF-capable switch
L3 traffic flowing
Route processor (RP) switchover
Supervisor switchover
Supervisor switchover
Switch retains adjacency table
Router clears adjacency table
Router retains adjacency table
Switch forwards routing updates

NSF-awareness introduced in 12.2(20)EWA
No route flaps! L3 traffic flowing!
BRKRST-3067
Switch waits for routing updates
Route flap! L3 traffic stops flowing!
Cisco Public
Supervisor
Switchover!
Router forwards
routing in
updates
NSF-capability
introduced
12.2(31)SG
for
classic
and 12.2(44)SG
for
No
routesupervisors
flaps! L3 traffic
flowing!
Supervisor 6-E.
117
Troubleshooting NSF
Sanity Check
Is SSO configured and operational?

show redundancy states
Is NSF configured?
show running-config | begin router
Using NSF-aware software and hardware?

12.2(20)EWA through 12.2(25)SG (always on by default)
IP base or enterprise services image

Supervisor Engines II-Plus, II-Plus+TS, II-Plus+10GE, IV, V, V-10GE
Catalyst 4948 and Catalyst 4948-10GE
Using NSF-capable software and hardware?

12.2(31)SG and beyond
Enterprise services IOS image
Supervisor Engines V, V-10GE, Catalyst 4948-10GE
BRKRST-3067
Cisco Public
118
Troubleshooting NSF
Digging Deeper: OSPF
Switch-1# show running-config | begin router
router ospf 1
log-adjacency-changes
nsf
network 10.0.0.0 0.255.255.255 area 0
network 20.0.0.0 0.255.255.255 area 0
Switch-1
Switch-2
Switch-1# show ip ospf 1 | begin Non-Stop

Non-Stop Forwarding enabled, last NSF restart 00:01:24 ago (took 35 secs)
IETF NSF helper support enabled
Cisco NSF helper support enabled
Switch-1# show cef state capabilities | include NSF
CEF NSF capable:
yes
Switch-1# show cef state | include NSF

CEF NSF sync:
BRKRST-3067
enabled/running
Cisco Public
119
Appendix
Catalyst 4500
Command Summaries
HA Redundancy
ISSU
Unicast Forwarding
Catalyst 6500
BRKRST-3067
Cisco Public
120
ISSU Overview
Performing In-Service Software Upgrades
Enables software upgrades without down-time

Built on top of SSO and NSF
Requires images to be compatible
show issu comp-matrix stored
1
ACTIVE
OLD
STANDBY
OLD
BRKRST-3067
load
version
ACTIVE
OLD
run
version
STANDBY
NEW
Cisco Public
STANDBY
OLD
commit
version
STANDBY
NEW
ACTIVE
ACTIVE
NEW
NEW
121
Troubleshooting ISSU
Sanity Check
Is SSO configured and operational?

show redundancy states
Both images support ISSU?

12.2(31)SGA onward
Does the hardware match?

show module
Is autoboot configured?
show bootvar
Configuration register ends in 2
Are both images saved on the supervisors?

BRKRST-3067
Cisco Public
122
Interpreting Error Messages
issu loadversion
%% Loadversion CLI can be run only from

Init State
%% Active config-register doesn't have

0x2 as the low order nibble
%% Active and Standby image names
should be same
issu runversion
%% Runversion CLI can be run only from

Loadversion or RunVersion-SwitchOver
State
%% Standby unit ID [ 1 ] is wrong;
expected [ 2 ]
%% User is specifying the old image

for Active1
1 unit ID [ 2 ] is wrong;
%%
Activecommitversion
issu
expected [ 1 ]
show issu state
issu acceptversion
%% Acceptversion CLI can be run only from

RunVersion or LoadVersion-SwitchOver
State
%% Acceptversion CLI can be run only

from RunVersion or LoadVersionSwitchOver State
%% Active unit ID [ 1 ] is wrong;

expected [ 2 ]
%% Active unit ID [ 1 ] is wrong;

expected [ 2 ]
1. Similar messages available for standby supervisor.

BRKRST-3067
Cisco Public
123
Detailed ISSU State Transitions
BRKRST-3067
Cisco Public
124
Debugging Supervisor Sync Failures
Occurs when command is not supported by standbys IOS version
Results in RPR mode
Unconfigure mismatched commands to synchronise supervisors and
return to SSO mode
Catalyst-4507# issu loadversion 1 bootflash:cat4500-ipbase-mz.122-31.SGA3 2
slavebootflash:cat4500-ipbase-mz.122-31.SGA3
<<< Wait for standby to fully boot >>>

<<< Wait for startup-config to synchronize to standby >>>
Mar
5 23:29:33.127: Config Sync: Bulk-sync failure due to Servicing Incompatibility. Please check full list of mismatched commands via:
show issu config-sync failures mcl

Mar
5 23:29:33.127: Config Sync: Starting lines from MCL file:
interface GigabitEthernet7/1
! <submode> "interface"
- dot1x host-mode multi-domain
! </submode> "interface
BRKRST-3067
Cisco Public
125
Appendix
Catalyst 4500
Command Summaries
HA Redundancy
ISSU
Unicast Forwarding
Catalyst 6500
BRKRST-3067
Cisco Public
126
Troubleshooting Unicast Forwarding

Verify Layer 3 Information
172.32.1.1
L2 Commands for Access Layer Switch
show
show
show
show
show
interface <interface> counters

spanning-tree vlan
platform hardware stp vlan
mac address-table interface
platform hardware mac-address-table address
Si
L2 Commands for Distribution Layer Switch
Si
show spanning-tree vlan <vlan> interface

show platform hardware stp vlan <vlan> interface
Si
L3 Commands for Distribution and Core Layer Switches
show
show
show
show
show
show
interface
ip route
arp
ip cef
adjacency
platform hardware ip route [ipv4] network/host
Si
Si
Access
L2
Distribution
L2/L3
Core
L3
Distribution
L2/L3
Access
L2
172.16.1.1
1. Cisco Catalyst 4900M and 4948E also uses TCAM4, so you can use the same troubleshooting commands.
BRKRST-3067
Cisco Public
127
Appendix
Catalyst 4500
Command Summaries
HA Redundancy
ISSU
Unicast Forwarding
Catalyst 6500
BRKRST-3067
Cisco Public
128
Generic Online Diagnostics

What Is Gold?
Gold defines a common framework for
diagnostics operations across Cisco
platforms running Cisco IOS Software
Goal: check the health of hardware components
and verify proper operation of the system data
plane and control plane at run-time and boot-time
Si
Provides a common CLI and scheduling for field

diagnostics including:
Bootup Tests (includes online insertion)
Health Monitoring Tests (background non-disruptive)
On-Demand Tests (disruptive and Non-disruptive)
User Scheduled Tests (disruptive and Non-disruptive)
CLI access to data via Management Interface
BRKRST-3067
Cisco Public
129

How Does Gold Work?
Diagnostic packet switching
tests verify that the system
is operating correctly:
Forwarding
Engine
Is the supervisor control plane and

forwarding plane functioning properly?
Line
Card
Is the standby supervisor ready to

take over?
Fabric
Forwarding
Engine
Are line cards forwarding packets

properly?
Active Supervisor
Are all ports working?
Standby Supervisor
CPU
Is the backplane connection working?

Other types of diagnostics tests
including memory and error
correlation tests are also available
BRKRST-3067
Line
Card
Cisco Public
130

What Type of Failure Does Gold Detect?
Diagnostics capabilities
built in hardware
Depending on hardware,
Gold can catch:
Port failure
Bent backplane connector
Bad fabric connection
Malfunctioning forwarding engines
Stuck control plane
Bad memory
BRKRST-3067
Cisco Public
131

Diagnostic Operation
Run During System Bootup, Line Card
OIR Or Supervisor Switchover
Makes Sure Faulty Hardware Is Taken
out of Service
Boot-Up Diagnostics
Switch(config)# diagnostic bootup level complete
Runtime Diagnostics
Health-Monitoring
Switch(config)# diagnostic monitor module 5 test 2

Switch(config)# diagnostic monitor interval module 5 test 2 00:00:15
On-Demand
Switch# diagnostic start module 4 test 8

Module 4: Running test(s) 8 may disrupt normal system
operation
Do you want to continue? [no]: y
Switch# diagnostic stop module 4
Scheduled
Switch(config)# diagnostic schedule module 4 test 1

port 3 on Jan 3 2005 23:32
Switch(config)# diagnostic schedule module 4 test 2
daily 14:45
BRKRST-3067
Cisco Public
Non-Disruptive Tests Run in

the Background
Serves As HA Trigger
All diagnostics tests can be run on
demand, for troubleshooting
purposes. It can also be used as a
pre-deployment tool.
Schedule Diagnostics Tests, for
Verification and Troubleshooting
Purposes
132

Using Diagnostics as a Pre-Deployment Tool
The Order in Which Tests Are Run Matters
Run diagnostics first on line cards, then on supervisors
Run packet switching tests first, run memory tests after
Switch# diagnostic start module 6 test all
Module 6: Running test(s) 8 will require resetting the line card after the test has completed
Module 6: Running test(s) 1-2,5-9 may disrupt normal system operation
Do you want to continue? [no]: yes
*Mar 25 22:43:16: %DIAG-SP-6-TEST_RUNNING: Module 6: Running TestTransceiverIntegrity{ID=1} ...
*Mar 25 22:43:16: %DIAG-SP-3-TEST_SKIPPED: Module 6: TestTransceiverIntegrity{ID=1} is skipped
*Mar 25 22:43:16: %LINK-5-CHANGED: Interface GigabitEthernet6/1, changed state to administratively down
*Mar 25 22:43:16: %DIAG-SP-6-TEST_RUNNING: Module 6: Running TestLoopback{ID=2} ...
*Mar 25 22:43:16: %DIAG-SP-6-TEST_RUNNING: Module 6: Running TestAsicMemory{ID=8} ...
*Mar 25 22:43:16: SP: ******************************************************************
*Mar 25 22:43:16: SP: * WARNING:
*Mar 25 22:43:16: SP: * ASIC Memory test on module 6 may take up to 2hr 30min.
*Mar 25 22:43:16: SP: * During this time, please DO NOT perform any packet switching.
*Mar 25 22:43:16: SP: ******************************************************************
<snip>
Switch# diagnostic start module 5 test all
Module 5: Running test(s) 27-30 will power-down line cards and standby supervisor should be power-down manually and supervisor
should be reset after the test
Module 5: Running test(s) 26 will shut down the ports of all linecards and supervisor should be reset after the test
Module 5: Running test(s) 3,5,8-10,19,22-23,26-31 may disrupt normal system operation
Do you want to continue? [no]: yes
<snip>
BRKRST-3067
Cisco Public
133

Catalyst Gold Operation Example
Switch# show diagnostic content mod 5
Module 5: Supervisor Engine 720 (Active)
<snip>
Testing Interval
ID
Test Name
Attributes
==== ================================== ============

1) TestScratchRegister -------------> ***N****A***
(day hh:mm:ss.ms)
=================
000 00:00:30.00
2) TestSPRPInbandPing --------------> ***N****A***
000 00:00:15.00
3) TestTransceiverIntegrity --------> **PD****I***
not configured
4) TestActiveToStandbyLoopback -----> M*PDS***I***
not configured
5) TestLoopback --------------------> M*PD****I***
not configured
6) TestNewIndexLearn ---------------> M**N****I***
not configured
7) TestDontConditionalLearn --------> M**N****I***
not configured
8) TestBadBpduTrap -----------------> M**D****I***
not configured
9) TestMatchCapture ----------------> M**D****I***
not configured
10) TestProtocolMatchChannel --------> M**D****I***
not configured
11) TestFibDevices ------------------> M**N****I***
not configured
12) TestIPv4FibShortcut -------------> M**N****I***
not configured
13) TestL3Capture2 ------------------> M**N****I***
not configured
14) TestIPv6FibShortcut -------------> M**N****I***
not configured
15) TestMPLSFibShortcut -------------> M**N****I***
not configured
16) TestNATFibShortcut --------------> M**N****I***
not configured
17) TestAclPermit -------------------> M**N****I***
not configured
18) TestAclDeny ---------------------> M**N****A***
000 00:00:05.00
19) TestQoSTcam ---------------------> M**D****I***
not configured
BRKRST-3067
<snip>
Cisco Public
134

Catalyst Gold Operation Example (Cont.)
20) TestL3VlanMet -------------------> M**N****I***
not configured
n/a
21) TestIngressSpan -----------------> M**N****I***
not configured
n/a
22) TestEgressSpan ------------------> M**D****I***
not configured
n/a
23) TestNetflowInlineRewrite --------> C*PD****I***
not configured
n/a
24) TestFabricSnakeForward ----------> M**N****I***
not configured
n/a
25) TestFabricSnakeBackward ---------> M**N****I***
not configured
n/a
26) TestTrafficStress ---------------> ***D****I**T
not configured
n/a
27) TestFibTcamSSRAM ----------------> ***D*X**IR**
not configured
n/a
28) TestAsicMemory ------------------> ***D*X**IR**
not configured
n/a
29) TestNetflowTcam -----------------> ***D*X**IR**
not configured
n/a
30) ScheduleSwitchover --------------> ***D****I***
not configured
n/a
31) TestFirmwareDiagStatus ----------> M**N****I***
not configured
n/a
32) TestAsicSync --------------------> ***N****A***
000 00:00:15.00 10
Pay extra attention to Memory tests:

Memory tests can take hours to
complete and a reset is required after
running these tests!
BRKRST-3067
Cisco Public
135

Switch# show diagnostic result mod 7
Current bootup diagnostic level: complete
Module 7: CEF720 24 port 1000mb SFP
Overall Diagnostic Result for Module 7 : MINOR ERROR
Diagnostic level at card bootup: complete
Test results: (. = Pass, F = Fail, U = Untested)
1) TestTransceiverIntegrity:
Port
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
----------------------------------------------------------------------------
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
2) TestLoopback:
Port
---------------------------------------------------------------------------.
BRKRST-3067
Cisco Public
136

r1# show diagnostic description module 5 test ?
<1-33> Test ID Number
ID
Test Name [On-Demand Test Attributes]
--- ------------------------------------------1 TestScratchRegister
[***N****]
2 TestSPRPInbandPing
[***N****]
3 TestTransceiverIntegrity
[**PD****]
4 TestActiveToStandbyLoopback
[M*PDS***]
5 TestLoopback
[M*PD****]
6 TestNewIndexLearn
[M**N****]
<snip>
r1# show diagnostic description module 5 test 2
TestSPRPInbandPing :
By default, this test is enabled as health-monitoring test.
The SP-RP Inband test catches most of the runtime software driver
and hardware issues on supervisors. This is done by using diagnostic
packet tests exercising the layer 2 forwarding engine, the L3-4
forwarding engine, and the replication engine along the path from
the Switch Processor to the Route Processor.
Packets are sent at an interval of 15 seconds and 10 consecutive
failures of the SP-RP Inband test result in failover to the
redundant supervisor (default).
BRKRST-3067
Cisco Public
137

Recommendations
Bootup diagnostics:
Set level to complete
On demand diagnostics:
Use as a pre-deployment tool: run complete diagnostics
before putting hardware into production environment
Use as a troubleshooting tool when suspecting
hardware failure
Si
Scheduled diagnostics:
Schedule key diagnostics tests periodically
Schedule all non-disruptive tests periodically
Health-monitoring diagnostics:
Key tests running by default
Enable additional non-disruptive tests for specific functionalities
enabled in your network: IPv6, MPLS, NAT
BRKRST-3067
Cisco Public
138
BRKRST-3067
Cisco Public
139

BRKRST 3067

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

BRKRST 3067

Caricato da

Copyright:

Formati disponibili

Troubleshooting Catalyst 4K and 6K

Catalyst 4500-E Series Switches

2012 Cisco and/or its affiliates. All rights reserved.

Supervisor Engine 7-E, 6-E, 6L-E, V-10GE, V, IV,

TCAMs for ACLs, QoS, L3 forwarding

NetFlow (NFL) for statistics gathering

Transparent Line Cards

2012 Cisco and/or its affiliates. All rights reserved.

Front Panel Ports

Supervisor and Blocking Line Card Block

Next Generation Supervisors

2012 Cisco and/or its affiliates. All rights reserved.

Interface Packet Drops

Catalyst 4500-E Series Switches

2012 Cisco and/or its affiliates. All rights reserved.

Catalyst 4500 Series Switches

Troubleshooting Interface Drops

Packet drops in transmit direction

Packet drops in receive direction

Front Panel Ports

Minimal buffer on receiving queue

2012 Cisco and/or its affiliates. All rights reserved.

Troubleshooting Interface Drops

Default queue 8 when QoS is disabled

Consider implementing a QoS policy to share traffic amongst queues

2012 Cisco and/or its affiliates. All rights reserved.

Troubleshooting Interface Drops

SUP6-E#show platform software interface g2/47 all | inc Busy

%C4K_HWPORTMAN-4-BLOCKEDTXQUEUE: Blocked transmit queue HwTxQId7 on Switch

Error log displayed when BusyBit is set

2012 Cisco and/or its affiliates. All rights reserved.

Troubleshooting Interface Drops

2012 Cisco and/or its affiliates. All rights reserved.

Troubleshooting Interface Drops

Sup6-E# show interface gi1/13 counter all | begin Rx-No

Sup6-E# show platform software interface g1/13 stub stat | in Overrun

Enable flow control receive on the peer

2012 Cisco and/or its affiliates. All rights reserved.

Interface Packet Drops

Catalyst 4500-E Series Switches

2012 Cisco and/or its affiliates. All rights reserved.

Catalyst 4500 Series Switches

Troubleshooting High CPU

STP, CDP, PAgP, VTP, DTP, routing

Program dynamic entries into hardware

Manage access to the switch

Manage system components

2012 Cisco and/or its affiliates. All rights reserved.

Troubleshooting High CPU

Packets copied to CPU but originally switched in

Packets punted to CPU for processing

Packets sent to the CPU for forwarding

2012 Cisco and/or its affiliates. All rights reserved.

Troubleshooting High CPU

SUP7-E# show processes cpu sorted detail

SUP6-E#sh proc cpu sorted

2012 Cisco and/or its affiliates. All rights reserved.

Troubleshooting High CPU

LoPri: exceeds CPU target

SUP6-E#show platform health

2012 Cisco and/or its affiliates. All rights reserved.

Troubleshooting High CPU

2012 Cisco and/or its affiliates. All rights reserved.