IOS XR Operational Tools

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

Presentation Goals
Introduce debugging tools available in IOS-XR
Provide a chance to practice using the tools
Short lab time after each set of commands

XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

Agenda
Processes
Memory
System Monitoring

XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

Process Management Terms

Process
An executable portion of code run within its own memory space

Thread
Sub-process managed by the parent process
Responsible for executing a portion of the overall process
A process may contain one or more threads
Only runs when the parent process is allocated runtime
Example - OSPF thread which handles hello receipt and
transmission, one per neighbor

Dynamically Linked Library

Reusable section of code that can be loaded at runtime
XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

Process Identification
Job ID (JID)
Assigned when process first run
Globally unique identifies the process
No change if process restarts

Process ID (PID)
Assigned when instance of process is created
Globally unique identifies the process
New PID assigned if process restarts

Thread ID (TID)
TID numbering restarts at 1 for each process

Name
Some commands allow process identification by name
XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

show process command options

RP/0/RP0/CPU0:CRS#show processes ?
Reference process by job id
<0-4294967295> job id
WORD
Name of the executable
aborts
Show process aborts
all
Show process data for all processes
blocked
Show detail for reply/send/mutex blocked processes.
boot
Show process boot info
Display process on other nodes
cpu
Show CPU use per process
distribution
Show distribution of processes
dynamic
Show process data for dynamically created processes
failover
Show process failover info
family
Show process family information.
files
Show file and channel use per process
location
location to display
log
Show process log
mandatory
Show process data for mandatory processes
memory
Show memory use per process
pidin
Show processes using QNX pidin command
searchpath
Show the search path
signal
Show signal use for processes.
startup
Show process data for processes created at startup
threadname
Show thread names.

XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

show process command output

RP/0/RP0/CPU0:CRS#show process ospf
JID and PID
Job Id: 262
PID: 209102
Executable path: /disk0/hfr-rout-3.3.1/bin/ospf
Instance #: 1
Automatic restart?
Version ID: 00.00.0000
Respawn: ON
Respawn count: 1
Max. spawns per minute: 12
Last started: Thu Jul 20 15:39:20 2006
Process state: Run
Package state: Normal
Started on config: cfg/gl/ipv4-ospf/proc/1/ord_z/config
core: TEXT SHAREDMEM MAINMEM
Max. core: 0
Placement: ON
startup_path: /pkg/startup/ospf.startup
Thread IDs
Ready: 13.338s
Available: 17.353s
SNMP Process
Process cpu time: 2.702 user, 0.188 kernel, 2.890 total
contains 8 threads
JID
TID Stack pri state
HR:MM:SS:MSEC NAME
which operate
262
1
60K 10 Receive
0:00:02:0672 ospf
under JID 288
262
2
60K 10 Receive
0:00:00:0012 ospf
262
3
60K 10 Receive
0:00:00:0012 ospf
XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

Monitoring Top Processes

Note: Remote execution & auto updating display

RP/0/RP0/CPU0:CRS#monitor processes location 0/2/CPU0
125 processes; 493 threads; 2201 channels, 3365 fds
CPU states: 93.1% idle, 5.8% user, 1.0% kernel
Memory: 2048M total, 1607M avail, page size 4K
JID TIDS Chans
57
11
19
70
12
52
1
20 129
200
6
16
125
6
34
135
3
7
172
4
25
171
16
66
184
2
12
185
5
30
137
5
22
248
9
32
101
5
22
205
3
5
195
1
8
178
5
19
74
10 114
224
2
6
XR Training / lwigley

FDs Tmrs
13
6
9
3
94
1
17
4
30
6
8
2
27
7
40
14
49
3
28
5
53
9
16
13
57
8
10
1
34
3
36
5
22
4
7
2

2005 Cisco Systems, Inc. All rights reserved.

MEM
12M
624K
0
564K
976K
124K
276K
684K
872K
544K
524K
2M
552K
152K
204K
440K
568K
236K

HH:MM:SS
0:18:53
0:01:47
502:07:10
0:14:43
0:25:48
0:03:00
0:02:07
0:00:16
0:00:08
0:00:08
0:00:21
0:23:24
0:00:01
0:00:10
0:00:00
0:00:00
0:00:00
0:00:00

CPU
2.95%
1.74%
1.02%
0.85%
0.13%
0.03%
0.03%
0.03%
0.03%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%

NAME
eth_server
qnet
procnto-600-smp
qsm
egressq
hbagent
mstats_svr
pse_driver
pifibm_server
pla_server
hfr_pm
wdsysmon
arp
SNMP Process
reddrv_listener
contains 8 threads
ppp_socket
which operate
nfsvr under JID 288
sysmgr
sysmgr
Cisco Public

Stopping, Restarting, and Crashing Processes

Note: Lab use only

process shutdown shut down and do not restart

Graceful termination

process start start a shut down process

process restart restart a process
Works if process is running or shutdown
Graceful termination

process crash simulate a crash

Process will restart automatically if Respawn=ON
Non-graceful termination

XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

Process Restart Example

RP/0/RP0/CPU0:CRS#show proc qnet
Initial respawn count shows
Job Id: 74
process hasnt restarted
PID: 32795
Executable path: /hfr-os-3.3.1/sbin/qnet
Instance #: 1
Args: transport=enet,conn_est_retries=3
Version ID: 00.00.0000
Restart a few times
Respawn: ON
Respawn count: 1

PID changes, JID stays same

RP/0/RP0/CPU0:CRS#process restart 74
RP/0/RP0/CPU0:CRS#process restart 74
RP/0/RP0/CPU0:CRS#show proc 74
Job Id: 74
Respawn count increases
PID: 7061531
Executable path: /hfr-os-3.3.1/sbin/qnet
Instance #: 1
Args: transport=enet,conn_est_retries=3
Version ID: 00.00.0000
Reason for restart
Respawn: ON
Respawn count: 3
Max. spawns per minute: 12
Last started: Thu Aug 31 07:13:37 2006
Process state: Run (last exit due to SIGTERM)
XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

10

Process Shutdown
RP/0/RP0/CPU0:CRS#process shut ospf
RP/0/RP0/CPU0:CRS#show process ospf
Job Id: 262
PID: 6983888
Executable path: /disk0/hfr-rout-3.3.1/bin/ospf
Instance #: 1
Process not running
Version ID: 00.00.0000
Respawn: ON
Respawn count: 4
Max. spawns per minute: 12
Last started: Thu Aug 31 07:02:32 2006
Process state: Killed (last exit status : 1)
Package state: Normal
Registered item(s): cfg/gl/ipv4-ospf/proc/
core: TEXT SHAREDMEM MAINMEM
Max. core: 0
Placement: ON
startup_path: /pkg/startup/ospf.startup
Ready: 6.368s
Available: 10.385s
RP/0/RP0/CPU0:CRS#process start ospf
XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

11

XR Processes Mini-lab

Try the process commands on your lab router

show process
process shutdown shut down and do not restart
Graceful termination

process start start a shut down process

process restart restart a process
Works if process is running or shutdown
Graceful termination

process crash simulate a crash

Process will restart automatically if Respawn=ON
Non-graceful termination

monitor process real time list

Sorted by memory or CPU usage
? to adjust parameters
XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

12

System Monitoring and Online Help

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

13

System Monitoring
Monitoring
show tech
show system verify
monitor interface
monitor controller

Online Help
describe
man

XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

14

show tech variations

RP/0/RP0/CPU0:CRS#show tech ?
bcdl
Output show commands of interest for bcdl debugging
cef
Output show commands of interest for CEF debugging
file
Specify a valid file name (e.g. disk0:tmp.log)
gsp
Output show commands of interest for gsp debugging
lrd
Output show commands of interest for lrd debugging
ospfv3
Output show commands for OSPFv3 debugging
password
Include password in output
pfi
Output show commands of interest for PFI debugging
placement Gather lots of information about process placement
platform
show tech-support platform output
rib
Show IP RIB related information
routing
Routing show tech-support output
terminal
Send output to terminal

XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

15

Monitor One Interface

monitor interface command

RP/0/RP1/CPU0:CRS1# monitor interface mgmtEth 0/rp1/CPU0/0
CRS1
Monitor Time: 00:00:52
SysUptime: 16:33:16
MgmtEth0/RP1/CPU0/0 is up, line protocol is up Encapsulation ARPA
Traffic Stats:(5 minute rates)
Input Packets:
Input pps:
Input Bytes:
Input Kbps:
Output Packets:
Output pps:
Output Bytes:
Output Kbps:
Errors Stats:
Input Total:
Input CRC:
Input Frame:
Input Overrun:
Output Total:
Output Underrun:

313326
5
34467898
3
37633
0
2034463
0

Delta
17
1033
4
244

1
0
0
0
0
0

0
0
0
0
0
0

Quit='q', Freeze='f', Thaw='t', Clear='c', Interface='i',

Detail='d', Brief='b', Next='n', Prev='p'
XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

16

Monitor All Interfaces

monitor interface all command

RP/0/RP1/CPU0:CRS1# monitor interface all
CRS1
Interface

Monitor Time: 00:00:55

Encap

SysUptime: 16:35:50

MgmtEth0/RP0/CPU0/0 ARPA

Input
pps
4

Output
pps
0

Input
Kbps
2

Output
Kbps
0

MgmtEth0/RP1/CPU0/0 ARPA

POS0/4/0/0

HDLC

1230

542210

POS0/4/0/3

HDLC

POS0/4/0/6

PPP

POS0/4/0/7

HDLC

1230

542210

Quit='q', Freeze='f', Thaw='t', Clear='c', Next set='n', Prev set='p'

XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

17

Online Help

Requires -doc package

RP/0/RP1/CPU0:CRS#man keyword mpls
Following Commands Matched for 'mpls'
FEATURE
COMMAND
is-is
mpls traffic-eng
is-is
mpls traffic-eng router-id (is-is)
is-is
show isis mpls traffic-eng adjacency-log
is-is
show isis mpls traffic-eng advertisements
is-is
show isis mpls traffic-eng tunnel
ospf
mpls traffic-eng area (ospf)
ospf
mpls traffic-eng router-id (ospf)
ospf
show ospf mpls traffic-eng
quality-of-service match mpls experimental topmost
quality-of-service set mpls experimental topmost
RP/0/RP1/CPU0:CRS#man command show cef ipv4
COMMAND
show cef ipv4
DESCRIPTION
To display the IPv4 Cisco Express Forwarding (CEF) table, use the show
cef ipv4 command in EXEC mode.
show cef ipv4 [prefix [mask] | type instance] [detail] [location node-id]
XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

18

Details about commands (permissions)

describe command

RP/0/5/CPU0:GSR1# describe show install active

Package:
c12k-base
c12k-base V3.5.0[00] Base Package for 124xx
Vendor : Cisco Systems
Desc
: Base Package for 124xx
Build : Built on Tue Jun 12 20:36:49 UTC 2007
Source : By zamboni in
/auto/srcarchive2/production/3.5.0/c12k/workspace for c2.95.3-p8
Card(s): RP, DRP, DRPSC, OC3-POS-4, OC12-POS, GE-3, OC12-POS-4,
OC48-POS, E3-OC48-POS, E3-OC12-POS-4, E3-OC3-POS-16, E3-OC3-POS-8, E3OC3-POS-4, E3-OC48-CH, E3-OC12-CH-4, E3-OC3-CH-16, E3-GE-4, E3-OC3-ATM-4,
E3-OC12-ATM-4, E5-CEC, E5-CEC-v2, SE-SEC
Restart information:
Required permissions
Default:
parallel impacted processes restart
Component:
installmgr V[main/472]

On the box installation program

User needs ALL of the following taskids:

XR Training / lwigley

pkg-mgmt 2005(READ)
Cisco Systems, Inc. All rights reserved.

Cisco Public

19

XR Monitoring and Help Mini-lab

Try the following commands on your lab router

Monitoring
show tech
show system verify
monitor interface
monitor controller

Online Help
describe
man

XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

20

Verifying the System

SUMMARY STEPS
1. admin
2. show platform [node-id]

9. show context

3. show version
4. show running-config
5. show logging

11. show memory heap summary {job-id | all}

6. show environment
7. show context
8. exit

10. show memory summary detail location all

12. top processes
13. show running-config
14. show system verify start

show system verify report

15. show {ipv4 | ipv6} interface brief

16. Show install active

XR Training / lwigley

2005 Cisco Systems, Inc. All rights reserved.

Cisco Public

21

Presentation_ID
XR Training
/ lwigley

Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved.
2004,
2005 Cisco

Cisco Public

22

22