Troubleshooting

March 2003

Troubleshooting 1

Objectives
By the end of this module you will be able to:
Locate specific information using the tools available
Isolate problems to and within NetCache
Apply diagnostic procedures to correct NetCache
problems
Use specific commands to analyze NetCache
performance
Use floppy boot options

March 2003

Troubleshooting 2

Troubleshooting

Network Problem Isolation

NetCache problem isolation
Configuration errors
Network problems
URL access problems
Recovering lost passwords

March 2003

Troubleshooting 3

Network Problem Isolation

Remove
browser proxy
configs

Enter a URL
in browser

Page
returned?

No

Problem with:
browser
network
ISP
host

Yes
Problem with
browser or
NetCache

March 2003

Troubleshooting 4

Network Problems

ifconfig -a
netstat -i
netstat -r
netdiag -nv
netdiag -v
traceroute

verify configuration
look for excessive errors
examine routing tables
quick check of OSI layers 1, 3, and 4
displays test as they are performed
verifies network path

March 2003

Troubleshooting 5

ifconfig Command
netcache> ifconfig -a
e0: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.41.66.140 netmask 0xffffff00 broadcast 10.41.66.255
ether 00:a0:98:00:97:ed (100tx-fd-up)
lo: flags=948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188
inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1

netcache> ifconfig e0
e0: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.41.66.140 netmask 0xffffff00 broadcast 10.41.66.255
ether 00:a0:98:00:97:ed (100tx-fd-up)

ifconfig

Changing address or mask info

Set media type
Set MTU
Enable (disable) interfaces
Display configurations

March 2003

Troubleshooting 6

netstat -i

netcache> netstat -i
Name
Mtu Network
e0
1500 10.41.66/24
lo
9188 127
netcache>

Address
edu2-C760
localhost

Ipkts
752k
0

Ierrs
0
0

March 2003

Opkts
3k
0

Oerrs
0
0

Collis
0
0

Queue
0
0

Troubleshooting 7

netstat -r
netcache > netstat -r
Routing tables
Internet:
Destination
Interface

Gateway

Flags

Refs

Use

default

10.41.66.1

UGS

238

e0

10.41.66/24

link#1

UC

e0

10.41.66.1

link#1

UHL

e0

localhost

localhost

UH

lo

edu2-C760>

March 2003

Troubleshooting 8

netdiag command
Analyzes the statistics continuously gathered by the network protocol
code and performs various tests of the protocol code
Options
-s
-v
-d
-b
-e
-p
-I interface
-n
-t
-a

Summary of the results

Verbose output
Debug output
Non-interactive usage
Print error codes
Tests the interface layer (i.e. NIC device drivers).
Tests only the specified interface - use with the -p option
Tests for the network layer (i.e. IP).
Tests for the transport layer (i.e. TCP and UDP).
Tests overall network subsystem.

netdiag
perform network diagnostics
SYNOPSIS
netdiag [ -s|v|d ] [ -nbate ] [ -p [ -I interface ] ]
netdiag -?
Description
The netdiag command analyzes the statistics continuously gathered by the network
protocol code and (if required) performs various tests to ensure the sanity of operation of
the protocol code. It displays the results of the analysis (and any tests performed) along
with suggested remedial actions (if any problems are found). The netdiag command
analyses almost all of the statistics displayed by the many forms of the netstat command
for aberrant values.
The first form presents allows the user to specify what subset(s) of the networking
subsystem to address and what kind of output to produce. The various options that
influence this command form are described in detail below.
The second form displays the usage message.
Options
-s

Print only a summary of the results of the various checks and tests

March 2003

Troubleshooting 9

performed.
-v

Verbose output. Print a description of the various checks and subtests as

they are performed.

-d

Debug output. Print a very detailed description of the various check and
subtests as they are performed. This option is for use by the expert user, or
someone who understands the various checks and tests that netdiag
performs.

-b

Non-interactive usage. The command should not prompt for user input.
This is useful when using netdiag from an automated script.

-e

Print error codes. Print a numeric error code with each message. This is
useful when using netdiag in consultation with a NetApp personnel.

-p

Perform diagnostic checks and tests for the interface layer (i.e. the various
NIC device drivers).

-I interface

Perform diagnostic checks and tests only for the specified interface. The
interface names that netdiag understands are the same as displayed by the
ifconfig command. This option is meaningful only in conjunction with the
-p option.

-n

Perform diagnostic checks and tests for the network layer (i.e. IP).

-t

Perform diagnostic checks and tests for the transport layer (i.e. TCP and
UDP).

-a

Perform diagnostic checks and tests to verify the sanity of the overall
network subsystem.

March 2003

Troubleshooting 10

netdiag Command
netcache >
Performing
Performing
Performing

netdiag
physical layer diagnostics.....OK
network layer diagnostics.....OK
transport layer diagnostics.....OK

March 2003

Troubleshooting 11

netdiag -v
netcache> netdiag -v
Performing physical layer diagnostics.....
Checking interface e0....
Checking interface e0 for hardware status....
Checking interface e0 for link status....
Checking interface e0 for a duplex mismatch
Interface e0 OK
Checking interface e0 to see if it is wedged
Checking interface e0 for misc errors
Checking interface e0 for capacity problems
OK
Performing network layer diagnostics.....
IP is OK
Performing transport layer diagnostics.....
Checking for problems with current TCP connections...
Checking for problems with recent TCP connections...
TCP is OK
UDP is OK

March 2003

Troubleshooting 12

Traceroute

netcache> traceroute
Usage: traceroute [-nrv] [-w wait] [-m max_ttl] [-p port#]
[-q nqueries][-t tos] [-s src_addr] host [data_size]

traceroute
Use the traceroute command to display the route packets take to a network host.
Syntax:
traceroute [-nrv] [-w <wait>] [-m <max_ttl>] [-p <port_number>] [-q nqueries] [-t tos] [s src_addr] host datasize

March 2003

Troubleshooting 13

Traceroute Sample

netcache> traceroute 198.95.226.1

traceroute to 198.95.226.1 (198.95.226.1), 30 hops max, 40 byte packets
1 wiley-120.sim.netapp.com (10.120.4.1) 2.836 ms 4.070 ms 1.239 ms
2 rocky.eng.netapp.com (10.56.0.5) 2.081 ms 1.664 ms 1.294 ms
3 10.34.0.57 (10.34.0.57) 1.226 ms 2.880 ms 1.032 ms
4 10.55.0.57 (10.55.0.57) 1.109 ms 1.473 ms 0.970 ms
5 10.32.0.42 (10.32.0.42) 1.787 ms 2.183 ms 1.031 ms
6 10.32.0.38 (10.32.0.38) 2.971 ms 2.770 ms 2.778 ms
7 pos2-int.netapp.com (10.1.3.35) 1.713 ms * 0.725 ms
8 198.95.226.1 (198.95.226.1) 2.922 ms * 2.595 ms
netcache>

March 2003

Troubleshooting 14

Resolving DNS Problems

Use ping or nslookup from an another
device to verify network DNS
ping s will return the destination IP

Check nameserver
Setup > DNS - General

Check NetCache DNS configuration

March 2003

Troubleshooting 15

DNS Troubleshooting
Cannot access URLs
outside firewall:
Make sure DNS
server can resolve
internet names

Cannot access URLs

inside firewall:
Make sure DNS
server is able to
resolve the specific
URL

March 2003

Troubleshooting 16

Flushing Cache and DNS

Utilities > Cache Objects > Flush

March 2003

Troubleshooting 17

Isolate NetCache Problems

Access
NetCache
GUI?

No

Telnet access?

No

Console
access?

No

Troubleshoot
hardware

Yes
Yes

NetCache is
running

Yes
Troubleshoot
configuration

March 2003

No

Troubleshooting 18

Configuration Errors
Most common configuration errors:
Damaged configuration file
Misconfigured network interface
Misconfigured IP address

March 2003

Troubleshooting 19

Cannot Access URLs

Log files
Authentication
Access controls
DNS configuration
Check automatic proxy configuration

March 2003

Troubleshooting 20

Cannot Access Specific URL

Test the server:
Ping the server
No response indicates a
server problem
netCache> ping www.espn.com
espn.com is alive

March 2003

Troubleshooting 21

Recovering Access to GUI

Cannot access
NetCache GUI

URL and Port

correct?

Yes

Interfaces
correclty
configured?

Yes

Telnet
Access?

Yes

No
No

March 2003

No

Reboot

No

Error
Messages?

Problem
persists?

Yes

Yes

Take
corrective
action

Call Tech
Support

Troubleshooting 22

Clearing Telnet Sessions

http://netcache:3132/alt/kickoff

Clearing telnet sessions

Telnet has a time-out feature, and occasionally, your session will time-out, but the
NetCache still considers the session active. Because only one telnet session is allowed,
this open session can block you from accessing the NetCache via telnet. The
http://netcache:3132/alt/kickoff URL provides you with a tool to close any open telnet
sessions on the NetCache.

March 2003

Troubleshooting 23

ok prompt

Access
console

Halt
appliance

'ok' prompt

OK prompt
Occasionally you will need to access the OK prompt to set or unset environment
variables or to access the five option maintenance menu.

March 2003

Troubleshooting 24

Command Line Diagnostics

Floppy boot device
Three ways to
access the 1-5
Floppy Boot Menu
Use Floppy Disks
Enter ^C when
prompted during boot
Set the floppy-boot?
environment variable
to true

Flash boot device

(C2100)
Use GRUB (GRand
Unified Bootloader)
menu to access
diagnostics
Same diagnostics as
with floppy boot

There are two ways to enter the 1-5 Floppy Boot Mode without the use of floppy disks
1) During the boot process, ONTAP will display:
press ^C for Special Boots
pressing the <CTRL> and C keys simultaneously while this is displayed will enter the
Floppy Boot menu
2) The NetCache can be also be set to floppy boot automatically. This can be done while
troubleshooting. However, do not leave this environment variable set during normal
operations as it will prevent the filer from booting without user intervention.
To set the NetCache for automatic floppy boot do the following:
netcache> halt
ok setenv floppy-boot? true
ok boot
The NetCache will then prompt you to floppy boot or normal boot. To unset this variable
do the following:
netcache> halt
ok unsetenv floppy-boot?

March 2003

Troubleshooting 25

Floppy Boot Commands

NetApp Release 5.1: Fri Mar 9 02:42:52 PST 2001

Copyright (c) 1992-2001 Network Appliance, Inc.
Starting boot on Mon Apr 16 17:40:59 GMT 2001
Mon Apr 16 17:41:16 GMT [net_e0:info]: Ethernet e0: Link up.
Waiting for WAFL init to complete.
(1)
(2)
(3)
(4)
(5)

Normal boot.
Boot without starting system services.
Change password.
Initialize all disks.
Maintenance mode boot.

Selection (1-5)?

(1)

Normal Boot

Use Normal Boot to run the filer normally, but from a

diskette.

(2)

Boot without /etc/rc

Use Boot without /etc/rc to troubleshoot and repair

configuration problems.
Booting without /etc/rc causes the filer to use only
default options settings, disregard all options settings you
put in /etc/rc, and disable some services, such as syslog.

(3)

Change Password

(4)

Initialize all disks

(5)

Maintenance mode boot

Used to change console password

Use Maintenance mode boot to go into Maintenance

mode and perform some volume and disk operations and
get detailed volume and disk information. Maintenance
mode is special for the following reasons:
Most normal functions, including file-system operations,
are disabled.
A limited set of commands is available for diagnosing
and repairing disk and volume problems.
Note: You exit Maintenance mode with the halt
command.

March 2003

Troubleshooting 26

GRUB Menu
GRUB

version 0.5.96.1

(621K lower / 1571584K upper memory)

+--------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
| Data ONTAP [backup]
[23;4HThe highlighted entry will be booted a
| Data ONTAP
a
l
[23;4H
Service menu
Booting 'Service menu'
root

(hd0,2)

Filesystem type is fat, partition type 0x1

configfile /x86/service.lst
GRUB

version 0.5.96.1

(621K lower / 1571584K upper memory)

+-------------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
| Network Appliance Diagnostics
|

March 2003

Troubleshooting 27

C2100 Diagnostics Commands

all
mb
mem
nvram
ether
fddi
scsi
fcal
atm
gbe
snet
trk
dafs
adpch

Run all system diagnostics

C2100 motherboard diagnostic
Main memory diagnostic
NVRAM diagnostic
Ethernet controller diagnostic
FDDI controller diagnostic
SCSI controller diagnostic
FCAL controller diagnostic
ATM controller diagnostic
GBE controller diagnostic
Servernet controller diagnostic
Troika controller diagnostic
Emulex VI(GBE) controller diagnostic
Adaptec SCSI controller diagnostic

March 2003

Troubleshooting 28

Printenv
ok printenv
Variable Name
Value
pci-probe-list
0,4,5,1,3
diag-switch?
false
fcode-debug?
true
.
.
.
netcache?
true
last-OS-booted-wafl-ver 13
last-OS-booted-ver
NetApp Release 5.1X6
floppy-boot?
true
java?
true

Default Value
0,4,5,1,3
false
true

Not available on C110X appliances

March 2003

Troubleshooting 29

floppy-boot? false
Set environment var floppy-boot? false
avoids the need for intervention in the
event of a panic or reboot

Not available on C110X appliances

March 2003

Troubleshooting 30

Rebooting a Down NetCache

Record
LCD
messages

ok prompt?

No

Cycle
power

ok prompt?

No

Boot from
floppies

Problem
resolved?

Yes

Done

No
Enter
'boot'

Yes

March 2003

Call tech
support

Troubleshooting 31

Maintenance Mode Option

Selection (1-5)? 5
You have selected the maintenance boot option:
the system has booted in maintenance mode allowing the
following operations to be performed:
?
disk
fcstat
halt
help
storage
vol
scsi
disk_fw_update_fix
disk_list
fcdiag
fctest
Mon Apr
version
Mon Apr
Mon Apr
> *>

Type "help <command>" for more details.

16 22:36:38 GMT [mgr.boot.new_OS:notice]: This system is running a new
of NetCache (now NetApp Release 5.1X6)
16 22:36:38 GMT [mgr.boot.reason_ok:notice]: System rebooted.
16 22:36:39 GMT [console_login_mgr:info]: root logged in from console

March 2003

Troubleshooting 32

Disk Option

To troubleshoot hardware or disk problems

netcache> disk
usage: disk <options>
Options are:
fail
<disk_name>
remove <disk_name>
swap
unswap
scrub { start | stop } unfail <disk_name>
-

fail a file system disk

remove a spare disk
prepare (quiet) bus for swap
undo disk swap and resume service
start or stop disk scrubbing
make a failed disk a spare again

March 2003

Troubleshooting 33

Disk Failures
No hot spare

Hot spare

Single disk failure:

degraded mode
autoshutdown in 24
hours
hourly error messages
Two disk failure:
autoshutdown
reinstall filesystem
restore backup configs

Single disk failure:

hot spare replaces
failed disk
logged in messages
log

View hot spare status at:

Maintenance > Disk > Management

March 2003

Troubleshooting 34

Disk Considerations
WAFL writes across the RAID group
Adding a new disk will unbalance the
write process and have an impact on
performance

March 2003

Troubleshooting 35

Troubleshooting Exercises

15 minutes in length
Use breakout rooms
Instructor will visit all rooms
Broadcast announcement 5 minutes
prior to regroup
Stay focussed, start telnet, start GUI
Share microphones, or no one else can
be heard

March 2003

Troubleshooting 36

Troubleshooting
Objectives
When you have completed this module, you will be able to do the following:

Determine if problems are internal or external to the NetCache

Isolate and resolve NetCache problems

Exercise Overview
The exercise will allow you to practice using the techniques and tools taught in this
course to isolate Netcache problems.

Time Estimate: 15 minutes

Required Hardware, Software, and Tools
Hardware
NetCache
Workstation
Software

NetCache 5.4 or later

ContentReporter
Browser clients
Special configuration files:
ts1.cfg
ts2.cfg
ts3.cfg
ts4.cfg

March 2003

Troubleshooting 37

Exercise: Problem 1
For this and the following problems, you will be asked to import a configuration. These
configurations have errors and it is your job identify the error, locate the cause and
recommend corrective action. At the end of each problem you will reset the NetCache to
its default settings.
1.

Access the NetCache command line

2.

Enter the following:

Sunnyvale:
NetCache>set < ftp://10.32.70.20/ts1.cfg
Elsewhere:
NetCache>set < ftp://64.94.95.10/ts1.cfg

3.

Log into your NetCache via the GUI.

4.

Open a browser client and proxy through your NetCache.

5.

Do you detect any symptom(s) of irregular performance?

6.

Record any symptoms here.

What might cause these indications?

8.

Confirm your diagnosis

9.

What is the corrective action?

10.

If possible, implement the corrective action

11.

Restore from backup or config z to return to the default configurations.

March 2003

Troubleshooting 38

Exercise: Problem 2.
1.

Access the NetCache command line

2.

Enter the following:

Sunnyvale:
NetCache>set < ftp://10.32.70.20/ts2.cfg
Elsewhere:
NetCache>set < ftp://64.94.95.10/ts2.cfg

3.

Log into your NetCache via the GUI.

4.

Open a browser client and proxy through your NetCache.

5.

Do you detect any symptom(s) of irregular performance?

6.

Record any symptoms here.

What might cause these indications?

8.

Confirm your diagnosis

9.

What is the corrective action?

10.

If possible, implement the corrective action

11.

Restore from backup or config z to return to the default configurations.

March 2003

Troubleshooting 39

Exercise: Problem 3
1.

Access the NetCache command line

2.

Enter the following:

Sunnyvale:
NetCache>set < ftp://10.32.70.20/ts3.cfg
Elsewhere:
NetCache>set < ftp://64.94.95.10/ts3.cfg

3.

Log into your NetCache via the GUI.

4.

Open a browser client and proxy through your NetCache.

5.

Do you detect any symptom(s) of irregular performance?

6.

Record any symptoms here.

What might cause these indications?

8.

Confirm your diagnosis

9.

What is the corrective action?

10.

If possible, implement the corrective action

11.

Restore from backup or config z to return to the default configurations.

March 2003

Troubleshooting 40

Exercise: Problem 4
1.

Access the NetCache command line

2.

Enter the following:

Sunnyvale:
NetCache>set < ftp://10.32.70.20/ts4.cfg
Elsewhere:
NetCache>set < ftp://64.94.95.10/ts4.cfg

3.

Log into your NetCache via the GUI.

4.

Open a browser client and proxy through your NetCache.

5.

Do you detect any symptom(s) of irregular performance?

6.

Record any symptoms here.

What might cause these indications?

8.

Confirm your diagnosis

9.

What is the corrective action?

10.

If possible, implement the corrective action

11.

Restore from backup or config z to return to the default configurations.

March 2003

Troubleshooting 41

March 2003

Troubleshooting 42