Sonoma Troubleshooting Guide.41a

A1000, A3000, A3500, A3500fc, RSM2000
Pre-call information gathering

techniques.
Anyone who has ever worked this product knows how quickly things can go bad. This
short TOI is intended to help you locate information within Sun, ask the right questions
before going onsite and to be able to identify potential problems and gather information
to help resolve problems quickly.
This also assumes you have some familiarity with the product and don't need help
installing or bringing up or working with the GUI.
We'll start with a brief (very brief) overview of what the products are...
All are the same from firmware and application perspective just one or two controllers
and a varying amount of disks. They use a hardware RAID controller designed by LSI.
They all have battery backup that retains data in cache in the event of a power outage,
they do not de-stage data to disk until power is restored. They all have data cache.
A1000
Single controller unit made from a single D1000, either 8 disk version or 12 disk version.
A3000 (formerly known as RSM2000)

A dual controller unit that uses the older RSM (Tabasco) trays. This was always a five
tray unit.
A3500
A dual controller unit that uses D1000 trays.
A3500fc
Same as A3500 but is the only fiber channel model.
The A3500 and A3500fc can come three ways,
1X5 One controller pair and five disk trays in a rack.
***Sun Personnel Use Only. *** 1

2X7 Two controller pairs and seven trays in one rack, each controller pair gets two entire
D1000's and three half D1000's.
3X15 Three controller pairs and 15 D1000's in two racks.
Listed next are some important web sites with valuable information.
http://storage.east/mreid
This is Mark Reids web page, lots of good data and a firmware to Raid Manager
application table that is invaluable for finding mismatches.
http://acts.ebay/storage/A3x00
This is Bob De Guc's page, it has a good section on nvsram and lots of other good
information.
http://webhome.ebay/A3x00
This is the Network Storage – Dilbert & Sonoma. Tons of great resources, a must see.
http://cpre-emea.uk/cgi-bin/df125853/lsiproc.tcl
This is a web based decoder for lsiser data.
http://pts-americas.west/nws/products/A3500/index.html
The PTS web page.
And of course.....
http://sunsolve.central.sun.com
Always get the latest patch matrix, Infodoc 43483, (formerly Early Notifier 20029) and
your patches here.

****Before any service call on this product it is best to do some basic research using
in band data gathering techniques. These methods are discussed here.****
1. VERIFY with data what version Raid Manager is being used with pkginfo -l SUNWosar .
2. VERIFY with data the firmware version for the application is appropriate using
module profile or raidutil -c.
3. Collect a module profile, Start RM6 GUI got to,configuration >File>Save Module
Profile, and check nvsram. If using 6.22 update1, the nvsram, listed as “Product
Revision” should be 0003 if not the upgrade is incomplete and this needs to be fixed.
4. Collect explorer data to confirm patch levels against Infodoc 43483and check
messages. In the explorer under /disks/sonoma you will find output of many raid manger
commands which can be used for trouble shooting.
Also, in the disks/sonoma/usr_lib_osa directory you will find the rmlog.log files. THESE
ARE CRUCIAL. With this file you can see the errors that raid manger has been reporting
with a simple tool, rmlogscan. REVIEW THE ERROR MESSAGES. Frequently there
will be more than one of these files that has been renamed, get it also.
5. IF THERE IS A MISMATCH BETWEEN FIRMWARE AND APP OR NVSRAM,

CORRECT THIS FIRST THEN PROCEED WITH SERVICE.
6. Use the information gathered to decide how to proceed, look at the over all health and
layout of the storage. Always check to see if they have layed out luns in a fashion that
will give tehm most redundancy. In larger arrays you should be able to survive a tray
failure without data loss. Use the module profile and plot the layout.
This concludes the inband method of data gathering. See examples on following pages.
1. VERIFY with data what version Raid Manager is being used.
pkginfo -l SUNWosar Sun Firmware Firmware Minimum-Maximum

Release Version Revision Patch Solaris Release
Number
VERSION: 1.0,REV=06.00 6.0 02.04.1d - 2.4 - 2.5.1

VERSION=6.01,REV=00.19 6.1 02.04.04.1 - 2.5.1 - 2.6
VERSION: 6.01,REV=01.04 6.1.1 02.05.06.33** 106513-04** 2.5.1 - Sol 7
VERSION: 6.01,REV=01.11 6.1.1 u1 02.05.06.33** 106513-04** 2.5.1 - Sol 7
VERSION: 6.01,REV=01.11 6.1.1 u2* 02.05.06.33** 106513-04** 2.5.1 - Sol 7
VERSION=6.22,REV=01.14 6.22 03.01.03.60 (.63??) 108834-07 (-09??) 2.5.1 11/97 - Sol 7
VERSION=6.22,REV=01.14 6.22 03.01.03.60 (.63??) 108553-07 (-09??) Sol 8
VERSION=6.22,REV=01.54 6.22.1 03.01.04.75 112125-04 2.6 - Sol 7
VERSION=6.22,REV=01.54 6.22.1 03.01.04.75 112126-05 Sol 8 - Sol 9

NOTE1: There are two different versions of "universal firmware"
02.05.05.00 and 02.05.06.32 neither of these versions are supported to
run on. You need to get the correct controller firmware downloaded.
* NOTE: Both 6.1.1 u1 and u2 report the same information from the
pkginfo command BUT if you do a "pkgparam SUNWosafw |grep 106513 " only
6.1.1u2 will come back with anything.
** NOTE: The patch 106513-05 with 02.05.06.39 firmware, was pulled

because of Sun Alert 24267
Patches 108834-09 and 108553-09 have both been pulled because of a

"Product boss Bulletin" See Bug IDs: 4453774and 4468699.
2. VERIFY with data the firmware version for the application is appropriate.
Below is firmware to application table.

http://storage.east/mreid/hints/slides/slide-20_firmware_history.html
RM6 Version Obsolete Application Boot Special

Firmware Firmware Instructions
RM6.0 02.04.1d 02.04.01.00 See Note 1

RM6.1 02.04.04.01 02.04.04.00
RM6.1.1 02.05.02.09 02.05.01.00
RM6.1.1-U1 02.05.02.11 02.05.01.00
RM6.1.1-U1+106513-02 Obsolete 02.05.02.14 02.05.01.00
RM6.1.1-U1+106707-01 Obsolete 02.05.02.15 02.05.01.00 Cluster Version
RM6.1.1-U1+106513-03 Obsolete 02.05.02.32 02.05.06.32
RM6.1.1-U1+106707-02 02.05.02.32 02.05.06.32 Cluster Version
RM6.1.1-U1+106513-04 02.05.06.33 02.05.06.32
RM6.1.1-U1+106513-05 Bad Patch 02.05.06.39 02.05.06.32
RM6.1.1-U1+108555-01 02.05.06.60 02.05.06.32 See Note 2
RM6.1.1-U2 02.05.02.32 02.05.06.32
RM6.1.1-U2+106513-03 Obsolete 02.05.02.32 02.05.06.32
RM6.1.1-U2+106707-02 02.05.02.32 02.05.06.32
RM6.1.1-U2+106513-04 Obsolete 02.05.06.33 02.05.06.32
RM6.1.1-U2+106513-05 Bad Patch 02.05.06.39 02.05.06.32
RM6.1.1-U2+108555-01 02.05.06.60 02.05.06.32 See Note 2
RM6.22 03.01.02.35 03.01.02.33 See Note 3
RM6.22+108834-03 Obsolete 03.01.03.54 03.01.03.00 See Note 3
RM6.22+108834-07 03.01.03.60 03.01.03.04 See Note 3
RM6.22+108834-09 Bad Patch 03.01.03.63 03.01.03.04 See Note 3
RM6.22+108553-03 Obsolete 03.01.03.54 03.01.03.00 See Note 3
RM6.22+108553-07 03.01.03.60 03.01.03.04 See Note 3
RM6.22+108553-09 Bad Patch 03.01.03.63 03.01.03.04 See Note 3
RM6.22-U1 Obsolete 03.01.04.68 03.01.04.00
RM6.22-U1+112125-01 Obsolete 03.01.04.71 03.01.04.00
RM6.22-U1+112126-01 Obsolete 03.01.04.71 03.01.04.00
RM6.22-U1+112125-03 03.01.04.75 03.01.04.00
RM6.22-U1+112126-03 03.01.04.75 03.01.04.00
Universal FW 02.05.05.00 02.05.00.00 See Note 4
Universal FW 02.05.06.32 02.05.06.32 See Note 4

Obsolete: The obsolete column references the patch, and is letting
you know a newer patch is available. When ever possible you should
be running the most current version of the firmware.
NOTE 1: If your controller FW is below 02.04.04.xx you must

upgrade to this level first before upgrading to RM6.1.1 and FW
02.05.02.xx.
NOTE 2: Patch 108555-01 requires that patch 106513-05 is installed

including the FW download to the controllers. This patch is only
available via CTE escalation.
NOTE 3: Before upgrading to any FW 03.01.xx.xx you must first be

at FW version 02.05.06.32. You can obtain the 02.05.06.32 firmware
from the SUNWosafw package included with RM 6.22.
NOTE 4: There are special rules for downloading to very early

versions of the FW (02.04.04.xx and below) that must be followed.
Just as in NOTE 1 above you must go through an intermediate
version. Also you must be sure to download the application FW
(*.apd) prior to the boot FW (*.bwd). This is the opposite way
that it is done when using the default settings in the GUI. This
is best done via the fwutil command from the command line. A FIN
will be released shortly.
Bad Patch If you see these firmware levels read the READMES and proceed
accordingly.
3. Collect a module profile, Start RM6 GUI got to,configuration >File>Save Module
Profile, and check nvsram. If using 6.22 update1, the nvsram, listed as “Product
Revision” should be 0003 if not the upgrade is incomplete and this needs to be fixed.
Profile for sykes_001
Profile Generated By: RAID Manager Release 06.22.01.54
Host: sykes
Controllers:
Name Serial Number Mode Logical Units

A (c4t5d0) 1T02284242 Active 1
Number of Drives = 6
Detailed Controller Information for sykes_001

Parameters Controller A (c4t5d0)
Board Name: Series 3
Board ID: 3240
Board Serial Number: 1T02284242
Product ID: StorEDGE A1000
Product Serial Number: 1T02284242
Vendor ID: Symbios
Date of Manufacture: 07/24/00
Product Revision: 0003
Host Interface: Parallel SCSI
Host-side ID (dec): 5
Boot Level: 03.01.04.00
Firmware Level: 03.01.04.75
Fibre Channel Level: -
Cache/Processor Size: 16/8 MB
Drives:
Detailed Drive Information for sykes_001
Location Capacity (MB) Status Vendor Product ID
[1,0] 8637 Optimal SEAGATE ST39173W SUN9.0G

[2,2] 8637 Spare-Stdby SEAGATE ST39173W SUN9.0G
Detailed Drive Information for sykes_001 (continued)
Location Firmware Version Serial Number Date Code
[1,0] 4315 LMF0882800 030599

[2,0] 4315 LMF0756000 030599
[1,1] 2815 LM88484200 083101
[2,1] 4315 LMF2385500 030599
[1,2] 4315 LMF2402000 030599
[2,2] 4315 LMD8689400 020499

Logical Units (LUNs):
Detailed LUN Information for sykes_001
Capacity
LUN Controller (MB) RAID Level
0 c4t5d0 34389 5
Detailed LUN Information for sykes_001 (continued)
LUN Associated Drives
0 [1,0] [2,0] [1,1] [2,1] [1,2]
Detailed LUN Information for sykes_001 (continued)
Segment Write Cache Cache Without

LUN Size Cache Mirroring Batteries Status
(Blocks)
0 64 Enabled Disabled Enabled Optimal
Drive Group Information for sykes_001
No. No. Total Remaining

of RAID of Capacity Capacity
Group LUNS Level Drives (MB) (MB)
Hot Spare - - 1 - -
1 1 5 5 34389 0
Logical Unit Information for sykes_001
Device RAID Cap.

LUN Group Name Level (MB) Status
0 1 c4t5d0 5 34389 Optimal
4. Collect explorer data to confirm patch levels against Infodoc 43483and check
messages. In the explorer under /disks/sonoma you will find output of many raid
manger commands which can be used for trouble shooting.
Also, in the disks/sonoma/usr_lib_osa directory you will find the rmlog.log files.
THESE ARE CRUCIAL. With this file you can see the errors that raid manger has
been reporting with a simple tool, rmlogscanmsg. REVIEW THE ERROR
MESSAGES. Frequently there will be more than one of these files that has been
renamed, get it also.

The utility to do this is called rmlogscanmsg, it can be found on Mark Reids page. The
tool will make the rmlog.log readable and will also translate the messages from
/var/adm/messages if there are raid manager messages there.
./rmlogscanmsg
USAGE: @(#)rmlogscan 2.0
rmlogscanmsg -l [-m] [-f logfile] | -f logfile | -m
-l run on live system, defaults to read /usr/lib/osa/rmlog.log
-f logfile - rmlog.log filename
-f is required for offline checks
if -f and -l are used together, the -f logfile takes precedence
-m message log format, default is /var/adm/messages
The following example is from an escalation that multiple controllers had been replaced
on. The rmlog.log showed the real problem all the time. A failed environmental card in a
D1000. VERY IMPORTANT to read the logs.
./rmlogscanmsg -f rmlog.log
Analysis of rmlog.log for
==========================================================================
Date Time Controller, LUN, Status, Sense Info

ASC/ASC , ASC/ASC TEXT
FRU , ERROR info , Status
==========================================================================
07/03/2002 17:42:43 c3t5d0s0 1T93600641 Message text: Successfully formatted logical unit 4 at RAID 5
with a capacity of 68938 MB.
11/15/2002 03:22:12 c3t5d3s0 1T93600641, 3, Optimal, Unit Attention
ASC/ASCQ A0 00 Write-Back Cache (With Mirroring) Could Not Be Enabled
02/08/2003 05:37:06 c3t5d2 1T93600641 Message text: The controller firmware version for controller
c3t5d2 (module: aurora_001) is 2.05.06. The recommended firmware version for this application is
3.00.00 or higher.\n
02/08/2003 05:37:06 c4t4d0 1T10400808 Message text: The controller firmware version for controller
c4t4d0 (module: aurora_001) is 2.05.06. The recommended firmware version for this application is
3.00.00 or higher.\n
02/08/2003 05:52:09 c3t5d2 1T93600641, 2, Optimal, Unit Attention
ASC/ASCQ 3F C7 Non-Media Component Failure
Sub-enclosure Group Power Supply , Failed

ASC/ASCQ 3F D9 Battery Near Expiration
ASC/ASCQ 0C 00 Unrecoverable Write Error
ASC/ASCQ A1 00 Write-Back Cache (With Mirroring) Could Not Be Enabled - Cache Size
ASC/ASCQ 95 02 Controller Removal/Replacement Detected
Sub-enclosure Group , Channel 2 Enclosure 1, Environmental Services Electronics, Failed
ASC/ASCQ 3F C3 Channel Failure
Controller Drive Interface Group
02/08/2003 08:15:45 c3t5d4 1T93600641, 4, Dead Multiple failures , Unit Attention
ASC/ASCQ 3F C3 Channel Failure
Controller Drive Interface Group
02/08/2003 08:22:38 c3t5d0 1T93600641 Message text: Successfully deleted LUN 4\n
02/08/2003 08:22:46 c3t5d3 1T93600641 Message text: Drive at [1,8] status changed to optimal\n
02/08/2003 08:42:01 c3t5d0 1T93600641 Message text: Successfully formatted logical unit 1 at RAID 1
ASC/ASCQ 3F 80 Drive Failed Due To Write Failure
Drive Group, Channel 2 Disk 8

ASC/ASCQ 3F 86 Drive Failed Due To Format Failure
Controller Buffer Group
ASC/ASCQ A4 00 Global Hot Spare Drive Smaller Than At Least One Drive In The Subsystem

02/08/2003 14:56:23 c3t5d0 1T93600641, 0, Optimal, Illegal Request
ASC/ASCQ 91 00 General Mode Select Error
02/08/2003 14:58:24 c3t5d0 1T93600641, 0, Optimal, Illegal Request
ASC/ASCQ 91 00 General Mode Select Error
02/08/2003 17:28:23 c3t5d0 1T93600641, 0, Optimal, Hardware Error
ASC/ASCQ 45 00 Selection Time-out On A Destination Bus
ASC/ASCQ A7 00 Recovered Data Buffer Memory Error
02/09/2003 08:30:50 c3t5d0 1T93600641 Message text: No parity errors detected on aurora_001.

02/24/2003 08:48:36 c4t4d1 1T10400808, 1, Degraded Reconstructing , Unit Attention
03/04/2003 11:25:08 c4t4d1 1T10400808 Message text: \nThe controller listed has detected that the
alternate controller or connection has failed in aurora_001. Please use the Recovery Guru in the Recovery
Application as soon as possible to resolve this problem.

03/25/2003 15:00:50 c4t4d0 1T10400808 Message text: The controller listed has detected that the
alternate controller has returned to an online status in aurora_001
END
Rev. 41a 01/28/2004
Inquires and especially corrections to chris.coffin@sun.com su: Sonoma Trouble Guide

Sonoma Troubleshooting Guide.41a

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Sonoma Troubleshooting Guide.41a

Caricato da

Copyright:

Formati disponibili

A1000, A3000, A3500, A3500fc, RSM2000

Pre-call information gathering

A3000 (formerly known as RSM2000)

The A3500 and A3500fc can come three ways,

1X5 One controller pair and five disk trays in a rack.

***Sun Personnel Use Only. *** 1

3X15 Three controller pairs and 15 D1000's in two racks.

This is a web based decoder for lsiser data.

The PTS web page.

***Sun Personnel Use Only. *** 2

5. IF THERE IS A MISMATCH BETWEEN FIRMWARE AND APP OR NVSRAM,

1. VERIFY with data what version Raid Manager is being used.

pkginfo -l SUNWosar Sun Firmware Firmware Minimum-Maximum

VERSION: 1.0,REV=06.00 6.0 02.04.1d - 2.4 - 2.5.1

***Sun Personnel Use Only. *** 3

** NOTE: The patch 106513-05 with 02.05.06.39 firmware, was pulled

Patches 108834-09 and 108553-09 have both been pulled because of a

Below is firmware to application table.

RM6 Version Obsolete Application Boot Special

RM6.0 02.04.1d 02.04.01.00 See Note 1

***Sun Personnel Use Only. *** 4

NOTE 1: If your controller FW is below 02.04.04.xx you must

NOTE 2: Patch 108555-01 requires that patch 106513-05 is installed

NOTE 3: Before upgrading to any FW 03.01.xx.xx you must first be

NOTE 4: There are special rules for downloading to very early

Profile for sykes_001

Profile Generated By: RAID Manager Release 06.22.01.54

Name Serial Number Mode Logical Units

Detailed Controller Information for sykes_001

***Sun Personnel Use Only. *** 5

Board Name: Series 3

Board ID: 3240

Board Serial Number: 1T02284242

Product ID: StorEDGE A1000

Product Serial Number: 1T02284242

Vendor ID: Symbios

Date of Manufacture: 07/24/00

Product Revision: 0003

Host Interface: Parallel SCSI

Boot Level: 03.01.04.00

Firmware Level: 03.01.04.75

Fibre Channel Level: -

Cache/Processor Size: 16/8 MB

Detailed Drive Information for sykes_001

Location Capacity (MB) Status Vendor Product ID

[1,0] 8637 Optimal SEAGATE ST39173W SUN9.0G

Detailed Drive Information for sykes_001 (continued)

Location Firmware Version Serial Number Date Code

[1,0] 4315 LMF0882800 030599

***Sun Personnel Use Only. *** 6

Detailed LUN Information for sykes_001

Detailed LUN Information for sykes_001 (continued)

LUN Associated Drives

0 [1,0] [2,0] [1,1] [2,1] [1,2]

Detailed LUN Information for sykes_001 (continued)

Segment Write Cache Cache Without

0 64 Enabled Disabled Enabled Optimal

Drive Group Information for sykes_001

No. No. Total Remaining

Logical Unit Information for sykes_001

Device RAID Cap.

***Sun Personnel Use Only. *** 7

Analysis of rmlog.log for

Date Time Controller, LUN, Status, Sense Info

Sun Personnel Use Only. 1

Sun Personnel Use Only. 2

Sun Personnel Use Only. 3

Sun Personnel Use Only. 4

Sun Personnel Use Only. 5

Sun Personnel Use Only. 6

Sun Personnel Use Only. 7

Sun Personnel Use Only. 8

Sun Personnel Use Only. 9

Sun Personnel Use Only. 10

Sun Personnel Use Only. 11

Sun Personnel Use Only. 12

Sun Personnel Use Only. 13