HP 3PAR Service Processor Troubleshooting

Guide

Abstract
This guide provides information for troubleshooting an HP 3PAR Service Processor.

HP Part Number: QR483-96013

Published: December 2012

 Copyright 2011, 2012 Hewlett-Packard Development Company, L.P.

Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial
Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under
vendor's standard commercial license.
The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express
warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall
not be liable for technical or editorial errors or omissions contained herein.
Acknowledgments
Intel, Itanium, Pentium, Intel Inside, and the Intel Inside logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries
in the United States and other countries.
Microsoft, Windows, Windows XP, and Windows NT are U.S. registered trademarks of Microsoft Corporation.
Adobe and Acrobat are trademarks of Adobe Systems Incorporated.
Java and Oracle are registered trademarks of Oracle and/or its affiliates.
UNIX is a registered trademark of The Open Group.

Contents
1 Introduction...............................................................................................4
Supported Service Processors.....................................................................................................4

2 Guidelines for Troubleshooting the Service Processor.......................................5

Troubleshooting Guidelines........................................................................................................5
Troubleshooting Tools................................................................................................................5
Audit and Logging Information..............................................................................................5
SPLOR................................................................................................................................6
Running a SPLOR from SPOCC.........................................................................................6
Using a SPLOR................................................................................................................6

3 Troubleshooting Service Processor Issues........................................................8

Troubleshooting IP Address Problems...........................................................................................8
Duplicate IP Address ...........................................................................................................8
Page Not Found Message..................................................................................................8

4 Support and Other Resources.......................................................................9

Contacting HP..........................................................................................................................9
HP 3PAR documentation............................................................................................................9
Typographic conventions.........................................................................................................12
HP 3PAR branding information.................................................................................................12

5 Documentation feedback...........................................................................13
A Service Processor Process Overview............................................................14
B Service Processor Messages in SPMAINT.....................................................15
C IP Ports Used by HP 3PAR..........................................................................17

Contents

1 Introduction
This guide provides information for troubleshooting an HP 3PAR Service Processor (SP).

Supported Service Processors

The following table lists supported SPs.
Table 1 Supported Service Processors
Part Number

Description

SP ID Range

F970 0017 01

Service Processor, 2U (Cal Digital)

SP00001SP00199

F970 0085

Service Processor, 1U (DELL 650)

SP00300SP00399

F970 0088

Service Processor, 1U (DELL 750)

SP00400SP00899

F975 0009 50 R5

Service Processor, 1U (WINTEC)

SP00900SP01999

F979 200051

Service Processor, 1U (SuperMicro)

SP02000SP02999

F975-200010

Service Processor, 1U (SuperMicro II)

SP03000SPxxxx

QR516A

Service Processor, 1U (Proliant

DL360e)

Varies. See note.

NOTE:

For the SP 4.1 release, the SP serial number range uses one of the following formats:

7-character SP ID: The literal SP string (2 characters) + 5 digits (for example, SP09997).
This ID format is used for legacy SPs.

12-character SP ID: The literal SP string (2 characters) + 10 alphanumeric characters

containing the HP serial number (for example, SP5463AB1253). This ID format is used for
SPs that run on HP Proliant DL360e.

41-character SP ID: The literal SP string (2 characters) + 32 alphanumeric characters + 7

dashes (-). The dashes delimit the 32 character alphanumeric strings into group of 4 characters
(for example, SP1111-bbbb-3333-DDDD-5555-6666-7777-8888). This ID format is used for
the virtual SP.

Introduction

2 Guidelines for Troubleshooting the Service Processor

This chapter provides troubleshooting guidelines and explains how to use this document for
troubleshooting Service Processor (SP) issues.

Troubleshooting Guidelines
General troubleshooting guidelines are as follows:

Identify the problem or problems.

Assess the situation.

Determine how the criticality of the problem. If more than one problem exists, determine which
problem is the most significant in terms of impact.

Collect as much information as possible relevant to the problem.

Generate an action plan and then execute it.

If the action does not resolve the problem, escalate.

Troubleshooting Tools
You can use the following tools to collect data from the SP:

Audit and Logging Information - Provides audit information regarding SP usage.

SPLOR - Gathers files to diagnose SP issues.

Audit and Logging Information

HP 3PAR SP Audit Information is contained in the SP_USER_ACTIVITY log file, which provides
the following audit information:

Users who accessed the SP

Logon and logoff times

The functionality used, such as Interactive CLI, Added System, and Started Guided Maintenance

Available through SPOCC-Reports, this file is gathered as part of a SPLOR and can be viewed by
HP Support personnel using Service Tools and Technical Support (STaTS).
Output examples:
yy/mm/dd 15:12:48
yy/mm/dd 15:13:20
yy/mm/dd 15:14:47
InServ: myInServ.
yy/mm/dd 16:20:56
myInServ
yy/mm/dd 16:22:58
yy/mm/dd 16:35:36

Remote Operation requested by userID as SP user #

#( userID-pts0 )/spmaint -- spmaint starting
#( userID-pts0 )/spmaint -- InFormOS validating update status -RC: 0
#( userID-pts0 )/spmaint -- interactive CLI requested for InServ:
#( userID-pts0 )/spmaint -- csstatus launched for myInServ
Remote Operation terminated for userID as SP user #

The actual commands that were executed can be found in other logs that are available to HP 3PAR
Support personnel.
HP 3PAR SP command logging includes the following:

Commands that were executed using SPMAINT are logged in SPMNTLOG, which is available
as a unique file in STaTS.

Commands that were initiated from the SP GUI (SPOCC) are logged in SPOCCMNTLOG.
This log is available as part of SPLOR.
Troubleshooting Guidelines

SPLOR
A SPLOR gathers files to help diagnose SP issues. You run a SPLOR from SPOCC.

Running a SPLOR from SPOCC

1.
2.
3.
4.

Log in to SPOCC as user 3parcust. Enter the IP address of the SP in a web browser, and
then press ENTER.
Click Support on the left-side of the screen.
Under Service Processor, click Launch SPLOR.
To view the SPLOR results, click View SPLOR Contents.

Using a SPLOR
A SPLOR provides the information for troubleshooting an SP. The following table provides a list of
file names and descriptions in the SPLOR.
Table 2 SPLOR file names and description

File name

Description

Dial-numbers

Connection Portal modem number

dmesg

Boot messages

dserrors

ST Process errors

dumplist

Sptransfer, SPup2date, stAnalyze, stBackDoor, stGrpNot

etc_filelist

STlogger, stMonitor, stNotify

etc_hosts

SP hosts file

etc_httpd_conf_httpd

Apache web server configuration file

etc_network

Network configuration information

etc_network_ifcfg-eth0

Eth0 network configuration information (SP Public network

interface)

etc_nsswitch.conf

Name service switch configuration file

etc_ntp.conf

Network Time Protocol (NTP) configuration file

etc_sp_fw

SP Firewall configuration file

etc_syslog_conf

Syslog configuration file

etc_xinetd_rsync

rsync service configuration

etc_xinetd_tftp

tftp service configuration

filter

Event filter file used by the spevent process to control

collection of storage system event files

gdda

Global values used for control purposes

installWork/pkgLog

SP OS Update log file

installWork/SPOSupdlog

SP OS Update log file

ipcs

Interprocess communication status

MessageLog340.100407.144344.txt

SP Message log

MessageLog341.txt

Active SP Message log

mgetty_login_config

Login configuration; describes which logins are allowed

Guidelines for Troubleshooting the Service Processor

Table 2 SPLOR file names and description (continued)

File name

Description

netstat_ifconfig

Network status information for connections, protocol, ports,

and sockets

network_routes

Network route information

platform

SP hardware configuration information

ps

Process status information

rmsgq

Process message queue file listings

rpms

Software package manager list

sp_tmp_installWork_filelst

Internal files used during upgrades and installations

sp_user_activity

SP user activity log file

sp_var_install_filelist

Internal files used during upgrades and installations

spconfig.100404.042520

SP configuration file (useful if an SP needs to be rebuilt)

spinHistory

Software installation history log file

splor

Report generated from "splor" file execution

spmntlog

SP maintenance log

spmntlog.xfer

SP maintenance log regarding transfers

top

Process activity statistics

var_lib_pgsql

Postgres database file listing

var_lib_pgsql_logfile

Postgres database log file

var_log_boot

Logs from SP booting up

var_log_cron

Cron job logfile

var_log_httpd_access_log

HTTP access log file

var_log_httpd_error_log

HTTP error log file

var_log_maillog

Log of mails to customer for events

var_log_messages

Linux message log file

var_log_secure

Authentication log for secure service

var_log_spooler

Printer log

weekly

Listing of weekly data file collection

WorkqMaint

Work queue statistics

Troubleshooting Tools

3 Troubleshooting Service Processor Issues

This chapter provides instructions for troubleshooting general problems that may occur with a
Service Processor (SP).

Troubleshooting IP Address Problems

Duplicate IP Address
If the wizard cannot configure the permanent IP address you entered because it is already in use:
1. Click Stop to stop the Apply Settings process and return to the Service Processor Setup wizard.
2. Click the Prev button until you reach Step 2: SP Networking.
3. Determine an available IPv4 address to use for the Service Processor, and then enter that IP
address in the IP Address text box.
4. Click Next until you return to Step 7: Apply Settings. (You do not need to re-enter any other
Service Processor settings.)
The wizard will automatically begin to apply the settings again.

Page Not Found Message

If the Service Processor displays a Page Not Found or similar message, the permanent Service
Processor IP address you entered is not a valid address. You must set up the Service Processor
again.
1. Restart the SP Setup wizard:

If you are using a physical Service Processor, return to the Set IP Address wizard and set
up a new, valid IP address.
After you have set up the new IP address, use that new permanent IP address to restart
the Service Processor Setup wizard (in other words, enter the URL https://<permanent
SP IP address>/sp/SpSetupWizard.html).

2.

If you are using a virtual Service Processor (VSP), use the temporary SP IP address to
restart the Service Processor Setup wizard (in other words, enter the URL https://
<temporary SP IP address>/sp/SpSetupWizard.html).

Proceed through the Service Processor Setup wizard, re-entering your Service Processor settings.
(The Service Processor ID is already set; you do not need to reset the ID.)

Troubleshooting Service Processor Issues

4 Support and Other Resources

Contacting HP
For worldwide technical support information, see the HP support website:
http://www.hp.com/support
Before contacting HP, collect the following information:

Product model names and numbers

Technical support registration number or Service Agreement ID (if applicable)

Product serial numbers

Error messages

Operating system type and revision level

Detailed questions

Specify the type of support you are requesting:

HP 3PAR storage system

Support request

HP 3PAR StoreServ 7200 and 7400 Storage systems

StoreServ 7000 Storage

HP 3PAR StoreServ10000 Storage systems

3PAR or 3PAR Storage

HP 3PAR T-Class storage systems

HP 3PAR F-Class storage systems

HP 3PAR documentation
For information about:

See:

Supported hardware and software platforms

The Single Point of Connectivity Knowledge for HP

Storage Products (SPOCK) website:
http://www.hp.com/storage/spock

Locating HP 3PAR documents

The HP 3PAR support website:

http://www.hp.com/3par/support_manuals
HP 3PAR storage system software

Storage concepts and terminology

HP 3PAR StoreServ Storage Concepts Guide

Using the HP 3PAR Management Console (GUI) to configure HP 3PAR Management Console User's Guide
and administer HP 3PAR storage systems
Using the HP 3PAR CLI to configure and administer storage
systems

HP 3PAR Command Line Interface Administrators

Manual

CLI commands

HP 3PAR Command Line Interface Reference

Analyzing system performance

HP 3PAR System Reporter Software User's Guide

Installing and maintaining the Host Explorer agent in order

to manage host configuration and connectivity information

HP 3PAR Host Explorer Users Guide

Creating applications compliant with the Common Information HP 3PAR CIM API Programming Reference
Model (CIM) to manage HP 3PAR storage systems
Migrating data from one HP 3PAR storage system to another HP 3PAR-to-3PAR Storage Peer Motion Guide

Contacting HP

10

For information about:

See:

Configuring the Secure Service Custodian server in order to

monitor and control HP 3PAR storage systems

HP 3PAR Secure Service Custodian Configuration Utility

Reference

Using the CLI to configure and manage HP 3PAR Remote

Copy

HP 3PAR Remote Copy Software Users Guide

Updating HP 3PAR operating systems

HP 3PAR Upgrade Pre-Planning Guide

Identifying storage system components, troubleshooting

information, and detailed alert information

HP 3PAR F-Class, T-Class, and StoreServ 10000 Storage

Troubleshooting Guide

Installing, configuring, and maintaining the HP 3PAR Policy

Server

HP 3PAR Policy Server Installation and Setup Guide

Support and Other Resources

HP 3PAR Policy Server Administration Guide

For information about:

See:
Planning for HP 3PAR storage system setup

Hardware specifications, installation considerations, power requirements, networking options, and cabling information
for HP 3PAR storage systems
HP 3PAR 7200 and 7400 storage systems

HP 3PAR StoreServ 7000 Storage Site Planning Manual

HP 3PAR 10000 storage systems

HP 3PAR StoreServ 10000 Storage Physical Planning

Manual
HP 3PAR StoreServ 10000 Storage Third-Party Rack
Physical Planning Manual

Installing and maintaining HP 3PAR 7200 and 7400 storage systems

Installing 7200 and 7400 storage systems and initializing
the Service Processor

HP 3PAR StoreServ 7000 Storage Installation Guide

Maintaining, servicing, and upgrading 7200 and 7400

storage systems

HP 3PAR StoreServ 7000 Storage Service Guide

Troubleshooting 7200 and 7400 storage systems

HP 3PAR StoreServ 7000 Storage Troubleshooting Guide

Maintaining the Service Processor

HP 3PAR Service Processor Software User Guide

HP 3PAR StoreServ 7000 Storage SmartStart Software

Users Guide

HP 3PAR Service Processor Onsite Customer Care

(SPOCC) User's Guide
HP 3PAR host application solutions
Backing up Oracle databases and using backups for disaster HP 3PAR Recovery Manager Software for Oracle User's
recovery
Guide
Backing up Exchange databases and using backups for
disaster recovery

HP 3PAR Recovery Manager Software for Microsoft

Exchange 2007 and 2010 User's Guide

Backing up SQL databases and using backups for disaster

recovery

HP 3PAR Recovery Manager Software for Microsoft SQL

Server Users Guide

Backing up VMware databases and using backups for

disaster recovery

HP 3PAR Management Plug-in and Recovery Manager

Software for VMware vSphere User's Guide

Installing and using the HP 3PAR VSS (Volume Shadow Copy HP 3PAR VSS Provider Software for Microsoft Windows
Service) Provider software for Microsoft Windows
User's Guide
Best practices for setting up the Storage Replication Adapter HP 3PAR Storage Replication Adapter for VMware
for VMware vCenter
vCenter Site Recovery Manager Implementation Guide
Troubleshooting the Storage Replication Adapter for VMware HP 3PAR Storage Replication Adapter for VMware
vCenter Site Recovery Manager
vCenter Site Recovery Manager Troubleshooting Guide
Installing and using vSphere Storage APIs for Array
Integration (VAAI) plug-in software for VMware vSphere

HP 3PAR VAAI Plug-in Software for VMware vSphere

User's Guide

HP 3PAR documentation

11

Typographic conventions
Table 3 Document conventions
Convention

Element

Bold text

Keys that you press

Text you typed into a GUI element, such as a text box
GUI elements that you click or select, such as menu items, buttons,
and so on

Monospace text

File and directory names

System output
Code
Commands, their arguments, and argument values

<Monospace text in angle brackets> Code variables

Command variables
Bold monospace text

Commands you enter into a command line interface

System output emphasized for scannability

WARNING! Indicates that failure to follow directions could result in bodily harm or death, or in
irreversible damage to data or to the operating system.
CAUTION:
NOTE:

Indicates that failure to follow directions could result in damage to equipment or data.

Provides additional information.

Required
Indicates that a procedure must be followed as directed in order to achieve a functional and
supported implementation based on testing at HP.

HP 3PAR branding information

12

The server previously referred to as the "InServ" is now referred to as the "HP 3PAR StoreServ
Storage system."

The operating system previously referred to as the "InForm OS" is now referred to as the "HP
3PAR OS."

The user interface previously referred to as the "InForm Management Console (IMC)" is now
referred to as the "HP 3PAR Management Console."

All products previously referred to as 3PAR products are now referred to as "HP 3PAR"
products.

Support and Other Resources

5 Documentation feedback
HP is committed to providing documentation that meets your needs. To help us improve the
documentation, send any errors, suggestions, or comments to Documentation Feedback
(docsfeedback@hp.com). Include the document title and part number, version number, or the URL
when submitting your feedback.

13

A Service Processor Process Overview

The following list provides an overview of the Service Processor (SP) processes.

SPCOLLECT - Periodically collects the storage system CLI data. The process also creates files
that contain the collected data, including the following file types:

Status

Config

Evtlog

Perform

Environ

EEPROM

Biosmsg

The file types correlate to the file extensions found in Service Tools and Technical Support
(STaTS). The period of collection varies with each file type.

14

SPCOMMCTLR - Controls the communication from the SP to the Connection Portal. This process
checks the configuration of Customer Controlled Access and determines the method of transfer:
Ethernet or modem.

SPNETMON - Monitors the SP network and gathers active network interface statistics.

SPTRANSFER - Manages the transfer of files from the SP to the Connection Portal. This process
interfaces with modem or an Ethernet network to accomplish transfer.

SPUP2DATE - Monitors Connection Portal for software updates staged to transfer to the SP.

STANALYZE - Analyzes event files created by spevent. Develops the text abstract that describes
the event or alert. This process sends a message to the notification process.

STBACKDOOR - Sends information about the SP process problems.

STGRPNOT - Scans notification database and issues any grouped notifications that have met
their timing parameters. Low-urgency notifications are grouped before they are sent.

STLOGGER - A process used by all other processes to record process logging information.
This information is stored in the MessageLog.

STMONITOR - Monitors various services and checks for normal operation. This process issues
back-door notifications for abnormal conditions.

STNOTIFY - Sends email notifications to users who have subscribed to the local notification
feature.

Service Processor Process Overview

B Service Processor Messages in SPMAINT

This appendix describes Service Processor (SP) messages in spmntlog file.
Acronyms used in this solution:

CCA - Customer Controlled Access (formerly called Big Yellow Switch [BYS])

CP - Connection Portal SP - Service Processor

ST - Service Tools group

When an SP transfers (or attempts to transfer) files from the customer site to the Connection Portal,
it logs messages to the spmntlog file.
The SP message table provides the following information:

The message that is logged in the spmntlog file

The meaning of each message

Remedial action that may be taken

NOTE: The following table is not an exhaustive list of the messages you may see. If you see a
message that is not on the list and that is causing a problem, contact HP Support.
Table 4 SP messages in SPMAINT
Message

Meaning

Remedial Action

Initializing

The code is doing some housekeeping

that does not require external
communications. If it persists beyond
30 seconds, something is probably
amiss.

If remote access is available, run a

SPLOR.
If remote access is not available, send
an FE onsite to log on to the SP console
and capture any errors. If the problem
persists, escalate to your HP 3PAR
Authorized Service Provider.

gddabatch failed on get for The text DB on the SP failed to

If the problem persists, escalate to your
produce the SP serial number, and we HP 3PAR Authorized Service Provider.
SP_ID
cannot continue without a serial
number. This is a logic failure and a
fatal error.
It is very unlikely that a user will
encounter this message.
Failed to get state of xfer Trying to query the Communication
If the problem persists, escalate to your
Controller to determine whether
HP 3PAR Authorized Service Provider.
media. Reason:...
transfers are allowed. This issue results
from a logic failure.
It is unlikely that a user will encounter
this message.
keyChange GetVar
varXFER_MEDIA failed...

SPcommCtlr is not providing the

If the problem persists, escalate to your
HP 3PAR Authorized Service Provider.
media (Ethernet or modem) that is
needed to do the exchange. This is a
logic failure and a fatal error.

keyChange GetVar varBYS

failed...

Program cannot get the state of the

Customer Controlled Access (CCA),
which indicates whether it is legal to
do transfers. This is a logic failure.

If the problem persists, escalate to your

HP 3PAR Authorized Service Provider.

keyChange BYS state is OFF Customer Controlled Access (CCA) is Change the CCA to allow outbound or
- sleeping for state change in the OFF state. Communications with bidirectional traffic (depending on your
the Connection Portal are not allowed preference).
in either direction, and a key

15

Table 4 SP messages in SPMAINT (continued)

Message

Meaning

Remedial Action

exchange is not allowed either. The

CCA will remain in this state
indefinitely, retrying every 60 seconds.
Testing connection...

CCA is OK, and the SP is attempting Verify network connectivity. If the

to contact the CP using SSH to perform problem persists, escalate to your HP
some trivial operation to make sure
3PAR Authorized Service Provider.
that a connection is possible. This
should be a short lived state.
If this state remains for more than five
minutes, the issue could be a logic
flaw. It could also be a network issue,
but generally, if the network
connection is not available, SSH times
out, in which case the system would
be in a different state.

sleeping for modem problems Transfer media is via modem and

there was a problem retrying the
- ...
operation. This could be due to a
modem issue in some older SPs or a
faulty phone line.
sleeping for firewall
access!

The CCA state is OK, but the firewall

is still not configured to allow the
communication to the CP. This is a
logic flaw.

Reboot the SP.

Check the phone line.

If the problem persists, escalate to your

HP 3PAR Authorized Service Provider.

keyChange: state OK, but ssh CCA is appropriately set for

Verify the firewall configuration and/or
resolve any network routing issues.
connection fails - sleeping communications to occur, but SP
cannot reach the Connection Portal.
This issue is most likely due to
customer firewall or routing problems.
CCA will stay in this state forever
(re-trying every two minutes) until the
SSH connection works.
Performing exchange

Everything is OK for the SP to talk to

the CP. The SP is about to invoke the
backend process to install the new
key.

spcmd communication problem A miscommunication between the SP

and CP has occurred. This message
- sleeping
represents a number of possible but
obscure issues.

16

Service Processor Messages in SPMAINT

No action required.

If the problem persists, escalate to your

HP 3PAR Authorized Service Provider.

C IP Ports Used by HP 3PAR

Table 5 (page 17) describes the TCP/IP port assignments for communication between various
components.
Table 5 TCP/IP port assignments
Port

Usage

Flow of Traffic

22: The Secure Shell (SSH) Protocol

in Legacy Mode

Used for storage system monitoring

and configuration through SSH
connections by the following
components:

Third-party SSH client <--> HP 3PAR

Service Processor

443: The Secure Socket Layer (SSL)

Protocol in Secure Service Agent
Mode

HP 3PAR Service Processor

HP 3PAR OS CLI Client

Third-party SSH client <--> HP 3PAR

Storage System CLI client
HP 3PAR Connection Portal <--> HP
3PAR Service Processor

HP 3PAR Connection Portal

80: World Wide Web (WWW) for
HTTP in Legacy Mode
443: WWW for HTTPS in Secure
Service Agent Mode
5781: 3PAR Event Reporting Service

Used by the HP 3PAR Service

HP 3PAR Service Processor --> WWW
Processor component to communicate
using the HTTP or HTTPS protocol.

Passes unsolicited events from the

Service Processor <--> Storage System
storage system and is used by the HP Event Service
3PAR Service Processor component.
(This port was used for HP 3PAR
Management Console events prior to
OS version 2.3.1)

5782: 3PAR Management Service

(unsecured)

Used by the following components for

storage system monitoring and
configuration over an unsecured
channel:
HP 3PAR Management Console
HP 3PAR OS CLI
HP 3PAR Recovery Manager
HP 3PAR System Reporter
HP 3PAR Service Processor

HP 3PAR IMC <--> HP 3PAR Storage

System CLI Server
HP 3PAR CLI <--> HP 3PAR Storage
System CLI Server
Recovery Manager <--> HP 3PAR
Storage System CLI Server
HP 3PAR System Reporter <--> HP 3PAR
Storage System CLI Server
HP 3PAR Service Processor --> HP 3PAR
Storage System CLI Server

(Port 2540 was used prior to SP

version 2.2.4.)

NOTE: If configured to do so, the storage system is capable of communication with a

customer-defined host via UDP port 514 (syslog).

17