Sei sulla pagina 1di 148

Technical

Information

SDU TROUBLESHOOTING

SOFTWARE RELEASE 2.16.3.X


SDU
CDMA2000 1X

ENGLISH
APR 2004
68P09258A80B
SPECIFICATIONS SUBJECT TO CHANGE WITHOUT NOTICE

Notice
While reasonable efforts have been made to assure the accuracy of this document, Motorola, Inc. assumes no liability resulting from any
inaccuracies or omissions in this document, or from use of the information obtained herein. The information in this document has been
carefully checked and is believed to be entirely reliable. However, no responsibility is assumed for inaccuracies or omissions. Motorola,
Inc. reserves the right to make changes to any products described herein and reserves the right to revise this document and to make
changes from time to time in content hereof with no obligation to notify any person of revisions or changes. Motorola, Inc. does not
assume any liability arising out of the application or use of any product, software, or circuit described herein; neither does it convey
license under its patent rights or the rights of others.
It is possible that this publication may contain references to, or information about Motorola products (machines and programs),
programming, or services that are not announced in your country. Such references or information must not be construed to mean
that Motorola intends to announce such Motorola products, programming, or services in your country.

Copyrights
This instruction manual, and the Motorola products described in this instruction manual may be, include or describe copyrighted
Motorola material, such as computer programs stored in semiconductor memories or other media. Laws in the United States and
other countries preserve for Motorola and its licensors certain exclusive rights for copyrighted material, including the exclusive
right to copy, reproduce in any form, distribute and make derivative works of the copyrighted material. Accordingly, any
copyrighted material of Motorola and its licensors contained herein or in the Motorola products described in this instruction manual
may not be copied, reproduced, distributed, merged or modified in any manner without the express written permission of Motorola.
Furthermore, the purchase of Motorola products shall not be deemed to grant either directly or by implication, estoppel, or
otherwise, any license under the copyrights, patents or patent applications of Motorola, as arises by operation of law in the sale of a
product.

Usage and Disclosure Restrictions


License Agreement
The software described in this document is the property of Motorola, Inc and its licensors. It is furnished by express license
agreement only and may be used only in accordance with the terms of such an agreement.

Copyrighted Materials
Software and documentation are copyrighted materials. Making unauthorized copies is prohibited by law. No part of the software or
documentation may be reproduced, transmitted, transcribed, stored in a retrieval system, or translated into any language or
computer language, in any form or by any means, without prior written permission of Motorola, Inc.

High Risk Activities


Components, units, or thirdparty products used in the product described herein are NOT faulttolerant and are NOT designed,
manufactured, or intended for use as online control equipment in the following hazardous environments requiring failsafe
controls: the operation of Nuclear Facilities, Aircraft Navigation or Aircraft Communication Systems, Air Traffic Control, Life
Support, or Weapons Systems (High Risk Activities). Motorola and its supplier(s) specifically disclaim any expressed or implied
warranty of fitness for such High Risk Activities.

Trademarks

MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are
the property of their respective owners.

Copyright 2003 Motorola, Inc.

Javat Technology and/or J2MEt: Java and all other Javabased marks are trademarks or registered trademarks of Sun
Microsystems, Inc. in the U.S. and other countries.
UNIXR: UNIX is a registered trademark of The Open Group in the United States and other countries.

REV091302
Table of Contents
SDU Troubleshooting
Software Release 2.16.3.x

List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii


List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
General Safety . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Chapter 1: Introduction
About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1
SDU Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5

Chapter 2: SDU Card Reset Procedure


SDU Card Reset Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1

Chapter 3: Login/Passwords
Log in to SDU Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1

Chapter 4: Data Collection


Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1

Chapter 5: SDU Cage Reset Procedure


SDU Cage Reset Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1

Chapter 6: SDU General Status


SDU General Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-1

Chapter 7: SDU Initialization Failure Scenarios


SDU Initialization Failure Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1

APR 2004 SDU Troubleshooting i


Table of Contents continued

Chapter 8: ISB/SPROC Troubleshooting


ISB/SPROC Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-1
Verifying ISB/SPROC Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-2
Troubleshooting ISB/SPROC Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-3
Troubleshooting ISB/SPROC Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-9

Chapter 9: SDU Synchronization Failures


SDU Synchronization Failure Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-1

Chapter 10: BPP Troubleshooting


BPP Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-1
Verifying BPP Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-2
Troubleshooting BPP Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-3

Chapter 11: CCA Troubleshooting


CCA Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-1
Verifying CCA Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-2
Troubleshooting CCA Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-3

Chapter 12: OMCR Failure Scenarios


OMCR Failure Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-1

Chapter 13: System Initialization Problems


System Initialization Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-1

Chapter 14: Call Processing Failures


Call Processing Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-1

Chapter 15: CFC Failures


SDU CFC Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15-1

ii SDU Troubleshooting APR 2004


List of Figures
SDU Troubleshooting
Software Release 2.16.3.x

Figure 1-1: SDU Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2


Figure 1-2: SDU Cage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3
Figure 1-3: SDU Cage Card Location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3
Figure 1-4: SDU Top I/O Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-4
Figure 5-1: SDU Main Breaker Panel (MBP) . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1
Figure 5-2: SDU Power Supply Modules and Circuit Breaker Modules . . . . . . 5-2

APR 2004 SDU Troubleshooting iii


List of Tables
SDU Troubleshooting
Software Release 2.16.3.x

Table 1-1: Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5


Table 7-1: SPROC Initialization Does Not Complete . . . . . . . . . . . . . . . . . . . . 7-2
Table 8-1: SPROC Unable to get IP address from ISB . . . . . . . . . . . . . . . . . . . 8-4
Table 8-2: ISB Fails to establish Gigabit Link . . . . . . . . . . . . . . . . . . . . . . . . . . 8-6
Table 8-3: Common OMCR Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-10
Table 8-4: SDU Communications Alarm 2822100:
Communication Fault, Physical Link Connection Error
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-11
Table 8-5: SDU Quality of Service Alarm 2823220:
Quality of Serv Fault, Packet Routing Capacity Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-11
Table 8-6: SDU Processing Error Alarm 2823390:
Processing Fault, Initialization Failure Troubleshooting Procedure . . . . . . . . . 8-12
Table 8-7: SDU Processing Error Alarm 2823420:
Processing Fault, Initialization Failure Troubleshooting Procedure . . . . . . . . . 8-13
Table 8-8: SDU Device Out of Service Alarm 2825000:
Device Out of Service Contact Lost Troubleshooting Procedure . . . . . . . . . . 8-13
Table 8-9: SDU Device Out of Service Alarm 2825040:
Device Out of Service Device Locked Troubleshooting Procedure . . . . . . . . 8-14
Table 8-10: SDU Alarm 2826555:
Application Heartbeat Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . 8-15
Table 8-11: SDU Alarm 2826560:
Application Messaging Failure Troubleshooting Procedure . . . . . . . . . . . . . . . 8-15
Table 8-12: SDU BPP Alarm 2826570:
Communication Failure Application Messaging Failure
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-16
Table 8-13: SDU Alarm 2826580:
Application Messaging Failure Troubleshooting Procedure . . . . . . . . . . . . . . . 8-16
Table 8-14: SDU Alarm 2826720:
Application Initialization Failure Troubleshooting Procedure . . . . . . . . . . . . . . 8-17
Table 8-15: SDU Alarm 2826730:
Application Initialization Failure Troubleshooting Procedure . . . . . . . . . . . . . . 8-17
Table 8-16: SDU Alarm 2826740:
Internal SCTP Stack Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . . . 8-18
Table 8-17: SDU Alarm 2826770:
Capacity Greatly Reduced Troubleshooting Procedure . . . . . . . . . . . . . . . . . 8-18
iv SDU Troubleshooting APR 2004
List of Tables continued

Table 8-18: SDU Alarm 2826850:


Application Platform Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . . 8-19
Table 8-19: SDU Alarm 2826870:
Incomplete Dataload Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . 8-19
Table 8-20: SDU Alarm 2826500:
Communication Failure Lost Connection with the MM
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-20
Table 8-21: SDU Alarm 2826520:
Communication Failure Link Disabled Troubleshooting Procedure . . . . . . 8-21
Table 8-22: SDU Alarm 2826530:
Communication Failure Lost Communication with the PDSN
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-21
Table 9-1: Troubleshoot SDU Heartbeat Failure . . . . . . . . . . . . . . . . . . . . . . . . 9-2
Table 10-1: Regenerate Corrupted Software . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-3
Table 10-2: SDU BPP Alarm 2826000:
Application Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . 10-4
Table 10-3: SDU BPP Alarm 2826001:
Application Error Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 10-4
Table 10-4: SDU BPP Alarm 2826010:
Call processing Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . 10-5
Table 10-5: SDU BPP Alarm 2826011:
Call Processing Error Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . 10-5
Table 10-6: SDU BPP Alarm 2826020:
Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-6
Table 10-7: SDU BPP Alarm 2826030:
CPU Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-6
Table 10-8: SDU BPP Alarm 2826040:
Hardware Component Error Troubleshooting Procedure . . . . . . . . . . . . . . . . . . 10-7
Table 10-9: SDU BPP Alarm 2826050:
Hardware Component Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . 10-7
Table 10-10: SDU BPP Alarm 2826060:
Partial Call Processing Capacity Loss Troubleshooting Procedure . . . . . . . . . . 10-8
Table 10-11: SDU BPP Alarm 2826070:
Full Call Processing Capacity Loss Troubleshooting Procedure . . . . . . . . . . . . 10-8
Table 10-12: SDU BPP Alarm 2826080:
Switch Processing Error Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . 10-9
Table 10-13: SDU BPP Alarm 2826090:
ISB Link Loss Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 10-9
Table 10-14: SDU BPP Alarm 2826100:
File Access Error Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 10-10
Table 10-15: SDU BPP Alarm 2826110:
High Temperature Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 10-10
Table 10-16: SDU BPP Alarm 2826111:
Very High Temperature Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . 10-11
Table 10-17: SDU BPP Alarm 2826140:
CPU Software Initialization Error Troubleshooting Procedure . . . . . . . . . . . . . 10-11
APR 2004 SDU Troubleshooting v
List of Tables continued

Table 10-18: SDU BPP Alarm 2826150:


CPU Software Initialization Failure Troubleshooting Procedure . . . . . . . . . . . . 10-12
Table 10-19: SDU Communications Alarms 2821801 and
2821802: Communication Fault, Packet Bus Connection no. <12>
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-12
Table 11-1: SDU Processing Error Alarm 2823380:
Processing Fault, Illegal Configuration Change Troubleshooting Procedure . . 11-3
Table 11-2: SDU Processing Error Alarm 2823410:
Processing Fault, Initialization Failure Troubleshooting Procedure . . . . . . . . . 11-4
Table 11-3: SDU Equipment Alarm 2823670:
Equipment Fault, Device Removal Detected Troubleshooting Procedure . . . . . 11-4
Table 11-4: SDU Equipment Alarm 2823690:
Equipment Fault, Hardware Fault Troubleshooting Procedure . . . . . . . . . . . . . 11-5
Table 11-5: SDU Equipment Alarm 2823790:
Equipment Fault, Subcomponent Fault Troubleshooting Procedure . . . . . . . . . 11-6
Table 11-6: SDU Environmental Alarm 2823900:
Environmental Fault, Power Supply Input Circuit Fault
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-6
Table 11-7: SDU Environmental Alarm 2824400:
Environmental Fault, AC Mains Under Volt Threshold Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-7
Table 11-8: SDU Environmental Alarm 2824440:
Environmental Fault, Bus Under Voltage Threshold Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-7
Table 11-9: SDU Environmental Alarm 2824460:
Environmental Fault, Cabinet Sensor Threshold Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-8
Table 11-10: SDU Environmental Alarm 2824480:
Environmental Fault, External Sensor Threshold Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-8
Table 11-11: SDU Environmental Alarm 2824700:
Environmental Fault, Temperature High Threshold Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-9
Table 11-12: SDU Environmental Alarm 2824740:
Environmental Fault, Temperature Very High Threshold Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-10
Table 11-13: SDU Communications Alarms 2821351 through 2821362:
Communication Fault, Electronic ID 1wire Bus Connection no. <112>
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-11
Table 11-14: SDU Communications Alarms 2821801 and 2821802:
Communication Fault, Packet Bus Connection no. <12> Troubleshooting
Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-11

vi SDU Troubleshooting APR 2004


Foreword

Scope of manual
This manual is intended for use by cellular telephone system
craftspersons in the day-to-day operation of Motorola cellular system
equipment and ancillary devices.
This manual is not intended to replace the system and equipment
training offered by Motorola, although it can be used to supplement or
enhance the knowledge gained through such training.

Obtaining Manuals
To view, download, or order manuals (original or revised), visit the
Motorola Lifecycles Customer web page at http://services.motorola.com,
or contact your Motorola account representative.
If Motorola changes the content of a manual after the original printing
date, Motorola publishes a new version with the same part number but a
different revision character.

Text conventions
The following special paragraphs are used in this manual to point out
information that must be read. This information may be set-off from the
surrounding text, but is always preceded by a bold title in capital letters.
The four categories of these special paragraphs are:

NOTE
Presents additional, helpful, non-critical information that
you can use.

IMPORTANT

* Presents information to help you avoid an undesirable


situation or provides additional information to help you
understand a topic or concept.

CAUTION
Presents information to identify a situation in which
damage to software, stored data, or equipment could occur,
thus avoiding the damage.

WARNING
Presents information to warn you of a potentially
hazardous situation in which there is a possibility of
personal injury.

APR 2004 SDU Troubleshooting vii


Foreword continued

The following typographical conventions are used for the presentation of


software information:
S In text, sans serif BOLDFACE CAPITAL characters (a type style
without angular strokes: for example, SERIF versus SANS SERIF)
are used to name a command.
S In text, typewriter style characters represent prompts and the
system output as displayed on an operator terminal or printer.
S In command definitions, sans serif boldface characters represent
those parts of the command string that must be entered exactly as
shown and typewriter style characters represent command output
responses as displayed on an operator terminal or printer.
S In the command format of the command definition, typewriter
style characters represent the command parameters.

Reporting manual errors


To report a documentation error, call the CNRC (Customer Network
Resolution Center) and provide the following information to enable
CNRC to open an MR (Modification Request):
the document type
the manual title, part number, and revision character
the page number(s) with the error
a detailed description of the error and if possible the proposed solution
Motorola appreciates feedback from the users of our manuals.

Contact us
Send questions and comments regarding user documentation to the email
address below:
cdma.documentation@motorola.com
Motorola appreciates feedback from the users of our information.

Manual banner definitions


A banner (oversized text on the bottom of the page, for example,
PRELIMINARY) indicates that some information contained in the
manual is not yet approved for general customer use.

24-hour support service


If you have problems regarding the operation of your equipment, please
contact the Customer Network Resolution Center (CNRC) for immediate
assistance. The 24 hour telephone numbers are:

North America +18004335202


Europe, Middle East, Africa +44 (0) 1793565444
Asia Pacific +861088417733
Japan & Korea . . . . . . . . . . . +81354633550
Latin American Countries +5112124020

For further CNRC contact information, contact your Motorola account


representative.

viii SDU Troubleshooting APR 2004


General Safety

Remember! . . . Safety
depends on you!!
The following general safety precautions must be observed during all
phases of operation, service, and repair of the equipment described in
this manual. Failure to comply with these precautions or with specific
warnings elsewhere in this manual violates safety standards of design,
manufacture, and intended use of the equipment. Motorola, Inc. assumes
no liability for the customers failure to comply with these requirements.
The safety precautions listed below represent warnings of certain dangers
of which we are aware. You, as the user of this product, should follow
these warnings and all other safety precautions necessary for the safe
operation of the equipment in your operating environment.

Ground the instrument


To minimize shock hazard, the equipment chassis and enclosure must be
connected to an electrical ground. If the equipment is supplied with a
three-conductor ac power cable, the power cable must be either plugged
into an approved three-contact electrical outlet or used with a
three-contact to two-contact adapter. The three-contact to two-contact
adapter must have the grounding wire (green) firmly connected to an
electrical ground (safety ground) at the power outlet. The power jack and
mating plug of the power cable must meet International Electrotechnical
Commission (IEC) safety standards.

NOTE
Refer to Grounding Guideline for Cellular Radio
Installations 68P81150E62.

Do not operate in an explosive


atmosphere
Do not operate the equipment in the presence of flammable gases or
fumes. Operation of any electrical equipment in such an environment
constitutes a definite safety hazard.

Keep away from live circuits


Operating personnel must:
S not remove equipment covers. Only Factory Authorized Service
Personnel or other qualified maintenance personnel may remove
equipment covers for internal subassembly, or component
replacement, or any internal adjustment.
S not replace components with power cable connected. Under certain
conditions, dangerous voltages may exist even with the power cable
removed.
S always disconnect power and discharge circuits before touching them.
APR 2004 SDU Troubleshooting ix
General Safety continued

Do not service or adjust alone


Do not attempt internal service or adjustment, unless another person,
capable of rendering first aid and resuscitation, is present.

Use caution when exposing or


handling the CRT
Breakage of the CathodeRay Tube (CRT) causes a high-velocity
scattering of glass fragments (implosion). To prevent CRT implosion,
avoid rough handling or jarring of the equipment. The CRT should be
handled only by qualified maintenance personnel, using approved safety
mask and gloves.

Do not substitute parts or


modify equipment
Because of the danger of introducing additional hazards, do not install
substitute parts or perform any unauthorized modification of equipment.
Contact Motorola Warranty and Repair for service and repair to ensure
that safety features are maintained.

Dangerous procedure
warnings
Warnings, such as the example below, precede potentially dangerous
procedures throughout this manual. Instructions contained in the
warnings must be followed. You should also employ all other safety
precautions that you deem necessary for the operation of the equipment
in your operating environment.

WARNING
Dangerous voltages, capable of causing death, are present in this
equipment. Use extreme caution when handling, testing, and
adjusting .

x SDU Troubleshooting APR 2004


Revision History

Manual Number
68P09258A80B

Manual Title
SDU Troubleshooting
Software Release 2.16.3.x

Version Information
The following table lists the manual version, date of version, and
remarks on the version.

Version Date of Issue Remarks


Level
B APR 2004 Added notes stating that logging into SDU boards should be done during
a maintenance window.

APR 2004 SDU Troubleshooting xi


1

Chapter 1: Introduction

Table of Contents

About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1


About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1
Associated Manuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1
SDU Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
SDU Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5
Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5

APR 2004 SDU Troubleshooting


1 Table of Contents continued

Notes

SDU Troubleshooting APR 2004


About this Manual 1

About this Manual


This manual includes procedures for troubleshooting the hardware
platform and associated software for the CDMA SDU (Selection
Distribution unit) system version 2.16.3.

IMPORTANT

* The procedures in this manual end when the symptom(s)


that caused the alarm(s) has been corrected. They DO NOT
supercede Scheduled Preventive Maintenance procedures.

IMPORTANT

* Some of the procedures in this manual must be performed


only during a scheduled maintenance window. Logging
into the SPROC, ISB, and BPP devices to perform
troubleshooting activities should be performed only during
a maintenance window to avoid system disruption.

NOTE
Additional performance tests may be required to insure
that the SDU still meets ATP specifications after any
device is replaced. Refer to the applicable Field
Replacement Unit (FRU) and ATP manual for this
information.

Associated Manuals
The following manuals and online help systems are referenced in this
manual.
S AN & SDU FRU (Field Replacement Procedures)
S CDMA2000 1X RAN Hardware Installation
S CDMA2000 1X RAN Functional Description
S CDL/CFC Reference Manual
S SMAP Operators Guide
S Cellular System Administration Guide (Online Help)
S System Command Reference (Online Help)
S System Output Messages (Online Help)
S SDU ATP (Acceptance Test Procedure)

APR 2004 SDU Troubleshooting 1-1


1
SDU Overview

Overview
The SDU performs the Selection/Distribution Function and Packet
Control Function for the 2.16.1.x network. For details, refer to the
CDMA2000 1X RAN Functional Description.

SDU Frame
The following figures illustrate the SDU frame, card cage, and top I/O
Panel.

Figure 1-1: SDU Frame

75 75 75 75
MAIN BREAKER PANEL

POWER SUPPLY
20 20

20 20 MODULES (4)
20 20

20 20

20 20
CIRCUIT BREAKER
MODULES (2)

UPPER SDU CAGE

FAN MODULES

LOWER SDU CAGE

FANS MODULES

NOTE:
1. Frame shown with doors removed
2. System configuration determines if upper cage is used.

1-2 SDU Troubleshooting APR 2004


SDU Overview continued
1

Figure 1-2: SDU Cage

Figure 1-3: SDU Cage Card Location

ISB card has a SPROC card mounted on it


BPP

CCA ISB ISB CCA

APR 2004 SDU Troubleshooting 1-3


1 SDU Overview continued

Figure 1-4: SDU Top I/O Panel

1-4 SDU Troubleshooting APR 2004


Acronyms 1

Acronyms
Table 1-1 defines acronyms used in this manual.

Table 1-1: Acronyms


Item Description
AN Access Node
BPP Bearer Payload Processor
CBM Circuit Breaker Module
CCA Customer Cabinet Alarms
DVM Digital Volt Meter
FRU Field Replaceable Unit
GND Ground
ISB Interface Switch Board
MBP Main Breaker Panel
MMI ManMachine Interface
PDU Power Distribution Unit
PSM Power Supply Module
RTN Return
SDU Selection Distribution Unit
SPROC System Processor
STP Shielded Twisted Pair
T1C Tier 1 Controller
NOTE
T1C is a different name for the ISB/SPROC.
UTP Unshielded Twisted Pair

APR 2004 SDU Troubleshooting 1-5


1 Acronyms continued

Notes

1-6 SDU Troubleshooting APR 2004


Chapter 2: SDU Card Reset Procedure
2

Table of Contents

SDU Card Reset Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1


Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1
Hardware vs. Software Reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1
Location of Reset Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1
Resetting the Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1
Lock/Unlock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
Verify Successful Reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2

APR 2004 SDU Troubleshooting


Table of Contents continued

Notes

SDU Troubleshooting APR 2004


SDU Card Reset Procedure

Overview
Many SDU BPP and SDU CCA alarm troubleshooting procedures 2
require a reset to resolve the alarm.
Generally, the alarm troubleshooting procedures conform to the
following variations:
S Reset not required
S Reset only required if the systems automatic reset fails
S Reset is up to the operator to perform when to reset depends upon
the urgency of the alarm.

WARNING
When manually resetting SDU cards, the call processing
resources will temporarily be unavailable. If possible,
attempt the reset during an appropriate maintenance
window (determine the severity of the alarm).

Hardware vs. Software Reset


When resetting cards to resolve an alarm, use the hardware reset method
(frontpanel reset switch) as opposed to the software reset method (e.g.
an OMCR reset command).

Location of Reset Switches


The BPP cards and CCA cards have hardware reset switches. The reset
switch is located in an unlabeled pinhole in the front panel of the card.

Resetting the BPPs/CCAs


To reset the card, insert an appropriately sized, nonconductive tool
(such as a toothpick), then press straight in with very light pressure.

Resetting the ISB/SPROCs


Since the ISB/SPROC does not have a frontpanel reset switch, it is
only possible to reset the ISB/SPROC by using the lock/unlock
commands from the OMCR, and by reseating the board (to do so refer
to the RAN Hardware Installation).

WARNING
Never reseat or reset the SPROC for any reason, unless
Motorola CNRC has given the instruction to do so. If this
is the case, first lock the SPROC from the OMCR.

APR 2004 SDU Troubleshooting 2-1


SDU Card Reset Procedure continued

Lock/Unlock
When manually resetting cards (as opposed to an automatic reset by the
2 system), use the OMCR lock and unlock commands. Lock removes the
card from service and unlock restores the card to service. For details,
refer to the System Command Reference.

Verify Successful Reset

NOTE
This case applies to both an automatic system reset and a
manual reset.

To verify that the reset of the card completes successfully:


S The cards frontpanel LED transitions from red (alarm state) to
blinking green to indicate that it is resetting. Once the reset completes,
the LED appears solid green.
S The OMCR should issue a state change event indicating that the card
has come back into service. You can use an OMCR Display
command to verify this. For example, if you reset a BPP card with
SDF software, the OMCR will indicate that the SDF is in service.

2-2 SDU Troubleshooting APR 2004


Chapter 3: Login/Passwords

Table of Contents

Log in to SDU Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1


Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1
Accessing Devices Using the OMCR . . . . . . . . . . . . . . . . . . . . . . . . . 3-1
Retrieving Information from Circuit BTS Configurations . . . . . . . . . . 3-3
Accessing the MMI Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4
Recovery Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-5

APR 2004 SDU Troubleshooting


Table of Contents continued

Notes

SDU Troubleshooting APR 2004


Log in to SDU Cards

Overview
Occasionally, an SDU troubleshooting procedure requires logging in to a
card. The following information provides directions on how to login to a
device remotely without accessing the front panel of the device. This
section explains how to log in to the various SDU cards.

IMPORTANT 3
* Logging into the SPROC, ISB, and BPP devices should be
performed only during a maintenance window to avoid
system disruption.

Accessing Devices Using the


OMCR
Most of the SDU devices can be accessed remotely (not on site using the
front panel MMI interface) using the OMCR. This action makes it easy
to gain access to the device and collect data. The following procedures
provide the information needed to access the SDU devices using the
OMCR.

Locating SDU IP Addresses


At the OMCR enter the following command:
S DISP SDU<sduid> IPADDRESS
The output of the display command provides the following information:

Table 3-1: Interpreting and Retrieving IP Addresses from the


DISPLAY Command Output
Item Description
SDU1 Management Link IP address of the
active SPROC
SDU1 Router 2 IP address of the
active ISB
SDU2 Router 2 IP address of the
standby ISB
BPP<frameid><cageid><slotid> BCP IP of LOGIN
ADDRESS of SDF
or PCF
BPP<frameid><cageid><slotid> CPE IP address of the first
DSP of the SDF or
PCF

APR 2004 SDU Troubleshooting 3-1


Log in to SDU Cards continued

WARNING
Any of the following actions could prevent future access to
the SDU boards:
S Never use the telnet command to log into any of the
SDU boards.
S Never use the Exit command to logout of any of the
SDU boards.

3 S Never used CTRL+X to logout of the SDU boards.

SPROC
At the OMCR use the following command to access to the SPROC.
S rlogin SDU1 MANAGEMENT_LINK<IP address>
To start a CLI session, type CLI at the prompt.
To exit the SPROC type LOGOUT at the prompt.

ISB
At the OMCR use the following command to access to the ISB.
S rlogin SDU1 ROUTERX<IP address>
To start a CLI session, type CLI at the prompt.
To exit the SPROC type LOGOUT at the prompt.

BPP
At the OMCR use the following command to access to the BPP.
S rlogin SDUBCP<IP address>
To start a CLI session, type CLI at the prompt.
To exit the SPROC type LOGOUT at the prompt.

3-2 SDU Troubleshooting APR 2004


Log in to SDU Cards continued

Retrieving Information from


Circuit BTS Configurations
The following two items contain directions on how to display items from
a circuit BTS configuration site.

Locating the PSI Address (for Circuit BTS Configurations)


At the OMCR enter the following command:
S DISP XC<xcid> STATUS ISO ALL PKTIF 3
Determine which PKTIF by entering the following command:
S DISP PKTIF<cageid>>slotid> PKTIFCONF
Display the PSI IP address of the PKTIF by entering the following
command:
S DISP PSI<cageid><slotid> BOARDCONF

PKTIF (on the Transcoder)


At the OMCR enter the following command:
S DISP XCX STATUS ISO ALL PKTIF
S DISP PKTIFXX PKTIFCONF
S DISP PSI## ipaddress
user: psi
password: rulesdata
S To start a CLI session enter: psiStart
To exit enter CTRL+}
send escape
quit

APR 2004 SDU Troubleshooting 3-3


Log in to SDU Cards continued

Accessing the MMI Console


The MMI console is used to enter commands. Each SDU card has an
MMI console connection.

IMPORTANT

* Logging into the SPROC, ISB, and BPP devices should be


performed only during a maintenance window to avoid
system disruption.
3

Configuration
To log in to SDU cards, you will need to obtain a PC with a Serial port,
and a terminal emulation program such as Hyperterminal. A
standalone dumb terminal can also be used.
The settings consist of:
S Baud Rate: 9600 Bps
Data: 8 Bit
S Parity: None
S Stop bit: 1
S Flow control: none
S VT100 Emulation
It is necessary to use a Null Modem cable to connect the PC to the
ISB/SPROC.

NOTE
S The MMI Console Interface is started when the network
element is booted; it is not necessary to manually start
the MMI Console Interface.
S After the terminal/PC is connected to the card, you may
need to press the <Enter> key several times.

S SPROC
Login: motorola
Password: developer
Exit MMI Password: partymoto

NOTE
Use startMMI to enter MMI mode again.

3-4 SDU Troubleshooting APR 2004


Log in to SDU Cards continued

S BPP
Login: motorola
Password: developer
Secure MMI: altivec
S CCA
Login: motorola
Password: developer
S ISB 3
Login: motorola
Password: developer
Enter logout to quit.

Recovery Procedure
If the rlogin command for the BPP command does not work, the
following procedure can be performed at the site.
Connect a serial port to the front panel of the BPP board.
At the prompt enter the following:
S lock
S shell Restart
If the lock and restart commands do not work, perform an lock/unlock
command to see if the session can be recovered.

APR 2004 SDU Troubleshooting 3-5


Log in to SDU Cards continued

Notes

3-6 SDU Troubleshooting APR 2004


Chapter 4: Data Collection

Table of Contents

Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1


Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1
Overview of Black Box Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1
Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1
4
Running collectbox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
Running bboxlisten . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3
Accessing the Collected Black Boxes . . . . . . . . . . . . . . . . . . . . . . . . . . 4-4

APR 2004 SDU Troubleshooting


Table of Contents continued

Notes

SDU Troubleshooting APR 2004


Data Collection

Data Collection
When contacting Motorola CNRC for assistance with SDU problems,
consider collecting the following data:
S Log current status information from the OMCR
S Optionally, also collect black box data from the OMCR.
This section describes the process for collecting the black box data.

Overview of Black Box


Programs

collectbox
The collectbox program is intended to be used on an OMCR to collect 4
debug information from the SDU. The debug information being gathered
is called a blackbox (airplane lingo) and contains information recorded
on the controller (SPROC).

bboxlisten
The bboxlisten program collects the blackbox logs from the SDU in the
case of an SPROC reset. Bboxlisten also provides some automatic bbox
file maintenance on the OMCR.

Prerequisites
The OMCR Software Installation should include the black box
programs for SDU. Installing software on the OMCR is beyond the
scope of this manual.
The following is a brief overview of the install process for reference
purposes: untar collect.tar into a temporary directory. Run installbbox
from the temporary directory to copy the executables and scripts to the
correct directory on the OMCR. Run cronbbox INSTALL to start the
server.

APR 2004 SDU Troubleshooting 4-1


Data Collection continued

Running collectbox

When to Run collectbox


In a nonreset situation, run collectbox to collect data from the SDU.

Running collectbox
From the OMCR, use the following command to run collectbox:

collectbox [ne ne# | ip ipaddress]


[p | f filter]
[active | standby | both ]
[d directory]

4
Setting the ne# (network element id or sdu number) or ipaddress is
mandatory. Optional parameters can be entered in any order.

NOTE
If not specified, the ne # is resolved from ip address. If the
ip address is not specified, the OMCR will pull the
address from the MIB based on the ne # specified.

Optional Parameters
Optional parameters:
S [f filter] Grabs the full blackbox using the specified filter, the
blackbox is not reset as part of the collection. The default behavior is
to collect the filtered blackbox from the active side. If neither the f or
p parameters are specified, the full unfiltered blackbox is collected.
Default behavior for the full blackbox is collection from the active
side.
S [p] Grabs the preserved blackboxes from the last reset. The default
behavior is to grab the preserved blackboxes from both the active and
the standby controllers. If neither the f or p parameters are
specified, the full unfiltered blackbox is collected. Default behavior
for the full blackbox is collection from the active side.
S [d directory] Specify the directory for output to be stored on the
OMCR. The default directory when none is specified is
/sc/spool/bbox.
S [active] Collect the blackbox from only the active side.
S [standby] Collect the blackbox from only the standby side.
S [both] Collect the blackbox from both sides.

4-2 SDU Troubleshooting APR 2004


Data Collection continued

Examples
The following examples illustrate the use of collectbox.
S collectbox ne 5
Collects a full blackbox from the active side of SDU5.
S collectbox ne 8 p
Collects the preserved blackboxes from both sides of SDU8.
S collectbox ne 1 f SNMP
Collects a filtered blackbox based on the keyword SNMP from the
active side of SDU1.
S collectbox ip 10.4.5.1 p standby
Collects the preserved blackboxes from the standby side of the SDU
with a management address of 10.4.5.1.

Description of Operation 4

This command does not require the CLI to be up when the command is
executed. The command will attempt to find the SDU in the database
and determine the correct IP address to connect to. The command will
normally ping the platform before attempting to collect the specified
blackbox. The command will display the name and location of the file
collected. Any temporary files created on the platform will be cleaned
up. This program telnets to the platform which grabs console output
temporarily from the front panel serial port.

Running bboxlisten

When to Run bboxlisten


When the blackbox server at the OMCR detects that the SPROC has
reset, bboxlisten automatically runs collectbox to collect data from the
SDU.

Running bboxlisten
No user intervention is required for the automatic upload of black boxes.

Description of Operation
The server will wake up when an SDU sends it a message after an
SPROC reset. The server will check every 30 seconds for collection
requests and invoke the collection script. The script will attempt to
upload the preserved black boxes from either SPROC (active and
standby). The script will turn off nagging on the SPROC so no more
requests to upload are generated from this reset. The script will check the
number of stored black boxes on the OMCR and clean up if needed.

APR 2004 SDU Troubleshooting 4-3


Data Collection continued

Accessing the Collected Black


Boxes
The black box output is cleaned up on the OMCR and stored in the
/sc/spool/bb directory. The collected black boxes are named based on the
sdu#, time of collection, active/standby, and original file name on
platform.
The following is an example:
/sc/spool/bb/s210041939a0.out
/sc/spool/bb/s210041939s0.out

s2 SDU2
1004 October 4
1939 7:39 PM
4 a0 preserved file /data/blackbox/0 from
active side
s1 preserved file /data/blackbox/1 from
standby side
sh preserved file
/data/blackbox/hardResetBBox from standby side
af0 preserved file /active/init/hungbbox0
from standby side

4-4 SDU Troubleshooting APR 2004


Chapter 5: SDU Cage Reset Procedure

Table of Contents

SDU Cage Reset Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1


Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1
SDU Cage Power Supply Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1
Cage Reset Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2
Cage Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2

APR 2004 SDU Troubleshooting


Table of Contents continued

Notes

SDU Troubleshooting APR 2004


SDU Cage Reset Procedure

Introduction
If the SDU should become completely unresponsive for some reason, the
method of last resort for recovery would be a power cycle of the SDU
cage (which in turn would power cycle every FRU in the SDU cage).

SDU Cage Power Supply


Overview
This section provides a high level overview of the SDUs power supply
and distribution in order to familiarize the user with the location and
purpose of the various power supplies and circuit breakers on the SDU
frame.
There are two power supplies per SDU cage, for a total of 4 per frame.
Each power supply pair (2 vertically stacked on the left and the right)
feeds a group of 7 circuit breakers which in turn feed 7 power zones
(or groups of slots) in a given cage.
There are 7 pushpull switches that correspond to the 7 circuit breakers
(running vertically alongside the power supplies). However, there is a
metal plate over these to prevent them from being manually opened. 5
These should not be used to perform a cage reset.
In addition to the 7 circuit breakers per cage, there are also 2 circuit
breakers per cage which correspond to the main power supplies to the
cage. These are located near the top of the front side of the SDU cage on
the panel labeled MAIN BREAKER PANEL. There is one circuit
breaker pushpull switch for each power supply and two power supplies
for each cage.

Figure 5-1: SDU Main Breaker Panel (MBP)

75 75 75 75

APR 2004 SDU Troubleshooting 5-1


SDU Cage Reset Procedure continued

Figure 5-2: SDU Power Supply Modules and Circuit Breaker Modules

20 20

20 20

20 20

20 20

20 20

Cage Reset Procedure

5 To cycle power to a given cage, the two circuit breaker switches for the
two power supplies to that cage must be pulled. The left two switches
control power to the lower cage and the right two switches control power
to the upper cage. Again, both switches must be pulled since the power
supply feeds are redundant.
Once both breaker switches are pulled, wait 1015 seconds and then
push the same two switches back in towards the frame. At this point, the
system should initialize completely and return to operation.

Cage Recovery
Once both circuit breaker switches are pushed back in for the Power
Supply, the SDU Frame will begin to initialize the SPROC and other
Payloads. During this process, the OMCR (NE Manager) gets an Alarm
which indicates the Loss of Communication with SDU Frame. Once
the NE (SPROC) completes the initialization and the ISB learns the
Routes from Access Node, then the SPROC will start sending the
SYNC Required to the OMCR. The OMCR also clears the Loss of
Communication Alarm. Then, the OMCR SYNCs with NE. Once the
NE SYNCs up with OMCR, the SDU Frame will be in the Operational
State.

5-2 SDU Troubleshooting APR 2004


Chapter 6: SDU General Status

Table of Contents

SDU General Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-1


General SDU Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-1

APR 2004 SDU Troubleshooting


Table of Contents continued

Notes

SDU Troubleshooting APR 2004


SDU General Status

General SDU Status


The following LEDs states indicate basic SDU status:
S Once the SDU boots up after a reset, the LED lights on the front panel
of an SDU frame should appear solid green.
S If the LED light on a board appears red, the board is having an alarm.
S If the LED light on a board appears blinking green, the board is
resetting.
S If the LED light is in a constant blinking green, the board may not be
provisioned at the moment.

Boot progress
To monitor the boot progress of a board in an SDU frame, a console
needs to be connected to the board. Without a console connection, only
the LEDs can provide a limited status. Once the LED light turns into
solid green, the board is finished rebooting.

State Determination
Once the SDU finishes rebooting, the OMCR can be used to check the
state of the SDU. For OMCR Status/Display Commands, please refer
to the System Command Reference.

APR 2004 SDU Troubleshooting 6-1


SDU General Status continued

Notes

6-2 SDU Troubleshooting APR 2004


Chapter 7: SDU Initialization Failure Scenarios

Table of Contents

SDU Initialization Failure Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1


Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1
The SPROC (either active or standby) fails to join in
the cluster and is unable to boot up . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-1
SPROC (either active or standby) initialization does not complete . . . 7-2
The SPROC (either active or standby) displays
going unmanaged, and resets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-3
Hung threads occur on the SPROC (active or standby)
and the SPROC resets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-4
The software load on one SPROC (either active or standby)
becomes corrupt, and the SPROC reboots . . . . . . . . . . . . . . . . . . . . . . . 7-5
A failure of the active LAN causes the SPROC to reset . . . . . . . . . . . . 7-6

APR 2004 SDU Troubleshooting


Table of Contents continued

Notes

SDU Troubleshooting APR 2004


SDU Initialization Failure Scenarios

Introduction
This section contains troubleshooting procedures for SDU initialization
failure scenarios.

NOTE
Refer to chapter 6 for descriptions of LED states during
various boot progress stages.

WARNING
Never reseat or reset the SPROC for any reason, unless
Motorola CNRC has given the instruction to do so. If this
is the case, first lock the SPROC from the OMCR.

The SPROC (either active or


standby) fails to join in the
cluster and is unable to boot
up

Symptom
If the SPROC (either active or standby) fails to join in the cluster and is
unable to boot up, use the following information to troubleshoot the
problem.

Possible Causes
Possible causes for this symptom include one of the following:
S LAN failure 7
S While the OMCR is configuring the ISBs, the SPROCs unmanaged
timeout timer expires.

Recovery Action
Reset the SPROC, and verify that the ISBs are in a stable state.

APR 2004 SDU Troubleshooting 7-1


SDU Initialization Failure Scenarios continued

SPROC (either active or


standby) initialization does not
complete

Symptom
The SPROC initialization does not complete if the SPROC console
displays node not in database and then resets.

Possible Causes
Possible causes for this symptom include one of the following:
S The configuration data file (baseNECDF .xml) for the SPROC either
contains errors or is corrupted.
For example, the configuration data may include an incorrect cage or
slot number.
S The SPROC is not properly seated.

Recovery Action
Perform the following recovery actions.

Table 7-1: SPROC Initialization Does Not Complete


Step Action
1 If the SPROC is not correctly seated, correctly seat the
SPROC.
NOTE
Refer to the CDMA2000 1X RAN Hardware Installation for
details.
2 If the SPROC is correctly seated, regenerate the SDU Data
7 Files from the OMCR.
Enter the following commands from the OMCR CLI.
NOTE
Refer to the CDMA2000 1X Cellular System Administration
Guide for additional details.
2a generate sdu# datafiles <Enter>
2b activate sdu# <Enter>

7-2 SDU Troubleshooting APR 2004


SDU Initialization Failure Scenarios continued

The SPROC (either active or


standby) displays going
unmanaged, and resets

Symptom
If the SPROC console displays going unmanaged and then resets,
this indicates that the SPROC lost management control and is
performing a reset to clear the underlying problem.

Possible Causes
This symptom may be due to one of the following causes:
S With a 2N redundant setup, if both the ISBs reset and initialize again,
one of the SPROCs will display going unmanaged and reset as
a result of a split brain resolution process. There is a possibility that
the node which was a manager (the active one) may also reset.
S Both ISBs are resetting.

Recovery Action
There is no recovery action. The SPROC automatically resets itself to
resolve the problem.

APR 2004 SDU Troubleshooting 7-3


SDU Initialization Failure Scenarios continued

Hung threads occur on the


SPROC (active or standby) and
the SPROC resets

Symptom

NOTE
A hung thread is like a hanging software task, only it is a
thread instead of a process.

If the hung thread problem occurs on the SDU, the SPROC console
displays something like the following output:
[D2D37D8] PANIC: Starting Emergency Exit
Handlers
[D2D37D8] PANIC: Called Default Emergency Exit
Handler
[D2D37D8] Writing Emergency Exit Trace
Information for 1 threads
[D2D37D8] goahead/hap Error: HUNG THREADS
DETECTED: srk tid = 14000011 (OS id = c43fa70)
[D2D37D8] Default EEH: PANIC for srk tid =
14000011 (OS id = c43fa70).
[D2D37D8] Finished Default Emergency Exit
Handler
[D2D37D8] PANIC: Finished Emergency Exit
Handlers
[D2D37D8] PANIC: Closing SRK
[D2E3D50] Closing Extensions
[D2E3D50] RIM Extension Shutting Down.
NEI : NEIX: Possible GA hung thread. Stack trace
7 in black box.Rebooting in 30 secs...
The SPROC will reboot.
If the standby SPROC has a higher slot number than the active SPROC,
and the hung thread problem occurs on the active SPROC, this
problem may cause a site reset (if the standby SPROC loses arbitration
for the cluster manager role).

Possible Causes
A GA hung thread is the cause. The GoAhead software shuts itself
down, which causes the SPROC to reboot.

7-4 SDU Troubleshooting APR 2004


SDU Initialization Failure Scenarios continued

Recovery Action
A full SDU reset from the OMCR may help to clear out the hung thread
problem, but use caution.

WARNING
A full SDU reset will cause a site outage. If possible,
perform the reset during an appropriate maintenance
window.

The software load on one


SPROC (either active or
standby) becomes corrupt, and
the SPROC reboots

Symptom
If the SPROC software becomes corrupted, the SPROC will reboot.

Possible Causes
See the Symptom description.

Recovery Action
No recovery action is necessary.
The SDU automatically resolves the problem using one of the following
methods:
S For a single SPROC node setup, the SPROC will reboot ten times,
then it will switch the partition and attempt to initialize using the other
partition.
7
S For a 2N redundant SPROC setup, if the other node becomes the
manager, the node running the corrupted software load will: become
the nonpotential manager; FTP the load from the manager; and
reset itself after switching the partition.

APR 2004 SDU Troubleshooting 7-5


SDU Initialization Failure Scenarios continued

A failure of the active LAN


causes the SPROC to reset

Symptom
If the active LAN fails, the active SPROC will transition to the OOS
state, and the redundant SPROC will take over.
The SPROC console displays output similar to the following:
NEIX: active interface x with an ip address
x.x.x.x and subnet x failed

Possible Causes
The ISB connected to the active LAN is not functioning properly.

Recovery Action
None. This is a normal behavior.

7-6 SDU Troubleshooting APR 2004


Chapter 8: ISB/SPROC Troubleshooting

Table of Contents

ISB/SPROC Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-1


Overview of ISB/SPROC Troubleshooting . . . . . . . . . . . . . . . . . . . . . . 8-1
Verifying ISB/SPROC Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-2
Verifying ISB/SPROC Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-2
Logging in to the ISB/SPROC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-2
Troubleshooting ISB/SPROC Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-3
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-3
ISB Code Load Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-3
The ISB is unable to provide DHCP service to the SPROC . . . . . . . . . 8-4
The ISB reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-5
ISB reset causes payloads to reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-5
The ISB fails to establish the gigabit link . . . . . . . . . . . . . . . . . . . . . . . 8-6
During SPROC failover, the ISB automatically reset . . . . . . . . . . . . . . 8-7
Pulling out the standby ISB and the standby SPROC
causes the payloads to reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-7
The Active ISB reset causes the SPROC (active and standby) to reset 8-7
The Standby ISB reset causes SPROC to reset . . . . . . . . . . . . . . . . . . . 8-8
Troubleshooting ISB/SPROC Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-9
ISB/SPROC Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-9
SDU ISB Communications Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-11
SDU ISB Quality of Service Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-11
SDU ISB Processing Error Alarm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-12
SDU SPROC Device Out of Service Alarms . . . . . . . . . . . . . . . . . . . . 8-13
SPROC Application Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-15
SPROC Communication Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-20
8

APR 2004 SDU Troubleshooting


Table of Contents continued

Notes

SDU Troubleshooting APR 2004


ISB/SPROC Troubleshooting

Overview of ISB/SPROC
Troubleshooting
This chapter deals specifically with the procedures for troubleshooting
the ISB/SPROC cards in the SDU.

APR 2004 SDU Troubleshooting 8-1


Verifying ISB/SPROC Status

Verifying ISB/SPROC Status


When troubleshooting the ISB/SPROC, it is recommended to verify the
current ISB/SPROC status by checking LEDs and verifying board states
using the OMCR Display/Status commands.
For details, refer to chapter 6, General SDU Status.

Logging in to the ISB/SPROC


Some procedures may require logging in to the ISB/SPROC directly.
Refer to chapter 3 for details on how to log in to SDU boards.

IMPORTANT

* Logging into the SPROC, ISB, and BPP devices should be


performed only during a maintenance window to avoid
system disruption.

8-2 SDU Troubleshooting APR 2004


Troubleshooting ISB/SPROC Failures

Overview
This section contains troubleshooting procedures for various ISB failure
scenarios.

WARNING
Never reseat or reset the SPROC for any reason, unless
Motorola CNRC has given the instruction to do so. If this
is the case, first lock the SPROC from the OMCR.

ISB Code Load Failure

Symptom
The ISB is unable to download the software load from the SPROC.

Possible Causes
The possible cause may be one of the following. The ISB:
S Is new from factory;
S Does not have a programmed route to the SPROC;
S Is not receiving the GETCODELOAD from the proxy component.

Recovery Action
Contact CNRC for assistance.

APR 2004 SDU Troubleshooting 8-3


Troubleshooting ISB/SPROC Failures continued

The ISB is unable to provide


DHCP service to the SPROC

Symptom
The SPROC is unable to get IP service from the ISB.

Possible Causes
Ensure that the debug ethernet port of the SPROC isnt connected to an
outside network. If, when the SPROC boots up, the console display
indicates it is loading up on the debug interface, the debug port is
connected to the network.

Recovery Action
Perform the recovery actions in Table 8-1.

Table 8-1: SPROC Unable to get IP address from ISB


Step Action
1 Ensure that the front panel Ethernet Jack does not have a
cable connected to it. The front panel Ethernet Jack is the
debug port.
2 Reboot the SPROC and ISB. Refer to chapter 2 for details on
resets.

8-4 SDU Troubleshooting APR 2004


Troubleshooting ISB/SPROC Failures continued

The ISB reset

Symptom
The symptoms include at least one of the following:
S The ISB resets
S The SPROC console displays keep alive timed out for the ISB.

Possible Causes
The SPROC reset the ISB, because the SPROC did not receive any
acknowledgement from the ISB.

Recovery Action
None. The automatic reboot should correct the problem.

ISB reset causes payloads to


reset

Symptom
The ISB reset causes payloads (BPPs, CCAs) to reset.

Possible Causes
If the ISB failover didnt occur quickly enough, the payloads will also
reset.

Recovery Action
None. The automatic ISB reset should correct the problem.

APR 2004 SDU Troubleshooting 8-5


Troubleshooting ISB/SPROC Failures continued

The ISB fails to establish the


gigabit link

Symptom
The ISB fails to establish the gigabit link.

Possible Causes
Possible causes include:
S Cable failure
S Span I/O failure
S The SPROC database was not updated.

Recovery Action
Check/replace cables and/or span I/O (swap out); check/update the
SPROC database for gigabit links.

Table 8-2: ISB Fails to establish Gigabit Link


Step Action
1 At the OMCR, verify the settings for the
MLS_SDU_CON links, the Gigabit connections from the
AN to the SDU.
If necessary, correct or change those settings.
NOTE
If you change these settings, additional steps from the
OMCR will be necessary. For example, if you modify a
configuration from the OMCR, you need to regenerate the
data files (the Generate Data File command) and reactive the
devices.
For more details, and the complete procedural context, refer
to the System Command Reference and the Cellular System
Administration Guide.
2 If it is not a configuration problem, continue at the next step.

8 3 Check and if necessary replace the gigabit cable(s).


4 If it is not a cable problem, swap out the span I/O. Refer to
the SDU FRU manual for the applicable procedure.

8-6 SDU Troubleshooting APR 2004


Troubleshooting ISB/SPROC Failures continued

During SPROC failover, the ISB


automatically reset

Symptom
During the SPROC failover, the ISB automatically reset.

Possible Cause
The route updates were not fast enough.

Recovery Action
None. The automatic reset should correct the problem.

Pulling out the standby ISB


and the standby SPROC
causes the payloads to reset

Symptom
The Payloads reset after a Dual standby pull.

Possible Causes
The route updates were not fast enough.

Recovery Action
None. The automatic reset should correct the problem.

The Active ISB reset causes


the SPROC (active and
standby) to reset

Symptom
The active ISB reset causes both SPROCs to reset.

Possible Causes
The route updates were not fast enough. 8

Recovery Action
None. The automatic reset should correct the problem.

APR 2004 SDU Troubleshooting 8-7


Troubleshooting ISB/SPROC Failures continued

The Standby ISB reset causes


SPROC to reset

Symptom
When the standby ISB reset, the SPROC also reset.

Possible Causes
The route updates were not fast enough.

Recovery Action
None. The automatic reset should correct the problem.

8-8 SDU Troubleshooting APR 2004


Troubleshooting ISB/SPROC Alarms

ISB/SPROC Alarms
This section contains troubleshooting procedures for
ISB/SPROCrelated alarms.
Though the ISB or SPROC reports these alarms, in some cases another
card may be causing the underlying problem.

WARNING
Never reseat or reset the SPROC for any reason, unless
Motorola CNRC has given the instruction to do so. If this
is the case, first lock the SPROC from the OMCR.

NOTE
In addition to maintaining troubleshooting logs, you may
also want to notify CNRC of problems if they frequently
reappear.

NOTE
For a more detailed description of each alarm, refer to
System Output Messages. Also, if you are not sure when
you need to resolve the problem, refer to the alarm
documentation (in the System Output Messages) for the
severity level.

APR 2004 SDU Troubleshooting 8-9


Troubleshooting ISB/SPROC Alarms continued

Table 8-3: Common OMCR Commands


Use OMCR command
Verify that the device is Display/Status commands for SDU cards:
present and operational status sdusdu#
status sdubppsdubpp#
status sduccasdu#sducage#sducca#
status sdusdu#sducage#sduisb#
status sdusdu#sducage#sdusproc#
status sdfrgsdu#
status pcfrgsdu#
Verify that the links are display mmsdfralinksdu#cbsc#mmsdfralink#
functional or correctly mmsdfralinkconf
configured display mmpcfralinksdu#cbsc#mmpcfralink# status
display mmpcfralinksdu#cbsc#mmpcfralink#
mmpcfralinkconf
status sdulinksdulink#
display
sdupdsnlinksdu#sdupdsncluster#sdupdsnlink#
pdsnlinkconf
NOTE
For information on editing, deleting and reprovisioning devices, refer to the Cellular
System Administration Guide.

8-10 SDU Troubleshooting APR 2004


Troubleshooting ISB/SPROC Alarms continued

SDU ISB Communications


Alarms

2822100 Communication Fault, Physical Link Connection


Error
This alarm is reported when a Communications Protocol Error causes a
fault on the Physical Link Connection.
Perform the following procedure to resolve this alarm.

Table 8-4: SDU Communications Alarm 2822100:


Communication Fault, Physical Link Connection Error Troubleshooting Procedure
Step Action
1 Check the physical link connection at the SDU and the farend connection of the physical link at the
AN IP Switch frame (Catalyst MLS chassis). Determine whether it is a physical link connectivity
error or a bad cable.
Refer to the CDMA2000 1X RAN Hardware Installation manual for details on physical cabling and
connectivity between the SDU and AN.

2 If it is not a physical link problem, replace the ISB. Refer to the CDMA2000 1X AN & SDU FRU for
details.

SDU ISB Quality of Service


Alarms

2823220 Quality of Service Fault, Packet Routing Capacity


Exceeded
This alarm is reported when heavy traffic and/or congestion causes the
Packet Routing Processor on the ISB card to reach its operating limit.
Perform the following procedure to resolve this alarm.

Table 8-5: SDU Quality of Service Alarm 2823220:


Quality of Serv Fault, Packet Routing Capacity Exceeded Troubleshooting Procedure
Step Action
8
1 Add additional processing capacity to the site or reduce the traffic loading.

APR 2004 SDU Troubleshooting 8-11


Troubleshooting ISB/SPROC Alarms continued

2823221 Quality of Service Fault, Packet Routing Capacity


Severely Exceeded
This alarm is reported when the systems Packet Routing Queue cannot
process the volume of traffic directed to it. The alarm indicates that the
Packet Routing Queue Threshold has been exceeded because the Packet
Routing Capacity was severely exceeded.
Perform the procedure in Table 8-5 to resolve this alarm.

SDU ISB Processing Error


Alarm

2823390 Processing Fault, Initialization Failure


This alarm is reported when a configuration or customization error
causes an Initialization failure of a hardware device.
Perform the following procedure to resolve this alarm.

Table 8-6: SDU Processing Error Alarm 2823390:


Processing Fault, Initialization Failure Troubleshooting Procedure
Step Action
1 The system automatically attempts to reset the card. The card will initialize if the automatic reset is
successful.
If the card initializes, no further action is necessary.
If the card fails to initialize, continue at the next step.
NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state.

2 If the card fails to initialize, attempt a hardware reset.

3 If the reset fails, lock and replace the card using the procedures contained in the CDMA2000 1X AN &
SDU FRU manual.

8-12 SDU Troubleshooting APR 2004


Troubleshooting ISB/SPROC Alarms continued

2823420 Processing Fault, OSPF Inactivity Timer Expired


This alarm indicates an OSPF Inactivity Timer Expired fault due to a
Software Program Error. This alarm represents a series and the last two
digits of the Alarm ID indicates the connection number. Typical range:
2342023422.
Perform the following procedure to resolve this alarm.

Table 8-7: SDU Processing Error Alarm 2823420:


Processing Fault, Initialization Failure Troubleshooting Procedure
Step Action
1 If there are (in addition to this alarm) alarms on the farend ISB, or if other BPPs and CCAs are also
reporting similar alarms, the problem is most likely with the ISB. In that case, the ISB problems
should be addressed first. Otherwise the problem is probably local to the BPP, and the BPP should be
reset at a convenient time.

2 If the problem continues, replace the BPP. Refer to the CDMA2000 1X AN & SDU FRU.

SDU SPROC Device Out of


Service Alarms

IMPORTANT

* If a card in the SDU is out of service, such as a BPP or


CCA, the SPROC reports the alarm.

2825000 Device Out of Service Contact Lost


This alarm is raised when the network element is unable to communicate
with a device. This may result from a failed device or the absence of a
device. A failure to communicate with a device constitutes a failure of a
device.
The CCA provides SDU monitoring; the BPP provides either selection
or packet data functions; the ISB provides transport, while the SPROC
provides the main SDU applications.
8
Perform the following procedure to resolve this alarm.

Table 8-8: SDU Device Out of Service Alarm 2825000:


Device Out of Service Contact Lost Troubleshooting Procedure
Step Action
1 Verify that the device specified in the alarm output is present and operational, using the applicable
OMCR CLI commands.

2 If the device is not present or operational, replace it with an operational device. Refer to the
CDMA2000 1X AN & SDU FRU manual.

APR 2004 SDU Troubleshooting 8-13


Troubleshooting ISB/SPROC Alarms continued

2825030 Device Out of Service Software Failure


This alarm is reported when a software application fails to respond to the
system heartbeat that polls the applications for operational status.
No immediate action is required. Monitor the device and ensure that it is
successfully restored to service.
If devices are reset often due to application failures, requesting a new
software load may be beneficial.

2825040 Device Out of Service Device Locked


This alarm is reported when the operator locks a device.
Perform the following procedure to resolve this alarm.

Table 8-9: SDU Device Out of Service Alarm 2825040:


Device Out of Service Device Locked Troubleshooting Procedure
Step Action
1 Once the device is unlocked using the unlock command, the alarm will clear.
Refer to the Cellular System Administration Guide for details on the unlock command.

2825050 Device Out of Service Diagnostics


This alarm is reported when the operator runs an OutofService
Diagnostic Test on a device by using the test command.
No operator action is needed. When the Diagnostic Test is complete, the
alarm will clear automatically.

8-14 SDU Troubleshooting APR 2004


Troubleshooting ISB/SPROC Alarms continued

SPROC Application Alarms

2826555 Application Heartbeat Failure


A lowlevel software element on the SPROC which controls the link has
failed.
Perform the following procedure to resolve this alarm.

Table 8-10: SDU Alarm 2826555:


Application Heartbeat Failure Troubleshooting Procedure
Step Action
1 The system disables the affected MMSDFRALINK or MMPCFRALINK. To clear the alarm, it is
necessary to delete and reprovision the link from the OMCR.

NOTE
For details, refer to the Cellular System Administration Guide for topics that deal with the SDULINK
software object and its subelements (the MMSDFRALINK and MMPCFRALINK).

2826560 Application Messaging Failure


An error has occurred in the software on the SPROC. Application
messaging among tasks on the SPROC has been corrupted to some
degree. The error may be in an application attempting to establish
onboard communications. Other causes may be bind or send failures in
the onboard message system.
Perform the following procedure to resolve this alarm.

Table 8-11: SDU Alarm 2826560:


Application Messaging Failure Troubleshooting Procedure
Step Action
1 Wait for the alarm to clear.
The card resets in an attempt to recover. The standby SPROC becomes the active SPROC during the
reset. The alarm clears when the SPROC is reset.

2 The failed SPROC becomes the standby SPROC. If the problem is not resolved in the reset, the 8
standby SPROC continues to fail. In this case, replace the card.
Refer to the CDMA2000 1X AN & SDU FRU manual for details.

APR 2004 SDU Troubleshooting 8-15


Troubleshooting ISB/SPROC Alarms continued

2826570 Application Messaging Failure


An error has occurred in the software on the SPROC. Application
messaging among tasks on the SPROC has been corrupted to some
degree. The error may be in an application attempting to establish
onboard communications. Other causes may be bind or send failures in
the onboard message system.
Perform the following procedure to resolve this alarm.

Table 8-12: SDU BPP Alarm 2826570:


Communication Failure Application Messaging Failure Troubleshooting Procedure
Step Action
1 The system disables the affected SDUSPROC. To clear the alarm, it is necessary to delete the
SDUSPROC from the OMCR and reprovision in an attempt to bring it back into service.

NOTE
For details, refer to the Cellular System Administration Guide and System Command Reference.

2826580 Application Messaging Failure


An error has occurred in the software on the SPROC. Application
messaging among tasks on the SPROC has been corrupted to some
degree. The error may be in an application attempting to establish
onboard communications. Other causes may be bind or send failures in
the onboard message system.
Perform the following procedure to resolve this alarm.

Table 8-13: SDU Alarm 2826580:


Application Messaging Failure Troubleshooting Procedure
Step Action
1 The system disables the affected device shown in the alarm. To clear the alarm, it is necessary to delete
the affected device from the OMCR and reprovision in an attempt to bring it back into service.
For details, refer to the Cellular System Administration Guide and System Command Reference.

8 NOTE
For details, refer to the Cellular System Administration Guide and System Command Reference.

8-16 SDU Troubleshooting APR 2004


Troubleshooting ISB/SPROC Alarms continued

2826720 Application Initialization Failure


An internal software error has occurred on the SPROC. A software
application controlling the link implicated in this alarm failed to
complete its initialization.
Perform the following procedure to resolve this alarm.

Table 8-14: SDU Alarm 2826720:


Application Initialization Failure Troubleshooting Procedure
Step Action
1 The system disables the affected MMSDFRALINK, or MMPCFRALINK, or SDUPDSNLINK. To
clear the alarm, it is necessary to delete the affected link type from the OMCR and reprovision in an
attempt to bring it back into service.

NOTE
For details, refer to the Cellular System Administration Guide and System Command Reference.

2826730 Application Initialization Failure


An internal software failure has occurred on the SPROC in the onboard
communications protocol software. This board will be unable to
complete initialization.
Perform the following procedure to resolve this alarm.

Table 8-15: SDU Alarm 2826730:


Application Initialization Failure Troubleshooting Procedure
Step Action
1 Wait for the alarm to clear. The card resets in an attempt to recover. The alarm clears when the failed
SPROC recovers.
NOTE
If the standby SPROC fails to initialize, the active SPROC is not affected.
If the active SPROC has this alarm, a failover occurs.

2 The failed SPROC becomes the standby SPROC. If the problem is not resolved in the reset, the
standby SPROC continues to fail. In this case, replace the card. 8
Refer to the CDMA2000 1X AN & SDU FRU manual for details.

APR 2004 SDU Troubleshooting 8-17


Troubleshooting ISB/SPROC Alarms continued

2826740 Internal SCTP Stack Failure


An internal software failure has occurred on the SPROC in the
application controlling the implicated link. The software was unable to
bind to the onboard communications protocol software.
Perform the following procedure to resolve this alarm.

Table 8-16: SDU Alarm 2826740:


Internal SCTP Stack Failure Troubleshooting Procedure
Step Action
1 The system disables the affected MMSDFRALINK, or MMPCFRALINK. To clear the alarm, it is
necessary to delete the affected link type from the OMCR and reprovision in an attempt to bring it
back into service.
For details, refer to the Cellular System Administration Guide and System Command Reference.

NOTE
For details, refer to the Cellular System Administration Guide and System Command Reference.

2826750 IP Failure
An internal software failure occurred in the protocol stack on the
SPROC in the onboard communications protocol software.

Perform the procedure in Table 8-15 to resolve this alarm.

2826760 Multicast Failure


An application on the SPROC was unable to join a multicast group.
Perform the procedure in Table 8-15 to resolve this alarm.

2826770 Capacity Greatly Reduced


Call capacity has been greatly reduced
Perform the following procedure to resolve this alarm.

8 Table 8-17: SDU Alarm 2826770:


Capacity Greatly Reduced Troubleshooting Procedure
Step Action
1 Check the status and alarms for all SDF boards, and correct problems as indicated for those alarms.
Resolving the SDF problems should restore the capacity.

8-18 SDU Troubleshooting APR 2004


Troubleshooting ISB/SPROC Alarms continued

2826780 Capacity Reduced


Call capacity was reduced.
Perform the procedure in Table 8-17 to resolve this alarm.

2826830 Application Platform Failure


The software associated with the device encountered an error in
initializing.
Perform the procedure in Table 8-14 to resolve this alarm.

2826840 Application Platform Failure


The software associated with the SPROC encountered a failure in
initializing.
Perform the procedure in Table 8-15 to resolve this alarm.

2826850 Application Platform Failure


A memory allocation failure occurred.
Perform the following procedure to resolve this alarm.

Table 8-18: SDU Alarm 2826850:


Application Platform Failure Troubleshooting Procedure
Step Action
1 To clear the alarm, it is necessary to lock, delete, and reprovision the BPP board to attempt bringing
it back into service. Refer to the Cellular System Administration Guide and System Command
Reference for details.

2826870 Incomplete Dataload


Some or all of the objects that the SDU requires for initialization are not
present.
Perform the following procedure to resolve this alarm.

Table 8-19: SDU Alarm 2826870: 8


Incomplete Dataload Troubleshooting Procedure
Step Action
1 The SDU is disabled.
To clear the alarm, regenerate the objects for download. This can be accomplished with the Generate
SDU Data File CLI command at the OMCR. Refer to Table 10-1, step 2.

APR 2004 SDU Troubleshooting 8-19


Troubleshooting ISB/SPROC Alarms continued

2826880 All Active Calls Released


A transient error caused all active calls to be released.
No action is necessary. Calls that originate after the alarm are
unaffected.

SPROC Communication
Alarms

2826500 Communication Failure, Lost Connection with MM


The connection at the protocollevel between the MM and SDU could
not be established or was dropped, although Access Node connectivity
is intact.
Perform the following procedure to resolve this alarm.

Table 8-20: SDU Alarm 2826500:


Communication Failure Lost Connection with the MM Troubleshooting Procedure
Step Action
1 No action necessary. The system will take the link out of service. The MM will try to reestablish the
link. When the link is reestablished, the alarm will clear.
NOTE
The following link types exist between the SDU and the MM: MMSDFRALINK (for SDU SDF
resource allocation) and MMPCFRALINK (for SDU PCF resource allocation). For details on
configuring and displaying the status of these links, refer to the Cellular System Administration Guide
and/or the System Command Reference.

2 If other links are failing locally, a problem may exist with the network connection or the ISB. In this
case, check the ISB for alarms and check the farend MM for failures.

2826510 Communication Failure, Lost Communication with


the MM
Communication at the applicationlevel between the MM and SDU
could not be established or was dropped, although Access Node
connectivity is intact.
8
Perform the Table 8-20 procedure to resolve this alarm.

8-20 SDU Troubleshooting APR 2004


Troubleshooting ISB/SPROC Alarms continued

2826520 Communication Failure, Link Disabled


This alarm indicates that the MM has reported that the link is disabled.
Perform the following procedure to resolve this alarm.

Table 8-21: SDU Alarm 2826520:


Communication Failure Link Disabled Troubleshooting Procedure
Step Action
1 No action is necessary. The system will take the link out of service. The MM will try to reestablish
the link. When the link is reestablished, the alarm will clear.
NOTE
The following link types exist between the SDU and the MM: MMSDFRALINK (for SDU SDF
resource allocation) and MMPCFRALINK (for SDU PCF resource allocation). For details on
configuring and displaying the status of these links, refer to the Cellular System Administration Guide
and/or the System Command Reference.

2 Check the farend MM for lock or failure status. Refer to the Cellular System Administration Guide
and System Command Reference for details on displaying the status of the MM.

2826530 Communication Failure, Lost Communication with


the PDSN
Communication between the SDU and PDSN could not be established or
has been lost. This alarm implicates the link to the PDSN.
Perform the following procedure to resolve this alarm.

Table 8-22: SDU Alarm 2826530:


Communication Failure Lost Communication with the PDSN Troubleshooting Procedure
Step Action
1 Wait for the alarm to clear as the system automatically attempts to reestablish the SDUPDSNLINK.
The alarm will clear when the SDU reestablishes the SDUPDSNLINK, the call processing link
between the SDU and PDSN.
From the OMCR, you can verify the status of the SDUPDSNLINK.
NOTE 8
The SDUPDSNLINK is the call processing link between the SDU and the PDSN. Until this link is
reestablished, the SDU cannot assign new calls to the PDSN. For more details on SDUPDSNLINK,
refer to the Cellular System Administration Guide and System Command Reference.

2 Check for other external communication failures.

3 Verify that the PDSN is functioning normally from the OMCR.


For more details on PDSN status (e.g. connectivity to the AN, the status of SDUPDSNLINK, etc.)
refer to the Cellular System Administration Guide and System Command Reference. For other aspects
of PDSN functionality, refer to the PDSN vendor documentation.

APR 2004 SDU Troubleshooting 8-21


Troubleshooting ISB/SPROC Alarms continued

Notes

8-22 SDU Troubleshooting APR 2004


Chapter 9: SDU Synchronization Failures

Table of Contents

SDU Synchronization Failure Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-1


Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-1
SDUOMCR Audit Synchronization failure . . . . . . . . . . . . . . . . . . . . 9-1
SDU fails to send sanity (heartbeat) message to OMCR . . . . . . . . . . . 9-1
SDU fails OMCR reset command . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-2

APR 2004 SDU Troubleshooting


Table of Contents continued

Notes

SDU Troubleshooting APR 2004


SDU Synchronization Failure Scenarios

Introduction
This chapter contains SDU synchronization failure scenarios and the
applicable recovery actions.

SDUOMCR Audit
Synchronization failure

Symptom
Any command from the OMCR to the SDU fails.

Possible Cause
The SDU is not in a state where it can synchronize with OMCR (or the
SDU is not connected to the OMCR).

Recovery Action
To verify that the SDU and OMCR are not in sync, type
snmpConnStateShow at the SPROC command line.

IMPORTANT

* Logging into the SPROC should be performed during a


scheduled maintenance window to avoid system disruption.

At the OMCR, initiate the audit command to the SPROC. Refer to the
System Command Reference for details.

SDU fails to send sanity


(heartbeat) message to OMCR

Symptom
The SDU fails to send the sanity (heartbeat) message to the OMCR
every two minutes.

NOTE
The OMCR monitors heartbeat messages to verify that it
is still in contact with the device.

Possible Cause
9
Connectivity does not exist between the SDU and the OMCR.

Recovery Action
If the OMCR is not receiving heartbeat messages from the SDU,
perform the following procedure.
APR 2004 SDU Troubleshooting 9-1
SDU Synchronization Failure Scenarios continued

Table 9-1: Troubleshoot SDU Heartbeat Failure


Step Action
1 Log in to the SPROC and enter:
snmpConnStateShow
The command output will indicate whether the SPROC is
communicating with the OMCR.
* IMPORTANT
Logging into the SPROC should be performed during a
scheduled maintenance window to avoid system disruption.
2 If step 1 indicates that the SPROC is not communicating with
the OMCR, enter the ping command and note the results:
S Ping the SPROC from the OMCR;
S Ping the OMCR from the SPROC.
3 If the results of steps 1 and 2 indicate that the SPROC and
OMCR are not communicating (and there is no existing
network link failure, for example, an AN link alarm) perform
a manual synchronization.
Log in to the OMCR CLI and enter the following command:
sync sdu# all
4 If the OMCR does not recover the SDU heartbeat, contact
CNRC.

SDU fails OMCR reset


command

Symptom
The OMCR cannot reset the SDU or the particular SDU card.

Possible Cause
The OMCR command does not reach the SDU.
This could be due to a transient error or a connectivity problem.

Recovery Action
Retry the OMCR command.
If it still doesnt work, determine whether there are any connectivity
9 problems (are there alarms for contact being lost with SDU).
Troubleshoot those issues based on the applicable alarm procedure.

9-2 SDU Troubleshooting APR 2004


Chapter 10: BPP Troubleshooting

Table of Contents

BPP Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-1


Overview of BPP Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-1
Verifying BPP Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-2
Verifying BPP Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-2
Troubleshooting BPP Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-3
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-3
Procedure for Regenerating Corrupted Software . . . . . . . . . . . . . . . . . 10-3
Troubleshooting BPP Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-4

10
APR 2004 SDU Troubleshooting
Table of Contents continued

Notes

10
SDU Troubleshooting APR 2004
BPP Troubleshooting

Overview of BPP
Troubleshooting
This chapter deals specifically with the procedures for troubleshooting
the BPP cards in the SDU.

10
APR 2004 SDU Troubleshooting 10-1
Verifying BPP Status

Verifying BPP Status


Whenever troubleshooting the BPP cards, it is recommended to verify
the current BPP status by checking LEDs and verifying board states
using the OMCR Display/Status commands.
For details, refer to chapter 6, General SDU Status.

10
10-2 SDU Troubleshooting APR 2004
Troubleshooting BPP Alarms

Overview
This section contains troubleshooting procedures for BPPrelated
alarms.
These alarms deal with the BPP hardware when implemented with
Packet Control Function (PCF) software and Selection Distribution
Function (SDF) software.
The following procedures address the alarm number, alarm name, and
procedure.

NOTE
In addition to maintaining troubleshooting logs, you may
also want to notify CNRC of problems if they frequently
reappear.

NOTE
For a more detailed description of each alarm, refer to
System Output Messages. Also, if you are not sure when
you need to resolve the problem, refer to the alarm
documentation (in the System Output Messages) for the
severity level.

Procedure for Regenerating


Corrupted Software
If an alarm troubleshooting procedure requires replacing invalid or
corrupt software, use the following procedure.

Table 10-1: Regenerate Corrupted Software


Step Action
1 Reinstall the software load as required by the alarm (alarm
dealing with invalid or corrupt software load or files).
2 After reinstalling software, it is necessary to regenerate the
SDU datafiles:
S From the OMCR, type generate sdu#
datafiles <Enter>
S From the OMCR, type activate sdu# <Enter>
NOTE
Refer to the Cellular System Administration Guide for
additional assistance.
10
APR 2004 SDU Troubleshooting 10-3
Troubleshooting BPP Alarms continued

Troubleshooting BPP Alarms

2826000 Application Failure


This alarm is reported when communication is lost with a critical, but
noncall processing task on a processor on the alarming BPP Card.
Perform the following procedure to resolve this alarm.

Table 10-2: SDU BPP Alarm 2826000:


Application Failure Troubleshooting Procedure
Step Action
1 The system automatically attempts to reset the card. The card will initialize if the automatic reset is
successful.
If the card initializes, no further action is necessary.
If the card fails to initialize, continue at the next step.
NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state. Also, there should be an attribute value change indicating that the
SDU capacity (either SDF or PCF) has increased by one BPPs worth of capacity.

2 Reset the hardware using the reset switch on the front panel of the specific BPP card.

3 If the hardware reset fails, lock and replace the card using the procedures contained in the CDMA2000
1X AN & SDU FRU manual.

2826001 Application Error


This alarm indicates a nonfatal runtime software error.
Perform the following procedure to resolve this alarm.

Table 10-3: SDU BPP Alarm 2826001:


Application Error Troubleshooting Procedure
Step Action
1 Verify whether the software load is valid and complete. To do so, check that the current software load
is actually the one intended to be running, and that all of the pertinent software files are present. Refer
to the SDU Software Installation documentation (in the Release Binder) for details.

2 If the software load is corrupt, replace it with a valid load and codeload the card with the new
software. Refer to the SDU Software Installation documentation (in the Release Binder) for details.

3 If the software load is valid, reset the BPP card.

10
10-4 SDU Troubleshooting APR 2004
Troubleshooting BPP Alarms continued

2826010 Call Processing Failure


This alarm is reported when communication is lost with a Call
Processing software program on a processor on a BPP Card.
Perform the following procedure to resolve this alarm.

Table 10-4: SDU BPP Alarm 2826010:


Call processing Failure Troubleshooting Procedure
Step Action
1 The system automatically attempts to reset the card. The card will initialize if the automatic reset is
successful.
If the card initializes, no further action is necessary.
If the card fails to initialize, continue at the next step.
NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state. Also, there should be an attribute value change indicating that the
SDU capacity (either SDF or PCF) has increased by one BPPs worth of capacity.

2 Reset the hardware using the reset switch on the front panel of the specific BPP card.

3 If the hardware reset fails, lock and replace the card using the procedures contained in the CDMA2000
1X AN & SDU FRU manual.

2826011 Call Processing Error


This alarm indicates an error in a call processing task on a processor. It
is most likely due to a software error.
Perform the following procedure to resolve this alarm.

Table 10-5: SDU BPP Alarm 2826011:


Call Processing Error Troubleshooting Procedure
Step Action
NOTE
As this alarm has minimal impact, the reset can be delayed almost indefinitely.

1 If the alarm does not clear, reset the card.

10
APR 2004 SDU Troubleshooting 10-5
Troubleshooting BPP Alarms continued

2826020 General Software Failure


This alarm is reported when a software program on the CPU of a BPP
Card fails.
Perform the following procedure to resolve this alarm.

Table 10-6: SDU BPP Alarm 2826020:


Failure Troubleshooting Procedure
Step Action
1 The system automatically attempts to reset the card. The card will initialize if the automatic reset is
successful.
If the card initializes, no further action is necessary.
If the card fails to initialize, continue at the next step.
NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state. Also, there should be an attribute value change indicating that the
SDU capacity (either SDF or PCF) has increased by one BPPs worth of capacity.

2 If the reset fails, reset the card using the reset switch on the front panel of the specific BPP card.

3 If the hardware reset fails, lock and replace the card using the procedures contained in the CDMA2000
1X AN & SDU FRU manual.

2826030 CPU Failure


This alarm indicates a processor failure. It is mostly likely due to a
hardware error.
Perform the following procedure to resolve this alarm.

Table 10-7: SDU BPP Alarm 2826030:


CPU Failure Troubleshooting Procedure
Step Action
1 The system automatically resets the card. No action is necessary, unless the card fails to initialize. If
the card fails to initialize, attempt a hardware reset using the reset switch on the frontpanel of the
BPP card.

2 If the hardware reset fails, lock and replace the card. Refer to the CDMA2000 1X AN & SDU FRU for
details.

10
10-6 SDU Troubleshooting APR 2004
Troubleshooting BPP Alarms continued

2826040 Hardware Component Error


This alarm is reported when an onboard (nonDSP) component
experiences a Runtime error.
Perform the following procedure to resolve this alarm.

Table 10-8: SDU BPP Alarm 2826040:


Hardware Component Error Troubleshooting Procedure
Step Action
1 If the additional text in the alarm output indicates that the board should be replaced and returned,
replace the card and return it to Motorola. To replace the card, refer to the CDMA2000 1X AN & SDU
FRU manual.

2 If the additional text does not indicate that the board should be replaced and returned, reset the card.

3 If the card fails to reinitialize properly after a reset, or if the alarm continues to occur, the card should
be replaced.

2826050 Hardware Component Failure


This alarm is reported when an onboard (nonDSP) component
experiences a runtime failure.
Perform the following procedure to resolve this alarm.

Table 10-9: SDU BPP Alarm 2826050:


Hardware Component Failure Troubleshooting Procedure
Step Action
1 The system automatically attempts to reset the card. The card will initialize if the automatic reset is
successful.
If the card initializes, no further action is necessary.
If the card fails to initialize, continue at the next step.
NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state. Also, there should be an attribute value change indicating that the
SDU capacity (either SDF or PCF) has increased by one BPPs worth of capacity.

2 If the card fails to initialize, reset the hardware using the reset switch on the front panel of the specific
BPP card.

3 If the reset fails, or if the problem recurs several times, lock and replace the card using the procedures
contained in the CDMA2000 1X AN & SDU FRU manual.

10
APR 2004 SDU Troubleshooting 10-7
Troubleshooting BPP Alarms continued

2826060 Partial Call Processing Capacity Loss


This alarm is reported when one or more DSPs on the alarming BPP
Card are unavailable.
This alarm indicates that some of the call processing capacity of the
card has been lost. The unavailability may be due to a failure of a
callprocessing component or of another component.
Perform the following procedure to resolve this alarm.

Table 10-10: SDU BPP Alarm 2826060:


Partial Call Processing Capacity Loss Troubleshooting Procedure
Step Action
1 Reset the card at the earliest opportunity.
NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state. Also, there should be an attribute value change indicating that the
SDU capacity (either SDF or PCF) has increased by one BPPs worth of capacity.

2 If the reset fails, lock and replace the card using the procedures contained in the CDMA2000 1X AN &
SDU FRU manual.

2826070 Full Call Processing Capacity Loss


This alarm is reported when all DSPs are unavailable.
This alarm indicates that all of the call processing capacity of the card
has been lost.
Perform the following procedure to resolve this alarm.

Table 10-11: SDU BPP Alarm 2826070:


Full Call Processing Capacity Loss Troubleshooting Procedure
Step Action
1 The system automatically resets the card. The card should be replaced if the automatic reset does not
result in a complete recovery.
To replace the card, refer to the CDMA 2000 1X AN & SDU FRU manual.

NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state. Also, there should be an attribute value change indicating that the
SDU capacity (either SDF or PCF) has increased by one BPPs worth of capacity.

10
10-8 SDU Troubleshooting APR 2004
Troubleshooting BPP Alarms continued

2826080 Switch Processing Error


This alarm is reported when the onboard switch experiences a nonfatal
error.
Perform the following procedure to resolve this alarm.
No immediate action is necessary. If the problem persists, follow the
procedure below. Unless other symptoms (for instance, high percentage
of dropped calls on the card) indicate a serious problem, the action may
be performed whenever it is convenient.

Table 10-12: SDU BPP Alarm 2826080:


Switch Processing Error Troubleshooting Procedure
Step Action
1 If the problem persists, reset the card.

2 If a reset does not resolve the problem, replace the card.

2826090 ISB Link Loss


This alarm indicates a loss of the link with an ISB. The ISB is named in
additional text in the alarm output.
Perform the following procedure to resolve this alarm.

Table 10-13: SDU BPP Alarm 2826090:


ISB Link Loss Troubleshooting Procedure
Step Action
1 If there are any ISBtype alarms occurring concurrently with this particular alarm, address the ISB
alarms first. Resolving the ISB problems should also resolve the BPP problems triggering this alarm.

2 If there are no ISB alarms, but other cards are also reporting link alarms, the ISB is probably the
source of the problem. Reset the ISB at a convenient time. When the ISB recovers, the BPP alarm
should clear.

3 If the ISB and other cards appear to be fine, or if resolving the ISB problems does not clear the alarm,
reset the BPP at a convenient time. If the problem remains, replace the BPP.

10
APR 2004 SDU Troubleshooting 10-9
Troubleshooting BPP Alarms continued

2826100 File Access Error


This alarm is reported when an attempt to access a file located on the
alarming BPP fails.
Perform the following procedure to resolve this alarm.

Table 10-14: SDU BPP Alarm 2826100:


File Access Error Troubleshooting Procedure
Step Action
1 Reset the card.

2 If the problem persists, replace the card.

2826110 High Temperature


This alarm indicates that the temperature on the card has exceeded the
lower of two hightemperature thresholds.
Perform the following procedure to resolve this alarm.

Table 10-15: SDU BPP Alarm 2826110:


High Temperature Troubleshooting Procedure
Step Action
1 Determine whether the problem is a general SDU system problem or whether it is specific to just one
card.
The trouble is local to the one card if all of the following cases are true:
S the fans are working properly
S the air flow through the SDU cage is satisfactory
S the general environmental conditions around the frame are acceptable
S no other cards in the SDU frame are reporting temperaturerelated alarms

2 If the problem is specific to the one card, reset the card. If the problem persists, replace the card.

3 If the problem is not specific to the one card, it is necessary to assess and correct the general
environmental conditions (e.g. fan, airflow) surrounding the SDU frame. When the environmental
problem has been corrected, the alarm will clear.
The environmental conditions must meet the specifications outlined for the SDU frame in the
CDMA2000 1X RAN Hardware Installation manual.

4 If a card reports high temperature for an extended time and later exhibits unusual behavior, the card
may have been damaged and should be replaced.

10
10-10 SDU Troubleshooting APR 2004
Troubleshooting BPP Alarms continued

2826111 Very High Temperature


This alarm indicates that the temperature on the card has exceeded the
higher of two hightemperature thresholds.
Perform the following procedure to resolve this alarm.

Table 10-16: SDU BPP Alarm 2826111:


Very High Temperature Troubleshooting Procedure
Step Action
1 Follow the procedure in Table 10-15.

2826120 Change Request Error


This alarm is reported when the alarming BPP could not process a
Change Request. This alarm is extremely unlikely and is included here
for reference purposes only.
These alarms clear after the operator resets the card. Note that for all
scenarios, reset refers to a hardware reset (accomplished using the
frontpanel reset button on the alarming card).

2826130 Change Request Failure


This alarm is reported when the alarming BPP could not process a
Change Request. This alarm is extremely unlikely and is included here
for reference purposes only.
These alarms clear after the operator resets the card. Note that for all
scenarios, reset refers to a hardware reset (accomplished using the
frontpanel reset button on the alarming card).

2826140 CPU Software Initialization Error


This alarm is reported when a software program on the Bearer Payload
Processor (BPP) Card failed to properly initialize.
Perform the following procedure to resolve this alarm.

Table 10-17: SDU BPP Alarm 2826140:


CPU Software Initialization Error Troubleshooting Procedure
Step Action
1 If the alarm indicates Waiting for Personality, assign either a PCF or an SDF personality to the
BPP card. Refer to the Cellular System Administration Guide for details on assigning PCF or SDF
functionality to a BPP card.
Otherwise, skip to the next step.

2 Refer to Table 10-3.


10
APR 2004 SDU Troubleshooting 10-11
Troubleshooting BPP Alarms continued

2826150 CPU Software Initialization Failure


This alarm is reported when a software program on the BPP card failed
to properly initialize.
Perform the following procedure to resolve this alarm.

Table 10-18: SDU BPP Alarm 2826150:


CPU Software Initialization Failure Troubleshooting Procedure
Step Action
1 The system automatically resets the card. The alarm will clear after the card successfully initializes.

2 Verify that the software load is valid and complete. To do so, check that the current software load is
actually the one intended to be running, and that all of the pertinent software files are present. If the
software load is corrupt, replace it with a valid load and codeload the card with the new software.
NOTE
Refer to the SDU Software Installation documentation (in the Release Binder) for details.

2827801 through 27802, Packet Bus Connection


This alarm is reported when a LAN Error or Loss of Frame condition
causes a Packet Bus Connection Fault on the identified Packet Bus
Connection (1 or 2). The last two digits of the alarm number identify the
connection number. The devices that could cause this alarm are: Bearer
Payload Processor (BPP) Card and Cabinet and Customer Alarm (CCA)
Card.
Perform the following procedure to resolve this alarm.

Table 10-19: SDU Communications Alarms 2821801 and 2821802:


Communication Fault, Packet Bus Connection no. <12> Troubleshooting Procedure
Step Action
1 Physically check the SDU card (BPP or CCA) specified in the alarm output. Determine whether it is
properly connected into the SDU backplane.
Refer to the CDMA2000 1X RAN Hardware Installation manual for details on card placement and
connections.

2 If it is not a card connectivity problem, then it is necessary to replace the FRU identified in the alarm.
Refer to the CDMA2000 1X AN & SDU FRU manual for details.

10
10-12 SDU Troubleshooting APR 2004
11

Chapter 11: CCA Troubleshooting

Table of Contents

CCA Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-1


Overview of CCA Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-1
Verifying CCA Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-2
Verifying CCA Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-2
Troubleshooting CCA Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-3
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-3
Troubleshooting CCA Processing Faults . . . . . . . . . . . . . . . . . . . . . . . . 11-3
Troubleshooting CCA Equipment Faults . . . . . . . . . . . . . . . . . . . . . . . 11-4
Troubleshooting CCA Environmental Faults . . . . . . . . . . . . . . . . . . . . 11-7
Troubleshooting CCA Communication Faults . . . . . . . . . . . . . . . . . . . 11-11

APR 2004 SDU Troubleshooting


11 Table of Contents continued

Notes

SDU Troubleshooting APR 2004


CCA Troubleshooting 11

Overview of CCA
Troubleshooting
This chapter deals specifically with the procedures for troubleshooting
the CCA cards in the SDU.

APR 2004 SDU Troubleshooting 11-1


11
Verifying CCA Status

Verifying CCA Status


Whenever troubleshooting the CCA cards, it is recommended to verify
the current CCA status by checking LEDs and verifying board states
using the OMCR Display/Status commands.
For details, refer to chapter 6, General SDU Status.

11-2 SDU Troubleshooting APR 2004


Troubleshooting CCA Alarms 11

Overview
This section contains troubleshooting procedures for CCArelated
alarms.
The following procedures address the alarm number, alarm name, and
procedure.

NOTE
In addition to maintaining troubleshooting logs, you may
also want to notify CNRC of problems if they frequently
reappear.

NOTE
For a more detailed description of each alarm, refer to
System Output Messages. Also, if you are not sure when
you need to resolve the problem, refer to the alarm
documentation (in the System Output Messages) for the
severity level.

Troubleshooting CCA
Processing Faults

2823380 Illegal Configuration Change


This alarm is reported when corrupt data causes an illegal configuration
change. Device List: CCA, Fan Tray, PSM DC
Perform the following procedure to resolve this alarm.

Table 11-1: SDU Processing Error Alarm 2823380:


Processing Fault, Illegal Configuration Change Troubleshooting Procedure
Step Action
1 If this alarm occurs, contact CNRC.

APR 2004 SDU Troubleshooting 11-3


11 Troubleshooting CCA Alarms continued

2823410 Initialization Failure


This alarm indicates an Invalid Configuration Data Loaded fault due to a
Configuration or Customization error. Device List: cabinetSensor,
customerSensor, controlRelay
Perform the following procedure to resolve this alarm.

Table 11-2: SDU Processing Error Alarm 2823410:


Processing Fault, Initialization Failure Troubleshooting Procedure
Step Action
1 If this alarm occurs, contact CNRC.

Troubleshooting CCA
Equipment Faults

2823670 Device Removal Detected


This alarm is reported when a monitored Fan or Power Supply Module is
removed from the system.
Perform the following procedure to resolve this alarm.

Table 11-3: SDU Equipment Alarm 2823670:


Equipment Fault, Device Removal Detected Troubleshooting Procedure
Step Action
1 Verify that maintenance was being performed on the alarmed device. If not, the device requires
replacement. Refer to the CDMA2000 1X AN & SDU FRU manual.

2 If maintenance was being performed, either a fan tray or power supply module was removed from the
system. The specified device should be placed back into the system.
NOTE
When a device is removed from the system wait 10 seconds before replacement to allow for software
adjustments needed for EID reread (and other internal adjustments) upon reinsertion.

11-4 SDU Troubleshooting APR 2004


Troubleshooting CCA Alarms continued
11

2823690 Hardware Fault


This alarm is reported when a hardware fault occurs due to one or more
of the following reasons:
S Equipment malfunction
S Multiplexer problem
S I/O Device failure
Device List: CCA, ISB, SDUFANTRAY, SDUPSMDCDC.

Perform the following procedure to resolve this alarm.

Table 11-4: SDU Equipment Alarm 2823690:


Equipment Fault, Hardware Fault Troubleshooting Procedure
Step Action
1 Replace the FRU specified in the alarm output. Refer to the CDMA2000 1X AN & SDU FRU manual
for details.

2 If this alarm is a fan tray alarm, the fan tray alarm indicates a problem with the fan tray hardware
has occurred. When this alarm condition is detected on any fan all other fan trays in the system will
boost until the alarm condition clears.
Causes/debugging:
S Something is lodged in the fan, the voltage has dropped, or the motor is bad, causing the rotation of
the fan blades to stop or slow down.
S Fan alarms are produced by faulty fan units which should be replaced.
S The automatic boost of the fans may place the rotation speed within the alarm threshold which in
turn will clear the alarm. The fans will then unboost and the alarm will occur again creating a cycle
or bounce condition on the alarm and fan boost.

3 If this alarm is a power supply controller board alarm, this indicates a hardware failure and could be
caused by an overstress condition. Replace the hardware since it is not functional.

APR 2004 SDU Troubleshooting 11-5


11 Troubleshooting CCA Alarms continued

2823790 Subcomponent Fault


This alarm is reported when a subcomponent fails due to equipment
malfunction.
Perform the following procedure to resolve this alarm.

Table 11-5: SDU Equipment Alarm 2823790:


Equipment Fault, Subcomponent Fault Troubleshooting Procedure
Step Action
1 Replace the FRU specified in the alarm output. Refer to the CDMA2000 1X AN & SDU FRU manual
for details.

2 If it is a power supply fan failure alarm, check the following:


S Check the power supply fan rotor for blockage.
S Check the power supply air intake.
S Check the power supply drive signal to the fan(s), they could be low or lost.
S If this alarm occurs the power supply will overheat thereby generating the power supply high
temperature and very high temperature alarms and shutdown.

2823900 Power Supply Input Circuit Fault


A fault in the power supply input circuit due to a low bus voltage or a
blown fuse.
A power supply fuse alarm indicates all other power supply
alarms have been detected at the same time and the power supply
module is still present in the system.

Table 11-6: SDU Environmental Alarm 2823900:


Environmental Fault, Power Supply Input Circuit Fault Troubleshooting Procedure
Step Action
1 Check the input power to the frame, and the circuit breaker to the PSM. Trace the fuse to make sure
that it is connected.

2 Check the alignment and replace the connecting module if necessary.

11-6 SDU Troubleshooting APR 2004


Troubleshooting CCA Alarms continued
11

Troubleshooting CCA
Environmental Faults

2824400 AC Mains Under Volt Threshold Exceeded


This alarm is reported when the number of Undervoltage incidents
exceeds the AC Mains Under Volt Threshold value. A potential Power
Supply Module failure or a service disruption is possible.
A power supply voltage alarm indicates that the power into the
power supply module (AC mains, 48V, etc.) has been interrupted.

Perform the following procedure to resolve this alarm.

Table 11-7: SDU Environmental Alarm 2824400:


Environmental Fault, AC Mains Under Volt Threshold Exceeded Troubleshooting Procedure
Step Action
1 Verify that the input voltage is within the specified range for the SDU. (Refer to the CDMA2000 1X
SDU ATP manual for details on the acceptable input voltage range for the SDU).

2 Carefully check possible causes for this alarm. This alarm is indeterminate and could be caused by
power supply hardware, the AC filter, etc.

3 If the power supply voltage alarm condition also exists, this may indicate that the power supply
modules have been operating on batteries and the batteries are near being fully discharged.

4 If the power supply voltage alarm condition does not exist, this alarm indicates an unknown hardware
failure, so replace the power supply module.

2824440 Bus Under Voltage Threshold Exceeded


This alarm is reported when the number of Bus Undervoltage incidents
exceeds the Bus Under Voltage Threshold value. A potential Power
Supply Module failure or a service disruption is possible.
Perform the following procedure to resolve this alarm.

Table 11-8: SDU Environmental Alarm 2824440:


Environmental Fault, Bus Under Voltage Threshold Exceeded Troubleshooting Procedure
Step Action
1 Evaluate and correct the power situation by verifying and ensuring that the input voltage is within the
specified range for the SDU.
Refer to the CDMA2000 1X SDU ATP manual for details on the acceptable input voltage range for the
SDU.

APR 2004 SDU Troubleshooting 11-7


11 Troubleshooting CCA Alarms continued

2824460 Cabinet Sensor Threshold Exceeded


This alarm is reported when the number of Cabinet Sensor incidents
exceeds the Cabinet Sensor Threshold value.
Perform the following procedure to resolve this alarm.

Table 11-9: SDU Environmental Alarm 2824460:


Environmental Fault, Cabinet Sensor Threshold Exceeded Troubleshooting Procedure
Step Action
1 Evaluate and correct the environmental condition cited by the alarm.

NOTE
Cabinet sensor 1 alarm occurs due to the cabinet / cage door being open.
Cabinet sensors 2 5 have not yet been defined.

2824480 External Sensor Threshold Exceeded


This alarm is reported when the number of External Sensor incidents
exceeds the External Sensor Threshold value.
Perform the following procedure to resolve this alarm.

Table 11-10: SDU Environmental Alarm 2824480:


Environmental Fault, External Sensor Threshold Exceeded Troubleshooting Procedure
Step Action
1 Evaluate and correct the situation being alarmed by the customer defined sensors.

NOTE
Since the sensors are customer defined, the troubleshooting action will depend upon the specific
implementation. To troubleshoot this problem it is necessary to know how the sensors were wired for
the particular installation.

11-8 SDU Troubleshooting APR 2004


Troubleshooting CCA Alarms continued
11

2824700 Temperature High Threshold Exceeded


This alarm is reported when the number of high temperature incidents
exceeds the Temperature High Threshold value. This alarm indicates that
the Power Supply Module (PSM) has reached a minor over temperature
threshold. This alarm represents an early warning of potential
overheating in the power supplies. This alarm also indicates that the
lower of the two high temperature thresholds has been crossed.
Perform the following procedure to resolve this alarm.

Table 11-11: SDU Environmental Alarm 2824700:


Environmental Fault, Temperature High Threshold Exceeded Troubleshooting Procedure
Step Action
1 Correct this problem by cooling the SDU environment or by removing dust from the fan input.
Check the following:
S Check ambient temperature conditions.
S Check power supply and cage ventilation.
S Check internal power supply fans.
S Check fan tray modules.

NOTE
When working with hardware follow all safety precautions specified in the CDMA2000 1X RAN Data
Hardware Installation and CDMA2000 1X AN & SDU FRU manuals.

APR 2004 SDU Troubleshooting 11-9


11 Troubleshooting CCA Alarms continued

2824740 Temperature Very High Threshold Exceeded


This alarm is reported when the number of high temperature incidents
exceeds the Very Temperature High Threshold value.This alarm indicates
that the Power Supply Module (PSM) has overheated and shut down for
thermal overload protection. This is a critical condition because the
other power supplies are now carrying the load and are at high risk for
shutting down as well. This is true unless something is wrong, such as
a power supply fan failure, which is also a minor alarm.
Perform the following procedure to resolve this alarm.

Table 11-12: SDU Environmental Alarm 2824740:


Environmental Fault, Temperature Very High Threshold Exceeded Troubleshooting Procedure
Step Action
1 Evaluate and correct the environmental condition cited by the alarm:
S If this alarm is coupled with fan failure alarms, correcting the fan problem should resolve this issue.
S If this alarm is occurring without any fan failure indications, evaluate and correct the temperature
conditions which may be causing the power supplies to overheat.

2 To evaluate temperaturerelated conditions, check the following:


S Check ambient temperature conditions.
S Check power supply and cage ventilation.
S Check internal power supply fans.
S Check fan tray modules.

11-10 SDU Troubleshooting APR 2004


Troubleshooting CCA Alarms continued
11

Troubleshooting CCA
Communication Faults

2827351 through 27362, Electronic ID 1wire Bus Failure

This alarm is reported when a Communications Subsystem Failure


causes an Electronic ID 1Wire Bus Connection Fault on the identified
Bus Connection (1 through 12). The last two digits of the Alarm Number
identify the Connection Number.

The devices that could cause this alarm include:


S DC/DC Power Supply
Perform the following procedure to resolve this alarm.

Table 11-13: SDU Communications Alarms 2821351 through 2821362:


Communication Fault, Electronic ID 1wire Bus Connection no. <112> Troubleshooting Procedure
Step Action
1 Reseat the PSM. This alarm will occur if the PSM is either missing or not seated correctly.
NOTE
Refer to the appropriate documentation for installing/seating the PSM.

2 If you reseated the PSM and the alarm still occurs, contact CNRC.

2827801 through 27802, Packet Bus Connection

This alarm is reported when a LAN Error or Loss of Frame condition


causes a Packet Bus Connection Fault on the identified Packet Bus
Connection (1 or 2). The last two digits of the alarm number identify the
connection number. The devices that could cause this alarm are: Bearer
Payload Processor (BPP) Card and Cabinet and Customer Alarm (CCA)
Card.

Perform the following procedure to resolve this alarm.

Table 11-14: SDU Communications Alarms 2821801 and 2821802:


Communication Fault, Packet Bus Connection no. <12> Troubleshooting Procedure
Step Action
1 Physically check the SDU card (BPP or CCA) specified in the alarm output. Determine whether it is
properly connected into the SDU backplane.
Refer to the CDMA2000 1X RAN Hardware Installation manual for details on card placement and
connections.

2 If it is not a card connectivity problem, then it is necessary to replace the FRU identified in the alarm.
Refer to the CDMA2000 1X AN & SDU FRU manual for details.

APR 2004 SDU Troubleshooting 11-11


11 Troubleshooting CCA Alarms continued

Notes

11-12 SDU Troubleshooting APR 2004


Chapter 12: OMCR Failure Scenarios
12

Table of Contents

OMCR Failure Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-1


Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-1
OMCR Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-1

APR 2004 SDU Troubleshooting


Table of Contents continued

Notes

12

SDU Troubleshooting APR 2004


OMCR Failure Scenarios

Overview
This section provides a listing of OMCR alarms that apply for multiple 12
network elements but also apply to the SDU.
Refer to the System Output Messages manual for the alarm description
and operator action.

OMCR Alarms
The following list describes related OMCR alarms:
S ALARM: 2878 Platform Out of Service
S ALARM: 2879 Platform Communication Failure with OMCR
S ALARM: 28102 SNMP MIB Support Unavailable for Committed
Version
S ALARM:28103 Management Synchronization Failed with NE
S ALARM: 306168 OMCR /SDU Database Mismatch
S ALARM: 306170 OMCR GCMGR Queue to NE Inactive
S ALARM:286820 Synchronization Failed: Unexpected Response
from Network Element
S ALARM:286821 Synchronization Failed: No Response from
Network Element
S ALARM 28 6851 Config File Corrupted

APR 2004 SDU Troubleshooting 12-1


OMCR Failure Scenarios continued

Notes

12

12-2 SDU Troubleshooting APR 2004


Chapter 13: System Initialization Problems

Table of Contents

13

System Initialization Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-1


Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-1
SPROC Boot Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-1
OMCR Commissioning Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-2

APR 2004 SDU Troubleshooting


Table of Contents continued

Notes

13

SDU Troubleshooting APR 2004


System Initialization Problems

Introduction
This chapter contains troubleshooting procedures for system
initialization problems.

SPROC Boot Problems

SPROC Fails to Receive DHCP from the ISB


13
Refer to chapter 8, section ISB unable to provide DHCP service to
SPROC.

SPROC Reboots Due to Bad NE_LIF.xml


If the software loaded on the SPROC does not agree with the
NE_LIF.xml file, then the SPROC will fail to initialize and will reset.
The following line will be displayed during init indicating a problem.
NEI : NSIF files missing or files not consistant
with LIF in /active/software.
NEI : Node is Not Potential Master based on
internal criteria[dd99220]
In this situation, the NE_LIF.xml file (located in the /active/software
directory) needs to be regenerated and downloaded to the SPROC. Note
that the NSIFsprocsproc file should NEVER be deleted from the
NE_LIF file.
Contact CNRC.

SPROC Component Factory Startup Fails


The following output indicates this SPROC error:
OMIX: palPDRSopen ERROR S_errno_EAGAIN for
/active/config/dynamic/baseNECDF.xml

OMIX: palPDRSseek ERROR S_errno_EAGAIN


This problem occurs due to a problem with permissions on the
.../data/pdrs directory. This directory needs permissions open to create a
file called safe. The component factory failure message is:
NEI : sending Data to NEIP
Component Factory started.
usage value is 0

hapua: onGoActive

availability value is 255


usage value is 1
Component Factory failed.
availability value is 255

APR 2004 SDU Troubleshooting 13-1


System Initialization Problems continued

Call CNRC to resolve this problem.

OMCR Commissioning
Problems

SDU Sync Command Fails


If the SDU sync command fails during commissioning, investigate the
following:
13 S Verify that you can ping the SPROC external IP address from the
OMCR.
S Verify the active version on the OMCR (disp sdu# version).
S Verify that no errors are seen on the SPROC. Use the following
command to display more debug information on the SPROC:
SetOAMPLogDebug(1)

IMPORTANT

* Logging into the SPROC should be performed during a


scheduled maintenance window to avoid system disruption.

13-2 SDU Troubleshooting APR 2004


Chapter 14: Call Processing Failures

Table of Contents

Call Processing Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-1


Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-1
Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-1
SPROC MM RA/RM Links Not Active . . . . . . . . . . . . . . . . . . . . . . 14-1
14
No PCF Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-1
Problems with SDF Resource Assignment . . . . . . . . . . . . . . . . . . . . . . 14-1
Problems with cBTS Proxy Request . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-1
Problems with Circuit IOS Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-2
A Interface Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-2

APR 2004 SDU Troubleshooting


Table of Contents continued

Notes

14

SDU Troubleshooting APR 2004


Call Processing Failures

Introduction
This chapter describes SDUrelated call processing failures.

Use
This section describes the basic subsections of call processing related to
a system with the SDU architecture. If either of these points fail, CFCs
and/or alarms will occur. In that case, follow the applicable
troubleshooting procedures for those CFCs/alarms. Some
troubleshooting actions specified here may also be used to supplement
those procedures.

14
SPROC MM RA/RM Links Not
Active
The following SPROC MM links must be active or else calls cannot be
established and CFCs and alarms will occur. If such CFCs and/or alarms
occur, follow the related troubleshooting procedures.
S MMSDFRALINK (used for SDF resource setup)
S MMPCFRALINK (used for PCF resource setup)

No PCF Resources
PCF resources must be available to establish new SDU 1X packet data
calls.
If a CFC indicates that there are no PCF resources available, check the
following (OMCR CLI commands can accomplish this):
S Verify that the PDSN exists.
S Verify that the PCF resources are INS.
S Verify that the PCF CPEs (Channel Processing Elements) are INS.
Each PCF card has a certain number of CPEs.
S Verify that the SDUPDSNLINK is INS.

Problems with SDF Resource


Assignment
SDF resources must be available to establish any SDU 1X calls. If there
is a problem with the SDF assignment portion of call setup, a CFC will
occur. Refer to the applicable CFC for the appropriate troubleshooting
procedure.

Problems with cBTS Proxy


Request
The XC PKTIF must receive the cBTS Proxy Request in order to set up
the call on that end. If this doesnt happen, a CFC will occur.
APR 2004 SDU Troubleshooting 14-1
Call Processing Failures continued

Problems with Circuit IOS


Traffic
If CFCs occur due to problems with sending/receiving Circuit IOS traffic
between devices, you can use the following methods to further inspect
the problem.

Sniffer on the Gigabit Link


Use a sniffer tool on the gigabit link to see if bidirectional traffic frames
can be seen.

Reverse Link
To determine the traffic being seen on the sniffer, examine the 4th byte
of the data field.
14
4th Byte (Frame Type):
00 = Idle
7E = Invalid Speech
01 20 = Valid Speech

Forward Link
5th Byte (Frame Type):
04 = 8th Rate Frame
Other = Valid speech frames will vary in rate

Look at XC KSW Port Connections


From the OMP, the port audit command can be used to make sure that
the desired connections are being made. This can be compared to the
TDM ports used by the MMS and the PSICE to verify the proper
connection is in place.

A Interface Problems
The A interfaces related to SDU are A8/A9 (between SDU SDF and
SDU PCF) and A10/A11 (between the SDU PCF and the PDSN).
The following steps must occur or else CFCs will occur and packet data
calls will not be established. In such a case, refer to the troubleshooting
procedure for the applicable CFC.
S A9 Setup A8 must be sent by SDF to PCF
S A9 Setup A8 must be received and seen by PCF
S PCF must send A11 Registration Request to PDSN
S PDSN must send an A11 Registration Reply

NOTE
For details on A interface call setup, refer to the
CDMA2000 1X RAN Functional Description.

14-2 SDU Troubleshooting APR 2004


Chapter 15: CFC Failures

Table of Contents

SDU CFC Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15-1


Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15-1
Common Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15-2
Troubleshooting CFC Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15-5

15

APR 2004 SDU Troubleshooting


Table of Contents continued

Notes

15

SDU Troubleshooting APR 2004


SDU CFC Troubleshooting

Overview
Introduction
This chapter contains CFC troubleshooting procedures for SDU.

NOTE
When displaying status data from the OMCR, it may be
helpful to log the data to a file, especially if you plan on
contacting CNRC.

OMCR Commands and Database Timers


This section refers to the use of OMCR commands as well as call
processing timers. For details on the commands and timers, refer to the
Cellular System Administration Guide and System Command Reference.

CFCs and Call Flow Context


When using tools to isolate call flow breakdowns, such as sniffer logs,
SMAP, Mobile DM, and callproc, an advanced knowledge of the call
flow context surrounding the particular CFC(s) may be helpful.
For example, if you are troubleshooting CFC 06, it may help to be
familiar with the call processing context in which CFC 06 occurs when
investigating the possible causes, such as what call processing messages
are sent and received in that particular segment of call setup.
The CDL/CFC Reference manual contains call flows with CFCs plotted
accordingly.
In addition, refer to the CDL/CFC Reference manual for more detailed
descriptions of the CFCs.

SMAP and Mobile DM


Many troubleshooting procedures deal with operation of SMAP and a
Mobile Diagnostic Monitor (DM) to collect call processing messages
and verify what messages were actually sent and received on each end of
the connection. For details on SMAP operations, and descriptions of the
messages types and fields that can be logged (what is received on the
reverse link, what is sent on the forward link), refer to the SMAP Control
GUI and SMAP Data Analyzer Operators Guides. For details on the
same capabilities for the Mobile DM, refer to your vendor
documentation.

APR 2004 SDU Troubleshooting 15-1


SDU CFC Troubleshooting continued

Common Procedures
The following procedures are common throughout this chapter. Refer to
this section when performing these procedures.

Viewing BPP Spel Logs

IMPORTANT

* Logging into the BPP should be performed during a


scheduled maintenance window to avoid system disruption.

To view errors and events logged by the BPP, use the spel <option>
command at the BPP command line.
The command format is spel <option> where <option> may be:
S celm: Change the Event Log filter settings
S cl: Clear event log
S cls: Change Logging State
S cmm: Change realtime MMI filter settings
S cfs: Change outgoing MMI filter settings
S le: Log event
S ler: Log error (Used internally by event logger)
S se: Show events
S sfe: Show filtered events
S ser: Show errors
S selm: Show current Event Log filter settings
S slogs: Show current logging states of MMI and EL
For example, to view events reported by the BPP:
spel se
For example, to view errors reported by the BPP:
spel ser

Collecting Sniffer Logs


Sniffer tools are optional aids for troubleshooting. Refer to the
documentation for the sniffer application for details.

15-2 SDU Troubleshooting APR 2004


SDU CFC Troubleshooting continued

Checking the Service Option Database


In certain troubleshooting procedures, it may be necessary to check the
service option database.
From the OMCR, use the following commands to check for service
option settings:
S display servoptcbsc#servopt# alt displays the
alternate service options associated with the specified Service Option.
S display servoptcbsc#servopt# soserv command
displays the Service Option parameters for the specified Service
Option. For example, it displays the values such as XcdrCkt and
TchCkt (used in troubleshooting CFCs).
S The following commands check for forward and reverse frame rates
(they also display other Mux settings such as Radio Configuration):
for 1X: display servoptcbsc#servopt# so1xcrc
for IS95A/B:
display servoptcbsc#servopt# so95abcrc
Corresponding edit commands are available to correct the settings if
necessary.

IMPORTANT

* Refer to the System Command Reference before changing


these settings, and for the correct ranges. For some
parameters, it is recommended to consult Motorola first
before making any changes.

SdfCpePrintStore
Use the SdfCpePrintStore command to determine whether calls are being
allocated across all CPEs (Channel Processing Elements) on the SDF
card.
This command will also identify the IP addresses of the individual
CPEs.
At the SPROC, enter: SdfCpePrintStore

IMPORTANT

* Logging into the SPROC should be performed during a


scheduled maintenance window to avoid system disruption.

APR 2004 SDU Troubleshooting 15-3


SDU CFC Troubleshooting continued

For example:
[SDUSPROC2]> SdfCpePrintStore
***** Printing Cpe Store for SDF ******
//
SDF Cpe Index = 0
status = INS
ipAddress = a01013d
capacity = 3200
availableBwu = 3200
usedBwu = 0
//
//
SDF Cpe Index = 1
status = INS
ipAddress = a01013e
capacity = 3200
availableBwu = 3200
usedBwu = 0
//

15-4 SDU Troubleshooting APR 2004


SDU CFC Troubleshooting continued

Troubleshooting CFC Failures

CFC=3 RF Layer 2 Failure


This CFC is pegged when a RF Layer 2 error is reported by a device.
There are many reasons that a RF Layer 2 error might be reported.

Causes:
Detailed Cause Description
0x1114 Layer 2 Failure Reported by SDU DSP
0x1115 Layer 2 Failure Reported by SDU L2
0x1119 RF Layer 2 Failure

Troubleshooting
S Dump the BPP SPEL Log

IMPORTANT

* Logging into the BPP should be performed during a


scheduled maintenance window to avoid system disruption.

CFC=4 RF Loss
This CFC is generated when the SDF CPE detects an RF loss condition.
RF Loss is defined by L3CC on the SDF receiving a Traffic Channel
State Change indication for Invalid Speech (erasure) frames from a valid
speech state. This will start the RF Loss Timer, if valid speech frames
are not received before the timer expires, the call will be torn down for
RF Loss.

Causes:
Detailed Cause Description
0x111A RF Loss

Troubleshooting:
RF Optimization may be necessary if RF Losses occur often.

CFC=5 No TCH Preamble Detected


This CFC is generated when the BTS tears down the call due to not
seeing any valid Circuit IOS frames on the forward link. Forward Ckt
IOS frames are generated by the SDF DSP after the SDU has received
the CDMA Channel Assign and completed the cBTS Proxy messaging
(for circuit BTS case) or completed the RF resource messaging (Packet
BTS).
The other cause could be for MccCpT1 timer expiring on the BTS, this
timer monitors for the mobile appearing on the traffic channel. The
Mobile moves from the paging channel to the traffic channel when it
receives the CDMA Channel Assign message from the BTS. This
message specifies what traffic channel (TCH) to move to.

APR 2004 SDU Troubleshooting 15-5


SDU CFC Troubleshooting continued

Causes:
Detailed Cause Description
0x111B No TCH Preamble Detected
0x2005 MccCpT1 Expired

Troubleshooting:
CFC 05 indicates either an RF environment problem or a possible
internal BTS problem. Use the following options to narrow down the
problem.
Problem is due to the RF environment:
S If possible, collect a mobile DM (Diagnostic Monitor) log to see what
information the mobile is seeing and what actions it is taking.
S Compare with an SMAP log to get the full picture of what is sent and
received on both ends.
S Messages not received/acknowledged on either side indicate RF
environment problems are occurring. For example, the BTS may not
have received the TCH Preamble frames from the mobile, or the
mobile may have not received the message from the BTS prompting
the transmission of TCH Preamble frames e.g. did not receive the
Channel Assignment/Extended Channel Assignment message.
Problem is due to the network:
S Collect a sniffer log to determine what bearer frames are present.
S Reset the BTS, force a conf load.

CFC=6 No Strau Synch


The bearer path connection between the XC/SDU and an assigned traffic
channel could not be established during call setup.

Causes:
Detailed Cause Description
0x111C No STRAU Synch

Troubleshooting:
S Collect a sniffer log to determine what bearer frames are present.
S Collect a sniffer log at multiple points to determine the source of
disconnect; from the Catalystt to the SDU ISB, and the SDU ISB to
the SDU BPP.

15-6 SDU Troubleshooting APR 2004


SDU CFC Troubleshooting continued

CFC=7 CP Timeout Awaiting MS Acquisition


The XC/SDU subsystem did not acquire the mobile (did not receive null
frames from the mobile) during call setup.

Causes:
Detailed Cause Description
0x1121 CP Timeout Awaiting MS Acquisition

Troubleshooting:
S If CFC 05 occurs, then CFC 07 will also occur. If this is the case,
instead troubleshoot CFC 05.
S If CFC 05 did not occur, and CFC 07 did occur, then there is probably
a disconnect:
Collect a sniffer log to determine what bearer frames are present.
Collect a sniffer log at multiple points to determine the source of
disconnect; from the Catalyst to the SDU ISB, and the SDU ISB to
the SDU BPP.

CFC=12 CPP Call Setup Timeout


This CFC is pegged when one of a number of various call setup timers
expires during call setup.

Causes:
Detailed Cause Description
0x114C SDF Call Setup Timeout
0x114D No Vocoder Connect Received at XC
0x114E No connect order received
0x154E SDF Assignment Timeout. No SDF Resource
Update is received at the SPROC before

Troubleshooting:
S Verify that the SDF Call Setup Timer is shorter than the Vocoder
Connect Timer at the XC before troubleshooting this failure, as this
can be a misleading failure cause.
S Dump some SPEL events on the BPP to see if any errors are being
reported by the SDF.

IMPORTANT

* Logging into the BPP should be performed during a


scheduled maintenance window to avoid system disruption.

CFC=13 CP Timeout Awaiting Service Option Ack


For P_REV_IN_USE=2 mobiles (800MHz cellular band), the XC/SDU
fails to receive an L2 ACK for the Service Option Response Order.
For P_REV_IN_USE=1 (1.9GHz PCS band), P_REV_IN_USE=3
(800MHz cellular band) and above mobiles, the CPP fails to receive an
L2 ACK or Service Connect Completion message.

APR 2004 SDU Troubleshooting 15-7


SDU CFC Troubleshooting continued

Causes:
Detailed Cause Description
0x1123 No service option ack received from MS

Troubleshooting:
S From a Mobile DM (Diagnostic Monitor) tool, determine if the MS
received the Service Option Response Order/Service Connect message
and is responding with a L2 Ack./Service Connect Completion
message.
S Compare with SMAP data to see what was actually sent and received
by the BTS.
S Check the value of the XcdrCkt. ServType in the Servopt database for
the specified service option.
S Check the value of the TchCkt. Serv Type in the Servopt database for
the specified service option.
S Check the value of the Sdf Service Connect Complete timer:
At the OMCR CLI: display sdusdf1 sdfsn
S Check the value of the Setup Action Time, SDF SN Setup A Timer.
S Determine if the SDU (SDF L3CC) is sending a Service Option
Response Order/Service Connect message. You can use SMAP to
monitor these messages.
S Dump some SPEL events on the BPP to determine if there are any
errors being reported by the BPP:
Check to see if Timer 23 (Sdf Service Connect Complete Timer) is
expiring. This cant be done on a system under call load.
Check that L2 received the Service Option Response Order/Service
Connect message. This cant be done on a system under load.
Check that the BTS received the Scap Service Configuration
Change Request message.

IMPORTANT

* Logging into the BPP should be performed during a


scheduled maintenance window to avoid system disruption.

15-8 SDU Troubleshooting APR 2004


SDU CFC Troubleshooting continued

CFC=14 Not enough Mobile Status Information


Received
This CFC is pegged when there is no response from the Mobile to the
Status Request message on expiration of the Sdf Status Response timer,
or if the Mobile rejects the Status Request message by sending an MS
Reject Order message.
Causes:

Detailed Cause Description


0x1125 Not enough status information received from Mobile

Troubleshooting:
S From a Mobile DM tool, determine if the Mobile received the Status
Request message at least 3 times.
S From the Mobile DM tool, determine if the Mobile responded to the
Status Request message with a Status Response/MS Reject Order
message.
S Also determine whether the BTS received the Status Response using
SMAP.
S Check the value of the Sdf Status Response timer:
At the OMCR CLI: display sdusdf1 sdfsn
S Dump some SPEL events on the BPP to determine if there are any
errors being reported by BPP.

IMPORTANT

* Logging into the BPP should be performed during a


scheduled maintenance window to avoid system disruption.

S Collect a BTS TCH Audit Log to determine if the BTS is getting any
erased frames in the Status Response message.
S Check the RF Calibration to determine if the RF environment can be
further improved.

APR 2004 SDU Troubleshooting 15-9


SDU CFC Troubleshooting continued

CFC=15 Negotiation Failure


This CFC is pegged when the Mobile and the SDU are unable to arrive
at a mutually acceptable service configuration. This CFC is pegged on
expiration of the Sdf Negotiation timer, or if the Mobile rejects the
Service Request/Response/Connect /Option Response Order message by
sending a MS Reject order message.

Causes:
Detailed Cause Description
0x1124 No mutual service option/configuration

Troubleshooting
S Verify the value of the Sdf Negotiation Timer.
S From a Mobile DM tool and SMAP determine if the BTS is sending
and the Mobile is receiving a Service Request/Response message.
S From the Mobile DM tool and SMAP, determine the response of the
Mobile to the Service Request/Response message and whether the
response is received by the BTS.
The Mobile can respond with a Service Response/Request or a MS
Reject order or send no response at all.
S From the Mobile DM tool determine if the Mobile is receiving a
Service Connect/Option Response Order message.
S From the Mobile DM tool and SMAP, determine the response of the
Mobile to the Service Connect /Option Response Order message and
whether the BTS receives the response.
S Using SMAP, check to see if the SDU (SDF L3CC) sends out a
Service Connect Message.
S In the servopt database, check the Alternative Service Option list of
the service option for which negotiation is taking place.
S In the servopt database, check the IS95 A/B and 1x Forward and
Reverse frame rates for the service option for which negotiation is
taking place.
S Using a sniffer, it is also possible to check to see if there are any
entries in the log which indicate the proposable Service Option List is
empty. Check the service option list supported by the Mobile by
examining the logging of the Status Response message.

15-10 SDU Troubleshooting APR 2004


SDU CFC Troubleshooting continued

CFC=35 Resource Allocation Timeout


This CFC is pegged when the MM believes that it has a valid connection
with the SDURA, but does not receive a resource response message
before the MM Resource Response timer expires.

Causes:
Detailed Cause Description
0x1148 No response from SDURA

Troubleshooting:
Collect a sniffer log to verify that the MM is sending resource requests
over SCTP.

CFC=36 No SDU Resources Available


This CFC is pegged when the resource response from the SDU indicates
that all available SDF resources are congested.

Causes:
Detailed Cause Description
0x114A No SDF Resources available

Troubleshooting:
S If there is a congestion alarm occurring for the SDU, troubleshoot the
alarm.
S At the OMCR, use SDUrelated STATUS and DISPLAY commands
to verify BPP usage.
S Check the OMCR event logs to verify that the SDFRG (SDF
Resource Group) is not reporting capacity changes.
S Check the resource allocation at the SPROC, is it close to the limit for
the SDFs?
S At the SPROC, run SdfCpePrintStore . Are calls being
allocated across all CPEs?

IMPORTANT

* Logging into the SPROC should be performed during a


scheduled maintenance window to avoid system disruption.

APR 2004 SDU Troubleshooting 15-11


SDU CFC Troubleshooting continued

CFC=52 Equipment Failure at BSC


This CFC is pegged as an indication of a general equipment failure
somewhere at the CBSC.

Causes:
Detailed Cause Description
0x0101 MM RM Link Failure
0x0102 MM RM Session ID Change
0x0103 MM RM Deregistration
0x0104 MM RM ReRegistration Failure
0x154F SDU CP Internal Failure
0x1550 SDU CPE Failure
0x1551 SDU BPP Failure
0x154B PSISIG Failure
0x154C PSICE Failure
0x154D PSI SDU Failure
0x1552 Backhaul Bearer Loss
0x1553 Network Bearer Loss

Troubleshooting:
S Determine device failure based on detailed cause
S Collect log info (OMP logs, event logs, etc) dependent upon specific
device failing
S Address any alarms that are occurring. Follow the troubleshooting
procedures for the appropriate alarms.

CFC=60 Error Between MSC and BSC

Troubleshooting:
S Look at MM callproc logs.
S Use CDLs to try to isolate the problem to a device.

CFC=61 Protocol Error Between MM and XC/SDU


MmCpT14 Timeout waiting for response from XC or SDU.

Causes:
Detailed Cause Description
NA No PSILINK in MM DB
NA No SDFRA in MM DB

Troubleshooting:
S Look at MM callproc logs.
S Use CDLs to try to isolate a device.

15-12 SDU Troubleshooting APR 2004


SDU CFC Troubleshooting continued

CFC=62 XC Detected Error

Troubleshooting:
S Look at MM callproc logs.
S Use CDLs to try to isolate a device.

CFC=80 MM Internal Error


This CFC will be pegged if the MM has completed SDU registration and
the SDFRA Link is setup, but the MM is missing something else in its
database that is preventing the call from being setup properly.

Causes:
Detailed Cause Description
NA No PSILINK in MM DB
NA No SDFRA in MM DB

Troubleshooting:
S Look at MM callproc logs.

CFC=112 Pkt Oriented Data Call Setup Failure


This CFC will be pegged if the call setup fails during the PCFPDSN
setup messaging phase.

Causes:
Detailed Cause Description
0x1140 PCF Pkt Oriented Setup Timer Expired

Troubleshooting:
S Use a PDSN system monitoring tool to view where the A10/A11
(RP) control messaging experienced problems. This should help
isolate the problem to the PCF or the PDSN. Refer to the CDMA2000
1X RAN Functional Description for a detailed description of A10/A11
call setup processing.
S Investigate the status of the PDSN.
S Verify that the SDU PCF CPE IP addresses are in the PDSN VHA
pool. To determine the PCF CPE IP addresses, enter the following
command at the SPROC: SdfCpePrintStore

IMPORTANT

* Logging into the SPROC should be performed during a


scheduled maintenance window to avoid system disruption.

APR 2004 SDU Troubleshooting 15-13


SDU CFC Troubleshooting continued

CFC=113 Pkt Oriented Data Call Protocol Violation


This CFC will be pegged usually because the SDF TA8 inactivity timer
expired. This will also be pegged if the PPP session fails to establish.

Causes:
Detailed Cause Description
0x1141 Protocol error between MS and BS
0x1143 TA8 Inactivity Timer Expired

Troubleshooting:
S Troubleshoot the PDSN.
S Get a sniffer log to look at PPP messaging. If the CFC occurred due
protocol error, this means that the PPP session did not establish
between the mobile and the PDSN due to the fault of the mobile or
PDSN. For a description of what successful PPP setup looks like
between the mobile and the PDSN, refer to the CDMA2000 1X RAN
Functional Description.

CFC=138 No PSI_SDU Available


This CFC is pegged when the MM is not aware of any SDUs being
available.

Causes:
Detailed Cause Description
NA MM SDFRA link is OOS
NA SDU is OOS (no MMSDU connection)

Troubleshooting:
S Troubleshoot any alarms for MMSDFRALINK (the call processing
link between the MM and SDU SDFRA). Refer to the applicable
alarm procedure.
S Troubleshoot the SDU OOS alarm. Refer to the applicable alarm
procedure.
S Verify you can ping the SPROC from the MM. Refer to your IP
planning documentation for the applicable IP addresses.

15-14 SDU Troubleshooting APR 2004


SDU CFC Troubleshooting continued

CFC=139 No PSICE Available


This CFC will be pegged when all available PSICEs have been utilized
on the system.

Causes:
Detailed Cause Description
0x114B cBTS Proxy Resource Request Timeout
0x114F cBTS Proxy Resource Configure Timeout
0x1150 cBTS Proxy Resource Mismatch

Troubleshooting:
S Consider increasing PSICE capacity.
S To verify actual capacity/usage, you can enter Disp_psice at the
XC OMP to determine PSICE availability. You can also get a sniffer
log to look at the actual cBTS Proxy messaging.

CFC=142 No PDSN Resources Available


The A10/A11 interface was not setup due to no PDSN resources being
available.

Causes:
Detailed Cause Description
0x1D79 PDSN Resources Not Available

Troubleshooting:
S Consider increasing PDSN capacity/resources.

CFC=143 No PCF Resources Available


This CFC is pegged when PCFRA indicates that there are no available
PCF resources.

Causes:
Detailed Cause Description
0x1D32 No PCF Resources available

Troubleshooting:
S Consider increasing SDU PCF capacity/resources.

APR 2004 SDU Troubleshooting 15-15


SDU CFC Troubleshooting continued

CFC=146 A11 Registration Denied


This CFC is pegged when the PDSN is denying registration in the A11
Reg Reply message.

Causes:
Detailed Cause Description
0x1D80 PDSN Reg Denied Reason Unspecified
0x1D81 PDSN Reg Denied Administratively Prohibited
0x1D82 PDSN Reg Denied Insufficient Resources
0x1D83 PDSN Reg Denied MS Failed Authentication
0x1D85 PDSN Reg Denied Identification Mismatch
0x1D86 PDSN Reg Denied Poorly formed request
0x1D88 PDSN Reg Denied Unknown PDSN Address

Troubleshooting:
S Examine the PDSN for possible failures. Refer to vendor
documentation for details.

15-16 SDU Troubleshooting APR 2004


*68P09258A80B*
68P09258A80B
Technical
Information

SDU TROUBLESHOOTING

SOFTWARE RELEASE 2.16.3.X


SDU
CDMA2000 1X

ENGLISH
APR 2004
68P09258A80B
SDU TROUBLESHOOTING

68P09258A80B
APR 2004
ENGLISH
SOFTWARE RELEASE 2.16.3.X
SDU
CDMA2000 1X
Technical Information Products and Services

STANDARD MANUAL PRINTING INSTRUCTIONS

Part Number: 68P09258A80B Filename: APC:


Title: SDU Troubleshooting
Volume of Print Vendor: Date

STANDARD SPECIFICATIONS FOR REFERENCEDO NOT MODIFY

PAPER: 1st. LEVEL TABS: 2nd. LEVEL TABS: FINISHING:

Body: 70 lb.

Inside Cover: 65 lb. Cougar


Single Sided
5 Cuts
Single Sided
7 Cuts
3Ring Binder
SlantD

Tabs: 110 lb. Index Clear Mylar Clear Mylar 3Hole Punched

Binder Cover: Standard TED Pantone 2706C White


(5/16in. dia.)
Shrink Wrap Body

cover 10 pt. Carolina Black Ink Black Ink

Black ink for body, inside cover, and binder cover.

TAB and SHEET SIZE/QUANTITY


7X9 8.5x11 11x17 Sheets = (Total Pages) / 2

1st Level Tabs 2nd Level Tabs

NONSTANDARD SPECIFICATIONS
Tape Bound Corner Stitch

Other: Meet with manager to determine the deliverable.

SPECIAL INSTRUCTIONS

Potrebbero piacerti anche