Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Information
SDU TROUBLESHOOTING
ENGLISH
APR 2004
68P09258A80B
SPECIFICATIONS SUBJECT TO CHANGE WITHOUT NOTICE
Notice
While reasonable efforts have been made to assure the accuracy of this document, Motorola, Inc. assumes no liability resulting from any
inaccuracies or omissions in this document, or from use of the information obtained herein. The information in this document has been
carefully checked and is believed to be entirely reliable. However, no responsibility is assumed for inaccuracies or omissions. Motorola,
Inc. reserves the right to make changes to any products described herein and reserves the right to revise this document and to make
changes from time to time in content hereof with no obligation to notify any person of revisions or changes. Motorola, Inc. does not
assume any liability arising out of the application or use of any product, software, or circuit described herein; neither does it convey
license under its patent rights or the rights of others.
It is possible that this publication may contain references to, or information about Motorola products (machines and programs),
programming, or services that are not announced in your country. Such references or information must not be construed to mean
that Motorola intends to announce such Motorola products, programming, or services in your country.
Copyrights
This instruction manual, and the Motorola products described in this instruction manual may be, include or describe copyrighted
Motorola material, such as computer programs stored in semiconductor memories or other media. Laws in the United States and
other countries preserve for Motorola and its licensors certain exclusive rights for copyrighted material, including the exclusive
right to copy, reproduce in any form, distribute and make derivative works of the copyrighted material. Accordingly, any
copyrighted material of Motorola and its licensors contained herein or in the Motorola products described in this instruction manual
may not be copied, reproduced, distributed, merged or modified in any manner without the express written permission of Motorola.
Furthermore, the purchase of Motorola products shall not be deemed to grant either directly or by implication, estoppel, or
otherwise, any license under the copyrights, patents or patent applications of Motorola, as arises by operation of law in the sale of a
product.
Copyrighted Materials
Software and documentation are copyrighted materials. Making unauthorized copies is prohibited by law. No part of the software or
documentation may be reproduced, transmitted, transcribed, stored in a retrieval system, or translated into any language or
computer language, in any form or by any means, without prior written permission of Motorola, Inc.
Trademarks
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are
the property of their respective owners.
Javat Technology and/or J2MEt: Java and all other Javabased marks are trademarks or registered trademarks of Sun
Microsystems, Inc. in the U.S. and other countries.
UNIXR: UNIX is a registered trademark of The Open Group in the United States and other countries.
REV091302
Table of Contents
SDU Troubleshooting
Software Release 2.16.3.x
Chapter 1: Introduction
About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1
SDU Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5
Chapter 3: Login/Passwords
Log in to SDU Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1
Scope of manual
This manual is intended for use by cellular telephone system
craftspersons in the day-to-day operation of Motorola cellular system
equipment and ancillary devices.
This manual is not intended to replace the system and equipment
training offered by Motorola, although it can be used to supplement or
enhance the knowledge gained through such training.
Obtaining Manuals
To view, download, or order manuals (original or revised), visit the
Motorola Lifecycles Customer web page at http://services.motorola.com,
or contact your Motorola account representative.
If Motorola changes the content of a manual after the original printing
date, Motorola publishes a new version with the same part number but a
different revision character.
Text conventions
The following special paragraphs are used in this manual to point out
information that must be read. This information may be set-off from the
surrounding text, but is always preceded by a bold title in capital letters.
The four categories of these special paragraphs are:
NOTE
Presents additional, helpful, non-critical information that
you can use.
IMPORTANT
CAUTION
Presents information to identify a situation in which
damage to software, stored data, or equipment could occur,
thus avoiding the damage.
WARNING
Presents information to warn you of a potentially
hazardous situation in which there is a possibility of
personal injury.
Contact us
Send questions and comments regarding user documentation to the email
address below:
cdma.documentation@motorola.com
Motorola appreciates feedback from the users of our information.
Remember! . . . Safety
depends on you!!
The following general safety precautions must be observed during all
phases of operation, service, and repair of the equipment described in
this manual. Failure to comply with these precautions or with specific
warnings elsewhere in this manual violates safety standards of design,
manufacture, and intended use of the equipment. Motorola, Inc. assumes
no liability for the customers failure to comply with these requirements.
The safety precautions listed below represent warnings of certain dangers
of which we are aware. You, as the user of this product, should follow
these warnings and all other safety precautions necessary for the safe
operation of the equipment in your operating environment.
NOTE
Refer to Grounding Guideline for Cellular Radio
Installations 68P81150E62.
Dangerous procedure
warnings
Warnings, such as the example below, precede potentially dangerous
procedures throughout this manual. Instructions contained in the
warnings must be followed. You should also employ all other safety
precautions that you deem necessary for the operation of the equipment
in your operating environment.
WARNING
Dangerous voltages, capable of causing death, are present in this
equipment. Use extreme caution when handling, testing, and
adjusting .
Manual Number
68P09258A80B
Manual Title
SDU Troubleshooting
Software Release 2.16.3.x
Version Information
The following table lists the manual version, date of version, and
remarks on the version.
Chapter 1: Introduction
Table of Contents
Notes
IMPORTANT
IMPORTANT
NOTE
Additional performance tests may be required to insure
that the SDU still meets ATP specifications after any
device is replaced. Refer to the applicable Field
Replacement Unit (FRU) and ATP manual for this
information.
Associated Manuals
The following manuals and online help systems are referenced in this
manual.
S AN & SDU FRU (Field Replacement Procedures)
S CDMA2000 1X RAN Hardware Installation
S CDMA2000 1X RAN Functional Description
S CDL/CFC Reference Manual
S SMAP Operators Guide
S Cellular System Administration Guide (Online Help)
S System Command Reference (Online Help)
S System Output Messages (Online Help)
S SDU ATP (Acceptance Test Procedure)
Overview
The SDU performs the Selection/Distribution Function and Packet
Control Function for the 2.16.1.x network. For details, refer to the
CDMA2000 1X RAN Functional Description.
SDU Frame
The following figures illustrate the SDU frame, card cage, and top I/O
Panel.
75 75 75 75
MAIN BREAKER PANEL
POWER SUPPLY
20 20
20 20 MODULES (4)
20 20
20 20
20 20
CIRCUIT BREAKER
MODULES (2)
FAN MODULES
FANS MODULES
NOTE:
1. Frame shown with doors removed
2. System configuration determines if upper cage is used.
Acronyms
Table 1-1 defines acronyms used in this manual.
Notes
Table of Contents
Notes
Overview
Many SDU BPP and SDU CCA alarm troubleshooting procedures 2
require a reset to resolve the alarm.
Generally, the alarm troubleshooting procedures conform to the
following variations:
S Reset not required
S Reset only required if the systems automatic reset fails
S Reset is up to the operator to perform when to reset depends upon
the urgency of the alarm.
WARNING
When manually resetting SDU cards, the call processing
resources will temporarily be unavailable. If possible,
attempt the reset during an appropriate maintenance
window (determine the severity of the alarm).
WARNING
Never reseat or reset the SPROC for any reason, unless
Motorola CNRC has given the instruction to do so. If this
is the case, first lock the SPROC from the OMCR.
Lock/Unlock
When manually resetting cards (as opposed to an automatic reset by the
2 system), use the OMCR lock and unlock commands. Lock removes the
card from service and unlock restores the card to service. For details,
refer to the System Command Reference.
NOTE
This case applies to both an automatic system reset and a
manual reset.
Table of Contents
Notes
Overview
Occasionally, an SDU troubleshooting procedure requires logging in to a
card. The following information provides directions on how to login to a
device remotely without accessing the front panel of the device. This
section explains how to log in to the various SDU cards.
IMPORTANT 3
* Logging into the SPROC, ISB, and BPP devices should be
performed only during a maintenance window to avoid
system disruption.
WARNING
Any of the following actions could prevent future access to
the SDU boards:
S Never use the telnet command to log into any of the
SDU boards.
S Never use the Exit command to logout of any of the
SDU boards.
SPROC
At the OMCR use the following command to access to the SPROC.
S rlogin SDU1 MANAGEMENT_LINK<IP address>
To start a CLI session, type CLI at the prompt.
To exit the SPROC type LOGOUT at the prompt.
ISB
At the OMCR use the following command to access to the ISB.
S rlogin SDU1 ROUTERX<IP address>
To start a CLI session, type CLI at the prompt.
To exit the SPROC type LOGOUT at the prompt.
BPP
At the OMCR use the following command to access to the BPP.
S rlogin SDUBCP<IP address>
To start a CLI session, type CLI at the prompt.
To exit the SPROC type LOGOUT at the prompt.
IMPORTANT
Configuration
To log in to SDU cards, you will need to obtain a PC with a Serial port,
and a terminal emulation program such as Hyperterminal. A
standalone dumb terminal can also be used.
The settings consist of:
S Baud Rate: 9600 Bps
Data: 8 Bit
S Parity: None
S Stop bit: 1
S Flow control: none
S VT100 Emulation
It is necessary to use a Null Modem cable to connect the PC to the
ISB/SPROC.
NOTE
S The MMI Console Interface is started when the network
element is booted; it is not necessary to manually start
the MMI Console Interface.
S After the terminal/PC is connected to the card, you may
need to press the <Enter> key several times.
S SPROC
Login: motorola
Password: developer
Exit MMI Password: partymoto
NOTE
Use startMMI to enter MMI mode again.
S BPP
Login: motorola
Password: developer
Secure MMI: altivec
S CCA
Login: motorola
Password: developer
S ISB 3
Login: motorola
Password: developer
Enter logout to quit.
Recovery Procedure
If the rlogin command for the BPP command does not work, the
following procedure can be performed at the site.
Connect a serial port to the front panel of the BPP board.
At the prompt enter the following:
S lock
S shell Restart
If the lock and restart commands do not work, perform an lock/unlock
command to see if the session can be recovered.
Notes
Table of Contents
Notes
Data Collection
When contacting Motorola CNRC for assistance with SDU problems,
consider collecting the following data:
S Log current status information from the OMCR
S Optionally, also collect black box data from the OMCR.
This section describes the process for collecting the black box data.
collectbox
The collectbox program is intended to be used on an OMCR to collect 4
debug information from the SDU. The debug information being gathered
is called a blackbox (airplane lingo) and contains information recorded
on the controller (SPROC).
bboxlisten
The bboxlisten program collects the blackbox logs from the SDU in the
case of an SPROC reset. Bboxlisten also provides some automatic bbox
file maintenance on the OMCR.
Prerequisites
The OMCR Software Installation should include the black box
programs for SDU. Installing software on the OMCR is beyond the
scope of this manual.
The following is a brief overview of the install process for reference
purposes: untar collect.tar into a temporary directory. Run installbbox
from the temporary directory to copy the executables and scripts to the
correct directory on the OMCR. Run cronbbox INSTALL to start the
server.
Running collectbox
Running collectbox
From the OMCR, use the following command to run collectbox:
4
Setting the ne# (network element id or sdu number) or ipaddress is
mandatory. Optional parameters can be entered in any order.
NOTE
If not specified, the ne # is resolved from ip address. If the
ip address is not specified, the OMCR will pull the
address from the MIB based on the ne # specified.
Optional Parameters
Optional parameters:
S [f filter] Grabs the full blackbox using the specified filter, the
blackbox is not reset as part of the collection. The default behavior is
to collect the filtered blackbox from the active side. If neither the f or
p parameters are specified, the full unfiltered blackbox is collected.
Default behavior for the full blackbox is collection from the active
side.
S [p] Grabs the preserved blackboxes from the last reset. The default
behavior is to grab the preserved blackboxes from both the active and
the standby controllers. If neither the f or p parameters are
specified, the full unfiltered blackbox is collected. Default behavior
for the full blackbox is collection from the active side.
S [d directory] Specify the directory for output to be stored on the
OMCR. The default directory when none is specified is
/sc/spool/bbox.
S [active] Collect the blackbox from only the active side.
S [standby] Collect the blackbox from only the standby side.
S [both] Collect the blackbox from both sides.
Examples
The following examples illustrate the use of collectbox.
S collectbox ne 5
Collects a full blackbox from the active side of SDU5.
S collectbox ne 8 p
Collects the preserved blackboxes from both sides of SDU8.
S collectbox ne 1 f SNMP
Collects a filtered blackbox based on the keyword SNMP from the
active side of SDU1.
S collectbox ip 10.4.5.1 p standby
Collects the preserved blackboxes from the standby side of the SDU
with a management address of 10.4.5.1.
Description of Operation 4
This command does not require the CLI to be up when the command is
executed. The command will attempt to find the SDU in the database
and determine the correct IP address to connect to. The command will
normally ping the platform before attempting to collect the specified
blackbox. The command will display the name and location of the file
collected. Any temporary files created on the platform will be cleaned
up. This program telnets to the platform which grabs console output
temporarily from the front panel serial port.
Running bboxlisten
Running bboxlisten
No user intervention is required for the automatic upload of black boxes.
Description of Operation
The server will wake up when an SDU sends it a message after an
SPROC reset. The server will check every 30 seconds for collection
requests and invoke the collection script. The script will attempt to
upload the preserved black boxes from either SPROC (active and
standby). The script will turn off nagging on the SPROC so no more
requests to upload are generated from this reset. The script will check the
number of stored black boxes on the OMCR and clean up if needed.
s2 SDU2
1004 October 4
1939 7:39 PM
4 a0 preserved file /data/blackbox/0 from
active side
s1 preserved file /data/blackbox/1 from
standby side
sh preserved file
/data/blackbox/hardResetBBox from standby side
af0 preserved file /active/init/hungbbox0
from standby side
Table of Contents
Notes
Introduction
If the SDU should become completely unresponsive for some reason, the
method of last resort for recovery would be a power cycle of the SDU
cage (which in turn would power cycle every FRU in the SDU cage).
75 75 75 75
Figure 5-2: SDU Power Supply Modules and Circuit Breaker Modules
20 20
20 20
20 20
20 20
20 20
5 To cycle power to a given cage, the two circuit breaker switches for the
two power supplies to that cage must be pulled. The left two switches
control power to the lower cage and the right two switches control power
to the upper cage. Again, both switches must be pulled since the power
supply feeds are redundant.
Once both breaker switches are pulled, wait 1015 seconds and then
push the same two switches back in towards the frame. At this point, the
system should initialize completely and return to operation.
Cage Recovery
Once both circuit breaker switches are pushed back in for the Power
Supply, the SDU Frame will begin to initialize the SPROC and other
Payloads. During this process, the OMCR (NE Manager) gets an Alarm
which indicates the Loss of Communication with SDU Frame. Once
the NE (SPROC) completes the initialization and the ISB learns the
Routes from Access Node, then the SPROC will start sending the
SYNC Required to the OMCR. The OMCR also clears the Loss of
Communication Alarm. Then, the OMCR SYNCs with NE. Once the
NE SYNCs up with OMCR, the SDU Frame will be in the Operational
State.
Table of Contents
Notes
Boot progress
To monitor the boot progress of a board in an SDU frame, a console
needs to be connected to the board. Without a console connection, only
the LEDs can provide a limited status. Once the LED light turns into
solid green, the board is finished rebooting.
State Determination
Once the SDU finishes rebooting, the OMCR can be used to check the
state of the SDU. For OMCR Status/Display Commands, please refer
to the System Command Reference.
Notes
Table of Contents
Notes
Introduction
This section contains troubleshooting procedures for SDU initialization
failure scenarios.
NOTE
Refer to chapter 6 for descriptions of LED states during
various boot progress stages.
WARNING
Never reseat or reset the SPROC for any reason, unless
Motorola CNRC has given the instruction to do so. If this
is the case, first lock the SPROC from the OMCR.
Symptom
If the SPROC (either active or standby) fails to join in the cluster and is
unable to boot up, use the following information to troubleshoot the
problem.
Possible Causes
Possible causes for this symptom include one of the following:
S LAN failure 7
S While the OMCR is configuring the ISBs, the SPROCs unmanaged
timeout timer expires.
Recovery Action
Reset the SPROC, and verify that the ISBs are in a stable state.
Symptom
The SPROC initialization does not complete if the SPROC console
displays node not in database and then resets.
Possible Causes
Possible causes for this symptom include one of the following:
S The configuration data file (baseNECDF .xml) for the SPROC either
contains errors or is corrupted.
For example, the configuration data may include an incorrect cage or
slot number.
S The SPROC is not properly seated.
Recovery Action
Perform the following recovery actions.
Symptom
If the SPROC console displays going unmanaged and then resets,
this indicates that the SPROC lost management control and is
performing a reset to clear the underlying problem.
Possible Causes
This symptom may be due to one of the following causes:
S With a 2N redundant setup, if both the ISBs reset and initialize again,
one of the SPROCs will display going unmanaged and reset as
a result of a split brain resolution process. There is a possibility that
the node which was a manager (the active one) may also reset.
S Both ISBs are resetting.
Recovery Action
There is no recovery action. The SPROC automatically resets itself to
resolve the problem.
Symptom
NOTE
A hung thread is like a hanging software task, only it is a
thread instead of a process.
If the hung thread problem occurs on the SDU, the SPROC console
displays something like the following output:
[D2D37D8] PANIC: Starting Emergency Exit
Handlers
[D2D37D8] PANIC: Called Default Emergency Exit
Handler
[D2D37D8] Writing Emergency Exit Trace
Information for 1 threads
[D2D37D8] goahead/hap Error: HUNG THREADS
DETECTED: srk tid = 14000011 (OS id = c43fa70)
[D2D37D8] Default EEH: PANIC for srk tid =
14000011 (OS id = c43fa70).
[D2D37D8] Finished Default Emergency Exit
Handler
[D2D37D8] PANIC: Finished Emergency Exit
Handlers
[D2D37D8] PANIC: Closing SRK
[D2E3D50] Closing Extensions
[D2E3D50] RIM Extension Shutting Down.
NEI : NEIX: Possible GA hung thread. Stack trace
7 in black box.Rebooting in 30 secs...
The SPROC will reboot.
If the standby SPROC has a higher slot number than the active SPROC,
and the hung thread problem occurs on the active SPROC, this
problem may cause a site reset (if the standby SPROC loses arbitration
for the cluster manager role).
Possible Causes
A GA hung thread is the cause. The GoAhead software shuts itself
down, which causes the SPROC to reboot.
Recovery Action
A full SDU reset from the OMCR may help to clear out the hung thread
problem, but use caution.
WARNING
A full SDU reset will cause a site outage. If possible,
perform the reset during an appropriate maintenance
window.
Symptom
If the SPROC software becomes corrupted, the SPROC will reboot.
Possible Causes
See the Symptom description.
Recovery Action
No recovery action is necessary.
The SDU automatically resolves the problem using one of the following
methods:
S For a single SPROC node setup, the SPROC will reboot ten times,
then it will switch the partition and attempt to initialize using the other
partition.
7
S For a 2N redundant SPROC setup, if the other node becomes the
manager, the node running the corrupted software load will: become
the nonpotential manager; FTP the load from the manager; and
reset itself after switching the partition.
Symptom
If the active LAN fails, the active SPROC will transition to the OOS
state, and the redundant SPROC will take over.
The SPROC console displays output similar to the following:
NEIX: active interface x with an ip address
x.x.x.x and subnet x failed
Possible Causes
The ISB connected to the active LAN is not functioning properly.
Recovery Action
None. This is a normal behavior.
Table of Contents
Notes
Overview of ISB/SPROC
Troubleshooting
This chapter deals specifically with the procedures for troubleshooting
the ISB/SPROC cards in the SDU.
IMPORTANT
Overview
This section contains troubleshooting procedures for various ISB failure
scenarios.
WARNING
Never reseat or reset the SPROC for any reason, unless
Motorola CNRC has given the instruction to do so. If this
is the case, first lock the SPROC from the OMCR.
Symptom
The ISB is unable to download the software load from the SPROC.
Possible Causes
The possible cause may be one of the following. The ISB:
S Is new from factory;
S Does not have a programmed route to the SPROC;
S Is not receiving the GETCODELOAD from the proxy component.
Recovery Action
Contact CNRC for assistance.
Symptom
The SPROC is unable to get IP service from the ISB.
Possible Causes
Ensure that the debug ethernet port of the SPROC isnt connected to an
outside network. If, when the SPROC boots up, the console display
indicates it is loading up on the debug interface, the debug port is
connected to the network.
Recovery Action
Perform the recovery actions in Table 8-1.
Symptom
The symptoms include at least one of the following:
S The ISB resets
S The SPROC console displays keep alive timed out for the ISB.
Possible Causes
The SPROC reset the ISB, because the SPROC did not receive any
acknowledgement from the ISB.
Recovery Action
None. The automatic reboot should correct the problem.
Symptom
The ISB reset causes payloads (BPPs, CCAs) to reset.
Possible Causes
If the ISB failover didnt occur quickly enough, the payloads will also
reset.
Recovery Action
None. The automatic ISB reset should correct the problem.
Symptom
The ISB fails to establish the gigabit link.
Possible Causes
Possible causes include:
S Cable failure
S Span I/O failure
S The SPROC database was not updated.
Recovery Action
Check/replace cables and/or span I/O (swap out); check/update the
SPROC database for gigabit links.
Symptom
During the SPROC failover, the ISB automatically reset.
Possible Cause
The route updates were not fast enough.
Recovery Action
None. The automatic reset should correct the problem.
Symptom
The Payloads reset after a Dual standby pull.
Possible Causes
The route updates were not fast enough.
Recovery Action
None. The automatic reset should correct the problem.
Symptom
The active ISB reset causes both SPROCs to reset.
Possible Causes
The route updates were not fast enough. 8
Recovery Action
None. The automatic reset should correct the problem.
Symptom
When the standby ISB reset, the SPROC also reset.
Possible Causes
The route updates were not fast enough.
Recovery Action
None. The automatic reset should correct the problem.
ISB/SPROC Alarms
This section contains troubleshooting procedures for
ISB/SPROCrelated alarms.
Though the ISB or SPROC reports these alarms, in some cases another
card may be causing the underlying problem.
WARNING
Never reseat or reset the SPROC for any reason, unless
Motorola CNRC has given the instruction to do so. If this
is the case, first lock the SPROC from the OMCR.
NOTE
In addition to maintaining troubleshooting logs, you may
also want to notify CNRC of problems if they frequently
reappear.
NOTE
For a more detailed description of each alarm, refer to
System Output Messages. Also, if you are not sure when
you need to resolve the problem, refer to the alarm
documentation (in the System Output Messages) for the
severity level.
2 If it is not a physical link problem, replace the ISB. Refer to the CDMA2000 1X AN & SDU FRU for
details.
3 If the reset fails, lock and replace the card using the procedures contained in the CDMA2000 1X AN &
SDU FRU manual.
2 If the problem continues, replace the BPP. Refer to the CDMA2000 1X AN & SDU FRU.
IMPORTANT
2 If the device is not present or operational, replace it with an operational device. Refer to the
CDMA2000 1X AN & SDU FRU manual.
NOTE
For details, refer to the Cellular System Administration Guide for topics that deal with the SDULINK
software object and its subelements (the MMSDFRALINK and MMPCFRALINK).
2 The failed SPROC becomes the standby SPROC. If the problem is not resolved in the reset, the 8
standby SPROC continues to fail. In this case, replace the card.
Refer to the CDMA2000 1X AN & SDU FRU manual for details.
NOTE
For details, refer to the Cellular System Administration Guide and System Command Reference.
8 NOTE
For details, refer to the Cellular System Administration Guide and System Command Reference.
NOTE
For details, refer to the Cellular System Administration Guide and System Command Reference.
2 The failed SPROC becomes the standby SPROC. If the problem is not resolved in the reset, the
standby SPROC continues to fail. In this case, replace the card. 8
Refer to the CDMA2000 1X AN & SDU FRU manual for details.
NOTE
For details, refer to the Cellular System Administration Guide and System Command Reference.
2826750 IP Failure
An internal software failure occurred in the protocol stack on the
SPROC in the onboard communications protocol software.
SPROC Communication
Alarms
2 If other links are failing locally, a problem may exist with the network connection or the ISB. In this
case, check the ISB for alarms and check the farend MM for failures.
2 Check the farend MM for lock or failure status. Refer to the Cellular System Administration Guide
and System Command Reference for details on displaying the status of the MM.
Notes
Table of Contents
Notes
Introduction
This chapter contains SDU synchronization failure scenarios and the
applicable recovery actions.
SDUOMCR Audit
Synchronization failure
Symptom
Any command from the OMCR to the SDU fails.
Possible Cause
The SDU is not in a state where it can synchronize with OMCR (or the
SDU is not connected to the OMCR).
Recovery Action
To verify that the SDU and OMCR are not in sync, type
snmpConnStateShow at the SPROC command line.
IMPORTANT
At the OMCR, initiate the audit command to the SPROC. Refer to the
System Command Reference for details.
Symptom
The SDU fails to send the sanity (heartbeat) message to the OMCR
every two minutes.
NOTE
The OMCR monitors heartbeat messages to verify that it
is still in contact with the device.
Possible Cause
9
Connectivity does not exist between the SDU and the OMCR.
Recovery Action
If the OMCR is not receiving heartbeat messages from the SDU,
perform the following procedure.
APR 2004 SDU Troubleshooting 9-1
SDU Synchronization Failure Scenarios continued
Symptom
The OMCR cannot reset the SDU or the particular SDU card.
Possible Cause
The OMCR command does not reach the SDU.
This could be due to a transient error or a connectivity problem.
Recovery Action
Retry the OMCR command.
If it still doesnt work, determine whether there are any connectivity
9 problems (are there alarms for contact being lost with SDU).
Troubleshoot those issues based on the applicable alarm procedure.
Table of Contents
10
APR 2004 SDU Troubleshooting
Table of Contents continued
Notes
10
SDU Troubleshooting APR 2004
BPP Troubleshooting
Overview of BPP
Troubleshooting
This chapter deals specifically with the procedures for troubleshooting
the BPP cards in the SDU.
10
APR 2004 SDU Troubleshooting 10-1
Verifying BPP Status
10
10-2 SDU Troubleshooting APR 2004
Troubleshooting BPP Alarms
Overview
This section contains troubleshooting procedures for BPPrelated
alarms.
These alarms deal with the BPP hardware when implemented with
Packet Control Function (PCF) software and Selection Distribution
Function (SDF) software.
The following procedures address the alarm number, alarm name, and
procedure.
NOTE
In addition to maintaining troubleshooting logs, you may
also want to notify CNRC of problems if they frequently
reappear.
NOTE
For a more detailed description of each alarm, refer to
System Output Messages. Also, if you are not sure when
you need to resolve the problem, refer to the alarm
documentation (in the System Output Messages) for the
severity level.
2 Reset the hardware using the reset switch on the front panel of the specific BPP card.
3 If the hardware reset fails, lock and replace the card using the procedures contained in the CDMA2000
1X AN & SDU FRU manual.
2 If the software load is corrupt, replace it with a valid load and codeload the card with the new
software. Refer to the SDU Software Installation documentation (in the Release Binder) for details.
10
10-4 SDU Troubleshooting APR 2004
Troubleshooting BPP Alarms continued
2 Reset the hardware using the reset switch on the front panel of the specific BPP card.
3 If the hardware reset fails, lock and replace the card using the procedures contained in the CDMA2000
1X AN & SDU FRU manual.
10
APR 2004 SDU Troubleshooting 10-5
Troubleshooting BPP Alarms continued
2 If the reset fails, reset the card using the reset switch on the front panel of the specific BPP card.
3 If the hardware reset fails, lock and replace the card using the procedures contained in the CDMA2000
1X AN & SDU FRU manual.
2 If the hardware reset fails, lock and replace the card. Refer to the CDMA2000 1X AN & SDU FRU for
details.
10
10-6 SDU Troubleshooting APR 2004
Troubleshooting BPP Alarms continued
2 If the additional text does not indicate that the board should be replaced and returned, reset the card.
3 If the card fails to reinitialize properly after a reset, or if the alarm continues to occur, the card should
be replaced.
2 If the card fails to initialize, reset the hardware using the reset switch on the front panel of the specific
BPP card.
3 If the reset fails, or if the problem recurs several times, lock and replace the card using the procedures
contained in the CDMA2000 1X AN & SDU FRU manual.
10
APR 2004 SDU Troubleshooting 10-7
Troubleshooting BPP Alarms continued
2 If the reset fails, lock and replace the card using the procedures contained in the CDMA2000 1X AN &
SDU FRU manual.
NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state. Also, there should be an attribute value change indicating that the
SDU capacity (either SDF or PCF) has increased by one BPPs worth of capacity.
10
10-8 SDU Troubleshooting APR 2004
Troubleshooting BPP Alarms continued
2 If there are no ISB alarms, but other cards are also reporting link alarms, the ISB is probably the
source of the problem. Reset the ISB at a convenient time. When the ISB recovers, the BPP alarm
should clear.
3 If the ISB and other cards appear to be fine, or if resolving the ISB problems does not clear the alarm,
reset the BPP at a convenient time. If the problem remains, replace the BPP.
10
APR 2004 SDU Troubleshooting 10-9
Troubleshooting BPP Alarms continued
2 If the problem is specific to the one card, reset the card. If the problem persists, replace the card.
3 If the problem is not specific to the one card, it is necessary to assess and correct the general
environmental conditions (e.g. fan, airflow) surrounding the SDU frame. When the environmental
problem has been corrected, the alarm will clear.
The environmental conditions must meet the specifications outlined for the SDU frame in the
CDMA2000 1X RAN Hardware Installation manual.
4 If a card reports high temperature for an extended time and later exhibits unusual behavior, the card
may have been damaged and should be replaced.
10
10-10 SDU Troubleshooting APR 2004
Troubleshooting BPP Alarms continued
2 Verify that the software load is valid and complete. To do so, check that the current software load is
actually the one intended to be running, and that all of the pertinent software files are present. If the
software load is corrupt, replace it with a valid load and codeload the card with the new software.
NOTE
Refer to the SDU Software Installation documentation (in the Release Binder) for details.
2 If it is not a card connectivity problem, then it is necessary to replace the FRU identified in the alarm.
Refer to the CDMA2000 1X AN & SDU FRU manual for details.
10
10-12 SDU Troubleshooting APR 2004
11
Table of Contents
Notes
Overview of CCA
Troubleshooting
This chapter deals specifically with the procedures for troubleshooting
the CCA cards in the SDU.
Overview
This section contains troubleshooting procedures for CCArelated
alarms.
The following procedures address the alarm number, alarm name, and
procedure.
NOTE
In addition to maintaining troubleshooting logs, you may
also want to notify CNRC of problems if they frequently
reappear.
NOTE
For a more detailed description of each alarm, refer to
System Output Messages. Also, if you are not sure when
you need to resolve the problem, refer to the alarm
documentation (in the System Output Messages) for the
severity level.
Troubleshooting CCA
Processing Faults
Troubleshooting CCA
Equipment Faults
2 If maintenance was being performed, either a fan tray or power supply module was removed from the
system. The specified device should be placed back into the system.
NOTE
When a device is removed from the system wait 10 seconds before replacement to allow for software
adjustments needed for EID reread (and other internal adjustments) upon reinsertion.
2 If this alarm is a fan tray alarm, the fan tray alarm indicates a problem with the fan tray hardware
has occurred. When this alarm condition is detected on any fan all other fan trays in the system will
boost until the alarm condition clears.
Causes/debugging:
S Something is lodged in the fan, the voltage has dropped, or the motor is bad, causing the rotation of
the fan blades to stop or slow down.
S Fan alarms are produced by faulty fan units which should be replaced.
S The automatic boost of the fans may place the rotation speed within the alarm threshold which in
turn will clear the alarm. The fans will then unboost and the alarm will occur again creating a cycle
or bounce condition on the alarm and fan boost.
3 If this alarm is a power supply controller board alarm, this indicates a hardware failure and could be
caused by an overstress condition. Replace the hardware since it is not functional.
Troubleshooting CCA
Environmental Faults
2 Carefully check possible causes for this alarm. This alarm is indeterminate and could be caused by
power supply hardware, the AC filter, etc.
3 If the power supply voltage alarm condition also exists, this may indicate that the power supply
modules have been operating on batteries and the batteries are near being fully discharged.
4 If the power supply voltage alarm condition does not exist, this alarm indicates an unknown hardware
failure, so replace the power supply module.
NOTE
Cabinet sensor 1 alarm occurs due to the cabinet / cage door being open.
Cabinet sensors 2 5 have not yet been defined.
NOTE
Since the sensors are customer defined, the troubleshooting action will depend upon the specific
implementation. To troubleshoot this problem it is necessary to know how the sensors were wired for
the particular installation.
NOTE
When working with hardware follow all safety precautions specified in the CDMA2000 1X RAN Data
Hardware Installation and CDMA2000 1X AN & SDU FRU manuals.
Troubleshooting CCA
Communication Faults
2 If you reseated the PSM and the alarm still occurs, contact CNRC.
2 If it is not a card connectivity problem, then it is necessary to replace the FRU identified in the alarm.
Refer to the CDMA2000 1X AN & SDU FRU manual for details.
Notes
Table of Contents
Notes
12
Overview
This section provides a listing of OMCR alarms that apply for multiple 12
network elements but also apply to the SDU.
Refer to the System Output Messages manual for the alarm description
and operator action.
OMCR Alarms
The following list describes related OMCR alarms:
S ALARM: 2878 Platform Out of Service
S ALARM: 2879 Platform Communication Failure with OMCR
S ALARM: 28102 SNMP MIB Support Unavailable for Committed
Version
S ALARM:28103 Management Synchronization Failed with NE
S ALARM: 306168 OMCR /SDU Database Mismatch
S ALARM: 306170 OMCR GCMGR Queue to NE Inactive
S ALARM:286820 Synchronization Failed: Unexpected Response
from Network Element
S ALARM:286821 Synchronization Failed: No Response from
Network Element
S ALARM 28 6851 Config File Corrupted
Notes
12
Table of Contents
13
Notes
13
Introduction
This chapter contains troubleshooting procedures for system
initialization problems.
hapua: onGoActive
OMCR Commissioning
Problems
IMPORTANT
Table of Contents
Notes
14
Introduction
This chapter describes SDUrelated call processing failures.
Use
This section describes the basic subsections of call processing related to
a system with the SDU architecture. If either of these points fail, CFCs
and/or alarms will occur. In that case, follow the applicable
troubleshooting procedures for those CFCs/alarms. Some
troubleshooting actions specified here may also be used to supplement
those procedures.
14
SPROC MM RA/RM Links Not
Active
The following SPROC MM links must be active or else calls cannot be
established and CFCs and alarms will occur. If such CFCs and/or alarms
occur, follow the related troubleshooting procedures.
S MMSDFRALINK (used for SDF resource setup)
S MMPCFRALINK (used for PCF resource setup)
No PCF Resources
PCF resources must be available to establish new SDU 1X packet data
calls.
If a CFC indicates that there are no PCF resources available, check the
following (OMCR CLI commands can accomplish this):
S Verify that the PDSN exists.
S Verify that the PCF resources are INS.
S Verify that the PCF CPEs (Channel Processing Elements) are INS.
Each PCF card has a certain number of CPEs.
S Verify that the SDUPDSNLINK is INS.
Reverse Link
To determine the traffic being seen on the sniffer, examine the 4th byte
of the data field.
14
4th Byte (Frame Type):
00 = Idle
7E = Invalid Speech
01 20 = Valid Speech
Forward Link
5th Byte (Frame Type):
04 = 8th Rate Frame
Other = Valid speech frames will vary in rate
A Interface Problems
The A interfaces related to SDU are A8/A9 (between SDU SDF and
SDU PCF) and A10/A11 (between the SDU PCF and the PDSN).
The following steps must occur or else CFCs will occur and packet data
calls will not be established. In such a case, refer to the troubleshooting
procedure for the applicable CFC.
S A9 Setup A8 must be sent by SDF to PCF
S A9 Setup A8 must be received and seen by PCF
S PCF must send A11 Registration Request to PDSN
S PDSN must send an A11 Registration Reply
NOTE
For details on A interface call setup, refer to the
CDMA2000 1X RAN Functional Description.
Table of Contents
15
Notes
15
Overview
Introduction
This chapter contains CFC troubleshooting procedures for SDU.
NOTE
When displaying status data from the OMCR, it may be
helpful to log the data to a file, especially if you plan on
contacting CNRC.
Common Procedures
The following procedures are common throughout this chapter. Refer to
this section when performing these procedures.
IMPORTANT
To view errors and events logged by the BPP, use the spel <option>
command at the BPP command line.
The command format is spel <option> where <option> may be:
S celm: Change the Event Log filter settings
S cl: Clear event log
S cls: Change Logging State
S cmm: Change realtime MMI filter settings
S cfs: Change outgoing MMI filter settings
S le: Log event
S ler: Log error (Used internally by event logger)
S se: Show events
S sfe: Show filtered events
S ser: Show errors
S selm: Show current Event Log filter settings
S slogs: Show current logging states of MMI and EL
For example, to view events reported by the BPP:
spel se
For example, to view errors reported by the BPP:
spel ser
IMPORTANT
SdfCpePrintStore
Use the SdfCpePrintStore command to determine whether calls are being
allocated across all CPEs (Channel Processing Elements) on the SDF
card.
This command will also identify the IP addresses of the individual
CPEs.
At the SPROC, enter: SdfCpePrintStore
IMPORTANT
For example:
[SDUSPROC2]> SdfCpePrintStore
***** Printing Cpe Store for SDF ******
//
SDF Cpe Index = 0
status = INS
ipAddress = a01013d
capacity = 3200
availableBwu = 3200
usedBwu = 0
//
//
SDF Cpe Index = 1
status = INS
ipAddress = a01013e
capacity = 3200
availableBwu = 3200
usedBwu = 0
//
Causes:
Detailed Cause Description
0x1114 Layer 2 Failure Reported by SDU DSP
0x1115 Layer 2 Failure Reported by SDU L2
0x1119 RF Layer 2 Failure
Troubleshooting
S Dump the BPP SPEL Log
IMPORTANT
CFC=4 RF Loss
This CFC is generated when the SDF CPE detects an RF loss condition.
RF Loss is defined by L3CC on the SDF receiving a Traffic Channel
State Change indication for Invalid Speech (erasure) frames from a valid
speech state. This will start the RF Loss Timer, if valid speech frames
are not received before the timer expires, the call will be torn down for
RF Loss.
Causes:
Detailed Cause Description
0x111A RF Loss
Troubleshooting:
RF Optimization may be necessary if RF Losses occur often.
Causes:
Detailed Cause Description
0x111B No TCH Preamble Detected
0x2005 MccCpT1 Expired
Troubleshooting:
CFC 05 indicates either an RF environment problem or a possible
internal BTS problem. Use the following options to narrow down the
problem.
Problem is due to the RF environment:
S If possible, collect a mobile DM (Diagnostic Monitor) log to see what
information the mobile is seeing and what actions it is taking.
S Compare with an SMAP log to get the full picture of what is sent and
received on both ends.
S Messages not received/acknowledged on either side indicate RF
environment problems are occurring. For example, the BTS may not
have received the TCH Preamble frames from the mobile, or the
mobile may have not received the message from the BTS prompting
the transmission of TCH Preamble frames e.g. did not receive the
Channel Assignment/Extended Channel Assignment message.
Problem is due to the network:
S Collect a sniffer log to determine what bearer frames are present.
S Reset the BTS, force a conf load.
Causes:
Detailed Cause Description
0x111C No STRAU Synch
Troubleshooting:
S Collect a sniffer log to determine what bearer frames are present.
S Collect a sniffer log at multiple points to determine the source of
disconnect; from the Catalystt to the SDU ISB, and the SDU ISB to
the SDU BPP.
Causes:
Detailed Cause Description
0x1121 CP Timeout Awaiting MS Acquisition
Troubleshooting:
S If CFC 05 occurs, then CFC 07 will also occur. If this is the case,
instead troubleshoot CFC 05.
S If CFC 05 did not occur, and CFC 07 did occur, then there is probably
a disconnect:
Collect a sniffer log to determine what bearer frames are present.
Collect a sniffer log at multiple points to determine the source of
disconnect; from the Catalyst to the SDU ISB, and the SDU ISB to
the SDU BPP.
Causes:
Detailed Cause Description
0x114C SDF Call Setup Timeout
0x114D No Vocoder Connect Received at XC
0x114E No connect order received
0x154E SDF Assignment Timeout. No SDF Resource
Update is received at the SPROC before
Troubleshooting:
S Verify that the SDF Call Setup Timer is shorter than the Vocoder
Connect Timer at the XC before troubleshooting this failure, as this
can be a misleading failure cause.
S Dump some SPEL events on the BPP to see if any errors are being
reported by the SDF.
IMPORTANT
Causes:
Detailed Cause Description
0x1123 No service option ack received from MS
Troubleshooting:
S From a Mobile DM (Diagnostic Monitor) tool, determine if the MS
received the Service Option Response Order/Service Connect message
and is responding with a L2 Ack./Service Connect Completion
message.
S Compare with SMAP data to see what was actually sent and received
by the BTS.
S Check the value of the XcdrCkt. ServType in the Servopt database for
the specified service option.
S Check the value of the TchCkt. Serv Type in the Servopt database for
the specified service option.
S Check the value of the Sdf Service Connect Complete timer:
At the OMCR CLI: display sdusdf1 sdfsn
S Check the value of the Setup Action Time, SDF SN Setup A Timer.
S Determine if the SDU (SDF L3CC) is sending a Service Option
Response Order/Service Connect message. You can use SMAP to
monitor these messages.
S Dump some SPEL events on the BPP to determine if there are any
errors being reported by the BPP:
Check to see if Timer 23 (Sdf Service Connect Complete Timer) is
expiring. This cant be done on a system under call load.
Check that L2 received the Service Option Response Order/Service
Connect message. This cant be done on a system under load.
Check that the BTS received the Scap Service Configuration
Change Request message.
IMPORTANT
Troubleshooting:
S From a Mobile DM tool, determine if the Mobile received the Status
Request message at least 3 times.
S From the Mobile DM tool, determine if the Mobile responded to the
Status Request message with a Status Response/MS Reject Order
message.
S Also determine whether the BTS received the Status Response using
SMAP.
S Check the value of the Sdf Status Response timer:
At the OMCR CLI: display sdusdf1 sdfsn
S Dump some SPEL events on the BPP to determine if there are any
errors being reported by BPP.
IMPORTANT
S Collect a BTS TCH Audit Log to determine if the BTS is getting any
erased frames in the Status Response message.
S Check the RF Calibration to determine if the RF environment can be
further improved.
Causes:
Detailed Cause Description
0x1124 No mutual service option/configuration
Troubleshooting
S Verify the value of the Sdf Negotiation Timer.
S From a Mobile DM tool and SMAP determine if the BTS is sending
and the Mobile is receiving a Service Request/Response message.
S From the Mobile DM tool and SMAP, determine the response of the
Mobile to the Service Request/Response message and whether the
response is received by the BTS.
The Mobile can respond with a Service Response/Request or a MS
Reject order or send no response at all.
S From the Mobile DM tool determine if the Mobile is receiving a
Service Connect/Option Response Order message.
S From the Mobile DM tool and SMAP, determine the response of the
Mobile to the Service Connect /Option Response Order message and
whether the BTS receives the response.
S Using SMAP, check to see if the SDU (SDF L3CC) sends out a
Service Connect Message.
S In the servopt database, check the Alternative Service Option list of
the service option for which negotiation is taking place.
S In the servopt database, check the IS95 A/B and 1x Forward and
Reverse frame rates for the service option for which negotiation is
taking place.
S Using a sniffer, it is also possible to check to see if there are any
entries in the log which indicate the proposable Service Option List is
empty. Check the service option list supported by the Mobile by
examining the logging of the Status Response message.
Causes:
Detailed Cause Description
0x1148 No response from SDURA
Troubleshooting:
Collect a sniffer log to verify that the MM is sending resource requests
over SCTP.
Causes:
Detailed Cause Description
0x114A No SDF Resources available
Troubleshooting:
S If there is a congestion alarm occurring for the SDU, troubleshoot the
alarm.
S At the OMCR, use SDUrelated STATUS and DISPLAY commands
to verify BPP usage.
S Check the OMCR event logs to verify that the SDFRG (SDF
Resource Group) is not reporting capacity changes.
S Check the resource allocation at the SPROC, is it close to the limit for
the SDFs?
S At the SPROC, run SdfCpePrintStore . Are calls being
allocated across all CPEs?
IMPORTANT
Causes:
Detailed Cause Description
0x0101 MM RM Link Failure
0x0102 MM RM Session ID Change
0x0103 MM RM Deregistration
0x0104 MM RM ReRegistration Failure
0x154F SDU CP Internal Failure
0x1550 SDU CPE Failure
0x1551 SDU BPP Failure
0x154B PSISIG Failure
0x154C PSICE Failure
0x154D PSI SDU Failure
0x1552 Backhaul Bearer Loss
0x1553 Network Bearer Loss
Troubleshooting:
S Determine device failure based on detailed cause
S Collect log info (OMP logs, event logs, etc) dependent upon specific
device failing
S Address any alarms that are occurring. Follow the troubleshooting
procedures for the appropriate alarms.
Troubleshooting:
S Look at MM callproc logs.
S Use CDLs to try to isolate the problem to a device.
Causes:
Detailed Cause Description
NA No PSILINK in MM DB
NA No SDFRA in MM DB
Troubleshooting:
S Look at MM callproc logs.
S Use CDLs to try to isolate a device.
Troubleshooting:
S Look at MM callproc logs.
S Use CDLs to try to isolate a device.
Causes:
Detailed Cause Description
NA No PSILINK in MM DB
NA No SDFRA in MM DB
Troubleshooting:
S Look at MM callproc logs.
Causes:
Detailed Cause Description
0x1140 PCF Pkt Oriented Setup Timer Expired
Troubleshooting:
S Use a PDSN system monitoring tool to view where the A10/A11
(RP) control messaging experienced problems. This should help
isolate the problem to the PCF or the PDSN. Refer to the CDMA2000
1X RAN Functional Description for a detailed description of A10/A11
call setup processing.
S Investigate the status of the PDSN.
S Verify that the SDU PCF CPE IP addresses are in the PDSN VHA
pool. To determine the PCF CPE IP addresses, enter the following
command at the SPROC: SdfCpePrintStore
IMPORTANT
Causes:
Detailed Cause Description
0x1141 Protocol error between MS and BS
0x1143 TA8 Inactivity Timer Expired
Troubleshooting:
S Troubleshoot the PDSN.
S Get a sniffer log to look at PPP messaging. If the CFC occurred due
protocol error, this means that the PPP session did not establish
between the mobile and the PDSN due to the fault of the mobile or
PDSN. For a description of what successful PPP setup looks like
between the mobile and the PDSN, refer to the CDMA2000 1X RAN
Functional Description.
Causes:
Detailed Cause Description
NA MM SDFRA link is OOS
NA SDU is OOS (no MMSDU connection)
Troubleshooting:
S Troubleshoot any alarms for MMSDFRALINK (the call processing
link between the MM and SDU SDFRA). Refer to the applicable
alarm procedure.
S Troubleshoot the SDU OOS alarm. Refer to the applicable alarm
procedure.
S Verify you can ping the SPROC from the MM. Refer to your IP
planning documentation for the applicable IP addresses.
Causes:
Detailed Cause Description
0x114B cBTS Proxy Resource Request Timeout
0x114F cBTS Proxy Resource Configure Timeout
0x1150 cBTS Proxy Resource Mismatch
Troubleshooting:
S Consider increasing PSICE capacity.
S To verify actual capacity/usage, you can enter Disp_psice at the
XC OMP to determine PSICE availability. You can also get a sniffer
log to look at the actual cBTS Proxy messaging.
Causes:
Detailed Cause Description
0x1D79 PDSN Resources Not Available
Troubleshooting:
S Consider increasing PDSN capacity/resources.
Causes:
Detailed Cause Description
0x1D32 No PCF Resources available
Troubleshooting:
S Consider increasing SDU PCF capacity/resources.
Causes:
Detailed Cause Description
0x1D80 PDSN Reg Denied Reason Unspecified
0x1D81 PDSN Reg Denied Administratively Prohibited
0x1D82 PDSN Reg Denied Insufficient Resources
0x1D83 PDSN Reg Denied MS Failed Authentication
0x1D85 PDSN Reg Denied Identification Mismatch
0x1D86 PDSN Reg Denied Poorly formed request
0x1D88 PDSN Reg Denied Unknown PDSN Address
Troubleshooting:
S Examine the PDSN for possible failures. Refer to vendor
documentation for details.
SDU TROUBLESHOOTING
ENGLISH
APR 2004
68P09258A80B
SDU TROUBLESHOOTING
68P09258A80B
APR 2004
ENGLISH
SOFTWARE RELEASE 2.16.3.X
SDU
CDMA2000 1X
Technical Information Products and Services
Body: 70 lb.
Tabs: 110 lb. Index Clear Mylar Clear Mylar 3Hole Punched
NONSTANDARD SPECIFICATIONS
Tape Bound Corner Stitch
SPECIAL INSTRUCTIONS