So I should have posted these a while ago when I finished by CCNP Voice but better late than never

here are some

notes on the TVoice 8.0 subject material.
Test blueprint https://learningnetwork.cisco.com/docs/DOC-8990 (CCO Required)
4 Elements of CUCM Environment

Call Agent / CUCM

Network Infrastructure

Cisco Applications

CUPS

Unity Connections

UCCX
Voice Clients

Network Infrastructure

Routers

Switches

Firewalls

CUCM

Primary Call Agent

Voice Clients

Physical Phone

SoftPhone

3rd Party SIP Phone

Cisco Applications

CUPS

CUC

UCCX

Possible Sources of Problems

Complex deployments

Various protocols

Various endpoints and services

Network infrastructure

Benefits of Systematic troubleshooting method

Easier to identify potential problems

Learning effect

Understanding what is going on where there is no problem:

Tracking down internal processes such as call flows

Knowing corner case

Better ability to spot anomalies

Systematic documentation helps to solve future issues.

Problem Solving Model

Define Problem
Gather Facts
Consider Possibilities
Create Action Plan
Implement Action Plan
Observe Results
Did it resolve the problem no go back to start.
Yes
Problem resolved
Document the facts

Troubleshooting Tools

Cisco Unified Serviceability

Alarms

Setting Trace

CAR

Control Center

DNA

RTMT

Alerts

Viewing Traces

Syslog Viewer

Performance Monitoring

CLI

Show commands

Debug commands

Packet sniffer

Cisco Unified Operations Manager

Alarms
Local / External Syslog
SDI Trace
SDL Trace

Trace Information
SDI System Diagnostic Interface

Run-time events

IP Address

TCP Handles

Date / Time

SDL Signaling Distribution Layer

Call processing information from CUCM / CTI Manager

Used by Cisco TAC

Log4J Used for Java Applications

Two ways to configure traces

Activate Troubleshooting Traces
Configure Traces on a per server / per service level

Max number of devices traced (default is 12)

FQCN fully qualified calling number

CCM:StationInit

Sent from phone to CUCM

Button Presses
CCM|StationD
From CUCM to Phone

Change Phone display

Turn on MWI

CCM|Digit analysis
CCM|RouteList
CCM|RouteList
CCM|RouteListCdrc
CCM|RouteListCdrc

Dialed Number Analyzer

Used to find out if we could call from phone X to phone Y

RTMT Syslog Logs

System
Application
Security

Cisco Unified Reporting

GCFM Generic Call Filter Module
Call filter match-list (up to 16)
Used for filtering the debug output from a router for a specific ANI / DNIS

Phone registration process

Cisco ILP in line power

Fast Link Pulse FLP
802.3af POE

Most common problem is mismatch of inline power between endpoint and switch

Common DHCP issues

DHCP server down

DHCP does not have scope for the phone

All addresses leased out

IP Helper not setup

Common TFTP issues

Scope does not offer option 150

Option 150 points to wrong address

TFTP server on the address is stopped or hung

Try to Ping the phones IP address to see if the CM server can reach the phone via OS Admin or via CLI utils network
ping
Check TFTP config on phone / DHCP servers
Check Phone status messages

MGCP registers to CUCM, GW must be configured to reach out to CUCM and the domain name must be the same
as is configured in CUCM.

MGCP GW sends Restart in Progress

CUCM ACK
CUCM Audit Endpoint
GW ACK
CUCM Request Notification
GW ACK

GW Registered

MGCP Problems
IP connectivity problems
Wrong TFTP server / CUCM not running TFTP

IOS SW version incompatible

Missing or incorrect CUCM server
Incorrect IOS configuration

Show ccm-manager shows MGCP status

Show mgcp endpoint shows the endpoints that are controlled by ccm
Show mgcp connection shows the active connections (calls)
Show mgcp statistics shows the mgcp message stats
Show mgcp srtp displays information about SRTP connections
Debug mgcp enables debugs for the mgcp protocol
Debug ccm-manager events see the events as the MGCP gateway is attempting to register

H.323 / SIP do not register to CUCM. CUCM will always show unknown.

Dial peer monitoring

Show dial-peer voice
Debug voip dialpeer
H.323 GW monitoring
Show gateway
Debug h225 asn1 | events
Debug h225 q931
SIP GW Monitoring
Show sip-ua calls, connections, statistics, status
Show sip service

Delayed dial tone / CCM Admin page slow CUCM server over burdened

CPU Usage high

Service crash

Service Hung

Memory Leak

Syslog messages can be used to view alerts

RTMT can be used to view CPU / Memory Utilization

Phones not registering

Maybe CCM service is not running or TFTP service isnt running

Cant reach CCM admin page

Tomcat might not be running can be restarted via CLI
IS DNS resolving the ccm host name
Can we ping the CCM server
ACL / FW change?

Utils service list

Utils start Cisco Tomcat

Check Duplex / Speed for issues related to slow network

DB replication issues
Pub has a R/W copy of the DB Subs have R/O copy of the DB

Replication issue example: Changed phone setting, but it never shows up

IDS Replication From PUB to SUBS
Static information phone configuration
ICCS (Intra-cluster communication Service) Full mesh between all call processors (subscribers)
Dynamic runtime information

Why DB goes out of sync

PUB cant reach subscribers network issues
QoS not setup to prefer DB replication traffic
DNS issue
Server CPU is taxed

Error occurred during the DB replication process

Verify NW connectivity
Verify DNS resolution works
Make sure there is enough BW for replication and the RTT not greater than 80ms between pub and sub

RTMT / CLI can be used to verify replication state.

Replication state should be 2 which is good replication
Cisco Unified Reporting Unified CM Database Status

Utils dbreplication status

File view activelog cm/trace/.. To view log file
Utils dbreplication runtimestate shows realtime replication status across the servers

Utils dbreplication repair if a status of 0 or 4 after 4 or more hours use dbreplication reset to stop and restart
database replication.

Run the repair command at the publisher only

Wait until completed displayed from runtime state.

Go to subscriber, utils dbreplication stop stops the replication on the subscriber

Go to Publisher, utils dbreplication reset (subscriber name)

If that doesnt work a cluster reset is needed.

Stop dbreplication on all nodes

From the pub, utils dbreplication cluster reset

Troubleshooting LDAP Integration Issues

Can sync and auth against LDAP

Cisco DirSync must be activated / running

Cant mix LDAP server types

When user deleted in LDAP next CUCM sync will tombstone the user object. The user will be deleted after 24 hours.

Common issues:
Network connectivity
Utils network ping
Service username and password wrong
check
Service user doesnt have sufficient permissions
Needs read access to all AD user objects
Search base wrong
Check the DN
DirSync set to manual or too long a period
Check LDAP sync schedule in CUCM LDAP directory setup
DirSync not running
Check that DirSync service is activated and running on publisher
Some end-users missing
Check the search base would encompass where the user is located in LDAP

DirSync has Service Parameters that can be set.

H.323 GW issue
Call into H323 gateway just gives secondary dial-tone
Direct-inward-dial missing from inbound dial-peer

Call Setup issue

Fast busy
Missing / wrong Caller ID
No ring back

Dead air
One-way calling
Inefficient call routing
Unexpected second dial tone

Reasons:
Bad CSS / wrong dialed number
Bad digit manipulation
Phone unregistered
Codec mis-match / region misconfiguration
No MTP resources
Bad QoS / no QoS
Configured wrong in CUCM
RSVP configured wrong
CAC blocking call due to no enough BW
WAN having problems
ICT setup wrong between clusters
CCD not configured right
CUBE configured wrong

Single site call setup issues

CoS misconfigured
Bad digit manipulation
FW to VM and VM is down
Unregistered phone

CUCM call routing

Digit by digit analysis most specific match logic

IP Phone
SCCP Digit by Digit

SIP En bloc / KPML / SIP Dial Rules

Gateway
MGCP / SIP / H323 En block / overlap send and receiving

Trunk
SIP / H323 En bloc / overlap sending and receiving

Partitions and CSS allow / deny calls

A device can only call those numbers that are in PT that are a part of its CSS

<None> PT is always accessible regardless if the calling device has a CSS

Devices that have the <None> CSS can access only other devices that are in the Partition of <None>
CSS is an ordered list of PTs
If no best match is found in a CSS search or they are the same then the partition with a partial match that is listed first
wins.

Line / Device CSS is concatenated and Line is used first then device.
TOD applied to a PT effectively makes the PT disappear when its outside of the schedule.

You can use SDI traces to find CSS problems.

TCP handler assigned to the phone when registers identifies specific phone in trace output.
PSS = Partition search string

One Way Calling

Phone A can call phone B but phone B cannot call Phone A
Potential CSS issue on Phone B that doesnt include the PT of Phone As DN.

Phone A cannot forward calls to another phone.

CSS on the I phone does not include the destination DN
Destination is invalid

Specified Destination is unregistered

Cant forward to voicemail when not answered

CFNA is invalid or not specified
VM profile is wrong
VM ports are not available / busy

Can be viewed via RTMT by looking at the Hunt List for the voicemail HL

On-net multi-site calling issues

Overlapping dial plan
Site codes can be used to overcome digit overlap
Toll bypass settings
Local end CoS
Remote end CoS
Inter-cluster trunk settings
QoS
CAC Mechanisms
WAN Problems
QoS issues with signaling across WAN
CAC Mechanism prevents the call
Issues with CCD

PCMU G711 codec in SIP SDP

Trunks between CUCM

GK could stop calls vi CAC
Trunk may not have registered to GK
Bad digit manipulation

CUBE issues:

Misconfiguration
MTP not allocated for H.323 to SIP
CSS and partition issues
Incorrect digit manipulation
CAC failure

CCD Issues
Misconfigured SAF trunks
CSS and partition issues
Patterns not advertised
Digit / pattern manipulation issues
Maximum number of learned patterns exceeded

GK Functions
ARQ Can I place the call
ACF Yes you can place the call and here is the IP you should send call setup to

GK Issues
Config errors
Registration issues
Duplicate ID
Terminal excluded
Security denial
Invalid alias
CAC issues
ACF received but get a busy tone
ARJ is null, not enough bandwidth
ARJ received, called party not registered
No tech prefix or no E.164 address for the call

Debug h225 asn1

Show gatekeeper endpoints

Debug ras
Debug gatekeeper main 5
Show gatekeeper calls
Show gatekeeper status
Show gatekeeper zone prefix

CUCM not registered to GK

Network issues
GK misconfigured
Endpoint misconfigured
Duplicate H.323 ID
Endpoint not authorized to register

CUCM registered to gK but phones cant make calls

Connectivity between cluster issues
Dial plan misconfiguration
Incorrect IP address returned
Called party is not registered
Insufficient bandwidth due to CAC

GK calculates BW as double the Payload e.g. G729 8k time 2 = 16K and G711 128K

CUBE issues
CUCM misconfigured
CUBE misconfigured
Their configuration does not match
MTP not allocated, DTMF / Codec mismatch
CSS / PT issues
Incorrect digit manipulation on CUCM or CUBE
Call capacity exceeded, CAC failure

Show call active voice brief

Show voip rtp connection

Allow-connections are unidirectional

Allow-connections h323 to sip
Allow-connections sip to h323

Common off-site calling issues

Gateway config errors
Dial plan config errors
Route plan errors
Codec Issues
Location issues
GW issues
IOS config error
COR errors
Problems with QoS
Trunk issues
Invalid number dialed

MGCP GW troubleshooting
Go to router and verify mgcp commands
Ccm-manager commands
If PRI is Q931 backhauling configured

On CCM is the GW hostname configured properly

Check the endpoint configuration
Show ccm-manager
Show mgcp endpoint
Show mgcp statistic

Show mgcp connection

Debug mgcp error
Debug mgcp packet
Debug mgcp state

H323 GW Tshoot
Verify that the dial-peer commands are right
Verify that the voice class h225 timeout for tcp is set to 3 seconds
Use the preference command to determine the CUCM server order
Check that h323 binding is configured
Check RP / RG / RL settings in CUCM
Check the GW IP is correct in CUCM

Show dialplan number <number>

Debug vtsp session
Debug vtsp dsp
Debug voice dial
Show isdn status
Debug isdn q931
Debug voice ccapi inout
Debug voip dialpeer inout
Debug cch323 h225
Debug h225 q931

SIP GW troubleshooting
Voip dialpeers are configured correctly and have sipv2 enabled as well as a valid session target
SIP UA retry settings are verified
Proper DTMF relay is selected
SIP trunk in CUCM points to GW IP address
Show sip-ua status
Show sip status

Show sip statistics

Debug ccsip calls

Overlap receive is digit by digit analysis

When digit manipulation is done at RG level is not displayed on phone or in CDR

PT-S Pretransformed Source

PT-D Pretransformed Destination
T-S Transformed Source
T-D Transformed Destination

No digits at RP level over rides DDI at RG level

D Device
G Route Group
L Route List
P Route Pattern

Test voice translation-rule

No ringback on ip phone
PSTN is not providing ringback
H.323 GW is not cutting through audio

Dead air issue

Routing issues
IP ACL
Firewall fixup problems

Call drops midway

GW lost communication with CUCM

Remote end accidently hung up

The network had a connectivity event that affected the RTP path
System or Software error occurred

Outside dial tone only provided when all patterns left that possibly match have provide outside dial-tone

Issues with globalized call routing

Normal call routing issues plus
Unreachable internal number when calling inbound
Call back not possible
Bad transformations

Common SAF CCD issues

SAF no enabled
Wrong SAF external client credentials
Interface hellos are deactivated
SAF forwarders are not layer 2 adjacent, or wrong static neighbor configuration in place
Network issues
FW blocking traffic

External SAF client Is on a different platform than the SAF forwarder (CUCM)
Internal SAF client is on the same platform as the SAF forwarder (CUBE, CME, SRST)

Is SAF client registered?

Show eirgrp service-family ipv4 clients (detail)
RTMT can show SAF client statistics

Debug eirgrp service-family external-client messages

Show eigrp service-family ipv4 topology

Show eigrp service-family ipv4 events

Verify CCD learned patterns via RTMT > CUCM > Learned Patterns

Show voice saf dndb all

Patterns learned but marked as unreachable
Means there may be WAN issues
Wrong learned pattern prefix
Patterns placed in a PT not accessible via the CSS of the phones

The roaming sensitive settings applied if the physical location is different but the DMG is the same
The device mobility related settings are applied if the physical location is different but DMG is the same
The device mobility related settings do not change if the DMG is different

Device mobility CSS only affects the device CSS never the line CSS

Device mobility issue

Device mobility set to off on phone or service parameters
Problems with IP subnet configuration
Problems with CAC / codecs
SRST reference issues
Media resource Problems
QoS
CoS issues

To troubleshoot Device mobility look at the applied settings on the phone

Use CUCM DNA to check dial plan results

Line / device CSS necessary to ensure proper CSS when roaming.

Device default profile used when user profile does not match phone

Extension Mobility Issues

Various login / logout problems
Phone button problems
Phone service problems
Call routing problems
CoS problems
EMCC issues

Phone restarts instead of resetting usually caused by different User Locale than phone.
Line / Device CSS needed to ensure proper CSS configuration

Cisco Unified Mobility

If were using MVA the MVA service must be enabled
Must have mobility softkey published to phone
Dusting (*74) feature allows call to be moved from Remote destination to the desk phone without holding the caller.

IF you cant access MVA you probably have an VXML error on the gateway or wrong number assigned to the MVA
app.
If you cant reach destination numbers with MVA it may be an issue with the inbound CSS on the MVA gateway.
Or bad Dial-peer on the gateway.

Cant use enterprise features.

Enterprise features are turned off on the service parameter
DTMF relay wrong on the gateway.

Three presence states

Unregistered
On-hook
Off-hook

BLF speed dial and presence in call history lists

Type A phones do not support directory

Presence groups only affect Directory Lists

Subscribe CSS affects the BLF subscription field

For presence to work across SIP trunks

Accept Presence Subscription
Accept Unsolicited notification
Subscript CSS on the trunk
Presence Group on the trunk

BLF for Call lists enable at the enterprise parameter level

Unicast MoH doesnt scale well and is BW intensive.

Most common MoH issues

MoH resource not registered
MOH resources are currently in use
Media resource misconfigured
TOH is heard instead of MOH
Call disconnected when placed on hold
MOH audio is poor
Multicast MOH is expected but unicast gets used

RTMT can show MOH streams in use

Tone on Hold when it should be music

Region issue
CAC blocking the BW for MOH

Audio streams not available on the MOH server

MOH server not active
Media resources misconfigured

Debug IP PIM see multicast advertisements

Show ip mroute shows multicast routing table

MTPs are used for repacketization a-law to u-law

Bridge connections using the same codec but different sample sizes
H.323 supplementary services
H.323 Outbound fast start
MTP needed when endpoints do not have a common method for sending DTMF between them.

Two non-sip endpoints do not require MTP

SCCP Phone and H323 GW
Two SIP endpoints do not require MTP
All Cisco SIP endpoints support NTEs (RFC2833)
DTMF is sent direction between the endpoints using NTE
No MTP required for G711 Calls
Combination of SIP and non-sip endpoints might require MTP
Depends on the endpoint
CUCM dynamically allocates MTPs on a call by call basis.

Issues related to MTP

No supplementary services available on H.323
Call setup fails when MTP required
DTMF issues when NTO not supported by an endpoint
Issues with mixed SIP endpoints

General MTP issues

MTP cannot register to CUCM misconfigured / network problems.

MTP registered but not available to calls because of running out of resources / misconfiguration

Show dspfarm profile can be used to see available HW MTP resources

Show sccp connections shows mtp / sccp connections on a router

Meet-me
Dedicated DN for conference calls
Access controlled by CSS

Conference originator cannot add participants to Ad Hoc conference

Cant link conferences together
Cant setup meet-me conference
Cant join a meet-me conference
Conference participants drop

SCCP media resources are case sensitive

DSPfarm conference is shutdown by default

Conference originator determines the MGRL

Ad-hoc conf issues

No conf softkey
No conference bridge message shown on phone no mrgl / no available bridges
Maximum number of participants reached
Conf bridge doesnt support codec
CAC rejects the conference bridge leg
Conf bridge out of resources
Cisco IP Voice Media Streaming App service not running
Advanced ad hoc not supported

Network issues

RTMT has conference performance counters

An end user can not join a meet-me conference via the softkey. Only the originator needs to press the meet-me conf
key.

Show dspfarm dsp all

Device that cannot satisfy the end to end requirements (enforced by regions) is used. The device that requires the
higher codec will invoke the transcoder.
Transcoder issues
Calls between endpoints with different codecs setup fails
Transcoder is not registered to CUCM
Misconfigured
Network issues
Transcoder registered but is running out of resources / misconfigured
If the Cisco IP Voice media streaming app is not running the transcoder will not register
DSPFarm could be admin down
MRG / MRGL misconfigured

RTMT can show current Transcoder resources available

MTP is used to represent an RSVP agent

RSVP Agent problems
RSVP CAC blocks calls when BW available
RSVP CAC does not make BW reservations when expected
No reservation
RSVP CAC not enabled
RSVP agents cannot register.

MTP SCCP admin state should be up and RSVP should shoe enabled

If call cannot be made

Too many reservations
IP phones are not associated to RSVP agents
Current IP paths goes over network segments that dont have enough BW
RSVP per-flow BW needs to be set to 16kb/s more than actual codec will really require
Network connectivity issues

RTMT can be used to show RSVP connections and available BW / out of resources counters

Debug ip rsvp resv can be used to see RSVP messages

Show ip RSVP neighbor will show known enabled RSVP neighbors
Show ip rsvp installed detail

Local RSVP within cluster only

E2E means across SIP trunk to another cluster end to end RSVP
Send PRACK Enabled

SIP Preconditions call fails

Not enough BW
Phones arent associated with RSVP
RSVP configured wrong
Network connectivity problems
Dial-plan problem

Delay below 150ms one way is normally acceptable for voice

Jitter represents the variation in the delay of received packets
Echo most commonly caused by impedance mismatch between 2-wire and 4-wire interfaces.

Latency < 150ms one way

Jitter < 30ms one way

Loss < 1% one way
17-106 Kb/s guaranteed priority BW
150 b/s + layer 2 overhead guaranteed BW for voice control traffic per call.

QoS Policy Implementation

Classification ACL
Marking DSCP tag (EF, CS4)
Congestion management Policy-map
Congestion avoidance WRED
Policing and shaping
Link efficiency LFI / Compression etc.

Layer 2 QoS issues

Buffer congestion is an issue
Make sure voice is mapped to the expedited egress queue
Enable the priority queue ingress and egress on the switch

3T represents the drop threshold

SRR Shaped round robin
Shaped mode every queue get a certain % of the BW and are rate limited
Shared mode minimum guarantee of BW and allowed to expand.

ngress / egress queues use weight tail drop

Uses the frames assigned QoS label to subject it to different thresholds

01-03 = Queue 1 (priority) Threshold 3 (least droppable)

You can use the QRT softkey to collect information about a poor quality call.
Show call active voice can show if the DSP is filling gaps in the stream of a call

Echo
Loudness on talkers device set too high high input level
Call proceed through tail circuits that do not use echo cancellers
Call path has end to end delay that is not covered by echo cancellers
Attenuation misconfigured on voice gateways

Identify types of echo, loud, long or acoustic.

Try first to move the remote phone from acoustic sources
Try replacing speakerphone or handset with better quality and see if it clears

ERL Echo Return Loss

One-way audio
cRTP on one end but not both of the links
ACL blocking return RTP traffic
Early versions of NAT could block RTP
Routing issues