CIS 188 CCNP TSHOOT (Troubleshooting)

Chapter 1 Planning Maintenance for

Complex Networks
Rick Graziani
Cabrillo College
graziani@cabrillo.edu
Fall 2014

Chapter Roadmap
Applying Maintenance Maintenance Processes and
Methodologies
Procedures
Maintenance Models Network Task Identification
and Methodologies Network Maintenance
Determining
Planning
Procedures and Tools
Scheduling maintenance
for Maintenance
Formalizing ChangeModels
Control Procedures
Establishing Network
Documentation
Procedures
Establishing Effective
Communication
Defining Standardization
Planning for Disaster
Recovery
Network Monitoring and
Performance Measurement

Network Maintenance
Tools, Applications
and Resources
Fundamental Tools,
Applications and
Resources
CLI, debug, GUI,
Backup Servers, Log
Servers, Time
Servers
Configuration and
Documentation Tools
Logging Services
Network Monitoring and
Performance
Measurement Tools
Implementing Backup
and Restore
2
Disaster Recovery Tools

Without these processes and tools

Network engineers can get into trouble.

3

Radia Perlman

Troubleshooting analogy (life lessons)

Before solving the problem, know the problem you are trying to
solve.
4

Planning Maintenance for Complex Networks

Configuration and implementation is sexy

Planning and documenting, not as much.
5

Applying Maintenance
Methodologies
Maintenance Models and Methodologies
Determining Procedures and Tools for Maintenance
Models

Applying Maintenance Methodologies

Support and maintenance

Objective: Keep the network available with minimum service
disruption and at acceptable performance levels.
Solution: Structured network maintenance
7

Maintenance Models and Methodologies

Network engineers job description assumes these tasks:

Device installation and maintenance
Failure response
Network performance
Business procedures
Security

Interrupt Driven

In many smaller networks, the process is largely interrupt driven.

User has a problem
Application performance problems
Security incidents are reported
Disadvantage
Other tasks beneficial to the long-term health of the network may be delayed
or not done.
Tasks get done in the order requested instead of priority or urgency.
May result in more network downtime in the long run.
Cant avoid interrupt driven work entirely because of failures.
Can reduce them by proactively monitoring and managing the network. 9

No one every calls PG&E to say Thanks,

the lights are on!

10

Structured Network Maintenance

Alternative to interrupt-driven is structured network maintenance.

Advantages
Reduced Network Downtime: Discover problems before they happen
More cost effectiveness: Adequate budget for networking needs and
choosing proper equipment.
Better alignment with business objectives: Upgrades and major
maintenance jobs are not scheduled during critical business hours.
Higher network security: Observe network vulnerabilities and needs and
plan for strengthening network security.

11

Structured Network Methodologies

Several well-known methodologies:

IT Infrastructure Library (ITIL)
FCAPS - ISO (Fault management, Configuration management,
Accounting management, Performance Management, and
Security Management)
Telecommunications Management Network (TMN) - ITU-T
Cisco Lifecycle Services

12

Chapter Roadmap
Applying Maintenance Maintenance Processes and
Methodologies
Procedures
Maintenance Models Network Task Identification
and Methodologies Network Maintenance
Determining
Planning
Procedures and Tools
Scheduling maintenance
for Maintenance
Formalizing ChangeModels
Control Procedures
Establishing Network
Documentation
Procedures
Establishing Effective
Communication
Defining Standardization
Planning for Disaster
Recovery
Network Monitoring and
Performance Measurement

Network Maintenance
Tools, Applications
and Resources
Fundamental Tools,
Applications and
Resources
CLI, debug, GUI,
Backup Servers, Log
Servers, Time
Servers
Configuration and
Documentation Tools
Logging Services
Network Monitoring and
Performance
Measurement Tools
Implementing Backup
and Restore
13
Disaster Recovery Tools

Maintenance Processes and Procedures

Backup and
Recovery

Network engineers need to do the following:

Identify essential network maintenance tasks
Recognize the advantages of scheduled maintenance
Evaluate the key decision factors that affect change control
procedures
Describe the essential elements of network documentation
Plan for disaster recovery
Describe the importance of network monitoring and performance
measurement

14

Network Maintenance Task Identification

Network maintenance plans need to include procedures for:

Accommodating Adds, Moves, and Changes
Installation and configuration of new devices
Replacement of failed devices
Backup of device configurations and software Troubleshooting link
and device failures
Software upgrading or patching
Network monitoring
Performance measurement and capacity planning
Writing and updating documentation

15

Network Maintenance Planning

Scheduling Maintenance
Interrupt driven or part of maintenance cycle
Schedule guarantees that these tasks will be done regularly and will
not get lost in the busy day-to-day work schedule.
Change requests need not be handled immediately, but during the
next scheduled timeframe.
Allows you to properly prioritize tasks
Scheduled maintenance, tasks that are disruptive to the network
are scheduled during off-hours.

16

Formalizing Change-Control Procedures

Any change you make has an associated risk due to possible

mistakes, conflicts, or bugs.
Determine the impact of the change on the network
Balance this against the urgency of the change
High-impact changes are usually made during maintenance
windows that are specifically scheduled for this purpose

17

18

Formalizing Change-Control Procedures

Which types of change require authorization?
Which changes have to be done during a maintenance window and
which changes can be done immediately?
What kind of preparation needs to be done before executing a
change?
What kind of verification needs to be done to confirm that the
change was effective?
What documentation or communications need to be taken after a
successful change?
What actions should be taken when a change has unexpected
results or causes problems?

19

Establishing Network Documentation Procedures

Network drawings: Physical and logical structure

Connection documentation: Patches, connections to providers, and power
circuits
Equipment lists: All devices, part numbers, serial numbers, installed
software versions, software licenses, warranty/service information
IP address administration: IP subnets scheme and all IP addresses
Configurations: All current device configurations and archives
Design documentation: The why did we do that?

20

21

Establishing Effective Communications

Network maintenance is typically a job that is performed by a team

Communications is key
Who is making changes and when?
How does the change affect others?
What are the results of tests that were done and what
conclusions can be drawn?
One team member can be disruptive to the process handled by
another team member

22

Communications is Key!

23

Standardization

Important that the tasks are performed consistently:

Are logging and debug timestamps set to local time or
Coordinated Universal Time (UTC)?
Should access lists end with explicit "deny all"?
In an IP subnet, is the first or the last valid IP address allocated
to the local gateway?
Different methods can lead to confusion, especially during
troubleshooting.

24

Planning for Disaster Recovery

Always consider the possibility of device failure

Building redundancy into the network at critical points and
eliminating single points of failure.
Due to budgetary limitations, it is not always possible to make every
single link, component, and device redundant
Natural disasters: flood or fire in the server room
25

Planning for Disaster Recovery

To replace a failed device:

Replacement hardware
The current software version for the device
The current configuration for the device
The tools to transfer the software and configuration to the device
Licenses (if applicable)
Knowledge of the procedures to install software, configurations
and licenses

26

Network Monitoring and Performance

Measurement

Helps you transform your network maintenance process to a less

interrupt-driven, more methodical approach
Important to choose the variables to be monitored and measured.
Interface status
interface load
CPU load
memory usage
More sophisticated metrics include delay, jitter, and packet loss

27

Cabrillo CIS/CS Datacenter Documentation

Project?

28

Chapter Roadmap
Applying Maintenance Maintenance Processes and
Methodologies
Procedures
Maintenance Models Network Task Identification
and Methodologies Network Maintenance
Determining
Planning
Procedures and Tools
Scheduling maintenance
for Maintenance
Formalizing ChangeModels
Control Procedures
Establishing Network
Documentation
Procedures
Establishing Effective
Communication
Defining Standardization
Planning for Disaster
Recovery
Network Monitoring and
Performance Measurement

Network Maintenance
Tools, Applications
and Resources
Fundamental Tools,
Applications and
Resources
CLI, debug, GUI,
Backup Servers, Log
Servers, Time
Servers
Configuration and
Documentation Tools
Logging Services
Network Monitoring and
Performance
Measurement Tools
Implementing Backup
and Restore
29
Disaster Recovery Tools

Network Maintenance Tools, Applications

and Resources

Choose the tools, applications and resources for doing network

maintenance in an efficient manner.
Commands
Software
Servers
Some of these we will take a brief look at in this chapter and more
detail later.
Some in later chapters.

30

Basic components of
a network
maintenance toolkit

The basic components of a network maintenance toolkit are:

CLI device management
show commands
debug commands
Embedded Device Manager (EEM)
IP SLA commands

31

Basic components of
a network
maintenance toolkit

The basic components of a network maintenance toolkit are:

Graphical User Interface (GUI) based device management (free)
Cisco Configuration Professional (CCP)
Secure Device Manager (SDM)
Cisco Configuration Assistant (CCA)
Cisco Network Assistant
CiscoWorks (not so free)

32

Basic components of
a network
maintenance toolkit

The basic components of a network maintenance toolkit are:

Backup server
TFTP
FTP (FTPS)
HTTP (HTTPS)
Secure Copy Protocol (SCP)

33

Basic components of
a network
maintenance toolkit

The basic components of a network maintenance toolkit are:

Log server
Syslog

34

Basic components of
a network
maintenance toolkit

The basic components of a network maintenance toolkit are:

Time server
Network Time Protocol (NTP)

35

Brief look at NTP

Vital that the clocks of the network devices are properly set and
synchronized.
Ensures correct timestamps on logging and debug output.
Stratum 1 server: a server that is directly connected to an
authoritative time source such as a radio or atomic clock (
www.time.gov and www.worldtimeserver.com/atomic-clock ).
Statum 2 server: A server that synchronizes its clock to a stratum 1
server.
Etc.

36

Brief Look at NTP

service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime localtime show-timezone
!
clock timezone PST -8
clock summer-time PDT recurring 2 Sun Mar 2:00 1 Sun Nov 2:00
!
ntp server 10.1.220.3

Uses the ntp server command

Clock is synchronized to a single timeserver with IP address 10.1.220.3.
Time zone is Pacific Standard Time (PST), -8 hour offset to Universal Time
Coordinated (UTC).
Change to daylight savings time on the 2nd Sunday in March at 2:00 am
Change back to standard time on the first Sunday in November at 2:00 am.
System logging is configured to use the local date and time in the time
stamps and to include the time zone in the time stamp.
For log entries generated by debugs, the settings are similar, but
milliseconds are included in the timestamps for greater accuracy.

37

NTP Online Sample Chapter

http://oreilly.com/catalog/hardcisco/chapter/ch10.html
38

Configuration and Documentation Tools

Many web-based (online) maintenance tools and resources that can

be helpful during the planning and implementation of network
maintenance procedures.
Configuration and Documentation Tools
Dynamic Configuration Tool
Cisco Feature Navigator No longer required with IOS 15
SNMP Object Navigator
Cisco Power Calculator

39

Configuration and Documentation Tools

Dynamic Configuration Tool: This tool aids you in creating

hardware configurations.
It verifies compatibility of the hardware and software you select, and
it gives you a complete Bill of Materials (BoM) that lists all the
necessary part numbers.
https://apps.cisco.com/qtc/config/html/configureHomeGuest.html

40

Configuration and Documentation Tools

Cisco Feature Navigator: This tool allows you to quickly find the
right Cisco IOS software release for the features you want to run on
your network.
Not needed with IOS 15
http://tools.cisco.com/ITDIT/CFN/

41

Configuration and Documentation Tools

SNMP Object Navigator: The Simple Network Management Protocol

(SNMP) Navigator translates SNMP Object Identifiers (OID) into object
names.
This tool also allows you to download SNMP Management Information Base
(MIB) files and to verify the supported MIBs in a particular Cisco IOS
Software version.
http://tools.cisco.com/Support/SNMP/do/BrowseOID.do?local=en

42

Configuration and Documentation Tools

Cisco Power Calculator: This tool calculates the power supply

requirements for a particular Power over Ethernet (PoE) hardware
configuration.
Needs Cisco CCO account login

43

Configuration and Documentation Tools

Documentation tools:
Wiki: A wiki combines easy web-based access with intuitive editing
capabilities.

44

Documentation

Issue tracking system: A.k.a. trouble ticket, support ticket, or

incident ticket system.
Allows issues to be:
Logged, tracked, and documented.
Better communications and escalation of incidents
Allows a team of people to work on the same incidents in an
efficient manner.
Can build a historical database of problems, their treatments, and
the resolutions.

45

Logging Services
Events on networking devices can be logged.
Various events
Various levels of severity
Events are logged to:
Console (default)
Console display
Buffer
Server
Examples
Interfaces up or down
Configuration changes
Routing protocol adjacencies

46

Logging
Services

Logging severity levels on Cisco Systems devices are as follows:

(0) Emergencies
(1) Alerts
(2) Critical
(3) Errors
(4) Warnings
(5) Notifications
(6) Informational
(7) Debugging
By default, all messages from level 0 to 7 are logged to the console

47

Logging Services

Console
You can also adjust the logging severity level of the console.
By default, all messages from level 0 to 7 are logged to the console;
You can configure the severity level as an optional parameter:
loggingconsolelevel
Limits the logging of messages displayed on the console terminal to the
specified level and (numerically) lower levels more severe.
You can enter the level number or level name.

48

Logging Services

Buffer
loggingbuffered[buffersize|level]
May or may not be the default
By default, messages of all severity levels are logged to buffer.
showloggingDisplays the content of the buffer
The buffer is circular, meaning that when the buffer has reached its
maximum capacity, the oldest messages will be discarded to allow the
logging of new messages.
49

Logging Services

Server
loggingipaddresscommand
Some IOS version it is logginghost
By default, only messages of severity level 6 or lower will be logged to the
syslog server.
This can be changed by entering the logging trap level command.
50

Network Monitoring and Performance

Measurement Tools

GUI- and CLI-based device management tools are used to examine

individual devices after the problem is noticed.
Network monitoring system continuously checks your network devices
availability and status:
Detect possible problems as soon as they occur
Sometimes before they even become apparent to end users
Uses protocols such as SNMP and ICMP
Cisco IOS Netflow technology can be leveraged to monitor devices and traffic
Gray area between network monitoring and performance measurement

51

Motivations for
measuring network
performance

The three main motivations for measuring network performance are as follows:
Capacity planning:
Create a baseline network traffic
Recognize trends in traffic growth
Predict when you need to upgrade links before congestion and performance
problems.
Diagnosing performance problems: difficult to troubleshoot because hard to
quantify and often intermittent in nature.
Application X is really slow lately.
What is causing the problem? Where is it occurring?
SLA compliance: Guaranteeing a level of service to others through an SLA or
certain level of service by a provider, need to have a method to measure.

52

Measuring network
performance

Typical statistics gathered include:

Packet and byte counters on interfaces
Device CPU and memory utilization
Round Trip Time (RTT)
Jitter
Packet loss
Analyzed or graphed using products such as:
Cisco Internetwork Performance Monitor (IPM) - part of CiscoWorks
LAN Management Solution
Multi Router Traffic Grapher (MRTG)

53

Implementing Backup and Restore Services

Essential element of any network maintenance toolkit

Simplest and most commonly implemented service is TFTP
No configuration on network devices
More secure protocols such as FTP, SCP, and HTTP or HTTPS
For all of these protocols, the credentials can be specified as
part of the Uniform Resource Locator (URL) that is used with the
copy command.
54

Copy command
R1# copy startup-config ftp://backup:san-fran@10.1.152.1/R1-test.cfg
Address or name of remote host [10.1.152.1]?
Destination filename [R1-test.cfg]?
Writing R1-test.cfg !
2323 bytes copied in 0.268 secs (8668 bytes/sec)

copy[/erase]sourceurldestinationurl
Copy the startup-configuration to a FTP server at 10.1.152.1
Create a file named RO1-test.cfg
username backup and password san-fran
The username and password are specified by placing the username and
password as username:password@before the server name or IP
address in the URL.
For SCP, HTTP and HTTPS you would use a similar syntax, replacing the
URL prefix ftp:// with scp://, http:// or https://
55

Copy command
R1(config)# ip ftp username backup
R1(config)# ip ftp password san-fran
R1(config)# exit
R1# copy startup-config ftp://10.1.152.1/R1-test.cfg
Address or name of remote host [10.1.152.1]?
Destination filename [R1-test.cfg]?
Writing R1-test.cfg !
2323 bytes copied in 0.304 secs (7641 bytes/sec)

Specifying the username and password on the command line is

somewhat cumbersome and suffers from the fact that the password
is displayed in clear text on the screen.
username and password can be specified in the configuration

56

Archive command
R1(config)# archive
R1(config-archive)# path flash:/config-archive/$h-config
R1(config-archive)# write-memory
R1(config-archive)# time-period 10080
Configuration Replace and Configuration Rollback A feature for the creation of
configuration archives, introduced in Cisco IOS Software Release 12.3(7)T.
The only mandatory parameter is the base file path.
Local or network path
Optional variables:
$h for the devices hostname in the filename
$t to include a time and date stamp in the filename
writememoryoption - Triggers an archive copy of the running configuration to
be created any time the running configuration is copied to NVRAM.
timeperiodminutesoption - Each time the time period elapses, a copy of
the running configuration will be archived.
*archiveconfig Manually saves a copy of the current running
57
configuration to the Cisco IOS configuration archive

Archive command
R1(config)# archive
R1(config-archive)# path flash:/config-archive/$h-config
R1(config-archive)# write-memory
R1(config-archive)# time-period 10080
R1# show archive
There are currently 3 archive configurations saved.
The next archive file will be named flash:/config-archive/R1config-4
Archive #

Name

0
1

flash:/config-archive/R1-config-1

flash:/config-archive/R1-config-2

flash:/config-archive/R1-config-3 <- Most Recent

58

Configure
Replace

R1# configure terminal

Enter configuration commands, one per line. End with CNTL/Z.
R1(config)# hostname TEST
TEST(config)# ^Z
TEST# configure replace flash:config-archive/R1-config-3 list
This will apply all necessary additions and deletions
to replace the current running configuration with the
contents of the specified configuration file, which is
assumed to be a complete configuration, not a partial
configuration. Enter Y if you are sure you want to proceed. ?
[no]: yes
!Pass 1
!List of Commands:
no hostname TEST
hostname RO1
end
Total number of passes: 1
Rollback Done

configurereplacetargeturl[list][force][timeseconds][nolock]

configure replace - Allows you to replace the currently running configuration

on the router with a saved configuration.
Compares the running configuration with the configuration file
Creates a list of differences
Set of Cisco IOS configuration commands are generated that changes the
existing running configuration to the replacement configuration.

59

R1#configureterminal
Enterconfigurationcommands,oneperline.EndwithCNTL/Z.
R1(config)#hostnameTEST
TEST(config)#^Z
TEST#configurereplaceflash:configarchive/R1config3list
Thiswillapplyallnecessaryadditionsanddeletions
toreplacethecurrentrunningconfigurationwiththe
contentsofthespecifiedconfigurationfile,whichis
assumedtobeacompleteconfiguration,notapartial
configuration.EnterYifyouaresureyouwanttoproceed.?[no]:yes
!Pass1
!ListofCommands:
nohostnameTEST

Changes shown with list option

hostnameRO1
end
Totalnumberofpasses:1
RollbackDone

The command option list is added to the configure replace

command in order to show the configuration commands that are
being applied by the configuration replacement.

60

Disaster Recovery Tools

Successful disaster recovery is dependent on existence of the following:

Up to date configuration backups
Up to date software backups
Up to date hardware inventories
Configuration and software provisioning tools
As parts of the fundamental network maintenance toolkit, TFTP, FTP, SCP,
HTTP and HTTPS server are useful for creating backups of the
configuration and operating system of a router or switch.
61

CIS 188 CCNP TSHOOT (Troubleshooting)

Chapter 1 Planning Maintenance for
Complex Networks
Rick Graziani
Cabrillo College
graziani@cabrillo.edu