Troubleshoot Workbook 21 January, 2012

TS5 - VRF

Troubleshooting Guidelines
This section is comprised of a set of troubleshooting scenarios. You have a maximum of 2 hours to complete the section. The final score of this section is combined with the Configuration sections to comprise your final Pass or Fail status on the given lab exam. A candidate is required to pass both sections to achieve Cisco CCIE certification. You will be presented with preconfigured routers and Frame-Relay switches in the topology. DO NOT change the following configuration on the devices.
Hostname Enable password "cisco" Console line configuration For all of the authentication configuration in the lab, password is "cisco" unless changed to introduce a break. Do NOT change AAA configuration unless explicitly stated in a question. Points are awarded for finding AND fixing inserted faults in the presented fully configured topology. An inserted fault is an introduced break for a scenario that was previously working. Depending on the scenario, fixing the inserted faults could require multiple command lines on the same or multiple devices. The resolution of one incident may depend on the resolution of previous incident(s). The dependency will not be visible if the tickets are resolved in sequence. There are NO physical faults introduced in the presented topology. Do NOT change any routing protocol boundaries. Refer to the provided diagram. DO NOT REMOVE ANY FEATURE CONFIGURED IN ORDER TO RESOLVE AN INCIDENT, YOU MUST RESOLVE MISCONFIGURATION RATHER THAN REMOVING IT ALL (examples: Access-lists, PBR, CoPP, MQC, etc.) Static and default routes are NOT permitted unless preconfigured. These restrictions include floating static and those generated by routing protocols. Routes to Null0 that are generated of a dynamic routing protocol solution are permitted. Tunneling and policy-routing are NOT permitted unless preconfigured. Dynamic Frame Relay mappings are NOT permitted. Points will be deducted for every incident in which candidate uses a prohibited solution. Candidates have control of all required devices in the topology. If required to verify the reachability from a host machine during the lab exam, use the ping command with source option on the router that is shown connected to the subjected host in the diagram.

Q1 IP SLA.

[2 Points]

The IP Service Level Agreement configured between R14 and R9 is not working as expected Fix problem so that it matches the following outputs:
While you are resolving this issue, you are not allowed to create any new interfaces. Refer to the Troubleshooting guidelines to determine if your solution is appropriate.
R14# sh ip sla statistics IPSLAs Latest Operation Statistics R9# sh ip sla responder General IP 10 IPSLA operation id:SLA Responder on Control port 1967 General IP SLA Responder is: Enabled Latest RTT: 17 milliseconds Number Latest Recent Latest of controlstart time: 13:58:21 EST Tue Dec errors: 0 operation message received: 170 Number of 18 2012 sources: return code: OK operation

10.1.1.14 [14:05:06.661 EST Tue Dec 18 2012] Number of successes: 9 10.1.1.14 [14:05:01.666 EST Tue Dec 18 2012] Number of failures: 0 10.1.1.14 live: Forever Operation time to [14:04:56.661 EST Tue Dec 18 2012] 10.1.1.14 [14:04:51.666 EST Tue Dec 18 2012] 10.1.1.14 [14:04:46.666 EST Tue Dec 18 2012] Recent error sources: Permanent Port IP SLA Responder Permanent Port IP SLA Responder is: Enabled tcpConnect Responder: IP Address 10.1.1.9 Port 1026

Diagram involves getting tcpConnect thing with population the table in show ip sla statistics between R14 as querier and R9 as responder using Ports TCP 1025 (source) and 1026 (destination). Both Routers are in the same AS.

AS 65222 EIGRP 222

172.16.14.X/30

R15 .2 E0/1 .6 E0/0 VL1415 .1 E0/0 R14 IP SLA Querier .9 E0/1 E1/1 E1/0 E0/3 E0/2 VL1617 .14 E0/0 E1/2 E1/3 VL1517 E0/0 E0/1 .5 E0/2 .13 E0/1 R17 .17 E0/0

SW3

VL1416 .10 E0/1

R16 IP SLA responder .18 E0/1 PE

R9

Possible errors are:

A) Port and IP Adress wrong, can be swaped. B) Schedule Life not configured (must be forever). C) Check for eventual access-lists. D) IP sla responder missing.

-3-

Explanations:
R9
ip sla responder tcp-connect ip address 10.1.1.9 port 1026

R14
ip sla 9 tcp-connect 10.1.1.14 1025 source-ip 10.1.1.9 source-port 1026 <<<< Port and IP Address wrong ! ip sla schedule 9 start-time now <<<< Change this to ip sla schedule 9 life forever start-time now

Q2 BGP.
R14 from AS 65222 is not able to reach a Host on R20 on AS65333 Fix problem so that R14 can ping R20:
R14# ping 10.1.1.20 so lo0 While you are resolving this issue, you are not allowed to create any new interfaces. Refer to the Troubleshooting guidelines to determine if your solution is appropriate.

[3 Points]

/0 E1 .90

1E .6
E .17 3 1/

0 0/

2 .6

/1 E1

Hint: Point to this ticket is to get route on RR R4 going to R9. Hint: Ping Continuosly from Source to Destination and check when problem gets solved

Possible errors are:

A) BGP Session missing between R5 (RR) and R12 B) RR R5 Has a Route to R9 whereas RR R4 has not. Check Cluster-IDs!!
With 4 Clusters:
Wrong Cluster-id on R4: Must be unique. Change it to 100.1.1.4

-4-

Q3 IPv6 Phone.
R19 is acting as an IPv6 phone. Fix problem so that the IPv6 Phone can reach R13 on AS65004:
Phone# ping XX:XX:XX::23 so loX While you are resolving this issue, you are not allowed to configure Auto-Tunnel feature. Refer to the Troubleshooting guidelines to determine if your solution is appropriate.

[2 Points]

R11 to R13 IPv6 Tunnel is DOWN,

Problem is NOT related to MPLS Cloud!! Possibly wrong Tunnel Mode Possibly loopback interface or a router is not routing protocols Check Ipv6 Address auto-configuration on IPv6 Phone. Possible OSPF neighbor problem. OSPF should be enabled through the tunnel Posible Access-List implicitly denying Protocols 47 (GRE IPv4) or 41 (GRE IPv6IP).
Hint: Problem is in Tunnel Config. Configuring auto-tunnel leads to pings.

Possible errors are:

A) IPv6 Phone is missing interface level command ipv6 address autoconfig default. Check for default keyword B) IPv6 Phone is missing command ipv6 unicast-routing C) Wrong Tunnel interface mode. Remove command tunnel mode mpls traffic-eng and set it to tunnel mode ipv6ip or tunnel mode gre" D) Tunnel Source missing on interface. Add interface level command tunnel source loopback0 E) Wrong Destination IP Address configured on interface Tunnel F) Missing ipv6 ospf 1 area 0 interface level command within Tunnel interface. G) Check for Posible Access-List implicitly denying Protocols 47 (GRE IPv4) or 41 (IPv6IP)

OSP Fa

rea 6

.29 E 1/0

.6 2

E1 /

.6 1
.17 / E1 3

E0 /0

.9 0 /0 E1

-5-

Explanations:
R19 (IPv6 Phone)
ipv6 unicast-routing ! interface Ethernet0/0 ipv6 enable ipv6 address autoconfig default
<<<< Missing!! - ADD

<<<< Missing!! - ADD

R11 / R13
ipv6 unicast-routing ! interface Tunnel1 ip address 100.1.1.9 255.255.255.0 ipv6 address 2000:89::9/64 ipv6 ospf 1 area 0 tunnel source Loopback0 tunnel destination 88.1.1.1 tunnel mode mpls traffic-eng tunnel mode ipv6ip

<<<< Missing!! - ADD <<<< Missing!! - ADD <<<< Wrong!! - CHANGE <<<< Wrong!! - REMOVE <<<< Missing!! - ADD

-6-

Q4 DNS.

[2 Points]

Ping from R20 to www.abc.com should resolve and reach the Web Server on the same AS. Packet count under ZBF map should increase with the ping traffic as shown in the output:
While you are resolving this issue, you are not allowed to create any new interfaces. Refer to the Troubleshooting guidelines to determine if your solution is appropriate.
R20# ping www.abc.com Translating "www.abc.com" ... domain server (10.1.1.22) [OK] Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 4.4.2.2, timeout is 2 seconds: !!!! Success rate is 100 percent (5/5), round-trip min / avg / max = 40/61/76 ms

R29# show policy-map type inspect zone-pair sessions Policy exists on zp ZBF Zone-pair: ZBF Service-policy inspect: ZBF Class-map: HTTP (match-any) Match: protocol http 0 packets, 0 bytes 30 second rate 0 bps inspect 0 packets, 0 bytes Class-map: DNS (match-any) Match: protocol dns 2 packets, 72 bytes 30 second rate 0 bps inspect 2 packets, 72 bytes Class-map: ICMP (match-any) Match: protocol icmp 5 packets, 500 bytes 30 second rate 0 bps inspect 5 packets, 500 bytes Class-map: class-default (match-any) Match: any Pass 362 packets, 15302 bytes

Zone Base Firewall is involved

EIGRP AS 333
172.10.10.X/29

.2 E0/0

SW4
Web Server www.abc.com Lo SW4 192.168.133.100

.17 E0/1 PE

.18 E0/1 R20 .1 E0/0 Ping www.abc.com

.3 E0/0

.10 E0/0 R21 ZBF .9 E0/1

R22

DNS Server DMZ 192.168.233.100

-7-

Possible errors are:

DNS Section: A) Client's ip domain-lookup is not configured B) Server-side ip host www.cisco.com X.X.X.X is not configured C) Access-List on SW4 is blocking ICMP Traffic. ZBF Section: E) Command match protocol dns missing in ZBF Config F) Command match protocol icmp missing in ZBF Config G) Zone Security applied incorrectly to interfaces. H) The DNS Server and Web Server IP addresses are reversed

R29
ip name-server 10.1.1.22 ip domain-lookup
<<<< Missing - ADD

R31
ip host www.cisco.com 4.2.2.2 ip dns server
<<<< Missing - ADD

-8-

Q5 PPP Multilink.
Ping from R25 Loopback0 should reach a user located on R27 Fix the Network so R25 Loopback 0 can ping R27:
R25# ping 10.1.1.27 source loopback0 While you are resolving this issue, you are not allowed to create any new interfaces. Refer to the Troubleshooting guidelines to determine if your solution is appropriate.

[2 Points]

R25 S0/2

S0/1

Multilink PPP MD5 S1/1

S1/0 .1 E0/0 R26 DHCP/NAT

RIP v2
R27

.2 E0/0 192.168.20.0/30

Multilink interface is down PPP is not configured correctly across multilink and multilink is missing group statement

Possible errors are:

A) R26 PPP multilink has ppp chap hostname/password mismatch with its adjacent router on username B) Username on R25 and R26 is incorrect along with password (Password must be CCIE). C) R25 and R26 are missing the multilink group command D) Authentication Commands missing under interface configuration. You may have to save and reload both Multilink routers after you think that your config is correct.

Explanations:
R25
username R26 password cisco ! interface Serial0/0/0 description PPP-Multilink-1 bandwidth 2048 no ip address encapsulation ppp ppp multilink ppp multilink group 1 no clock rate 2000000 no cdp enable
<<<< Missing - ADD

R26
interface Multilink1 ip address x.x.x.x 255.255.255.252 ip nat outside ip inspect monitor out ip virtual-reassembly ppp multilink ppp multilink group 1 no cdp enable ! interface Serial0/0/1 description PPP-Multilink-2 bandwidth 2048 no ip address encapsulation ppp ppp multilink ppp multilink group 1 no fair-queue no clock rate 2000000 no cdp enable ppp authentication chap pap ppp pap sent-username myrouter password CC1E

<<<< Missing - ADD <<<< Missing - ADD

<<<< Missing - ADD <<<< Missing - ADD

-9-

- 10 -

Q6 Frame-Relay QoS.

[2 Points]

Traffic that is marked with IP Precedence 5/ToS 160 coming from R26 must reach R23 Fix problem so that the extended ping result in 100% success:
While you are resolving this issue, you are not allowed to create any new interfaces. Refer to the Troubleshooting guidelines to determine if your solution is appropriate.
R26# ping Target IP address: 10.1.1.23 Repeat count [5]: 5 Extended commands [n]: y Source address or interface: Type of service [0]: 160 Set DF bit in IP header? [No]: Validate reply data? [No]: Data pattern [0xABCD]: Loose, Strict, Record, Timestamp, Verbose [none]: Sweep range of sizes [n]: Type escape sequence to abort Sending 10000, 100-byte ICMP Echos to 10.1.1.23, timeout is 2 seconds: !!!!

R25# sh policy-map int s0/0/0 | be DLCI 254 Serial0/0/0: DLCI 254 Service-policy output: POLICY Class-map: VOICE (match-all) 0 packets, 0 bytes 5 minute offered rate 0 bps, drop rate 0 bps Match: ip precedence 5 police: cir 12000 bps, bc 3000 bytes conformed 0 packets, 0 bytes; actions: transmit exceeded 0 packets, 0 bytes; actions: drop conformed 0 bps, exceed 0 bps QoS Set dscp ef Packets marked 0 Class-map: class-default (match-any) 0 packets, 0 bytes 5 minute offered rate 0 bps, drop rate 0 bps Match: any Queueing queue limit 64 packets (queue depth/total drops/no-buffer drops) 0/0/0 (pkts output/bytes output) 0/0 shape (average) cir 80000, bc 320, be 320 target shape rate 80000 lower bound cir 0, adapt to fecn 0

- 11 -

Multicast Boundary PE R13 .13 E0/1 2001:CC1E:ABCD:10:10:10:0:X/125

AS 65004 OSPF 3 Area 0

MD5 Auth 10.10.10.X/29

Local Service Provider (ISP)

.14 E0/0 R23

MSDP Anycast RP 198.23.23.23 .1 S1/0 234 235 S0/0 S0/1 253 .3 S0/0 R25 S0/2 S0/1 254 .17 E0/0 .2 S1/0 R24 S1/0 .1 E0/0 S1/1 R26 DHCP/NAT S0/2 245 243 FR1 R28 .18 E0/0 Video Streamer 224.28.28.28

BGP AS 65004

QoS DLCI

Multilink PPP MD5

RIP v2
R27

.2 E0/0 192.168.20.0/30

Scenario 1: MQC Class-Based Shaping:

Possible errors are:
A) Policy-Map not applied under Frame-Relay map-class B) CIR is too small and results in packet loss. Raise it to higher value with command shape average 96000 C) Match IP Precedence 5 missing under configured Class-Map.

Explanations:
Nested MQC CB-Shaping over FR
class-map VOICE match ip precedence 5 ! policy-map VOICE class VOICE priority percent 10 ! policy-map CISCO class class-default shape average 8000 shape adaptive 8000 service-policy VOICE ! map-class frame-relay CCIE service-policy output CISCO ! interface Se0/0 frame-relay interface-dlci 206 class CCIE
<<<< Missing - ADD

<<<< Raise it to 96000 <<<< Nested call POLICY-MAP VOICE

<<<< Missing - ADD

<<<< interface call MAP-CLASS CCIE

- 12 -

- 13 -

Q7 MSDP Multicast on Frame Relay.

PC2 in has to get a Multicast Stream from R28 in AS65004. Fix problem so the ping results:
R28# ping 224.28.28.28 re 5 While you are resolving this issue, you are not allowed to create any new interfaces. Refer to the Troubleshooting guidelines to determine if your solution is appropriate.

[2 Points]

Possible errors are:

A) R13 has access-list blockin multicast traffic B) R25 has access-list blockin multicast traffic through Policy in Control Plane. Make ACL deny traffic. C) R25 is missing interface level command ip pim nbma-mode D) Video streaming server R26 is missing "ip pim sparse-dense-mode" and "ip pim auto-rp listener" E) Auto-RP not configured. F) Wrong DR elected (R25). Raise Auto-RP Priority on R23 Frame Relay interface. G) Missing command ip pim auto-rp listener on involved routers

.6

E1 /

.6
.17 /3 E1

E0

/0

. 90 /0 E1

- 14 -

Explanations:
Note: In AS65004 there is frame relay area running multicast with multicast boundaries denying 224.0.1.39 and 224.0.1.40.

MSDP Peering is UP. Use of Auto-RP 224.0.1.39 and 224.0.1.40 is denied at border. Whether this is an error or not remains to be verified, because in boundaries denying 39 and 40 is usually part of the RP control mechanism solution.

R25
access-list 100 permit ip any host 224.23.23.23 ! class-map DRP match access-group 100 ! policy-map DRM class DRP drop ! control-plane service-policy input DRM ! interface Serial0/1 ip address 10.1.48.1 255.255.255.248 ip pim nbma-mode ip pim sparse-mode encapsulation frame-relay frame-relay interface-dlci 100 frame-relay interface-dlci 200 ! interface loopback10 ip pim sparse-dense-mode ! ip pim send-rp-announce Loopback10 scope 16 ip pim send-rp-discovery Loopback10 scope 16
<<<< Make it deny (2 -ves= +ve)

<<<< Missing

<<<< Missing <<<< Missing

Auto RP filter by UDP 496 port

ip access-list extended UDP deny udp any eq pim-auto-rp 224.0.1.0 0.0.0.255 eq pim-auto-rp permit ip any any

Standard access-list applied to multicast boundary

access-list 10 deny 224.0.1.39 access-list 10 deny 224.0.1.40 access-list 10 permit any ! int Serial0/0 ip multicast boundary 10 in

- 15 -

Q8 IGP Routing (OSPF to BGP Redistribution).

Traffic going from R32 must reach 4.2.2.2 going through R1 over the internet Fix problem so that the extended ping result in 100% success:
While you are resolving this issue, you are not allowed to redistribute bgp into ospf.. Refer to the Troubleshooting guidelines to determine if your solution is appropriate.

[3 Points]

PC2# ping 4.2.2.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds: !!!! Success rate is 100 percent (5/5), round-trip min / avg / max = 40/61/76 ms

Possible errors are:

A) BGP Peering between R1 and R2, or R1 and R3 not configured. Configure it. Configuration Details are provided in Question. B) Hosts interface IP Address is Wrong; it cannot reach its configured Next-Hop. Fix it. C) Aggregated Route towards 4.2.2.2 is learned by R4 but it is suppressed. Remove suppression keyword. D) Network 10.0.0.0/8 is advertised as a summary. Remove network summarization. E) Route-Map on R5 with next-hop interface null0 set for prefix 4.0.0.0/8. Fix it. F) Network 10.1.1.0 not announced int OSPF Area 0 G) Missconfigured LoopBack on R13 with IP Address 4.2.2.2/32 Shutdown interface Loopback Clear arp-cache on all the way troughout the traceroute path

Explanations:

- 16 -

Q9 MPLS.

(2 Errors)

[3 Points]

Client connected to R34 in ACMEs Branch Office (AS65111) has to reach Server R31 in ACME HeadQuarters. Fix problem:
RR# While you are resolving this issue, you are not allowed to create any new interfaces. Refer to the Troubleshooting guidelines to determine if your solution is appropriate.
AS 65111

ACME Corp Network office (branch)

RIP v2
172.16.13.X/30 Client R34 .2 E0/0
Deafult RIP Route

Internet AS 65535
.6 E0/1 R33 .5 E0/2
vrf ACME RD=111:111 Extended Backbone

.1 E0/0

R1 .1 E0/0 .9 E0/1

PE R8 .26 E0/1 .25 E0/1

.85 E0/0 .86 E0/1 R6

0 /0 .9 4 E

Global Telecom Provider (ISP)

PE

BGP AS 65001
.89 E0/2
.90 E1/

.78 E0/0
a2

R11

192.168.10.X/30 OSPF 3 Area 0 .2 E0/0 .33 E0/1

192.168.10.X/30
3 Fa E0 / OSP
re

R7 . 74 E0 .18 E0/0 /

.73

.10 E0/0 .34 E0/1

.77

0 E0/ 3

.37 RR

RR

IPv6 Tunnel OSPF Area 0

R2

Cluster ID E0/2 100.1.1.2

R3 Cluster ID 100.1.1.3 .41 E0/2

E0 /0 E1 /1

R12 PE

RR .38 E0/0 Cluster ID

100.1.1.4

.45 E0/1 .54 E1/1

P OS r ea Fa 5

.46 E0/1 R5 0/2 .6 E .49 E0/3 MSDP Anycast RP 198.23.23.23

VLAN 48

R4 .21 E0/3

.6

Cluster ID 100.1.1.5

RR .42 E0/0

OS

PF

e ar

.29 E1/0

.5

.1 3

E0/

2
VLAN 4

E1 /0
Fa rea

PE

VLAN 20

OS P

.6
a3

1
/2 .9 3 E1 /3 E1 .17

.14
4

.53 E0/0 R9

OSPF 3 Area 1 E0/0 SVI .22 E0/1

E0 / 0

AS 65AAA
10.10.20.X/30

.17 E0/2
Backdoor Link to R9 VRF Static route pointed to R29

E0/0 SVI .50 E0/1 E1/0 E1/1

PE

R13

OSPF 5 Area 0
VLAN 6

SW1
ea 6 F ar

SW2
E0/3 E0/2

.26 SVI 56 .2 SVI 6 .29 SVI 60

E1/3 E0/1 .30 E0/2 .14 E0/3 0 0/ 1E .2

.1 0

.1 E0/0 R31 Server E0/0 E1/0

VLAN 56

E0/2 10.1.1.1/24

SW6
E1/1

E1/2
/2 E0 3 / E0 E0 /2

R30
VLAN 10

O SP

E0/3

VLAN 11

VLAN 12

E0 /1

E0/0 R32 PC1 PC2

E1/0
VLAN 5

E0 /3

E1/1

.9 E

. 18 0/ 1

.5 SVI User
.25 SVI 56

SW5
E1/3

.3 0 E

E0/1 E1/2

.6 E0/2 .13 E0/3 R29

VRF Static route pointed to R30

0/0

E0 /0

10.1.1.100/24 vrf ACME PE

PC SVI 11 .57 SVI

PC SVI 12 .65 SVI IGMP Join

Main Link to R10

.22 E0/1 R10

ACME Corp Headquarter

Check BGP to RIP Route Redistribution Route not getting from BGP to last RIP Router in the queue. R8 has a best route to R9 going through R10 based on lower IP Address of R9 Missing Route between R8 and two Routers in the series.

Possible errors are:

A) R8 and/or R9 have MPLS configured with label protocol TDP. Change it to MPLS Label Protocol LDP. B) R8 is a PE connected to R6 and R7. R6 and R7 do not forward labels because they miss ip cef command. Add it C) MPBGP Neighbor isnt active on PE. Use VPNv4 Address Family level command neighbor X.X.X.X active to fix it. D) R4 has MPLS MTU size wrongly configured on interfaces facing R9 and R10.

Explanations:
R4
interface mpls mtu ! interface mpls mtu Ethernet3/0 100 Ethernet4/0 100
<<<< WRONG - FIX

<<<< WRONG - FIX

- 17 -

Q10 MST.
User has to ping a Server in two hops. Fix problem:
RR# While you are resolving this issue, you are not allowed to modify the configuration of SW6.. Refer to the Troubleshooting guidelines to determine if your solution is appropriate.

[2 Points]

E0

/2

Possible errors are:

A) VTP Sync, Trunk config, STP, VLANs, etc B) Both SW5 and SW6 are configured as VTP Client. Configure SW5 as Server. C) VTP Password Mismatch. D) SW6 is configured not to allow Vlan56 across its e3/1 trunk interface. Adjust priority on SW5 e3/1 for Vlan56 E) Servers VTP Revision Number is too low; Client does not synchronize. Raise VTP Revision Number on Server. F) Interface VLAN 56 is configured as passive in the OSPF Process. Fix it

E0 /

.1

0 0/ 1E .2

E0 /1

/ E0 2 E0 /3

- 18 -