Troubleshooting BGP Net Workers, 2001)

Networkers 2001, Australia
March 28-30, Brisbane

2000, Cisco Systems, Inc.
Troubleshooting BGP
Phil Smith
Presentation_ID 2001, Cisco Systems, Inc. 2000, Cisco Systems, Inc.
BGP in Large Scale Networks
Scalable Stable Simple

Avoid the Problem in the First Place

Use simple configurations
maintain a consistent policy throughout the AS
Promote stable networks

nail-down your routes use loopback interfaces
Grow into your network

use peer-groups and RRs for scalability
Agenda
Basic Tools Peer Establishment UPDATE Exchange Selection Algorithm Route Reflectors Route Flap Damping
Tool Time
Basic Tools
2210 1351_06_2000_c2 Inc.2000, Cisco Systems, Inc. 2000, Cisco Systems,
BGP Troubleshooting Tools
show commands debug output Log messages
show Commands
router#show ip bgp ? A.B.C.D IP prefix <network>/<length>, e.g., 35.0.0.0/8 A.B.C.D Network in the BGP routing table to display cidr-only Display only routes with non-natural netmasks community Display routes matching the communities community-list Display routes matching the community-list dampened-paths Display paths suppressed due to dampening filter-list Display routes conforming to the filter-list flap-statistics Display flap statistics of routes inconsistent-as Display only routes with inconsistent origin ASs neighbors Detailed information on TCP and BGP neighbor connections paths Path information peer-group Display information on peer-groups quote-regexp Display routes matching the AS path "regular expression" regexp Display routes matching the AS path regular expression summary Summary of BGP neighbor status | Output modifiers <cr>
show Commands (Cont.)
router#show ip bgp neighbors x.x.x.x ? advertised-routes Display the routes advertised to a BGP neighbor dampened-routes Display the dampened routes received from neighbor flap-statistics Display flap statistics of the routes learned from neighbor paths Display AS paths learned from neighbor received Display information received from a BGP neighbor received-routes Display the received routes from neighbor routes Display routes learned from neighbor | Output modifiers <cr>
The BGP Table
router#show ip bgp BGP table version is 9, local router ID is 7.72.6.1 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network *> 3.0.0.0 *> 5.0.0.0 *> 6.0.0.0 * i *> 7.0.0.0 *> 8.0.0.0/5 *> 17.0.0.0 * i *> 23.0.0.0 * i *> 35.0.0.0 * i Next Hop 0.0.0.0 0.0.0.0 6.72.6.2 6.72.6.2 0.0.0.0 0.0.0.0 6.72.6.2 6.72.6.2 6.72.6.2 6.72.6.2 6.72.6.2 6.72.6.2 Metric LocPrf Weight Path 0 32768 i 0 32768 i 4294967294 0 200 i 4294967294 100 0 200 i 0 32768 i 0 32768 i 4294967294 0 200 i 4294967294 100 0 200 i 4294967294 0 200 i 4294967294 100 0 200 i 4294967294 0 200 i 4294967294 100 0 200 i
10
The BGP Table (Cont.)
router#show ip bgp 6.0.0.0 BGP routing table entry for 6.0.0.0/8, version 2 Paths: (2 available, best #1) Advertised to non peer-group peers: 7.25.14.4 7.72.6.3 7.75.7.1 200 6.72.6.2 from 6.72.6.2 (7.72.6.2) Origin IGP, metric 4294967294, localpref 100, valid, external, best 200 6.72.6.2 from 7.75.7.1 (7.75.7.1) Origin IGP, metric 4294967294, localpref 100, valid, internal
11
show ip bgp Summary
router#show ip bgp summary BGP router identifier 7.72.6.1, local AS number 100 BGP table version is 9, main routing table version 9 8 network entries and 12 paths using 1176 bytes of memory 3 BGP path attribute entries using 144 bytes of memory 1 BGP AS-PATH entries using 24 bytes of memory BGP activity 8/0 prefixes, 12/0 paths Neighbor 6.72.6.2 7.25.14.4 7.72.6.3 7.75.7.1 V 4 4 4 4 AS MsgRcvd MsgSent 200 6885 6882 300 6882 6883 100 6880 6886 100 6884 6885 TblVer InQ OutQ Up/Down State/PfxRcd 9 0 0 4d18h 4 9 0 0 4d18h 0 9 0 0 4d18h 0 9 0 0 4d18h 4
12
show ip bgp neighbors

router#show ip bgp neighbors 6.72.6.2 BGP neighbor is 6.72.6.2, remote AS 200, external link Index 1, Offset 0, Mask 0x2 BGP version 4, remote router ID 7.72.6.2 BGP state = Established, table version = 9, up for 4d21h Last read 00:00:56, last send 00:00:48 Hold time 180, keepalive interval 60 seconds Neighbor NLRI negotiation: Configured for unicast routes only Peer negotiated unicast and multicast routes Exchanging unicast routes only Received route refresh capability(old) from peer Minimum time between advertisement runs is 30 seconds Received 7044 messages, 0 notifications, 0 in queue Sent 7041 messages, 0 notifications, 0 in queue Prefix advertised 4, suppressed 0, withdrawn 0 Route refresh request: received 0, sent 0 Inbound path policy configured Incoming update prefix filter list is in-filter Outgoing update prefix filter list is out-filter Route map for incoming advertisements is ebgp-router-in Route map for outgoing advertisements is ebgp-router-out Connections established 1; dropped 0 Last reset never Number of unicast/multicast prefixes received 4/0 External BGP neighbor may be up to 255 hops away. Connection state is ESTAB, I/O status: 1, unread input bytes: 0 Local host: 3.72.6.1, Local port: 179 Foreign host: 6.72.6.2, Foreign port: 11014
13
debug ip bgp
router#debug ip bgp ? A.B.C.D BGP neighbor address dampening BGP dampening events BGP events keepalives BGP keepalives updates BGP updates <cr>
Remembercan be dangerous!
Use only in the lab or if advised by the TAC
To make a little safer:

logging buffered <size> no logging console
14
Session Establishment (debug ip bgp )

16:06:30: BGP: 7.72.6.1 sending OPEN, version 4 16:06:31: BGP: 7.72.6.1 OPEN rcvd, version 4 16:06:31: BGP: 7.72.6.1 rcv OPEN w/ OPTION parameter len: 12 16:06:31: BGP: 7.72.6.1 rcv OPEN w/ option parameter type 2 (Capability) len 6 16:06:31: BGP: 7.72.6.1 OPEN has CAPABILITY code: 1, length 4 16:06:31: BGP: 7.72.6.1 OPEN has MP_EXT CAP for afi/safi: 1/1 16:06:31: BGP: 7.72.6.1 rcv OPEN w/ option parameter type 2 (Capability) len 2 16:06:31: BGP: 7.72.6.1 OPEN has CAPABILITY code: 128, length 0 16:06:31: BGP: 7.75.7.1 passive open 16:06:31: BGP: 7.75.7.1 OPEN rcvd, version 4 16:06:31: BGP: 7.75.7.1 sending OPEN, version 4 16:06:31: BGP: 7.75.7.1 rcv OPEN w/ OPTION parameter len: 12 16:06:31: BGP: 7.75.7.1 rcv OPEN w/ option parameter type 2 (Capability) len 6 16:06:31: BGP: 7.75.7.1 OPEN has CAPABILITY code: 1, length 4 16:06:31: BGP: 7.75.7.1 OPEN has MP_EXT CAP for afi/safi: 1/1 16:06:31: BGP: 7.75.7.1 rcv OPEN w/ option parameter type 2 (Capability) len 2 16:06:31: BGP: 7.75.7.1 OPEN has CAPABILITY code: 128, length 0
15
Session Establishment (debug ip bgp events)

17:31:39: BGP: 7.72.6.1 went from Idle to Active 17:32:00: BGP: 7.72.6.1 went from Active to OpenSent 17:32:00: BGP: 7.72.6.1 went from OpenSent to OpenConfirm 17:32:00: BGP: 7.72.6.1 went from OpenConfirm to Established
17:31:59: BGP: 7.75.7.1 went from Idle to Active 17:32:00: BGP: 7.75.7.1 went from Active to Idle 17:32:00: BGP: 7.75.7.1 went from Idle to Connect 17:32:00: BGP: 7.75.7.1 went from Connect to OpenSent 17:32:00: BGP: 7.75.7.1 went from OpenSent to OpenConfirm 17:32:00: BGP: 7.75.7.1 went from OpenConfirm to Established
16
Looking at the Updates

router#debug ip bgp updates? <1-199> Access list <1300-2699> Access list (expanded range) <cr> router#debug ip bgp x.x.x.x updates? <1-199> Access list <1300-2699> Access list (expanded range) <cr>
Use an access-list to limit the output!

17
debug ip bgp Updates
Peer Address
Prefix Being Advertised
NEXT_HOP
BGP: 6.72.6.2 computing updates, neighbor version 0, table version 13, starting at 0.0.0.0 BGP: 6.72.6.2 send UPDATE 3.0.0.0/8, next 3.72.6.1 BGP: , metric 0, path 100 BGP: 6.72.6.2 send UPDATE 5.0.0.0/8 (chgflags: 0x0), next 3.72.6.1 BGP: 6.72.6.2 send UPDATE 7.0.0.0/8 (chgflags: 0x0), next 3.72.6.1 BGP: 6.72.6.2 1 updates enqueued (average=56, maximum=56) BGP: 6.72.6.2 update run completed, ran for 0ms, neighbor version 0, start version 13, throttled to 13, check point net 0.0.0.0
18
debug ip bgp Updates (Cont.)

BGP: 6.72.6.2 rcv 494, path BGP: 6.72.6.2 rcv BGP: 6.72.6.2 rcv BGP: 6.72.6.2 rcv BGP: 6.72.6.2 rcv UPDATE 200 UPDATE UPDATE UPDATE UPDATE w/ attr: nexthop 6.72.6.2, origin i, metric about about about about 6.0.0.0/8 17.0.0.0/8 23.0.0.0/8 35.0.0.0/8
Peer Address Prefixes in the Same UPDATE
Attributes Apply to All Prefixes
BGP: 6.72.6.2 rcv UPDATE w/ attr: nexthop 6.72.6.2, origin i, metric 294, path 200 100 BGP: 6.72.6.2 rcv UPDATE about 3.0.0.0/8 -- DENIED due to: as-path contains our own AS; BGP: 6.72.6.2 rcv UPDATE about 7.0.0.0/8 -- DENIED due to: as-path contains our own AS;
19
Logging Neighbor Changes

Generate a log message whenever a BGP neighbor changes state, also indicate reason for reset Syntax (router subcommand):
[no] bgp log-neighbor-changes Typical log messages:
%BGP-5-ADJCHANGE: neighbor x.x.x.x Up %BGP-5-ADJCHANGE: neighbor x.x.x.x Down-Remote AS changed
20
show ip bgp neighbors x.x.x.x
router#show ip bgp neighbors 7.75.7.1 BGP neighbor is 7.75.7.1, remote AS 200, external link ... Received 194 messages, 1 notifications, 0 in queue Sent 194 messages, 0 notifications, 0 in queue Prefix advertised 0, suppressed 0, withdrawn 0 Route refresh request: received 0, sent 0 Connections established 7; dropped 7 Last reset 00:04:11, due to BGP Notification received, hold time expired Number of unicast/multicast prefixes received 0/0 External BGP neighbor may be up to 255 hops away. No active TCP connection
21
Come Meet the Neighbors!

Peer Establishment
22
Peer Establishment
Routers establish a TCP session
Port 179permit in ACLs IP connectivity (route from IGP)
OPEN messages are exchanged

Peering addresses must match the TCP session Local AS configuration parameters Capabilities negotiation
23
Common Problems
Sessions are not established

No IP reachability Incorrect configuration Peering addresses OPEN parameters
24
Cant Establish Session Symptoms

routerA#show ip bgp summary BGP router identifier 7.72.6.1, local AS number 100 BGP table version is 4, main routing table version 4 6 network entries and 6 paths using 774 bytes of memory 2 BGP path attribute entries using 96 bytes of memory 1 BGP AS-PATH entries using 24 bytes of memory BGP activity 6/0 prefixes, 6/0 paths Neighbor 6.72.6.2 7.25.14.4 7.72.6.3 7.75.7.1 V 4 4 4 4 AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 200 0 0 0 0 0 never Idle 300 4 5 4 0 0 00:01:43 0 100 0 0 0 0 0 never Active 100 7 5 4 0 0 00:01:55 3
The peering session is not established!

State may change between active, idle and connect
25
Cant Establish Session Troubleshooting I
Is the remote-as assigned correctly?

Local AS router bgp 100 neighbor 6.72.6.2 remote-as 200 neighbor 7.72.6.3 remote-as 100 eBGP Peer
iBGP Peer
26
Cant Establish Session Troubleshooting I (Cont.)

Verify IP connectivity
check the routing table use ping/trace to verify two way reachability inspect for ACLs in the path to the neighbor
routerA#show ip route 7.72.6.3 Routing entry for 7.72.6.3/32 Known via "ospf 123, distance 110, metric 87, type intra area Last update from 27.27.27.254 on POS5/0, 00:09:33 ago Routing Descriptor Blocks: * 27.27.27.254, from 7.72.6.3, 00:09:33 ago, via POS5/0 Route metric is 87, traffic share count is 1 routerA#ping 7.72.6.3 Sending 5, 100-byte ICMP Echos to 7.72.6.3, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 28/30/32 ms
27

routerA#debug ip bgp BGP debugging is on 10:51:02: BGP: 7.72.6.3 open active, delay 6864ms 10:51:09: BGP: 7.72.6.3 open active, local address 27.27.27.253 10:51:09: BGP: 7.72.6.3 open failed: Connection refused by remote host
Is the remote router configured for BGP?

What IP address is the remote router configured to receive?
router bgp 100 no synchronization bgp log-neighbor-changes neighbor 7.72.6.1 remote-as 100
28

The TCP session is always sourced from the closest IP address to the destination!
27.27.27.254
Configuration:
Router A router bgp 100 neighbor 27.27.27.254 remote-as 100 Router C router bgp 100 neighbor 27.27.27.253 remote-as 100
27.27.27.253
If redundant paths exist, use loopback interfaces to establish the session.
29

router bgp 100 neighbor 7.72.6.3 remote-as 100 neighbor 7.72.6.3 update-source Loopback0
Information sourced from the IP address in interface Loopback0
routerA#debug ip tcp transactions 11:19:48: BGP: 7.72.6.3 open active, delay 9916ms 11:19:53: TCP: sending RST, seq 0, ack 3098129121 11:19:53: TCP: sent RST to 7.7.7.6:11719 from 7.72.6.1:179
Solution:
make sure both routers source the information from the appropriate interface
30

routerA#show ip bgp summary BGP router identifier 7.72.6.1, local AS number 100 BGP table version is 4, main routing table version 4 6 network entries and 6 paths using 774 bytes of memory 2 BGP path attribute entries using 96 bytes of memory 1 BGP AS-PATH entries using 24 bytes of memory BGP activity 6/0 prefixes, 6/0 paths Neighbor 6.72.6.2 7.25.14.4 7.72.6.3 7.75.7.1 V 4 4 4 4 AS MsgRcvd MsgSent 200 0 0 300 385 385 100 42 49 100 388 385 TblVer InQ OutQ Up/Down State/PfxRcd 0 0 0 never Idle 4 0 0 06:22:17 0 4 0 0 00:00:15 0 4 0 0 06:22:30 3
The eBGP session is still having trouble!

31
Cant Establish Session Troubleshooting II

Verify IP connectivity
check the routing table use ping/trace to verify two way reachability
routerA#show ip route 6.72.6.2 %Network not in table routerA#configure terminal Enter configuration commands, one per line. End with CNTL/Z. routerA(config)#ip route 6.72.6.2 255.255.255.255 1.1.1.5 routerA#ping 6.72.6.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 6.72.6.2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
32
Cant Establish Session Troubleshooting II (Cont.)

Peering with a loopback interface
Advantages Interface is always up Multiple physical paths may exist to reach it Disadvantages Physical link failure may take longer to detect
33

routerA#debug ip bgp routerA#debug ip tcp transactions 13:25:30: TCP: sending RST, seq 0, ack 2030100669 13:25:30: TCP: sent RST to 6.72.6.2:11041 from 3.72.6.1:179
router bgp 100 neighbor 6.72.6.2 remote-as 200 neighbor 6.72.6.2 update-source Loopback1
Neighbour is trying to peer with this IP address
The debug output indicates the neighbours configured peering address

34

13:33:30: TCP: sending RST, seq 0, ack 2510129645 13:33:30: TCP: sent RST to 6.72.6.2:11045 from 3.72.6.1:179
Hint: by default, eBGP peers should be directly connected

in this case, the peering address doesnt match a connected interface in the local router
35

routerA#show ip bgp neighbors 6.72.6.2 BGP neighbor is 6.72.6.2, remote AS 200, external link Index 1, Offset 0, Mask 0x2 BGP version 4, remote router ID 0.0.0.0 BGP state = Idle, table version = 0 Last read 00:00:06, last send never Hold time 180, keepalive interval 60 seconds Neighbor NLRI negotiation: Configured for unicast routes only Minimum time between advertisement runs is 30 seconds Received 0 messages, 0 notifications, 0 in queue Sent 0 messages, 0 notifications, 0 in queue Prefix advertised 0, suppressed 0, withdrawn 0 Route refresh request: received 0, sent 0 Connections established 0; dropped 0 Last reset never Number of unicast/multicast prefixes received 0/0 External BGP neighbor not directly connected. No active TCP connection
36

router bgp 100 neighbor 6.72.6.2 remote-as 200 neighbor 6.72.6.2 ebgp-multihop 255 neighbor 6.72.6.2 update-source Loopback1
At this point, the session should come up
37

routerA#show ip bgp summary BGP router identifier 7.72.6.1, local AS number 100 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down 6.72.6.2 4 200 10 26 0 0 0 never
State/PfxRcd Active
Still having trouble!

Connectivity issues have already been checked and corrected.
38

14:06:37: BGP: 6.72.6.2 open active, local address 3.72.6.1 14:06:37: BGP: 6.72.6.2 went from Active to OpenSent 14:06:37: BGP: 6.72.6.2 sending OPEN, version 4 14:06:37: BGP: 6.72.6.2 received NOTIFICATION 2/2 (peer in wrong AS) 2 bytes 0001 14:06:37: BGP: 6.72.6.2 remote close, state CLOSEWAIT 14:06:37: BGP: service reset requests 14:06:37: BGP: 6.72.6.2 went from OpenSent to Idle 14:06:37: BGP: 6.72.6.2 closing
If an error is detected, a notification is sent and the session is closed

In this case the remote router had a bad configuration
39
OPEN Message
0
1 2 3 4 5 6 7
9 10 11 12 13 14 15
16
17 18 19 20 21 22 23
24
25 26 27 28 29 30 31
32
Version My Autonomous System Hold Time BGP Identifier Opt. Parm. Len. Optional Parameters
40
Capabilities Negotiation
Allows for the advertisement of capabilities (type 2) Backwards compatible
New error subcode introduced to indicate which capabilities are not supportedthe session must be reset
Capability Code (1 Octet) Capability Length (1 Octet) Capability Value (Variable)
RFC2842 May2000 (Proposed Standard)
41
Wheres the Beef?

UPDATE Exchange
42
UPDATE Exchange
Once the session has been established, UPDATEs are exchanged

all the locally known routes only the bestpath is advertised
Incremental UPDATE messages are exchanged afterwards

43
Propagation Decisions
bestpath received from eBGP peer

advertise to all peers
bestpath received from iBGP peer

advertise only to eBGP peers a full iBGP mesh must exist
44
Common Problems
Missing routes
No iBGP full mesh Filters: routes are not received/sent
Slow convergence
45
UPDATE Filters
Type of filters
Prefix filters AS_PATH filters Community filters Any attribute may be used in a route-map
Applied incoming and/or outgoing

46
Missing Routes Troubleshooting Steps

Determine which filters are applied to the BGP session
show ip bgp neighbors x.x.x.x Look at the configuration
Examine the route and pick out the relevant attributes

show ip bgp x.x.x.x
47
Missing Routes Troubleshooting Steps (Cont.)
Compare the route against the filters If no match is found

Use route-refresh or soft-reconfiguration Filter the updates through an ACL to determine where the problem is
48
Missing RoutesSymptoms
Missing 4.0.0.0/8 in 7.75.7.1 (routerA)
not received from 7.72.6.3 (routerB)
routerB#sh ip bgp nei 7.75.7.1 advertised-routes | include 4.0.0.0 *> 4.0.0.0 0.0.0.0 0 32768 i
routerB shows that the route was advertised to routerA!
49
Missing RoutesTroubleshooting
routerA#show access-lists 10 Standard IP access list 10 permit 4.0.0.0 routerA#debug ip bgp 7.72.6.3 updates 10 BGP updates debugging is on for access list 10 for neighbor 7.72.6.3 routerA#clear ip bgp 7.72.6.3 in 01:22:41: BGP: 7.72.6.3 rcv UPDATE w/ attr: nexthop 7.72.6.3, origin i, metric 0, path 200 01:22:41: BGP: 7.72.6.3 rcv UPDATE about 4.0.0.0/8 -- DENIED due to: distribute/prefix-list;
50
Missing Routes Troubleshooting (Cont.)

router bgp 100 no synchronization bgp log-neighbor-changes neighbor 7.72.6.3 remote-as 200 neighbor 7.72.6.3 ebgp-multihop 255 neighbor 7.72.6.3 update-source Loopback0 neighbor 7.72.6.3 prefix-list filter in ! ip prefix-list filter seq 5 deny 4.0.0.0/8 ip prefix-list filter seq 10 permit 0.0.0.0/0 le 32
51
Slow ConvergenceSymptoms
The eBGP peering is established, but convergence is not complete even after several hours
routerA#show ip bgp summary ... Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 150.10.10.1 4 100 3550 3570 847 0 206 05:53:51 100
Possible causes
Remote router is not healthy (OutQ) Lower layer problems (IP)
52
Slow Convergence Troubleshooting

routerA#show ip route 150.10.10.1 Routing entry for 150.10.10.1/32 Routing Descriptor Blocks: 10.105.1.71, from 150.20.20.1, 00:06:14 ago, via POS2/1/0 * 156.1.1.1, from 150.20.20.1, 00:06:14 ago, via POS2/1/1 routerA#ping 150.10.10.1 Sending 5, 100-byte ICMP Echos to 150.10.10.1: !!!!! Reply to request 0 Record route: (156.1.1.2) (195.5.5.1) (10.105.1.134) (150.10.10.1) (10.105.1.76) (195.5.5.2) (156.1.1.1) (211.211.211.1) <*> Reply to request 1 Record route: (10.105.1.69) (140.10.50.5) (150.10.10.1) (140.10.50.6) (10.105.1.71) (211.211.211.1) <*>
Success is 100 percent, round-trip min/avg/max = 4/64/296 ms
Ping with route record option.

53
Slow Convergence Troubleshooting (Cont.)

eBGP Peering
OC-3
OC-3
A OC-3 OC-3 T3 router bgp 100 neighbor 150.10.10.1 remote-as 200 neighbor 150.10.10.1 ebgp-multihop 2 neighbor 150.10.10.1 update-source Loopback0
B OC-3
Longest path has more than 2 hops to the destination. Use higher TTL!
54
Pick One, Only One!

Route Selection Process
55
Route Selection Process
A common policy should be maintained across the AS to guarantee loop-free operation

Not all routers may select the same path
Filters may be used to modify or add attributes, affecting the selection algorithm
56
Common Problems
Inconsistent decision/policy
MED External paths Communities By default, communities are not propagated neighbor x.x.x.x send-community
57
Inconsistent Decision Symptom I

The bestpath changes every time the peering is reset.
routerA#sh ip bgp 160.100.0.0 BGP routing table entry for 160.100.0.0/16, version 40 Paths: (3 available, best #3, advertised over IBGP, EBGP) 100 204.146.33.10 from 204.146.33.10 (204.146.33.1) Origin IGP, metric 0, localpref 100, valid, internal 300 204.146.33.66 from 204.146.33.66 (204.146.33.2) Origin IGP, metric 20, localpref 100, valid, internal 300 204.146.33.6 from 204.146.33.6 (10.4.1.1) Origin IGP, metric 30, valid, external, best
58
Inconsistent Decision Symptom I (Cont.)

routerA#sh ip bgp 160.100.0.0 BGP routing table entry for 160.100.0.0/16, version 2 Paths: (3 available, best #3, advertised over EBGP) 100 204.146.33.10 from 204.146.33.10 (204.146.33.1) Origin IGP, metric 0, localpref 100, valid, internal 300 204.146.33.6 from 204.146.33.6 (10.4.1.1) Origin IGP, metric 30, valid, external 300 204.146.33.66 from 204.146.33.66 (204.146.33.2) Origin IGP, metric 20, localpref 100, valid, internal, best
Same paths, but different result!

59
Inconsistent Decision Symptom I (Cont.)

routerA#sh ip bgp 160.100.0.0 BGP routing table entry for 160.100.0.0/16, version 12 Paths: (3 available, best #3, advertised over EBGP) 300 204.146.33.6 from 204.146.33.6 (10.4.1.1) Origin IGP, metric 30, valid, external 300 204.146.33.66 from 204.146.33.66 (204.146.33.2) Origin IGP, metric 20, localpref 100, valid, internal 100 204.146.33.10 from 204.146.33.10 (204.146.33.1) Origin IGP, metric 0, localpref 100, valid, internal, best
Different resultagain!!
60
Deterministic MED
By default, the prefixes are compared in order of arrival

it may result in inconsistent decisions use bgp deterministic-med the bestpath is recalculated as soon as the command is entered enable in all the routers in the AS
61
Deterministic MEDOperation
The paths are ordered by peer-AS The bestpath for each group is selected The overall bestpath results from comparing the winners in each group
62
Deterministic MEDResult
routerA#sh ip bgp 160.100.0.0 BGP routing table entry for 160.100.0.0/16, version 15 Paths: (3 available, best #1, advertised over EBGP) 100 204.146.33.10 from 204.146.33.10 (204.146.33.1) Origin IGP, metric 0, localpref 100, valid, internal, best 300 204.146.33.66 from 204.146.33.66 (204.146.33.2) Origin IGP, metric 20, localpref 100, valid, internal 300 204.146.33.6 from 204.146.33.6 (10.4.1.1) Origin IGP, metric 30, valid, external
The bestpath will always be the same!

63
Inconsistent Decision Symptom II

The bestpath changes every time the peering is reset
routerA#show ip bgp 7.0.0.0 BGP routing table entry for 7.0.0.0/8, version 15 Paths: (2 available, best #2) Not advertised to any peer 200 1.1.1.5 from 1.1.1.5 (1.1.1.1) Origin IGP, metric 0, localpref 100, valid, external 200 21.21.21.254 from 21.21.21.254 (7.75.7.1) Origin IGP, metric 0, localpref 100, valid, external, best
64
Inconsistent Decision Symptom II (Cont.)

routerA#show ip bgp 7.0.0.0 BGP routing table entry for 7.0.0.0/8, version 17 Paths: (2 available, best #2) Not advertised to any peer 200 21.21.21.254 from 21.21.21.254 (7.75.7.1) Origin IGP, metric 0, localpref 100, valid, external 200 1.1.1.5 from 1.1.1.5 (1.1.1.1) Origin IGP, metric 0, localpref 100, valid, external, best
The oldest external is the bestpath.

All other attributes are the same Stability enhancement!
65
Route Reflectors
Playing with Mirrors
66
Route Reflectors
Provide additional control to allow router to advertise (reflect) iBGP learned routes to other iBGP peers
Method to reduce the size of the iBGP mesh
Normal BGP speakers can coexist

Only the RR has to support this feature
67
Route ReflectorsTerminology
Non-Client Route Reflector
Clusters Clients Clients
Lines Represent Both Physical Links and BGP Logical Connections

68
Reflection Decisions
Once the best path is selected:

From non-client reflect to all clients From client reflect to all non-clients AND other clients From eBGP peer reflect to all clients and non-clients
69
Common Problems
Missing routes Routing loops and close calls
70
Missing RoutesSymptoms
At least one route is missing from at least one router in the network.
routerA#show ip bgp 4.0.0.0 % Network not in table routerA#show ip bgp summary BGP router identifier 7.25.14.4, local AS number 100 BGP table version is 1, main routing table version 1 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 7.72.6.2 4 100 7 7 1 0 0 00:04:18 0
71
Missing RoutesTroubleshooting
Check routers for filters
routerA# router bgp 100 no synchronization neighbor 7.72.6.2 remote-as 100
routerB# router bgp 100 no synchronization bgp cluster-id 0.0.0.5 neighbor 7.25.14.4 remote-as 100 neighbor 7.25.14.4 route-reflector-client neighbor 7.72.6.1 remote-as 100 routerB#show ip bgp 4.0.0.0 % Network not in table
routerC
72
Missing Routes Troubleshooting I

Follow the path where the routes should have been learned
routerC# router bgp 100 no synchronization Same bgp cluster-id 0.0.0.5 Cluster-ID! neighbor 7.72.6.2 remote-as 100 neighbor 7.75.7.1 remote-as 100 neighbor 7.75.7.1 route-reflector-client routerC#show ip bgp | include 4.0.0.0 *>i4.0.0.0 7.72.6.3 0
100
0 200 i
73
Missing Routes Troubleshooting II

Alternative way to find duplicate cluster-id
use route-refresh + debug ip bgp updates ACL
routerB#clear ip bgp 7.72.6.1 in 21:45:40: BGP: 7.72.6.1 rcv UPDATE w/ attr: nexthop 7.72.6.3, origin i, localpref 100, metric 0, path 2 21:45:40: BGP: 7.72.6.1 rcv UPDATE about 4.0.0.0/8 -DENIED due to: reflected from the same cluster;
74
Missing Routes Troubleshooting (Cont.)

Clusters with multiple RRs
If the RRs have the same cluster-id, all the clients must peer with all the reflectors
If not needed for administration, dont assign a cluster-id for added flexibility!
RR
RR
Lines Represent Both Physical Links and BGP Logical Connections
75
Routing LoopSymptom
routerD#traceroute 7.1.1.1 1 1.1.1.2 24 msec 24 msec 40 msec

rtrB rtrC
Loop!
2 156.1.1.1 28 msec 48 msec 24 msec 3 156.1.1.2 24 msec 24 msec 24 msec 4 156.1.1.1 28 msec 28 msec 24 msec 5 156.1.1.2 28 msec 28 msec 28 msec 6 156.1.1.1 28 msec 28 msec 32 msec
Lines Represent Physical Connections
76
Routing LoopTroubleshooting
Verify Routing Information
routerC#show ip bgp 7.0.0.0 BGP routing table entry for 7.0.0.0/8 1 150.10.10.1 (metric 115) from 150.10.10.1 (150.20.20.1) Origin IGP, valid, external, best routerC#show ip route 150.10.10.1 Routing entry for 150.10.10.1/32 Routing Descriptor Blocks: * 156.1.1.1, from 150.20.20.1, via Ethernet2/1/1 routerB#show ip bgp 7.0.0.0 BGP routing table entry for 7.0.0.0/8 1 156.1.1.2 from 156.1.1.2 (212.212.212.1) Origin IGP, valid, internal, best routerB#show ip route 156.1.1.2 Routing entry for 156.1.1.0/24 Routing Descriptor Blocks: * directly connected, via Ethernet1
Addresses on Same Subnet!

77
Routing Loop Troubleshooting (Cont.)
Check configuration
routerC# router bgp 134 neighbor 150.10.10.1 remote-as 1 neighbor 150.10.10.1 ebgp-multihop 255 neighbor 150.10.10.1 update-source Loopback0 neighbor 156.1.1.1 remote-as 134 neighbor 156.1.1.1 route-reflector-client neighbor 156.1.1.1 next-hop-self ! ip route 150.10.10.1 255.255.255.255 s0 250 A-RR
B D
C-RR

78
Routing Loop Troubleshooting (Cont.)
Solution
Establish the eBGP peering permanently through the backup link Use LOCAL_PREF or MED to break any tie!
A-RR
B D
C-RR

79
Close CallSymptom
The bestpath is not being followed to an external destination
routerA#show ip route 4.4.4.4 Routing entry for 4.0.0.0/8 Known via "bgp 1", distance 200, metric 0 Tag 2, type internal Last update from 6.72.6.3 00:25:45 ago Routing Descriptor Blocks: * 6.72.6.3, from 7.75.7.1, 00:25:45 ago Route metric is 0, traffic share count is 1 AS Hops 1, BGP network version 0 Expected to go out though the NEXT_HOP in the update.
80
Close CallSymptom (Cont.)

routerA#traceroute 4.4.4.4 1 2 3 4 5 6 1.1.1.4 8.25.14.3 172.18.176.1 161.44.0.56 161.44.0.18 4.4.4.4 0 8 8 8 8 4 msec msec msec msec msec msec 0 4 4 8 8 5 msec msec msec msec msec msec 0 4 4 8 4 4 msec msec msec msec msec msec
All eBGP peers are configured with their interface address

The NEXT_HOP is expected in the trace
81
Close Call Troubleshooting (Cont.)

Verify configuration
Check for alternate routes
router bgp 100 no synchronization neighbor 7.75.7.1 remote-as 100 neighbor 7.75.7.1 update-source Loopback0
routerC
routerA#show ip bgp 4.0.0.0 BGP routing table entry for 4.0.0.0/8, version 2 Paths: (1 available, best #1) Not advertised to any peer 200 6.72.6.3 (metric 103) from 7.75.7.1 (7.75.7.1) Origin IGP, metric 0, localpref 100, valid, internal, best
82

router bgp 100 no synchronization bgp log-neighbor-changes neighbor 7.72.6.1 remote-as 100 neighbor 7.72.6.1 update-source Loopback0 neighbor 7.72.6.1 route-reflector-client neighbor 7.72.6.2 remote-as 100 neighbor 7.72.6.2 update-source Loopback0 neighbor 7.72.6.2 route-reflector-client neighbor 6.72.6.3 remote-as 200
routerA routerB
routerC is a RR with two clients

83

routerC#show ip bgp 4.0.0.0 BGP routing table entry for 4.0.0.0/8, version 2 200, (Received from a RR-client) 8.25.14.3 (metric 3) from 7.72.6.1 (7.72.6.1) Origin IGP, metric 0, localpref 100, valid, internal 200 6.72.6.3 (metric 2) from 6.72.6.3 (6.72.6.3) Origin IGP, metric 0, localpref 100, valid, external, best
routerC#traceroute 4.4.4.4 1 2 3 4 6.72.6.3 161.44.0.56 161.44.0.18 4.4.4.4 0 0 0 0 msec msec msec msec 4 4 4 4 msec msec msec msec 0 0 0 0 msec msec msec msec
Expected Path!
84

routerB#traceroute 4.4.4.4 1 2 3 4 5 8.25.14.3 172.18.176.1 161.44.0.48 161.44.0.15 4.4.4.4 8 msec 16 msec 8 msec 16 msec 12 msec 16 msec 12 msec 16 msec 12 msec 16 msec 12 msec 16 msec 8 msec 8 msec 8 msec
Output from the other client

follows the same exit as routerA
85

routerB#show ip bgp 4.0.0.0 BGP routing table entry for 4.0.0.0/8, version 13 200 8.25.14.3 (metric 2) from 8.25.14.3 (8.25.14.3) Origin IGP, metric 0, localpref 100, valid, external, best 200 6.72.6.3 (metric 50) from 7.75.7.1 (7.75.7.1) Origin IGP, metric 0, localpref 100, valid, internal
routerB is following the correct path!

86

The logical connection between routerC (RR) and routerA provides the route
The physical path is followed
D
Logical Connection 4.0.0.0/8
B C C
87
Route Flap Damping

Stabilising the Network
2210 1351_06_2000_c2 2000, Cisco Systems, Inc. 2001, Cisco Systems, Inc.
www.cisco.com
88
Route Flap Damping

Route flap
Going up and down of path or change in attribute
BGP WITHDRAW followed by UPDATE = 1 flap eBGP neighbour going down/up is NOT a flap
Ripples through the entire Internet Wastes CPU
Damping aims to reduce scope of route flap propagation
Implementation documented in RFC2439

89
Route Flap Damping Operation

Add penalty (1000) for each flap Exponentially decay penalty
half life determines decay rate
Penalty above suppress-limit

do not advertise route to BGP peers
Penalty decayed below reuse-limit

re-advertise route to BGP peers
90
Route Flap Damping

4000 Suppress limit 3000
Penalty
2000 Reuse limit 1000
0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Time
Network Announced
Network Not Announced
Network Re-announced
91
Route Flap Damping Operation

Only applied to inbound announcements from eBGP peers Alternate paths still usable Controlled by:
Half-life (default 15 minutes) reuse-limit (default 750) suppress-limit (default 2000) maximum suppress time (default 60 minutes)
92
Route Flap Damping Configuration

Fixed damping
router bgp 100 bgp dampening [<half-life> <reuse-value> <suppress-penalty> <maximum suppress time>]
Selective and variable damping

bgp dampening [route-map <name>] route-map <name> permit 10 match ip address prefix-list FLAP-LIST set dampening [<half-life> <reuse-value> <suppress-penalty> <maximum suppress time>] ip prefix-list FLAP-LIST permit 192.0.2.0/24 le 32
93
Care required when setting parameters Penalty must be less than reuse-limit at the maximum suppress time Maximum suppress time and half life must allow penalty to be larger than suppress limit
94

Examples -
bgp dampening 30 750 3000 60
reuse-limit of 750 means maximum possible penalty is 3000 - no prefixes suppressed as penalty cannot exceed suppress-limit
Examples -
reuse-limit of 2000 means maximum possible penalty is 8000 - suppress limit is easily reached
95

Examples -
reuse-limit of 500 means maximum possible penalty is 2000 - no prefixes suppressed as penalty cannot exceed suppress-limit
Examples -
reuse-limit of 750 means maximum possible penalty is 6000 - suppress limit is easily reached
96
Route Flap Damping Maths!
Maximum possible value of penalty is
Always make sure that suppress-limit chosen is LESS than max-penalty otherwise there will be no route damping
97
Route Flap Damping Status

Flapping prefixes
gw>sh ip bgp flap-statistics BGP table version is 8444024, local router ID is 202.12.29.64 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network *> 63.74.112.0/22 *> 63.74.120.0/22 *d 63.109.97.0/24 *> 63.140.98.0/24 *> 64.254.102.0/24 h 66.0.0.0 *> 66.9.38.0/23 *d 66.40.0.0/18 *d 139.78.0.0 *d 147.39.200.0/24 h 192.31.83.0 ... From 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21 Flaps Duration Reuse 3 3 12 4 2 2 3 11 7 7 3 00:18:08 00:18:08 Path 1221 16779 1 701 13593 1221 16779 1 701 13593
00:42:07 00:28:20 1221 16779 1 701 17106 00:29:48 00:03:11 00:14:43 00:21:22 1221 5696 15322 1221 16779 1 701 17307 1221 16779 1833 701 1239 4231 1221 16779 1 3356 16449
00:35:00 00:26:30 1221 11388 00:10:53 00:30:50 1221 5727 1 2828 7911 7294 5078 00:41:02 00:17:10 1221 5727 7018 701 702 1913 1559 00:05:11 1221 16779 1 2828 7911 7294 5078
98
Route Flap Damping Status

Damped prefixes
gw>sh ip bgp dampened-paths BGP table version is 8443716, local router ID is 202.12.29.64 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network *d 202.8.238.0 *d 216.148.175.0 *d 212.9.128.0/19 *d 200.23.202.0 *d 148.219.0.0 *d 203.10.1.0 *d 206.137.4.0/22 *d 195.54.160.0/19 *d 192.146.206.0 ...
From 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21 203.62.252.21
Reuse
Path
00:04:10 1221 5727 9658 9386 i 00:05:50 1221 5727 1740 11282 11282 i 00:09:30 1221 16779 1833 1299 1755 8787 i 00:12:20 1221 1 3561 6503 3454 3454 3454 i 00:13:40 1221 1 1239 8151 278 i 00:13:20 1221 2764 2764 2764 2764 9543 i 00:14:10 1221 1 3561 6347 18435 i 00:16:20 1221 5727 1239 1755 8525 ? 00:21:50 1221 5727 1239 5078 5078 i
99
Route Flap Damping Monitoring

Monitoring route flaps
gw#debug ip bgp damp BGP dampening debugging is on
Mar 9 15:49:26.443 AEST: BGP: charge penalty for 204.134.66.0/24 path 1221 16779 1 1239 3912 with halflife-time 15 reuse/suppress 750/2000 Mar 9 15:49:26.443 AEST: 1000 ... Mar 9 15:49:26.667 AEST: BGP: charge penalty for 203.17.144.0/22 path 1221 5727 7018 3561 7474 7569 7570 4738 7572 7571 7476 with halflife-time 15 reuse/suppress 750/2000 Mar 9 15:49:26.667 AEST: 2766 flapped 5 times since 00:03:39. New penalty is flapped 1 times since 00:00:00. New penalty is
Mar 9 15:49:26.667 AEST: BGP: suppress 203.17.144.0/22 path 1221 5727 7018 3561 7474 7569 7570 4738 7572 7571 7476 for 00:28:10 (penalty 2766) Mar 9 15:49:26.671 AEST: halflife-time 15, reuse/suppress 750/2000
100
Route Flap Damping Monitoring

Monitoring route flaps (more)
Mar 9 16:00:32.021 AEST: BGP: unsuppressed 203.255.232.0/24, path 1221 6461 3786 1237 Mar 9 16:03:38.358 AEST: BGP: unsuppressed 204.228.71.0/24, path 1221 16779 1 3908 14041 12145 Mar 9 16:03:38.358 AEST: BGP: unsuppressed 204.228.64.0/24, path 1221 16779 1 3908 14041 12145 Mar 9 16:03:38.358 AEST: BGP: unsuppressed 198.59.93.0/24, path 1221 16779 1 3908 14041 12145 Mar 9 16:03:38.358 AEST: BGP: unsuppressed 198.59.87.0/24, path 1221 16779 1 3908 14041 12145 Mar 9 16:04:10.570 AEST: BGP: unsuppressed 207.40.160.0/20, path 1221 16779 1 1239 1791 Mar 9 16:04:10.570 AEST: BGP: unsuppressed 207.40.64.0/20, path 1221 16779 1 1239 1791 Mar 9 16:04:20.994 AEST: BGP: unsuppressed 203.91.145.0/24, path 1221 16779 1 701 703 9237 Mar 9 16:04:51.914 AEST: BGP: unsuppressed 203.168.64.0/20, path 1221 1 3561 4755 17632
101
Route Flap Damping Operations
Be aware:
Many ISPs use flap damping Unreachable remote networks may be due to damping, not disconnections Damping applies to flapping prefixes, not unstable connections
102
Summary
2210 1351_06_2000_c2 2000, Cisco Systems, Inc. 1999, Cisco Systems, Inc.
www.cisco.com
103 103
Summary/Tips
Isolate the problem! Use ACLs when enabling debug commands Enable bgp log-neighbor-changes IP reachability must exist for sessions to be established
Learned from IGP Make sure the source and destination addresses match the configuration
104
Summary/Tips
Use loopback interfaces for stability and where multiple paths exist Use common filters
Keep them simple!
Maintain a consistent policy throughout the AS Use deterministic-med

105
Summary/Tips
Select the appropriate knob/attribute for the job
Learn the decision algorithm
Route reflectors
Follow the physical topology Define a cluster-id only if administratively needed
BGP Damping
Useful for enhancing stability of Internet connected networks
106
Please Complete Your Evaluation Form

Troubleshooting BGP
2210 1351_06_2000_c2 Inc.1999, Cisco Systems, Inc. 2000, Cisco Systems, 107
2210 1351_06_2000_c2
108

Troubleshooting BGP Net Workers, 2001)

Caricato da

Informazioni sul documento

Descrizione originale:

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Troubleshooting BGP Net Workers, 2001)

Caricato da

Copyright:

Formati disponibili

Networkers 2001, Australia

March 28-30, Brisbane

Presentation_ID 2001, Cisco Systems, Inc. 2000, Cisco Systems, Inc.

BGP in Large Scale Networks

Scalable Stable Simple

Avoid the Problem in the First Place

Promote stable networks

Grow into your network

2210 1351_06_2000_c2 Inc.2000, Cisco Systems, Inc. 2000, Cisco Systems,

BGP Troubleshooting Tools

show commands debug output Log messages

2000, Cisco Systems, Inc.

show Commands (Cont.)

2000, Cisco Systems, Inc.

The BGP Table

2000, Cisco Systems, Inc.

The BGP Table (Cont.)

2000, Cisco Systems, Inc.

show ip bgp Summary

2000, Cisco Systems, Inc.

show ip bgp neighbors

To make a little safer:

Session Establishment (debug ip bgp )

Session Establishment (debug ip bgp events)

2000, Cisco Systems, Inc.

Looking at the Updates

Use an access-list to limit the output!

debug ip bgp Updates

Prefix Being Advertised

debug ip bgp Updates (Cont.)

Peer Address Prefixes in the Same UPDATE

Attributes Apply to All Prefixes

Logging Neighbor Changes

2000, Cisco Systems, Inc.

show ip bgp neighbors x.x.x.x

2000, Cisco Systems, Inc.

Come Meet the Neighbors!

2210 1351_06_2000_c2 Inc.2000, Cisco Systems, Inc. 2000, Cisco Systems,

OPEN messages are exchanged

Sessions are not established

2000, Cisco Systems, Inc.

Cant Establish Session Symptoms

The peering session is not established!

Cant Establish Session Troubleshooting I

Is the remote-as assigned correctly?

2000, Cisco Systems, Inc.

Cant Establish Session Troubleshooting I (Cont.)

Cant Establish Session Troubleshooting I (Cont.)

Is the remote router configured for BGP?

Cant Establish Session Troubleshooting I (Cont.)

If redundant paths exist, use loopback interfaces to establish the session.

Cant Establish Session Troubleshooting I (Cont.)

Information sourced from the IP address in interface Loopback0

Cant Establish Session Symptoms

The eBGP session is still having trouble!

Cant Establish Session Troubleshooting II

Cant Establish Session Troubleshooting II (Cont.)

Cant Establish Session Troubleshooting II (Cont.)

Neighbour is trying to peer with this IP address

The debug output indicates the neighbours configured peering address

Cant Establish Session Troubleshooting II (Cont.)

Hint: by default, eBGP peers should be directly connected

Cant Establish Session Troubleshooting II (Cont.)