Expert BGP Troubleshooting: Five Broken Scenarios, Ticket Style

The CCIE troubleshooting section does not ask you to explain BGP. It hands you a broken network and a clock. The difference between passing and failing is not knowing more BGP - it is having a repeatable method for turning a symptom into a cause in under ten minutes.

This article is five broken scenarios, each built and broken for real in a CML lab, each with the actual diagnostic output and the actual fix. Every one of them is a mistake that has taken down a production network. Work through them the way you would work a ticket. For the underlying theory, see the complete BGP guide.

The method, before the tickets

Every BGP problem is one of exactly four things, and they must be checked in this order:

The single most under-used command in BGP troubleshooting is this one:

show ip bgp neighbors <peer> | section Policy

It gives you a running count of every prefix the router discarded and why. Route-map. Prefix-list. AS-path loop. Maxprefix. It is a confession, and the router volunteers it. Use it in every ticket.

One prerequisite: to see what a neighbour sent you before your policy chewed it, you need soft-reconfiguration inbound on that neighbour. Turn it on when you start troubleshooting (it costs memory, so turn it off after).

Ticket 1: "The iBGP session keeps resetting, but it's up"

Symptom: R1 and R2 are iBGP peers in AS 65001, using loopbacks. The session is Established. But the log is full of resets and the operator swears something is wrong.

Diagnosis. The summary looks healthy:

R2#show ip bgp summary
Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
1.1.1.1         4        65001       7       6       56    0    0 00:00:01        8

Established, 8 prefixes. But look at the uptime: one second. And now the detail:

R2#show ip bgp neighbors 1.1.1.1 | include BGP state|Last reset|Local host|Foreign host
  BGP state = Established, up for 00:00:01
  Last reset 00:00:11, due to Active open failed
Local host: 2.2.2.2, Local port: 179
Foreign host: 1.1.1.1, Foreign port: 43797

Last reset ... due to Active open failed. R2's own attempt to open the session failed. The session that is currently up was opened by R1 - you can see it in the ports: R2's local port is 179, meaning R2 is the passive side. R2 accepted an inbound connection; it never successfully made an outbound one.

Cause. R2 is missing neighbor 1.1.1.1 update-source Loopback0. When R2 initiates, it sources from the physical interface, 10.0.12.2. R1 expects a connection from 2.2.2.2 and rejects it. But R1 does have update-source configured, so R1's outbound connection arrives correctly sourced from 1.1.1.1, and R2 happily accepts it. The session works, and the misconfiguration is masked.

Remove update-source from both sides and the mask comes off:

R1#show ip bgp summary
Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
2.2.2.2         4        65001       0       0        1    0    0 00:00:13 Idle

R1#show ip bgp neighbors 2.2.2.2 | include BGP state|Last reset|No active TCP
  BGP state = Idle, down for 00:00:14
  Last reset 00:00:14, due to Active open failed
  No active TCP connection

Fix: neighbor x.x.x.x update-source Loopback0 on both peers.

Lesson: "Active open failed" on an Established session means the session is only up because the other router dialled. That is a half-broken config waiting for the day the other router reboots first.

Ticket 2: "The routes are in the BGP table but not the routing table"

Symptom: R2 has received 100.100.100.0/24 over iBGP from R1. It is in show ip bgp. It is not in show ip route. Traffic to it is dropped.

Diagnosis. Two flags tell the whole story:

R2#show ip bgp 100.100.100.0/24
BGP routing table entry for 100.100.100.0/24, version 0
Paths: (1 available, no best path)
  100
    10.0.13.2 (inaccessible) from 1.1.1.1 (1.1.1.1)
      Origin IGP, metric 0, localpref 200, valid, internal

R2#show ip bgp | include 100.100
 * i  100.100.100.0/24 10.0.13.2                0    200      0 100 i

no best path. (inaccessible). And in the summary line there is a * but no >.

Cause. The next-hop is 10.0.13.2 - an address on the link between R1 and the ISP. R2 has no route to it. The IGP does not carry the eBGP link subnets (and should not). BGP requires a resolvable next-hop before a path can become best, so the route sits in the table doing nothing.

R1 is missing neighbor 2.2.2.2 next-hop-self. Without it, R1 passes the external next-hop through to its iBGP peer unchanged.

Fix:

router bgp 65001
 address-family ipv4
  neighbor 2.2.2.2 next-hop-self

The trap. This one is far nastier than it looks, because eBGP multipath hides it completely. In the lab, R1 had maximum-paths 2 configured. With multipath active, removing next-hop-self changed nothing at all - R2 still received next-hop 1.1.1.1, because a router advertising a multipath route to an iBGP peer sets itself as the next-hop (there is no single external next-hop to pass along). Only after removing maximum-paths 2 did the failure appear.

So: on any edge router with eBGP multipath, a missing next-hop-self is invisible - right up until the day one circuit fails, the prefix drops to a single path, and your entire internal routing collapses. Configure next-hop-self explicitly on every iBGP session from an eBGP edge router, whether or not it appears to be needed. See the multipath article for the full walkthrough.

Ticket 3: "We only get one route from the provider"

Symptom: R2 peers with ISP-B (AS 200). The provider insists they are sending three prefixes. R2 has one.

Diagnosis. The prefix count in the summary already says something is being dropped:

R2#show ip bgp summary
Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.25.2       4          200      89      85      112    0    0 00:36:30        1

With soft-reconfiguration inbound enabled, compare what arrived against what survived:

R2#show ip bgp neighbors 10.0.25.2 received-routes
     Network          Next Hop            Metric LocPrf Weight Path
 *    5.5.5.5/32       10.0.25.2                0             0 200 i
 *    6.6.6.6/32       10.0.25.2                              0 200 300 i
 *    6.6.60.0/24      10.0.25.2                              0 200 300 i
Total number of prefixes 3

R2#show ip bgp neighbors 10.0.25.2 routes
     Network          Next Hop            Metric LocPrf Weight Path
 *>   5.5.5.5/32       10.0.25.2                0    150      0 200 i
Total number of prefixes 1

Three in, one out. The provider is telling the truth. Now ask the router why:

R2#show ip bgp neighbors 10.0.25.2 | section Policy
  Local Policy Denied Prefixes:    --------    -------
    route-map:                            0          2

Two prefixes denied by a route-map, inbound.

Cause. Someone wrote an inbound route-map to set local-preference on a specific prefix, and forgot that a route-map ends in an implicit deny:

route-map ISPB-IN permit 10
 match ip address prefix-list ISPB-ALLOW
 set local-preference 150

! ...and nothing else. Everything not matching ISPB-ALLOW is denied.

Fix: a bare permit clause at the end.

route-map ISPB-IN permit 20

Lesson: this is, by a wide margin, the most common BGP outage in the world. Any time you attach a route-map to a neighbour to set something, you have also, silently, attached a filter. Every route-map used for attribute manipulation needs a terminating permit. Every single one.

Ticket 4: "The neighbour is advertising it, we're not receiving it, and there's no filter"

Symptom: R5 (ISP-B) demonstrably advertises 10.10.10.0/24 to R2. R2 does not have it. There is no inbound filter of any kind on R2.

Diagnosis. First confirm the sender is not lying:

R5#show ip bgp neighbors 10.0.25.1 advertised-routes
     Network          Next Hop            Metric LocPrf Weight Path
 *>   1.1.1.1/32       10.0.56.2                     200      0 300 100 65001 i
 *>   10.10.10.0/24    10.0.56.2                     200      0 300 100 65001 i

It is being sent. Now look at R2's policy counters:

R2#show ip bgp neighbors 10.0.25.2 | section Policy
  Local Policy Denied Prefixes:    --------    -------
    AS_PATH loop:                       n/a         30
    Bestpath from this peer:              9        n/a
    Other Policies:                       4        n/a
    Total:                               13         30

There is the answer, in one line. AS_PATH loop: 30.

Cause. Look at the AS-path R5 is sending: 300 100 65001. R2 is in AS 65001. That prefix originated in R2's own AS, travelled out through ISP-A, across AS 300, through ISP-B, and is now coming home. BGP's fundamental loop-prevention rule fires: if my own ASN appears in the AS-path of an inbound update, discard it silently.

This is not a bug. This is BGP working exactly as designed and protecting you from a routing loop. But it is invisible unless you look at the right counter, because the route never enters the BGP table - not even the pre-policy soft-reconfiguration copy. The AS-path check happens at parse time, before storage.

The real question is why the prefix is looping at all. Something upstream is transiting routes it should not. In the lab, AS 300 was leaking: it re-advertised routes learned from ISP-A to ISP-B, turning itself into unintended transit. The fix is not on R2. The fix is an outbound filter at AS 300 restricting it to its own prefixes.

When you actually want to accept it: in an MPLS L3VPN, a customer with the same AS number at multiple sites legitimately sees its own ASN in the path. That is what allowas-in and as-override exist for - covered in the transport articles under the MPLS pillar.

Ticket 5: "IPv4 is fine, IPv6 is completely dead"

Symptom: R1 and R3 have both an IPv4 and an IPv6 eBGP session. IPv4 is perfect. The IPv6 session will not stay up and R1 has lost all its IPv6 routes.

Diagnosis.

R1#show bgp ipv6 unicast summary
Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
2001:DB8:13::2  4          100       0       0        1    0    0 00:00:12 Idle

R1#show bgp ipv6 unicast
     Network          Next Hop            Metric LocPrf Weight Path
 *>   2001:DB8:10::/64 ::                       0         32768 i

Idle. Zero messages sent, zero received. And only R1's own locally-originated prefix remains - everything learned from the peer is gone.

Meanwhile IPv4 is entirely unaffected:

R1#show ip bgp summary
Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.13.2       4          100      27      36      108    0    0 00:14:08        3

Cause. On R3, someone removed the activation:

router bgp 100
 address-family ipv6
  no neighbor 2001:DB8:13::1 activate

With no bgp default ipv4-unicast in effect (as it should be in any multiprotocol design), a neighbour that is not activated in any address family has nothing to negotiate. The OPEN carries no usable AFI/SAFI capability, and the session drops to Idle. It does not stay up-but-empty. It goes down.

Fix:

router bgp 100
 address-family ipv6
  neighbor 2001:DB8:13::1 activate
  neighbor 2001:DB8:13::1 send-community both

Lesson: a BGP neighbour is defined globally but must be activated per address family. Every multiprotocol BGP problem starts with checking activate. Zero MsgSent and zero MsgRcvd on an Idle session is the signature - the routers are not even trying to talk.

The commands, collected

! Session layer
show ip bgp summary
show ip bgp neighbors <peer> | include BGP state|Last reset|No active TCP|Local host|Foreign host

! What is being sent
show ip bgp neighbors <peer> advertised-routes

! What is being received, and what survived policy
show ip bgp neighbors <peer> received-routes     (needs soft-reconfiguration inbound)
show ip bgp neighbors <peer> routes
show ip bgp neighbors <peer> | section Policy    <-- the confession

! Why is it not in the RIB
show ip bgp <prefix>                             (look for: no best path, (inaccessible), the > flag)
show ip route <prefix>
show ip cef <prefix>

! Multiprotocol
show bgp ipv6 unicast summary
show bgp ipv6 unicast <prefix>
show bgp ipv6 unicast labels                     (6PE)

Try it yourself

Build the lab, then break it. Each of these produces a distinct, findable signature. Give yourself ten minutes per fault and do not look at the config until you have named the cause from show output alone:

1. Remove update-source from one side of an iBGP session, then from both. Note the difference.
2. Remove next-hop-self from an eBGP edge router with multipath on, then turn multipath off. Watch the failure appear from nowhere.
3. Attach an inbound route-map with a match and a set and no terminating permit.
4. Remove an upstream AS's outbound filter so it starts transiting your own prefixes back to you.
5. De-activate an IPv6 neighbour on one side only.
6. Bonus: set neighbor x maximum-prefix 2 and watch the session tear itself down. Find the counter that tells you why.

Key takeaways

• Work the layers in order: session up, prefix sent, prefix accepted, prefix used. Do not skip.
• show ip bgp neighbors <peer> | section Policy tells you exactly how many prefixes were discarded and by what. Run it first, always.
• received-routes vs routes, with soft-reconfiguration inbound, is how you separate "they did not send it" from "we threw it away".
• "Active open failed" on an Established session means only the far end can dial. Fix it before it fixes you.
• (inaccessible) plus no best path is always a next-hop problem, and usually a missing next-hop-self - which eBGP multipath will hide from you.
• An implicit deny at the end of an attribute-setting route-map is the most common self-inflicted BGP outage there is.
• AS-path loop rejections never enter the BGP table. The only evidence is the counter.
• Multiprotocol BGP: activate per address family, or the session does not come up at all.

That closes the expert BGP series. The full cluster index, from first principles to here, lives on the BGP pillar guide.