Expert Transport Troubleshooting: MPLS and DMVPN Ticket Scenarios

Transport technologies fail in layers, and the art of troubleshooting them is knowing which layer to suspect. An MPLS L3VPN has an IGP, LDP, MP-BGP, VRFs, and a label stack, any one of which can break while the others look fine. A DMVPN has an underlay, NHRP, IPsec, mGRE, and a routing protocol on top - five moving parts that all have to work for a spoke to reach a spoke.

This article is five transport faults, each grounded in the real behaviour we saw building the MPLS and DMVPN labs for this series, with the diagnostic commands that isolate each layer. For the theory, see the MPLS guide and the DMVPN guide.

The method: isolate the layer

For MPLS L3VPN, work bottom-up:

For DMVPN, also bottom-up: underlay reachability → NHRP registration → IPsec SA → mGRE tunnel → routing protocol. A break at any layer looks like "the tunnel is down" from the top.

Ticket 1: "The customer VPN route is in BGP but not forwarding"

Symptom. A remote customer prefix appears in show bgp vpnv4 unicast all on the PE, imports into the VRF, but traffic to it is dropped.

Diagnosis. The route is present, so BGP and RT import are fine. The problem is the label stack. Check whether the VPN prefix has a valid label path:

PE1#show mpls forwarding-table vrf CUST-A 2001:DB8:BB::22/128
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Outgoing   or Tunnel Id     Switched      interface
None       19         2001:DB8:BB::22/128[V]  0       Et0/2      10.0.12.2

Here the out-label is 19 and there is an outgoing interface - good. If instead you see No Label or no entry, the VPN label was assigned by BGP but there is no transport LSP to carry it. That means LDP is broken between the PEs.

Cause and fix. Almost always a broken LDP LSP to the remote PE loopback. Check show mpls ldp neighbor and confirm the loopback is a /32 in the IGP with an LDP binding. A very common cause is mpls ip missing on a core interface, or the PE loopback advertised as something other than a /32 (LDP will not build a host LSP to a summarised loopback). The two-label stack needs both labels; a present VPN label with a missing transport label forwards nowhere.

Ticket 2: "Two customer sites with the same AS can't reach each other"

Symptom. Site A and Site B of the same customer, both AS 65100, cannot exchange routes over the L3VPN. Everything else works.

Diagnosis. The route is being dropped by BGP's AS-path loop check. On the receiving CE it simply is not there:

CE2#show ip route 11.11.11.11
% Network not in table

And the PE facing that CE shows the drop in its policy counters:

PE2#show ip bgp vpnv4 vrf CUST-A neighbors 10.2.2.2 | section Policy
  Local Policy Denied Prefixes:    --------    -------
    Bestpath from this peer:              4        n/a

Cause. The customer reuses AS 65100 at both sites. A route from Site A arrives at Site B with 65100 already in its AS-path, and Site B rejects it as a loop.

Fix. as-override on the PE (rewrites the customer AS to the provider AS) or allowas-in on the CE (tolerates the own-AS in the path). Full detail, including the SoO you may need on top for dual-homed sites, in BGP as the PE-CE protocol.

Ticket 3: "The DMVPN tunnel is up but spokes can't reach each other"

Symptom. show dmvpn shows all peers up, the hub is reachable, but spoke-to-spoke traffic fails.

Diagnosis. Check what the spoke thinks the next hop to the other spoke is:

SPOKE1#show ip route 12.12.12.12
  * 10.0.0.12, from 10.0.0.2, via Tunnel0

If the next hop is the other spoke (10.0.0.12) - good, that is Phase 3 working. If the next hop is the hub (10.0.0.1), the hub is not preserving the spoke as the next hop.

Cause and fix. Missing no ip next-hop-self eigrp on the hub's tunnel. Without it, the hub advertises other spokes' routes with itself as the next hop, so all spoke-to-spoke traffic hairpins through the hub (and if the hub blocks it, fails entirely). Also confirm no ip split-horizon eigrp on the hub, or spokes never learn each other's routes at all. Both live on the hub tunnel - see EIGRP over DMVPN.

Ticket 4: "The DMVPN failed over its routing but traffic still black-holes"

Symptom. A dual-hub DMVPN. HUB1 fails. EIGRP reconverges via HUB2 - the routing table shows a valid path. And spoke-to-spoke traffic is dead anyway.

SPOKE1#ping 10.2.2.1 source Loopback0 repeat 20
....................
Success rate is 0 percent (0/20)

Diagnosis. The routing looks fine, so look at NHRP, not the RIB:

SPOKE1#show dmvpn
     2 100.64.1.1             10.0.0.1    UP 00:03:18     S
                             10.0.0.12    UP 00:01:24    I2
     1 100.64.2.1             10.0.0.2    UP 00:03:19     S

The I2 state on the spoke-to-spoke entry (10.0.0.12) is the tell. It is a stale, incomplete shortcut - the direct tunnel to SPOKE2 was resolved through HUB1, and HUB1 is now dead, so the mapping is broken.

Cause. The NHRP holdtime was left at its default of 7200 seconds. The stale shortcut would not age out for two hours.

Fix, immediate:

SPOKE1#clear ip nhrp
SPOKE1#ping 10.2.2.1 source Loopback0 repeat 5
!!!!!
Success rate is 100 percent (5/5)

Fix, permanent: ip nhrp holdtime 300 on every tunnel, so stale shortcuts age out in five minutes and re-resolve via the surviving hub automatically. This is the single most common reason a "redundant" dual-hub DMVPN does not actually fail over. Full walkthrough in dual-hub DMVPN design.

Ticket 5: "The IPsec tunnel won't come up after a config change"

Symptom. After changing the crypto config on a DMVPN tunnel, the tunnel is down and will not recover.

Diagnosis. Check whether the tunnel was administratively shut by the change itself. When you change the IPsec protection profile on IOS XE, it tells you exactly what it did:

interface Tunnel0
 tunnel protection ipsec profile IPSEC-IKEV2
% Shutting down Tunnel0 interface due to IPsec tunnel protection modification.
% Please run "no shutdown" after config change to bring up the interface.

Cause. Changing the tunnel protection profile forces IOS XE to bounce the tunnel, and it stays shut until you bring it back up. If you missed the message, the tunnel is simply administratively down.

Fix.

interface Tunnel0
 shutdown
 no shutdown

If the tunnel still will not come up after the bounce, the crypto itself is mismatched. Check the IKE version and parameters on both ends: show crypto ikev2 sa (state should be READY) or show crypto isakmp sa (state QM_IDLE). A spoke on IKEv2 and a hub on IKEv1 will never negotiate. Detail in DMVPN with IKEv1 vs IKEv2.

The commands, collected

! MPLS L3VPN - bottom-up
show ip route <remote-PE-loopback>         (IGP)
show mpls ldp neighbor                       (LDP)
show mpls forwarding-table                   (label stack)
show bgp vpnv4 unicast all summary           (MP-BGP)
show bgp vpnv4 unicast all labels            (VPN labels)
show ip route vrf CUST-A                     (VRF import)
show ip bgp vpnv4 vrf CUST-A neighbors x | section Policy   (drops)

! DMVPN - bottom-up
ping <hub-NBMA>                              (underlay)
show dmvpn                                    (NHRP state - watch for I2)
show crypto ikev2 sa   /  show crypto isakmp sa   (IPsec)
show ip nhrp                                  (mappings)
show ip route <remote-spoke>                 (routing + next hop)
show ip eigrp neighbors                       (routing adjacency)

Key takeaways

• Transport faults are layered. Isolate the layer before you fix anything - IGP, LDP, MP-BGP, VRF for MPLS; underlay, NHRP, IPsec, routing for DMVPN.
• VPN prefix present but not forwarding = a broken transport LSP. The two-label stack needs both labels; check LDP.
• Same-AS customer sites failing = BGP AS-path loop drop. Fix with as-override (PE) or allowas-in (CE).
• Spoke-to-spoke failing with all peers up = missing no ip next-hop-self (or split-horizon) on the hub tunnel.
• Dual-hub DMVPN that reconverges routing but still black-holes = a stale NHRP shortcut (I2) from the default 7200s holdtime. Set ip nhrp holdtime 300.
• IPsec tunnel down after a config change = IOS XE bounced it on the protection-profile change. shutdown / no shutdown, then check for an IKE version mismatch.

That closes the expert transport series, and with it the CCIE transport domain. The full cluster indexes live on the MPLS pillar and the DMVPN pillar.