GRE Tunnel Troubleshooting: Recursive Routing and Five More Failures

This is the field guide for diagnosing GRE tunnel failures on Cisco IOS XE. Every failure mode is paired with its symptoms, the show commands that confirm it, and the fix, roughly most common first. If you are 30 minutes into an incident, work down the list and stop when something matches.

The recursive-routing section is worth reading even if nothing is broken, because it is the failure engineers most often misdiagnose. It is built around a real capture: a two-router CML lab on IOS XE 17.18.2 where the tunnel is made to eat its own transport, with the log sequence, the routing table before and after, and three fixes.

This is part of the PingLabz GRE Tunnels: The Complete Guide. For protocol theory and configuration, start there.

First Checks: The Five-Minute Triage

Before deep debugging, run these five commands. They tell you which sub-system is broken.

R1# show interface Tunnel0 | include Tunnel|line proto|MTU|Keepalive|source|dest
Tunnel0 is up, line protocol is up
  MTU 17916 bytes, ...
  Keepalive set (10 sec), retries 3
  Tunnel source 198.51.100.1, destination 203.0.113.1
  Tunnel transport MTU 1476 bytes

R1# show ip route 203.0.113.1
Routing entry for 203.0.113.1/32
  Known via "static", distance 1, metric 0
  Routing Descriptor Blocks:
  * 198.51.100.2
      Route metric is 0, traffic share count is 1

R1# ping 203.0.113.1 source 198.51.100.1
!!!!!
Success rate is 100 percent (5/5)

R1# ping 10.0.0.2 source 10.0.0.1
!!!!!
Success rate is 100 percent (5/5)

R1# ping 192.168.2.1 source 192.168.1.1 size 1400 df-bit
!!!!!
Success rate is 100 percent (5/5)

If all five pass, the tunnel works at the IP level. Failures point at the broken layer:

Recursive Routing

Symptom: Tunnel flaps every 30 seconds to a few minutes. Log shows %TUN-5-RECURDOWN: Tunnel0 temporarily disabled due to recursive routing.

Cause: The route to the tunnel destination IP is itself learned through the tunnel. The router cannot encapsulate packets to the tunnel destination if the only way it knows to reach the destination is via the tunnel.

What the router is actually complaining about

A tunnel interface is not a real port. To forward out of Tunnel0 the router builds the outer IP header, looks up the tunnel destination, and stacks the tunnel's adjacency on whatever real interface that lookup returns. When the best route to the destination resolves out of the tunnel, the adjacency stacks on itself and the tunnel carries its own transport. CEF catches the loop while building the chain, which is why the tell-tale line lands just before the tunnel drops.

It flaps rather than fails because shutting the tunnel fixes it. Tunnel down, neighbor lost, bad route withdrawn, underlay route reappears, tunnel up, neighbor re-forms, bad route re-learned. A very reliable oscillator.

Administrative distance is the trigger

Two paths to the tunnel destination exist, and the router is not choosing between sane and insane. It chooses by the preference value each routing source carries, and nothing in that comparison knows one path runs through the tunnel it is about to break.

In the lab below the underlay is OSPF (AD 110) and EIGRP runs over the tunnel advertising the endpoint loopbacks (AD 90). Internal EIGRP beats OSPF, so the tunnel-learned route wins the instant the neighbor comes up and the tunnel dies. Swap the protocols and the same topology never recurses, which is why one team swears the design is fine while another watches a tunnel flap every 40 seconds.

The real lesson: recursion is not caused by the tunnel. It is caused by advertising the tunnel destination prefix into the routing protocol running over the tunnel. If that prefix never travels inside the tunnel, the RIB has one candidate and nothing to lose to.

The repro, captured

Two iol-xe routers on IOS XE 17.18.2. Physical link 10.0.12.0/30, Tunnel0 on 172.16.0.0/30 sourced from and destined to the peer loopback, OSPF on the physical link so the tunnel comes up, then EIGRP 100 over Tunnel0 advertising those loopbacks.

R1# show ip interface brief | include Tunnel0
Tunnel0                172.16.0.1      YES TFTP   up                    up
R1# show ip route 2.2.2.2
Routing entry for 2.2.2.2/32
  Known via "ospf 1", distance 110, metric 11
  * 10.0.12.2, from 2.2.2.2, via Ethernet0/0

via Ethernet0/0 is what a healthy tunnel destination looks like: a real interface, not Tunnel0. Then EIGRP forms over the tunnel and re-learns 2.2.2.2 at AD 90.

*DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.16.0.2 (Tunnel0) is up: new adjacency
*ADJ-5-PARENT: Midchain parent maintenance for IP midchain out of Tunnel0 - looped chain attempting to stack
*TUN-5-RECURDOWN: Tunnel0 temporarily disabled due to recursive routing
*LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
*DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 172.16.0.2 (Tunnel0) is down: interface down

Those five lines are one causal chain. The neighbor comes up, the bad route installs, and %ADJ-5-PARENT ... looped chain attempting to stack is CEF failing to build the adjacency. That line always lands immediately before RECURDOWN. The tunnel then drops and takes the neighbor with it, setting up the next cycle. The best diagnostic is not a debug, it is show ip route against the tunnel destination.

Fix 1: pin the destination with a static route

A static route carries AD 1, which beats anything a routing protocol can offer, so the underlay path wins permanently:

R1(config)# ip route 2.2.2.2 255.255.255.255 10.0.12.2
R1# show ip route 2.2.2.2
Routing entry for 2.2.2.2/32
  Known via "static", distance 1, metric 0
  * 10.0.12.2
R1# show ip interface brief | include Tunnel0
Tunnel0                172.16.0.1      YES TFTP   up                    up

Known via "static", distance 1 is the proof, and the tunnel stayed up and stable from there. In this article's addressing that is ip route 203.0.113.1 255.255.255.255 198.51.100.2.

Configure it on both ends, because recursion is per-router and fixing R1 alone leaves R2 flapping. Make it a /32 so it wins on prefix length as well as distance. And a static next hop does not follow underlay reconvergence, so on a multi-path underlay point it at something stable.

Fix 2: keep the transport out of the overlay protocol

The static route treats the symptom. Not advertising the tunnel endpoints into the tunnel's own routing protocol removes the cause, and scales better because there is no per-destination line to forget.

Cheapest version: leave the endpoint loopbacks out of the overlay protocol's network statements. Where they arrive by redistribution or a summary you do not control, filter them outbound on the tunnel instead, for example an EIGRP distribute-list prefix NO-TRANSPORT out Tunnel0 denying the endpoint /32s. OSPF is harder inside an area, since you cannot filter LSAs between routers in one area, so keep the transport prefixes in another area and filter at the ABR.

Fix 3: put the transport in its own VRF

The structural fix makes recursion impossible rather than unlikely. Put the underlay interface and the tunnel source in their own VRF (the front-door VRF, or fVRF) and resolve the destination there with tunnel vrf UNDERLAY, while the tunnel interface stays in the global table. The lookup now happens in a table the overlay protocol cannot reach into, so no amount of careless redistribution over the tunnel can produce a recursive route. The DMVPN cluster guide covers the pattern at scale, where a hub NBMA address leaking into the overlay EIGRP breaks every spoke at once.

Whichever fix you pick, add keepalive 5 3 while you are in there. It will not prevent recursion, but it catches a tunnel that is up for the wrong reasons, which matters because a GRE tunnel reports up/up while the far end is dead.

Firewall Blocking IP Protocol 47

The classic "works in lab, fails in production" failure.

Symptom: Tunnel0 shows up / up. Underlay ping between tunnel sources works. Overlay ping fails with no response.

Cause: A stateful firewall in the underlay drops IP protocol 47 because it is not TCP, UDP or ICMP. It sees GRE as a non-standard protocol and drops it silently.

Diagnosis:

R1# show interface Tunnel0 | include packets
     Input  packets : 0
     Output packets : 5234

Output packets growing with input packets at zero is the unmistakable sign that traffic is leaving but nothing is coming back. Confirm on the underlay: SPAN both endpoints' WAN interfaces and look for IP protocol 47 in each direction.

Fix: Allow IP protocol 47 between the two tunnel-source IPs on every firewall in the path.

! On a Cisco ASA / FTD
access-list OUTSIDE-IN extended permit gre host 203.0.113.1 host 198.51.100.1
access-list INSIDE-OUT extended permit gre host 198.51.100.1 host 203.0.113.1

MTU and Fragmentation Problems

Symptom: Small packets and pings work. Some applications are fine, others stall, time out or load partially, and HTTPS to specific sites hangs.

Cause: Inner packets are too large after GRE and IPsec encapsulation. Without MSS clamping, TCP endpoints negotiate a segment size on a 1500-byte assumption, and full-size segments are either dropped (DF=1 plus filtered ICMP is a PMTUD black hole) or fragmented.

Diagnosis:

R1# ping 10.0.0.2 size 1400 df-bit
!!!!!
R1# ping 10.0.0.2 size 1500 df-bit
M.M.M
Success rate is 0 percent (0/5)

"M" is "could not fragment", exactly what a real DF=1 packet hits.

Fix: Two lines, on both ends:

interface Tunnel0
 ip mtu 1400
 ip tcp adjust-mss 1360

The math, the IPv6 considerations, and how to walk the size up to find your real path MTU are all in why a GRE tunnel drops large packets but passes pings.

Keepalive Flap with IPsec Rekey

Symptom: Tunnel up most of the time, down briefly about once an hour, with logs showing Tunnel0 line protocol changed state to down then up.

Cause: GRE keepalives time out during the IPsec SA rekey window. The default IKEv2 lifetime is 3,600 seconds, which matches the symptom timing.

Diagnosis:

R1# show crypto ikev2 sa
Tunnel-id Local      Remote     fvrf/ivrf  Status
1         198.51.100.1/500  203.0.113.1/500  none/none  READY
      Life/Active Time: 3600/3540 sec   <- about to rekey

If "Active Time" is close to "Life" and the tunnel just flapped, the events are correlated.

Fix: Loosen the keepalive retry count or interval so it rides through a brief rekey window:

interface Tunnel0
 keepalive 10 5     ! 50-second timeout instead of 30

Or stagger the IPsec lifetimes so the two ends do not rekey at the same second, or move to BFD. If you are still weighing whether to wrap GRE in IPsec at all, the trade-off between a plain GRE tunnel and an encrypted one is worth settling before tuning timers around it.

OSPF Neighbor Stuck in INIT

Symptom: Tunnel up, overlay ping works, but the OSPF neighbor reaches INIT or 2-WAY and never FULL.

Cause: OSPF hellos are one-way. Either an ACL blocks 224.0.0.5 inbound on one end, or the authentication, area numbers or network types do not match.

Diagnosis:

R1# debug ip ospf hello
*Apr 30 14:35:12: OSPF: Send hello to 224.0.0.5 area 0 on Tunnel0 from 10.0.0.1
*Apr 30 14:35:22: OSPF: Send hello to 224.0.0.5 area 0 on Tunnel0 from 10.0.0.1
! No "Rcv hello" entries

If R1 only sends and never receives, R2's hellos are being dropped on the way. Check on R2:

R2# debug ip ospf hello
*Apr 30 14:35:14: OSPF: Send hello to 224.0.0.5 area 0 on Tunnel0 from 10.0.0.2
! On R2 you see hellos sent in both directions but R2 also sees no "Rcv hello"

Both ends sending and neither receiving means something in between is dropping the multicast. A standard "permit gre" rule passes GRE-encapsulated multicast, but custom ACLs interfere.

Fix: Verify both ends match on area, hello/dead timers, network type and authentication. Run show ip ospf interface Tunnel0 on each side and compare.

Tunnel0 Line Protocol Down

Symptom: Tunnel0 reports down/down or up/down. Configuration looks correct.

Causes (in order of likelihood):

1. The tunnel source IP does not exist on the local router. Either the configured source-interface is down, or the source IP was misspelled.
2. There is no route in the IP routing table to the tunnel destination.
3. The tunnel destination cannot be the same as the tunnel source.

If the tunnel never worked rather than stopped working, compare it against a known-good GRE tunnel build on IOS XE before debugging further.

Diagnosis:

R1# show interface Tunnel0 | include source|destination|line
Tunnel0 is up, line protocol is down
  Tunnel source 198.51.100.1, destination 203.0.113.1

R1# show ip route 203.0.113.1
% Network not in table

"Network not in table" is the smoking gun.

Fix: Add a route to the tunnel destination via whatever next hop the underlay requires.

R1(config)# ip route 203.0.113.1 255.255.255.255 198.51.100.2

Useful Debugs

Each debug is paired with what it shows. Run them on a lab tunnel first: on production they produce a lot of output.

For debug ip packet detail, restrict the scope tightly:

R1(config)# ip access-list extended DBG
R1(config-ext-nacl)# permit ip host 198.51.100.1 host 203.0.113.1
R1(config-ext-nacl)# permit ip host 203.0.113.1 host 198.51.100.1
R1# debug ip packet detail 100
! 100 references the access-list-extended ACL number; for named ACLs use 'list DBG'

Always undebug all when you are done. A debug left running in production has consumed many a router CPU.

Packet Capture Strategy

For problems that resist show-and-debug, capture the packets. Embedded Packet Capture on IOS XE writes to a buffer or a file:

R1(config)# ip access-list extended GRE-CAPTURE
R1(config-ext-nacl)# permit gre host 198.51.100.1 host 203.0.113.1
R1(config-ext-nacl)# permit gre host 203.0.113.1 host 198.51.100.1

R1# monitor capture CAP interface GigabitEthernet1 both
R1# monitor capture CAP access-list GRE-CAPTURE
R1# monitor capture CAP buffer size 5
R1# monitor capture CAP start
! ... wait for the issue to reproduce ...
R1# monitor capture CAP stop
R1# show monitor capture CAP buffer brief
R1# monitor capture CAP export bootflash:gre.pcap

Open the pcap in Wireshark. GRE shows as IP protocol 47, the header carries the encapsulated protocol type, and the inner packet decodes automatically. It is the fastest way to confirm the encapsulation is what you expected, that keepalives are being reflected, and that IPsec is wrapping the GRE correctly.

When to Escalate

If the tunnel works in lab but fails over a carrier path, suspect a middlebox you cannot reach:

• ISP CGNAT that mishandles IP protocol 47.
• Carrier MPLS L3VPN with ACLs that drop GRE.
• Customer-edge DPI firewall that rewrites GRE keepalive payloads.
• 5G links with a 1380-byte underlay MTU (drop ip mtu to 1300 to test).

Open a ticket with captures from both ends and timestamps that line up. Most carrier "GRE does not work" issues turn out to be a default deny on a transit firewall, fixed quickly once they see them.

Summary

GRE is robust enough that production failures fall into a few recognizable patterns: recursive routing, IP protocol 47 blocked, MTU mismatch, keepalive flap during IPsec rekey, and routing-protocol issues that look like GRE issues. Run the triage first, match the symptom, apply the fix.

• %TUN-5-RECURDOWN means the best route to the tunnel destination resolves out of the tunnel. Confirm with show ip route <destination>, not a debug.
• %ADJ-5-PARENT ... looped chain attempting to stack lands immediately before it. That pairing is the fingerprint.
• The cause is administrative distance: the overlay protocol's route (90 or 110) beats the underlay's. Never advertise the tunnel endpoints into the protocol running over the tunnel.
• Three fixes, increasingly permanent: a static /32 on both ends, filtering the transport out of the overlay, or a front-door VRF.
• The other failure modes: IP protocol 47 dropped, MTU, keepalive timing against IPsec rekey, one-way OSPF hellos, and a missing route to the destination.

If you bookmark one thing, bookmark the five-minute triage at the top. It separates "the tunnel is broken" from "the routing on top of the tunnel is broken," and that distinction shapes the rest of the session. The full GRE coverage is at the PingLabz GRE pillar.