DMVPN failures cluster hard. After the underlay itself, almost every broken cloud comes down to a spoke that cannot register, a tunnel that is up/up while silently eating packets, or shortcuts that never form. So instead of writing another symptom checklist, we broke a healthy Phase 3 lab (IOS XE 17.18, hub and two spokes) three different ways on purpose and captured what each failure actually looks like on the CLI, including the case where the hub logs nothing at all. This is the troubleshooting companion to the DMVPN complete guide; the healthy-state baseline it compares against is the Phase 3 configuration walkthrough.
The Layered Method
DMVPN stacks four layers, and each has its own verification command. Test in this order and you cannot get lost: the underlay (can tunnel sources ping each other?), then NHRP (show dmvpn and show ip nhrp nhs detail), then the routing protocol (show ip eigrp neighbors or OSPF equivalent), then IPsec if configured (show crypto session, covered in the IPsec article). The single most information-dense field in all of it is the State column of show dmvpn: UP means NHRP resolved and working, NHRP means the control plane is failing right now, and IKE/CRYPTO states mean the problem is a layer down in IPsec. A peer in state NHRP with the tunnel interface up/up is the signature of every failure below.
Baseline: What Healthy Looks Like
Burn this into memory, because every diagnosis is a diff against it. Hub with two registered spokes:
HUB# show dmvpn
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 198.51.100.1 10.0.0.2 UP 00:01:35 D
1 192.0.2.1 10.0.0.3 UP 00:01:31 DSpoke with a healthy NHS (state RE, requests and replies balanced):
SPOKE1# show ip nhrp nhs detail
10.0.0.1 RE priority = 0 cluster = 0 req-sent 1 req-failed 0 repl-recv 1 (00:00:23 ago)Failure 1: Wrong NHS Address (Registering Into the Void)
We pointed SPOKE2 at an NHS that does not exist (10.0.0.9 instead of 10.0.0.1), the kind of typo that ships in a copy-pasted branch template. The symptom set:
SPOKE2# show dmvpn
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 203.0.113.1 10.0.0.9 NHRP 00:00:07 S
SPOKE2# show ip nhrp nhs detail
10.0.0.9 E priority = 0 cluster = 0 req-sent 5 req-failed 0 repl-recv 0
Pending Registration Requests:
Registration Request: Reqid 6, Ret 16 NHS 10.0.0.9 expired (Tu0)
SPOKE2# show interface Tunnel0 | include line protocol
Tunnel0 is up, line protocol is upRead the tells: state NHRP not UP, NHS stuck in E (expecting replies), req-sent climbing with repl-recv frozen at zero, a pending registration marked expired, and the interface happily up/up the whole time (GRE interfaces do not go down for control plane failures; see GRE keepalives for why). The registration packets are arriving at the hub, but they are addressed to an NHS identity nobody owns, so nothing answers. Fix the ip nhrp nhs and ip nhrp map pair so both reference the hub's real tunnel IP and its real underlay address.
Failure 2: NHRP Authentication Mismatch (the Hub Tells You)
Next we set SPOKE2's authentication string to WRONGKEY while the hub expects PLZ123. From the spoke, the symptoms are identical to failure 1: state NHRP, requests unanswered, registrations expiring. This is the diagnostic trap of NHRP failures, since three different root causes present the same on the spoke side. The difference is on the hub. With debug nhrp error:
*Jul 11 18:52:12.182: NHRP: Receive Registration Request via Tunnel0 vrf: global(0x0), packet size: 108
*Jul 11 18:52:12.182: %DMVPN-3-DMVPN_NHRP_ERROR: Tunnel0: Recieved wrong authentication string for Registration Request , Reason: authentication failure (11) on (Tunnel: 10.0.0.3 NBMA: 203.0.113.1)The hub receives the registration, checks the cleartext auth string, and rejects it with a logged reason (yes, "Recieved" is misspelled in IOS XE, which at least makes it easy to grep). This is why the second move in any registration failure, after reading the spoke's NHS counters, is to look at the hub: an answering hub that refuses is loud, and its log line names the misbehaving spoke's NBMA address for you. Align ip nhrp authentication across the cloud (8 characters maximum) and registration recovers on the next retry.
Failure 3: Tunnel Key Mismatch (Nobody Tells You)
Finally, the cruel one. We set tunnel key 999 on SPOKE2 while everyone else runs tunnel key 100:
SPOKE2# show interface Tunnel0 | include line protocol|Key
Tunnel0 is up, line protocol is up
Key 0x3E7, sequencing disabled
SPOKE2# ping 10.0.0.1 repeat 3
...
Success rate is 0 percent (0/3)
SPOKE2# show ip nhrp nhs detail
10.0.0.1 E priority = 0 cluster = 0 req-sent 7 req-failed 0 repl-recv 0 (00:01:48 ago)
Pending Registration Requests:
Registration Request: Reqid 11, Ret 64 NHS 10.0.0.1 expired (Tu0)Tunnel up/up, plain overlay pings dead, registrations expiring, and here is the defining feature: the hub logs nothing. Zero. The GRE key lives in the GRE header, so key-mismatched packets are discarded by the GRE decapsulation code before NHRP ever sees them; there is no NHRP error to log because, as far as the hub's NHRP process knows, the spoke never spoke. When a spoke shows failure-1 symptoms but the hub shows no incoming registrations and no errors while other spokes work fine, diff the tunnel stanzas: tunnel key, tunnel source, and mode. The Key 0x3E7 line (999 in hex) in the interface output is your receipt.
The same silent-drop signature comes from a wrong tunnel source, transport ACLs eating GRE (protocol 47), or a middlebox stripping it, which is also where general GRE troubleshooting overlaps this article.
When Registration Works but Shortcuts Do Not
A distinct family: hub state UP everywhere, spoke-to-spoke traffic flows, but traceroutes never leave the hub path. That is not broken, it is Phase 1 behavior in what you thought was a Phase 3 cloud. Check the pair of commands that define Phase 3: ip nhrp redirect present on the hub tunnel, ip nhrp shortcut present on the spoke tunnels. Then confirm the machinery with counters instead of guesswork: show ip nhrp traffic on the spoke should show a received Traffic Indication (the redirect) and sent Resolution Requests, and success lands in the RIB as the %/[NHO] override:
SPOKE1# show ip route next-hop-override
D % 10.0.2.0/24 [90/102400640] via 10.0.0.1, 00:00:57, Tunnel0
[NHO][90/255] via 10.0.0.3, 00:00:09, Tunnel0If redirects are sent but shortcuts never install, the usual culprits are spoke-to-spoke underlay reachability (both spokes must reach each other's public addresses directly, which NAT loves to break) or a missing shortcut on just one side. The full message flow, including what each counter means, is in the NHRP deep dive, and the phase mechanics are in Phase 1 vs 2 vs 3.
Routing Adjacency Failures Wearing a DMVPN Mask
One more disguise worth naming: tunnels UP, NHRP clean, but no EIGRP or OSPF neighbors. Before touching the routing protocol, check ip nhrp map multicast on the spoke (missing means unicast works and multicast hellos vanish) and the hub's multicast handling. Then check the usual suspects: EIGRP split horizon on the hub (spokes neighbor fine but never see each other's routes), OSPF network type mismatches, and MTU mismatch stalling OSPF in EXSTART. The design-side prevention for all of these is in routing over DMVPN.
When IPsec Is in the Stack
Encrypted clouds add one layer to triage, and it slots in cleanly between the underlay and NHRP: if show dmvpn shows IKE-related states or the tunnel line protocol refuses to come up with tunnel protection configured, work the crypto ladder before touching NHRP. show crypto ikev2 sa answers whether negotiation completed (READY) or never started (no SA at all usually means UDP 500 is filtered or the peer address is wrong); show crypto ipsec sa counters answer whether traffic encrypts and decrypts symmetrically (encaps climbing while the peer's decaps stalls means ESP is dying in transit, typically a firewall); and show crypto session ties it together per tunnel. Mismatched proposals fail loudly in IKEv2 with a clear negotiation error, while PSK mismatches fail at IKE_AUTH. The one DMVPN-specific crypto trap: with tunnel protection applied on one side but not the other, GRE from the unprotected side arrives cleartext, gets rejected, and the symptom mimics failure 3's silence. The full crypto build and its verification ladder live in securing DMVPN with IPsec profiles.
Capturing Debugs That Actually Land Somewhere
A practical note from producing this article's captures, because it will bite anyone using automation or a jump host. On IOS XE, debug output goes to the console line by default, and most automation frameworks (PyATS included) disable console logging the moment they connect, so debug nhrp packet appears to produce nothing. The fix is to send debugs to the buffer and read them back:
SPOKE1(config)# logging buffered 200000 debugging
SPOKE1# debug nhrp packet
SPOKE1# ... trigger the event ...
SPOKE1# show logging | include NHRP
SPOKE1# undebug allThe second gotcha: NHRP only sends packets when it has a reason. Clearing the cache with clear ip nhrp does not necessarily force an immediate re-registration, so a debug window can legitimately capture silence. Bouncing the tunnel interface (or waiting out the registration timer) guarantees a registration exchange to observe. Always undebug all when done; debug nhrp packet on a busy hub is a self-inflicted denial of service.
The Compressed Runbook
Underlay pings between tunnel sources. show dmvpn state column on both ends. If state NHRP: show ip nhrp nhs detail on the spoke (req-sent versus repl-recv), then hub logs with debug nhrp error. Hub logging auth failures means strings mismatch; hub silent means GRE never arrives, so diff tunnel key, source, mode, and check transport filtering. If state UP but no shortcuts: verify redirect/shortcut pair, then show ip nhrp traffic for Traffic Indications, then spoke-to-spoke underlay reachability. If tunnels and NHRP are clean but routes are missing: multicast maps, split horizon, network types. Escalate to debug nhrp packet only once you know which conversation you are watching for, and remember debugs land in the buffer, not your SSH session, unless you ask.
Key Takeaways
Troubleshoot DMVPN as four layers and let show dmvpn's state column route you: NHRP state means registration or resolution, and the spoke-side symptoms are identical for wrong NHS, bad authentication, and key mismatch, so the differential diagnosis happens on the hub. An answering hub that logs authentication failures is a config diff away from fixed; a silent hub means GRE itself is dropping, with tunnel key mismatch as the classic cause since the interface stays up/up through all of it. Shortcut problems are almost always a missing redirect/shortcut command or spokes that cannot reach each other's underlay addresses. Keep the healthy baseline from the Phase 3 build handy, and the rest of the cluster in the DMVPN complete guide.
