Dual-Hub DMVPN: Redundancy Designs That Actually Fail Over

Everyone builds a second DMVPN hub for redundancy. Far fewer people test that the failover actually works - and the ones who do are often unpleasantly surprised. A dual-hub DMVPN can have both hubs up, both spokes registered, routing fully converged, and still black-hole spoke-to-spoke traffic for two hours after a hub fails, because of a single default timer nobody changed.

This article builds a dual-hub DMVPN in a CML lab, fails a hub, and shows exactly what breaks and why - including a genuine, non-obvious failure that we hit and fixed. For the fundamentals, start at the complete DMVPN guide.

Single cloud vs dual cloud

Single cloud is the simpler default and is what you want when both hubs are on the same transport. Dual cloud is for when you need to reason about, police, or fail between two independent transports.

The lab, up and running

Two hubs (HUB1 primary, HUB2 backup), two spokes, one INET underlay router, EIGRP over the mGRE cloud, and IPsec protecting every tunnel. All three NHRP peers register with HUB1:

HUB1#show dmvpn
Type:Hub, NHRP Peers:3,
     1 100.64.2.1             10.0.0.2    UP 00:00:31     D
     1 100.64.11.1           10.0.0.11    UP 00:00:27     D
     1 100.64.12.1           10.0.0.12    UP 00:00:22     D

Spoke-to-spoke works: SPOKE1 learns SPOKE2's LAN via both hubs, and the next hop is SPOKE2 directly - not a hub - thanks to no ip next-hop-self on the hub tunnels:

SPOKE1#show ip route 12.12.12.12
    10.0.0.12, from 10.0.0.1, via Tunnel0
  * 10.0.0.12, from 10.0.0.2, via Tunnel0

SPOKE1#show ip cef 12.12.12.12
12.12.12.12/32
  nexthop 10.0.0.12 Tunnel0
  nexthop 10.0.0.12 Tunnel0

Two equal-cost paths, both pointing at SPOKE2 as the next hop. This is a healthy Phase 3 DMVPN. (For the EIGRP-over-mGRE details - no split-horizon, no next-hop-self - see EIGRP over DMVPN.)

The failure that surprised us

We shut HUB1's internet link. EIGRP did its job: it dropped the HUB1 neighbour, and the route to SPOKE2 survived via HUB2. Routing reconverged. Everything looked fine in the routing table.

And spoke-to-spoke traffic died completely:

SPOKE1#ping 10.2.2.1 source Loopback0 repeat 20
....................
Success rate is 0 percent (0/20)

Zero. Not a brief reconvergence blip - total, sustained black-holing, with the routing table showing a perfectly good path via HUB2. The kind of thing that has you staring at a correct-looking show ip route wondering why the pings still fail.

The cause

The direct spoke-to-spoke shortcut to SPOKE2 had been resolved through HUB1. When HUB1 died, that NHRP mapping went stale - it still pointed at HUB1 as the resolution path:

SPOKE1#show dmvpn
     2 100.64.1.1             10.0.0.1    UP 00:03:18     S
                             10.0.0.12    UP 00:01:24    I2
     1 100.64.2.1             10.0.0.2    UP 00:03:19     S

See that I2 state on the SPOKE2 entry - a stale, incomplete shortcut still associated with HUB1's now-dead NBMA address. The routing table said "send to SPOKE2 directly", but the NHRP-to-NBMA mapping needed to build that direct tunnel was broken and pointing at a dead hub.

And here is the killer: it would not fix itself for two hours. The NHRP holdtime had been left at the default. Until that stale entry aged out, SPOKE1 kept trying to reach SPOKE2 through a resolution that no longer worked.

SPOKE1#clear ip nhrp
SPOKE1#ping 10.2.2.1 source Loopback0 repeat 5
!!!!!
Success rate is 100 percent (5/5)

A manual clear ip nhrp forced immediate re-resolution via HUB2 and connectivity came straight back. But you cannot rely on manually clearing NHRP on every spoke every time a hub fails.

The fix: a short NHRP holdtime

The default NHRP holdtime is 7200 seconds - two hours. That is far too long for a dual-hub design, because it is exactly how long a stale spoke-to-spoke shortcut resolved via a failed hub can persist. Set it short on every tunnel:

interface Tunnel0
 ip nhrp holdtime 300

With a 300-second holdtime, a shortcut resolved through a hub that dies ages out within five minutes and re-resolves through the surviving hub automatically. Five minutes is still not instant - during that window, spoke-to-spoke traffic for an already-resolved pair may black-hole - but it is recoverable without human intervention, and for the common case (spoke-to-hub-to-spoke fallback, and newly-resolved shortcuts) failover is much faster.

This is the single most important and most overlooked knob in a dual-hub DMVPN. The redundancy you built is only real if the NHRP holdtime lets stale shortcuts clear in a reasonable time. Every dual-hub design should set it explicitly; the default will bite you exactly when you least want it to - during a hub failure.

Making one hub primary

With both hubs advertising equal metrics, spokes load-share, which is usually not the intent. Make HUB1 primary and HUB2 a true backup with a delay adjustment or an offset list on the backup hub - one command, one router, applies to every spoke:

! On HUB2, make its advertised routes less attractive
router eigrp 100
 ! (named mode) offset-list, or on the interface:
interface Tunnel0
 delay 10000

Now spokes prefer HUB1 and keep HUB2 as a feasible successor - ready for instant EIGRP failover for the routing, while the NHRP holdtime governs how fast stale shortcuts recover. Both mechanisms matter, and they are independent. (Offset-list detail in EIGRP offset lists.)

The two independent failover mechanisms

This is the mental model that makes dual-hub DMVPN make sense. There are two things that have to fail over, and they are governed by different timers:

A DMVPN that fails over its routing in three seconds but keeps a stale NHRP shortcut for two hours has effectively not failed over at all for that spoke pair. Test both. Tune both.

Tune the EIGRP and IPsec timers too

• EIGRP hello/hold on the tunnel: the default on a multipoint interface is 60/180 - three minutes to notice a dead hub. Tune to 5/15, or pair with BFD, so the routing side fails over fast.
• IPsec DPD (Dead Peer Detection): so a spoke notices a dead hub's crypto session promptly and tears down the SA, rather than holding it until the SA lifetime expires.
• NHRP registration timer: spokes re-register with the NHS at one-third of the holdtime by default, so a shorter holdtime also means more frequent registration - a reasonable trade for faster recovery.

Key takeaways

• A dual-hub DMVPN has two failover mechanisms: routing (EIGRP timers, fast) and NHRP resolution (holdtime, slow by default). Both must work.
• We reproduced the classic failure: HUB1 dies, routing reconverges via HUB2, but a spoke-to-spoke shortcut resolved through HUB1 goes stale (I2) and black-holes traffic - with a correct-looking routing table.
• The cause is the default 7200-second NHRP holdtime. Set ip nhrp holdtime 300 on every tunnel so stale shortcuts age out and re-resolve via the surviving hub automatically.
• Make one hub primary with a delay or offset-list adjustment on the backup hub - one command, all spokes.
• Tune the EIGRP tunnel hello/hold (default 60/180 is far too slow) and IPsec DPD as well.
• Test the failover, not just the redundancy. Both hubs up is not the same as failover working.

Next: DMVPN with IKEv1 vs IKEv2, using the same lab. The full cluster index lives on the DMVPN pillar.