IPsec

FlexVPN Spoke-to-Spoke: Dynamic Tunnels Without DMVPN

The design, the config, and exactly how far we got on a real cat8000v. Hub-and-spoke worked. The NHRP shortcut never fired. Here is how you tell the difference.
FlexVPN spoke-to-spoke: real 2-hop traceroute through the hub and zero NHRP Traffic Indications
In: IPsec, VPN, FlexVPN, CCIE, Labs

Every hub-and-spoke IPsec VPN has the same structural flaw: two branch offices that want to talk to each other cannot. Not directly, anyway. Their traffic climbs the tunnel to the hub, gets decrypted, gets routed, gets re-encrypted, and comes back down a different tunnel. It works, but it burns CPU on a box that has better things to do and it adds a leg to the path that has nothing to do with where the packets are actually going.

FlexVPN's answer is NHRP redirect and shortcut switching: the hub tells the spokes "you two can reach each other directly", and the spokes build a dynamic IPsec tunnel between themselves. That is the design. This article walks through it properly, shows the exact configuration, and then tells you what happened when we built it on a Cisco cat8000v running IOS XE 17.18.02.

Here is the part most articles will not tell you: the shortcut never formed. Hub-and-spoke worked perfectly. IKEv2 route injection worked beautifully. Spoke-to-spoke traffic passed at 100 percent. But every packet hairpinned through the hub, and the NHRP counters on every device stayed at zero. We are not going to pretend otherwise, because if you are reading this at 2am with a traceroute that still says two hops, a sanitised success story is useless to you. What you need is a way to tell the difference between a shortcut that is working and a shortcut that never fired. That is section four, and it is the reason this article exists.

The Problem: The Hub Is In The Middle Of Everything

Our topology is a standard FlexVPN hub-and-spoke. A hub on 203.0.113.1, two spokes on 198.51.100.2 and 198.51.100.6, and an ISP router in between. The hub's LAN is 10.20.10.0/24, SPOKE1's is 10.30.10.0/24, SPOKE2's is 10.30.20.0/24. Both spokes are up, both got a tunnel address from the hub's pool, and the IKEv2 authorization policy pushed every route into place without a single line of OSPF, EIGRP, or BGP.

Now watch what happens when SPOKE1 pings SPOKE2's LAN:

SPOKE1#ping 10.30.20.1 source Loopback10 repeat 20
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (20/20), round-trip min/avg/max = 4/7/15 ms

Twenty out of twenty. It works. So what is the complaint? Trace the path:

SPOKE1#traceroute 10.30.20.1 source Loopback10 probe 1 timeout 1
Type escape sequence to abort.
Tracing the route to 10.30.20.1
  1 10.255.0.1 34 msec        <-- the HUB
  2 10.0.9.25 81 msec         <-- SPOKE2's tunnel address

Two hops. Hop one is 10.255.0.1, the hub's tunnel address. Hop two is 10.0.9.25, which is the address the hub's local pool handed to SPOKE2. Every single packet between those two branches is being decrypted by the hub, looked up in the routing table, and re-encrypted into the other spoke's tunnel.

That costs you three things:

  • Latency. The packet takes a detour through a third site. If your hub is in a datacentre in another region and your two spokes are in the same city, that detour is the entire round-trip time.
  • Hub bandwidth and CPU. Every spoke-to-spoke byte crosses the hub's WAN link twice (in and out) and gets processed by the crypto engine twice (decrypt and re-encrypt). Scale that across fifty branches and the hub is doing crypto for traffic it has no interest in.
  • Blast radius. The hub becomes a single point of failure for traffic that never needed to touch it.

None of that is a bug. It is exactly what a hub-and-spoke topology does. The question is whether you can teach the spokes to skip the middleman.

The Design That Is Supposed To Fix It

NHRP (Next Hop Resolution Protocol) is the mechanism. It is the same machinery that makes DMVPN Phase 3 work, and FlexVPN reuses it. Two knobs do the work, and they live on opposite ends of the tunnel.

ip nhrp redirect (on the hub)

When the hub receives a packet on an NHRP-enabled interface and forwards it back out an NHRP-enabled interface (which is precisely what a hairpin is), it recognises the pattern. Rather than silently continuing to forward, it sends an NHRP Traffic Indication back to the source. The message is essentially "I am forwarding this for you, but you could be reaching that destination directly. Go look it up."

This is the trigger for the whole process. If the redirect never fires, nothing downstream of it can happen.

ip nhrp shortcut virtual-template 1 (on the spoke)

The spoke receives the Traffic Indication and sends an NHRP Resolution Request for the destination prefix. That request reaches the far spoke, which answers with a Resolution Reply containing its NBMA address - the public, underlay IP it is reachable on. Now the originating spoke knows how to get to its peer directly, so it clones its own virtual-template into a fresh Virtual-Access interface, brings up a new IKEv2 and IPsec SA straight to that NBMA address, and installs a shortcut route in the NHRP cache.

The next packet takes the direct tunnel. The traceroute drops to one hop. The hub is out of the data path entirely, and it stays out until the cache entry ages out.

That is the theory, and it is a genuinely good design. Here is the configuration that implements it, taken verbatim from our lab.

The hub's virtual-template

interface Virtual-Template1 type tunnel
 ip unnumbered Loopback0
 ip nhrp network-id 1
 ip nhrp redirect
 tunnel source GigabitEthernet2
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile PLZ-FLEX-IPSEC

Note the tunnel source and tunnel mode lines. They are not optional. Leave them out and the cloned Virtual-Access comes up and immediately dies in a loop, throwing %DMVPN-5-NHRP_NETID_UNCONFIGURED at you while IKEv2 authenticates perfectly, which makes it look like a crypto problem when it is not. We cover that trap in the FlexVPN hub-and-spoke configuration walkthrough.

The spoke's tunnel

interface Tunnel0
 ip address negotiated
 ip nhrp network-id 1
 ip nhrp shortcut virtual-template 1
 tunnel source GigabitEthernet2
 tunnel mode ipsec ipv4
 tunnel destination 203.0.113.1
 tunnel protection ipsec profile PLZ-FLEX-IPSEC

Three things matter here. ip address negotiated means the tunnel IP comes from the hub's pool, so you never touch spoke addressing. ip nhrp network-id 1 must match the hub (it is a local significance value, but both ends need to consider themselves on the same NHRP network). And ip nhrp shortcut virtual-template 1 is the line that says "if you get a redirect, resolve it and build a tunnel by cloning template 1".

What Actually Happened

The configuration was correct. We can prove that, because on IOS XE the interesting question is not what is in the virtual-template, it is what got inherited by the dynamically-created Virtual-Access when the spoke connected. So look at the derived config, not the running config:

HUB#show derived-config interface Virtual-Access1
interface Virtual-Access1
 ip unnumbered Loopback0
 ip nhrp network-id 1
 no ip nhrp shortcut
 ip nhrp redirect                       <-- redirect IS there
 tunnel source GigabitEthernet2
 tunnel mode ipsec ipv4
 tunnel destination 198.51.100.2
 tunnel protection ipsec profile PLZ-FLEX-IPSEC

ip nhrp redirect is present on the live, cloned interface that is actually carrying the spoke's traffic. This is not a case of the template failing to apply. The feature is enabled exactly where it needs to be.

And yet NHRP never sent a single packet.

HUB#show ip nhrp traffic
Virtual-Access1: Max-send limit:10000Pkts/10Sec, Usage:0%
   Sent: Total 0
         0 Resolution Request  0 Resolution Reply  0 Registration Request
         0 Error Indication  0 Traffic Indication  0 Redirect Suppress
   Rcvd: Total 0
         0 Resolution Request  0 Resolution Reply  0 Registration Request

Zero Traffic Indications sent. The hub was hairpinning spoke-to-spoke traffic all day (we had just pushed twenty ICMP echoes through it and watched them succeed), and it never once generated the redirect that is supposed to start the shortcut process.

Predictably, the spoke side is a desert:

SPOKE1#show ip nhrp
                                          (empty)

SPOKE1#show ip nhrp traffic
Tunnel0: Max-send limit:10000Pkts/10Sec, Usage:0%
   Sent: Total 0
         0 Resolution Request  0 Resolution Reply  0 Registration Request

An empty NHRP cache and zero Resolution Requests sent. That is entirely consistent: the spoke was never told to resolve anything, so it did not.

What we tried

For completeness, here is what we changed and what it did not fix:

  • Added ip nhrp redirect to the spoke tunnels and virtual-templates as well as the hub. No change. (A spoke that builds shortcuts can itself need to redirect in some topologies, so it was worth trying.)
  • Added ip nhrp nhs 10.255.0.1 nbma 203.0.113.1 multicast to the spoke Tunnel0, giving the spoke an explicit next-hop-server statement pointing at the hub. Counters stayed at zero.
  • Sustained data traffic between the spoke LANs, in case the redirect needed more than a handful of packets to trigger. It did not help.
  • Cleared and rebuilt the IKEv2 SAs so the Virtual-Access interfaces were freshly cloned with every knob already in place. Same result.

The conclusion, stated plainly: on cat8000v running IOS XE 17.18.02, in this topology, the NHRP shortcut did not establish. The hub never generated a Traffic Indication, so the spokes were never prompted to resolve, so no direct tunnel was ever built. Spoke-to-spoke connectivity worked, but only via the hairpin.

We are not going to speculate wildly about the root cause. This is platform and version behaviour, and it is exactly the kind of thing you should verify on your own gear and your own image before you design around it. What we can give you is the diagnostic method, which is worth more than a guess.

How To Tell Whether Yours Is Actually Working

This is the reusable part. There are four independent checks, and they fail in a specific order. If the first one is zero, the other three cannot possibly succeed, which is enormously useful because it tells you where to stop looking.

1. Hub: Traffic Indication counter
Command: show ip nhrp traffic
Working: Traffic Indication is NON-ZERO and climbs as spoke-to-spoke traffic flows.
Ours: 0 Traffic Indication. Nothing sent, ever.
2. Spoke: NHRP cache
Command: show ip nhrp
Working: A cache entry for the far spoke's LAN prefix, mapped to that spoke's NBMA (public) address.
Ours: Empty. No entries at all.
3. Spoke: IKEv2 SA count
Command: show crypto ikev2 sa
Working: A SECOND SA, direct to the other spoke's public IP, alongside the hub SA.
Ours: One SA only, to the hub.
4. Spoke: the path itself
Command: traceroute spoke LAN to spoke LAN
Working: Drops from 2 hops to 1 once the shortcut is up (the first probe may still show the hub).
Ours: Stayed at 2 hops. Hub, then spoke.

The single most important line in that grid is the first one. If your hub's Traffic Indication counter is zero, the redirect is not being generated, and no amount of spoke-side configuration will fix it. Engineers lose hours here, because the symptom presents on the spoke (empty cache, no second SA, two-hop traceroute) and the instinct is to start adjusting the spoke. Do not. The spoke is behaving correctly. It is waiting to be told to resolve, and it is never being told.

Start at the hub, confirm the redirect is actually inherited by the Virtual-Access with show derived-config interface Virtual-Access1, and then watch that counter while you push traffic. If it moves, the mechanism is alive and you have a spoke-side or reachability problem. If it does not move, the mechanism is not alive and you are debugging the wrong device.

The Honest Recommendation

Here is where we land after building this for real.

FlexVPN's hub-and-spoke is excellent, and you should use it. The IKEv2 route injection in particular is one of the more elegant things in the Cisco VPN toolkit: the hub hands out a tunnel address from a local pool and pushes the entire prefix list into the spoke's routing table through an IKEv2 authorization policy. No overlay routing protocol. No adjacencies to babysit. If you want the full build, the config, and the gotchas, that is FlexVPN site-to-site configuration, and the conceptual grounding is in FlexVPN explained.

For proven spoke-to-spoke today, DMVPN Phase 3 does this reliably. Same NHRP redirect and shortcut mechanism, same design intent, and we have it working and captured on real gear. If your requirement is "branches must talk directly to each other and I need it in production this quarter", DMVPN Phase 3 is the path with the shorter risk profile. The full cluster, including the phase-by-phase behaviour and the NHRP cache entries you should expect to see, lives at the PingLabz DMVPN guide. It is worth your time.

And validate the FlexVPN shortcut on your own platform before you bet a design on it. Not because the feature is bad, but because "it works on the datasheet" and "it works on the image in my rack" are different claims, and only one of them keeps your migration on schedule. Build the two-spoke lab, push traffic, and watch show ip nhrp traffic on the hub. You will know inside five minutes.

Key Takeaways

  • Hub-and-spoke hairpinning is real and measurable. Our spoke-to-spoke traceroute showed two hops: the hub at 10.255.0.1, then the far spoke at 10.0.9.25. Connectivity was 100 percent, but every packet was decrypted and re-encrypted by a router that had no business being in the path.
  • The fix is a two-sided mechanism. ip nhrp redirect on the hub generates the Traffic Indication; ip nhrp shortcut virtual-template 1 on the spoke turns that into a Resolution Request and a direct dynamic tunnel. Both ends need a matching ip nhrp network-id.
  • Check the derived config, not the template. On FlexVPN the interface that carries traffic is a cloned Virtual-Access. show derived-config interface Virtual-Access1 is how you prove the NHRP knobs actually landed on it.
  • On cat8000v 17.18.02, in our topology, the shortcut never formed. The redirect was correctly inherited, the counters stayed at zero, the NHRP cache stayed empty, and the traceroute stayed at two hops. We tried spoke-side redirect, an explicit NHS statement, sustained traffic, and full SA rebuilds. None of it changed the result.
  • Debug from the hub outward. A zero Traffic Indication counter means the redirect is not firing, and nothing you configure on the spoke can compensate for that. Four checks, in order: hub counter, spoke cache, spoke SA count, traceroute hop count.
  • DMVPN Phase 3 is the proven route to spoke-to-spoke today. Use FlexVPN for its hub-and-spoke and its IKEv2 route injection, which are genuinely first-rate, and validate the shortcut path on your own gear before you commit to it.

None of this diminishes FlexVPN. It is the most modern IPsec VPN framework Cisco ships, and the parts of it that worked in our lab worked cleanly and with less configuration than any equivalent. But a lab is where you find out which parts those are, and the difference between an engineer and a brochure is that the engineer tells you what the box actually did.

Written by
More from Ping Labz
Great! You’ve successfully signed up.
Welcome back! You've successfully signed in.
You've successfully subscribed to Ping Labz.
Your link has expired.
Success! Check your email for magic link to sign-in.
Success! Your billing info has been updated.
Your billing was not updated.