GETVPN Redundancy: COOP Key Servers and Rekey Survival

The first question anyone sensible asks about GETVPN is: "so the whole thing depends on one router?" It is a fair question. The key server holds the group policy, generates the keys, and pushes them out. Lose it and, on the face of it, you have lost the VPN.

The answer is more interesting than it looks. Half the worry is misplaced (the data plane genuinely does not care if the key server dies, and we shut one down to prove it), and the half that is real is fixed with COOP: cooperative key servers, two or more, with priorities and an election. But COOP has a trap that almost everyone hits, that no show command flags as a failure, and that will silently kill your entire group at the next rekey interval. We walked into it deliberately and captured exactly what the router says.

This is part three of the GETVPN series, following GETVPN Explained (the model and the header-preservation capture) and the GETVPN configuration walkthrough. If GETVPN itself is new to you, start there. Both sit under the IPsec VPN pillar. Everything below is real output from Cisco IOS XE (cat8000v 17.18.02): KS1 at 10.100.0.11, KS2 at 10.100.0.12, group members GM1 and GM2, on a shared private WAN core.

Start Here: The Data Plane Survives a Dead Key Server

Before configuring any redundancy at all, it is worth understanding precisely what you lose when a key server dies, because most people badly overestimate it.

We shut down KS1's core-facing interface. The key server was gone. Then, on a group member, with no key server reachable:

GM1#ping 10.30.10.1 source Loopback10 repeat 5
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 6/9/20 ms

One hundred percent. Encrypted. With the key server dead.

The key server is a control plane element. It is not in the data path. This is the single most misunderstood fact about GETVPN, and it falls directly out of the architecture: group members do not tunnel to the key server, they register with it. Once a member holds the TEK, it encrypts to every other member directly across the WAN. The key server is not carrying a single byte of user traffic.

So what do you actually lose when the key server goes away?

That is a degradation on a timer, not an outage. The group keeps encrypting until the TEK lifetime runs out with no rekey to replace it. Your GETVPN does not fall over the instant the key server reboots, which changes how you should think about the risk: it is a control-plane availability problem, not a data-plane one.

Which is exactly why COOP exists. Not to keep traffic flowing (that already works), but to keep the control plane alive so the group can rekey, re-register, and grow.

Configuring COOP

COOP adds a redundancy sub-block under server local in the GDOI group. Each key server gets a local priority and a list of its COOP peers. Highest priority wins the election and becomes Primary.

On KS1 (priority 100), pointing at KS2:

crypto gdoi group PLZ-GETVPN
 server local
  redundancy
   local priority 100
   peer address ipv4 10.100.0.12

On KS2 (priority 75), pointing at KS1, and otherwise carrying an identical group configuration (same identity number, same policy ACL, same transform set, same rekey parameters). Group members already listed both servers, which is why the member config in the previous article had two server address ipv4 lines:

crypto gdoi group PLZ-GETVPN
 identity number 4321
 server address ipv4 10.100.0.11
 server address ipv4 10.100.0.12

Members try the servers in list order. Registration is not tied to the Primary: a member can register to a Secondary perfectly happily. Only rekeys come from the Primary.

The ordering gotcha: address before redundancy

Type the COOP block on a key server that does not yet have its local GDOI address configured and IOS stops you flat:

KS2(config-gdoi-local-server-red)#local priority 75
% ERROR: Local server address must be configured first

address ipv4 must be configured before redundancy. This makes sense once you see it (COOP peering sources itself from the key server's local GDOI address, so there is nothing to peer from until that address exists), but it catches people because it is the reverse of how the config reads back afterwards, and because a copy-paste of a working config from another box goes in line by line in the order you paste it. Put address ipv4 10.100.0.11 in first, then add the redundancy block.

The Election, in Real Output

With both key servers up, show crypto gdoi ks coop is where you confirm the relationship. On KS1, the higher priority:

KS1#show crypto gdoi ks coop
Crypto Gdoi Group Name :PLZ-GETVPN
    Local Address: 10.100.0.11
    Local Priority: 100
    Local KS Role: Primary   , Local KS Status: Alive
    Local KS version: 1.0.27
    Primary Timers:
        Primary Refresh Policy Time: 20
        Remaining Time: 14
    Peer Sessions:
    Session 1:
        Peer Address: 10.100.0.12
        Peer KS Role: Secondary , Peer KS Status: Dead
        IKE status: Established

And on KS2, the lower priority, which is the more informative view because it sees the whole picture:

KS2#show crypto gdoi ks coop
    Local Address: 10.100.0.12
    Local Priority: 75
    Local KS Role: Secondary , Local KS Status: Alive
    Peer Address: 10.100.0.11
    Peer Version: 1.0.27
    Peer Priority: 100
    Peer KS Role: Primary   , Peer KS Status: Alive
        IKE status: Established

Priority 100 beats priority 75, so KS1 is Primary and KS2 is Secondary. IKE status: Established confirms the two key servers have an IKE SA between them (COOP peering is itself protected, which is why both key servers need ISAKMP keys for each other).

Note the Primary Refresh Policy Time: 20 on KS1. That is the announcement interval. The Primary periodically reasserts itself, and the Secondary's failure to hear those announcements is what triggers an election.

Failover: KS2 Promotes Itself

We shut KS1's core interface. Here is the real syslog sequence from KS2, in order:

KS2#show logging
%GDOI-5-COOP_KS_ADD: 10.100.0.11 added as COOP Key Server in group PLZ-GETVPN.
%GDOI-4-COOP_KS_UNAUTH: Contact from unauthorized KS 10.100.0.11 in group PLZ-GETVPN at
    local address 10.100.0.12 (Possible MISCONFIG of peer/local address)
%GDOI-5-COOP_KS_ELECTION: KS entering election mode in group PLZ-GETVPN (Previous Primary = NONE)
%GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.100.0.11 in group PLZ-GETVPN transitioned to Primary
    (Previous Primary = NONE)
%GDOI-3-COOP_KS_UNREACH: Cooperative KS 10.100.0.11 Unreachable in group PLZ-GETVPN.
    IKE SA Status = Failed to establish.
%GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.100.0.12 in group PLZ-GETVPN transitioned to Primary
    (Previous Primary = 10.100.0.11)

Walk it in order:

• COOP_KS_ADD is KS2 discovering its peer.
• COOP_KS_UNAUTH is the trap. Park it, it gets its own section below, and it is the most important line on this page.
• COOP_KS_ELECTION and the first COOP_KS_TRANS_TO_PRI are the initial election, correctly resolving in favour of KS1 (priority 100).
• COOP_KS_UNREACH is KS1 dying. IKE SA Status = Failed to establish, because the interface is down.
• The second COOP_KS_TRANS_TO_PRI is the payoff: KS 10.100.0.12 transitioned to Primary (Previous Primary = 10.100.0.11). The priority-75 secondary promoted itself.

And the post-failover state confirms it:

KS2#show crypto gdoi ks coop
    Local Address: 10.100.0.12
    Local Priority: 75
    Local KS Role: Primary   , Local KS Status: Alive
    Peer Address: 10.100.0.11
    Peer KS Role: Primary   , Peer KS Status: Dead

Local KS Role: Primary on the box with priority 75. Note the slightly unsettling Peer KS Role: Primary, Peer KS Status: Dead: KS2 has not forgotten that KS1 was Primary, it has simply marked it dead and taken over. When KS1 comes back with priority 100, it will reclaim the Primary role. COOP is preemptive, which is normally what you want (your best-connected key server should hold the role) but it does mean a flapping key server flaps the Primary role with it.

The Trap: COOP Key Servers Must Share the Same RSA Rekey Key

This is the section to read twice. It is the failure mode that separates a GETVPN deployment that survives its first rekey from one that dies quietly at 3am, and it is the reason we built this lab wrong on purpose.

We gave KS2 its own RSA rekey keypair, generated locally, instead of exporting KS1's and importing it. Everything looked fine. COOP peered. The election ran. The failover worked. Every show command above was captured on that setup. Nothing in show crypto gdoi ks coop says "your keys are wrong."

But the router did tell us, once, in the log:

%GDOI-4-COOP_KS_UNAUTH: Contact from unauthorized KS 10.100.0.11 in group PLZ-GETVPN
    at local address 10.100.0.12 (Possible MISCONFIG of peer/local address)

That line is the tell. Note how misleading the parenthetical hint is: "Possible MISCONFIG of peer/local address" sends you off checking IP addresses, which are perfectly correct. The actual cause is that the two key servers do not share an RSA rekey keypair, so they cannot authenticate each other's rekey authority.

Why it matters (and why no show command catches it)

Here is the mechanism, and once you see it the whole thing is obvious.

When a group member registers, it receives the key server's public key as part of the group policy. From then on, every rekey message the member receives is signed, and the member verifies that signature against the public key it holds. That is the entire anti-spoofing story for rekeys: without it, anyone who could reach a group member could push them a forged key.

Now suppose KS2 has its own, different RSA keypair. Consider the sequence:

The cruelty of this failure is the timing. The mistake is made at build time. The COOP election works. The failover test passes. Someone signs off the change. And the group dies at the next rekey interval after a key server failover, which could be weeks later, in the middle of the night, with the on-call engineer looking at a perfectly healthy-looking show crypto gdoi ks coop that says Primary, Alive.

And the group members are doing exactly the right thing. From their point of view, an unrecognised key server is signing rekeys, which is precisely the attack the signature exists to stop. The security mechanism is working. You just misconfigured which key it is protecting.

The fix: export from KS1, import on KS2

Both key servers must hold the same RSA rekey keypair. On KS1, where the key was generated (with exportable, which you must have specified at generation time, because you cannot add it afterwards):

crypto key export rsa PLZ-REKEY-KEY pem terminal 3des <passphrase>

That prints the PEM-encoded public and private keys to the terminal, with the private key encrypted under 3DES and your passphrase. Copy the whole block, then import it on KS2 with the matching crypto key import rsa command and the same passphrase and label. Both key servers now sign with the same key, so a rekey from either one verifies against the public key every member holds. Point rekey authentication mypubkey rsa PLZ-REKEY-KEY at that shared label on both.

Two practical notes. First, this is why the exportable keyword in the key generation step of the configuration article is not optional: forget it and your only remedy is to regenerate the key, which forces every member to re-register. Second, you are transporting a private key over a terminal session. Use SSH, use a strong passphrase, and clear your scrollback.

The verification nobody does

Because no show command reports "COOP key mismatch," the only reliable proof is behavioural. Do this in a maintenance window, on a lab, before you do it in production:

• Set a short rekey lifetime (ours was rekey lifetime seconds 900, deliberately short so the lab rekeys while you watch).
• Fail the primary key server.
• Wait past one full rekey interval with the secondary as Primary.
• Check Rekeys received in show crypto gdoi on a member. It must increment.
• Scan the key server logs for COOP_KS_UNAUTH. If it is there, your keys do not match, no matter how healthy the COOP state looks.

A failover test that only checks "did the secondary become Primary" and "does the ping still work" will pass on a completely broken deployment. Both of those things pass on ours, and ours was wrong.

Design Notes for Production

• Two key servers is the sensible baseline, and IOS supports more. Put them in different sites, on different power, and reachable from every member.
• Priority sets the winner, and preemption is on. Highest priority takes Primary and reclaims it when it returns. Give your best-connected key server the highest number.
• Members should list every key server in their server address ipv4 lines. They try them in order and can register to a Secondary quite happily.
• Key servers need ISAKMP keys for each other, not just for the members. COOP peering runs over an IKE SA (IKE status: Established).
• Keep the group config identical across key servers. Same identity number, same policy ACL, same transform set, same rekey parameters, same RSA rekey key. A COOP pair that disagrees about policy will hand different members different policies depending on which one they registered to.
• Watch for %GDOI-3-COOP_KS_UNREACH and %GDOI-4-COOP_KS_UNAUTH in your monitoring. The first is a genuine outage. The second is the silent killer above, and it deserves an alert.

Key Takeaways

• The data plane survives a dead key server. With KS1 shut down, GM1 pinged GM2's LAN at 100 percent, encrypted. The key server is control plane and is never in the data path. What you lose is rekeys, re-registration, and policy changes, so the group degrades on a timer rather than dropping.
• COOP fixes the control plane. A redundancy block under server local, a local priority, and a peer address ipv4. Highest priority becomes Primary.
• address ipv4 must come before redundancy. Otherwise: % ERROR: Local server address must be configured first.
• Failover is real and it works. The syslog runs COOP_KS_ELECTION, COOP_KS_UNREACH, COOP_KS_TRANS_TO_PRI, and the priority-75 secondary promotes itself to Primary.
• COOP key servers MUST share the same RSA rekey keypair. Export it from the primary (crypto key export rsa ... pem terminal 3des) and import it on the secondary. Generate it exportable or you will not be able to.
• %GDOI-4-COOP_KS_UNAUTH: Contact from unauthorized KS ... is the tell. Its hint about a "MISCONFIG of peer/local address" is misleading. The real cause is mismatched rekey keys.
• Mismatched keys kill the group silently. Members verify rekey signatures against the public key they hold. A secondary signing with a different key gets its rekeys rejected by every member, and the group dies at the next rekey interval, long after the change window closed.
• Test past a full rekey interval, not just the election. A failover test that checks only "Primary?" and "ping?" passes on a broken deployment. Check that Rekeys received increments.

That completes the GETVPN series: the model and the header-preservation capture, the IOS XE build, and now redundancy. All of it captured live on cat8000v 17.18.02. For the rest of the family (IKEv2, crypto maps, VTIs, DMVPN, FlexVPN, certificate authentication, and VRF-aware designs), the IPsec VPN pillar is the index.