The IKEv1 SA stays at QM_IDLE, connection 1001 is still active, and every ping between the LANs fails. That is the useful starting point when IPsec Phase 2 is not coming up on Cisco: check what happened to the data SAs and the traffic that should trigger them.
This Cisco IOL lab changes one item at a time: the transform set, PFS group, crypto ACL, NAT exemption and IPsec lifetime. Four changes stop forwarding. The fifth does not: with lifetimes of 3600 and 120 seconds, IOS negotiates the lower value and keeps carrying traffic through a rekey.
All results below come from the same measured lab. Configuration and output excerpts retain their original lines; a line containing only ... marks omitted lines. For a broader fault-isolation workflow, keep the PingLabz IPsec VPN troubleshooting guide alongside this exercise.
Topology and addressing
HQ and BRANCH run classic crypto-map IKEv1 on Cisco IOL 17.18.2, using the vrnetlab/cisco_iol:17.18.02 image. A third IOL router provides transit. The two Alpine 3.24.2 hosts generate LAN-to-LAN traffic. These results apply to that image and the policy shown below.
The ISP has connected WAN routes only, with no routes to either private LAN. Captures on both ISP data links show whether packets cross the transit network as ESP or leave HQ in cleartext.
hq-host HQ isp BRANCH branch-host
10.74.10.10 --- E0/2 10.74.10.1
E0/1 192.0.2.1 --- E0/1 192.0.2.2
E0/2 198.51.100.1 --- E0/1 198.51.100.2
E0/2 10.74.20.1 --- 10.74.20.20
<----------- LAN-to-LAN IPsec over the ISP ----------->
| Node | Data interface | Address | Route or role |
|---|---|---|---|
| hq-host | eth1 | 10.74.10.10/24 | 10.74.20.0/24 via 10.74.10.1 |
| HQ | Ethernet0/2 | 10.74.10.1/24 | HQ LAN gateway |
| HQ | Ethernet0/1 | 192.0.2.1/30 | Default via 192.0.2.2; crypto map and NAT outside |
| ISP | Ethernet0/1 | 192.0.2.2/30 | HQ-facing WAN |
| ISP | Ethernet0/2 | 198.51.100.1/30 | BRANCH-facing WAN |
| BRANCH | Ethernet0/1 | 198.51.100.2/30 | Default via 198.51.100.1; crypto map |
| BRANCH | Ethernet0/2 | 10.74.20.1/24 | Branch LAN gateway |
| branch-host | eth1 | 10.74.20.20/24 | 10.74.10.0/24 via 10.74.20.1 |
Begin with those interfaces and routes working and console access to both VPN peers. The walkthrough starts at the crypto configuration. For the terminology behind Main Mode, Quick Mode and data SAs, see IPsec explained: IKE phases.
Baseline crypto configuration
Phase 1 uses AES-256, SHA-256, pre-shared authentication and DH group 14, with an 86400-second policy lifetime. Phase 2 uses ESP AES-256/SHA-HMAC in tunnel mode, PFS group 14 and a 3600-second lifetime.
On HQ, the relevant running configuration is:
crypto isakmp policy 10
encryption aes 256
hash sha256
authentication pre-share
group 14
crypto isakmp key <PSK_REDACTED> address 198.51.100.2
...
crypto ipsec transform-set PL-TS esp-aes 256 esp-sha-hmac
mode tunnel
crypto ipsec transform-set PL-BAD esp-aes esp-sha-hmac
mode tunnel
...
crypto map PL-MAP 10 ipsec-isakmp
set peer 198.51.100.2
set transform-set PL-TS
set pfs group14
match address PL-VPN
!
...
interface Ethernet0/1
ip address 192.0.2.1 255.255.255.252
ip nat outside
crypto map PL-MAP
!
interface Ethernet0/2
ip address 10.74.10.1 255.255.255.0
ip nat inside
!
...
ip nat inside source list PL-NAT interface Ethernet0/1 overload
ip route 0.0.0.0 0.0.0.0 192.0.2.2
...
ip access-list extended PL-NAT
10 deny ip 10.74.10.0 0.0.0.255 10.74.20.0 0.0.0.255
20 permit ip 10.74.10.0 0.0.0.255 any
ip access-list extended PL-VPN
10 permit ip 10.74.10.0 0.0.0.255 10.74.20.0 0.0.0.255
On BRANCH, the peer address and crypto ACL reverse direction:
crypto isakmp policy 10
encryption aes 256
hash sha256
authentication pre-share
group 14
crypto isakmp key <PSK_REDACTED> address 192.0.2.1
...
crypto ipsec transform-set PL-TS esp-aes 256 esp-sha-hmac
mode tunnel
crypto ipsec transform-set PL-BAD esp-aes esp-sha-hmac
mode tunnel
...
crypto map PL-MAP 10 ipsec-isakmp
set peer 192.0.2.1
set transform-set PL-TS
set pfs group14
match address PL-VPN
!
...
interface Ethernet0/1
ip address 198.51.100.2 255.255.255.252
crypto map PL-MAP
!
interface Ethernet0/2
ip address 10.74.20.1 255.255.255.0
!
...
ip route 0.0.0.0 0.0.0.0 198.51.100.1
...
ip access-list extended PL-VPN
10 permit ip 10.74.20.0 0.0.0.255 10.74.10.0 0.0.0.255
Supply the same private lab PSK on both peers in place of the redacted placeholder. PL-TS is the working transform set. PL-BAD is a pre-created AES-128 alternative used only for round A; its presence alone does not break anything because the baseline map selects PL-TS.
HQ already runs NAT overload. The first entry in PL-NAT exempts traffic to the branch LAN from translation. A deny in this NAT classification ACL means “do not translate this traffic.” It does not drop the packet. Keep that ACL separate from PL-VPN, which selects traffic for IPsec.
The baseline map output on HQ confirms the effective lifetime, including the default that does not appear as an explicit lifetime command in the running configuration:
show crypto map
Crypto Map IPv4 "PL-MAP" 10 ipsec-isakmp
...
Security association lifetime: 4608000 kilobytes/3600 seconds
Prove the working baseline
From hq-host, run ping -c 15 -W 1 10.74.20.20 to send 15 pings with a one-second reply timeout. After clearing the data SAs, the first attempt gets 14 replies:
PING 10.74.20.20 (10.74.20.20): 56 data bytes
...
--- 10.74.20.20 ping statistics ---
15 packets transmitted, 14 packets received, 6% packet loss
On HQ, Phase 1 remains active after the test:
show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
198.51.100.2 192.0.2.1 QM_IDLE 1001 ACTIVE
The corresponding IPsec output has matching LAN identities, positive counters, nonzero SPIs and active ESP SAs:
show crypto ipsec sa
...
local ident (addr/mask/prot/port): (10.74.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.74.20.0/255.255.255.0/0/0)
current_peer 198.51.100.2 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 14, #pkts encrypt: 14, #pkts digest: 14
#pkts decaps: 14, #pkts decrypt: 14, #pkts verify: 14
...
#send errors 0, #recv errors 0
...
current outbound spi: 0x3F4EB86(66382726)
PFS (Y/N): Y, DH group: group14
...
inbound esp sas:
spi: 0xBCC9EE9A(3167350426)
transform: esp-256-aes esp-sha-hmac ,
...
sa timing: remaining key lifetime (k/sec): (4220351/3584)
...
Status: ACTIVE(ACTIVE)
...
outbound esp sas:
spi: 0x3F4EB86(66382726)
transform: esp-256-aes esp-sha-hmac ,
...
sa timing: remaining key lifetime (k/sec): (4220351/3584)
...
Status: ACTIVE(ACTIVE)
On branch-host, the reverse command ping -c 5 -W 1 10.74.10.10 succeeds:
PING 10.74.10.10 (10.74.10.10): 56 data bytes
...
--- 10.74.10.10 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
HQ’s encapsulation and decapsulation counters then rise from 14 to 19. The ISP captures contain 38 ESP frames on each link: 19 in each direction, with no cleartext ICMP. Zeek’s HQ-facing connection record independently identifies IP protocol 50, ESP:
{"ts":1792197814.195043,"uid":"C4qAzg2GDz8QCUsfi9","id.orig_h":"192.0.2.1","id.orig_p":0,"id.resp_h":"198.51.100.2","id.resp_p":0,"proto":"unknown_transport","duration":23.302730798721313,"orig_bytes":0,"resp_bytes":0,"conn_state":"OTH","local_orig":true,"local_resp":true,"missed_bytes":0,"orig_pkts":19,"orig_ip_bytes":2888,"resp_pkts":19,"resp_ip_bytes":2888,"ip_proto":50}
Zeek does not decrypt the tunnel or establish the IKE state. Use its protocol and packet observations with the router output and endpoint replies. Counts from a capture window also need not equal a router’s accumulated counters.
Keep Phase 1 in place while testing Phase 2
Each round starts from the restored baseline and forces fresh Phase 2 negotiation by clearing the peer’s IPsec SAs on both routers. At HQ’s privileged EXEC prompt:
clear crypto sa peer 198.51.100.2
At BRANCH’s privileged EXEC prompt:
clear crypto sa peer 192.0.2.1
The IKE SA is left in place. Connection 1001 remains QM_IDLE and active throughout the fault tests. That makes the distinction between Phase 1 and Phase 2 observable.
For the initial probe in each fault round, both peers briefly enable these debugs at privileged EXEC, with logging buffered and console logging disabled:
debug crypto isakmp
debug crypto ipsec
Stop debugging after that bounded probe, then inspect the buffered log:
undebug all
show logging
The timestamps matter: the buffers retain earlier rounds. Old packet totals can survive a clear on either peer, even when no current ESP SAs exist. The fault excerpts below use HQ’s post-probe counters; check the change in those totals, current SPIs and active SAs together.
Round A: The crypto map selects the wrong transform set
Fault: BRANCH selects PL-BAD (AES-128) while HQ continues to offer PL-TS (AES-256).
At BRANCH’s crypto-map configuration prompt, the injected command is:
set transform-set PL-BAD
The 15-packet HQ-to-branch probe fails completely:
PING 10.74.20.20 (10.74.20.20): 56 data bytes
...
--- 10.74.20.20 ping statistics ---
15 packets transmitted, 0 packets received, 100% packet loss
HQ still has the original IKE SA, but its data counters stay at zero and its outbound SPI is zero:
show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
198.51.100.2 192.0.2.1 QM_IDLE 1001 ACTIVE
show crypto ipsec sa
...
local ident (addr/mask/prot/port): (10.74.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.74.20.0/255.255.255.0/0/0)
current_peer 198.51.100.2 port 500
PERMIT, flags={origin_is_acl,ipsec_sa_request_sent}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
...
#send errors 15, #recv errors 0
...
current outbound spi: 0x0(0)
PFS (Y/N): N, DH group: none
BRANCH’s debug rejects the proposed transform:
*Oct 17 00:44:12.645: IPSEC(ipsec_process_proposal): invalid transform proposal received:
{esp-aes 256 esp-sha-hmac }
*Oct 17 00:44:12.645: ISAKMP-ERROR: (1001):IPSec policy invalidated proposal with error 256
*Oct 17 00:44:12.645: ISAKMP-ERROR: (1001):phase 2 SA policy not acceptable! (local 198.51.100.2 remote 192.0.2.1)
The diagnosis clue is the proposed AES-256 transform against a map selecting the AES-128 set. On this run, that produces error 256. Compare the actual algorithms and key lengths in the selected sets; the set names alone tell you little.
Both ISP captures contain three UDP/500 frames and zero ESP frames during the fault. The packet summary omits the ESP field when none were counted. Here is the HQ-facing summary excerpt:
{
"isp-eth1.pcap": {
"counts": {
"frames": 11,
"ike_udp500": 3,
"ike_exchange_5": 2,
"ike_exchange_32": 1
},
"identical_ike_payload_repeats": 0,
"repeated_ike_payloads": [],
"cleartext_icmp_examples": []
},
...
}
To fix it, select the working set again under BRANCH’s existing crypto map:
set transform-set PL-TS
Clear the peer IPsec SAs on both routers as above, then probe again. Recovery was delayed in this round: the first forward retry still lost all 15 packets. A reverse initiation from branch-host then received four of five replies:
PING 10.74.10.10 (10.74.10.10): 56 data bytes
...
--- 10.74.10.10 ping statistics ---
5 packets transmitted, 4 packets received, 20% packet loss
HQ then has active ESP SAs and four packets in each direction:
show crypto ipsec sa
...
local ident (addr/mask/prot/port): (10.74.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.74.20.0/255.255.255.0/0/0)
current_peer 198.51.100.2 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 4, #pkts encrypt: 4, #pkts digest: 4
#pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4
...
#send errors 0, #recv errors 0
...
current outbound spi: 0x1F87489A(528959642)
PFS (Y/N): Y, DH group: group14
...
inbound esp sas:
spi: 0xABF05D8B(2884656523)
transform: esp-256-aes esp-sha-hmac ,
...
sa timing: remaining key lifetime (k/sec): (4356510/3594)
...
Status: ACTIVE(ACTIVE)
...
outbound esp sas:
spi: 0x1F87489A(528959642)
transform: esp-256-aes esp-sha-hmac ,
...
sa timing: remaining key lifetime (k/sec): (4356510/3594)
...
Status: ACTIVE(ACTIVE)
The completed recovery captures contain eight ESP frames on each link, four per direction. Those observations establish encrypted forwarding before the next round; they do not establish a particular retry timer. Later repairs first run ping -c 3 -W 1 10.74.10.10 on branch-host as a reverse warm-up, then perform the forward probe and steady five-packet checks in both directions.
Round B: PFS groups disagree
Fault: BRANCH’s crypto map uses PFS group 19 while HQ’s map stays at group 14.
The change on BRANCH is:
set pfs group19
HQ-to-branch ping fails again:
PING 10.74.20.20 (10.74.20.20): 56 data bytes
...
--- 10.74.20.20 ping statistics ---
15 packets transmitted, 0 packets received, 100% packet loss
The IKE SA remains active, while HQ has no usable outbound data SA:
show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
198.51.100.2 192.0.2.1 QM_IDLE 1001 ACTIVE
show crypto ipsec sa
...
local ident (addr/mask/prot/port): (10.74.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.74.20.0/255.255.255.0/0/0)
current_peer 198.51.100.2 port 500
PERMIT, flags={origin_is_acl,ipsec_sa_request_sent}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
...
#send errors 15, #recv errors 0
...
current outbound spi: 0x0(0)
PFS (Y/N): N, DH group: none
BRANCH sees the offered group 14, then rejects the IPsec policy:
*Oct 17 00:45:23.645: ISAKMP: (1001): group is 14
...
*Oct 17 00:45:23.646: IPSEC(ipsec_process_proposal): invalid transform proposal flags -- 0x81
*Oct 17 00:45:23.646: ISAKMP-ERROR: (1001):IPSec policy invalidated proposal with error 1024
*Oct 17 00:45:23.646: ISAKMP-ERROR: (1001):phase 2 SA policy not acceptable! (local 198.51.100.2 remote 192.0.2.1)
The clue is the difference between the offered PFS group and the group configured under BRANCH’s map. Phase 1 still uses DH group 14 on both peers; changing PFS in the crypto map changes the Phase 2 requirement. Error 1024 is the observed rejection here, rather than a universal lookup code for every Cisco platform.
Both capture windows contain three UDP/500 frames and no ESP. The transforms and LAN selectors have already been restored, so the single changed PFS group explains this failure. The HQ-facing packet summary retains the complete counts object:
{
"isp-eth1.pcap": {
"counts": {
"frames": 8,
"ike_udp500": 3,
"ike_exchange_5": 2,
"ike_exchange_32": 1
},
"identical_ike_payload_repeats": 0,
"repeated_ike_payloads": [],
"cleartext_icmp_examples": []
},
...
}
Under BRANCH’s crypto map, restore:
set pfs group14
After the peer IPsec clears and reverse warm-up, the forward probe gets all 15 replies:
PING 10.74.20.20 (10.74.20.20): 56 data bytes
...
--- 10.74.20.20 ping statistics ---
15 packets transmitted, 15 packets received, 0% packet loss
HQ’s post-probe counters and outbound SPI now show encrypted forwarding:
show crypto ipsec sa
...
local ident (addr/mask/prot/port): (10.74.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.74.20.0/255.255.255.0/0/0)
current_peer 198.51.100.2 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 17, #pkts encrypt: 17, #pkts digest: 17
#pkts decaps: 17, #pkts decrypt: 17, #pkts verify: 17
...
#send errors 0, #recv errors 0
...
current outbound spi: 0x22AB59E7(581655015)
PFS (Y/N): Y, DH group: group14
The later steady five-packet tests succeed in both directions. Each ISP link records 52 ESP frames in the full recovery window, including warm-up and verification traffic.
Round C: The proxy IDs do not match the protected LANs
Fault: BRANCH’s map selects an ACL protecting 10.74.20.0/24 to 10.74.99.0/24, instead of the actual HQ LAN, 10.74.10.0/24.
The effective faulty configuration on BRANCH is:
crypto map PL-MAP 10 ipsec-isakmp
set peer 192.0.2.1
set transform-set PL-TS
set pfs group14
match address PL-VPN-BAD
!
...
ip access-list extended PL-VPN-BAD
10 permit ip 10.74.20.0 0.0.0.255 10.74.99.0 0.0.0.255
The host probe fails, with Phase 1 still up and no outbound ESP SPI at HQ:
PING 10.74.20.20 (10.74.20.20): 56 data bytes
...
--- 10.74.20.20 ping statistics ---
15 packets transmitted, 0 packets received, 100% packet loss
show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
198.51.100.2 192.0.2.1 QM_IDLE 1001 ACTIVE
show crypto ipsec sa
...
local ident (addr/mask/prot/port): (10.74.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.74.20.0/255.255.255.0/0/0)
current_peer 198.51.100.2 port 500
PERMIT, flags={origin_is_acl,ipsec_sa_request_sent}
#pkts encaps: 27, #pkts encrypt: 27, #pkts digest: 27
#pkts decaps: 27, #pkts decrypt: 27, #pkts verify: 27
...
#send errors 15, #recv errors 0
...
current outbound spi: 0x0(0)
PFS (Y/N): N, DH group: none
HQ’s 27 encapsulated and decapsulated packets are retained totals: both were already 27 before this probe. The failed test adds 15 send errors, while the outbound SPI stays at zero. In this output, the PFS field describes the absent data SA; compare configured PFS requirements in the crypto maps.
BRANCH logs the identities for which it cannot find a matching policy:
*Oct 17 00:47:50.956: map_db_find_best did not find matching mapcrypto_mapdb_find_map Fail to find matching policy ivrf 0, fvrf 0, flags 129, ike_profile NULL, local_proxy 10.74.20.0, remote_proxy 10.74.10.0, km_local 198.51.100.2, km_remote 192.0.2.1, km_local_port 39, km_remote_port 44633
*Oct 17 00:47:50.956: ISAKMP-ERROR: (1001):IPSec policy invalidated proposal with error 32
*Oct 17 00:47:50.957: ISAKMP-ERROR: (1001):phase 2 SA policy not acceptable! (local 198.51.100.2 remote 192.0.2.1)
The useful clue is the local and remote proxy pair: 10.74.20.0 and 10.74.10.0. Proxy IDs describe the protected traffic selectors. BRANCH’s selected ACL expects 10.74.99.0 instead, so it cannot accept this LAN-to-LAN proposal. Mirroring the ACL means swapping source and destination networks while retaining the corresponding masks and protocol scope.
The fault captures contain two UDP/500 frames per link and zero ESP. This is a policy rejection, consistent with the missing selector match. The HQ-facing packet summary is:
{
"isp-eth1.pcap": {
"counts": {
"frames": 8,
"ike_udp500": 2,
"ike_exchange_32": 1,
"ike_exchange_5": 1
},
"identical_ike_payload_repeats": 0,
"repeated_ike_payloads": [],
"cleartext_icmp_examples": []
},
...
}
To repair it, enter BRANCH’s existing crypto-map configuration mode and restore the original ACL reference:
match address PL-VPN
Then leave crypto-map mode and remove the unused bad ACL in global configuration mode:
no ip access-list extended PL-VPN-BAD
The tested approach switches the map to a separate ACL; it does not edit an ACL while that ACL is attached to the map. After clearing the peer IPsec SAs and warming the reverse direction, the forward probe succeeds:
PING 10.74.20.20 (10.74.20.20): 56 data bytes
...
--- 10.74.20.20 ping statistics ---
15 packets transmitted, 15 packets received, 0% packet loss
HQ’s counters rise from 27 to 44 in each direction, and its outbound SPI becomes nonzero:
show crypto ipsec sa
...
local ident (addr/mask/prot/port): (10.74.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.74.20.0/255.255.255.0/0/0)
current_peer 198.51.100.2 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 44, #pkts encrypt: 44, #pkts digest: 44
#pkts decaps: 44, #pkts decrypt: 44, #pkts verify: 44
...
#send errors 0, #recv errors 0
...
current outbound spi: 0x738276BC(1937929916)
PFS (Y/N): Y, DH group: group14
Steady five-packet checks succeed in both directions, and each recovery capture contains 52 ESP frames.
Round D: NAT changes the source before crypto selection
Fault: HQ loses sequence 10, the VPN exemption in PL-NAT, leaving the general overload permit in place.
This is the resulting NAT classification ACL on HQ:
ip access-list extended PL-NAT
20 permit ip 10.74.10.0 0.0.0.255 any
All 15 pings fail. HQ still reports the same IKE SA, but this time its IPsec send-error counter also stays at zero:
PING 10.74.20.20 (10.74.20.20): 56 data bytes
...
--- 10.74.20.20 ping statistics ---
15 packets transmitted, 0 packets received, 100% packet loss
show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
198.51.100.2 192.0.2.1 QM_IDLE 1001 ACTIVE
show crypto ipsec sa
...
local ident (addr/mask/prot/port): (10.74.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.74.20.0/255.255.255.0/0/0)
current_peer 198.51.100.2 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
...
#send errors 0, #recv errors 0
...
current outbound spi: 0x0(0)
PFS (Y/N): N, DH group: none
The NAT translation table supplies the decisive clue:
show ip nat translations
Pro Inside global Inside local Outside local Outside global
icmp 192.0.2.1:1024 10.74.10.10:91 10.74.20.20:91 10.74.20.20:1024
The inside source, 10.74.10.10, becomes 192.0.2.1. That source no longer matches HQ’s crypto ACL for 10.74.10.0/24 to 10.74.20.0/24, so the traffic does not trigger a new Quick Mode exchange.
There is no new proposal-rejection message for this probe. The short HQ log excerpt records the configuration change and a node purge; neither line diagnoses NAT on its own:
*Oct 17 00:48:56.782: %SYS-5-CONFIG_I: Configured from console by console
*Oct 17 00:49:07.744: ISAKMP: (1001):purging node 84203615
Do not reuse the older transform, PFS or proxy errors still in the buffer as an explanation for this round. The translation and WAN packets are the evidence that distinguishes it.
On the HQ-facing ISP link, Zeek records 15 cleartext echo requests from the translated address, with no echo replies in that connection:
{"ts":1792198142.363111,"uid":"CLwZ2h3d2e7dvShV05","id.orig_h":"192.0.2.1","id.orig_p":8,"id.resp_h":"10.74.20.20","id.resp_p":0,"proto":"icmp","duration":14.003781080245972,"orig_bytes":840,"resp_bytes":0,"conn_state":"OTH","local_orig":true,"local_resp":true,"missed_bytes":0,"orig_pkts":15,"orig_ip_bytes":1260,"resp_pkts":0,"resp_ip_bytes":0,"ip_proto":1}
The packet summary counts 30 cleartext ICMP frames there: 15 requests and 15 destination-unreachable messages from the ISP. The branch-facing link has no cleartext ICMP, and neither link has ESP. The ISP has no route to the branch LAN; the translated requests never cross to BRANCH.
On HQ, restore the exemption under ip access-list extended PL-NAT:
10 deny ip 10.74.10.0 0.0.0.255 10.74.20.0 0.0.0.255
Return to privileged EXEC and clear the old dynamic translations in this lab:
clear ip nat translation *
Then force fresh peer IPsec SAs and repeat the traffic checks. After reverse warm-up, the forward probe succeeds:
PING 10.74.20.20 (10.74.20.20): 56 data bytes
...
--- 10.74.20.20 ping statistics ---
15 packets transmitted, 15 packets received, 0% packet loss
show crypto ipsec sa
...
local ident (addr/mask/prot/port): (10.74.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.74.20.0/255.255.255.0/0/0)
current_peer 198.51.100.2 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 17, #pkts encrypt: 17, #pkts digest: 17
#pkts decaps: 17, #pkts decrypt: 17, #pkts verify: 17
...
#send errors 0, #recv errors 0
...
current outbound spi: 0x9E816F1D(2659282717)
PFS (Y/N): Y, DH group: group14
The steady five-packet probes pass in both directions, with 52 ESP frames and no cleartext ICMP on each link during recovery. For more on reading the translation table, see NAT/PAT troubleshooting on Cisco IOS XE.
Round E: The lifetime mismatch does not break the tunnel
Change under test: BRANCH’s IPsec lifetime becomes 120 seconds while HQ remains at 3600 seconds.
At BRANCH’s crypto-map configuration prompt, IOS accepts the shorter value and warns about it:
set security-association lifetime seconds 120
Warning! Value of 120 sec is lower than the recommended optimum value of 900 sec
The initial forward probe receives 14 of 15 replies, just as the baseline did. hq-host then runs ping -c 125 -W 1 10.74.20.20 and receives every reply:
PING 10.74.20.20 (10.74.20.20): 56 data bytes
...
--- 10.74.20.20 ping statistics ---
15 packets transmitted, 14 packets received, 6% packet loss
PING 10.74.20.20 (10.74.20.20): 56 data bytes
...
--- 10.74.20.20 ping statistics ---
125 packets transmitted, 125 packets received, 0% packet loss
Phase 1 remains active at the end of the longer test:
show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
198.51.100.2 192.0.2.1 QM_IDLE 1001 ACTIVE
HQ’s debug explicitly records the accepted responder lifetime:
*Oct 17 00:50:14.258: ISAKMP: (1001):processing NOTIFY RESPONDER_LIFETIME protocol 3
spi 2527534807, message ID = 2458003205, sa = 0x80007D416EA9BA60
*Oct 17 00:50:14.258: ISAKMP: (1001):processing responder lifetime
*Oct 17 00:50:14.258: ISAKMP: (1001): responder lifetime of 120s
IOS negotiated the lower, 120-second lifetime in this test. The first HQ data-SA snapshot shows 14 packets in each direction, active ESP SAs and 94 seconds remaining:
show crypto ipsec sa
...
local ident (addr/mask/prot/port): (10.74.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.74.20.0/255.255.255.0/0/0)
current_peer 198.51.100.2 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 14, #pkts encrypt: 14, #pkts digest: 14
#pkts decaps: 14, #pkts decrypt: 14, #pkts verify: 14
...
#send errors 0, #recv errors 0
...
current outbound spi: 0x96A71ED7(2527534807)
PFS (Y/N): Y, DH group: group14
...
inbound esp sas:
spi: 0x89F9FAC5(2314861253)
transform: esp-256-aes esp-sha-hmac ,
...
sa timing: remaining key lifetime (k/sec): (4277245/94)
...
Status: ACTIVE(ACTIVE)
...
outbound esp sas:
spi: 0x96A71ED7(2527534807)
transform: esp-256-aes esp-sha-hmac ,
...
sa timing: remaining key lifetime (k/sec): (4277245/94)
...
Status: ACTIVE(ACTIVE)
After the follow-up, the counters reach 139 per direction and both ESP SPIs have changed. HQ’s later snapshot is:
show crypto ipsec sa
...
local ident (addr/mask/prot/port): (10.74.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.74.20.0/255.255.255.0/0/0)
current_peer 198.51.100.2 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 139, #pkts encrypt: 139, #pkts digest: 139
#pkts decaps: 139, #pkts decrypt: 139, #pkts verify: 139
...
#send errors 0, #recv errors 0
...
current outbound spi: 0x74CEFD5B(1959722331)
PFS (Y/N): Y, DH group: group14
...
inbound esp sas:
spi: 0xBC3F4808(3158263816)
transform: esp-256-aes esp-sha-hmac ,
...
sa timing: remaining key lifetime (k/sec): (4251322/50)
...
Status: ACTIVE(ACTIVE)
...
outbound esp sas:
spi: 0x74CEFD5B(1959722331)
transform: esp-256-aes esp-sha-hmac ,
...
sa timing: remaining key lifetime (k/sec): (4251322/50)
...
Status: ACTIVE(ACTIVE)
The WAN capture corroborates forwarding and rekey: 278 ESP frames per link, six Quick Mode frames representing two three-message exchanges, and four distinct ESP SPIs across both directions. There is no cleartext ICMP. The HQ-facing summary excerpt is:
{
"isp-eth1.pcap": {
"counts": {
"frames": 320,
"ike_udp500": 8,
"ike_exchange_5": 2,
"ike_exchange_32": 6,
"esp": 278,
"esp 192.0.2.1 -> 198.51.100.2": 139,
"esp 198.51.100.2 -> 192.0.2.1": 139
},
"identical_ike_payload_repeats": 0,
"repeated_ike_payloads": [],
"cleartext_icmp_examples": []
},
...
}
The diagnosis clue is an accepted shorter lifetime alongside active SAs, changed SPIs and sustained replies. A different configured lifetime is not, by itself, proof of a Phase 2 failure. This result is specific to the tested image and policy combination.
There is no outage to repair in round E. To return BRANCH’s crypto map to the baseline lifetime, the recorded restoration command is:
set security-association lifetime seconds 3600
After the peer IPsec clears and reverse warm-up, the forward probe gets 15 of 15 replies; subsequent steady checks get five of five in both directions. Both recovery captures contain 52 ESP frames. Restoring 3600 seconds makes the configurations consistent; it does not explain away a failure that never occurred.
Quick reference: symptom to cause to fix
These are the pairings observed in this lab. Read error numbers with the surrounding proposal, selectors and current configuration.
| Symptom or debug clue | Cause in this test | Fix |
|---|---|---|
Active QM_IDLE; zero HQ data counters; transform rejection with error 256 |
BRANCH map selects AES-128 while HQ proposes AES-256 | Select PL-TS on BRANCH; clear peer IPsec SAs and verify traffic |
Active QM_IDLE; offered PFS group 14; policy rejection with error 1024 |
BRANCH map requires PFS group 19 | Restore PFS group 14 on BRANCH; renegotiate Phase 2 |
| Matching-policy lookup fails for the actual LAN pair; error 32 | BRANCH crypto ACL names 10.74.99.0/24 as the remote LAN | Restore the mirrored PL-VPN ACL reference |
| Active IKE SA; no new Quick Mode rejection; translated LAN source and cleartext WAN ICMP | HQ NAT exemption is missing | Restore sequence 10 in PL-NAT; clear stale translations and verify ESP |
| Responder lifetime accepted at 120 seconds; active ESP SAs; 125/125 follow-up replies | Lifetime mismatch negotiated successfully | No outage fix required; restore 3600 seconds for baseline consistency |
Takeaways and verification checklist
QM_IDLE alone cannot prove that a protected flow works. In rounds A–C, Quick Mode reaches BRANCH and gets rejected. In round D, NAT changes the packet so the crypto ACL does not select it. Both paths leave Phase 1 active and the host unable to reach the far LAN, but the fixes are different.
The recorded final state returns both peers to the baseline configuration. Steady five-packet probes pass in both directions, both peers have active ESP SAs, and each final ISP capture contains 48 ESP frames with no cleartext ICMP. That is the measured final verification, not a claim about a tunnel’s present state.
For a Phase 1 up, Phase 2 down investigation:
- Confirm the intended peer has an active IKE SA on both routers.
- Generate traffic between addresses covered by the crypto ACL, using the intended LAN source.
- Read the local and remote identities, current ESP SPIs and active SAs. Compare counters before and after the probe rather than trusting an old total.
- Compare the transform set actually selected by each map, including cipher strength, authentication and mode.
- Compare the Phase 2 PFS settings separately from the Phase 1 DH policy.
- Check that crypto ACLs describe the same protected traffic in opposite directions.
- Inspect NAT classification and translations if the packet never triggers negotiation.
- Read timestamped debug lines from the current attempt, then turn debugging off.
- Confirm ESP on both transit links and replies at the endpoints, including a reverse test.
- For a lifetime concern, inspect the negotiated value and test across rekey before calling it an outage.
Change one item, rebuild the data SAs, and keep testing until router state, WAN packets and endpoint results agree.