Almost everything written about IPsec starts with a config snippet. That is backwards. If you do not know what an SA is, why IKE runs in two phases, or which bytes of your packet actually get encrypted, then a crypto map is just a spell you cast and hope. This article is the conceptual foundation for the whole IPsec VPN cluster: what IPsec is, how the two-phase IKE model works, what a Security Association really is, and precisely what an eavesdropper on the transit link can and cannot see.
Everything below is grounded in a real CML lab: two Cisco IOS XE edge routers (cat8000v 17.18.02) either side of a service provider router, with real packet captures taken on the transit link.
IPsec Is a Framework, Not a Protocol
The first thing to unlearn: IPsec is not one protocol. It is a set of them, working at Layer 3, that together give you three properties for IP packets.
Two protocols deliver those properties on the wire (AH and ESP), and one negotiates the keys that make them possible (IKE). The split between "the thing that protects packets" and "the thing that agrees on how to protect packets" is the single most useful mental model in IPsec.
AH vs ESP: Only One of These Matters in Practice
AH (Authentication Header, IP protocol 51) and ESP (Encapsulating Security Payload, IP protocol 50) are the two protection protocols. In modern designs you will use ESP, essentially always. Here is why.
The killer difference is the NAT row. AH signs the outer IP header, so any NAT device on the path invalidates the signature and the packet is discarded. ESP deliberately leaves the outer header out of its integrity check, which is exactly what lets it cross the internet. Every capture in this article is ESP.
Tunnel Mode vs Transport Mode: Which Bytes Get Encrypted
This is where most people go vague, so let us be precise. ESP encrypts everything it encapsulates and nothing else. The difference between the two modes is what gets encapsulated.
Transport mode keeps the original IP header and protects only the payload behind it:
[ orig IP hdr ][ ESP hdr ][ TCP/UDP + data (encrypted) ][ ESP trailer ][ ESP auth ]
^------------ encrypted -------------^The source and destination host IPs stay visible on the wire. Transport mode is for host-to-host protection, and inside IOS XE it is what you use under a GRE tunnel or an SVTI (because there is already an outer delivery header, so a second one would be a waste of 20 bytes).
Tunnel mode encrypts the entire original packet, header included, and builds a brand new outer IP header addressed to the crypto endpoints:
[ new IP hdr ][ ESP hdr ][ orig IP hdr + TCP/UDP + data (encrypted) ][ ESP trailer ][ ESP auth ]
^------------------ encrypted ------------------^The original source and destination addresses are now ciphertext. On the wire, the packet appears to be a conversation between the two VPN gateways and nothing else. Our lab transform set says exactly this:
crypto ipsec transform-set PLZ-TS-AES256 esp-aes 256 esp-sha256-hmac
mode tunnelAnd IOS XE confirms it on the live SA with in use settings ={Tunnel, }. That single word is the difference between the ISP seeing your internal addressing and seeing nothing at all. We prove it with a capture further down.
Why IKE Runs in Two Phases
Before any packet can be protected, the two peers must agree on keys. That is IKE's job (Internet Key Exchange, UDP 500). The awkward part is the chicken-and-egg problem: negotiating keys securely requires a secure channel, and you do not have one yet.
IKE solves it by doing the job twice, for two different purposes:
crypto isakmp policy + the pre-shared keyshow crypto isakmp sashow crypto ipsec saThe payoff is efficiency. Phase 1 is expensive (Diffie-Hellman is heavy maths) but you do it once and keep it for an hour. Phase 2 is cheap and runs inside that already-encrypted channel, so you can rekey data keys often, or add a second protected flow, without redoing the expensive part. It also means a Phase 2 failure looks nothing like a Phase 1 failure, which is the single most useful fact when troubleshooting an IPsec VPN.
Watching It Happen: The Real Packet List
Here is a capture taken on the transit link between our edge router (203.0.113.1) and the remote peer (198.51.100.2), the moment protection was enabled and a ping was sent. This is the two-phase model, on the wire, in order:
No. Time Source Destination Proto Info
3 5.153552 203.0.113.1 198.51.100.2 ISAKMP Identity Protection (Main Mode)
6 5.617148 198.51.100.2 203.0.113.1 ISAKMP Identity Protection (Main Mode)
9 5.861495 203.0.113.1 198.51.100.2 ISAKMP Quick Mode
11 6.425161 203.0.113.1 198.51.100.2 ISAKMP Quick Mode
12 7.026032 203.0.113.1 198.51.100.2 ESP ESP (SPI=0x6d4428aa)
13 8.939238 198.51.100.2 203.0.113.1 ESP ESP (SPI=0x94cd6141)
14 12.002692 203.0.113.1 198.51.100.2 ESP ESP (SPI=0x6d4428aa)
15 12.003690 198.51.100.2 203.0.113.1 ESP ESP (SPI=0x94cd6141)
16 13.004286 203.0.113.1 198.51.100.2 ESP ESP (SPI=0x6d4428aa)
17 13.005508 198.51.100.2 203.0.113.1 ESP ESP (SPI=0x94cd6141)Read it top to bottom and the theory becomes concrete.
Packets 3 to 6: Main Mode. Wireshark labels IKEv1 Main Mode "Identity Protection", which is the whole point of it. Main Mode is six messages in three round trips. Messages 1 and 2 propose and accept the ISAKMP policy (encryption, hash, DH group, authentication method, lifetime). Messages 3 and 4 carry the Diffie-Hellman public values and nonces, and at the end of message 4 both peers independently derive the same shared secret without it ever crossing the link. Messages 5 and 6 are where each peer proves who it is (with the pre-shared key here), and those two messages are already encrypted with the keys derived in step 2. That is the identity protection: who you are never travels in the clear. Aggressive Mode collapses this into three messages and gives that up, which is why you avoid it.
Packets 9 and 11: Quick Mode. Phase 2. Everything in these packets is encrypted inside the Phase 1 channel, which is why the capture can tell you they are Quick Mode but not what is in them. Quick Mode negotiates the things that will actually protect user traffic: the transform set (AES-256 with SHA-256 HMAC), the mode (tunnel), the proxy identities (which subnets this SA covers), an optional fresh Diffie-Hellman exchange if PFS is on, and the SPIs each side will use.
Packets 12 onward: ESP. The negotiation is done. From here the ISP sees nothing but IP protocol 50 between two public IPs, and the SPI values that Quick Mode agreed on.
QM_IDLE Does Not Mean Idle
Once Phase 1 is complete, the ISAKMP SA reports its state:
EDGE1#show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
198.51.100.2 203.0.113.1 QM_IDLE 1001 ACTIVEQM_IDLE trips people up because it sounds like something is asleep. It is not. It means Main Mode finished successfully, the ISAKMP SA is established and authenticated, and the Quick Mode state machine is idle because it has nothing to do right now. QM_IDLE is the healthy steady state of Phase 1. It is what you want to see.
Critically, QM_IDLE says nothing about whether user traffic is flowing. Phase 1 up and Phase 2 dead is an extremely common failure (a transform set mismatch, a crypto ACL mismatch), and it looks exactly like a healthy tunnel in this command. That is why you never stop at show crypto isakmp sa.
What a Security Association Actually Is
An SA is a one-way contract: "traffic matching this selector, going in this direction, is protected with this algorithm, this key, and this SPI". The SPI (Security Parameter Index) is the 32-bit tag the receiver uses to look up which SA an inbound ESP packet belongs to. Because an SA is one-way, you always get them in pairs. Here is the live SA on the lab, trimmed:
interface: GigabitEthernet2
Crypto map tag: PLZ-CMAP, local addr 203.0.113.1
protected vrf: (none)
local ident (addr/mask/prot/port): (10.20.10.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (10.30.10.0/255.255.255.0/0/0)
current_peer 198.51.100.2 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 9, #pkts encrypt: 9, #pkts digest: 9
#pkts decaps: 9, #pkts decrypt: 9, #pkts verify: 9
#send errors 0, #recv errors 0
local crypto endpt.: 203.0.113.1, remote crypto endpt.: 198.51.100.2
plaintext mtu 1438, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet2
current outbound spi: 0x4EA0F404(1319171076)
PFS (Y/N): Y, DH group: group14
inbound esp sas:
spi: 0xD0E19C7F(3504446591)
transform: esp-256-aes esp-sha256-hmac ,
in use settings ={Tunnel, }
sa timing: remaining key lifetime (k/sec): (4607998/3572)
Status: ACTIVE(ACTIVE)Read the SPIs. The current outbound spi is 0x4EA0F404 and the inbound esp sas block shows a completely different SPI, 0xD0E19C7F. That is not a bug. Each peer chooses the SPI for the traffic it will receive, and tells the other side to use it. Inbound and outbound are separate SAs with separate keys, and that is why the counters come in mirrored pairs (encaps/encrypt/digest on the way out, decaps/decrypt/verify on the way in). If you see encaps incrementing but decaps stuck at zero, you have a one-way tunnel, and you now know exactly which SA to go looking for.
Two other lines earn their keep. PFS (Y/N): Y, DH group: group14 means Perfect Forward Secrecy is on: Quick Mode ran its own Diffie-Hellman exchange, so cracking the Phase 1 key still would not hand an attacker the data keys. And plaintext mtu 1438 is ESP overhead made visible, the largest plaintext packet that fits before encryption pushes it past the 1500-byte path MTU. That number is behind a large share of "the tunnel is up but big transfers hang" tickets.
Proxy IDs: One Crypto ACL Line, One SA Pair
In a classic crypto-map design, what gets protected is defined by a crypto ACL (also called the proxy ID, or the traffic selector). It is not a filter that permits or denies. A permit line means "encrypt this", and each line becomes its own negotiated pair of SAs. Our lab ACL has two lines:
ip access-list extended PLZ-CRYPTO-ACL
permit ip 192.168.99.0 0.0.0.255 10.30.10.0 0.0.0.255
permit ip 10.20.10.0 0.0.0.255 10.30.10.0 0.0.0.255And the session detail proves the consequence:
EDGE1#show crypto session detail
Interface: GigabitEthernet2
Session status: UP-ACTIVE
Peer: 198.51.100.2 port 500 fvrf: (none) ivrf: (none)
IKEv1 SA: local 203.0.113.1/500 remote 198.51.100.2/500 Active
IPSEC FLOW: permit ip 10.20.10.0/255.255.255.0 10.30.10.0/255.255.255.0
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 10 drop 0 life (KB/Sec) 4607998/3516
Outbound: #pkts enc'ed 10 drop 0 life (KB/Sec) 4607999/3516
IPSEC FLOW: permit ip 192.168.99.0/255.255.255.0 10.30.10.0/255.255.255.0
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 7 drop 0 life (KB/Sec) 4607999/3562
Outbound: #pkts enc'ed 7 drop 0 life (KB/Sec) 4607999/3562One ISAKMP SA (Phase 1, shared). Two IPSEC FLOWs, one per ACL line, each showing Active SAs: 2 (inbound plus outbound). Two ACL lines produced four IPsec SAs and their own independent counters. This is also why proxy IDs must match exactly on both ends: your permit ip A B has to be the mirror image of the peer's permit ip B A, or Quick Mode is rejected. The full crypto-map build, including how to keep those ACLs symmetrical, is covered in site-to-site IPsec with crypto maps.
The Tunnel Is Built On Demand (and the First Packets Die)
With a crypto map there is no tunnel sitting there waiting. The SA is created only when a packet matches the crypto ACL, and building it takes time (Main Mode plus Quick Mode, six plus three messages, with two Diffie-Hellman computations). The packets that trigger it are consumed. Watch the very first ping on a freshly configured tunnel:
EDGE1#ping 10.30.10.1 source Loopback10 repeat 5
.....
Success rate is 0 percent (0/5)All five ICMP echoes were used up triggering IKE. Immediately after, with the SA now established, the same ping is perfect:
EDGE1#ping 10.30.10.1 source Loopback10 repeat 10
Sending 10, 100-byte ICMP Echos to 10.30.10.1, timeout is 2 seconds:
Packet sent with a source address of 10.20.10.1
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 10/21/48 msThis is not a fault. It is how on-demand IPsec works, and it is the reason a help desk should never diagnose a VPN on the first ping. Ping twice.
The Proof: The Original IP Header Really Does Vanish
Go back to the capture. Before protection was applied to this link, the transit router could read everything, including the inner host addresses:
No. Time Source Destination Proto Info
2 1.461513 192.168.99.100 10.30.10.1 ICMP Echo (ping) request id=0x004e, seq=1/256, ttl=63
3 1.462810 10.30.10.1 192.168.99.100 ICMP Echo (ping) reply id=0x004e, seq=1/256, ttl=255After tunnel-mode ESP, the same ping across the same link looks like this and nothing more:
12 7.026032 203.0.113.1 198.51.100.2 ESP ESP (SPI=0x6d4428aa)
13 8.939238 198.51.100.2 203.0.113.1 ESP ESP (SPI=0x94cd6141)No 192.168.99.100. No 10.30.10.1. No ICMP, no sequence numbers, no protocol identification at all. The only metadata left in the clear is the outer IP header (two public endpoints), the fact that this is IP protocol 50, and the SPI. That is tunnel mode doing its job, and it is worth noting what it does NOT hide: an observer still knows the two gateways are talking, roughly how much, and when. IPsec gives you confidentiality of content, not of the relationship. If you need the routing protocol and multicast to ride inside as well, that is the job of GRE over IPsec.
Why 3DES, MD5 and DH Group 2 Are Off the Table
You will still find 3DES, MD5 and DH group 2 in old configs and in half the tutorials online. Do not copy them. 3DES has a 64-bit block size (vulnerable to Sweet32 birthday-bound collisions on long-lived sessions) and it is slow. MD5 is broken for collision resistance. DH group 2 is a 1024-bit modulus, comfortably inside the reach of a well-funded precomputation attack. The lab config here uses the modern baseline you should default to: AES-256, SHA-256, and DH group 14 (2048-bit), with PFS on so each Phase 2 rekey gets fresh key material.
Key Takeaways
- IPsec is a framework. ESP protects packets, IKE negotiates the keys. Use ESP; AH cannot survive NAT because it signs the outer IP header.
- Tunnel mode encrypts the original IP header and builds a new one addressed to the crypto endpoints. Transport mode keeps the original header exposed and is what you use under GRE or an SVTI.
- IKE is two phases for a reason. Phase 1 (Main Mode, 6 messages) builds an authenticated channel; Phase 2 (Quick Mode, 3 messages) runs inside it and negotiates the keys that protect user data. Expensive once, cheap thereafter.
- QM_IDLE is healthy. It means Phase 1 is up and Quick Mode has nothing pending. It tells you nothing about whether user traffic is being encrypted.
- SAs are unidirectional. Inbound and outbound have different SPIs and different keys. Always read both counter sets:
encaps/encrypt/digestout,decaps/decrypt/verifyin. - One crypto ACL line equals one SA pair. Two permit lines gave us two IPSEC FLOWs and four SAs. Proxy IDs must mirror exactly on both peers or Quick Mode is rejected.
- Crypto-map tunnels are built on demand. The first packets that match the ACL are consumed triggering IKE. A 0/5 first ping followed by 10/10 is normal, not a fault.
- The wire tells the truth. After tunnel-mode ESP, the transit provider sees two public IPs, protocol 50, and an SPI. Nothing else.
Next in the IPsec VPN cluster: build it for real with site-to-site IPsec crypto maps, carry routing protocols and multicast inside the tunnel with GRE over IPsec, and learn to read the failures fast with the IPsec VPN troubleshooting guide.
