GRE over IPsec MTU Black Hole: MSS 1360 vs a 1400-Byte Path

On a GRE over IPsec tunnel across a 1400-byte provider MTU, the commonly quoted ip mtu 1400 and ip tcp adjust-mss 1360 still stalled large downloads, because the path only carried 1302-byte inner packets. I built this in my home lab on IOL in Containerlab. The black hole appeared only with tunnel path-mtu-discovery on both peers, crypto ipsec df-bit set, and provider ICMP unreachables suppressed. The default-looking setup fragmented and worked. The fix was ip mtu 1302 and ip tcp adjust-mss 1262 on the tunnel.

Three small pings passed, and the small HTTP page loaded. A 2 MB file request received HTTP 200, then delivered zero body bytes until the 12-second timeout. It did that three times.

The default-looking configuration did not black-hole. ESP left with DF clear, the provider fragmented, and downloads completed. Applying crypto ipsec df-bit set alone also failed to reproduce the stall. The stall needed all three conditions at once, with the 1400-byte provider IP MTU applied in both directions.

The GRE tunnel MTU guide covers the theory. Here the useful result is the measured boundary: 1302 bytes of inner IPv4 passed; 1303 silently disappeared. The eventual repair used tunnel MTU 1302 and TCP MSS 1262.

On the west Linux host, the small checks returned:

PING 10.37.20.10 (10.37.20.10) 56(84) bytes of data.
64 bytes from 10.37.20.10: icmp_seq=1 ttl=62 time=4.53 ms
64 bytes from 10.37.20.10: icmp_seq=2 ttl=62 time=5.42 ms
64 bytes from 10.37.20.10: icmp_seq=3 ttl=62 time=6.15 ms

--- 10.37.20.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 4.530/5.365/6.145/0.660 ms
PL37-small-page

The large object at http://10.37.20.10/random.bin was exactly 2,097,152 bytes, or 2 MiB. From the west host, each request used a 12-second limit and recorded the HTTP code, body bytes, elapsed seconds and download speed. Its three failed downloads returned:

trial=1
code=200 bytes=0 seconds=12.002313 speed=0
trial=2
code=200 bytes=0 seconds=12.001430 speed=0
trial=3
code=200 bytes=0 seconds=12.001982 speed=0

An HTTP status alone would have marked this test successful. Counting the body bytes exposed the failure.

Topology, addressing and MTUs

The path was west host → west router → provider → east router → east host. Tunnel37 joined the edge routers across the provider; captures covered both LANs and both provider-facing links.

The three routers used vrnetlab/cisco_iol:17.18.02; the recorded IOL version was 17.18.2. Both Linux hosts used alpine:3.24.2. Host data interfaces were explicitly set to 1500 bytes so an unexpected virtual-link MTU would not become another variable.

Each host routed 10.37.0.0/16 through its LAN gateway. The edge routers used default routes toward the provider and static remote-LAN routes through Tunnel37. No routing protocol or NAT-T was involved; the WAN carried protocol 50 ESP.

One platform quirk mattered immediately: IOL rejected a physical-interface MTU change, but accepted the IP MTU setting. The provider therefore continued to display Ethernet MTU 1500 in its interface summary while its forwarding IP MTU was 1400. Read the IP-interface output when checking this bottleneck.

Baseline configuration and the attempts that passed

The west peer's recorded IKEv2 and IPsec configuration follows, with the pre-shared key redacted. It used AES-CBC-256, SHA256 integrity and group 19 for IKEv2, plus tunnel-mode ESP with AES-256 and SHA256-HMAC. No PFS was configured in the IPsec profile. The IKEv2 explanation covers those negotiation pieces separately.

crypto ikev2 proposal MTU37-IKE 
 encryption aes-cbc-256
 integrity sha256
 group 19
!
crypto ikev2 policy MTU37-POL 
 proposal MTU37-IKE
!
crypto ikev2 keyring MTU37-KEY
 peer MTU37-PEER
  address 198.18.37.6
  pre-shared-key local <redacted-psk>
  pre-shared-key remote <redacted-psk>
 !
!
!
crypto ikev2 profile MTU37-IKE-PROFILE
 match identity remote address 198.18.37.6 255.255.255.255 
 authentication remote pre-share
 authentication local pre-share
 keyring local MTU37-KEY
...
crypto ipsec transform-set MTU37-ESP esp-aes 256 esp-sha256-hmac 
 mode tunnel
crypto ipsec df-bit set
!
crypto ipsec profile MTU37-PROTECT
 set transform-set MTU37-ESP 
 set ikev2-profile MTU37-IKE-PROFILE

This excerpt is from the eventual fault state, so it includes the retained global DF setting. For the initial passing state, that setting and tunnel PMTUD were absent. The west data interfaces and routes in the fault state were:

interface Tunnel37
 ip address 10.37.255.1 255.255.255.252
 tunnel source Ethernet0/1
 tunnel destination 198.18.37.6
 tunnel path-mtu-discovery
 tunnel protection ipsec profile MTU37-PROTECT
!
...
interface Ethernet0/1
 ip address 198.18.37.1 255.255.255.252
!
interface Ethernet0/2
 ip address 10.37.10.1 255.255.255.0

ip route 0.0.0.0 0.0.0.0 198.18.37.2
ip route 10.37.20.0 255.255.255.0 Tunnel37

On the east router, the same crypto proposal, transform and profile were used, with remote peer and identity 198.18.37.1. Its LAN and WAN addresses are in the table; its tunnel and routes were:

interface Tunnel37
 ip address 10.37.255.2 255.255.255.252
 tunnel source Ethernet0/1
 tunnel destination 198.18.37.1
 tunnel path-mtu-discovery
 tunnel protection ipsec profile MTU37-PROTECT

ip route 0.0.0.0 0.0.0.0 198.18.37.5
ip route 10.37.10.0 255.255.255.0 Tunnel37

For the complete tunnel-building walkthrough, use GRE over IPsec. This test kept that routing and encryption arrangement fixed while changing MTU and DF behavior.

The provider's final fault configuration was:

interface Ethernet0/1
 ip address 198.18.37.2 255.255.255.252
 no ip unreachables
 ip mtu 1400
!
interface Ethernet0/2
 ip address 198.18.37.5 255.255.255.252
 no ip unreachables
 ip mtu 1400
!

Both data interfaces suppressed ICMP unreachables directly; there was no filtering ACL. The progression matters because simply lowering the provider MTU did not produce the intended failure. The initial configuration readback did not expose a named crypto DF default; DF-clear describes the observed packets.

In the symmetric test, one east-origin 1500-byte ESP datagram appeared downstream as IP fragments of 124 and 1396 bytes. The first had MF set; the second had fragment offset 13. DF was clear. Fragmentation was carrying the traffic successfully in this setup.

There were two setup problems before these comparisons: the HTTP server needed repair, and the first capture attempt failed because the data-link peers were in the node network namespaces. The preliminary transfer completed after the server repair, but that failed capture produced no packet evidence. The later successful captures were closed with zero reported kernel drops.

The fault: DF reaches the provider, feedback does not

There was no explicit tunnel IP MTU override or TCP MSS clamp in the fault state. The change that finally reproduced the black hole was enabling tunnel PMTUD on both edge tunnels. The west CLI recorded:

west-rtr(config)#interface Tunnel37
west-rtr(config-if)#tunnel path-mtu-discovery
west-rtr(config-if)#end

The global crypto ipsec df-bit set command was still present. Together with the two 1400-byte provider egress IP MTUs and suppressed provider unreachables, this produced the stalled downloads. This run did not isolate tunnel PMTUD with the global crypto DF setting removed, so it does not establish whether that global setting was necessary.

With tunnel PMTUD enabled, inner DF-set probes produced DF-set ESP packets. DF-clear traffic could still fragment and pass. That distinction explains how the same path supported some traffic during the failure.

The west IPsec SA remained active. Its counters included:

    #pkts encaps: 7499, #pkts encrypt: 7499, #pkts digest: 7499
    #pkts decaps: 12413, #pkts decrypt: 12413, #pkts verify: 12413
...
    #send errors 0, #recv errors 0

     local crypto endpt.: 198.18.37.1, remote crypto endpt.: 198.18.37.6
     plaintext mtu 1438, path mtu 1500, ip mtu 1500, ip mtu idb Ethernet0/1

These are cumulative counters, including earlier successful transfers. They establish that encryption and decryption had occurred; they do not prove delivery of the stalled file. The reported 1500-byte crypto path MTU also failed to reveal the hidden provider bottleneck. For a tunnel that cannot establish SAs at all, the IPsec troubleshooting workflow addresses that earlier stage.

During the stall, the provider's IP statistics showed:

  Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
         6199 fragmented, 12398 fragments, 17 couldn't fragment
         0 invalid hole

The 17 couldn't-fragment events supported the MTU diagnosis. The same snapshot showed 17 total output drops on the provider's west interface and zero on its east interface. These totals were not cleared between phases; neither interface's counter alone describes the entire test.

The WAN capture comparison located the loss more precisely. Of 56 oversized, DF-set ESP packets leaving the east peer, none of the matching SPI/sequence pairs appeared on the west link. A separate west-to-east sweep found 28 oversized DF-set packets at their origin and zero matching pairs at the east link. No ICMP type 3/code 4 appeared on either WAN capture in those phases.

All 56 east-origin oversized DF packets had outer IP length 1500 and frame length 1514. One unchanged capture row is below. Its columns begin with frame number, relative time, frame length, source, destination, IP length, DF, MF, fragment offset, protocol, SPI and sequence:

23,2.062055000,1514,198.18.37.6,198.18.37.1,1500,True,False,0,50,0x23b1400c,12446,,,,,,,

That absence concerns provider feedback on the WAN. There was a separate local ICMP error for a 1500-byte inner probe, described below. It would be wrong to call the entire test ICMP-free.

Find it: sweep DF sizes below the local ceiling

The encapsulation budget was inner IPv4 → GRE IPv4 plus GRE → tunnel-mode ESP → fresh outer IPv4. The provider limited that final IP packet to 1400 bytes. Comparing only the original packet with 1400 misses everything wrapped around it. For why GRE goes inside ESP here rather than the reverse, see GRE over IPsec vs IPsec over GRE.

The first large Linux probe used 1472 data bytes with DF set. It met a local 1414-byte tunnel IP limit and received an MTU error from 10.37.10.1. Allowing fragmentation at that size passed. Neither result measured the hidden provider ceiling. A local error can also populate the host's PMTU cache, so the recorded tests included route-cache flushes before relevant comparisons.

I then swept below 1414, using total IP sizes on IOS and ICMP data sizes on Linux. The ping DF-bit guide explains the 28-byte difference. Here, IOS size 1302 equals Linux data size 1274 plus 20 bytes of IPv4 and 8 of ICMP.

The router test crossed the boundary one byte at a time:

west-rtr#ping 10.37.20.10 source Ethernet0/2 size 1302 df-bit repeat 2 timeout 1
Type escape sequence to abort.
Sending 2, 1302-byte ICMP Echos to 10.37.20.10, timeout is 1 seconds:
Packet sent with a source address of 10.37.10.1 
Packet sent with the DF bit set
!!
Success rate is 100 percent (2/2), round-trip min/avg/max = 5/5/6 ms
west-rtr#ping 10.37.20.10 source Ethernet0/2 size 1303 df-bit repeat 2 timeout 1
Type escape sequence to abort.
Sending 2, 1303-byte ICMP Echos to 10.37.20.10, timeout is 1 seconds:
Packet sent with a source address of 10.37.10.1 
Packet sent with the DF bit set
..
Success rate is 0 percent (0/2)

On the west host, ping -c 2 -W 1 -M do -s 1274 10.37.20.10 tested the passing boundary. Changing the data size to 1275 tested the next byte. The recorded output was:

data-size=1274
PING 10.37.20.10 (10.37.20.10) 1274(1302) bytes of data.
1282 bytes from 10.37.20.10: icmp_seq=1 ttl=62 time=5.61 ms
1282 bytes from 10.37.20.10: icmp_seq=2 ttl=62 time=5.56 ms

--- 10.37.20.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 5.558/5.582/5.606/0.024 ms
data-size=1275
PING 10.37.20.10 (10.37.20.10) 1275(1303) bytes of data.

--- 10.37.20.10 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1019ms

The full sweep failed at inner sizes 1400, 1360, 1340, 1320 and 1310, passed at 1300, 1301 and 1302, then failed at 1303 and 1304. That brackets a usable 1302-byte inner IPv4 ceiling for these DF probes.

Measure the IPsec overhead from the frames

The LAN and WAN capture lengths supplied the packet budget below. Frame length includes the 14-byte Ethernet header, with no VLAN tag or FCS in these captures. The provider's IP MTU must be compared with outer IP length, not frame length.

For reference, these unchanged WAN rows span the transition. Their columns are frame number, relative time, frame length, source, destination, IP length, DF, MF, fragment offset, protocol, SPI and sequence, followed by unused fields:

45,28.933314000,1402,198.18.37.1,198.18.37.6,1388,True,False,0,50,0x41f40255,7549,,,,,,,
46,28.936829000,1402,198.18.37.6,198.18.37.1,1388,False,False,0,50,0x23b1400c,12529,,,,,,,
47,29.934658000,1402,198.18.37.1,198.18.37.6,1388,True,False,0,50,0x41f40255,7550,,,,,,,
48,29.937853000,1402,198.18.37.6,198.18.37.1,1388,False,False,0,50,0x23b1400c,12530,,,,,,,
49,29.943880000,1418,198.18.37.1,198.18.37.6,1404,True,False,0,50,0x41f40255,7551,,,,,,,
50,30.962722000,1418,198.18.37.1,198.18.37.6,1404,True,False,0,50,0x41f40255,7552,,,,,,,

For an inner IPv4 packet of N bytes, GRE contributes 24 bytes. The configured ESP stack adds an 8-byte header, 16-byte IV, padding, a 2-byte trailer, a 16-byte integrity field and a fresh 20-byte IPv4 header. That is 86 fixed bytes plus 0–15 bytes of padding. Transport mode would reuse the GRE IPv4 header instead of adding a fresh 20-byte outer header; see tunnel mode vs transport mode for GRE over IPsec. The SA reports the 16-byte IV; the frame series corroborates the integrity budget. ESP was not decrypted, so the padding values are derived from those lengths and the configured CBC transform.

The resulting IPsec overhead calculation is outer IP = 60 + 16 × ceil((N + 26) / 16). All 20 host-probe measurements in the sweep matched it.

At N=1302, GRE plus the inner packet occupies 1326 bytes; the trailer takes that to 1328, already aligned to 16 bytes. The outer IP packet is 1388. At N=1303, 1329 bytes must round up to 1344, giving an outer packet of 1404. One extra inner byte required another cipher block.

The 1400-byte provider MTU leaves room for 83 blocks after the fixed outer fields: floor((1400−60)/16)=83. Those 1328 encrypted bytes leave 1302 for the original IPv4 packet after GRE and the trailer. The unused 12 underlay bytes cannot fit another 16-byte block.

Why tunnel IP MTU 1400 and MSS 1360 still failed

I tested both commonly quoted settings on both peers. The file request still returned:

code=200 bytes=0 seconds=6.001599

An inner packet of 1400 bytes became 1500 outside in this capture. That cannot cross a 1400-byte provider link with DF set. Likewise, TCP MSS 1360 permits a base IPv4/TCP packet of 1400 before GRE and ESP. Neither value matched this measured path.

Fix it: MTU 1302, MSS 1262, then check the wire

The repair used the measured inner ceiling and subtracted the base IPv4/TCP headers for MSS: 1302−20−20=1262. TCP options are accounted for within the packet budget; they are not subtracted from the advertised MSS a second time. The following change was applied to Tunnel37 on both peers:

west-rtr(config)#interface Tunnel37
west-rtr(config-if)#ip mtu 1302
west-rtr(config-if)#ip tcp adjust-mss 1262
west-rtr(config-if)#end

The saved post-fix configs still carried tunnel path-mtu-discovery alongside ip mtu 1302 on both tunnels. Cisco's GRE and IPsec fragmentation guide recommends using both commands together: ip mtu leaves room for the GRE and IPsec overhead, and tunnel PMTUD can lower the tunnel MTU further only when ICMP comes back from a smaller link. This provider suppressed those unreachables, so the static value had to match the measured ceiling.

New TCP handshakes showed the rewrite on the LAN captures, where the header remained visible. In the combined-fix test, the west LAN recorded:

7,2.014358000,74,10.37.10.10,10.37.20.10,60,True,False,0,6,,,,,,0x0002,1460,0,
8,2.017794000,74,10.37.20.10,10.37.10.10,60,True,False,0,6,,,,,,0x0012,1262,0,

The east LAN recorded the corresponding handshake:

7,2.013441000,74,10.37.10.10,10.37.20.10,60,True,False,0,6,,,,,,0x0002,1262,0,
8,2.013496000,74,10.37.20.10,10.37.10.10,60,True,False,0,6,,,,,,0x0012,1262,0,

The west SYN's MSS changed from 1460 to 1262. Its TCP flags are 0x0002; the SYN-ACK uses 0x0012. Four captured handshakes showed that west-origin rewrite. The server already advertised 1262 on its own LAN in this phase, so these rows alone do not demonstrate a reverse-direction rewrite.

All three complete downloads then passed:

trial=1
code=200 bytes=2097152 seconds=0.944552 speed=2220787
trial=2
code=200 bytes=2097152 seconds=0.986193 speed=2126836
trial=3
code=200 bytes=2097152 seconds=0.905304 speed=2316921

The downloaded file's SHA-256 matched the server's: 726a646e9ea363585026c14a3d8be22f06f5da95409ac193f601431a148a8c56. These are home-lab transfer timings, not throughput benchmarks.

Both fixed WAN captures contained 8062 ESP packets, with maximum outer IP length 1388 and Ethernet frame length 1402. Neither contained oversized ESP or outer IP fragments. The repeated Linux boundary test now showed:

PING 10.37.20.10 (10.37.20.10) 1274(1302) bytes of data.
1282 bytes from 10.37.20.10: icmp_seq=1 ttl=62 time=5.01 ms
1282 bytes from 10.37.20.10: icmp_seq=2 ttl=62 time=5.66 ms

--- 10.37.20.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 5.014/5.337/5.661/0.323 ms
PING 10.37.20.10 (10.37.20.10) 1275(1303) bytes of data.
From 10.37.10.1 icmp_seq=1 Frag needed and DF set (mtu = 1302)

--- 10.37.20.10 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1002ms

The usable size stayed 1302. The next byte now elicited an explicit error from the local tunnel ingress instead of disappearing at the provider. On this IOL image, router-originated oversized DF pings still printed dots; the Linux error and packet captures supplied the distinction.

Before IPsec became active, the tunnel IP/transport MTU was 1476; it changed automatically to 1414 when protection became active. The large interface maximum changed from 17916 to 17854. After repair, the tunnel's IP-interface output confirmed 1302, while its transport-MTU line remained 1414 and the SA still reported path MTU 1500. Those displays describe different limits. The measured forwarding result is what validated the repair.

What MSS alone and IP MTU alone actually did

With the explicit tunnel MTU removed and MSS 1262 retained on both peers, TCP passed in 0.860541 seconds. The oversized non-TCP DF ping still silently failed:

code=200 bytes=2097152 seconds=0.860541
PING 10.37.20.10 (10.37.20.10) 1275(1303) bytes of data.

--- 10.37.20.10 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1020ms

This separate test captured both endpoints originating MSS 1460. On the west LAN:

1,0.000000000,74,10.37.10.10,10.37.20.10,60,True,False,0,6,,,,,,0x0002,1460,0,
2,0.005154000,74,10.37.20.10,10.37.10.10,60,True,False,0,6,,,,,,0x0012,1262,0,

On the east LAN:

1,0.000000000,74,10.37.10.10,10.37.20.10,60,True,False,0,6,,,,,,0x0002,1262,0,
2,0.000184000,74,10.37.20.10,10.37.10.10,60,True,False,0,6,,,,,,0x0012,1460,0,

The west SYN reached the east LAN with 1262, and the east SYN-ACK reached the west LAN with 1262. That proves clamping in both directions. MSS solved the tested TCP transfer; it did not change the 1303-byte ICMP packet's size.

With tunnel MTU 1302 retained and MSS clamping removed from both peers, the transfer also passed:

code=200 bytes=2097152 seconds=0.772531
PING 10.37.20.10 (10.37.20.10) 1275(1303) bytes of data.
From 10.37.10.1 icmp_seq=1 Frag needed and DF set (mtu = 1302)

There is a qualification: the server's SYN-ACK already advertised MSS 1262 before crossing the tunnel, and endpoint TCP metrics had not been flushed. The west SYN remained 1460 on both LANs. This result proves a passing transfer in that state and local MTU feedback for oversized DF ICMP. It does not prove fresh TCP PMTUD recovery. I restored both measured settings after these comparisons.

MTU checks I run on every tunnel

• Compare a small request with a complete large transfer, recording body bytes and elapsed time.
• Sweep DF sizes around the pass/fail boundary; distinguish local MTU feedback from silent transit loss.
• Check outer IP lengths, DF and fragments on both WAN sides, and MSS on both LAN sides.
• Retest after changing one setting, accounting for cached endpoint state.

The tracepath PMTUD lab covers another way to investigate missing MTU feedback. No tracepath result is claimed for this GRE/IPsec run.

Verification checklist

• Confirm both tunnel interfaces, remote-LAN routes and active IPsec SAs before interpreting size-dependent loss.
• Verify the provider IP MTU and unreachable policy, plus the two tunnel PMTUD settings used for this reproduction.
• Reproduce 1302 passing and 1303 failing with DF before repair; distinguish Linux data size from IOS total size.
• Confirm MSS 1262 in new handshakes and complete 2,097,152-byte downloads after repair.
• Repeat the boundary test: 1274 Linux data bytes pass, while 1275 produces local MTU 1302 feedback with the explicit tunnel MTU.
• Check outer IP lengths and fragments alongside counters. The GRE troubleshooting guide covers broader tunnel faults.

Related tunnel labs

• GRE tunnel MTU — Review the encapsulation budget and MTU concepts.
• GRE over IPsec — Build the routed, encrypted tunnel.
• Ping MTU and the DF bit — Compare Linux payload sizes with IOS packet sizes.
• tracepath and PMTUD — Follow path MTU discovery and missing feedback.
• IPsec VPN troubleshooting — Check negotiation and protected traffic.
• GRE tunnel troubleshooting — Investigate tunnel state and forwarding faults.
• GRE vs IPsec vs GRE over IPsec — Pick the right tunnel type before tuning its MTU.

FAQ

Does default GRE over IPsec create this MTU black hole?

It did not in this run. Outer ESP DF stayed clear and provider fragmentation allowed downloads. The successful reproduction added tunnel PMTUD on both peers while retaining the global crypto DF setting, with provider IP MTU 1400 and unreachables suppressed. The crypto DF setting alone did not reproduce it.

Why did ip tcp adjust-mss 1360 fail?

It allowed inner TCP packets larger than this path's measured 1302-byte ceiling. The combined 1400/1360 test stalled with zero body bytes. MSS 1262 matched the measured IPv4 budget and passed the TCP transfer.

Why does one byte change outer length from 1388 to 1404?

AES-CBC alignment. Inner size 1302 fit the current encrypted block budget with zero padding; 1303 required 15 padding bytes and another 16-byte block. That step crossed the 1400-byte provider IP MTU.

Is an MSS clamp enough for all traffic?

The MSS-only test repaired TCP and demonstrated both handshake rewrites. A 1303-byte DF ICMP probe still silently dropped. Adding the explicit tunnel IP MTU produced local MTU feedback for that oversized probe; it addressed behavior the TCP clamp could not change.