Conntrack and SNAT: Fix a Stale NAT Entry Without a Flush

An existing connection keeps working after its SNAT rule is removed. A new connection fails. You put the rule back, and one retry still fails while a fresh source port works. Looking only at the current NAT rules cannot explain all three results: the router also has connection state created under earlier rules.

This lab follows the packets and conntrack tuples through those changes. It keeps one TCP session open, removes a single SNAT rule, tests a new flow, restores the rule, and repairs only the failed entry. The original working session remains intact throughout.

Keep the service, change the return-path requirement

Start with the three namespaces, echo-service.py, and plf_filter table from the nftables trace lab. Remove its deliberate deny and temporary tracing table first. The routed baseline must return an ACK from the server on TCP port 8080.

• Client: 192.0.2.10:41001 on plf-client.
• Router: 192.0.2.1 on lan0, 198.51.100.1 on wan0, in plf-router.
• Server: 198.51.100.10:8080 on plf-server.

Tested with: Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, nftables 1.1.6, conntrack-tools 1.4.9, iproute2 6.19.0, tcpdump 4.99.6, Python 3.14.4. Install the conntrack command if it is missing:

sudo apt-get install conntrack

Inside the router, retain an explicit stateful filter rule independently of the NAT rule:

sudo ip netns exec plf-router nft insert rule inet plf_filter services \
  ct state established,related counter accept comment '"lab-stateful-allow"'

This rule does not replace the existing port allow. It accepts established/related traffic in this base chain; the first SYN still reaches the port rule, and another base chain could still deny either packet. In this test it also keeps conntrack processing active when the SNAT rule is removed.

Now remove only the server's default route:

sudo ip -n plf-server route del default
sudo ip -n plf-server route show

The server retains its connected 198.51.100.0/24 route. It can reply to 198.51.100.1 directly, but it has no route to the client's 192.0.2.10. SNAT will make the routed request appear to come from the router's server-facing address. This deliberate route condition makes missing translation observable.

Save nat.nft:

table ip plf_nat {
    chain post {
        type nat hook postrouting priority 100; policy accept;
        oifname "wan0" ip saddr 192.0.2.10 ip daddr 198.51.100.10 tcp dport 8080 counter snat to 198.51.100.1 comment "lab-service-snat"
    }
}

sudo ip netns exec plf-router nft -c -f nat.nft
sudo ip netns exec plf-router nft -f nat.nft

The ip family keeps this IPv4 example explicit. The rule matches the server-bound flow leaving wan0 and sets its source address to 198.51.100.1. There was no source-port collision in this run, so the source port stayed the same; do not assume NAT always preserves it.

Record conntrack events and both packet views

In a monitoring terminal, start the filtered event stream before connecting:

sudo ip netns exec plf-router conntrack -E -p tcp \
  --orig-src 192.0.2.10 --orig-dst 198.51.100.10 \
  --dport 8080 -o timestamp

-E watches events from now onward. Use -L to inspect existing state; starting an event monitor after a connection is established does not recreate its earlier events. Both operations run in the router namespace, where the translation is applied.

In separate capture terminals:

sudo ip netns exec plf-router tcpdump --immediate-mode -nni lan0 'tcp port 8080'

sudo ip netns exec plf-router tcpdump --immediate-mode -nni wan0 'tcp port 8080'

Keep the echo server running inside plf-server. In a client terminal, open the interactive mode from the trace lab's program:

sudo ip netns exec plf-client python3 -u echo-service.py hold --source-port 41001

After the connected message, type before-rule-removal and press Enter. The measured response was ACK before-rule-removal. Leave this process open; later text entered into this same terminal uses the same TCP connection.

The event stream showed NEW with SYN_SENT, then updates through SYN_RECV to ESTABLISHED and [ASSURED]. A filtered state lookup captured the entry:

sudo ip netns exec plf-router conntrack -L -p tcp \
  --orig-src 192.0.2.10 --orig-dst 198.51.100.10 \
  --sport 41001 --dport 8080

This is the measured tuple excerpt, with the timeout and unrelated fields omitted:

ESTABLISHED
src=192.0.2.10 dst=198.51.100.10 sport=41001 dport=8080
src=198.51.100.10 dst=198.51.100.1 sport=8080 dport=41001 [ASSURED]

The first address/port group is the original direction before translation: client to server. The second describes the expected reply direction: server to the translated address. Reverse that reply group to understand the forward packet on the server-facing link.

The captures confirmed it. On lan0, the request had source 192.0.2.10:41001; on wan0, source 198.51.100.1:41001. The response was addressed to 198.51.100.1:41001 on wan0 and to 192.0.2.10:41001 on lan0. Matching payload text and TCP sequence ranges tied the two views to the same exchange. Translation changed the address, not the message.

Remove the rule while keeping the connection

List the current SNAT rule with its handle:

sudo ip netns exec plf-router nft -a list chain ip plf_nat post

For the measured run, lab-service-snat had handle 2. Delete the handle from your own listing, leaving the table and chain:

sudo ip netns exec plf-router nft delete rule ip plf_nat post handle 2

In the still-open client terminal, type after-rule-removal. It returned ACK after-rule-removal. The conntrack lookup still showed the same original and reply tuples, and wan0 still showed source 198.51.100.1 for this session.

The nftables NAT guide explains why: the initial packet establishes a NAT binding, and later packets use that binding. They do not search the current NAT rules again for every packet. Removing a rule therefore does not, by itself, rewrite an existing flow's binding.

There is a useful prerequisite behind this result. An initial minimal run had no separate ct rule. In that ruleset the SNAT rule was the only thing holding the namespace's IPv4 conntrack hooks registered: nftables nat and ct expressions each take a reference, and the kernel unregisters the hooks when the last one is removed (see nf_conntrack_proto.c). Deleting that rule left the old entry listed, but new packets were no longer tracked or translated; nft list hooks showed the conntrack hooks disappear. The explicit stateful rule above keeps them registered, and the repeat produced the persistent translation shown here.

Compare a fresh flow and repair the rule

From another client terminal, open a new tuple:

sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 41002 --message new-without-nat

It timed out after the program's two-second timeout. The server-facing capture showed the new SYN with untranslated source 192.0.2.10, and the server had no route for its reply. The selected conntrack entry showed:

SYN_SENT
src=192.0.2.10 dst=198.51.100.10 sport=41002 dport=8080 [UNREPLIED]
src=198.51.100.10 dst=192.0.2.10 sport=8080 dport=41002

[UNREPLIED] means conntrack has not seen a matching reply. It does not identify the cause by itself. Here, the untranslated SYN, server route table, and missing reply provide the cause; in another network, a downstream drop could produce the same state.

Restore the SNAT rule, then use a fresh source port:

sudo ip netns exec plf-router nft add rule ip plf_nat post \
  oifname '"wan0"' ip saddr 192.0.2.10 ip daddr 198.51.100.10 \
  tcp dport 8080 counter snat to 198.51.100.1 comment '"lab-service-snat"'
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 41003 --message new-after-repair

This returned ACK new-after-repair. Its reply tuple used 198.51.100.1 again. The original held connection also continued to work. That checks two different things: the retained old binding and the rule applied to a newly tracked flow.

Why the same failed tuple still times out

Retry source port 41002 while its failed entry is still present:

sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 41002 --message same-tuple-after-rule-repair

The measured retry still timed out. Its SYN_SENT [UNREPLIED] entry retained the untranslated reply destination 192.0.2.10. This flow had been tracked while no SNAT rule matched, and that earlier no-translation binding continued to govern the tuple. A new socket process did not necessarily create a new tracked flow: the addresses, ports, and protocol matched an existing entry.

For this isolated failed test flow, remove exactly that entry:

sudo ip netns exec plf-router conntrack -D -p tcp \
  --orig-src 192.0.2.10 --orig-dst 198.51.100.10 \
  --sport 41002 --dport 8080
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 41002 --message same-tuple-after-state-delete

The delete reported one entry. The next attempt received ACK same-tuple-after-state-delete, and the tuple now showed the translated reply destination. In the original held terminal, old-flow-still-working also received its ACK. No global conntrack flush was needed.

Deleting state can interrupt a live connection, so first verify the exact original tuple and that it belongs to the failed test. A broad flush would discard unrelated flows and would obscure whether this repair was actually selective. The conntrack manual documents the listing, event, and tuple-filtered delete operations used here.

Conntrack's ESTABLISHED and [ASSURED] flags describe transport tracking; the echoed line is the separate evidence that the application exchanged data. For a connection that still fails after the tuple is correct, follow the route and listener diagnosis. To start the route/listener walkthrough in routed mode, restore the server's default route and remove only the lab NAT table:

sudo ip -n plf-server route add default via 198.51.100.1
sudo ip netns exec plf-router nft delete table ip plf_nat