> ## Content Index
> Fetch the complete content index at: https://www.pinglabz.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Conntrack and SNAT: Fix a Stale NAT Entry Without a Flush
- URL: https://www.pinglabz.com/conntrack-snat-state-lab/
- Published: 2026-10-13T06:25:20.000Z
- Updated: 2026-10-13T06:25:20.000Z
- Description: Keep one session open while changing SNAT. Compare original and reply tuples, reproduce stale no-translation state, and repair one flow.
- Author: Jaime
- Tags: Linux, Troubleshooting, Netfilter, Packet Analysis, Linux Labs

An existing connection keeps working after its SNAT rule is removed. A new connection fails. You put the rule back, and one retry still fails while a fresh source port works. Looking only at the current NAT rules cannot explain all three results: the router also has connection state created under earlier rules.

This lab follows the packets and conntrack tuples through those changes. It keeps one TCP session open, removes a single SNAT rule, tests a new flow, restores the rule, and repairs only the failed entry. The original working session remains intact throughout.

## Keep the service, change the return-path requirement

Start with the three namespaces, `echo-service.py`, and `plf_filter` table from the [nftables trace lab](https://www.pinglabz.com/nftables-monitor-trace-packet-drop/). Remove its deliberate deny and temporary tracing table first. The routed baseline must return an `ACK` from the server on TCP port `8080`.

- Client: `192.0.2.10:41001` on `plf-client`.
- Router: `192.0.2.1` on `lan0`, `198.51.100.1` on `wan0`, in `plf-router`.
- Server: `198.51.100.10:8080` on `plf-server`.

Tested with: Ubuntu 26.04.1 LTS, kernel `7.0.0-38-generic`, nftables `1.1.6`, conntrack-tools `1.4.9`, iproute2 `6.19.0`, tcpdump `4.99.6`, Python `3.14.4`. Install the conntrack command if it is missing:

```bash
sudo apt-get install conntrack
```

Inside the router, retain an explicit stateful filter rule independently of the NAT rule:

```bash
sudo ip netns exec plf-router nft insert rule inet plf_filter services \
  ct state established,related counter accept comment '"lab-stateful-allow"'
```

This rule does not replace the existing port allow. It accepts established/related traffic in this base chain; the first SYN still reaches the port rule, and another base chain could still deny either packet. In this test it also keeps conntrack processing active when the SNAT rule is removed.

Now remove only the server's default route:

```bash
sudo ip -n plf-server route del default
sudo ip -n plf-server route show
```

The server retains its connected `198.51.100.0/24` route. It can reply to `198.51.100.1` directly, but it has no route to the client's `192.0.2.10`. SNAT will make the routed request appear to come from the router's server-facing address. This deliberate route condition makes missing translation observable.

Save `nat.nft`:

```nft
table ip plf_nat {
    chain post {
        type nat hook postrouting priority 100; policy accept;
        oifname "wan0" ip saddr 192.0.2.10 ip daddr 198.51.100.10 tcp dport 8080 counter snat to 198.51.100.1 comment "lab-service-snat"
    }
}
```

```bash
sudo ip netns exec plf-router nft -c -f nat.nft
sudo ip netns exec plf-router nft -f nat.nft
```

The `ip` family keeps this IPv4 example explicit. The rule matches the server-bound flow leaving `wan0` and sets its source address to `198.51.100.1`. There was no source-port collision in this run, so the source port stayed the same; do not assume NAT always preserves it.

## Record conntrack events and both packet views

In a monitoring terminal, start the filtered event stream before connecting:

```bash
sudo ip netns exec plf-router conntrack -E -p tcp \
  --orig-src 192.0.2.10 --orig-dst 198.51.100.10 \
  --dport 8080 -o timestamp
```

`-E` watches events from now onward. Use `-L` to inspect existing state; starting an event monitor after a connection is established does not recreate its earlier events. Both operations run in the router namespace, where the translation is applied.

In separate capture terminals:

```bash
sudo ip netns exec plf-router tcpdump --immediate-mode -nni lan0 'tcp port 8080'
```

```bash
sudo ip netns exec plf-router tcpdump --immediate-mode -nni wan0 'tcp port 8080'
```

Keep the echo server running inside `plf-server`. In a client terminal, open the interactive mode from the trace lab's program:

```bash
sudo ip netns exec plf-client python3 -u echo-service.py hold --source-port 41001
```

After the connected message, type `before-rule-removal` and press Enter. The measured response was `ACK before-rule-removal`. Leave this process open; later text entered into this same terminal uses the same TCP connection.

The event stream showed `NEW` with `SYN_SENT`, then updates through `SYN_RECV` to `ESTABLISHED` and `[ASSURED]`. A filtered state lookup captured the entry:

```bash
sudo ip netns exec plf-router conntrack -L -p tcp \
  --orig-src 192.0.2.10 --orig-dst 198.51.100.10 \
  --sport 41001 --dport 8080
```

This is the measured tuple excerpt, with the timeout and unrelated fields omitted:

```text
ESTABLISHED
src=192.0.2.10 dst=198.51.100.10 sport=41001 dport=8080
src=198.51.100.10 dst=198.51.100.1 sport=8080 dport=41001 [ASSURED]
```

The first address/port group is the original direction before translation: client to server. The second describes the expected reply direction: server to the translated address. Reverse that reply group to understand the forward packet on the server-facing link.

The captures confirmed it. On `lan0`, the request had source `192.0.2.10:41001`; on `wan0`, source `198.51.100.1:41001`. The response was addressed to `198.51.100.1:41001` on `wan0` and to `192.0.2.10:41001` on `lan0`. Matching payload text and TCP sequence ranges tied the two views to the same exchange. Translation changed the address, not the message.

## Remove the rule while keeping the connection

List the current SNAT rule with its handle:

```bash
sudo ip netns exec plf-router nft -a list chain ip plf_nat post
```

For the measured run, `lab-service-snat` had handle `2`. Delete the handle from your own listing, leaving the table and chain:

```bash
sudo ip netns exec plf-router nft delete rule ip plf_nat post handle 2
```

In the still-open client terminal, type `after-rule-removal`. It returned `ACK after-rule-removal`. The conntrack lookup still showed the same original and reply tuples, and `wan0` still showed source `198.51.100.1` for this session.

The [nftables NAT guide](https://wiki.nftables.org/wiki-nftables/index.php/Performing%5FNetwork%5FAddress%5FTranslation%5F%28NAT%29?ref=pinglabz.com) explains why: the initial packet establishes a NAT binding, and later packets use that binding. They do not search the current NAT rules again for every packet. Removing a rule therefore does not, by itself, rewrite an existing flow's binding.

There is a useful prerequisite behind this result. An initial minimal run had no separate `ct` rule. In that ruleset the SNAT rule was the only thing holding the namespace's IPv4 conntrack hooks registered: nftables `nat` and `ct` expressions each take a reference, and the kernel unregisters the hooks when the last one is removed (see [nf\_conntrack\_proto.c](https://github.com/torvalds/linux/blob/v7.0/net/netfilter/nf%5Fconntrack%5Fproto.c?ref=pinglabz.com)). Deleting that rule left the old entry listed, but new packets were no longer tracked or translated; `nft list hooks` showed the conntrack hooks disappear. The explicit stateful rule above keeps them registered, and the repeat produced the persistent translation shown here.

## Compare a fresh flow and repair the rule

From another client terminal, open a new tuple:

```bash
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 41002 --message new-without-nat
```

It timed out after the program's two-second timeout. The server-facing capture showed the new SYN with untranslated source `192.0.2.10`, and the server had no route for its reply. The selected conntrack entry showed:

```text
SYN_SENT
src=192.0.2.10 dst=198.51.100.10 sport=41002 dport=8080 [UNREPLIED]
src=198.51.100.10 dst=192.0.2.10 sport=8080 dport=41002
```

`[UNREPLIED]` means conntrack has not seen a matching reply. It does not identify the cause by itself. Here, the untranslated SYN, server route table, and missing reply provide the cause; in another network, a downstream drop could produce the same state.

Restore the SNAT rule, then use a fresh source port:

```bash
sudo ip netns exec plf-router nft add rule ip plf_nat post \
  oifname '"wan0"' ip saddr 192.0.2.10 ip daddr 198.51.100.10 \
  tcp dport 8080 counter snat to 198.51.100.1 comment '"lab-service-snat"'
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 41003 --message new-after-repair
```

This returned `ACK new-after-repair`. Its reply tuple used `198.51.100.1` again. The original held connection also continued to work. That checks two different things: the retained old binding and the rule applied to a newly tracked flow.

## Why the same failed tuple still times out

Retry source port `41002` while its failed entry is still present:

```bash
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 41002 --message same-tuple-after-rule-repair
```

The measured retry still timed out. Its `SYN_SENT [UNREPLIED]` entry retained the untranslated reply destination `192.0.2.10`. This flow had been tracked while no SNAT rule matched, and that earlier no-translation binding continued to govern the tuple. A new socket process did not necessarily create a new tracked flow: the addresses, ports, and protocol matched an existing entry.

For this isolated failed test flow, remove exactly that entry:

```bash
sudo ip netns exec plf-router conntrack -D -p tcp \
  --orig-src 192.0.2.10 --orig-dst 198.51.100.10 \
  --sport 41002 --dport 8080
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 41002 --message same-tuple-after-state-delete
```

The delete reported one entry. The next attempt received `ACK same-tuple-after-state-delete`, and the tuple now showed the translated reply destination. In the original held terminal, `old-flow-still-working` also received its ACK. No global conntrack flush was needed.

Deleting state can interrupt a live connection, so first verify the exact original tuple and that it belongs to the failed test. A broad flush would discard unrelated flows and would obscure whether this repair was actually selective. The [conntrack manual](https://netfilter.org/projects/conntrack-tools/conntrack-manpage.html?ref=pinglabz.com) documents the listing, event, and tuple-filtered delete operations used here.

Conntrack's `ESTABLISHED` and `[ASSURED]` flags describe transport tracking; the echoed line is the separate evidence that the application exchanged data. For a connection that still fails after the tuple is correct, follow the [route and listener diagnosis](https://www.pinglabz.com/firewall-allows-tcp-connection-fails/). To start the route/listener walkthrough in routed mode, restore the server's default route and remove only the lab NAT table:

```bash
sudo ip -n plf-server route add default via 198.51.100.1
sudo ip netns exec plf-router nft delete table ip plf_nat
```