> ## Content Index
> Fetch the complete content index at: https://www.pinglabz.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Linux Firewall Allows TCP but It Fails: Route, Listener, NAT
- URL: https://www.pinglabz.com/firewall-allows-tcp-connection-fails/
- Published: 2026-10-13T06:25:22.000Z
- Updated: 2026-10-13T06:25:22.000Z
- Description: Reproduce a missing route, stopped listener, loopback binding, and later firewall drop. Choose the next check from the packet’s last known point.
- Author: Jaime
- Tags: Troubleshooting, Linux, Firewall, Packet Analysis, Linux Labs

Opening TCP port `8080` does not create a listener, choose a client route, or repair the server's return path. An allow counter can increase during a failed connection because it describes one filtering decision along that path.

This walkthrough separates four failures in a small Linux lab: a missing client route, a stopped service, a service bound only to loopback, and a later firewall drop. A related NAT test adds a fifth pattern: the SYN reaches the server, but the reply tuple and return route are wrong. Each fault is introduced on a healthy path and repaired before the next one.

## Establish the path and the expected response

Use the `plf-client`, `plf-router`, and `plf-server` namespaces and `echo-service.py` from the [nftables trace lab](https://www.pinglabz.com/nftables-monitor-trace-packet-drop/). This article starts in its routed configuration, with no SNAT: client `192.0.2.10` reaches server `198.51.100.10:8080` through the router's `lan0` and `wan0`. Both hosts have a default route through their local router address, and IPv4 forwarding is enabled in `plf-router`.

Tested with: Ubuntu 26.04.1 LTS, kernel `7.0.0-38-generic`, iproute2 `6.19.0`, nftables `1.1.6`, tcpdump `4.99.6`, conntrack-tools `1.4.9`, Python `3.14.4`. All probes stayed on isolated namespace links. No Cisco image or physical network was needed for these cases.

Start the service in a server terminal:

```bash
sudo ip netns exec plf-server python3 -u echo-service.py server
```

In a diagnostic terminal, check the client's lookup and the server's listener, then request a response:

```bash
sudo ip -n plf-client route get 198.51.100.10
sudo ip netns exec plf-server ss -lntp 'sport = :8080'
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40100 --message healthy
```

The lookup chose `192.0.2.1` on `eth0`; the listener was on `198.51.100.10:8080`; the client received `ACK healthy`. Router-side captures on both interfaces contained the complete ten-packet exchange. That is the baseline these failures will be compared with.

For each test, start captures before the probe in two additional terminals:

```bash
sudo ip netns exec plf-router tcpdump --immediate-mode -nni lan0 'tcp port 8080'
```

```bash
sudo ip netns exec plf-router tcpdump --immediate-mode -nni wan0 'tcp port 8080'
```

Use a fresh source port for each case. It distinguishes the current attempt from a lingering session or retransmission. Stop each foreground capture with Ctrl+C after recording the result. The [tcpdump reference](https://www.pinglabz.com/tcpdump-for-network-engineers/) and [socket statistics guide](https://www.pinglabz.com/ss-command-linux/) cover the surrounding syntax.

## No SYN leaves: inspect the client route

Remove only the client's default route:

```bash
sudo ip -n plf-client route del default
sudo ip -n plf-client route get 198.51.100.10
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40101 --message missing-client-route
```

The lookup reported `Network is unreachable`. The client immediately returned `OSError`, errno `101`, with the same message. Neither router capture contained a TCP packet for the attempt.

That combination points upstream of the router's allow rule. The client cannot select a route to this destination. Looking for a forward-chain deny cannot explain an attempt that never reached that chain.

Restore the route and test the application again:

```bash
sudo ip -n plf-client route add default via 192.0.2.1
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40201 --message route-restored
```

The measured recovery returned `ACK route-restored`. The [Linux routing reference](https://www.pinglabz.com/ip-route-linux/) explains why a destination-specific lookup is more useful here than simply noting that the interface has an address.

An empty capture alone is weaker evidence. It can also mean the capture used the wrong interface, filter, or namespace, or started too late. Here, the exact failing route lookup and immediate local error support the diagnosis independently.

## SYN reaches the server, RST returns: check the listener

Stop the foreground echo server with Ctrl+C. Check its namespace and repeat the attempt:

```bash
sudo ip netns exec plf-server ss -lntp 'sport = :8080'
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40102 --message no-listener
```

`ss` showed no matching listener. Both router captures recorded one SYN toward the server and one RST/ACK back. The client reported `ConnectionRefusedError`, errno `111`.

This case did not silently lose the SYN. It reached the server-side network stack, which returned a reset because the destination socket was not listening. The capture on `wan0` recorded the reset about `59` microseconds after the SYN in this VM run; that timing is incidental, while the direction and flags explain the refusal.

A firewall reject rule or an intermediary can also send a TCP reset. In this controlled case, the empty listener listing, the deliberately stopped process, and the router captures establish the cause together.

Restart the server in its terminal, then retry:

```bash
sudo ip netns exec plf-server python3 -u echo-service.py server
```

```bash
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40202 --message listener-restored
```

The client received `ACK listener-restored`. There was no need to loosen the router's policy to fix a service that was not running.

## A listener exists, but only on loopback

Stop the server again. Start the same program with its bind address changed:

```bash
sudo ip netns exec plf-server python3 -u echo-service.py server --bind 127.0.0.1
```

Inspect the actual local address, not just the presence of port `8080`:

```bash
sudo ip netns exec plf-server ss -lntp 'sport = :8080'
```

The relevant measured line was:

```text
LISTEN 0 128 127.0.0.1:8080 0.0.0.0:*
```

The process is alive, but its listener belongs to the server namespace's loopback address. A packet addressed to `198.51.100.10:8080` does not match that socket.

```bash
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40103 --message loopback-listener
```

The result looked like the stopped-service case: `ConnectionRefusedError`, one SYN, and one returning RST/ACK at both router interfaces. The packet trace alone did not distinguish the two faults. `ss` did.

Stop the loopback instance, restart with the intended service address, and verify:

```bash
sudo ip netns exec plf-server python3 -u echo-service.py server --bind 198.51.100.10
```

```bash
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40203 --message binding-restored
```

The response was `ACK binding-restored`. Binding to a specific lab address is sufficient; opening the service on every address is not needed for this test. Always run `ss` where the application lives: a listener on the VM's main network stack is a different socket from one inside `plf-server`.

## SYN enters the router but never exits: trace the later decision

With the correctly bound service running, add the scoped deny to the router's later forward chain:

```bash
sudo ip netns exec plf-router nft add rule inet plf_filter later \
  ip saddr 192.0.2.10 ip daddr 198.51.100.10 tcp dport 8080 \
  counter drop comment '"lab-late-drop"'
```

Load the narrowly scoped `trace.nft` from the trace lab (this is the first case in this article that uses it), and start its monitor before probing:

```bash
sudo ip netns exec plf-router nft -f trace.nft
sudo ip netns exec plf-router nft -a monitor trace
```

```bash
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40104 --message later-drop
```

The client timed out. The new SYN attempts were visible on `lan0`, absent on `wan0`, and traced through `services` with an accept before `later` dropped them. The server still had the correct listener. This is the case where the firewall evidence identifies the interruption.

The small capture also included traffic from an earlier failed tuple. Filter by `tcp.port == 40104` when counting this attempt; the total counter or all packets on port `8080` can include other probes. That is why matching a trace ID and a tuple is stronger than reading a cumulative rule counter in isolation.

List the later chain and remove the handle attached to the deny. In this capstone run, it was `5`:

```bash
sudo ip netns exec plf-router nft -a list chain inet plf_filter later
sudo ip netns exec plf-router nft delete rule inet plf_filter later handle 5
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40105 --message recovered
```

The client received `ACK recovered`, and both interfaces again recorded the complete ten-packet exchange. Use your own current handle if it differs. After diagnosis, stop the monitor and remove the temporary `plf_debug` tracing table as described in the trace lab.

## SYN passes, but the reply path is wrong

The [conntrack/SNAT lab](https://www.pinglabz.com/conntrack-snat-state-lab/) produces a different timeout with the same client and service. There, the server has no route to `192.0.2.10`, and the router needs SNAT to make the request's source reachable on the server subnet.

When its SNAT rule was absent, the SYN appeared on both router interfaces. Conntrack recorded `SYN_SENT [UNREPLIED]` with reply destination `192.0.2.10`; the server lacked a route to that address. The packet had passed the forward allow rule, but the required return path had not been established.

Restoring the rule fixed a fresh tuple. A previously failed tuple retained its no-translation state until that selected entry was removed. The sibling lab shows the exact tuple filters, two capture views, and application recovery. Use its evidence before concluding that `[UNREPLIED]` means a firewall drop; the flag only says the tracker has not seen a matching reply.

## Choose the next check from the packet's last known point

The measured cases give a practical sequence:

- Immediate `Network is unreachable`, failing client route lookup, and no SYN at router ingress: repair the client lookup first.
- SYN crosses the router and a RST comes back: inspect the server listener address and any explicit reject rules. In this lab, `ss` separated no listener from loopback-only binding.
- SYN appears at router ingress but not egress: follow the selected nft trace through every relevant base chain and hook.
- SYN crosses the router without a reply: inspect the server's return route and the conntrack/NAT tuples on both sides.
- Handshake and request succeed but no useful application response arrives: continue at the application. An `ESTABLISHED` flag is not the response itself.

Keep the same destination, service, and bounded probe across the repair. The useful final result is an application response on the path you meant to restore.

Once TCP reaches a TLS listener, continue with [the OpenSSL SNI, hostname, and chain lab](https://www.pinglabz.com/openssl-sni-hostname-certificate-chain-lab/). If the server certificate verifies but the server rejects its client, use [the mutual TLS client-certificate lab](https://www.pinglabz.com/openssl-mtls-client-certificate-rejected/) to check the missing identity, issuer, or certificate purpose.

For a SYN visible on the server interface but missing at local input, follow [the reverse-path filtering investigation](https://www.pinglabz.com/linux-rp-filter-asymmetric-routing-lab/). When small IPv6 requests work but uploads stall, use [the Packet Too Big and cached-PMTU lab](https://www.pinglabz.com/ipv6-packet-too-big-tcp-stall-lab/) to check the feedback and sender packet size.