> ## Content Index
> Fetch the complete content index at: https://www.pinglabz.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# nft monitor trace: Find Which nftables Rule Drops a Packet
- URL: https://www.pinglabz.com/nftables-monitor-trace-packet-drop/
- Published: 2026-10-13T06:25:19.000Z
- Updated: 2026-10-13T06:25:19.000Z
- Description: Follow one SYN through an allow and a later deny. Use a scoped nft trace, rule handles, and captures on both sides of an isolated Linux router.
- Author: Jaime
- Tags: Linux, Troubleshooting, Firewall, Netfilter, Linux Labs

The allow rule's counter increases, but the client still times out. That can happen when the packet passes one nftables base chain and a later chain drops it. The first `accept` tells you where evaluation stopped in that chain. It does not tell you that the packet reached the server.

This lab reproduces that exact case. A TCP echo service works through a Linux router, a second forward chain blocks it, and `nft monitor trace` follows the same SYN through both decisions. Captures on the router's two interfaces check where the packet disappears. Removing the single deny rule restores the exchange.

## Build a small routed path

The lab uses three network namespaces on one Linux VM:

- `plf-client`: `eth0`, `192.0.2.10/24`, gateway `192.0.2.1`.
- `plf-router`: `lan0`, `192.0.2.1/24`; `wan0`, `198.51.100.1/24`; IPv4 forwarding enabled inside this namespace.
- `plf-server`: `eth0`, `198.51.100.10/24`, gateway `198.51.100.1`; TCP service on port `8080`.

There is no NAT in this first investigation. Each host has a return path through the router. These are documentation addresses on isolated veth links; the namespaces have no uplink to the VM's management network. The [namespace and bridge lab](https://www.pinglabz.com/linux-namespace-network-lab/) explains the underlying network stacks. This smaller topology gives the firewall just one routed flow to handle.

Tested with: Ubuntu 26.04.1 LTS, kernel `7.0.0-38-generic`, iproute2 `6.19.0`, nftables `1.1.6`, tcpdump `4.99.6`, Python `3.14.4`. On Ubuntu, install the reader tools if needed:

```bash
sudo apt-get install iproute2 iputils-ping nftables tcpdump python3
```

Save the following as `setup.sh` on the Linux VM. It refuses existing names so a previous lab is not silently replaced:

```bash
#!/usr/bin/env bash
set -euo pipefail
for ns in plf-client plf-router plf-server; do
  if ip netns list | awk '{print $1}' | grep -qx "$ns"; then
    echo "Refusing to replace existing namespace: $ns" >&2
    exit 1
  fi
done
for dev in plfc0 plfr0 plfw0 plfs0; do
  if ip link show dev "$dev" >/dev/null 2>&1; then
    echo "Refusing to replace existing interface: $dev" >&2
    exit 1
  fi
done
for ns in plf-client plf-router plf-server; do
  ip netns add "$ns"
  ip -n "$ns" link set lo up
done
ip link add plfc0 type veth peer name plfr0
ip link set plfc0 netns plf-client
ip link set plfr0 netns plf-router
ip -n plf-client link set plfc0 name eth0
ip -n plf-router link set plfr0 name lan0
ip link add plfw0 type veth peer name plfs0
ip link set plfw0 netns plf-router
ip link set plfs0 netns plf-server
ip -n plf-router link set plfw0 name wan0
ip -n plf-server link set plfs0 name eth0
ip -n plf-client addr add 192.0.2.10/24 dev eth0
ip -n plf-router addr add 192.0.2.1/24 dev lan0
ip -n plf-router addr add 198.51.100.1/24 dev wan0
ip -n plf-server addr add 198.51.100.10/24 dev eth0
ip -n plf-client link set eth0 up
ip -n plf-router link set lan0 up
ip -n plf-router link set wan0 up
ip -n plf-server link set eth0 up
ip -n plf-client route add default via 192.0.2.1
ip -n plf-server route add default via 198.51.100.1
ip netns exec plf-router sysctl -w net.ipv4.ip_forward=1
echo 'Created isolated plf-client, plf-router and plf-server; source files preserved.'
```

Run it once and inspect the resulting routes:

```bash
sudo bash setup.sh
sudo ip -n plf-client route get 198.51.100.10
sudo ip -n plf-server route get 192.0.2.10
```

The first lookup should choose `192.0.2.1` through the client's `eth0`; the second should choose `198.51.100.1` through the server's `eth0`. If either lookup fails, repair that route before diagnosing a firewall decision. The [ip route reference](https://www.pinglabz.com/ip-route-linux/) covers these lookups.

## Start a service that proves an application exchange

A completed TCP handshake is useful, but this lab also checks that the application reads a line and answers it. Save this small test program as `echo-service.py` in your working directory:

```python
#!/usr/bin/env python3
"""A local lab protocol: echo one line as 'ACK <line>'. No external traffic."""
import argparse, json, socket, sys, threading
p = argparse.ArgumentParser()
p.add_argument('mode', choices=['server', 'client', 'hold'])
p.add_argument('--bind', default='198.51.100.10')
p.add_argument('--port', type=int, default=8080)
p.add_argument('--source-port', type=int, default=40010)
p.add_argument('--message', default='baseline')
a = p.parse_args()

def output(**values):
    print(json.dumps(values), flush=True)
if a.mode == 'server':
    s = socket.socket()
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind((a.bind, a.port))
    s.listen()
    output(event='listening', address=s.getsockname())

    def handle(c, peer):
        with c:
            f = c.makefile('rb')
            for line in f:
                output(event='request', peer=peer, message=line.decode().rstrip())
                c.sendall(b'ACK ' + line)
    while True:
        c, peer = s.accept()
        threading.Thread(target=handle, args=(c, peer), daemon=True).start()
else:
    try:
        with socket.socket() as s:
            s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
            s.settimeout(2)
            s.bind(('192.0.2.10', a.source_port))
            s.connect(('198.51.100.10', a.port))
            output(event='connected', local=s.getsockname(), remote=s.getpeername())
            f = s.makefile('rb')
            lines = sys.stdin if a.mode == 'hold' else [a.message + '\n']
            for line in lines:
                s.sendall(line.encode())
                answer = f.readline()
                output(event='reply', message=answer.decode().rstrip())
    except OSError as e:
        output(event='error', type=type(e).__name__, errno=e.errno, message=str(e))
        sys.exit(1)
```

In a server terminal, start it inside the server namespace and leave that terminal open:

```bash
sudo ip netns exec plf-server python3 -u echo-service.py server
```

In another terminal, check the listener and make the baseline request:

```bash
sudo ip netns exec plf-server ss -lntp 'sport = :8080'
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40010 --message baseline
```

The measured client connected from `192.0.2.10:40010` and received `ACK baseline`. The server logged the same client address and message. That checks routing in both directions, the listening socket, and an actual request/response. Keep that successful result; a later timeout only becomes informative when you know what changed.

## Create two forward chains

Save this as `filter.nft`:

```nft
table inet plf_filter {
    chain services {
        type filter hook forward priority 0; policy accept;
        ip saddr 192.0.2.10 ip daddr 198.51.100.10 tcp dport 8080 counter accept comment "lab-service-allow"
    }
    chain later {
        type filter hook forward priority 10; policy accept;
    }
}
```

Check and load it only inside the router namespace:

```bash
sudo ip netns exec plf-router nft -c -f filter.nft
sudo ip netns exec plf-router nft -f filter.nft
sudo ip netns exec plf-router nft -a -y list table inet plf_filter
```

Both chains use the `forward` hook because the destination is another host. A service on the router itself would follow `input` instead. Lower priority numbers run first, so `services` at `0` precedes `later` at `10`. The empty later chain currently accepts by policy, and the baseline exchange still works.

Add the fault to the later chain:

```bash
sudo ip netns exec plf-router nft add rule inet plf_filter later \
  ip saddr 192.0.2.10 ip daddr 198.51.100.10 tcp dport 8080 \
  counter drop comment '"lab-late-drop"'
```

This is a silent drop, not a reject. It matches only client-to-server TCP traffic on port `8080` in the isolated router. The [nftables guide](https://www.pinglabz.com/nftables-linux/) explains tables, handles, and rule syntax; here the ordering of two base chains is the part under test.

## Enable nft monitor trace for this flow

`nft monitor trace` only receives trace events for packets whose `nftrace` flag has been set. Starting the monitor alone does not instrument the flow. Save this as `trace.nft`:

```nft
table inet plf_debug {
    chain selected {
        type filter hook prerouting priority -301; policy accept;
        iifname "lan0" ip saddr 192.0.2.10 ip daddr 198.51.100.10 tcp dport 8080 meta nftrace set 1
    }
}
```

The early prerouting chain marks only packets arriving on `lan0` for this destination and port. It lets them continue to routing and forwarding. It does not trace the reply direction, ARP, or unrelated traffic.

```bash
sudo ip netns exec plf-router nft -c -f trace.nft
sudo ip netns exec plf-router nft -f trace.nft
sudo ip netns exec plf-router nft -a monitor trace
```

Leave that monitor running. In two more terminals, capture on either side of the router:

```bash
# Router ingress from the client
sudo ip netns exec plf-router tcpdump --immediate-mode -nni lan0 'tcp port 8080'
```

```bash
# Router egress toward the server
sudo ip netns exec plf-router tcpdump --immediate-mode -nni wan0 'tcp port 8080'
```

Start both captures before the next request. These points bracket forwarding; the ingress capture can see a packet before the forward filter drops it. The [tcpdump reference](https://www.pinglabz.com/tcpdump-for-network-engineers/) covers saving the same observations to a PCAP.

## Follow one SYN through accept and drop

Use a new source port so this attempt is easy to identify:

```bash
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40011 --message late-drop
```

The measured result was `TimeoutError: timed out`. The ingress capture recorded two SYN attempts from source port `40011`; the egress capture recorded none. Both captures reported zero kernel drops, and their captured counts matched the filter-received counts.

The trace supplied the reason. These are selected rule lines from one measured packet; packet header lines are omitted:

```text
trace id 6b50c914 inet plf_debug selected rule ... meta nftrace set 1 # handle 2 (verdict continue)
trace id 6b50c914 inet plf_filter services rule ... accept comment "lab-service-allow" # handle 3 (verdict accept)
trace id 6b50c914 inet plf_filter later rule ... drop comment "lab-late-drop" # handle 4 (verdict drop)
```

The shared trace ID ties these lines to one packet. A retransmitted SYN has its own trace ID, so do not combine unrelated lines just because their addresses match. Follow the hook, chain, and verdict for a single ID first.

The allow counter rose from `6` to `8` packets during the failed attempt, while the new drop rule counted `2` packets and `120` bytes. Those are the same two attempts at successive stages, not four separate lost packets. The earlier accept did its job in the `services` chain; it simply was not the last decision.

The [nft manual](https://www.netfilter.org/projects/nftables/manpage.html?ref=pinglabz.com) describes this behavior: an accept ends the current base chain, while a later base chain can still drop the packet. A drop ends further ruleset evaluation for that packet. The [tracing guide](https://wiki.nftables.org/wiki-nftables/index.php/Ruleset%5Fdebug/tracing?ref=pinglabz.com) documents enabling `nftrace` and monitoring the resulting events.

## Remove the deny and verify recovery

List the current rule handles:

```bash
sudo ip netns exec plf-router nft -a list chain inet plf_filter later
```

In the measured run, the deny rule was handle `4`. Use the handle attached to `lab-late-drop` in your own listing; handles can differ in another ruleset.

```bash
sudo ip netns exec plf-router nft delete rule inet plf_filter later handle 4
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40012 --message recovery
```

The client received `ACK recovery`. Both capture points recorded the complete ten-packet exchange, including its handshake, data, and close, and the trace no longer contained the late drop. Recovery checked the same application path rather than substituting a ping for a TCP result.

After the investigation, stop the monitors and foreground captures with Ctrl+C and remove only the temporary tracing table:

```bash
sudo ip netns exec plf-router nft delete table inet plf_debug
```

The routed lab and allow policy can stay in place for the [conntrack and SNAT investigation](https://www.pinglabz.com/conntrack-snat-state-lab/). If the trace accepts the flow but the application still fails, continue with [route, listener, and NAT checks](https://www.pinglabz.com/firewall-allows-tcp-connection-fails/). The Linux trace names the filter decision; the [Linux-to-Cisco drop lab](https://www.pinglabz.com/linux-cisco-packet-drop-lab/) uses captures and an IOS ACL counter to locate a similar interruption across a Cisco router.