nft monitor trace: Find Which nftables Rule Drops a Packet

The allow rule's counter increases, but the client still times out. That can happen when the packet passes one nftables base chain and a later chain drops it. The first accept tells you where evaluation stopped in that chain. It does not tell you that the packet reached the server.

This lab reproduces that exact case. A TCP echo service works through a Linux router, a second forward chain blocks it, and nft monitor trace follows the same SYN through both decisions. Captures on the router's two interfaces check where the packet disappears. Removing the single deny rule restores the exchange.

Build a small routed path

The lab uses three network namespaces on one Linux VM:

• plf-client: eth0, 192.0.2.10/24, gateway 192.0.2.1.
• plf-router: lan0, 192.0.2.1/24; wan0, 198.51.100.1/24; IPv4 forwarding enabled inside this namespace.
• plf-server: eth0, 198.51.100.10/24, gateway 198.51.100.1; TCP service on port 8080.

There is no NAT in this first investigation. Each host has a return path through the router. These are documentation addresses on isolated veth links; the namespaces have no uplink to the VM's management network. The namespace and bridge lab explains the underlying network stacks. This smaller topology gives the firewall just one routed flow to handle.

Tested with: Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, iproute2 6.19.0, nftables 1.1.6, tcpdump 4.99.6, Python 3.14.4. On Ubuntu, install the reader tools if needed:

sudo apt-get install iproute2 iputils-ping nftables tcpdump python3

Save the following as setup.sh on the Linux VM. It refuses existing names so a previous lab is not silently replaced:

#!/usr/bin/env bash
set -euo pipefail
for ns in plf-client plf-router plf-server; do
  if ip netns list | awk '{print $1}' | grep -qx "$ns"; then
    echo "Refusing to replace existing namespace: $ns" >&2
    exit 1
  fi
done
for dev in plfc0 plfr0 plfw0 plfs0; do
  if ip link show dev "$dev" >/dev/null 2>&1; then
    echo "Refusing to replace existing interface: $dev" >&2
    exit 1
  fi
done
for ns in plf-client plf-router plf-server; do
  ip netns add "$ns"
  ip -n "$ns" link set lo up
done
ip link add plfc0 type veth peer name plfr0
ip link set plfc0 netns plf-client
ip link set plfr0 netns plf-router
ip -n plf-client link set plfc0 name eth0
ip -n plf-router link set plfr0 name lan0
ip link add plfw0 type veth peer name plfs0
ip link set plfw0 netns plf-router
ip link set plfs0 netns plf-server
ip -n plf-router link set plfw0 name wan0
ip -n plf-server link set plfs0 name eth0
ip -n plf-client addr add 192.0.2.10/24 dev eth0
ip -n plf-router addr add 192.0.2.1/24 dev lan0
ip -n plf-router addr add 198.51.100.1/24 dev wan0
ip -n plf-server addr add 198.51.100.10/24 dev eth0
ip -n plf-client link set eth0 up
ip -n plf-router link set lan0 up
ip -n plf-router link set wan0 up
ip -n plf-server link set eth0 up
ip -n plf-client route add default via 192.0.2.1
ip -n plf-server route add default via 198.51.100.1
ip netns exec plf-router sysctl -w net.ipv4.ip_forward=1
echo 'Created isolated plf-client, plf-router and plf-server; source files preserved.'

Run it once and inspect the resulting routes:

sudo bash setup.sh
sudo ip -n plf-client route get 198.51.100.10
sudo ip -n plf-server route get 192.0.2.10

The first lookup should choose 192.0.2.1 through the client's eth0; the second should choose 198.51.100.1 through the server's eth0. If either lookup fails, repair that route before diagnosing a firewall decision. The ip route reference covers these lookups.

Start a service that proves an application exchange

A completed TCP handshake is useful, but this lab also checks that the application reads a line and answers it. Save this small test program as echo-service.py in your working directory:

#!/usr/bin/env python3
"""A local lab protocol: echo one line as 'ACK <line>'. No external traffic."""
import argparse, json, socket, sys, threading
p = argparse.ArgumentParser()
p.add_argument('mode', choices=['server', 'client', 'hold'])
p.add_argument('--bind', default='198.51.100.10')
p.add_argument('--port', type=int, default=8080)
p.add_argument('--source-port', type=int, default=40010)
p.add_argument('--message', default='baseline')
a = p.parse_args()

def output(**values):
    print(json.dumps(values), flush=True)
if a.mode == 'server':
    s = socket.socket()
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind((a.bind, a.port))
    s.listen()
    output(event='listening', address=s.getsockname())

    def handle(c, peer):
        with c:
            f = c.makefile('rb')
            for line in f:
                output(event='request', peer=peer, message=line.decode().rstrip())
                c.sendall(b'ACK ' + line)
    while True:
        c, peer = s.accept()
        threading.Thread(target=handle, args=(c, peer), daemon=True).start()
else:
    try:
        with socket.socket() as s:
            s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
            s.settimeout(2)
            s.bind(('192.0.2.10', a.source_port))
            s.connect(('198.51.100.10', a.port))
            output(event='connected', local=s.getsockname(), remote=s.getpeername())
            f = s.makefile('rb')
            lines = sys.stdin if a.mode == 'hold' else [a.message + '\n']
            for line in lines:
                s.sendall(line.encode())
                answer = f.readline()
                output(event='reply', message=answer.decode().rstrip())
    except OSError as e:
        output(event='error', type=type(e).__name__, errno=e.errno, message=str(e))
        sys.exit(1)

In a server terminal, start it inside the server namespace and leave that terminal open:

sudo ip netns exec plf-server python3 -u echo-service.py server

In another terminal, check the listener and make the baseline request:

sudo ip netns exec plf-server ss -lntp 'sport = :8080'
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40010 --message baseline

The measured client connected from 192.0.2.10:40010 and received ACK baseline. The server logged the same client address and message. That checks routing in both directions, the listening socket, and an actual request/response. Keep that successful result; a later timeout only becomes informative when you know what changed.

Create two forward chains

Save this as filter.nft:

table inet plf_filter {
    chain services {
        type filter hook forward priority 0; policy accept;
        ip saddr 192.0.2.10 ip daddr 198.51.100.10 tcp dport 8080 counter accept comment "lab-service-allow"
    }
    chain later {
        type filter hook forward priority 10; policy accept;
    }
}

Check and load it only inside the router namespace:

sudo ip netns exec plf-router nft -c -f filter.nft
sudo ip netns exec plf-router nft -f filter.nft
sudo ip netns exec plf-router nft -a -y list table inet plf_filter

Both chains use the forward hook because the destination is another host. A service on the router itself would follow input instead. Lower priority numbers run first, so services at 0 precedes later at 10. The empty later chain currently accepts by policy, and the baseline exchange still works.

Add the fault to the later chain:

sudo ip netns exec plf-router nft add rule inet plf_filter later \
  ip saddr 192.0.2.10 ip daddr 198.51.100.10 tcp dport 8080 \
  counter drop comment '"lab-late-drop"'

This is a silent drop, not a reject. It matches only client-to-server TCP traffic on port 8080 in the isolated router. The nftables guide explains tables, handles, and rule syntax; here the ordering of two base chains is the part under test.

Enable nft monitor trace for this flow

nft monitor trace only receives trace events for packets whose nftrace flag has been set. Starting the monitor alone does not instrument the flow. Save this as trace.nft:

table inet plf_debug {
    chain selected {
        type filter hook prerouting priority -301; policy accept;
        iifname "lan0" ip saddr 192.0.2.10 ip daddr 198.51.100.10 tcp dport 8080 meta nftrace set 1
    }
}

The early prerouting chain marks only packets arriving on lan0 for this destination and port. It lets them continue to routing and forwarding. It does not trace the reply direction, ARP, or unrelated traffic.

sudo ip netns exec plf-router nft -c -f trace.nft
sudo ip netns exec plf-router nft -f trace.nft
sudo ip netns exec plf-router nft -a monitor trace

Leave that monitor running. In two more terminals, capture on either side of the router:

# Router ingress from the client
sudo ip netns exec plf-router tcpdump --immediate-mode -nni lan0 'tcp port 8080'

# Router egress toward the server
sudo ip netns exec plf-router tcpdump --immediate-mode -nni wan0 'tcp port 8080'

Start both captures before the next request. These points bracket forwarding; the ingress capture can see a packet before the forward filter drops it. The tcpdump reference covers saving the same observations to a PCAP.

Follow one SYN through accept and drop

Use a new source port so this attempt is easy to identify:

sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40011 --message late-drop

The measured result was TimeoutError: timed out. The ingress capture recorded two SYN attempts from source port 40011; the egress capture recorded none. Both captures reported zero kernel drops, and their captured counts matched the filter-received counts.

The trace supplied the reason. These are selected rule lines from one measured packet; packet header lines are omitted:

trace id 6b50c914 inet plf_debug selected rule ... meta nftrace set 1 # handle 2 (verdict continue)
trace id 6b50c914 inet plf_filter services rule ... accept comment "lab-service-allow" # handle 3 (verdict accept)
trace id 6b50c914 inet plf_filter later rule ... drop comment "lab-late-drop" # handle 4 (verdict drop)

The shared trace ID ties these lines to one packet. A retransmitted SYN has its own trace ID, so do not combine unrelated lines just because their addresses match. Follow the hook, chain, and verdict for a single ID first.

The allow counter rose from 6 to 8 packets during the failed attempt, while the new drop rule counted 2 packets and 120 bytes. Those are the same two attempts at successive stages, not four separate lost packets. The earlier accept did its job in the services chain; it simply was not the last decision.

The nft manual describes this behavior: an accept ends the current base chain, while a later base chain can still drop the packet. A drop ends further ruleset evaluation for that packet. The tracing guide documents enabling nftrace and monitoring the resulting events.

Remove the deny and verify recovery

List the current rule handles:

sudo ip netns exec plf-router nft -a list chain inet plf_filter later

In the measured run, the deny rule was handle 4. Use the handle attached to lab-late-drop in your own listing; handles can differ in another ruleset.

sudo ip netns exec plf-router nft delete rule inet plf_filter later handle 4
sudo ip netns exec plf-client python3 echo-service.py client \
  --source-port 40012 --message recovery

The client received ACK recovery. Both capture points recorded the complete ten-packet exchange, including its handshake, data, and close, and the trace no longer contained the late drop. Recovery checked the same application path rather than substituting a ping for a TCP result.

After the investigation, stop the monitors and foreground captures with Ctrl+C and remove only the temporary tracing table:

sudo ip netns exec plf-router nft delete table inet plf_debug

The routed lab and allow policy can stay in place for the conntrack and SNAT investigation. If the trace accepts the flow but the application still fails, continue with route, listener, and NAT checks. The Linux trace names the filter decision; the Linux-to-Cisco drop lab uses captures and an IOS ACL counter to locate a similar interruption across a Cisco router.