Linux rp_filter: Diagnose Asymmetric Routing Packet Drops

The SYN was visible in tcpdump on the server. The listening socket was present, and both nftables chains had accept policies. The connection still timed out.

In this lab, the server's best route back to the client used a different interface from the one receiving the SYN. Strict IPv4 reverse-path filtering rejected that source before the packet reached the local input chain. We'll identify the drop with captures and counter deltas, then compare an intentional asymmetric path with a repaired symmetric route.

Two cables, one listening address

The client and server each have two veth interfaces:

• eth0: client 192.0.2.10/24, server 192.0.2.20/24.
• alt0: client 198.51.100.10/24, server 198.51.100.20/24.
• The service listens on 192.0.2.20:8080. Requests arrive through the first cable.

There is no router, NAT, default route, or uplink. A later server-side host route sends replies to 192.0.2.10 through the second cable. These documentation addresses stay inside the namespaces.

Tested with: Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, iproute2 6.19.0, nftables 1.1.6, tcpdump 4.99.6, Python 3.14.4. You need those tools and sudo. The namespace lab explains veth and namespace basics; use a fresh directory for this separate topology.

Build the path and prove the application

Create scripts/ and evidence/ in that directory. Save this as scripts/setup-rpf.sh. It refuses existing lab names rather than replacing them.

#!/usr/bin/env bash
set -euo pipefail
for ns in plrpf-client plrpf-server; do
  if ip netns list | awk '{print $1}' | grep -qx "$ns"; then
    echo "Existing namespace: $ns; stop before reusing it" >&2; exit 1
  fi
done
for dev in rpf-c0 rpf-s0 rpf-c1 rpf-s1; do
  if ip link show "$dev" >/dev/null 2>&1; then
    echo "Existing interface: $dev; stop before reusing it" >&2; exit 1
  fi
done
for ns in plrpf-client plrpf-server; do
  ip netns add "$ns"
  ip -n "$ns" link set lo up
done
ip link add rpf-c0 type veth peer name rpf-s0
ip link add rpf-c1 type veth peer name rpf-s1
ip link set rpf-c0 netns plrpf-client
ip link set rpf-c1 netns plrpf-client
ip link set rpf-s0 netns plrpf-server
ip link set rpf-s1 netns plrpf-server
ip -n plrpf-client link set rpf-c0 name eth0
ip -n plrpf-client link set rpf-c1 name alt0
ip -n plrpf-server link set rpf-s0 name eth0
ip -n plrpf-server link set rpf-s1 name alt0
ip -n plrpf-client addr add 192.0.2.10/24 dev eth0
ip -n plrpf-server addr add 192.0.2.20/24 dev eth0
ip -n plrpf-client addr add 198.51.100.10/24 dev alt0
ip -n plrpf-server addr add 198.51.100.20/24 dev alt0
for ns in plrpf-client plrpf-server; do
  for dev in eth0 alt0; do ip -n "$ns" link set "$dev" up; done
done
ip netns exec plrpf-client sysctl -w net.ipv4.conf.all.rp_filter=0 net.ipv4.conf.eth0.rp_filter=2 net.ipv4.conf.alt0.rp_filter=2
ip netns exec plrpf-server sysctl -w net.ipv4.conf.all.rp_filter=0 net.ipv4.conf.eth0.rp_filter=1 net.ipv4.conf.alt0.rp_filter=1

Run it as root with sudo bash scripts/setup-rpf.sh. The client's two interfaces use loose validation so they can accept the intentionally asymmetric reply later. The server starts with strict validation on each interface and all=0; that combination still validates incoming traffic. These are explicit settings in the lab namespaces, not assumptions about your distribution's defaults.

Save this as scripts/echo-rpf.py:

#!/usr/bin/env python3
import socket
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('192.0.2.20', 8080))
s.listen()
print('listening 192.0.2.20:8080', flush=True)
while True:
    c, peer = s.accept()
    with c:
        c.settimeout(5)
        data = c.recv(1024)
        c.sendall(b'ACK ' + data)
        print(peer, data.decode().strip(), flush=True)

In terminal A on the Linux host, start it in the server namespace:

sudo ip netns exec plrpf-server python3 -u scripts/echo-rpf.py

Leave that terminal open. Save this client as scripts/client-rpf.py; it binds a visible source port and requires the expected application response.

#!/usr/bin/env python3
import json,socket,sys,time
port=int(sys.argv[1]);label=sys.argv[2];start=time.monotonic()
s=socket.socket();s.settimeout(2);s.bind(('192.0.2.10',port))
try:
    s.connect(('192.0.2.20',8080));s.sendall(label.encode());reply=s.recv(1024).decode()
    assert reply=='ACK '+label
    print(json.dumps({'label':label,'source_port':port,'reply':reply,'seconds':time.monotonic()-start}))
except Exception as e:
    print(json.dumps({'label':label,'source_port':port,'error':type(e).__name__,'message':str(e),'seconds':time.monotonic()-start}));sys.exit(1)
finally:s.close()

In terminal B, confirm the listener and run the baseline:

sudo ip netns exec plrpf-server ss -lntp 'sport = :8080'
sudo ip netns exec plrpf-server ip route get 192.0.2.10 from 192.0.2.20
sudo ip netns exec plrpf-client python3 scripts/client-rpf.py 44001 rpf-baseline

The lookup used eth0, and the client received ACK rpf-baseline. The server logged that message. This checks a real exchange before changing the route. For listener checks, see the ss reference.

Observe admission before changing the route

Save this as scripts/rpf-observe.nft:

table inet rpf_observe {
 chain early { type filter hook prerouting priority -300; policy accept;
  ip saddr 192.0.2.10 ip daddr 192.0.2.20 tcp dport 8080 tcp flags & (syn | ack) == syn counter comment "lab-prerouting-syn"
 }
 chain local { type filter hook input priority 0; policy accept;
  ip saddr 192.0.2.10 ip daddr 192.0.2.20 tcp dport 8080 tcp flags & (syn | ack) == syn counter comment "lab-input-syn"
 }
}

Validate and load it only in the server namespace:

sudo ip netns exec plrpf-server nft -c -f scripts/rpf-observe.nft
sudo ip netns exec plrpf-server nft -f scripts/rpf-observe.nft
sudo ip netns exec plrpf-server nft list table inet rpf_observe
sudo ip netns exec plrpf-server nstat -az TcpExtIPReversePathFilter

The rules count initial SYNs at prerouting and local input. They contain no drop verdict. Keep the before values; the relevant measurement is the change during one attempt, not the namespace's cumulative total. nstat -az requests absolute counters and includes zero-valued entries.

In two additional terminals, capture each server interface before the attempt:

sudo ip netns exec plrpf-server tcpdump --immediate-mode -U -nn -i eth0 -w evidence/strict-eth0.pcap 'tcp port 8080'

sudo ip netns exec plrpf-server tcpdump --immediate-mode -U -nn -i alt0 -w evidence/strict-alt0.pcap 'tcp port 8080'

Wait for each capture's listening message. Stop captures with Ctrl+C after the attempt and retain their statistics. Use different filenames for later cases. The tcpdump guide covers capture filters.

Make the best reverse path disagree with ingress

In terminal B, add the more specific server route:

sudo ip -n plrpf-server route add 192.0.2.10/32 via 198.51.100.10 dev alt0
sudo ip netns exec plrpf-server ip route get 192.0.2.10 from 192.0.2.20
sudo ip netns exec plrpf-server sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.eth0.rp_filter
sudo ip netns exec plrpf-client python3 scripts/client-rpf.py 44002 rpf-strict-asymmetric
sudo ip netns exec plrpf-server nft list table inet rpf_observe
sudo ip netns exec plrpf-server nstat -az TcpExtIPReversePathFilter

The selected route became:

192.0.2.10 from 192.0.2.20 via 198.51.100.10 dev alt0

The client timed out after its two-second bound. In the saved run, eth0 recorded two SYNs from port 44002 and no SYN/ACK. alt0 recorded no TCP packet for that attempt. The prerouting SYN counter increased by two; the input SYN counter did not increase. TcpExtIPReversePathFilter increased by two. Both captures reported zero kernel drops and matching captured/filter-received counts.

That combination places the failure after the early observation and before local input, with the kernel's reverse-path counter identifying the rejected checks. Seeing a packet in an ingress capture does not establish that a socket accepted it. This also differs from the later nftables-chain drop: here neither counting rule denied the packet.

In this simple main-table lab, the lookup exposes the mismatch. With policy routing, marks, VRFs, or multiple routing tables, check the actual rule and table lookup too.

Check rp_filter on both all and the interface

The Linux 7.0 sysctl documentation defines 1 as strict validation and 2 as loose validation. Strict mode checks the best reverse interface; loose mode checks whether the source is reachable. The kernel uses the numeric maximum of all and the receiving interface's setting.

Test why changing only eth0 to zero may leave strict validation active:

sudo ip netns exec plrpf-server sysctl -w net.ipv4.conf.all.rp_filter=1 net.ipv4.conf.eth0.rp_filter=0
sudo ip netns exec plrpf-client python3 scripts/client-rpf.py 44003 rpf-all-strict-interface-zero

This attempt also timed out. Its prerouting and reverse-path counter deltas were two, with zero input SYNs. The effective value was still max(1,0)=1.

If the asymmetric route is intentional, test loose validation on this one receiving lab interface:

sudo ip netns exec plrpf-server sysctl -w net.ipv4.conf.eth0.rp_filter=2
sudo ip netns exec plrpf-client python3 scripts/client-rpf.py 44004 rpf-loose-asymmetric

The client received ACK rpf-loose-asymmetric. Both SYN counters increased by one; the reverse-path counter did not increase. Captures showed the request SYN on server eth0 and its SYN/ACK on alt0. That proves the asymmetric path worked with these validation settings. Because the maximum is numeric, max(1,2)=2; the all=1 setting does not override a per-interface 2 with strict mode.

Loose validation has a weaker source-reachability condition. Use the route design and source-validation policy to decide whether it belongs on a real interface.

Repair the route while retaining strict validation

For the final comparison, restore strict validation and remove only the deliberately added host route:

sudo ip netns exec plrpf-server sysctl -w net.ipv4.conf.all.rp_filter=0 net.ipv4.conf.eth0.rp_filter=1
sudo ip -n plrpf-server route del 192.0.2.10/32 via 198.51.100.10 dev alt0
sudo ip netns exec plrpf-server ip route get 192.0.2.10 from 192.0.2.20
sudo ip netns exec plrpf-client python3 scripts/client-rpf.py 44005 rpf-strict-route-repaired

The route again selected eth0, and the client received ACK rpf-strict-route-repaired. Each SYN counter increased by one with no reverse-path counter increase. Stop the foreground test service with Ctrl+C when finished.

For a SYN that reaches the interface but goes no further, compare the reverse lookup, effective validation setting, hook counters, and socket state. The route/listener/firewall capstone covers the surrounding checks. If the handshake and small requests work but a larger upload stalls, continue with the IPv6 Packet Too Big lab, where the missing evidence is feedback to the sender.