DNSSEC Broken Trust Chain: Debug a DS Mismatch with delv

A signed answer can look completely healthy in dig and still fail DNSSEC validation. In this lab, www.secure.lab.test. kept returning 203.0.113.42 with an RRSIG after one hexadecimal digit in its parent DS record changed. delv rejected it with broken trust chain.

The useful part is finding which link failed. We will validate the same answer before the fault, inspect the parent DS and child DNSKEY, then restore the correct delegation. An unsigned sibling provides a second comparison: an insecure answer and a bogus answer have different reasons behind them.

If the header and sections are unfamiliar, start with the dig guide. For what an application does before it ever sends DNS, use the Linux resolver path.

The chain we are testing

The client at 192.0.2.10 connects directly to an authoritative BIND server at 192.0.2.20, using UDP or TCP port 1053. Two network namespaces share a veth link. Neither has a default route.

The server hosts three zones:

• lab.test. is signed and supplies the local trust anchor.
• secure.lab.test. is signed. Its parent publishes a DS derived from the child KSK, the key-signing key.
• plain.lab.test. is unsigned. Its signed parent supplies proof that the delegation has no DS.

delv performs validation on the client. It asks this one server for the A, DNSKEY, DS, and denial-of-existence records it needs. This is a private chain anchored at lab.test., not an Internet delegation or a recursive resolver deployment. Hosting parent and children together makes the experiment small while retaining the DS-to-DNSKEY check.

Tested with: Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, BIND and delv 9.20.24, tcpdump 4.99.6. The BIND 9.20.24 manual explains delv's local validator and the -a and +root trust-anchor options used below. For the namespace mechanics, see the namespace lab.

Prepare the isolated lab

Run the setup as your ordinary user on a disposable Linux lab VM with sudo access. These commands target Ubuntu 26.04 package names. Keep the same working directory through setup; use new namespace names if these already exist.

DNS_LAB_ROOT=${DNS_LAB_ROOT:-"$HOME/dns-lab"}
DNS_CLIENT=${DNS_CLIENT:-pldns-client}
DNS_SERVER=${DNS_SERVER:-pldns-server}
mkdir "$DNS_LAB_ROOT"
cd "$DNS_LAB_ROOT"
umask 077
mkdir keys zones pki evidence vendor

Install the utilities, then extract the distribution's named executable locally. This avoids installing the BIND server service. Check ldd for any not found dependency before continuing; the matching BIND libraries come with bind9-utils.

sudo apt-get install -y --no-install-recommends \
  bind9-utils knot-dnsutils openssl tcpdump tshark iproute2 \
  libjemalloc2 liblmdb0 libmaxminddb0 libnghttp2-14 \
  liburcu8t64 libxml2-16
(cd vendor && apt download bind9 && dpkg-deb -x bind9_*.deb extracted)
ldd vendor/extracted/usr/sbin/named
vendor/extracted/usr/sbin/named -V

Connect the two namespaces. The route output should contain the connected 192.0.2.0/24 network and no default route. Queries will use the server's address explicitly.

sudo ip netns add "$DNS_CLIENT"
sudo ip netns add "$DNS_SERVER"
sudo ip link add pldnsr-c type veth peer name pldnsr-s
sudo ip link set pldnsr-c netns "$DNS_CLIENT"
sudo ip link set pldnsr-s netns "$DNS_SERVER"
sudo ip -n "$DNS_CLIENT" link set pldnsr-c name eth0
sudo ip -n "$DNS_SERVER" link set pldnsr-s name eth0
sudo ip -n "$DNS_CLIENT" addr add 192.0.2.10/24 dev eth0
sudo ip -n "$DNS_SERVER" addr add 192.0.2.20/24 dev eth0
for ns in "$DNS_CLIENT" "$DNS_SERVER"; do
  sudo ip -n "$ns" link set lo up
  sudo ip -n "$ns" link set eth0 up
  sudo ip -n "$ns" route show
done

Generate keys, zones, and the deliberate fault

Use separate KSK and ZSK files for each signed zone. A ZSK signs ordinary records; this small lab also signs the DNSKEY set with the available zone keys. Algorithm 13 is ECDSAP256SHA256. The .private files stay in keys; the DNSKEY .key files are public.

parent_ksk=$(dnssec-keygen -K keys -a ECDSAP256SHA256 -L 60 -f KSK lab.test.)
parent_zsk=$(dnssec-keygen -K keys -a ECDSAP256SHA256 -L 60 lab.test.)
child_ksk=$(dnssec-keygen -K keys -a ECDSAP256SHA256 -L 60 -f KSK secure.lab.test.)
child_zsk=$(dnssec-keygen -K keys -a ECDSAP256SHA256 -L 60 secure.lab.test.)
dnssec-dsfromkey -2 "keys/$child_ksk.key" > zones/child.ds

Create the answer records and the parent delegation. The child DS is generated from the child KSK rather than copied from this article: every rerun produces different keys and digests.

cat > zones/secure.zone <<'EOF'
$ORIGIN secure.lab.test.
$TTL 60
@ IN SOA ns.lab.test. hostmaster.lab.test. (2026100701 60 60 3600 60)
@ IN NS ns.lab.test.
www IN A 203.0.113.42
EOF
cat > zones/plain.zone <<'EOF'
$ORIGIN plain.lab.test.
$TTL 60
@ IN SOA ns.lab.test. hostmaster.lab.test. (2026100701 60 60 3600 60)
@ IN NS ns.lab.test.
www IN A 203.0.113.43
EOF
cat > zones/parent-good.zone <<'EOF'
$ORIGIN lab.test.
$TTL 60
@ IN SOA ns.lab.test. hostmaster.lab.test. (2026100701 60 60 3600 60)
@ IN NS ns.lab.test.
ns IN A 192.0.2.20
secure IN NS ns.lab.test.
plain IN NS ns.lab.test.
EOF
cat zones/child.ds >> zones/parent-good.zone

Make a second parent file with one changed digest digit. Then sign both parents and the unchanged child. The bogus parent must itself have valid signatures; otherwise we would be testing a bad parent signature instead of a DS mismatch.

python3 - <<'PYFAULT'
from pathlib import Path
p = Path('zones/parent-good.zone')
lines = p.read_text().splitlines()
for i, line in enumerate(lines):
    fields = line.split()
    if 'DS' in fields:
        digest = fields[-1]
        fields[-1] = ('0' if digest[0] != '0' else '1') + digest[1:]
        lines[i] = ' '.join(fields)
Path('zones/parent-bogus.zone').write_text('\n'.join(lines) + '\n')
PYFAULT
for zone in parent-good parent-bogus secure; do
  origin=lab.test.
  [ "$zone" != secure ] || origin=secure.lab.test.
  dnssec-signzone -S -n 1 -a -K "$PWD/keys" \
    -o "$origin" -s -3600 -e +604800 \
    -f "zones/$zone.zone.signed" "zones/$zone.zone"
done

-S finds the zone's keys, -a verifies generated signatures, and -n 1 bounds signing to one worker. Signatures start an hour before signing and expire about seven days later. Check the VM's UTC clock and regenerate the signed files before a later rerun.

Trust the parent KSK explicitly:

python3 - "$parent_ksk" <<'PYANCHOR'
from pathlib import Path
import sys
line = next(s for s in Path('keys/' + sys.argv[1] + '.key').read_text().splitlines()
            if ' DNSKEY ' in s)
fields = line.split()
i = fields.index('DNSKEY')
flags, protocol, algorithm = fields[i+1:i+4]
key = ''.join(fields[i+4:])
Path('anchor.conf').write_text(
    f'trust-anchors {{ "lab.test." static-key {flags} {protocol} {algorithm} "{key}"; }};\n')
PYANCHOR

This anchor is the root of this experiment's trust. A normal lookup rooted at . will not discover the private lab.test. chain. Do not substitute the child key as the anchor: that would bypass the parent delegation we are trying to test.

Serve the healthy zones and validate locally

Build two configurations. Both serve exactly the same child zone; only the selected signed parent differs. dnssec-validation no applies to this authoritative server. The client-side delv validator remains enabled.

for mode in good bogus; do
cat > "named-$mode.conf" <<EOF
options {
 directory "$PWD/zones";
 listen-on port 1053 { 192.0.2.20; };
 listen-on-v6 { none; };
 recursion no;
 dnssec-validation no;
 allow-query { 192.0.2.10; };
 allow-transfer { none; };
 notify no;
 pid-file "$PWD/evidence/named.pid";
 session-keyfile "$PWD/keys/session.key";
 managed-keys-directory "$PWD/keys";
};
controls { };
zone "lab.test." { type primary; file "parent-$mode.zone.signed"; };
zone "secure.lab.test." { type primary; file "secure.zone.signed"; };
zone "plain.lab.test." { type primary; file "plain.zone"; };
EOF
named-checkconf -z "named-$mode.conf"
done

In a separate terminal on the VM, enter the lab directory and start the healthy server in the foreground:

DNS_LAB_ROOT=${DNS_LAB_ROOT:-"$HOME/dns-lab"} DNS_CLIENT=${DNS_CLIENT:-pldns-client} DNS_SERVER=${DNS_SERVER:-pldns-server}
cd "$DNS_LAB_ROOT"
sudo ip netns exec "$DNS_SERVER" \
  "$PWD/vendor/extracted/usr/sbin/named" \
  -g -n 1 -u "$USER" -c "$PWD/named-good.conf"

Wait for running in the server log. The first line of each new-terminal block reuses the lab variables (with the same defaults), so every terminal targets the same directory and namespaces. -u "$USER" lets the process read your owner-only lab files after dropping privileges. It is a foreground lab process, bound to the server namespace's address.

In another terminal, query and validate:

DNS_LAB_ROOT=${DNS_LAB_ROOT:-"$HOME/dns-lab"} DNS_CLIENT=${DNS_CLIENT:-pldns-client} DNS_SERVER=${DNS_SERVER:-pldns-server}
cd "$DNS_LAB_ROOT"
sudo ip netns exec "$DNS_CLIENT" delv \
  @192.0.2.20 -p 1053 -a "$PWD/anchor.conf" +root=lab.test \
  www.secure.lab.test. A +rtrace +vtrace +trust

Selected stdout from the measured healthy case:

; fully validated
www.secure.lab.test.	60	IN	A	203.0.113.42

The trace on stderr also showed the parent DNSKEY becoming secure, the child's DS validating under that parent, and the child key set becoming trusted. Finally, the A RRset's signature verified. +rtrace shows record fetches; +vtrace shows the validation steps.

Capture those fetches at the client if you want packet evidence. Start this before the query in another terminal, wait for listening on eth0, and press Ctrl-C after the query completes:

DNS_LAB_ROOT=${DNS_LAB_ROOT:-"$HOME/dns-lab"} DNS_CLIENT=${DNS_CLIENT:-pldns-client} DNS_SERVER=${DNS_SERVER:-pldns-server}
cd "$DNS_LAB_ROOT"
sudo ip netns exec "$DNS_CLIENT" tcpdump \
  --immediate-mode -U -i eth0 -nn -s 0 \
  -w "$PWD/evidence/secure.pcap" \
  "host 192.0.2.20 and (udp port 1053 or tcp port 1053)"

Immediate mode matters for such short exchanges: -U flushes packets already delivered to tcpdump, but does not eliminate libpcap's packet-buffer timeout. Check the final capture statistics before treating a tiny file as evidence that no queries occurred.

Break the DS and locate the failed step

Press Ctrl-C in the healthy server terminal. Restart the same command with named-bogus.conf; wait for running. Nothing in the child zone or its A record has changed.

First ask for the raw answer:

sudo ip netns exec "$DNS_CLIENT" dig \
  @192.0.2.20 -p 1053 www.secure.lab.test. A \
  +norecurse +dnssec +time=2 +tries=1

The measured response had status: NOERROR, flags: qr aa, the same A record, and an RRSIG. +dnssec requests DNSSEC records through the DO bit. Their presence does not establish that the chain validates, and this authoritative response did not set AD.

Repeat the earlier delv command. Selected stderr:

;; validating secure.lab.test/DNSKEY: no DNSKEY matching DS
;; validating secure.lab.test/DNSKEY: no valid signature found (DS)
;; broken trust chain resolving 'www.secure.lab.test/A/IN': 192.0.2.20#1053
;; resolution failed: broken trust chain

Read the preceding steps too. The trace validated the parent DNSKEY and the signed DS RRset before reporting no DNSKEY matching DS for the child. That narrows this fault to the relationship between a trusted DS and the child keys.

Inspect that relationship directly:

sudo ip netns exec "$DNS_CLIENT" dig \
  @192.0.2.20 -p 1053 secure.lab.test. DS +dnssec +norecurse
sudo ip netns exec "$DNS_CLIENT" dig \
  @192.0.2.20 -p 1053 secure.lab.test. DNSKEY +dnssec +norecurse
dnssec-dsfromkey -2 "keys/$child_ksk.key"
cat zones/child.ds

The last two commands run in the original setup shell where child_ksk is still defined. In a new shell, select the child's public KSK file by the 257 DNSKEY flags rather than assuming a key filename from another run.

The measured key tag was 40514, algorithm 13, digest type 2 (SHA-256). The correct digest began 7EB676F0; the published bogus digest began 0EB676F0. The rest of the digest, child DNSKEY set, and signed A answer were unchanged. A matching key tag and algorithm are not enough when the digest differs.

On this installed delv, the failed validation printed resolution failed: broken trust chain but returned shell exit code 0. For this test, check the validation message and expected answer together; do not automate success using only $? or only grep for the address.

Distinguish unsigned from bogus, then repair

The unsigned branch behaves differently. In either server mode, run:

sudo ip netns exec "$DNS_CLIENT" delv \
  @192.0.2.20 -p 1053 -a "$PWD/anchor.conf" +root=lab.test \
  www.plain.lab.test. A +rtrace +vtrace +trust

Measured stdout:

; unsigned answer
www.plain.lab.test.	60	IN	A	203.0.113.43

The trace validated the parent's NSEC proof that plain.lab.test. had no DS. With this authenticated proof of an unsigned delegation, the answer is insecure. The signed child with a mismatched DS is bogus because its expected chain cannot be established. These categories follow RFC 4033's validation states; removing validation from the client would not demonstrate either result.

For recovery, stop the bogus server and restart named-good.conf. Run the same delv command for www.secure.lab.test. again. The measured result returned to fully validated with 203.0.113.42. Each check used a fresh delv process, so it did not inherit a previous invocation's cache.

If you investigate this symptom elsewhere, keep the anchor, UTC clock, actual resolver, DS digest, and child keys visible. A missing answer, an unreachable server, an expired signature, and this mismatched delegation need different evidence. Here, the healthy parent signature plus the failed DS-to-DNSKEY match identifies the deliberate fault.

Next, use the plain DNS, DoT, and DoH comparison with these same zones. It demonstrates how verified TLS can deliver a DNS answer whose DNSSEC chain still fails.