> ## Content Index
> Fetch the complete content index at: https://www.pinglabz.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# DNSSEC Broken Trust Chain: Debug a DS Mismatch with delv
- URL: https://www.pinglabz.com/dnssec-broken-chain-delv-lab/
- Published: 2026-10-13T06:25:28.000Z
- Updated: 2026-10-13T06:25:28.000Z
- Description: Change one digit in a signed parent DS, follow delv to the failed child key check, and tell a bogus chain from an unsigned delegation.
- Author: Jaime
- Tags: DNS, Troubleshooting, Linux, Linux Labs

A signed answer can look completely healthy in `dig` and still fail DNSSEC validation. In this lab, `www.secure.lab.test.` kept returning `203.0.113.42` with an RRSIG after one hexadecimal digit in its parent DS record changed. `delv` rejected it with `broken trust chain`.

The useful part is finding which link failed. We will validate the same answer before the fault, inspect the parent DS and child DNSKEY, then restore the correct delegation. An unsigned sibling provides a second comparison: an insecure answer and a bogus answer have different reasons behind them.

If the header and sections are unfamiliar, start with [the dig guide](https://www.pinglabz.com/dig-command/). For what an application does before it ever sends DNS, use [the Linux resolver path](https://www.pinglabz.com/linux-dns-resolution/).

## The chain we are testing

The client at `192.0.2.10` connects directly to an authoritative BIND server at `192.0.2.20`, using UDP or TCP port `1053`. Two network namespaces share a veth link. Neither has a default route.

The server hosts three zones:

- `lab.test.` is signed and supplies the local trust anchor.
- `secure.lab.test.` is signed. Its parent publishes a DS derived from the child KSK, the key-signing key.
- `plain.lab.test.` is unsigned. Its signed parent supplies proof that the delegation has no DS.

`delv` performs validation on the client. It asks this one server for the A, DNSKEY, DS, and denial-of-existence records it needs. This is a private chain anchored at `lab.test.`, not an Internet delegation or a recursive resolver deployment. Hosting parent and children together makes the experiment small while retaining the DS-to-DNSKEY check.

Tested with: Ubuntu 26.04.1 LTS, kernel `7.0.0-38-generic`, BIND and `delv` `9.20.24`, tcpdump `4.99.6`. The [BIND 9.20.24 manual](https://bind9.readthedocs.io/en/v9.20.24/manpages.html?ref=pinglabz.com#delv-dns-lookup-and-validation-utility) explains `delv`'s local validator and the `-a` and `+root` trust-anchor options used below. For the namespace mechanics, see [the namespace lab](https://www.pinglabz.com/linux-namespace-network-lab/).

## Prepare the isolated lab

Run the setup as your ordinary user on a disposable Linux lab VM with sudo access. These commands target Ubuntu 26.04 package names. Keep the same working directory through setup; use new namespace names if these already exist.

```bash
DNS_LAB_ROOT=${DNS_LAB_ROOT:-"$HOME/dns-lab"}
DNS_CLIENT=${DNS_CLIENT:-pldns-client}
DNS_SERVER=${DNS_SERVER:-pldns-server}
mkdir "$DNS_LAB_ROOT"
cd "$DNS_LAB_ROOT"
umask 077
mkdir keys zones pki evidence vendor
```

Install the utilities, then extract the distribution's `named` executable locally. This avoids installing the BIND server service. Check `ldd` for any `not found` dependency before continuing; the matching BIND libraries come with `bind9-utils`.

```bash
sudo apt-get install -y --no-install-recommends \
  bind9-utils knot-dnsutils openssl tcpdump tshark iproute2 \
  libjemalloc2 liblmdb0 libmaxminddb0 libnghttp2-14 \
  liburcu8t64 libxml2-16
(cd vendor && apt download bind9 && dpkg-deb -x bind9_*.deb extracted)
ldd vendor/extracted/usr/sbin/named
vendor/extracted/usr/sbin/named -V
```

Connect the two namespaces. The route output should contain the connected `192.0.2.0/24` network and no default route. Queries will use the server's address explicitly.

```bash
sudo ip netns add "$DNS_CLIENT"
sudo ip netns add "$DNS_SERVER"
sudo ip link add pldnsr-c type veth peer name pldnsr-s
sudo ip link set pldnsr-c netns "$DNS_CLIENT"
sudo ip link set pldnsr-s netns "$DNS_SERVER"
sudo ip -n "$DNS_CLIENT" link set pldnsr-c name eth0
sudo ip -n "$DNS_SERVER" link set pldnsr-s name eth0
sudo ip -n "$DNS_CLIENT" addr add 192.0.2.10/24 dev eth0
sudo ip -n "$DNS_SERVER" addr add 192.0.2.20/24 dev eth0
for ns in "$DNS_CLIENT" "$DNS_SERVER"; do
  sudo ip -n "$ns" link set lo up
  sudo ip -n "$ns" link set eth0 up
  sudo ip -n "$ns" route show
done
```

## Generate keys, zones, and the deliberate fault

Use separate KSK and ZSK files for each signed zone. A ZSK signs ordinary records; this small lab also signs the DNSKEY set with the available zone keys. Algorithm 13 is ECDSAP256SHA256\. The `.private` files stay in `keys`; the DNSKEY `.key` files are public.

```bash
parent_ksk=$(dnssec-keygen -K keys -a ECDSAP256SHA256 -L 60 -f KSK lab.test.)
parent_zsk=$(dnssec-keygen -K keys -a ECDSAP256SHA256 -L 60 lab.test.)
child_ksk=$(dnssec-keygen -K keys -a ECDSAP256SHA256 -L 60 -f KSK secure.lab.test.)
child_zsk=$(dnssec-keygen -K keys -a ECDSAP256SHA256 -L 60 secure.lab.test.)
dnssec-dsfromkey -2 "keys/$child_ksk.key" > zones/child.ds
```

Create the answer records and the parent delegation. The child DS is generated from the child KSK rather than copied from this article: every rerun produces different keys and digests.

```bash
cat > zones/secure.zone <<'EOF'
$ORIGIN secure.lab.test.
$TTL 60
@ IN SOA ns.lab.test. hostmaster.lab.test. (2026100701 60 60 3600 60)
@ IN NS ns.lab.test.
www IN A 203.0.113.42
EOF
cat > zones/plain.zone <<'EOF'
$ORIGIN plain.lab.test.
$TTL 60
@ IN SOA ns.lab.test. hostmaster.lab.test. (2026100701 60 60 3600 60)
@ IN NS ns.lab.test.
www IN A 203.0.113.43
EOF
cat > zones/parent-good.zone <<'EOF'
$ORIGIN lab.test.
$TTL 60
@ IN SOA ns.lab.test. hostmaster.lab.test. (2026100701 60 60 3600 60)
@ IN NS ns.lab.test.
ns IN A 192.0.2.20
secure IN NS ns.lab.test.
plain IN NS ns.lab.test.
EOF
cat zones/child.ds >> zones/parent-good.zone
```

Make a second parent file with one changed digest digit. Then sign both parents and the unchanged child. The bogus parent must itself have valid signatures; otherwise we would be testing a bad parent signature instead of a DS mismatch.

```bash
python3 - <<'PYFAULT'
from pathlib import Path
p = Path('zones/parent-good.zone')
lines = p.read_text().splitlines()
for i, line in enumerate(lines):
    fields = line.split()
    if 'DS' in fields:
        digest = fields[-1]
        fields[-1] = ('0' if digest[0] != '0' else '1') + digest[1:]
        lines[i] = ' '.join(fields)
Path('zones/parent-bogus.zone').write_text('\n'.join(lines) + '\n')
PYFAULT
for zone in parent-good parent-bogus secure; do
  origin=lab.test.
  [ "$zone" != secure ] || origin=secure.lab.test.
  dnssec-signzone -S -n 1 -a -K "$PWD/keys" \
    -o "$origin" -s -3600 -e +604800 \
    -f "zones/$zone.zone.signed" "zones/$zone.zone"
done
```

`-S` finds the zone's keys, `-a` verifies generated signatures, and `-n 1` bounds signing to one worker. Signatures start an hour before signing and expire about seven days later. Check the VM's UTC clock and regenerate the signed files before a later rerun.

Trust the parent KSK explicitly:

```bash
python3 - "$parent_ksk" <<'PYANCHOR'
from pathlib import Path
import sys
line = next(s for s in Path('keys/' + sys.argv[1] + '.key').read_text().splitlines()
            if ' DNSKEY ' in s)
fields = line.split()
i = fields.index('DNSKEY')
flags, protocol, algorithm = fields[i+1:i+4]
key = ''.join(fields[i+4:])
Path('anchor.conf').write_text(
    f'trust-anchors {{ "lab.test." static-key {flags} {protocol} {algorithm} "{key}"; }};\n')
PYANCHOR
```

This anchor is the root of this experiment's trust. A normal lookup rooted at `.` will not discover the private `lab.test.` chain. Do not substitute the child key as the anchor: that would bypass the parent delegation we are trying to test.

## Serve the healthy zones and validate locally

Build two configurations. Both serve exactly the same child zone; only the selected signed parent differs. `dnssec-validation no` applies to this authoritative server. The client-side `delv` validator remains enabled.

```bash
for mode in good bogus; do
cat > "named-$mode.conf" <<EOF
options {
 directory "$PWD/zones";
 listen-on port 1053 { 192.0.2.20; };
 listen-on-v6 { none; };
 recursion no;
 dnssec-validation no;
 allow-query { 192.0.2.10; };
 allow-transfer { none; };
 notify no;
 pid-file "$PWD/evidence/named.pid";
 session-keyfile "$PWD/keys/session.key";
 managed-keys-directory "$PWD/keys";
};
controls { };
zone "lab.test." { type primary; file "parent-$mode.zone.signed"; };
zone "secure.lab.test." { type primary; file "secure.zone.signed"; };
zone "plain.lab.test." { type primary; file "plain.zone"; };
EOF
named-checkconf -z "named-$mode.conf"
done
```

In a separate terminal on the VM, enter the lab directory and start the healthy server in the foreground:

```bash
DNS_LAB_ROOT=${DNS_LAB_ROOT:-"$HOME/dns-lab"} DNS_CLIENT=${DNS_CLIENT:-pldns-client} DNS_SERVER=${DNS_SERVER:-pldns-server}
cd "$DNS_LAB_ROOT"
sudo ip netns exec "$DNS_SERVER" \
  "$PWD/vendor/extracted/usr/sbin/named" \
  -g -n 1 -u "$USER" -c "$PWD/named-good.conf"
```

Wait for `running` in the server log. The first line of each new-terminal block reuses the lab variables (with the same defaults), so every terminal targets the same directory and namespaces. `-u "$USER"` lets the process read your owner-only lab files after dropping privileges. It is a foreground lab process, bound to the server namespace's address.

In another terminal, query and validate:

```bash
DNS_LAB_ROOT=${DNS_LAB_ROOT:-"$HOME/dns-lab"} DNS_CLIENT=${DNS_CLIENT:-pldns-client} DNS_SERVER=${DNS_SERVER:-pldns-server}
cd "$DNS_LAB_ROOT"
sudo ip netns exec "$DNS_CLIENT" delv \
  @192.0.2.20 -p 1053 -a "$PWD/anchor.conf" +root=lab.test \
  www.secure.lab.test. A +rtrace +vtrace +trust
```

Selected stdout from the measured healthy case:

```text
; fully validated
www.secure.lab.test.	60	IN	A	203.0.113.42
```

The trace on stderr also showed the parent DNSKEY becoming secure, the child's DS validating under that parent, and the child key set becoming trusted. Finally, the A RRset's signature verified. `+rtrace` shows record fetches; `+vtrace` shows the validation steps.

Capture those fetches at the client if you want packet evidence. Start this before the query in another terminal, wait for `listening on eth0`, and press Ctrl-C after the query completes:

```bash
DNS_LAB_ROOT=${DNS_LAB_ROOT:-"$HOME/dns-lab"} DNS_CLIENT=${DNS_CLIENT:-pldns-client} DNS_SERVER=${DNS_SERVER:-pldns-server}
cd "$DNS_LAB_ROOT"
sudo ip netns exec "$DNS_CLIENT" tcpdump \
  --immediate-mode -U -i eth0 -nn -s 0 \
  -w "$PWD/evidence/secure.pcap" \
  "host 192.0.2.20 and (udp port 1053 or tcp port 1053)"
```

Immediate mode matters for such short exchanges: `-U` flushes packets already delivered to tcpdump, but does not eliminate libpcap's packet-buffer timeout. Check the final capture statistics before treating a tiny file as evidence that no queries occurred.

## Break the DS and locate the failed step

Press Ctrl-C in the healthy server terminal. Restart the same command with `named-bogus.conf`; wait for `running`. Nothing in the child zone or its A record has changed.

First ask for the raw answer:

```bash
sudo ip netns exec "$DNS_CLIENT" dig \
  @192.0.2.20 -p 1053 www.secure.lab.test. A \
  +norecurse +dnssec +time=2 +tries=1
```

The measured response had `status: NOERROR`, `flags: qr aa`, the same A record, and an RRSIG. `+dnssec` requests DNSSEC records through the DO bit. Their presence does not establish that the chain validates, and this authoritative response did not set AD.

Repeat the earlier `delv` command. Selected stderr:

```text
;; validating secure.lab.test/DNSKEY: no DNSKEY matching DS
;; validating secure.lab.test/DNSKEY: no valid signature found (DS)
;; broken trust chain resolving 'www.secure.lab.test/A/IN': 192.0.2.20#1053
;; resolution failed: broken trust chain
```

Read the preceding steps too. The trace validated the parent DNSKEY and the signed DS RRset before reporting `no DNSKEY matching DS` for the child. That narrows this fault to the relationship between a trusted DS and the child keys.

Inspect that relationship directly:

```bash
sudo ip netns exec "$DNS_CLIENT" dig \
  @192.0.2.20 -p 1053 secure.lab.test. DS +dnssec +norecurse
sudo ip netns exec "$DNS_CLIENT" dig \
  @192.0.2.20 -p 1053 secure.lab.test. DNSKEY +dnssec +norecurse
dnssec-dsfromkey -2 "keys/$child_ksk.key"
cat zones/child.ds
```

The last two commands run in the original setup shell where `child_ksk` is still defined. In a new shell, select the child's public KSK file by the `257` DNSKEY flags rather than assuming a key filename from another run.

The measured key tag was `40514`, algorithm `13`, digest type `2` (SHA-256). The correct digest began `7EB676F0`; the published bogus digest began `0EB676F0`. The rest of the digest, child DNSKEY set, and signed A answer were unchanged. A matching key tag and algorithm are not enough when the digest differs.

On this installed `delv`, the failed validation printed `resolution failed: broken trust chain` but returned shell exit code **0**. For this test, check the validation message and expected answer together; do not automate success using only `$?` or only `grep` for the address.

## Distinguish unsigned from bogus, then repair

The unsigned branch behaves differently. In either server mode, run:

```bash
sudo ip netns exec "$DNS_CLIENT" delv \
  @192.0.2.20 -p 1053 -a "$PWD/anchor.conf" +root=lab.test \
  www.plain.lab.test. A +rtrace +vtrace +trust
```

Measured stdout:

```text
; unsigned answer
www.plain.lab.test.	60	IN	A	203.0.113.43
```

The trace validated the parent's NSEC proof that `plain.lab.test.` had no DS. With this authenticated proof of an unsigned delegation, the answer is insecure. The signed child with a mismatched DS is bogus because its expected chain cannot be established. These categories follow [RFC 4033's validation states](https://datatracker.ietf.org/doc/html/rfc4033?ref=pinglabz.com#section-5); removing validation from the client would not demonstrate either result.

For recovery, stop the bogus server and restart `named-good.conf`. Run the same `delv` command for `www.secure.lab.test.` again. The measured result returned to `fully validated` with `203.0.113.42`. Each check used a fresh `delv` process, so it did not inherit a previous invocation's cache.

If you investigate this symptom elsewhere, keep the anchor, UTC clock, actual resolver, DS digest, and child keys visible. A missing answer, an unreachable server, an expired signature, and this mismatched delegation need different evidence. Here, the healthy parent signature plus the failed DS-to-DNSKEY match identifies the deliberate fault.

Next, use [the plain DNS, DoT, and DoH comparison](https://www.pinglabz.com/dns-dot-doh-kdig-lab/) with these same zones. It demonstrates how verified TLS can deliver a DNS answer whose DNSSEC chain still fails.