Ping Labz

OpenSSL s_client: Debug SNI, Hostname, and Chain Errors

Keep the destination fixed while changing SNI, the expected hostname, and the chain. Verify each fault with strict OpenSSL checks.

OpenSSL TLS checks in three cards: client destination and SNI, server certificate selection, and client hostname and chain verification.

A TCP connection completes. The service sends a certificate. Then the HTTPS request fails.

That gets filed as “SSL broken,” but several different checks happen between the TCP handshake and the first HTTP response. The server has to choose a certificate, that certificate has to cover the name you intended to reach, and the client has to build a chain to a trusted root.

I tested those checks separately in an isolated Linux lab. The same address served a working certificate chain, a fallback certificate, and a leaf certificate with its intermediate missing. Each failure had a different explanation, even when OpenSSL printed the same error number.

If TCP itself never completes, start with the route, listener, and firewall lab. This article starts after that boundary: the client reaches the TLS listener.

Three inputs that look like one hostname

Keep these inputs separate when testing a virtual host:

  • -connect 192.0.2.20:8443 selects the destination address and port.
  • -servername app.test sends Server Name Indication, or SNI. This lab server uses that hint to choose its certificate.
  • -verify_hostname app.test checks whether the certificate covers the name you expected.

Sending SNI does not perform the hostname check. Connecting to an IP address also does not mean the certificate must contain that IP: here we deliberately connect by address while verifying the DNS identity app.test.

The HTTP Host header is another input. It arrives inside the TLS connection, after certificate selection and verification. Giving that header the right value cannot repair a certificate that already failed verification.

The OpenSSL s_client reference documents the separate SNI and verification options. For an HTTP-focused test against a chosen address, the existing curl guide covers --resolve and request timing.

Build an isolated TLS path

The topology is a direct veth pair between two Linux network namespaces:

plt-client                         plt-server
eth0 · 192.0.2.10/24  ← veth →     eth0 · 192.0.2.20/24
                                   :8443 — leaf + intermediate
                                   :8444 — leaf only

There is no router, DNS lookup, NAT, or uplink in this test. Both listeners deliberately use TLS 1.3. Tested with: kernel 7.0.0-38-generic, OpenSSL 3.5.5, Python 3.14.4. You need Bash, iproute2, OpenSSL, Python with TLS 1.3 support, and privileges to create namespaces. Packet inspection later also uses tcpdump and tshark.

On the Linux lab host, create a fresh directory and work from it in every terminal:

mkdir tls-lab
cd tls-lab

Save the following as setup.sh, then run sudo bash setup.sh. It refuses existing namespace or interface names rather than overwriting a previous lab.

#!/usr/bin/env bash
set -euo pipefail
for ns in plt-client plt-server; do
  if ip netns list | awk '{print $1}' | grep -qx "$ns"; then
    echo "Refusing existing namespace: $ns" >&2
    exit 1
  fi
done
for dev in pltc0 plts0; do
  if ip link show dev "$dev" >/dev/null 2>&1; then
    echo "Refusing existing interface: $dev" >&2
    exit 1
  fi
done
ip netns add plt-client
ip netns add plt-server
ip -n plt-client link set lo up
ip -n plt-server link set lo up
ip link add pltc0 type veth peer name plts0
ip link set pltc0 netns plt-client
ip link set plts0 netns plt-server
ip -n plt-client link set pltc0 name eth0
ip -n plt-server link set plts0 name eth0
ip -n plt-client addr add 192.0.2.10/24 dev eth0
ip -n plt-server addr add 192.0.2.20/24 dev eth0
ip -n plt-client link set eth0 up
ip -n plt-server link set eth0 up

Make the certificates reproducible

The trusted path is app.test → Server Intermediate → Server Root. A second certificate covers default.test. An unrelated root lets us test the client trust store without changing the server.

Save the following as server-pki.sh, then run bash server-pki.sh from the lab directory. It creates short-lived disposable certificates and local private keys. Keep those keys in the lab; the client commands need only the public root certificate. Nothing is added to the operating system’s trust store.

#!/usr/bin/env bash
set -euo pipefail
umask 077
if [[ -e pki ]]; then
  echo 'Refusing to overwrite existing pki directory.' >&2
  exit 1
fi
mkdir pki
key() {
  openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out "pki/$1.key"
}
root() {
  key "$1"
  openssl req -new -x509 -key "pki/$1.key" -sha256 -days 14 \
    -subj "/CN=$2" -addext 'basicConstraints=critical,CA:TRUE' \
    -addext 'keyUsage=critical,keyCertSign,cRLSign' -out "pki/$1.crt"
}
root server-root 'PingLabz Lab Server Root'
root other-root 'PingLabz Lab Other Root'
key server-intermediate
openssl req -new -key pki/server-intermediate.key \
  -subj '/CN=PingLabz Lab Server Intermediate' -out pki/server-intermediate.csr
cat > pki/intermediate.ext <<'EOF'
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
EOF
openssl x509 -req -in pki/server-intermediate.csr -CA pki/server-root.crt \
  -CAkey pki/server-root.key -set_serial 100 -days 7 -sha256 \
  -extfile pki/intermediate.ext -out pki/server-intermediate.crt
leaf() {
  local name="$1" dns="$2" serial="$3"
  key "$name"
  openssl req -new -key "pki/$name.key" -subj "/CN=$dns" -out "pki/$name.csr"
  cat > "pki/$name.ext" <<EOF
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature
extendedKeyUsage=serverAuth
subjectAltName=DNS:$dns
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
EOF
  openssl x509 -req -in "pki/$name.csr" -CA pki/server-intermediate.crt \
    -CAkey pki/server-intermediate.key -set_serial "$serial" -days 5 -sha256 \
    -extfile "pki/$name.ext" -out "pki/$name.crt"
  cat "pki/$name.crt" pki/server-intermediate.crt > "pki/$name-chain.pem"
}
leaf app app.test 101
leaf default default.test 102
openssl verify -CAfile pki/server-root.crt -untrusted pki/server-intermediate.crt pki/app.crt pki/default.crt

The final offline check should print pki/app.crt: OK and pki/default.crt: OK. Certificate dates depend on when you generate them; regenerate in a fresh directory if these short-lived certificates expire.

Each *-chain.pem contains the leaf followed by the intermediate. The trusted root stays in the client’s -CAfile; sending a root from the server would not make an unrelated client trust it.

Serve the right chain and a broken chain

Save this as tls-service.py. The SNI callback chooses app.test only when that exact name arrives. Missing or other names deliberately fall back to default.test. That fallback is this lab’s policy; another server may reject an unknown name instead.

#!/usr/bin/env python3
"""Tiny lab HTTP service: SNI selects app.test; every other name uses default.test."""
import argparse, json, pathlib, socket, ssl

parser = argparse.ArgumentParser()
parser.add_argument('--port', type=int, default=8443)
parser.add_argument('--leaf-only', action='store_true')
args = parser.parse_args()
pki = pathlib.Path('pki')

def context(name):
    ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
    ctx.minimum_version = ssl.TLSVersion.TLSv1_3
    ctx.maximum_version = ssl.TLSVersion.TLSv1_3
    cert = pki / (name + ('.crt' if args.leaf_only else '-chain.pem'))
    ctx.load_cert_chain(cert, pki / (name + '.key'))
    return ctx

default_ctx = context('default')
app_ctx = context('app')

def choose_context(sock, name, initial):
    selected = 'app' if name == 'app.test' else 'default'
    sock.context = app_ctx if selected == 'app' else default_ctx
    print(json.dumps({'event': 'sni', 'name': name, 'selected': selected}), flush=True)

default_ctx.sni_callback = choose_context
with socket.socket() as listener:
    listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    listener.bind(('192.0.2.20', args.port))
    listener.listen()
    print(json.dumps({'event': 'listening', 'port': args.port, 'leaf_only': args.leaf_only}), flush=True)
    while True:
        raw, peer = listener.accept()
        raw.settimeout(3)
        try:
            with default_ctx.wrap_socket(raw, server_side=True) as tls:
                request = tls.recv(4096)
                body = b'PingLabz TLS lab OK\n'
                tls.sendall(b'HTTP/1.0 200 OK\r\nContent-Length: ' + str(len(body)).encode() + b'\r\n\r\n' + body)
                print(json.dumps({'event': 'http', 'peer': peer, 'version': tls.version(), 'request': request.splitlines()[0].decode(errors='replace')}), flush=True)
                tls.unwrap()
        except (ssl.SSLError, OSError) as error:
            raw.close()
            print(json.dumps({'event': 'error', 'peer': peer, 'reason': str(error)}), flush=True)

Run the full-chain listener in one lab-host terminal and the leaf-only listener in another. Both terminals must be in the lab directory so the certificate paths resolve.

# Terminal 1: complete server chain
sudo ip netns exec plt-server python3 -u tls-service.py

# Terminal 2: deliberately missing intermediate
sudo ip netns exec plt-server python3 -u tls-service.py --port 8444 --leaf-only

Establish a strict working baseline

From a third lab-host terminal, send a real HTTP request. The timeout bounds the command, and -ign_eof keeps OpenSSL reading after the piped request ends so it can receive the response and TLS close notification.

printf 'GET / HTTP/1.0\r\nHost: app.test\r\nConnection: close\r\n\r\n' |
  sudo ip netns exec plt-client timeout 12s openssl s_client \
    -connect 192.0.2.20:8443 -tls1_3 \
    -servername app.test -verify_hostname app.test \
    -CAfile pki/server-root.crt -no-CApath -no-CAstore \
    -verify_return_error -showcerts -state -ign_eof
echo "s_client exit: $?"

Selected output from my baseline:

Certificate chain
 0 s:CN=app.test
   i:CN=PingLabz Lab Server Intermediate
 1 s:CN=PingLabz Lab Server Intermediate
   i:CN=PingLabz Lab Server Root
Verification: OK
Verify return code: 0 (ok)
HTTP/1.0 200 OK
PingLabz TLS lab OK
s_client exit: 0

That proves more than a TCP connection. The expected identity verified, the chain reached the explicit trusted root, and the application returned a response.

-showcerts shows the certificates the server sent. Treat it as a supplied certificate list, not proof that the list is trusted. The successful verification is a separate result.

Remove SNI while keeping the expected name

Repeat the baseline command, replacing -servername app.test with -noservername. Keep -verify_hostname app.test, the root, and the port unchanged.

The service selected its fallback certificate. These are the relevant client lines:

Certificate chain
 0 s:CN=default.test
   i:CN=PingLabz Lab Server Intermediate
verify error:num=62:hostname mismatch
Verify return code: 62 (hostname mismatch)
s_client exit: 1

The server log recorded "name": null, "selected": "default". The CA was still correct; the selected identity was wrong. The HTTP request still carried Host: app.test, but it could not get past the failed handshake.

Restore -servername app.test and the original command succeeds. For a real service, investigate the SNI sent by the client and the certificate assigned to that virtual host before replacing trust roots.

Send the right SNI but verify the wrong name

Now keep -servername app.test and change only -verify_hostname app.test to -verify_hostname wrong.test.

depth=0 CN=app.test
verify error:num=62:hostname mismatch
Verify return code: 62 (hostname mismatch)
s_client exit: 1

This has the same error number as the missing-SNI case, but the server supplied app.test. Its Subject Alternative Name covers app.test, not wrong.test.

Inspect the public certificate on the lab host:

openssl x509 -in pki/app.crt -noout -subject -issuer -ext subjectAltName

If the supplied certificate is the intended one, compare its SAN with the actual service name. Repair the certificate or client endpoint configuration as appropriate. Disabling hostname verification would conceal the mismatch.

Separate an untrusted root from a missing intermediate

First repeat the baseline on port 8443 with -CAfile pki/other-root.crt. The server still sends its leaf and intermediate, but the client trusts an unrelated root:

depth=1 CN=PingLabz Lab Server Intermediate
verify error:num=20:unable to get local issuer certificate
Verify return code: 20 (unable to get local issuer certificate)
s_client exit: 1

Return to -CAfile pki/server-root.crt, then change the destination to 192.0.2.20:8444. That listener sends only the leaf:

Certificate chain
 0 s:CN=app.test
   i:CN=PingLabz Lab Server Intermediate
depth=0 CN=app.test
verify error:num=20:unable to get local issuer certificate
s_client exit: 1

The error text alone cannot distinguish these cases. In the first, validation reached the supplied intermediate at depth 1 but could not reach a trusted issuer. In the second, validation stopped at the leaf at depth 0 because its intermediate was absent.

Confirm the missing piece offline:

# Fails: the trusted root cannot directly issue this leaf.
openssl verify -CAfile pki/server-root.crt \
  -verify_hostname app.test pki/app.crt

# Passes: provide the intermediate as an untrusted chain-building input.
openssl verify -CAfile pki/server-root.crt \
  -untrusted pki/server-intermediate.crt \
  -verify_hostname app.test pki/app.crt

The second command returned pki/app.crt: OK. -untrusted supplies candidate intermediate certificates; it does not turn them into trust anchors. The OpenSSL verification documentation explains the distinction.

For the leaf-only listener, repair the served chain by using the full-chain listener on 8443. The restored strict request returned HTTP 200 with verification code 0. On a deployed service, configure the leaf plus required intermediate certificates, then verify what the listener actually sends.

A 200 response can still come from an untrusted server

For one controlled comparison, use the baseline command with the unrelated root and remove only -verify_return_error.

My result was:

verify error:num=20:unable to get local issuer certificate
Verify return code: 20 (unable to get local issuer certificate)
HTTP/1.0 200 OK
PingLabz TLS lab OK
s_client exit: 0

OpenSSL reported the trust failure and continued as a diagnostic client. A script that checked only the process exit status or HTTP response would have called this healthy.

Keep -verify_return_error, the expected hostname, and the intended trust source together in a validation command. Inspect the final response as well. A timeout exit of 124 means the bounded command expired, not that a certificate verified.

Use the capture for the question it can answer

To check which SNI left the client, start this in another lab-host terminal, run one probe, then press Ctrl-C in the capture terminal:

sudo ip netns exec plt-server tcpdump --immediate-mode -U -n \
  -i eth0 -s 0 -w sni.pcap 'tcp port 8443 or tcp port 8444'

tshark -r - -Y 'tls.handshake.type == 1' -T fields \
  -e frame.number -e ip.src -e tcp.dstport \
  -e tls.handshake.extensions_server_name < sni.pcap

In the baseline capture, frame 4 was a ClientHello from 192.0.2.10 to port 8443 with SNI app.test. The missing-SNI capture had an empty SNI field. Both captures contained a TCP SYN and SYN-ACK before the TLS exchange.

This lab does not use Encrypted ClientHello. With TLS 1.3, the later certificate messages and alerts are encrypted, so these undecrypted captures cannot tell you which certificate or rejection reason was inside them. Use s_client output and server logs for that evidence. Packet captures establish transport progress and the visible ClientHello inputs.

Across the seven server-verification cases, each saved capture had matching file, captured, and filter-received packet counts with zero kernel drops. The final strict baseline passed again.

Choose the next check from the failed layer

Start with the address and port, then the supplied certificate, expected name, and chain-building inputs. A wrong fallback certificate points you toward SNI or virtual-host configuration. The intended certificate with the wrong SAN points toward the endpoint name or issuance. A missing issuer needs both the server-sent list and the client trust source checked.

If server verification succeeds but the service demands a client identity, continue with the mutual TLS rejection lab. That moves the verification decision to the server and reproduces three distinct client-certificate failures.

For the same certificate hostname and CA checks on a DNS endpoint, compare plain DNS, DoT, and DoH. Its companion DNSSEC chain investigation follows a signed parent DS to the child key check, keeping endpoint authentication and record validation visible.

Share Share Share Share Share Email

Read next