How Linux Resolves Names: resolv.conf, systemd-resolved, nsswitch

Here is the sentence that ends more DNS arguments than any other: dig and ping do not resolve names the same way. dig speaks DNS to a nameserver. ping, curl, your browser and every other normal application go through the C library resolver, which consults a configuration file first and may never send a DNS packet at all.

So "but dig says it works" is not evidence that name resolution works. This article walks the whole path, in order, on a real host: nsswitch.conf, /etc/hosts, /etc/resolv.conf, the search list, and what changes when systemd-resolved gets involved. Every capture is genuine output from a Debian 13 host wired into a Cisco Modeling Labs topology with an authoritative server for pinglabz.lab at 10.77.3.10, including the resolver being deliberately broken and then fixed. It is part of the Linux networking commands cluster.

The order of operations

When an application calls getaddrinfo(), this is what happens:

dig, host and nslookup all skip steps 1 through 3 and jump straight to sending a packet. That is by design, and it is why they are the right tools for testing a nameserver and the wrong tools for testing whether an application will resolve a name.

What is in charge on this host

Start by finding out. Three commands answer it:

j@llmbits:~$ cat /etc/os-release | head -2
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"

j@llmbits:~$ ls -l /etc/resolv.conf
-rw-r--r-- 1 root root 78 Aug 16 15:32 /etc/resolv.conf

j@llmbits:~$ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 45.90.28.181
nameserver 45.90.30.181

j@llmbits:~$ systemctl is-active systemd-resolved
inactive

j@llmbits:~$ systemctl is-active NetworkManager
active

A regular file, not a symlink, with a "Generated by NetworkManager" comment and systemd-resolved inactive. So NetworkManager owns this file and rewrites it whenever a connection comes up. Whatever you hand-edit here is temporary, which is the first thing to know before you edit it.

The three arrangements you will meet in the wild:

nsswitch decides who gets asked

The hosts: line is short and every token on it matters:

j@llmbits:~$ grep -E '^(hosts|networks):' /etc/nsswitch.conf
hosts:          files mdns4_minimal [NOTFOUND=return] dns
networks:       files

Read left to right. files means /etc/hosts is consulted first. mdns4_minimal handles .local names via multicast DNS. [NOTFOUND=return] is an action item that says: if mDNS authoritatively says this name does not exist, stop, do not fall through to DNS. Then dns, which is the stub resolver.

That [NOTFOUND=return] is worth understanding because it is the reason .local names behave strangely on a network that also has a .local DNS zone. If you inherited an Active Directory domain named something.local, this line is why half your hosts cannot resolve it.

getent is the command that walks this whole chain, which makes it the honest test:

j@llmbits:~$ getent hosts pinglabz.com
178.128.137.126 pinglabz.com

j@llmbits:~$ getent ahosts pinglabz.com
178.128.137.126 STREAM pinglabz.com
178.128.137.126 DGRAM
178.128.137.126 RAW

j@llmbits:~$ getent hosts localhost
::1             localhost ip6-localhost ip6-loopback

Three things there. getent hosts is the plain lookup. getent ahosts shows the results per socket type, which is what getaddrinfo() actually returns to an application. And localhost resolved to ::1, not 127.0.0.1, straight out of /etc/hosts with IPv6 preferred. That last one causes real bugs when a service binds to 127.0.0.1 and a client connects to "localhost".

/etc/hosts beats DNS, and that is the whole trick

Add an entry and watch DNS become irrelevant:

j@llmbits:~$ sudo printf '10.77.2.10 www.pinglabz.lab\n' >> /etc/hosts

j@llmbits:~$ getent hosts www.pinglabz.lab
10.77.2.10      www.pinglabz.lab

j@llmbits:~$ ping -c 1 www.pinglabz.lab
PING www.pinglabz.lab (10.77.2.10) 56(84) bytes of data.
64 bytes from www.pinglabz.lab (10.77.2.10): icmp_seq=1 ttl=62 time=5.46 ms

Now change the file to something wrong and run both tools:

j@llmbits:~$ sudo sed -i 's|^10.77.2.10 www.pinglabz.lab$|10.99.99.99 www.pinglabz.lab|' /etc/hosts

j@llmbits:~$ getent hosts www.pinglabz.lab
10.99.99.99     www.pinglabz.lab

j@llmbits:~$ dig @10.77.3.10 www.pinglabz.lab +short
10.77.2.10

There it is, in four lines. The application layer says 10.99.99.99. DNS says 10.77.2.10. Both are correct answers to different questions, and the only one that matters to curl, ssh or your service is the first one. Any time somebody insists DNS is fine and the application disagrees, check /etc/hosts before you check anything else.

Two corollaries worth internalizing. A stale /etc/hosts entry from a migration three years ago will happily override a correct DNS record forever, with no TTL and no expiry. And there is no cache to flush: the file is read on every lookup, so an edit takes effect immediately for new lookups, though a long-running process that already resolved and cached the answer internally will not notice.

resolv.conf: nameservers, search and options

Point the stub at the lab server and the whole toolchain follows:

j@llmbits:~$ sudo printf 'nameserver 10.77.3.10\nsearch pinglabz.lab\noptions timeout:2 attempts:1\n' > /etc/resolv.conf

j@llmbits:~$ cat /etc/resolv.conf
nameserver 10.77.3.10
search pinglabz.lab
options timeout:2 attempts:1

j@llmbits:~$ getent hosts www.pinglabz.lab
2001:db8:77:2::10 www.pinglabz.lab

j@llmbits:~$ host srv1.pinglabz.lab
srv1.pinglabz.lab has address 10.77.3.10
srv1.pinglabz.lab has IPv6 address 2001:db8:77:3::10

j@llmbits:~$ dig www.pinglabz.lab +noall +answer +stats
www.pinglabz.lab.	3600	IN	A	10.77.2.10
;; Query time: 8 msec
;; SERVER: 10.77.3.10#53(10.77.3.10) (UDP)

Note that getent returned the IPv6 address and dig returned the IPv4 one. Not a contradiction: getent hosts prints the first result from getaddrinfo(), and RFC 6724 address selection prefers IPv6 when both are available. dig with no type asks for A only. Two tools, two questions, two correct answers.

The directives that matter:

The search list and ndots

With search pinglabz.lab set, short names work through the C library and through the DNS tools that opt in:

j@llmbits:~$ getent hosts www
2001:db8:77:2::10 www.pinglabz.lab

j@llmbits:~$ host srv1
srv1.pinglabz.lab has address 10.77.3.10
srv1.pinglabz.lab has IPv6 address 2001:db8:77:3::10

j@llmbits:~$ dig +search www +noall +answer
www.pinglabz.lab.	3600	IN	A	10.77.2.10

Without +search, dig asks for the literal name and the server refuses because www. is not in its zone:

j@llmbits:~$ dig www +noall +comments
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 37458
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

This one difference produces a specific and very common confusion: "ping www works but dig www fails, so DNS is broken." It is not. dig does not apply the search list unless you tell it to.

ndots controls when the search list is used at all. The default of 1 means any name with no dot gets the search suffix. Raise it and names with one dot get suffixed too, which is how Kubernetes makes service.namespace resolve, and also why a pod with ndots:5 sends four failed queries before trying api.github.com as an absolute name. On this host, raising it to 2 changes what happens to a name with one dot in it:

j@llmbits:~$ sudo printf 'nameserver 10.77.3.10\nsearch pinglabz.lab\noptions ndots:2 timeout:2 attempts:1\n' > /etc/resolv.conf

j@llmbits:~$ getent hosts _sip._udp || echo 'no result, as expected'
no result, as expected

The trailing-dot rule is the escape hatch. api.github.com. with a dot on the end is absolute and skips the search list entirely, regardless of ndots. In a latency-sensitive container, that trailing dot is a real optimization.

What a broken resolver looks like from userspace

Point /etc/resolv.conf at an address that does not answer, and see how each layer reports it:

j@llmbits:~$ sudo printf 'nameserver 10.77.9.9\noptions timeout:1 attempts:1\n' > /etc/resolv.conf

j@llmbits:~$ getent hosts www.pinglabz.lab; echo "exit status: $?"
exit status: 2

j@llmbits:~$ ping -c 1 www.pinglabz.lab
ping: www.pinglabz.lab: Temporary failure in name resolution

j@llmbits:~$ curl -sS -m 5 http://www.pinglabz.lab/ ; echo "exit status: $?"
curl: (6) Could not resolve host: www.pinglabz.lab
exit status: 6

Three different messages for one root cause. "Temporary failure in name resolution" is EAI_AGAIN, meaning the resolver could not get an answer, as opposed to "Name or service not known" which is EAI_NONAME and means it got a definitive NXDOMAIN. Those two strings are worth telling apart when you are reading somebody else's logs: the first is an infrastructure problem, the second is a data problem.

Add a working server as a second entry and the lookup succeeds, but that is not the whole story:

j@llmbits:~$ sudo printf 'nameserver 10.77.9.9\nnameserver 10.77.3.10\noptions timeout:1 attempts:1\n' > /etc/resolv.conf

j@llmbits:~$ time getent hosts www.pinglabz.lab
2001:db8:77:2::10 www.pinglabz.lab

real	0m0.014s

Fourteen milliseconds, because 10.77.9.9 returned ICMP host unreachable immediately rather than silently dropping the packet. That is the friendly failure mode. A resolver that black-holes queries instead of rejecting them costs you the full timeout value on every single lookup, and with the default timeout:5 attempts:2 a two-server list can take twenty seconds to fail. This is the single most common cause of "the application is slow" turning out to be DNS.

options rotate spreads the load rather than always starting from the top:

j@llmbits:~$ sudo printf 'nameserver 10.77.9.9\nnameserver 10.77.3.10\noptions timeout:1 attempts:1 rotate\n' > /etc/resolv.conf

j@llmbits:~$ time getent hosts srv1.pinglabz.lab
2001:db8:77:3::10 srv1.pinglabz.lab

real	0m0.011s

Seeing the queries on the wire

The final proof of which path a lookup took is the packet capture. Three getent lookups, captured on the lab interface:

j@llmbits:~$ sudo tcpdump -ni ens224 -c 6 udp port 53
07:04:05.768598 IP 10.77.0.100.58780 > 10.77.3.10.53: 13719+ AAAA? www.pinglabz.lab. (34)
07:04:05.775268 IP 10.77.3.10.53 > 10.77.0.100.58780: 13719*- 1/0/0 AAAA 2001:db8:77:2::10 (78)
07:04:05.781269 IP 10.77.0.100.48997 > 10.77.3.10.53: 24690+ AAAA? srv1.pinglabz.lab. (35)
07:04:05.786623 IP 10.77.3.10.53 > 10.77.0.100.48997: 24690*- 1/0/0 AAAA 2001:db8:77:3::10 (80)
07:04:05.792679 IP 10.77.0.100.38405 > 10.77.3.10.53: 300+ AAAA? mail.pinglabz.lab. (35)
07:04:05.798155 IP 10.77.3.10.53 > 10.77.0.100.38405: 300*- 0/1/0 (122)

Read the flags on the answers. The * means authoritative. The 1/0/0 is answer, authority and additional counts. The last exchange is 0/1/0: zero answers, one authority record. That is NODATA, because mail.pinglabz.lab has an A record but no AAAA. A capture with 0/1/0 and no error is not a failure, it is a name that exists without the record type you asked for.

If you run this and see no packets at all, the lookup was answered by /etc/hosts or by a local cache and never left the machine. That absence is itself the diagnosis.

When systemd-resolved is in charge

Install and enable it on the same host and the whole arrangement changes:

j@llmbits:~$ systemctl is-active systemd-resolved
active

j@llmbits:~$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Aug 19 07:05 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

j@llmbits:~$ head -3 /etc/resolv.conf; tail -3 /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
nameserver 127.0.0.53
options edns0 trust-ad
search .

/etc/resolv.conf is now a symlink, and the only nameserver in it is a loopback stub. Editing that file is pointless. The real configuration lives per network link inside the daemon:

j@llmbits:~$ ss -lntp 2>/dev/null | grep ':53 '
LISTEN 0      4096   127.0.0.53%lo:53        0.0.0.0:*
LISTEN 0      4096      127.0.0.54:53        0.0.0.0:*

Two listeners. 127.0.0.53 is the stub, which applies routing rules, caching and DNSSEC. 127.0.0.54 is the pass-through, which forwards without the extras. Applications go to .53 through resolv.conf; something that wants raw forwarding uses .54.

resolvectl status is the command that replaces reading a file:

j@llmbits:~$ resolvectl status
Global
         Protocols: +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (ens192)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
       DNS Servers: 45.90.28.181 45.90.30.181
     Default Route: yes

Link 3 (ens224)
    Current Scopes: LLMNR/IPv4
         Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
     Default Route: no

Per-link configuration is the whole point of systemd-resolved, and it is what makes VPN split DNS work correctly instead of by luck.

Split DNS in one command

Give the lab interface its own nameserver and a routing domain, and names split by suffix automatically:

j@llmbits:~$ sudo resolvectl dns ens224 10.77.3.10
j@llmbits:~$ sudo resolvectl domain ens224 pinglabz.lab '~pinglabz.lab'

j@llmbits:~$ resolvectl status ens224
Link 3 (ens224)
    Current Scopes: DNS LLMNR/IPv4
         Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
       DNS Servers: 10.77.3.10
        DNS Domain: ~pinglabz.lab
     Default Route: no

The tilde prefix is what makes this a routing domain rather than a search domain: it means "send queries for this suffix to this link's servers" without adding it to the search list. Now watch the split work:

j@llmbits:~$ resolvectl query www.pinglabz.lab
www.pinglabz.lab: 2001:db8:77:2::10                         -- link: ens224
                  10.77.2.10                                -- link: ens224

-- Information acquired via protocol DNS in 11.1ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network

j@llmbits:~$ resolvectl query pinglabz.com
pinglabz.com: 178.128.137.126                               -- link: ens192

-- Information acquired via protocol DNS in 21.8ms.
-- Data from: network

Lab names went out ens224 to the lab server. Public names went out ens192 to the upstream resolvers. One command each, no resolv.conf gymnastics, and the -- link: annotation tells you which path was taken. That annotation is the feature; it is information nothing else on the box will give you.

j@llmbits:~$ resolvectl domain
Global:
Link 2 (ens192):
Link 3 (ens224): ~pinglabz.lab

j@llmbits:~$ resolvectl dns
Global:
Link 2 (ens192): 45.90.28.181 45.90.30.181
Link 3 (ens224): 10.77.3.10

The cache is real, and visible

resolvectl query tells you where each answer came from. Run the same lookup twice:

j@llmbits:~$ resolvectl query mail.pinglabz.lab
mail.pinglabz.lab: 10.77.3.20                               -- link: ens224

-- Information acquired via protocol DNS in 12.6ms.
-- Data from: network

j@llmbits:~$ resolvectl query mail.pinglabz.lab
mail.pinglabz.lab: 10.77.3.20                               -- link: ens224

-- Information acquired via protocol DNS in 3.0ms.
-- Data from: cache

12.6ms from the network, 3.0ms from the cache, and the source is stated outright. sudo resolvectl flush-caches clears it, and the daemon logs that it did:

j@llmbits:~$ sudo journalctl -u systemd-resolved -n 10 --no-pager -o cat
Using system hostname 'llmbits'.
Started systemd-resolved.service - Network Name Resolution.
ens192: Bus client set default route setting: yes
ens192: Bus client set DNS server list to: 45.90.28.181, 45.90.30.181
ens224: Bus client set DNS server list to: 10.77.3.10
ens224: Bus client set search domain list to: pinglabz.lab, ~pinglabz.lab
Flushed all caches.
Flushed all caches.

One honest note from this capture: resolvectl statistics asked for interactive polkit authentication over a non-interactive SSH session and refused to run, even under sudo:

j@llmbits:~$ resolvectl statistics
Failed to issue io.systemd.Resolve.Monitor.DumpStatistics() varlink call: io.systemd.InteractiveAuthenticationRequired

That is a polkit policy on this build, not a broken resolver. On a console session it works. Worth knowing before you put it in a monitoring script.

The stub is a cache, and dig can see it

With systemd-resolved running, dig with no @ goes to 127.0.0.53 like everything else. That gives you a clean three-way comparison:

j@llmbits:~$ dig www.pinglabz.lab +noall +answer +stats
www.pinglabz.lab.	3600	IN	A	10.77.2.10
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; MSG SIZE  rcvd: 61

j@llmbits:~$ dig @127.0.0.53 www.pinglabz.lab +noall +answer +stats
www.pinglabz.lab.	3599	IN	A	10.77.2.10
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; MSG SIZE  rcvd: 61

j@llmbits:~$ dig @10.77.3.10 www.pinglabz.lab +noall +answer +stats
www.pinglabz.lab.	3600	IN	A	10.77.2.10
;; SERVER: 10.77.3.10#53(10.77.3.10) (UDP)
;; MSG SIZE  rcvd: 77

Look at the TTLs. 3600 straight from the authority, 3599 from the stub cache one second later. A TTL that counts down is proof you are reading a cache, and it works whether that cache is on your loopback or three networks away. Details of what else dig exposes are in the complete dig guide.

One rough edge to be aware of: per-link routing applies to reverse lookups too, and not always the way you expect.

j@llmbits:~$ resolvectl query 10.77.3.10
10.77.3.10: resolve call failed: Name '10.3.77.10.in-addr.arpa' not found

The reverse name is under in-addr.arpa, which does not match the ~pinglabz.lab routing domain on ens224, so the query went upstream instead of to the lab server that actually holds the PTR. To fix it you would add ~3.77.10.in-addr.arpa as a routing domain on that link. It is a good illustration of the rule: routing domains match on the query name, and reverse lookups have completely different names from forward ones.

A diagnostic order that works

FAQ

Why does ping resolve a name that dig cannot?

Because ping goes through nsswitch.conf and dig does not. The name is almost certainly in /etc/hosts, being answered by mDNS, or being completed by the search list that dig ignores by default. Test with getent hosts, which takes the same path ping does.

How do I flush the DNS cache on Linux?

First establish that there is one. A bare glibc stub resolver does not cache at all, so there is nothing to flush and every lookup goes to the wire. With systemd-resolved, use sudo resolvectl flush-caches. With nscd, sudo nscd -i hosts. With dnsmasq as a local cache, restart it. Long-running applications may also hold their own in-process cache that none of these touch, which is a genuinely different problem.

My changes to /etc/resolv.conf keep disappearing. Why?

Something owns the file. Read the first line: if it says NetworkManager, configure DNS with nmcli con mod <name> ipv4.dns ... instead. If it is a symlink into /run/systemd, use resolvectl or drop a config file in /etc/systemd/resolved.conf.d/. If dhclient is writing it, the fix is supersede domain-name-servers in dhclient.conf. Editing the generated file always loses.

What is 127.0.0.53 and why is it my only nameserver?

It is the systemd-resolved stub listener. Real upstream servers are configured per link inside the daemon, and resolv.conf just points every application at the stub. Run resolvectl status to see the servers actually in use.

What does ndots:5 do in Kubernetes, and why is it slow?

It makes any name with fewer than five dots get the search list applied first. That is what lets service.namespace resolve inside a cluster. The cost is that an external name like api.github.com (two dots) generates several failed cluster-internal queries before the absolute lookup is tried. Appending a trailing dot to external names in your config skips the whole search sequence.

Why do I get an IPv6 address when I wanted IPv4?

RFC 6724 address selection prefers IPv6 when a name has both A and AAAA records and the host has a usable IPv6 address. That is why getent hosts shows the AAAA and dig with no type shows the A. If IPv6 is configured but not actually routed, you get connection attempts that hang before falling back, which is where getent ahosts and a careful look at the interface addressing earn their keep.

Which config wins if both systemd-resolved and NetworkManager are running?

They cooperate rather than compete, in the normal case. NetworkManager learns DNS servers from DHCP or a profile and hands them to systemd-resolved over D-Bus, per link. You can see exactly that happening in the journal capture above: ens192: Bus client set DNS server list to: .... The daemon holds the configuration, NetworkManager supplies it.

Key takeaways

• dig tests a nameserver. getent hosts tests name resolution. They are different questions and only the second one matches what your application will do.
• nsswitch.conf decides the order. files before dns means /etc/hosts wins, permanently, with no TTL and no cache to flush.
• Read the first line of /etc/resolv.conf and check whether it is a symlink. That tells you who owns DNS on the host and therefore where to make a change that sticks.
• A resolver that silently drops queries costs you timeout x attempts x servers on every lookup. One that rejects them fails in milliseconds. This is the usual cause of "slow application" tickets.
• "Temporary failure in name resolution" is EAI_AGAIN, an infrastructure problem. "Name or service not known" is EAI_NONAME, a data problem.
• dig ignores the search list unless you pass +search. That alone explains most "works in ping, fails in dig" reports.
• With systemd-resolved, resolvectl status replaces reading a file, and resolvectl query tells you which link answered and whether it came from cache.
• A tilde-prefixed routing domain sends a suffix to one link's servers without adding it to the search list. It matches on the query name, so reverse lookups need their own in-addr.arpa entry.
• A TTL that counts down means you read a cache. A TTL at its configured value plus the aa flag means you reached an authority.

For the query tools themselves: dig for protocol detail, host for one-line answers and scriptable exit codes, and nslookup for interactive work and cross-platform habit. For the offensive view of the same infrastructure, see the DNS enumeration cluster. Everything in this cluster is indexed on the Linux networking commands pillar.