wget is the tool you reach for when the transfer has to finish without you watching it. It retries on its own, resumes where it stopped, follows links recursively and writes a progress log you can read afterward. That set of behaviors is why it is still installed on nearly every server despite curl being better at almost everything else.
This guide covers wget the way a network engineer uses it: as a reachability tester, a mirroring tool and an unattended fetcher on links that are not reliable. Every capture is real output from a Debian 13 host inside a Cisco Modeling Labs topology, pulling from an nginx server two hops away and a static site three hops away over OSPF-routed IOS XE. It is part of the Linux networking commands cluster.
The lab, and the version
The client is Debian 13 at 10.77.0.100. WEB1 (nginx) sits at 10.77.2.10 two hops away, SRV1 at 10.77.3.10 three hops away serving a small static tree and a 5 MB binary. Version first, because wget and wget2 are different programs with overlapping flags:
j@llmbits:~$ wget --version | head -3
GNU Wget 1.25.0 built on linux-gnu.
-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nlsThat feature line is worth a glance. A build without +https (still common in minimal container images and on old embedded gear) will silently fail on every TLS URL you give it.
The default behavior, and why it surprises people
Run wget with a URL and nothing else and it saves to a file in the current directory, printing a progress bar to stderr:
j@llmbits:~$ wget http://10.77.2.10/
--2026-08-19 08:02:30-- http://10.77.2.10/
Connecting to 10.77.2.10:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 896 [text/html]
Saving to: 'index.html'
0K 100% 26.1M=0s
2026-08-19 08:02:30 (26.1 MB/s) - 'index.html' saved [896/896]This is the opposite of curl, which writes to stdout. It is also why wget in a script quietly litters the working directory with index.html.1, index.html.2 and so on. Three flags fix that:
j@llmbits:~$ wget -qO - http://10.77.2.10/ | head -6
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
j@llmbits:~$ wget -q -O nginx-home.html http://10.77.2.10/ && ls -l nginx-home.html
-rw-rw-r-- 1 j j 896 Apr 7 04:37 nginx-home.html-q is quiet, -O - is "write to stdout", and -O name pins the output filename so repeated runs overwrite instead of accumulating.
The connectivity test nobody uses enough
--spider makes wget issue the request and check the response without downloading the body. Combined with -S it prints the full response headers, which makes it a perfectly good HTTP health check on a box where curl is not installed:
j@llmbits:~$ wget -S --spider http://10.77.2.10/
Spider mode enabled. Check if remote file exists.
--2026-08-19 08:02:30-- http://10.77.2.10/
Connecting to 10.77.2.10:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Server: nginx/1.29.8
Date: Wed, 19 Aug 2026 15:02:30 GMT
Content-Type: text/html
Content-Length: 896
Last-Modified: Tue, 07 Apr 2026 11:37:12 GMT
Connection: keep-alive
ETag: "69d4ec68-380"
Accept-Ranges: bytes
Length: 896 [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.Add -q and the exit status becomes the whole result, which is exactly what a monitoring script wants:
j@llmbits:~$ wget -q --spider http://10.77.2.10/ && echo UP || echo DOWN
UP
j@llmbits:~$ wget -q --spider --timeout=3 --tries=1 http://10.77.2.10:8080/ && echo UP || echo DOWN
DOWNNote the --tries=1. Without it, wget defaults to twenty retries, so a health check against a dead service can sit there for minutes before reporting the failure you already knew about.
Retries and timeouts are the whole point
The retry behavior is what makes wget worth keeping. It is also what makes it dangerous in a script if you leave it at defaults. Here is a fast failure against an address with no host behind it:
j@llmbits:~$ wget --tries=2 --timeout=3 http://10.77.9.9/
--2026-08-19 08:02:30-- http://10.77.9.9/
Connecting to 10.77.9.9:80... failed: No route to host.The router answered with an ICMP unreachable, so wget gave up immediately rather than burning the timeout. That is the same distinction covered in the curl article: an error that arrives fast is a network doing its job.
The four timing controls that matter, and sensible values for unattended use:
--tries=NDefault 20. Use 1 for health checks, 3 to 5 for real transfers, and --tries=0 only when you genuinely want infinite retries on a flaky WAN.
--timeout=NSets DNS, connect and read timeouts together. Setting only this covers the common cases; the three below let you split them.
--connect-timeout / --read-timeoutSplit them when the path is slow to establish but fine once running, such as a satellite or heavily loaded VPN link.
--waitretry=NBacks off between attempts, growing to N seconds. Stops a retry loop from hammering a server that is already struggling.
Rate limiting, so the download does not eat the link
Pulling an OS image down a branch circuit during business hours is a classic self-inflicted outage. --limit-rate is one flag and it works:
j@llmbits:~$ wget --limit-rate=300k -O big.bin http://10.77.3.10:8000/big.bin
--2026-08-19 08:02:30-- http://10.77.3.10:8000/big.bin
Connecting to 10.77.3.10:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5242880 (5.0M) [application/octet-stream]
Saving to: 'big.bin'
0K .......... .......... .......... .......... .......... 0% 1.58M 3s
500K .......... .......... .......... .......... .......... 10% 234K 16s
2500K .......... .......... .......... .......... .......... 49% 234K 9s
5100K .......... .......... 100% 37.3T=17s
2026-08-19 08:02:47 (303 KB/s) - 'big.bin' saved [5242880/5242880]Five megabytes at 303 KB/s measured against a 300 KB/s cap. The first line shows 1.58M because the limiter allows an initial burst before it settles, which is normal and worth knowing so you do not chase it.
That default dot progress bar is unreadable in a log file. Change the style rather than turning it off, so you keep the summary line:
j@llmbits:~$ wget --progress=dot:giga -O big.bin http://10.77.3.10:8000/big.bin 2>&1 | tail -6
Saving to: 'big.bin'
0K ..... 100% 2.64M=1.9s
2026-08-19 08:03:21 (2.64 MB/s) - 'big.bin' saved [5242880/5242880]Unlimited, the same file moved at 2.64 MB/s across three router hops. That is the ceiling this virtual path gives a single TCP flow, and it is a useful sanity number to have before anyone claims the download is slow.
Mirroring a site
Recursive retrieval is the feature curl does not have. Four flags cover almost every real use:
j@llmbits:~$ mkdir mirror && cd mirror && wget -r -np -k -nv http://10.77.3.10:8000/
2026-08-19 08:03:19 URL:http://10.77.3.10:8000/ [87/87] -> "10.77.3.10:8000/index.html" [1]
http://10.77.3.10:8000/robots.txt:
2026-08-19 08:03:19 ERROR 404: File not found.
2026-08-19 08:03:19 URL:http://10.77.3.10:8000/docs/ [285/285] -> "10.77.3.10:8000/docs/index.html" [1]
2026-08-19 08:03:19 URL:http://10.77.3.10:8000/about.html [47/47] -> "10.77.3.10:8000/about.html" [1]
2026-08-19 08:03:19 URL:http://10.77.3.10:8000/docs/backup.cfg [14/14] -> "10.77.3.10:8000/docs/backup.cfg" [1]
2026-08-19 08:03:19 URL:http://10.77.3.10:8000/docs/notes.txt [17/17] -> "10.77.3.10:8000/docs/notes.txt" [1]
FINISHED --2026-08-19 08:03:19--
Total wall clock time: 0.07s
Downloaded: 5 files, 450 in 0s (1.27 MB/s)Notice it fetched robots.txt first and got a 404. wget obeys robots.txt in recursive mode by default, which regularly confuses people whose mirror comes back empty. -e robots=off overrides it, on servers you own.
The -k flag rewrote the links so the copy browses offline:
j@llmbits:~$ cat ~/mirror/10.77.3.10:8000/index.html
<h1>PingLabz lab site</h1>
<a href="docs/index.html">docs</a><br>
<a href="about.html">about</a>The original said href="docs/". After conversion it points at the file that actually exists on disk.
-rRecursive. Default depth is 5 levels, controlled with -l N. -l 1 is usually what you meant.
-npNo parent. Without it, one link upward and you are mirroring the entire site instead of the subtree you asked for.
-kConvert links for local browsing. Applied after the crawl finishes, so an interrupted run leaves them unconverted.
-mMirror. Shorthand for -r -N -l inf --no-remove-listing. Powerful and easy to point at something enormous by accident, so always pair it with -np.
Batches, and only fetching what changed
A list of URLs in a file plus -i is the cleanest way to pull a set of firmware images or config backups in one pass:
j@llmbits:~$ cat urls.txt
http://10.77.2.10/
http://10.77.3.10:8000/about.html
http://10.77.3.10:8000/docs/notes.txt
j@llmbits:~$ wget -nv -P batch -i urls.txt
2026-08-19 08:03:21 URL:http://10.77.2.10/ [896/896] -> "batch/index.html" [1]
2026-08-19 08:03:21 URL:http://10.77.3.10:8000/about.html [47/47] -> "batch/about.html" [1]
2026-08-19 08:03:21 URL:http://10.77.3.10:8000/docs/notes.txt [17/17] -> "batch/notes.txt" [1]
FINISHED --2026-08-19 08:03:21--
Total wall clock time: 0.02s
Downloaded: 3 files, 960 in 0s (5.46 MB/s)-P dir sets the destination and -nv gives one line per file, which is the right verbosity for a cron job.
Timestamping with -N is how you make a repeated job cheap. It sends a conditional request and skips the transfer if the local copy is current:
j@llmbits:~$ wget -N http://10.77.2.10/
--2026-08-19 08:21:26-- http://10.77.2.10/
Connecting to 10.77.2.10:80... connected.
HTTP request sent, awaiting response... 304 Not Modified
File 'index.html' not modified on server. Omitting download.That 304 Not Modified is the server confirming your copy is current. Running an hourly sync of a large artifact repository with -N costs you one small request per file instead of the whole tree.
Resuming, and when it does not work
-c continues a partial download by asking for a byte range. It only works when the server supports ranges, which is signaled by the Accept-Ranges: bytes header you saw in the nginx response earlier. Servers that do not advertise it, including Python's built-in http.server, will restart the transfer from zero and wget will not warn you loudly about it. Check for the header before you rely on resume across a bad link.
Also worth knowing: -c combined with -O does not behave the way people expect, because -O truncates. If you want resume, let wget pick the filename or use -c with the file already in place under its natural name.
Headers, user agents and TLS
Some servers behave differently depending on who is asking. Both the user agent and arbitrary headers are one flag each:
j@llmbits:~$ wget -S --spider --user-agent='PingLabz-Probe/1.0' --header='X-Lab: phase4' http://10.77.2.10/ 2>&1 | head -8
Spider mode enabled. Check if remote file exists.
--2026-08-19 08:03:21-- http://10.77.2.10/
Connecting to 10.77.2.10:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Server: nginx/1.29.8
Date: Wed, 19 Aug 2026 15:03:21 GMT
Content-Type: text/htmlTLS validation is strict by default, and the failure message is more specific than most:
j@llmbits:~$ wget -O /dev/null https://10.77.3.10:8443/ 2>&1 | tail -4
Connecting to 10.77.3.10:8443... connected.
ERROR: The certificate of '10.77.3.10' is not trusted.
ERROR: The certificate of '10.77.3.10' doesn't have a known issuer.
The certificate's owner does not match hostname '10.77.3.10'
j@llmbits:~$ wget --no-check-certificate -qO - https://10.77.3.10:8443/ 2>/dev/null | head -4
<HTML><BODY BGCOLOR="#ffffff">
<pre>
s_server -accept 8443 -cert s.pem -wwwThree separate problems reported separately: untrusted, unknown issuer, and a name mismatch because the certificate carries a CN and we connected by IP. That is more useful than a single generic error, and it tells you which one to actually fix.
FAQ
Why does my recursive download come back empty?
Almost always robots.txt. wget honors it in recursive mode, so a Disallow: / stops the crawl before it starts. On a server you own, add -e robots=off. On one you do not own, respect it.
wget or curl for scripts?
wget when the job is "get this file, keep trying, do not bother me". curl when the job is "make this request and tell me exactly what happened". Health checks work with either; wget -q --spider is the shortest form and needs no extra flags to suppress output.
How do I stop wget from creating index.html.1?
Use -O name to pin the output file, or -N so it overwrites based on timestamp, or --no-clobber so it skips rather than duplicating. The default of numbering suffixes exists so an interactive user never loses data, and it is wrong for automation.
What is wget2 and should I use it?
A rewrite with HTTP/2, parallel connections and better compression support. It is faster on many small files. Flags mostly carry over but not entirely, so test before swapping it into an existing script. Most distributions still ship classic wget as wget.
Can wget send a POST?
Yes, with --post-data or --post-file, and you can set the content type with --header. It works, but for anything API-shaped curl gives you far better control over methods, status codes and response inspection.
How do I log an unattended download properly?
-o logfile (lowercase) redirects the progress output to a file, -a logfile appends. Combine with -b to background it, and --progress=dot:giga so the log stays readable. Do not confuse -o with -O: one is the log, the other is the output file.
Key takeaways
wget -q --spider URLis a complete HTTP health check in one line, and the exit status is the whole answer. Add--tries=1or it will retry twenty times before telling you.- Default retries are 20 and default recursion depth is 5. Both are wrong for automation. Set
--tries,--timeoutand-lexplicitly. --limit-rateis the difference between a firmware pull and a branch outage. Expect a short burst above the cap before it settles.- Recursive mode obeys robots.txt. An empty mirror is usually that, not a network problem.
-npbelongs on every recursive command, especially with-m, or one upward link turns a subtree into the whole site.-Nmakes repeated syncs cheap: a 304 response instead of a transfer.-conly resumes if the server sendsAccept-Ranges: bytes. Verify that header before you depend on resume over a bad link.
Next in this cluster: netcat for checking ports that are not HTTP at all, socat for relaying between them, and SSH tunneling for reaching things from the wrong side of a firewall. The full toolset is indexed on the Linux networking commands pillar.