/commands/wget/wget-download-list-of-files
wget -i urls.txt: download a list of URLs from a file
- tool
- wget
- versions
- GNU Wget 1.x
- os
- linux, macos, windows
wget -i urls.txt downloads every URL listed in urls.txt, one URL per line, in order. Combine it with -P DIR to save the files into a directory, -nc to skip files that already exist (so the command can be re-run after an interruption), and -c to resume partial downloads. URLs given on the command line as well are fetched first. With -i -, wget reads the list from standard input.
Commands
# urls.txt: one URL per line
cat > urls.txt <<'EOF'
https://example.com/data/2026-01.csv
https://example.com/data/2026-02.csv
https://example.com/data/2026-03.csv
EOF
# Download all of them into ./data, skip files already present
wget -nc -P data -i urls.txt
# Resume partial files, wait 1 s between requests, log to a file
wget -c --wait=1 -o wget.log -P data -i urls.txt
# URLs from another command
grep -o 'https://[^"]*\.pdf' page.html | sort -u | wget -nc -P pdfs -i -example.com stands for the real server. Keep the list free of empty lines with spaces, and make sure each line is a complete URL.
Options for batch downloads
| Option | Effect |
|---|---|
| -i FILE, --input-file=FILE | Read URLs from FILE; - means standard input |
| -B URL, --base=URL | Resolve relative links in the input file against URL |
| -F, --force-html | Treat the input file as HTML and download the links it contains |
| -nc | Skip URLs whose file already exists |
| -c | Resume partial files |
| -P DIR | Save all files under DIR |
| --wait=SECONDS | Pause between requests, to be gentle with the server |
| --random-wait | Vary the pause between 0.5 and 1.5 times --wait |
| -t N, --tries=N | Number of attempts per URL (default 20) |
| -o LOG / -a LOG | Write or append the log to LOG instead of the terminal |
| -nv, --no-verbose | One line per file in the log |
Exit status after a batch
| Code | Meaning |
|---|---|
| 0 | No problems |
| 1 | Generic error |
| 2 | Parse error, for example a bad option or input file |
| 3 | File I/O error |
| 4 | Network failure |
| 5 | SSL verification failure |
| 6 | Username or password authentication failure |
| 7 | Protocol error |
| 8 | Server issued an error response, such as 404 or 403 |
When several kinds of error occur in one run, the lower-numbered code wins (0 and 1 excepted), as the manual states. A batch with one 404 among a hundred successes still exits 8, and the log shows which URL failed.
Pitfalls
- Windows line endings. A list saved with CRLF endings can make wget request URLs ending in
%0Dand get 404 errors. Convert it withdos2unix urls.txtortr -d '\r'. - Duplicate names. Two URLs ending in the same file name (
.../a/data.csv,.../b/data.csv) givedata.csvanddata.csv.1, or with-ncthe second is skipped. Use-xto keep the directory structure. - Rate limits. Servers may answer
429or403to fast batches; add--wait. - PowerShell 5.1 maps
wgettoInvoke-WebRequest; usewget.exe.
Check the result
Count the downloaded files against the list: wc -l < urls.txt and ls data | wc -l. With -nv -o wget.log, grep -c ERROR wget.log counts the failures.
FAQ
Can I read the URLs from standard input?
Yes, use - as the file name, for example grep '\.pdf$' links.txt | wget -i -. To read a file that is literally named -, write ./-.
How do I download the files in parallel?
GNU Wget 1.x downloads one file at a time. Split the work with xargs, for example xargs -n 1 -P 4 wget -q -nc < urls.txt, which runs four wget processes, or use a tool designed for parallel transfers.
How do I see which URLs failed?
Write a log with -o wget.log (or -a to append) and search it for ERROR lines. wget's exit status reports only the most serious error of the whole run, not which URLs it affected.