/commands/wget/wget-recursive-download-directory
wget -r -np -nH --cut-dirs: download all files from a web directory
- tool
- wget
- versions
- GNU Wget 1.x
- os
- linux, macos, windows
wget -r -np -nH --cut-dirs=2 -R "index.html*" https://example.com/pub/data/ downloads every file below /pub/data/ from a server that shows a directory listing. -r follows the links recursively, -np (--no-parent) never goes above the starting directory, -nH drops the example.com/ folder, --cut-dirs=2 drops pub/data/ from the local paths, and -R "index.html*" deletes the listing pages after wget has read their links. Recursion stops at 5 levels by default; raise it with -l.
Commands
# All files below /pub/data/, saved without the host and pub/data folders
wget -r -np -nH --cut-dirs=2 -R "index.html*" https://example.com/pub/data/
# Only some file types, at most 2 levels deep, into ./downloads
wget -r -np -nH --cut-dirs=2 -l 2 -A "*.pdf,*.csv" -P downloads https://example.com/pub/data/
# Everything flat in one directory, no subfolders
wget -r -np -nd -R "index.html*" -P flat https://example.com/pub/data/The trailing / on the URL matters: without it wget treats data as a file and -np may be computed from /pub/ instead.
How the local path is built
| Options | File https://example.com/pub/data/2026/a.csv is saved as |
|---|---|
| -r | example.com/pub/data/2026/a.csv |
| -r -nH | pub/data/2026/a.csv |
| -r -nH --cut-dirs=1 | data/2026/a.csv |
| -r -nH --cut-dirs=2 | 2026/a.csv |
| -r -nd | a.csv (all files in one directory, duplicates get .1, .2) |
| -r -nH --cut-dirs=2 -P dl | dl/2026/a.csv |
Options
| Option | Effect |
|---|---|
| -r, --recursive | Follow links recursively |
| -l N, --level=N | Maximum depth (default 5; 0 or inf for no limit) |
| -np, --no-parent | Do not ascend to the parent directory |
| -nH | No host-name directory |
| --cut-dirs=N | Ignore N directory components of the URL path |
| -nd | No directories at all |
| -A LIST / -R LIST | Accept or reject file name suffixes or patterns, comma-separated |
| --reject-regex=RE | Skip URLs matching a regex, for example listing sort links containing ?C= |
| -N | Re-run later and fetch only files that changed |
| --wait=1 | Pause between requests |
| -e robots=off | Ignore robots.txt (only where you are allowed to) |
Pitfalls
- Sort links in listings. Apache listings contain links such as
?C=M;O=Athat produce many extraindex.html?C=...files.-R "index.html*"removes them;--reject-regex '\?C='avoids requesting them at all. - Rejected pages are still downloaded. wget needs the HTML to find links, so it downloads listing pages and deletes them afterwards when they match
-R. - Large trees. A wrong URL or a missing
-npcan crawl far more than intended. Try first with-l 1, or with--spiderto list URLs without saving them. - Server load. Use
--waitand avoid parallel runs against small servers.
For a whole website with its images and stylesheets for offline browsing, use the mirror options instead: see wget --mirror. wget prints a FINISHED summary with the number of files and bytes at the end; count the result with find . -type f | wc -l.
FAQ
Why does wget download only index.html?
Either the server returns no directory listing (only a page without links to the files), or robots.txt forbids crawling and wget obeys it. Check the listing in a browser. If you are allowed to fetch the files, -e robots=off disables the robots.txt check; respect the site's rules.
How many --cut-dirs do I need?
Count the directories in the URL path you want to drop. For https://host/pub/data/2026/ with -nH, --cut-dirs=2 saves into 2026/ and --cut-dirs=3 saves the files directly into the current directory.
Why does wget go up into the parent directory?
The listing has a Parent Directory link and -np was not given. --no-parent keeps the crawl below the starting URL. The URL must end with / so wget treats it as a directory.