/commands/wget/wget-mirror-website
wget --mirror --convert-links --page-requisites: download a website for offline use
The commands on this page have not been executed by our verification runner yet.updated
- tool
- wget
- versions
- GNU Wget 1.x
- os
- linux, macos, windows
wget --mirror --page-requisites --convert-links --adjust-extension --no-parent https://example.org/docs/ downloads every page under /docs/ together with the images, stylesheets and scripts they use, and rewrites the links so the copy works offline from your disk. --mirror turns on recursion with infinite depth and timestamping, --no-parent keeps wget from climbing above the start directory, and the files are saved under a folder named after the host.
Command
wget --mirror --page-requisites --convert-links --adjust-extension --no-parent \
--wait=1 --random-wait \
https://example.org/docs/Short form: wget -m -p -k -E -np -w 1 https://example.org/docs/. Open example.org/docs/index.html in a browser afterwards.
Options
| Option | Effect |
|---|---|
| -m, --mirror | Same as -r -N -l inf --no-remove-listing: recursive, timestamps, unlimited depth |
| -p, --page-requisites | Also download images, CSS and scripts needed to display each page |
| -k, --convert-links | After the download, rewrite links to point to the local copies |
| -E, --adjust-extension | Save HTML and CSS with .html/.css extensions so they open locally |
| -np, --no-parent | Never go to the parent directory of the start URL |
| -l N, --level=N | Limit the recursion depth (default 5 with -r, unlimited with -m) |
| -w N, --wait=N / --random-wait | Pause between requests to reduce load on the server |
| -H, --span-hosts with -D domains | Follow links to other hosts, limited to the listed domains |
| -R / -A | Reject or accept file name patterns, for example -R "*.zip,*.iso" |
| -nH, --cut-dirs=N | Do not create the host directory, drop N leading directories |
Things to know
- wget follows
robots.txtduring recursive downloads.-e robots=offdisables that; use it only on sites you are allowed to copy. - Pages built in the browser by JavaScript are saved as their initial HTML; wget does not run scripts, so such sites often come out empty.
- Rerunning the same
--mirrorcommand updates the copy: thanks to timestamping (-N) only files that changed on the server are downloaded again. - Images on a CDN host are skipped unless you add
-H -D cdn.example.org,example.org. - Logged-in areas need cookies:
--load-cookies cookies.txtwith a cookie file exported from the browser.
Sources
Related pages
flags · wgetwget -c (--continue): resume a partially downloaded filewget -c resumes an interrupted download by requesting only the missing bytes with an HTTP Range header. It needs server support; --tries retries on failure.flags · wgetwget -O file (--output-document): save the download under another namewget -O NAME saves the download as NAME instead of the URL file name, and -O - writes it to stdout. Capital -O is the output file; lowercase -o is the log.errors · wgetwget: unable to resolve host addresswget prints unable to resolve host address when the DNS lookup fails: a typo, no network or DNS server, a required proxy, or a container without working DNS.commands · wgetwget -r -np -nH --cut-dirs: download all files from a web directorywget -r -np -nH --cut-dirs=2 -R 'index.html*' downloads every file under a web directory listing. How local paths are built, filters, depth and pitfalls.