How to download a website's assets into a single archive for offline viewing
There are moments when you need to view a specific website even if your internet connection drops. This tool downloads the HTML, stylesheets, images, and JavaScript of a desired site and bundles them into a single archive. The result is an offline webpage that can be opened locally.
How wget and archiver work together
The tool runs the wget command in the background to collect all site elements. It uses the --mirror option to recursively download subpages and --convert-links to convert links to relative paths, allowing navigation offline. It also uses --page-requisites to fetch the CSS and images required to display the pages. The collected files are compressed using the archiver library and then sent to the user via a socket channel. This process proceeds as the server communicates in real time with the client.
Safety mechanisms to limit download scope
Accepting files without limit can lead to insufficient disk space. To prevent this, you can set the DOWNLOAD_QUOTA environment variable. The default is 100MB, which is passed to wget to ensure a single request does not exceed this size. This setting prevents the server from consuming excessive resources.
Prerequisites for execution
To run this tool, you need Node.js 16 or higher and wget installed on your system. The installation command for wget varies by operating system. Use apt install wget on Debian or Ubuntu, brew install wget on macOS, and winget install JernejSimoncic.Wget on Windows. After cloning the repository, run npm install and npm start to start the service on local port 3000. You can also use a different port by changing the PORT environment variable.
Points to check before use
This tool focuses on replicating the entire structure of a specific website. Therefore, if the target site is vast or has a lot of dynamic content, it may generate files larger than expected. You must also ensure that wget is correctly installed in the system path. Without wget, the download function will not work at all. If you plan to deploy to a cloud environment, you must configure an image or environment that includes wget.
