3

I am downloading about 5000 .htm pages from a website using Wget for Windows. The page sizes are between 500 KB and 1,5 MB

It takes 35 minutes to download 1000 files, and that means like 3 hours for 5,000 files. Is there a faster alternative? Maybe some program that can download multiple files at the time and maybe downloading the files in the memory and then saving them on disk in batches of 100 or 300 files?

The script looks like this:

wget -O 10001.htm https://www.wowhead.com/item=10001
wget -O 10002.htm https://www.wowhead.com/item=10002
..
wget -O 11000.htm https://www.wowhead.com/item=11000

I was using wowhead.com just as an example, because they have a lot of files, so I could create a good example. I don't need those files from wowhead, but sometimes I have to download thousands of files with similar sizes from other websites.

5
  • 5
    wget is as good as it gets, ba-dum-tssss! You need concurrency.
    – MonkeyZeus
    Jul 8 at 14:05
  • 1
    Starting for each download an own process slows down the whole process as open https connections can not be reused. Better learn a script language like Python and write a short program.
    – Robert
    Jul 8 at 20:47
  • Consider the load you're putting on the server, too. If it's a well-configured server serving static files it might barely notice. But if the pages are dynamically generated, downloading many pages quickly may noticeably affect the web site's performance. That's not very nice, and if the site operators notice it may get you in to trouble (up to criminal charges for a DDoS attack). I wouldn't go crazy with concurrency if I were you.
    – marcelm
    Jul 9 at 9:29
  • @MonkeyZeus - in aria2 the concurency is enabled by default and it is set to 5, I tried "aria2c -i links.txt" and "wget -i links.txt" and the result was 5 minutes versus 25 minutes. That means aria2c is 5 times faster by default, without bothering to split the download into 5 input files or 5 scripts
    – Joe Jobs
    Jul 10 at 2:21
  • So aria2 doesn't utilize wget to make requests?
    – MonkeyZeus
    Jul 10 at 11:04
10

Possibly consider using aria2 instead.

It comes with a bunch of features that make it superior to wget for this type of usage, including:

  • Support for running multiple downloads in parallel, optionally with per-download bandwidth limiting (this is a huge performance improvement if dealing with lots of small files, and also if the server enforces a per-connection bandwidth limiting (which is rather common for big file-hosting sites)).
  • Support for using multiple connections to download a single large file, if the web server supports it.
  • Support for resuming partial downloads if the web server supports it.
  • Support for pre-allocating space for files being downloaded (helps to make sure if dealing with lots of files that you don’t run yourself out of disk space, and can sometimes speed up actually writing the data to disk).
  • Built-in support for HTTP, HTTPS, FTP, FTPS, SFTP, BitTorrent, and Metalink.
  • The ability to run as a service and be controlled remotely via RPC, with a handful of third-party GUI interfaces making use of this (my personal recommendation for this would be AriaNg, which is a nice web UI for it).

It’s also completely free and has good cross-platform support.

8
  • 1
    As far as aria2 GUIs go, I'd recommend uGet. On the other hand, control via RPC is a great option for automated use too. Jul 8 at 16:39
  • 1
    @valisstillwithMonica uGet is a rather nice one too. I hadn’t listed it since last I knew it was only available for UNIX-like systems, but it looks like in the long time since I last looked they added Windows support. Jul 8 at 21:08
  • 1
    The most relevant feature here would be HTTP pipelining to reuse the same connection for multiple small requests, avoiding TCP connection startup overhead. (wowhead is a database of world of warcraft items, spells, etc, so those URLs are just small HTML.) Small enough to even have multiple requests in flight over one connection, given latency x bandwidth product. But yeah multiple connections in parallel might also help. Jul 9 at 7:11
  • 1
    @JoeJobs The web server itself may not support it. It’s technically optional functionality, and some servers explicitly turn it off. Jul 9 at 15:42
  • 2
    @JoeJobs: I think curl supports it (curl.se/docs/manpage.html). e.g. curl --remote-name 'https://www.wowhead.com/item=[10001-11000]' should do it if the server supports it. (I forget if there's a way to specify a text transformation to generate local file names from URLs, but that is the correct syntax for specifying multiple URLs via a counter pattern. You could just do a local file pattern-rename later separately.) Jul 9 at 18:03
7

Because the requests are independent, you can divide the script into smaller scripts that will execute concurrently. Making four approximately equal scripts will likely run in 1/4 the current time. As the number of simultaneous requests gets large, you will be limited on your end or by the host.

1
  • Indeed, I did split the batch file in 5 files, with 200 downloads in each file, and I got the 1000 files in 7 minutes, down from 35 minutes, so it was indeed 5 times faster
    – Joe Jobs
    Jul 10 at 1:05
4

It looks like you're starting a new instance of wget for each download. I usually store all of the URLs in a text file (one URL per line) and use wget's -i (input file) flag.

Something like: wget --no-clobber -i urls.txt

3
  • Indeed, I tried that and it reduced the download time from 35 minutes to 25 minutes. Simply by not starting and stopping wget 1000 times. The problem is that this way I can't set the file name for each download, which is not an issue in the above particular wowhead example but it is a problem with other downloads.
    – Joe Jobs
    Jul 9 at 0:59
  • @JoeJobs: And maybe by not opening a new TCP connection for each URL, if wget can use HTTP pipelining and the wowhead servers support it. And at least not needing a separate DNS lookup for each URL. Jul 9 at 7:07
  • I just found this - CURL SAYS BYE BYE TO PIPELINING APRIL 6, 2019 DANIEL STENBERG - daniel.haxx.se/blog/2019/04/06/curl-says-bye-bye-to-pipelining - it says HTTP/2 multiplexing is better than HTTP Pipelining
    – Joe Jobs
    Jul 9 at 20:08
3

You can use wget but to put it in background like this:

wget -O 10001.htm --background https://www.wowhead.com/item=10001
wget -O 10002.htm --background https://www.wowhead.com/item=10002
...

This will run all of them in background (and they will run on the same time) so you will speedup your process

4
  • 3
    Won't this possibly spawn 500o processes (nearly) at the same time, all running at the same time .....
    – albert
    Jul 8 at 8:19
  • @albert, IMHO no. I think to the moment OP start process 100 the first process/download will end (just estimation). Moreover I have on my Windows 11 currently >300 processes :) Jul 8 at 8:32
  • 11
    I'd worry more about triggering anti-DDOS measures of site than hitting process limit here. Jul 8 at 12:00
  • 1
    @valisstillwithMonica, this is also possible. But OP can create appropriate size of batches to avoid it, for example 5 or 10 simultaneous downloads. And for the record this will be DoS "attack", requests will come from one source. Jul 8 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.