3

I am downloading about 5000 .htm pages from a website using Wget for Windows. The page sizes are between 500 KB and 1,5 MB

It takes 35 minutes to download 1000 files, and that means like 3 hours for 5,000 files. Is there a faster alternative? Maybe some program that can download multiple files at the time and maybe downloading the files in the memory and then saving them on disk in batches of 100 or 300 files?

The script looks like this:

wget -O 10001.htm https://www.wowhead.com/item=10001
wget -O 10002.htm https://www.wowhead.com/item=10002
..
wget -O 11000.htm https://www.wowhead.com/item=11000

I was using wowhead.com just as an example, because they have a lot of files, so I could create a good example. I don't need those files from wowhead, but sometimes I have to download thousands of files with similar sizes from other websites.

Improve this question
5
  • 5
    wget is as good as it gets, ba-dum-tssss! You need concurrency.
    – MonkeyZeus
    Commented Jul 8, 2021 at 14:05
  • 1
    Starting for each download an own process slows down the whole process as open https connections can not be reused. Better learn a script language like Python and write a short program.
    – Robert
    Commented Jul 8, 2021 at 20:47
  • Consider the load you're putting on the server, too. If it's a well-configured server serving static files it might barely notice. But if the pages are dynamically generated, downloading many pages quickly may noticeably affect the web site's performance. That's not very nice, and if the site operators notice it may get you in to trouble (up to criminal charges for a DDoS attack). I wouldn't go crazy with concurrency if I were you.
    – marcelm
    Commented Jul 9, 2021 at 9:29
  • @MonkeyZeus - in aria2 the concurency is enabled by default and it is set to 5, I tried "aria2c -i links.txt" and "wget -i links.txt" and the result was 5 minutes versus 25 minutes. That means aria2c is 5 times faster by default, without bothering to split the download into 5 input files or 5 scripts
    – Joe Jobs
    Commented Jul 10, 2021 at 2:21
  • So aria2 doesn't utilize wget to make requests?
    – MonkeyZeus
    Commented Jul 10, 2021 at 11:04

4 Answers 4

Reset to default
10

Possibly consider using aria2 instead.

It comes with a bunch of features that make it superior to wget for this type of usage, including:

  • Support for running multiple downloads in parallel, optionally with per-download bandwidth limiting (this is a huge performance improvement if dealing with lots of small files, and also if the server enforces a per-connection bandwidth limiting (which is rather common for big file-hosting sites)).
  • Support for using multiple connections to download a single large file, if the web server supports it.
  • Support for resuming partial downloads if the web server supports it.
  • Support for pre-allocating space for files being downloaded (helps to make sure if dealing with lots of files that you don’t run yourself out of disk space, and can sometimes speed up actually writing the data to disk).
  • Built-in support for HTTP, HTTPS, FTP, FTPS, SFTP, BitTorrent, and Metalink.
  • The ability to run as a service and be controlled remotely via RPC, with a handful of third-party GUI interfaces making use of this (my personal recommendation for this would be AriaNg, which is a nice web UI for it).

It’s also completely free and has good cross-platform support.

Improve this answer
8
  • 1
    As far as aria2 GUIs go, I'd recommend uGet. On the other hand, control via RPC is a great option for automated use too.
    – val - disappointed in SE
    Commented Jul 8, 2021 at 16:39
  • 1
    @valisstillwithMonica uGet is a rather nice one too. I hadn’t listed it since last I knew it was only available for UNIX-like systems, but it looks like in the long time since I last looked they added Windows support.
    – Austin Hemmelgarn
    Commented Jul 8, 2021 at 21:08
  • 1
    The most relevant feature here would be HTTP pipelining to reuse the same connection for multiple small requests, avoiding TCP connection startup overhead. (wowhead is a database of world of warcraft items, spells, etc, so those URLs are just small HTML.) Small enough to even have multiple requests in flight over one connection, given latency x bandwidth product. But yeah multiple connections in parallel might also help.
    – Peter Cordes
    Commented Jul 9, 2021 at 7:11
  • 1
    @JoeJobs The web server itself may not support it. It’s technically optional functionality, and some servers explicitly turn it off.
    – Austin Hemmelgarn
    Commented Jul 9, 2021 at 15:42
  • 2
    @JoeJobs: I think curl supports it (curl.se/docs/manpage.html). e.g. curl --remote-name 'https://www.wowhead.com/item=[10001-11000]' should do it if the server supports it. (I forget if there's a way to specify a text transformation to generate local file names from URLs, but that is the correct syntax for specifying multiple URLs via a counter pattern. You could just do a local file pattern-rename later separately.)
    – Peter Cordes
    Commented Jul 9, 2021 at 18:03
7

Because the requests are independent, you can divide the script into smaller scripts that will execute concurrently. Making four approximately equal scripts will likely run in 1/4 the current time. As the number of simultaneous requests gets large, you will be limited on your end or by the host.

Improve this answer
1
  • Indeed, I did split the batch file in 5 files, with 200 downloads in each file, and I got the 1000 files in 7 minutes, down from 35 minutes, so it was indeed 5 times faster
    – Joe Jobs
    Commented Jul 10, 2021 at 1:05
4

It looks like you're starting a new instance of wget for each download. I usually store all of the URLs in a text file (one URL per line) and use wget's -i (input file) flag.

Something like: wget --no-clobber -i urls.txt

Improve this answer
3
  • Indeed, I tried that and it reduced the download time from 35 minutes to 25 minutes. Simply by not starting and stopping wget 1000 times. The problem is that this way I can't set the file name for each download, which is not an issue in the above particular wowhead example but it is a problem with other downloads.
    – Joe Jobs
    Commented Jul 9, 2021 at 0:59
  • @JoeJobs: And maybe by not opening a new TCP connection for each URL, if wget can use HTTP pipelining and the wowhead servers support it. And at least not needing a separate DNS lookup for each URL.
    – Peter Cordes
    Commented Jul 9, 2021 at 7:07
  • I just found this - CURL SAYS BYE BYE TO PIPELINING APRIL 6, 2019 DANIEL STENBERG - daniel.haxx.se/blog/2019/04/06/curl-says-bye-bye-to-pipelining - it says HTTP/2 multiplexing is better than HTTP Pipelining
    – Joe Jobs
    Commented Jul 9, 2021 at 20:08
3

You can use wget but to put it in background like this:

wget -O 10001.htm --background https://www.wowhead.com/item=10001
wget -O 10002.htm --background https://www.wowhead.com/item=10002
...

This will run all of them in background (and they will run on the same time) so you will speedup your process

Improve this answer
4
  • 3
    Won't this possibly spawn 500o processes (nearly) at the same time, all running at the same time .....
    – albert
    Commented Jul 8, 2021 at 8:19
  • @albert, IMHO no. I think to the moment OP start process 100 the first process/download will end (just estimation). Moreover I have on my Windows 11 currently >300 processes :)
    – Romeo Ninov
    Commented Jul 8, 2021 at 8:32
  • 11
    I'd worry more about triggering anti-DDOS measures of site than hitting process limit here.
    – val - disappointed in SE
    Commented Jul 8, 2021 at 12:00
  • 1
    @valisstillwithMonica, this is also possible. But OP can create appropriate size of batches to avoid it, for example 5 or 10 simultaneous downloads. And for the record this will be DoS "attack", requests will come from one source.
    – Romeo Ninov
    Commented Jul 8, 2021 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.