I am downloading about 5000 .htm pages from a website using Wget for Windows. The page sizes are between 500 KB and 1,5 MB
It takes 35 minutes to download 1000 files, and that means like 3 hours for 5,000 files. Is there a faster alternative? Maybe some program that can download multiple files at the time and maybe downloading the files in the memory and then saving them on disk in batches of 100 or 300 files?
The script looks like this:
wget -O 10001.htm https://www.wowhead.com/item=10001 wget -O 10002.htm https://www.wowhead.com/item=10002 .. wget -O 11000.htm https://www.wowhead.com/item=11000
I was using wowhead.com just as an example, because they have a lot of files, so I could create a good example. I don't need those files from wowhead, but sometimes I have to download thousands of files with similar sizes from other websites.