4

I have some 100,000 URLs stored in a text file, I need a multi-threaded downloader to download them all, preferably one that retries failed downloads. I use Windows 8.1.

I don't want the downloader to verify the validity of the URLs or check the sizes of the target files beforehand, it's time-consuming when the files are many and almost certainly crashes the downloader.
However, most popular downloaders do load all the URLs and check them before the download starts.

The closest I have now is "Mass Download", it loads URLs on the fly, everything seems spot-on except it doesn't fetch the right files (could not find out why).

They are webpages like http://kaze.shinshomap.info/series/ishigaki/8.html, I have no idea at all why Mass Download couldn't handle them, most download managers did quite well, except they crash before the download even starts as the number of files increases.

Do you have in mind any other downloaders that could download in bulk?

Improve this question
0

3 Answers 3

Reset to default
7

wget -c -i yourfile.txt should work. Run it once to download everything from yourfile.txt, which should be a text file with each url seperated by a new line, and run it again to make sure everything is done. If it crashes, just run it again.

You might choose to adjust the command line options to taste.

Improve this answer
1
  • 2
    Wget isn't standard on windows. Finding a port is trivial - I use the one that comes with gow
    – Journeyman Geek
    Apr 20, 2015 at 12:38
4

Use HtTrack, define a project and dump your list of URLs into it, tell it to not follow links so that only your URLs are downloaded.

It collects all the files in subfolders corresponding to the URLs, which may or may not be what you want.

It also has filters in case you only want certain file types.

Improve this answer
0
4

I used JDownloader in the past. Can also check if the file is still online and can interact with the clipboard (if you copy a link, it automaticly recognizes it without pasting)

And it's free :-)

Improve this answer
2
  • 1
    There actually already is a (BETA) JDownloader 2. But the OP asked for a downloader that does not verify the URL beforehand. I use JDownloader myself very often, but I'm not sure if there is a setting to disable URL-validation.
    – Alex
    Apr 22, 2015 at 7:32
  • I must have read too fast, I thought the OP didn't needed it but wouldn't mind if it did.
    – diedie2
    Apr 22, 2015 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.