0

I have a large list of URLs (150k) and I want a software (similar to httrack) to download the index.html file from the URLs.

The problem with httrack (Mac) is that it doesn't support importing URLs from a text file, therefore I reach maximum input character for terminal and cannot add all the URLs.

Both Mac and Windows are fine, but I prefer Mac.

  • What operating system? – Nicolas Raoul Nov 4 '16 at 6:54
  • @NicolasRaoul both Mac and Windows are fine, but I prefer Mac – adel rahimi Nov 4 '16 at 6:55
  • Thanks! For each URL, do you want the entire website, or just index.html? – Nicolas Raoul Nov 4 '16 at 6:59
  • @NicolasRaoul the index.html file is enough I want to be able to download page content (all text) and then I can index them in search platforms (like solr) to extract information from them. – adel rahimi Nov 4 '16 at 7:04
  • @adelrahimi: Does it really have to be a file named index.html (not all servers use/provide these), or do you mean the files the URLs point to? Or are these URLs all from the same site, and you know that it uses index.html for every page? ·· How does your data look like? Could it be that it contains http://example.com/foo/bar/hello.html, but you want the tool download http://example.com/ (or http://example.com/index.html specifically)? – unor Nov 4 '16 at 8:01
3

wget command can be used for this

wget --input-file=list.txt

this will store results in index.html, index.html.1, index.html.2 files. If you want to store results separately (e.g. create folder for each link you parse) then use the following format

wget --input-file=list.txt --force-directories
2

Just install python, if you are on windows - no need on OS-X or most Linux installs, then open a terminal and type pip install requests to get the requests library on some platforms you will need to prefix with sudo and you are good to go.

Assuming that your url file has one line per url:

python

import requests
n = 0
with open('url_file.txt') as infile:
     n += 1
     for line in infile:
         resp = requests.get(line.strip())
         if resp.ok:
             with(open('index_%06d.html' % n, 'wb') as outfile:
                  outfile.write(resp.content)
         else:
              print('Bad URL', line)

You could also use the Beautiful Soup library to extract the data that you need from the pages.

All of the above is Free, Gratis & Open Source plus is cross platform.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.