3

I need to save multiple pages to my harddisk.The pages have a pattern like this:

http://example.com/page1.html
http://example.com/page2.html
http://example.com/page3.html
.
.
.
.

I need to save a large number of webpages(about 5000).I tried using IDM but it hangs when I enter the links maybe due to the large number of pages.I also tried WinHTTrack but it saves the pages along with the unneccessary things such as pdf's from the site which i really don't need.

Can you recommend a software?thanks

4

You can use GNU wget:

  • free and open source
  • Linux, Windows, Mac (brew install wget)
  • CLI
  • tons of features such as the ability not to download PDFs.
2

I would suggest looking into the combination of python and scrapy this will enable you to construct a spider that will walk the web site(s) following the rules that you set on what content to download and save.

Points to consider:

  • Free & Open Source
  • Cross platform will run on Windows/Linux/Mac
  • Only download the content that you specify so saves bandwidth/time/space
  • Large StackOverflow & on-line communities
0

If you use Firefox, you can use DownThemAll! which is a very robust mass downloading tool.

Just add a new download job and make sure to add in the appropriate batch descriptor: enter image description here

Then all of the downloads will be added and automatically start. If a page is a 404, then dTa will just ignore without writing that page to the disk. It's that simple!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.