Batch saving webpages

Question

I need to save multiple pages to my harddisk.The pages have a pattern like this:

http://example.com/page1.html
http://example.com/page2.html
http://example.com/page3.html
.
.
.
.

I need to save a large number of webpages(about 5000).I tried using IDM but it hangs when I enter the links maybe due to the large number of pages.I also tried WinHTTrack but it saves the pages along with the unneccessary things such as pdf's from the site which i really don't need.

Can you recommend a software?thanks

Answer 1

You can use GNU wget:

• free and open source
• Linux, Windows, Mac (brew install wget)
• CLI
• tons of features such as the ability not to download PDFs.

Answer 2

I would suggest looking into the combination of python and scrapy this will enable you to construct a spider that will walk the web site(s) following the rules that you set on what content to download and save.

Points to consider:

• Free & Open Source
• Cross platform will run on Windows/Linux/Mac
• Only download the content that you specify so saves bandwidth/time/space
• Large StackOverflow & on-line communities

Answer 3

If you use Firefox, you can use DownThemAll! which is a very robust mass downloading tool.

Just add a new download job and make sure to add in the appropriate batch descriptor:

Then all of the downloads will be added and automatically start. If a page is a 404, then dTa will just ignore without writing that page to the disk. It's that simple!