Abstract

In my work I often have the need of document myself with many sources. Some of most useful ones rely on personal or corporate project blogs websites. Practically all of them are just simple websites with text and links, not really a great huge amount of dynamic javascript content.

Even if we are in 2015 where web hosting costs very few both personal and corporate websites are very subject to disappear, person working on that project got bored, corporate acquisitions or changes make the pages not reachable anymore or paid/hidden under a required account.

Not having the time for read everything doesn't mean that I have not the time for SAVE a COPY, a SNAPSHOT of that website, blog, or so on to have my emergency backup copy just in case..

TL; DR

I need a software or a library or something able to:

  • required: go deep on many level of links downloading the webpages but without creating redundancies
  • required: domain boundaries, in this way you can specify stuff like 16 level of links but not to be worried because you know the program will never exit from yourdomain.com
  • required: download all the files of any extensions: images, sounds, videos, pdf, zip, rar, iso, app, dmg, everything
  • preferred: able to work in multithread ( i mean spawn multiple requests together to improve the speed would be a nice bonus )
  • preferred: command line instead gui

Thank you in advance for all the suggester!

  • 1
    Any requirements concerning the OS it must run on? I can think of at least two command-line utilities meeting your requirements (though they might require you some learning to use them to your needs), namely wget and HTTrack (the latter also having a GUI, but you can use it from CLI). Both can be used to mirror complete websites, both work cross-platform. – Izzy Oct 27 '15 at 12:42
  • @Izzy I'm currently testing HTTrack, it seems nice but i still need to find it's limits. But about wget ? It's able to traverse many deep links, rewrite the urls and capable of high parallelism ? :O If it's full multiplatform is better, otherwise win or linux are ok – user3450548 Oct 27 '15 at 13:08
  • wget even is "pre-installed" with most Linux distros, considered almost a "core component". I've never used it to mirror complete websites (hence I've posted it as a comment, not an answer) – I've read it's capable of that, but cannot say so from my own experience for the reasons given. – Izzy Oct 27 '15 at 13:11
  • @Izzy thank you for your comment. I use sometimes wget also on window as binary executable but my level of knowledge about it being able to save successfully a whole website is similar to yours. I will need to investigate it a bit, but in the meanwhile hope for someone with a command line ready and some more details ;) – user3450548 Oct 27 '15 at 13:17
  • I'd say "How to use wget to mirror a complete website" (or something similar) would be a perfect fit for e.g. SuperUser :) – Izzy Oct 27 '15 at 13:48

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.