I'm interested in downloading the entire contents of a website (mirroring).

It took wget 8 hours to mirror a 3 GB website. I have also tried HTTrack, but the performance was the same. I've also checked with different websites (to rule out that specific server being the culprit), but they showed similar performance.

Are there any tools that would provide better performance?

  • Preferably for Linux, but OSx would also do.
  • If links are rewritten to work locally, that's fine (but not a must)
  • Are you sure the owner of that site agrees to a "total mirror"? If I find someone doing that on one of my sites, I'd see to ban him. Especially if you don't care to have the links rewritten, there are other methods to achieve that than over HTTP. I've e.g. sent the contents of my eBook server per DVD on request. Sucking Gigabytes in less than a few ours I'd consider abuse of my site. Just saying ;) – Izzy Jul 22 '16 at 6:59
  • @Izzy: There might be legal uses for that, though. – Nicolas Raoul Jul 22 '16 at 7:23
  • What kind of network connection do you have? – Nicolas Raoul Jul 22 '16 at 7:24
  • @NicolasRaoul I didn't say there aren't (just think of Archive.Org). But as a web master, I was unfortunate enough to find my sites duplicated without my consent, causing high bandwidth and slowing down regular users. Note that I'm not speaking of where the content reappeared later, "enriched" with ads – that would be the real legal part of it. I'm just speaking of the duplication process as such. – Izzy Jul 22 '16 at 7:26
  • I doubt the performance issue is in the software, it's more likely that the website reponds slowly, especially with the load your give it (really sophisticated sites might even rate-limit you on purpose). – Jan Doggen Jul 22 '16 at 8:55

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.