1

I'm interested in downloading the entire contents of a website (mirroring).

It took wget 8 hours to mirror a 3 GB website. I have also tried HTTrack, but the performance was the same. I've also checked with different websites (to rule out that specific server being the culprit), but they showed similar performance.

Are there any tools that would provide better performance?

  • Preferably for Linux, but OSx would also do.
  • If links are rewritten to work locally, that's fine (but not a must)
Improve this question
6
  • Are you sure the owner of that site agrees to a "total mirror"? If I find someone doing that on one of my sites, I'd see to ban him. Especially if you don't care to have the links rewritten, there are other methods to achieve that than over HTTP. I've e.g. sent the contents of my eBook server per DVD on request. Sucking Gigabytes in less than a few ours I'd consider abuse of my site. Just saying ;)
    – Izzy ♦
    Jul 22, 2016 at 6:59
  • @Izzy: There might be legal uses for that, though.
    – Nicolas Raoul ♦
    Jul 22, 2016 at 7:23
  • What kind of network connection do you have?
    – Nicolas Raoul ♦
    Jul 22, 2016 at 7:24
  • @NicolasRaoul I didn't say there aren't (just think of Archive.Org). But as a web master, I was unfortunate enough to find my sites duplicated without my consent, causing high bandwidth and slowing down regular users. Note that I'm not speaking of where the content reappeared later, "enriched" with ads – that would be the real legal part of it. I'm just speaking of the duplication process as such.
    – Izzy ♦
    Jul 22, 2016 at 7:26
  • I doubt the performance issue is in the software, it's more likely that the website reponds slowly, especially with the load your give it (really sophisticated sites might even rate-limit you on purpose).
    – user416
    Jul 22, 2016 at 8:55

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.