I want to download every file (via HTTP, HTTPS and FTP, HTML, PHP as delivered, JS, CSS, linked PDF, images) from a domain, its subdomains, every domain linked from the previously mentioned pages and their subdomains.

In case it's unclear what exactly I expect I provided an example in which I would want everything except for content from "unrelated.com". I was not able to find a crawler that let's me do that.


I would prefer a Linux command-line operated crawler that I can run from my VPS. I could live with Linux+GUI but I don't have Windows, Android or any Apple device/OS.

A software that outputs the links per page in an easily parsable list would also be sufficient, then I could write a shell script to select which link to download.

Changing the links in the HTML code (especially those that point to different domains) to point to my then local files would be nice but is not necessary.

  • Have you looked at wget and its mirroring options? Think it can do most of what you are asking and it should be already installed on most linux systems..
    – ivanivan
    Commented Jun 19, 2017 at 4:15

1 Answer 1


The python Scrapy library can do exactly what you are looking for:

  • No GUI You can run interactively within a python or ipython shell or you can write a script.
  • Save all of the data from the walked pages
  • Depth limitation
  • Rate limitation so as not to get kicked off or banned
  • Crawl a single or linked sites
  • Filter sites that you don't wish to crawl

Scrapy & Python are free, open source, cross platform tools.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.