I want to download every file (via HTTP, HTTPS and FTP, HTML, PHP as delivered, JS, CSS, linked PDF, images) from a domain, its subdomains, every domain linked from the previously mentioned pages and their subdomains.
In case it's unclear what exactly I expect I provided an example in which I would want everything except for content from "unrelated.com". I was not able to find a crawler that let's me do that.
I would prefer a Linux command-line operated crawler that I can run from my VPS. I could live with Linux+GUI but I don't have Windows, Android or any Apple device/OS.
A software that outputs the links per page in an easily parsable list would also be sufficient, then I could write a shell script to select which link to download.
Changing the links in the HTML code (especially those that point to different domains) to point to my then local files would be nice but is not necessary.