I need to obtain all the links present in a group of web pages.
Most of these pages are part of four websites, but some of these sites are not
domain.com but rather
domain.com/~site. Some of these pages have non-canonical duplicate URLs that don't redirect to the canonical one.
I want a software that runs (preferibly) on Linux or Windows, that I can provide a list of URLs, and it will generate a single, deduplicated, recursive list of all links it finds.
I've tried HTTrack but couldn't get it to behave as I need.
It would be great if it could follow HTTP redirects, JS redirects and HTML redirects (
I need a tool that will look at an URL and generate a text list of all links found (without downloading the pages). And then do that recursively.
Is there such a tool?