I need to obtain all the links present in a group of web pages.

Most of these pages are part of four websites, but some of these sites are not domain.com but rather domain.com/~site. Some of these pages have non-canonical duplicate URLs that don't redirect to the canonical one.

I want a software that runs (preferibly) on Linux or Windows, that I can provide a list of URLs, and it will generate a single, deduplicated, recursive list of all links it finds.

I've tried HTTrack but couldn't get it to behave as I need.

It would be great if it could follow HTTP redirects, JS redirects and HTML redirects (META http-equiv="refresh").


I need a tool that will look at an URL and generate a text list of all links found (without downloading the pages). And then do that recursively.

Is there such a tool?

  • you can tinker with scrapy – StefanS May 16 '16 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.