Website Link Checker for Linux

Question

Periodically, I need to check my websites for "link-rot". The plural already indicates, this is a job that cannot be done manually (to many sites and even more links to check), so I want a tool to assist me.

Must haves:

• Must run on Linux
• Must allow to define filters (for URLs/Server-Names/Domains not to check; RegEx would be great, simple "substring" is still fine)
• Must allow to filter the output (at least to "show errors and warnings only"; the more granular the better)

Strongly preferred:

• GUI and CLI interfaces (so I can run it manually with a nice interface, and also automatically from Cron – in which case, if possible, the GUI should be able to load the "results")
• Some statistics
• Efficient:
  • Should not check the same URL multiple times (but of course if broken report it for each page it's found on)
  • should not parse the same page multiple times¹
• Exact (as few "false negatives" as possible)

Nice to have:

• being able to specify which URL parameters to ignore²
• sending (formatted) reports by mail (if run from Cron)³
• Capable of scanning sites with auth requirement⁴
• Ability to exclude file-types from being scanned⁵

I've already tried:

• gUrlChecker: Seeing the GUI it seems to meet all requirements. But it ignored all filter settings (for hosts/URLs to skip; if an example is included how to do that, that answer is welcome – maybe I did something wrong, or the version of gUrlChecker I was using has a bug)
• LinkChecker: far to many "false negatives" (e.g. reports A redirects to A, i.e. to itselfs; checks the same page multiple times and reports its errors multiple times as well, reported "unreachable pages" (301, 401) which were clearly reachable (without "authorization"), no filtering of output (though its displaying of "errors and warnings only" is acceptable, I'd wish to have it show e.g. "errors only"). Again, this could be a bug meanwhile solved: as gUrlChecker, I've installed it from the Ubuntu repos, which are not always having the latest versions (ouch, indeed: 7.x in the repo, 9.3 on the project site – will test again with the latest version)

---

^{1: If e.g. on the scanned site, pages A, B and C link to Z (still on the scanned server itself, i.e. no external links), Z should be scanned only once, not 3 times as I've experienced it e.g. with LinkChecker}

^{2: If the site e.g. serves the same content in multiple languages, it makes no sense to scan all language variants (provided the links on them are identical). So I may e.g. wish to ignore the lang=XX parameter, and have the link checker consider a.php, a.php?lang=en, and a.php?lang=de the same page. This of course might be covered by the must-have filter with RegEx, provided the lang parameter is optional ;)}

^{3: Sure with Cron STDERR is captured, so the focus of that mail lies on "formatted". That might e.g. be ODF calc sheets (which then can be "filtered" using OpenOffice/LibreOffice).}

^{4: i.e. those sites requesting username/password to be accessed (HTTP response code 401); I just noticed LinkChecker added that with v7.9, gUrlChecker is capable of that as well. This mainly relates to the site to be scanned, not necessarily to external links (if both is supported, it should be configurable separately)}

^{5: if the link checker is e.g. capable of scanning contents of PDF, MSWord, whatever documents, it should be possible to turn that off: a website might hold "older document versions" for reference, where "outdated links" are considered "normal". Exclusion could take place by mime-type or file extension.}

Answer 1

As there were no recommendations, I've ended up using LinkChecker. While most of the cons I've listed with my question remained, using the newest version from the author's site went better than running the version that ships in the repo.

^{LinkChecker GUI and CLI (source: LinkChecker; click images for larger variants)}

How it matches my requirements

Must haves

• Must run on Linux: Yes, it does.
• Must allow to define filters: Partly. Pre-filtering (i.e. excluding things from the check) is partly possible, though not always intuitive. I found no way to filter the result list in the GUI.
• Must allow to filter the output: Again partly, see previous item. Once the result list is there, no further filtering is possible.

Strongly preferred

• GUI and CLI interfaces: Yes (see screenshots above). Even a Web interface (CGI) is provided.
• Some statistics: Not as detailed as I'd hoped for, but some stats are available (see lower-right corner in the GUI screenshot)
• Efficient: Sort of. Still far too many "duplicates"
• Exact (as few "false negatives" as possible): Here I was fighting with. Still a bit annoying, I couldn't get rid of all of them. It might be possible for "experienced users" – but it's definitely not intuitive for "first-timers"

Nice to have

• being able to specify which URL parameters to ignore: Here I succeeded only partly. There must be some trick, but I couldn't find it: it's possible to define patterns for URLs to ignore, but at some point I've stopped experimenting how to get that working with parameters:
  
  ^{Options window with URL patterns (click for larger variant)}
• sending (formatted) reports by mail (if run from Cron): Due to the not fully satisfying results with the GUI, I have not checked further with this.

• Capable of scanning sites with auth requirement: Not thoroughly tested, but this seems to be possible – to be configured in the linkcheckerrc file:

  [authentication]
  # Different user/password pairs for different URLs can be provided.
  […]
  

• Ability to exclude file-types from being scanned: I had not to deal with this, as LinkChecker seems to have not encountered any PDF or other file format it could scan.

Conclusion

While not exactly what I'm looking for, LinkChecker comes pretty close – most likely as close as I can get. If you encountered something better matching my needs, I'm looking forward to alternatives :)