Periodically, I need to check my websites for "link-rot". The plural already indicates, this is a job that cannot be done manually (to many sites and even more links to check), so I want a tool to assist me.
Must haves:
- Must run on Linux
- Must allow to define filters (for URLs/Server-Names/Domains not to check; RegEx would be great, simple "substring" is still fine)
- Must allow to filter the output (at least to "show errors and warnings only"; the more granular the better)
Strongly preferred:
- GUI and CLI interfaces (so I can run it manually with a nice interface, and also automatically from Cron – in which case, if possible, the GUI should be able to load the "results")
- Some statistics
- Efficient:
- Should not check the same URL multiple times (but of course if broken report it for each page it's found on)
- should not parse the same page multiple times1
- Exact (as few "false negatives" as possible)
Nice to have:
- being able to specify which URL parameters to ignore2
- sending (formatted) reports by mail (if run from Cron)3
- Capable of scanning sites with auth requirement4
- Ability to exclude file-types from being scanned5
I've already tried:
- gUrlChecker: Seeing the GUI it seems to meet all requirements. But it ignored all filter settings (for hosts/URLs to skip; if an example is included how to do that, that answer is welcome – maybe I did something wrong, or the version of gUrlChecker I was using has a bug)
- LinkChecker: far to many "false negatives" (e.g. reports A redirects to A, i.e. to itselfs; checks the same page multiple times and reports its errors multiple times as well, reported "unreachable pages" (301, 401) which were clearly reachable (without "authorization"), no filtering of output (though its displaying of "errors and warnings only" is acceptable, I'd wish to have it show e.g. "errors only"). Again, this could be a bug meanwhile solved: as gUrlChecker, I've installed it from the Ubuntu repos, which are not always having the latest versions (ouch, indeed: 7.x in the repo, 9.3 on the project site – will test again with the latest version)
1: If e.g. on the scanned site, pages A, B and C link to Z (still on the scanned server itself, i.e. no external links), Z should be scanned only once, not 3 times as I've experienced it e.g. with LinkChecker
2: If the site e.g. serves the same content in multiple languages, it makes no sense to scan all language variants (provided the links on them are identical). So I may e.g. wish to ignore the lang=XX
parameter, and have the link checker consider a.php
, a.php?lang=en
, and a.php?lang=de
the same page. This of course might be covered by the must-have filter with RegEx, provided the lang
parameter is optional ;)
3: Sure with Cron STDERR
is captured, so the focus of that mail lies on "formatted". That might e.g. be ODF calc sheets (which then can be "filtered" using OpenOffice/LibreOffice).
4: i.e. those sites requesting username/password to be accessed (HTTP response code 401); I just noticed LinkChecker added that with v7.9, gUrlChecker is capable of that as well. This mainly relates to the site to be scanned, not necessarily to external links (if both is supported, it should be configurable separately)
5: if the link checker is e.g. capable of scanning contents of PDF, MSWord, whatever documents, it should be possible to turn that off: a website might hold "older document versions" for reference, where "outdated links" are considered "normal". Exclusion could take place by mime-type or file extension.