We regularly have broken links to and from our website; I'd like to operate a tool to report them, preferably on Ubuntu Linux.
The tool would be a link checker, crawling the site and reporting broken URLs with the URLs of the referring pages.
Our site is far too big to allow starting up a link checker application and waiting for it to finish. So the actual crawling must be done in the background.
One approach is to run a command line link checker for Linux in an infinite loop. It will crawl the whole site and generate a report at the end. Once it's done, replace the previous report with the generated one and restart.
- When a URL breakage is detected, generating the report is usually hours or even days away. By that time, the page in question may have been fixed.
- The same pages will be checked over and over, even when they haven't changed in years.
- Puts a pretty high load on the server.
Which link checker (as I said, preferably available for Ubuntu) fixes this?
- Maintains a database of crawled URLs, with last access time, status, last modification time, response time, MD5 hash of the page content.
- Crawls URLs in the database continually, spreading the load on the server.
- Revisits changing pages sooner than unchanging pages.
- Revisits fast pages sooner than slow pages.
- Reports each broken link with last access time, referring page, last time it wasn't broken if available.
- Updates its report continually.