7

We regularly have broken links to and from our website; I'd like to operate a tool to report them, preferably on Ubuntu Linux.

The tool would be a link checker, crawling the site and reporting broken URLs with the URLs of the referring pages.

Our site is far too big to allow starting up a link checker application and waiting for it to finish. So the actual crawling must be done in the background.

One approach is to run a command line link checker for Linux in an infinite loop. It will crawl the whole site and generate a report at the end. Once it's done, replace the previous report with the generated one and restart.

Drawbacks:

  • When a URL breakage is detected, generating the report is usually hours or even days away. By that time, the page in question may have been fixed.
  • The same pages will be checked over and over, even when they haven't changed in years.
  • Puts a pretty high load on the server.

Which link checker (as I said, preferably available for Ubuntu) fixes this?

Requirements:

  • Maintains a database of crawled URLs, with last access time, status, last modification time, response time, MD5 hash of the page content.
  • Crawls URLs in the database continually, spreading the load on the server.
  • Revisits changing pages sooner than unchanging pages.
  • Revisits fast pages sooner than slow pages.
  • Reports each broken link with last access time, referring page, last time it wasn't broken if available.
  • Updates its report continually.
6
  • "the report is usually hours or even days away" - how large is your site? 3h*50MBit/s=67 GB. Given a large website with 512 kB (without images, you do not download images, just headers), this is 130.000 pages that can be checked in 3 hours. If you talk about days, that's 2.1 million pages. Are your assumptions correct? Commented Dec 13, 2016 at 10:49
  • 1
    Last time I counted, the logs showed requests for 3690360 different URLs. Commented Dec 13, 2016 at 13:07
  • Ok, maybe you want to add a requirement for distributed execution then (like multiple processes in different locations worldwide) Commented Dec 13, 2016 at 13:41
  • 1
    No, that would be part of the chosen solution, not part of the problem, and I don't think it would help much, either. I'm trying to avoid putting too much load on the server. Commented Dec 20, 2016 at 15:06
  • Is it all served by one server? Are those unique pages or does that count include API endpoints? I am intrigued by this.
    – Gordon
    Commented Jan 22, 2019 at 13:12

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.