7

We regularly have broken links to and from our website; I'd like to operate a tool to report them, preferably on Ubuntu Linux.

The tool would be a link checker, crawling the site and reporting broken URLs with the URLs of the referring pages.

Our site is far too big to allow starting up a link checker application and waiting for it to finish. So the actual crawling must be done in the background.

One approach is to run a command line link checker for Linux in an infinite loop. It will crawl the whole site and generate a report at the end. Once it's done, replace the previous report with the generated one and restart.

Drawbacks:

  • When a URL breakage is detected, generating the report is usually hours or even days away. By that time, the page in question may have been fixed.
  • The same pages will be checked over and over, even when they haven't changed in years.
  • Puts a pretty high load on the server.

Which link checker (as I said, preferably available for Ubuntu) fixes this?

Requirements:

  • Maintains a database of crawled URLs, with last access time, status, last modification time, response time, MD5 hash of the page content.
  • Crawls URLs in the database continually, spreading the load on the server.
  • Revisits changing pages sooner than unchanging pages.
  • Revisits fast pages sooner than slow pages.
  • Reports each broken link with last access time, referring page, last time it wasn't broken if available.
  • Updates its report continually.
  • "the report is usually hours or even days away" - how large is your site? 3h*50MBit/s=67 GB. Given a large website with 512 kB (without images, you do not download images, just headers), this is 130.000 pages that can be checked in 3 hours. If you talk about days, that's 2.1 million pages. Are your assumptions correct? – Thomas Weller Dec 13 '16 at 10:49
  • 1
    Last time I counted, the logs showed requests for 3690360 different URLs. – reinierpost Dec 13 '16 at 13:07
  • Ok, maybe you want to add a requirement for distributed execution then (like multiple processes in different locations worldwide) – Thomas Weller Dec 13 '16 at 13:41
  • 1
    No, that would be part of the chosen solution, not part of the problem, and I don't think it would help much, either. I'm trying to avoid putting too much load on the server. – reinierpost Dec 20 '16 at 15:06
  • Is it all served by one server? Are those unique pages or does that count include API endpoints? I am intrigued by this. – Gordon Jan 22 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.