I have an old list of URLs, and many are out-of-date.

Eliminating the URLs that return a "Page not Found" or "Server not found" error is easy, but how to detect the pages whose domain has been bought and is listed as "domain for sale", or just running a generic engine that just shows advertisements and links of thousands of similarly "squatted" domains?

  • Ideally a free open source software using an open database. A free web service is acceptable as well.
  • Must be able to work for a list of thousands of URLs, no time limit.

Domain for sale

Some domains are more difficult to detect as they try to emulate a legitimate website with many advertisements and sometimes malware. Such websites often try to generate content based on users' search keywords. Accessing the same site from two different countries often shows totally different content, which might be a way to spot them.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.