I have an old list of URLs, and many are out-of-date.
Eliminating the URLs that return a "Page not Found" or "Server not found" error is easy, but how to detect the pages whose domain has been bought and is listed as "domain for sale", or just running a generic engine that just shows advertisements and links of thousands of similarly "squatted" domains?
- Ideally a free open source software using an open database. A free web service is acceptable as well.
- Must be able to work for a list of thousands of URLs, no time limit.
Some domains are more difficult to detect as they try to emulate a legitimate website with many advertisements and sometimes malware. Such websites often try to generate content based on users' search keywords. Accessing the same site from two different countries often shows totally different content, which might be a way to spot them.