I need to perform usage analysis on a few websites based on no more than standard access logs, e.g. Apache's access_log. Entries have a timestamp, a URL path, a client IP address, a HTTP status, and a few more bits, but no USER_AGENT or HTTP_REFERER.

Most of the sites are low-volume, so the majority of visits is from bots: crawlers from search engine operators, SEO optimizers, vulnerability scanners, etcetera. Such automated visits must be excluded from the analysis as much as possible. 100% accuracy isn't required, but 90% would be nice.

Are any tools or resources available for doing this? E.g. does anyone maintain a public database of IP addresses used by crawlers, that I could use to filter them out by IP address?

PS: Linux-based, free solutions are preferred, but I'm interested if there's anything out there that tries to do this.

  • Could you show a snippet of the access.log? Commented Nov 30, 2020 at 17:14
  • The usual clarification request: OS this should run on, price limit in case of paid software? And yeah, most go by UA, though there might be some going by "known IP ranges", letting you set up specifications, or use external databases like those for GeoIP. You won't get 100%, though: "good bots" identify properly, "bad bots" often don't.
    – Izzy
    Commented Dec 1, 2020 at 0:39
  • If they go by UA, that is relevant, but it's not what I'm looking for in this question. Commented Dec 1, 2020 at 9:03


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.