It is sadly very common nowadays to be subject to various referrer spam logs obscuring real access_log statistics for the HTTP_REFERER environment variable, the Referer header field of HTTP.

Is there any software that can automatically determine and filter out such spam?

I'm using nginx, and I have a few domains in .ru, for example, many of which seem to receive the very same referer spam at the very same time; some of these domains aren't even used for any live site yet.

I'm using visitors as my web log analyser.

I'm thinking that there should be some post-HTTP-access pre-analyser software to perhaps have some number of the following features:

  • automatically download a list of these spam URLs from some central and trusted location, similar to how spamd authors provide a list of greytrapped addresses from their own location (UAlberta.Ca) where they run spamd by themselves

  • analyse local access_log from multiple similar domains in the same zone, to see if there are multiple consequent burst requests with the same Referer field from the same IP-address with distinct User-Agent fields and to distinct local domains, especially if no request is made without a foreign referrer

  • make pingback requests to the addresses supplied in Referer, and filter out results that have no backlink, potentially also looking for lack of thematic keywords, to prevent the news feed of frequently updated sites to be blacklisted

  • keep a database of spam referer addresses based on the above, checked daily to hourly through a cron job, and filter out requests with such referers from access_log, possibly by having access_log piped through such tool prior to the daily ingestion by the log analysis software

  • That sounds interesting. What OS should it run on? I myself have set up automated blacklisting by IP sources (list of known spammer IPs) with Apache to keep them out right from the beginning (i.e. before they access anything). If NGinx has that ability (to be fed by a list of "bad IPs", one per line, to not allow in), this might be a partial answer – provided the requests are kept out of access_log then.
    – Izzy
    Feb 24, 2015 at 10:23
  • @Izzy, yeah, I used to add them right into my pf(4) firewall, which is even more efficient than having these spammers talk to your web-server at all. However, they don't really cause any trouble for nginx, other than these log entries, so, I'm thinking that post-access_log-filtering might be a better idea, especially since many of these IPs change all the time. The software should run on OpenBSD/FreeBSD.
    – cnst
    Feb 24, 2015 at 16:28
  • The changing IPs are not a real issue to me, as the source file (list of spammer IPs) is auto-updated all 24h. But I agree in so far that it's not at all complete, and thus the other part might be required at least complementary. Though I don't care for what referer they put into the logs, as my stats are not public :)
    – Izzy
    Feb 24, 2015 at 16:34
  • 1
    @Izzy, Well, that's kinda exactly the issue -- you probably want to be defensive, not offensive here, as such, it sounds like a better approach to allow access, and then filter out offenders from the stats (e.g. by piping access_log through grep -v), otherwise, even though your stats is not public, it sounds like those offending referrers still show up in it (e.g. with the requests that originally prompt them to be blocked further down the line), defeating the purpose of the filtering.
    – cnst
    Feb 24, 2015 at 18:58


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.