I'm looking for a tool that would allow me to answer questions such as:

  • Most visited page(s) on a 10 minute interval (or even shorter), example: between 28/01/2016 20:00 and 28/01/2016 20:10
  • How many errors returned on that period (HTTP Codes essentially). Detail of the error would be nice to have as well.
  • Referrals for those GET requests

Must be "log based" (HTTPd access/error log) and not based on a tag (piece of JS code to track events) as that's not 100% reliable.

Can be either self-hosted or SaaS.

In essence, I need something like Awstats or Piwik but with the capability to go quite granular.

  • Self-hosted? Or SaaS?
    – Nicolas Raoul
    Jan 29 '16 at 7:47
  • You need the tracking JavaScript code, the tracking server, and the analysis tools, I guess?
    – Nicolas Raoul
    Jan 29 '16 at 7:48
  • @NicolasRaoul, doesn't matter really. The most important part in all of this I guess is the fact that it must be "log based" (HTTPd access/error log) and not based on a tag (piece of JS code to track events) as that's not 100% reliable.
    – fobia
    Jan 29 '16 at 7:59
  • Thanks! For Apache httpd I guess, or do you need to support other web servers? The log format might be different.
    – Nicolas Raoul
    Jan 29 '16 at 8:02
  • Apache, indeed.
    – fobia
    Jan 29 '16 at 8:05

If I want to figure things like that, I pick on GoAccess from the command line. It's amazingly fast and accepts input via pipe (of course also accepts a file name), so I can restrict it using grep to what I wish to see – e.g. grepping for data and hour to restrict analysis to that hour.

On the command line, GoAccess provides a curses like interface managing multiple sections, so you can easily dive into either UserAgents, top URLs, referrers, or whatever you're after:

GoAccess GoAccess
GoAccess (source: GoAccess; click images for larger variants)

As you can see, the command-line interface even lets you filter output to further narrow down your scope. And yes, I use it with Apache – so it should be fine with your environment.

So let's match your requirements:

  • Most visited page(s) on a 10 minute interval (or even shorter): possible, e.g. by piping grep output. If your log lines e.g. start with the pattern [Wed Oct 11 14:32:52 2000], you could get your 10 minute interval grepping for "Oct 11 13:1" (to cover everything between 13:10 and 13:19), piping the output to GoAccess – and then investigate the most visited pages as shown in section 2 of the first screenshot ("Top requests").
  • How many errors returned on that period: Similar to the previous point, different section ("HTTP Status Codes", "404 or Not Found")
  • Referrals for those GET requests: Same again, section "Referring Sites & URLs"

If you're interested in "automatic reports", GoAccess also supports other output variants: HTML (see here for example output), JSON and CSV are covered here which you then could evaluate with your own scripts (or run it by Cron to create "static HTML reports" to access remotely from anywhere). Also see this answer for a different view on things.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.