2

I've developed multiple scripts each handling different things, they all spit out a log that I would like to analyze and gather data/statistics from.

I've tried to find a custom log handler but either they have been unable to accept my seperator/delimiter and/or added extra unnecessary fields (Splunk - Does this for my logs) despite that I used , as separator/delimiter.

So, what I'd like to ask, is there a tool where I can analyze custom logs? And not just see the rows of data (That's what I use LogExpert for), I want to perhaps see graphs over the data / input.

To give a better understanding of what I'm trying to achieve; I have made a custom 404 logging, but since it's not apache writing it, the regular apache analyzers doesn't work. What I want to find is a tool that can read my custom 404 log and tell me what links have the most 404 via graph or some other graphic display.

Though that is just one of my custom logs, this is the most important one.

Required features

  • Windows 7 compatible
  • Filter data based on certain criteria
  • Graphical display over raw data / Filtered data
  • Recognize log structure by either First line of file (First line would be "header/field" names) or specified when importing the log

Attractive features (not but required)

  • Column sorted
  • Apply multiple filters
  • Save filters
  • Count / Sort data - Either by column or other defined measures.

I have tried the following programs and they were unable to do what I required.

Sorry for the long wall of text and any grammatical mistakes :)

Improve this question
1
  • An ELK (ElasticSearch, Logstash, Kibana) stack ? Quite resource-hungry, complicated to set up but once it's done it's excellent. Kibana is used as a web front end and can dynamically create graphs and views based on a search query.
    – user111
    Commented Jan 23, 2015 at 22:21

2 Answers 2

Reset to default
2

This sounds like a job for python + pandas.

  • Free
  • Easy to customise the readers
  • Lots of manipulations, selections and graphing available
  • Cross platform.
Improve this answer
7
  • Hi! Thanks for the help, it is indeed cable of a lot of things but as far as I was able to tell I had to code the logic behind the graphs / data extraction from the logs?
    – Rasmus Boie
    Commented Jan 26, 2015 at 8:17
  • You would be amazed how easy it is.
    – Steve Barnes
    Commented Jan 26, 2015 at 16:59
  • I tried to install it via Anaconda (continuum.io/downloads) and then ran it through the ipython-notebook program/browser thing. I might have done it wrong, but as far as I could see/tried, I had to code every bit of logic myself.
    – Rasmus Boie
    Commented Jan 28, 2015 at 7:33
  • 1
    I struggled quiet a bit with getting it to work and since it's not in a language I'm all familiar with it didn't make it easier, but I have given you a +1 for the answer none the less as it looks very good, I have however found a GUI to microsofts Log parser 2.2 (lizard-labs.com/log_parser_lizard.aspx) and that builds around Query which I'm more familiar with, and I will try to get that to work. But I will keep Pandas in mind as it seems VERY use able for other things as well
    – Rasmus Boie
    Commented Jan 28, 2015 at 8:16
  • 1
    @RasmusBoie it might be worth posting that as an answer to your own question so as to make the information available to others that come along see softwarerecs.stackexchange.com/help/self-answer
    – Steve Barnes
    Commented Jan 28, 2015 at 8:20
2

While digging around more I fell over Log Parser Lizard, it is essentially a GUI for Microsoft's Log Parser 2.2 and builds around using QUERY's to fetch the data you want, this allows you to also build graphs (Bar / Pie / Curves and so on) from the fetched data.

Link to Lizzard log: http://www.lizard-labs.com/log_parser_lizard.aspx

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.