Software for parsing and displaying log data in a structured way

Question

In my job it is common to work with a lot of custom, in-house log file formats generated by embedded devices. e.g.

[16:31][ERROR][controller.c,33]: Actuator not responding {I:30,t:40,}

[16:42][MESSAGE][controller.c,39]: Rollout incomplete, timer resetting with code <3>

[16:42][WARN][spool.c,22]: Folding procedure failed, stator is locked {b:4,t:45,deltat:3}

[16:53][WARN][comms.c,14]: CAN protocol timed out, did peer reset watchdog?

Reading a log file in a text editor is frustrating because there is no color-coding. Sometimes these logs are 1000s of lines long.

I am looking for a (ideally GUI) program that can open an arbitrary text file, and:

• Allow me to apply a simple regex to decode each line, i.e. decode the [16:31], and [ERROR] into simple timestamps

• Display decoded fields as columns in a table (so each line becomes part of the columns timestamp,severity,file,line,message,additional etc.). CSV export capability would be nice.

• Apply color coding (conditional formatting) on certain lines or severities

• Allow easy filtering to find specific lines

• Bonus: Allow easy plotting of decoded numerical columns in logs. For example, say there is an occasional message like [BATTERY] Vbat is 3.9V. It would be nice to have an easy way of selecting all lines with a [BATTERY] prefix, extracting the voltage number with a regex, and plotting it on a graph with timestamp on the x. It would also be convenient to be able to overlay multiple curves from disparate data sources.

Software I've tried for this include:

• Writing a parser in Python using pandas. This involves reading all lines into dataframe rows and using str.extract to decode structured fields into columns using regex. This is not ideal as I find I have to write a lot of code to parse and plot numerical data.

• A data analysis tool such as Logstash or Kibana. These seem overkill as they require the log files to be stored in a database first. Our company has no cloud solution and the logfiles are written to an SD card as .log plaintext files. These tools also require strong IT experience to set up, which I lack.

Any suggestions would be appreciated!

Answer 1

I wrote a log viewer loxx (as replacement of baretail supporting some more features - including simple graphic ouputss). I think it can fulfill most or all of your requirements. You can write also macros generating graphical output for your data. It supports timestamps, color highlighting and much more.

Have a try - any feedback is welcome!