2

I have a log file that looks like this:

06:24:35,524 WARN [ajp-bio-8009-exec-57804][c3p0] Temperature high
16:07:40,457 ERROR [ajp-bio-8009-exec-156][c3p0] Brain not found 
17:07:47,789 INFO [ajp-bio-8009-exec-57804][c3p0] Received salutation
17:07:47,885 ERROR [ajp-bio-8009-exec-156][c3p0] Brain not found 
17:07:50,875 ERROR [ajp-bio-8009-exec-156][c3p0] Brain not found 
17:08:02,746 ERROR [ajp-bio-8009-exec-156][c3p0] Brain not found 
20:27:51,179 WARN [ajp-bio-8009-exec-57804][c3p0] Temperature high

I want a tool to generate a graph showing the most frequent events, and their volume across time. That would make it easier to spot the peaks, and guess the causality between events.

The output could look like enter image description here (one small bar for each event occurrence or be more curve-like. Ideally it would allow switching between the two, as the first is better for small volumes and the second better for high volumes.

The graph should also show dates (if a GUI, on user action it could reveal the timestamp of each event) and the log message for each line/curve.

Bonus (optional):

  • Allows me to select what log messages I want to show (I often want to ignore some trivial messages that come often).
  • Correlates events (for instance if two messages always appear in pair, show the pair as a single message).
2

Use sed to insert a ; character just after the timestamp:

sed -i -e "s/^\(..\):\(..\):\(..\),\(...\) /\1:\2:\3,\4;/" $FILE

Open the CSV file with LibreOffice Calc, make sure to choose the right delimiter so that your timestamp is in the first column.

Example:

enter image description here

Convert the timestamp column from text type to date type, as described at https://unix.stackexchange.com/a/370227/2305

Save as a spreadsheet.

Create a new column between the timestamp and the message, and type this formula in the topmost cell:

=IF(COUNTIF(A2, "*Temperature high*"), "1", IF(COUNTIF(A2, "*Brain not found*"), "2", 0)

You get the idea, replace the default value 0 by as many IFs as you need. This is quite lousy, inspired from here, but I haven't been able to make a wildcard VLOOKUP to catch values that often contain random identifiers.

Copy this formula to all cells below.

Select the timestamp and code columns, and press Insert chart.

Select XY (Scatter)

Uncheck Display legend, click Finish

Click on the numbers of the vertical axis, right-click and select Format axis, set the intervals to 1.

Click on the points, right-click and select Format data series then on the Line tab reduce the width and increase the height.

This gives a chart like this:

enter image description here

Add a legend. Any tip on improving the formula part so that the legend gets used for parsing would be greatly appreciated!

  • Great answer! Why not accept it? – Mawg Aug 30 '18 at 9:52
  • 1
    @Mawg: It is all manual steps, with the spreadsheet formula being especially tedious to type. I hope some software will appear that does all of this automatically :-) – Nicolas Raoul Aug 30 '18 at 13:16
  • 1
    Ok, then leave it open. I suspect that I might one day want to follow those manual steps ;-) – Mawg Aug 30 '18 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.