Show unique log statements

Question

I have a list of errors from a log and I'd like to reduce it to a list of unique errors. Is there software to do this?

In an application log the same statement is often printed many times as that event happens many times or that part of the code is run many times. The statements will have different time values, thread names, etc. I would like to get one instance of each log line for all lines in the log. The lines don't have the same format so it would take a long time to manually create rules to match instances of the same statement. The software needs to figure out which lines are the same statement and remove the duplicates.

I'd prefer open source software for Linux or Windows but I'd be interested to know about all software. I may be willing to spend up to $1000 in future and I'm interested to know what's available over that.

Answer 1

Typically date, time, thread, etc. are at the start of a log line. Often, will either be a fixed count of characters, or delimited by some character like a comma or colon.

Simply take your log file, and use a combination of cut sort and uniq. For example, here's a line from an Apache2 webserver log

172.16.130.165 - - [10/Jun/2019:08:13:00 -0400] "GET /favicon.ico HTTP/1.1" 200 324 "-" "Safari/14607.2.6.1.1 CFNetwork/978.0.7 Darwin/18.6.0 (x86_64)"
66.249.88.24 - - [10/Jun/2019:08:41:19 -0400] "GET /merges/ HTTP/1.1" 200 3930 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon"
66.249.88.28 - - [10/Jun/2019:08:41:20 -0400] "GET /favicon.ico HTTP/1.1" 200 3583 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon"

Using a space as a delimiter, field 6 and beyond is what is interesting, so to make that unique and count them I could do

cat /var/log/apache2/myserver-access.log | cut -f 6- -d " " | sort | uniq -c