0

We have a solution where we have multiple data centers and per data center we have multiple servers with the same application. The application is handling stateless traffic and produces a lot of logging using syslog.

For troubleshooting we need to be able to combines these various log lines to a single output. These events always start with a UUID and end with a common terminator line. In between all lines with the same process ID are part of the output. For example:

server1 application[12345] ALERT: Start transaction dd03ed10-077c-4f81-88cf-1bf6ba479dce
server1 application[12345] INFO: Some relevant information
server1 application[12345] INFO: Some more relevant information
server1 application[12345] DEBUG: This is relevant too
server1 application[12345] ALERT: End of transaction

Or if multiple servers are involved, it would look like this:

server8 application[808080] ALERT: Start transaction dd03ed10-077c-4f81-88cf-1bf6ba479dce
server8 application[808080] INFO: Some relevant information
server2 application[222] ALERT: Start transaction dd03ed10-077c-4f81-88cf-1bf6ba479dce
server2 application[222] INFO: Some more relevant information
server8 application[808080] DEBUG: Multiple servers working together
server8 application[808080] ALERT: End of transaction
server2 application[222] ALERT: End of transaction

And in the worst case we have multiple transactions that need to be merged in the same log output.

server3 application[3321] ALERT: Start transaction 9d141a65-6e8f-44e9-8c48-81a7371e00fd
server3 application[3321] ALERT: Relevant extra transaction: dd03ed10-077c-4f81-88cf-1bf6ba479dce
server8 application[808080] ALERT: Start transaction dd03ed10-077c-4f81-88cf-1bf6ba479dce
server8 application[808080] INFO: Some relevant information
server3 application[3321] DEBUG: Some extra logging
server2 application[222] ALERT: Start transaction dd03ed10-077c-4f81-88cf-1bf6ba479dce
server2 application[222] INFO: Some more relevant information
server8 application[808080] DEBUG: Multiple servers working together
server8 application[808080] ALERT: End of transaction
server2 application[222] ALERT: End of transaction
server3 application[3321] ALERT: End of transaction

Question:

Which solutions are available to collect syslog, make it searchable (via an API or a web-interface) that would be able to combine these messages in such a way? An open-source solution would be prefered, but we are open for commercial software as well. We need to process about 1 to 5 GB of syslog daily and we require 3 months retention.

migrated from unix.stackexchange.com Jun 4 at 20:29

This question came from our site for users of Linux, FreeBSD and other Un*x-like operating systems.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.