0

We have a solution where we have multiple data centers and per data center we have multiple servers with the same application. The application is handling stateless traffic and produces a lot of logging using syslog.

For troubleshooting we need to be able to combines these various log lines to a single output. These events always start with a UUID and end with a common terminator line. In between all lines with the same process ID are part of the output. For example:

server1 application[12345] ALERT: Start transaction dd03ed10-077c-4f81-88cf-1bf6ba479dce
server1 application[12345] INFO: Some relevant information
server1 application[12345] INFO: Some more relevant information
server1 application[12345] DEBUG: This is relevant too
server1 application[12345] ALERT: End of transaction

Or if multiple servers are involved, it would look like this:

server8 application[808080] ALERT: Start transaction dd03ed10-077c-4f81-88cf-1bf6ba479dce
server8 application[808080] INFO: Some relevant information
server2 application[222] ALERT: Start transaction dd03ed10-077c-4f81-88cf-1bf6ba479dce
server2 application[222] INFO: Some more relevant information
server8 application[808080] DEBUG: Multiple servers working together
server8 application[808080] ALERT: End of transaction
server2 application[222] ALERT: End of transaction

And in the worst case we have multiple transactions that need to be merged in the same log output.

server3 application[3321] ALERT: Start transaction 9d141a65-6e8f-44e9-8c48-81a7371e00fd
server3 application[3321] ALERT: Relevant extra transaction: dd03ed10-077c-4f81-88cf-1bf6ba479dce
server8 application[808080] ALERT: Start transaction dd03ed10-077c-4f81-88cf-1bf6ba479dce
server8 application[808080] INFO: Some relevant information
server3 application[3321] DEBUG: Some extra logging
server2 application[222] ALERT: Start transaction dd03ed10-077c-4f81-88cf-1bf6ba479dce
server2 application[222] INFO: Some more relevant information
server8 application[808080] DEBUG: Multiple servers working together
server8 application[808080] ALERT: End of transaction
server2 application[222] ALERT: End of transaction
server3 application[3321] ALERT: End of transaction

Question:

Which solutions are available to collect syslog, make it searchable (via an API or a web-interface) that would be able to combine these messages in such a way? An open-source solution would be prefered, but we are open for commercial software as well. We need to process about 1 to 5 GB of syslog daily and we require 3 months retention.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.