5

TLDR: Our application generates file logs holding bits of information that we'd like to parse them and store in one place, in order to periodically consult them and compute metrics from them (in real time). I've tried a few things, but I I'm starting to get lost...


Basically, some operations on our networked application generate log files which holds structured entries like flowId (UUID), status (stepBegin, stepX, stepY, ..., stepEnd) or previousFlow (UUID or the previous flow). Each log file is created in a repertory named with its flowId (so there are a lot of file). We can then access each log to reconstruct a flow that happened, but it's rather tedious by hand.

What we need:

  1. Centralize and store logs (Bonus: free space by deleting logs in the same time).
  2. Allow to fetch and display related logs. For example: all sharing the same flowID, ordered by time. (Bonus: automated joints on specific fields like previousStepId)
  3. Compute metrics from multiple log entries. For instance: for every flowId, (date where status = "stepEnd") - (date where status = "stepBegin") would give us the execution time of the operation identified by this flowId. (Bonus: do it on the log stream itself to store it in a metrics temporal DB)

Constraints:

  • It has to be free, preferably open source.
  • Will be used inside a protected network, security is not a hard requirement
  • We'd like to keep using the file system for its benefits (fail-safe and asynchronous) but we have control over the java application producing logs, we can code a custom log handler if necessary.
  • We'd like to keep it simple but we will consider any solution (even involving lots of different third party libraries/softwares)
  • Ideally we'd like to visualize in our web application, so "embeddable" views would be perfect and "requestable" APIs would be nice; but third party GUI or web console are still ok.

What we have/tried

  • We have a prometheus + grafana setup running for other metrics (JMX feed) that we can mutualize. (for computed metrics for instance)
  • I made an filebeat + ELK stack setup to try. Filebeat consumes log files, logstash parses key/values from them to store it orderly in elastic search and kibana to visualize. It covers our 1st and 2nd needs but I could not compute metrics from multiple entries with ELK...

I'm looking into CEP (complex event processing), but I'm just a poor developer and rather lost in this operational jungle. Any advice would help, thanks!

4
  • I've posted an answer to a similar question here: softwarerecs.stackexchange.com/a/61662/3075 . The answer has a link to a paper (a master thesis from 2013).
    – knb
    Commented Jun 26, 2019 at 12:29
  • Thanks! Its a bit outdated and does not really covers my need for metrics computation though ...
    – Pierre
    Commented Jun 26, 2019 at 13:37
  • have a look at microprofile.io/project/eclipse/microprofile-opentracing Commented Jun 29, 2019 at 10:05
  • 1
    I looked at it and I don't see how it could cover my needs... I'm not looking for a framework I already have a 10+ years old code base, I'm not looking to migrate it :)
    – Pierre
    Commented Jul 1, 2019 at 8:46

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.