How to centralize, store and analyse spread out file logs?

Question

TLDR: Our application generates file logs holding bits of information that we'd like to parse them and store in one place, in order to periodically consult them and compute metrics from them (in real time). I've tried a few things, but I I'm starting to get lost...

---

Basically, some operations on our networked application generate log files which holds structured entries like flowId (UUID), status (stepBegin, stepX, stepY, ..., stepEnd) or previousFlow (UUID or the previous flow). Each log file is created in a repertory named with its flowId (so there are a lot of file). We can then access each log to reconstruct a flow that happened, but it's rather tedious by hand.

What we need:

1. Centralize and store logs (Bonus: free space by deleting logs in the same time).
2. Allow to fetch and display related logs. For example: all sharing the same flowID, ordered by time. (Bonus: automated joints on specific fields like previousStepId)
3. Compute metrics from multiple log entries. For instance: for every flowId, (date where status = "stepEnd") - (date where status = "stepBegin") would give us the execution time of the operation identified by this flowId. (Bonus: do it on the log stream itself to store it in a metrics temporal DB)

Constraints:

• It has to be free, preferably open source.
• Will be used inside a protected network, security is not a hard requirement
• We'd like to keep using the file system for its benefits (fail-safe and asynchronous) but we have control over the java application producing logs, we can code a custom log handler if necessary.
• We'd like to keep it simple but we will consider any solution (even involving lots of different third party libraries/softwares)
• Ideally we'd like to visualize in our web application, so "embeddable" views would be perfect and "requestable" APIs would be nice; but third party GUI or web console are still ok.

What we have/tried

• We have a prometheus + grafana setup running for other metrics (JMX feed) that we can mutualize. (for computed metrics for instance)
• I made an filebeat + ELK stack setup to try. Filebeat consumes log files, logstash parses key/values from them to store it orderly in elastic search and kibana to visualize. It covers our 1st and 2nd needs but I could not compute metrics from multiple entries with ELK...

I'm looking into CEP (complex event processing), but I'm just a poor developer and rather lost in this operational jungle. Any advice would help, thanks!