I have at my workplace rather large system, which aggregates and process incoming data(events). You can think about it like it is a something like relational DB.
We need to make some monitoring system to have possibility to get notification about something suspicious events (for example, some data sources may disconnect, or anything else can go wrong) - about everything that is impacting our input stream of events.
So, I come up with a few types of triggers that may be helpful:
- Count events for last period of time (1 minute, 5 minutes, 1 hour) and compare it with the same period before last period. Trigger an event in case the difference is too big
- Compare the same data with period at the past (for example, with the same period yesterday, or a week ago).
By 'count event' I may mean rather complex calculations: like unique users, events count , with some conditions (for example, unique users who older than 15 or events from 'android' platform) - all this statistics is easy to get from our storage by simple requests, it is not critical to perform this requests several times per minute.
So my question is what shall I do? What systems may I look at? Or Does my case is too specific, so I shall develop something like my own monitoring system?