I have bounded time series data. Currently this data is in stored in MongoDB, i.e. one document per bounded time series of up to 100000 data points, potentially an order of magniture more but that is certainly the limit. There are millions of these time series. A single data point can contain nested data, e.g.

    event_time: 12345
    value_a: 2
    value_b: 5
    value_c: "bar"
    nested: {
        value_d: 7

I need to be able to calculate statistics on these time series. Both, for single bounded ones, as well as on a number or all of theses bounded ones together. The statistics are calculated on operations like count, min, max, avg of the contents of a data point. Both overall as well as aggregation in buckets.

What needs to be fast (miliseconds): Statistics on a single time series. E.g. aggregation into daily buckets for 100000 data points that spawn over a year

What does not need to be fast: Analytics over a number or all of the time series.

I'm aware of MongoDBs native map-reduce support or aggregation pipeline but I had very bad experience with performance in the past. I'm not afraid to use Spark if need be. But I feel like this would be an overkill since the data where speed matters is bounded and rather small. And when the data becomes large, speed does not matter that much anymore. Also, why implement somthing that might already exist.

Although I'd like to keep MongoDB as a data store it is not cast in stone if it is making things overcomplicated. So the first question is whether Mongo is a terrible idea. If Mongo is in fact a terrible idea, what data store is recommended for this application?

Are there tools that run these kind of operations with MongoDB as a data source?

I found Cube which is not under active development anymore.

I looked into elasticsearch since it can be connected to Mongo and does provide the required operations, as far as I can tell. But I'm not sure this is the right tool for the job. Any experiences with using it for that use case?

  • Well, elasticsearch it is worth to try. Store your mongodb data to elasticsearch, and use kibana to visualizing your data. I'll leave an article from netsil about choosing time series database. My personal opinion, I'll choose druid. – Fajri Abdillah Nov 13 '16 at 2:41
  • Thank you for that link, it provides a nice overview. I probably have to either dismiss the idea of having only one data store or build the analytics myself. – rob Nov 14 '16 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.