Have been looking for real-time analytics database as a managed service that could scale horizontally and have column storage and support distributed joins. Data is not too big (around 2 billion rows increasing by 1 billion per year), but still too much for postgres/Aurora to aggregate in sub-second.
I have already investigated following products:
- memSql (pretty close, but it's closed source and pretty expensive. Also community is pretty quiet. So possible vendor lock here.)
- crate.io (not sure as data import is buggy and slow. Can't get the data into the cluster. This is separate issue.)
- Citus (Not viable as it has moved to Azure, AWS would be required.)
I have been running Memsql with 4 nodes (AWS m5.2xlarge). So looking something that can do aggregation queries (including dimension table join) for 2 billion rows under 1 seconds with similar hardware.
It should also be able to keep up as data grows, so after couple of billion more rows, throwing couple of more nodes into the mix should give similar sub-second performance.
Managed service would be best fit, but self-hosted is also an option, if superior db is found.