I'm going to receive a large group of CSV files daily. Additionally I have 1 billion records of data (from those CSV files) but its not a fixed number (1b records daily). They're going to be growing and I need to store them in a DB. Also there are some extra points:
- There's no update
- There's no join and relationship
- Select a bunch of rows and group by
- Write intensive is so more than read
- I don't need normalisation
I had a bench between MySQL InnoDB and MyISAM. MyISAM was more better than InnoDB (because I have no normalisation) but MySQL is not a good approach because I have no relation.
I also checked MongoDB but with 50GB of CSV data it used 150Gb of storage!
I know I can use CSV files but I need a database approach. A database has lots of tools (like simple GROUP BY queries) and advantages like updates, bug fixes, security stuff, read and write performance, replication etc.
So I think I need a NoSQL DB which can do distributed write and support above extra points. But I don't know which NoSQL one (ones) is better for me.
I'm using Linux (CentOS).