I am looking for a file-based database solution, where the original inputs are in csv format.
Key info:
- One complete dataset is a static collection of files. However, I will need to have multiple datasets over time.
- The dat files do not have dependencies(relations) with each other, hence no-sql solutions could work here.
- The total size of a dataset can be 1-4GB. If I split the dataset into labeled data, I can break the 4GB to maximum 100MB files. Meanwhile, in my solution I need to fetch info from many of those files.
From the sql domain, I see as a candidate Sqlite and from no-sql the MongoDb. Wrt to Sqlite, I have used it with small files and it seems to be a good candidate. I am not sure though how would it work, if the DB is up to 4GB. Wrt to MongoDb (or nosql solutions) I do not have yet experience.
Question
What would you use as a file-based database in this case? Alternatively, can you give some directions that can help me find a solution?