File-based database: Sqlite, MongoDb or else?

Question

I am looking for a file-based database solution, where the original inputs are in csv format.

Key info:

• One complete dataset is a static collection of files. However, I will need to have multiple datasets over time.
• The dat files do not have dependencies(relations) with each other, hence no-sql solutions could work here.
• The total size of a dataset can be 1-4GB. If I split the dataset into labeled data, I can break the 4GB to maximum 100MB files. Meanwhile, in my solution I need to fetch info from many of those files.

From the sql domain, I see as a candidate Sqlite and from no-sql the MongoDb. Wrt to Sqlite, I have used it with small files and it seems to be a good candidate. I am not sure though how would it work, if the DB is up to 4GB. Wrt to MongoDb (or nosql solutions) I do not have yet experience.

Question

What would you use as a file-based database in this case? Alternatively, can you give some directions that can help me find a solution?

Answer 1

Stick with what you know.

Linked from the FAQ, we find the limits page, which says

Maximum Database Size

Every database consists of one or more "pages". Within a single database, every page is the same size, but different database can have page sizes that are powers of two between 512 and 65536, inclusive. The maximum size of a database file is 2147483646 pages.

At the maximum page size of 65536 bytes, this translates into a maximum database size of approximately 1.4e+14 bytes (***140 terabytes****, or 128 tebibytes, or 140,000 gigabytes or 128,000 gibibytes).

Of course, you will need a 140 tB hard-disk for this, but a RAID array could handle it ;-)

Your 4gB will be no problem. Go Sqlite