1

I am looking for a file-based database solution, where the original inputs are in csv format.

Key info:

  • One complete dataset is a static collection of files. However, I will need to have multiple datasets over time.
  • The dat files do not have dependencies(relations) with each other, hence no-sql solutions could work here.
  • The total size of a dataset can be 1-4GB. If I split the dataset into labeled data, I can break the 4GB to maximum 100MB files. Meanwhile, in my solution I need to fetch info from many of those files.

From the sql domain, I see as a candidate Sqlite and from no-sql the MongoDb. Wrt to Sqlite, I have used it with small files and it seems to be a good candidate. I am not sure though how would it work, if the DB is up to 4GB. Wrt to MongoDb (or nosql solutions) I do not have yet experience.

Question

What would you use as a file-based database in this case? Alternatively, can you give some directions that can help me find a solution?

0

Stick with what you know.

Linked from the FAQ, we find the limits page, which says

Maximum Database Size

Every database consists of one or more "pages". Within a single database, every page is the same size, but different database can have page sizes that are powers of two between 512 and 65536, inclusive. The maximum size of a database file is 2147483646 pages.

At the maximum page size of 65536 bytes, this translates into a maximum database size of approximately 1.4e+14 bytes (***140 terabytes****, or 128 tebibytes, or 140,000 gigabytes or 128,000 gibibytes).

Of course, you will need a 140 tB hard-disk for this, but a RAID array could handle it ;-)

Your 4gB will be no problem. Go Sqlite

  • 1
    Thank you for the feedback. Do you see any added value of using MongoDB or similar? I would be keen to benchmark an alternative solution to sqlite. – user1221647 Jun 13 '19 at 15:08
  • That's opinion based & off-topic here - but easily searchable; cleverer people than me have already benchmarked it. Personally, I prefer Relational data, but YKmMv – Mawg says reinstate Monica Jun 13 '19 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.