I am searching a solution to store files of 1.5 GB. There is a decade of file per day.

I am searching a solution to store those files. I have already done some searches on HDFS and MongoDB. However, I do not manage to determine if they are good tools for my problem. And moreover, I can not find if I am missing another possible/better tool.

I have the following requirements:

  • New documents available in few seconds (maximum)
  • Solution which can be distributed
  • If the solution as a plugin in Logstash it is a plus

  • Open source

  • I need redundancy for the stored files
  • I need to pull the files from python
  • Shortest time possible to retrieve the files
  • I do not want from a cloud solution (e.g.: Amazon S3)
  • 1
    Have you considered mariadb? You can import CSV files directly into a database. Every programming language has database support so python access is a non-issue. You can have 2 or more mariadb installations. hostpresto.com/community/tutorials/…
    – cybernard
    Feb 18 '19 at 18:48
  • I didn't considered mariaDB I will take a look at it. Thanks.
    – Thomas R
    Feb 19 '19 at 7:45
  • besides creating indexes, fast storage will decrease the access time. A M.2 SSD can get 2500mb/s read and 1500mb/s write easy. Even faster ones exist up to 3500mb/s read and writes speeds approx 2500.(maybe more)
    – cybernard
    Feb 19 '19 at 13:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.