I am searching a solution to store files of 1.5 GB. There is a decade of file per day.
I am searching a solution to store those files. I have already done some searches on HDFS and MongoDB. However, I do not manage to determine if they are good tools for my problem. And moreover, I can not find if I am missing another possible/better tool.
I have the following requirements:
- New documents available in few seconds (maximum)
- Solution which can be distributed
If the solution as a plugin in Logstash it is a plus
- I need redundancy for the stored files
- I need to pull the files from python
- Shortest time possible to retrieve the files
- I do not want from a cloud solution (e.g.: Amazon S3)