I am looking for a DFS for unified data access and caching used files within a small cluster. We are currently using NFS to RAID node, and it seems not the best way to further scaling. So, I am looking for DFS with such work logic: All data is stored on the master node, and workers store only recently used data, i.e.:

  1. Worker connects to the cluster,
  2. It access some path like "dfs://test/test.png",
  3. File is downloaded to worker from master node,
  4. Next access will be to the local copy of the "test.png".

If a worker adds new files to DFS, it first creates the local copy and then loads it to the master node in background mode.

If a worker has no additional space for new files, it overwrites firstly accessed files. Most of the data is unstructured read-only files(pictures or videos as parts of datasets)

Which technology is most appropriate for such use case? Is it possible, using popular solutions, such as hadoop or alluxio?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.