1

I am looking for a DFS for unified data access and caching used files within a small cluster. We are currently using NFS to RAID node, and it seems not the best way to further scaling. So, I am looking for DFS with such work logic: All data is stored on the master node, and workers store only recently used data, i.e.:

  1. Worker connects to the cluster,
  2. It access some path like "dfs://test/test.png",
  3. File is downloaded to worker from master node,
  4. Next access will be to the local copy of the "test.png".

If a worker adds new files to DFS, it first creates the local copy and then loads it to the master node in background mode.

If a worker has no additional space for new files, it overwrites firstly accessed files. Most of the data is unstructured read-only files(pictures or videos as parts of datasets)

Which technology is most appropriate for such use case? Is it possible, using popular solutions, such as hadoop or alluxio?

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.