I am researching distributed file systems for a particular use-case.
Our use case involves high-latency between nodes with the requirement of fast writes.
The options I've seen so far do not seem to be suited for cases of high latency in between nodes (ie: GlusterFS) - they seem to be based on synchronous / blocking write models and so get extremely slow when the servers are in different geographic zones.
In our case the latencies range from 10ms to 200ms in the worst case between nodes. So those options are not in play.
Let me explain a bit more concretely our use case and setup:
- We have servers in different geographic zones (Chile, Miami, Frankfurt, Dallas).
- We want a distributed FS spanning all or most of these servers.
- Writing should be fast and non-blocking, as close as possible to a non-distributed fs.
- We are fine with handling uniqueness / conflicts directly in our code, the file objects stored will be assigned globally unique UUIDs using a probabilistic model / or geographic prefixes, making conflicts extremely unlikely or some other logic that is not the responsibility of the distributed FS.
- Reading can be slow if not reading from the original 'input' node, but ideally fast without much bandwidth usage if reading from the original node from which the file was input as clients will often read from where they wrote.
- We are not using AWS, Google or Azure for any of our servers and this is quite intentional (for reasons I'd prefer not to have to get into).
- The distributed FS will serve as a short term cache (3 months or so), and all data ingressed is immediately backed-up longer-term elsewhere, so some data loss within the distributed FS is tolerable every now and then if that makes it easier for us to achieve our other goals.
A few requirements:
- Open-source and free options only which we can mount on ubuntu 1604 or 1804.
- Ideally it should appear like a regular posix filesystem when mounted so that our services (some of which are 3rd party and we can't modify) can easily read and write.
- Should have some level of fault tolerance, but small amounts of data loss are OK if a node goes out just after a new file comes in.
- AWS, Azure or Google-based solutions are not an option so please don't invoke them unless it's very likely to be in some novel and unexpected way :).
I just noticed xtreemfs -- not sure if it fits yet, I would be interested in your thoughts!