Distributed filesystem with bit rot protection, erasure coding, scalability, fuse

Question

I have seen the question asked (in different forms) here, here, and perhaps the best one I found was here, but I do not think this is a duplicate because quite some time has past since those questions were asked, and my question has its own nuances that may help others in similar situations. Please hear me out.

Background

My question comes from there being many great distributed filesystems that advertise that they are amazing, but I suspect not all of them are what I need.

I have looked at this awesome list for suggestions on what is available and am not sure which one fits my needs.

Use case

The purpose for this server is to keep my data safe and available for general use. I will be using it to store my personal backups, and data stored and used by Nextcloud, Gogs, and anything else I self-host in the future.

What I am looking for

I am looking for a distributed filesystem that:

• protects against bit rot
• has erasure coding (or at least data duplication so drive failure doesn't disrupt usage)
• ability to scale
  • from 1 server to more later
  • from 2 HDDs to more later
• can connect via fuse

Powerful API and ease of use are big plusses.

My current hardware

This may not be important, but it may help with tips on implementation.

I currently have a Raspberry Pi, one 2 TB HDD and one 4 TB HDD. I plan to add one more 2 TB HDD in the near future, and more servers with many more HDDs in the far future (money is tight right now; am poor college student).

My currently proposed solution

I have researched this a lot, and I get this is a little over my head, but here's what I've got so far:

I'm thinking that Ceph is currently my best bet when it comes to flexibility and it seems stable.

My plan would be to to put BTRFS on the drives to handle bit rot, and then run Ceph as a single node cluster for later expansion.

Questions about how this would work

Some specific questions I have about my proposed setup:

1. I know that BTRFS can have bit rot protection, but is that by default? What do I need for it to be enabled?
2. I know that the inconsistency in drive size can be a problem (one 2 TB, one 4 TB), but will it work until I get another 2 TB drive?

Answer 1

Based on what you've said, I think you're probably right that Ceph is your best option, provided that you can give it enough processing power. However, BTRFS is probably not your best option for a storage backend. Ceph already has good options for detection and handling of bit-rot, and you will get much better performance by using those and running on top of XFS than you will using BTRFS.

Regarding your specific questions though:

1. Kind of. Checksumming is enabled by default, but you just get IO errors on failed checksums unless you explicitly use a storage profile in BTRFS itself that can recover. If you choose to use BTRFS< I would suggest just returning the IO errors, and let Ceph do the recovery, as it will usually recover faster.
2. It should work fine, provided you don't try to store more data than can fit on the smallest drive. Once you have enough that you can balance replicated copies, it should scale up just fine too (but make sure to have the system redistribute the storage usage when you add a new drive).

However, if you can't throw much computing power at it, Gluster probably is your best option, for the reasons outlined by davidgo in his reply, wiht the caveat that BTRFS is still useful when running Gluster.

As a point of comparison, Gluster is usually constrained on low-end systems by network performance, whereas Ceph is often (but not always) constrained by both networking and context switching performance. As a more specific example of a comparison, I have 2-node cluster consisting of two 250 USD Intel NUC's (NUC5PPYTH) with 4G of DDR3-1333 RAM each (which I'm ironically looking at replacing with a couple of Raspberry Pi 3's, because they should actually get better average performance). Running Gluster on top of BTRFS, I get about 20MB/s read and write speed. Running Ceph on top of BTRFS, it's roughly half that for read speed, and between half and one quarter for write speed, but they bottleneck on different things (Gluster is bottlenecked on the network, because the NIC's in the systems are mediocre at best, but Ceph pegs the CPU and the network).

Answer 2

I think you might want to reconsider your needs and priorities, because they are inconsistent. (Cheap, highly reliable, scalable - pick 2 at most) The most incongruous is talking about a Raspberry Pi and the bit rot. (Also, the links you provided are probably not that relevant in as much as they are not about general storage)

You may be able to "get further" by scrapping the requirement of redundancy built into the filesystem - you can achieve this on filesystems which would otherwise not support it by using software RAID. (Thats not to say the correct file system won't have this)

Similarly, you may want to drop the requirement for remote mounting using FUSE - you can achieve this either with NFS or SSHFS. (again, not saying you can't get this with your ultimate choice, just makes sense to strike the requirement off your list, because it can be worked arround quite easily)

I (unfortunately - I just put down a fair chunk of money on it) believe the "Gold Standard" solution for a scalable solution for your usage case is to run a ZFS system with RAID2Z- but to get ultimate reliability you need ECC Ram - and the system is not easily scalable between multiple nodes unless you use ISCSI.

Another solution (which is probably more realistic in terms of your requirements) would be something like Gluster. You can distribute the hard drives and have (distributed even) fault tolerance, and it runs fine even on a Raspberry Pi. Its also fairly straight forward to set up. This solution won't get around the write hole and is not as sophisticated as ZFS or BTRFS, but way more in line with what I suspect your goals are.