2

Consider a read-only filesystem - or, better, part of a filesystem: a directory tree. How can I get redundancy for that without going to full RAID techniques (mirroring or otherwise)?

As an example of what I'm considering: WinRAR and Parchive are two archiving formats that allow for the creation of "recovery volumes" that provide redundancy for a set of identically sized files. They use standard RAID techniques (Reed-Solomon codes or something else suitable) to provide recovery against a certain number of lost or corrupted files: E.g., given a set of 10 large files you can create 2 recovery volumes to go with them such that any 10 of the 12 files will recover the full set. Or you can pick other levels of redundancy, 1 recovery file, 5, whatever.

So what I'd like is that - but without needing to put the directory tree into an archive! Is anything like that possible? (I.e., given a read-only directory tree I'd like to get one (or more) additional files that provide a certain level of redundancy against that directory tree at the cost of 10-30% disk space).

Acceptable solutions would include leveraging operating system features, e.g., userspace file systems, or something. (I'm specifically not interested in mere filesystem checksumming techniques that would allow me to detect corruption: It's recovery I want.)

I'm willing to be operating system agnostic and even filesystem agnostic - that is, if there is a technique that is OS- or filesystem- specific, I'd be willing to switch to it.

7
  • 1
    Why not RAID? RAID exists for a long time, it works. There is also software RAID. Also consider: none of the techniques you mention will prevent data loss. If your disk breaks, only a backup can help you. Why not an archive/backup? Archives/backups exist for a long time, they work. Oct 1 '20 at 19:10
  • IMHO you're asking an XY-question. What problem are you really trying to solve? Oct 1 '20 at 19:11
  • 1
    you could be right, @ThomasWeller - I want to store a large filesystem of files on a (large) removable drive - but still have them easily accessible when I put the drive in. It's not an easily raidable situation. I have been doing it with WinRAR and recovery volumes. But it obviously isn't as easy to browse the drive or pick out individual files when everything is in a large archive. I'd like to get the redundancy of winrar/parchive without packing the files. 1/2
    – davidbak
    Oct 1 '20 at 19:26
  • I have thought of splitting the single drive into 5 or 6 partitions, of equal size, and using software RAID on them. (RAID 5 or RAID 6). Write performance would of course suck. Read performance would probably be not much worse (maybe it would be worse due to many extra seeks to the separate partitions). Convenience when plugging in the drive would be low ... But, maybe that's the best approach? 2/2
    – davidbak
    Oct 1 '20 at 19:27
  • And how should the redundancy be stored when the removable drive is plugged off? Oct 1 '20 at 19:36