I'm thinking of getting into producing video/audio. In other words, working with large files which are going to be created, worked on for a while, and then likely not touched again for a long time (originals and backups of the edited product). In order to

  • reduce the total size my everyday backup system (currently Apple Time Machine) has to deal with, and
  • protect against unnoticed damage over the long term (bit rot, accidental deletion/editing),

I'd like to establish a separate primary storage and semi-automatic backup system for these files, with the following properties:

  1. There's a working area which I can access to add files to, which is located on some disk that might be an external HD/SSD or a NAS device or whatever.
  2. I can at my convenience attach regular external hard drives (or another physical storage medium) to it to sync to them as backup copies.
  3. The system will alert me if any file is deleted or modified from the primary storage during backups or sooner, or just prevent the change from happening. I don't want truly “append-only”, but I want the default assumption to be that if a file is changed or deleted, without my taking special action, then this was a mistake which should not also end up erasing the backups if I do nothing.

You could think of this system as being somewhat like a DVCS such as Git; the notable difference I don't particularly want to support multiple versions of individual files (insofar as I have versions just naming them differently will do fine), and I do want the system to have good performance characteristics with large files and large total repository size (multiple terabytes). On the other hand, if there is a DVCS that can do this easily, that would suit me fine as long as it doesn't have too much unnecessary complexity to routinely deal with.

Clarifications and nice-to-haves:

  • It would be acceptable if the solution for (1) and (2) were separate from the solution for (3); for example, the former could be addressed by regular backup/sync/copy software and the latter could be a program that builds a catalog of files and checksums and compares against the previous version of the catalog. (In that particular case, it'd be neat if I could also browse the catalog without the storage being attached.)

  • Encryption is not a required feature. Neither is compression, since the large files will typically be incompressible.

  • Features related to archiving video/audio would be nice but the system should be able to store arbitrary file types.

  • Preferably, the backup copies should be accessible as ordinary files rather than needing to use the software that made them to read them.

  • It does not have to automatically make backups when I attach external storage; doing it on command is fine.

  • I'm also planning to upload selected files to my web server to make them public; if the system supported “this subdirectory alone is also synced there” that would be handy and save an extra step.

  • I'm open to suggestions for completely different workflows than the one I'm imagining, as long as the result still doesn't silently and irreversibly sync deletions, and is robust against damage.

Platform, compatibility, meta:

  • Should run on macOS and Linux. Does not need to preserve macOS-specific file attributes (though that might turn out to be useful).
  • Preferably, should be boring well-tested open source software that's been around for years and I just haven't heard of or thought of in this context.
  • While I'm not insistent on literally plugging in hard drives, the backups must not be to a cloud service; I want a solution that does not have recurring fees and is not going to abruptly stop working because of someone else's business decision or security mistake (only mine).

Options I've already thought of:

  • From reading the documentation, git-annex seems like it could do what I want. But it does a lot more than I want, too; all else being equal I'd like a tool with fewer configurations/states to be in. And instead of complaining about deletions or edits it would either sync them automatically (while preserving history so this is not irreversible) or require me to use both git-annex and git commands to record every file update (which would be acceptable but not optimal).
  • There are other "add large file handling to your favorite DVCS" tools; maybe one of them is simpler and more fitting than git-annex.
  • I briefly web searched for NAS devices and found that Synology NASes can be configured to have an add-only access control policy for specific users, and backup to external drives. However, the closest backup policy to what I want was "if a file is changed, write a renamed copy" rather than "if a file is changed or deleted, report an error", so as a system it's vulnerable to errors in the internal storage.
  • I could use rsync or any other sync tool for the backups and write myself a tool that generates and checks catalogs, and package that all together in a reasonably convenient script. But then it doesn't have any useful features that I don't get around to writing myself, and it would redundantly read files for the sync step and the verification step. (A suggestion for a catalog-making-and-checking tool would be interesting, leaving me only with the simpler problem of gluing it all together.)
  • Not used it myself, and it might not cover all your wishes – but Git Annex might be worth a look. Yes, it's especially for large files – and AFAIR from back when I read on it, you can add as much redundancy as you wish: other machines, other disks… And even decide which disk/machine shall cover what. Not sure it would alert you on deletion, though, or how far things can be automated with it – as I wrote, I haven't used it myself. – Izzy Mar 8 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.