2

At our institute we have large (~200TB) collections of binary data, where the file size may range from KB to GB, but mostly it is in the range of 1-100 MB. We need to store them so project groups can work on them (derive new data). Currently we are working on a directory structured file system with file naming conventions. However, as used to modern version control systems some features would be very handy:

  • commit hook (when new data arrives, execute some scripts, reject or accept commit, etc.)
  • search through meta data of the binary files
  • only keep the last three versions of a file once it has been replaced by a newer version (as we have a tape library in background for backup)
  • export subtree of directories for collaboration with other institutes

Ideally it would be best not to replace our filesystem based solution but to work on top of it. So we don't want to convert binary data into spurious (or proprietary) formats.

Ah and it should be on-premises software ideally open source. Thank you so much for your input.

Improve this question
4
  • Sounds like you're looking for something like git-annex? I haven't tried it myself, so I cannot say if it fully fits.
    – Izzy ♦
    Aug 21, 2017 at 8:49
  • Thanks, git-annex is quite cool, I will suggest it to a staff member, as he is juggling with 100..200 hard drives. But for my use case it is probably not a good fit, as our file server provides enough^TM space. From our IT I got another recommendation: ZFS with its snapshot mechanism. It is known to use only little overhead. But I think it is very coarse grained compared to a user based versioning system. Also export for data sharing is somehow limited. From online search I got also interested into SciTran, but it seems to be tailored to medical lab environments.
    – math
    Aug 22, 2017 at 12:05
  • If I'd know whether (and how well) it can deal with those file sizes, I'd throw in Gitea – which I'm using myself and am quite happy with. Think of it as a "self-hosted Github" with most of its features (repo, PRs, issues, wiki). AFAIK binary data isn't "diffed" anyway, and Git has something called "LFS" (LargeFileSupport) you can install on top of it.
    – Izzy ♦
    Aug 22, 2017 at 12:13
  • Take a look at Dropbox and Acronis solutions.
    – Nakilon
    Sep 3, 2017 at 23:29

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.