At our institute we have large (~200TB) collections of binary data, where the file size may range from KB to GB, but mostly it is in the range of 1-100 MB. We need to store them so project groups can work on them (derive new data). Currently we are working on a directory structured file system with file naming conventions. However, as used to modern version control systems some features would be very handy:
- commit hook (when new data arrives, execute some scripts, reject or accept commit, etc.)
- search through meta data of the binary files
- only keep the last three versions of a file once it has been replaced by a newer version (as we have a tape library in background for backup)
- export subtree of directories for collaboration with other institutes
Ideally it would be best not to replace our filesystem based solution but to work on top of it. So we don't want to convert binary data into spurious (or proprietary) formats.
Ah and it should be on-premises software ideally open source. Thank you so much for your input.