1

I have a bunch of big files that are appendonly, however get renamed quite often.

For efficient backup I'd like to detect renames in the source directory using file hashes, and replicate these in the backup directory, so that I can synchronize the added parts (several hundred GBs) in the following step using rsync --size-only --append

However all the common hash/checksum calculators (sum, cksum, md5sum, shasum, b2sum, b3sum, ...) only perform hashing for the whole files, which is prohibitive here with TB sized files on network shares.

Instead I'd like to do partial file hashing focusing on only a small part at the start of the files (a few KB, or MB at worst, only as big a part as necessary to avoid hash collisions in source).

Is there any command-line software with an output similar to the ones named above but allowing customization of the maximal number of bytes to consider for each file (maybe even allowing to specify a startpoint)? (For MS Windows I've found something called Hash, done by a person named Dan Mares, but nothing for Linux so far).

Presently I'm doing something along:

for a in *; do echo $a $(dd if=$a bs=8k count=1 status=none | b3sum --no-names); done

However, I would really like to avoid the additional processes necessary here (dd, subshell, for-loop).

1

You could for example use Python 3:

import hashlib
file_name='abc.txt'
with open(file_name) as file_to_check:
    # read 8192 bytes of the file
    data = file_to_check.read(8192)    
    # pipe contents of the file through
    md5_returned = hashlib.md5(data).hexdigest()
2
  • Thanks for the suggestion - one up for cuteness ;-) . However, to be honest, i want to save time, not spend more. The dd-b3sum pipeline has done its job (BTW: the BLAKE3 guys did a fantastic piece of work) way before the python interpreter environment even has completed starting up and VM begins chewing through the code (had tried that kind of approach previously). The subshell is rather cheap in comparison. (BTW: in your code above the file has to be opened binary). – jf1 Feb 4 at 16:46
  • Well, why not writing your own program in Rust or Go (or C) if you need speed? You could open-source it afterwards. – scenox Feb 4 at 19:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.