Efficient rename detection using partial file hash

Question

I have a bunch of big files that are appendonly, however get renamed quite often.

For efficient backup I'd like to detect renames in the source directory using file hashes, and replicate these in the backup directory, so that I can synchronize the added parts (several hundred GBs) in the following step using rsync --size-only --append

However all the common hash/checksum calculators (sum, cksum, md5sum, shasum, b2sum, b3sum, ...) only perform hashing for the whole files, which is prohibitive here with TB sized files on network shares.

Instead I'd like to do partial file hashing focusing on only a small part at the start of the files (a few KB, or MB at worst, only as big a part as necessary to avoid hash collisions in source).

Is there any command-line software with an output similar to the ones named above but allowing customization of the maximal number of bytes to consider for each file (maybe even allowing to specify a startpoint)? (For MS Windows I've found something called Hash, done by a person named Dan Mares, but nothing for Linux so far).

Presently I'm doing something along:

for a in *; do echo $a $(dd if=$a bs=8k count=1 status=none | b3sum --no-names); done

However, I would really like to avoid the additional processes necessary here (dd, subshell, for-loop).

Answer 1

You could for example use Python 3:

import hashlib
file_name='abc.txt'
with open(file_name) as file_to_check:
    # read 8192 bytes of the file
    data = file_to_check.read(8192)    
    # pipe contents of the file through
    md5_returned = hashlib.md5(data).hexdigest()