File Comparison Tool (Checksum based, Windows 2016/10)

Question

So I am looking for a tool that can compare files in folders based on checksums (this is common, not hard to find); however, my use-case is that the files can exist in pretty deep folder paths that can change, I am expected to compare them every few months and ONLY create a package of the different files. I don't care what folders the files are in, the same file can move between folders regularly and files wouldn't change names much, only content (so checksums are a must).

My issue is that almost all of the tools I can find do care about the folder paths when they compare folders, I don't and I actually want it to ignore the folder paths. I rather not develop anything or at least only have to develop a small part of the process to save time.

To be clear the order I am looking for things to happen are:

Program scans directory from 1/1/2020 (A).
Program scans directory from 4/1/2020 (B)
Finds all checksums in B that don't exist in A and make a new folder with the files that are different (C).

Any ideas? Also, this needs to only happen every 4 months and would cover only about 47gb (32,000 files). If it runs for 18 hours thats totally fine. I just need it to work.

Answer 1

I would suggest using a more sophisticated hashing than simple check sums, to avoid the possibility of hash collisions. Possible hashing values would SHA-1, SHA-256, etc.

You can do this with a few lines of python on almost any platform by using the built-in libraries, specifically os.walk to traverse your directory structure & hashlib to compute the hash values. You could even create a zip of the new/changed files with zipfile. Personally I would do something along the lines of:

#!python # the following code assumes python 3.8 or higher
import os
import datetime
import pickle # used to store the dictionary between runs
import hashlib

CHUNK_SIZE = 1024*1024  # A megabyte at a time adjust if necessary
TREE_ROOT = "/top/of/tree" # Where ever that is
SHA_FILE = "/some/other/path/tree_shas.pickle" # Adjust as needed

def hashfile(filepath):
    """ Calculate the hash of a single file """
    with open(filepath, 'rb') as infile:
        sha = hashlib.sha256()
        while chunk := infile.read(CHUNK_SIZE): # This will only work for python >3.8
            sha.update(chunk)
    return sha.digest()

# The above tested with a 12 MB file and took 39 msecs on my laptop

def check_tree(startfrom, last_shas):
   """ Check the contents of a tree against the sha values in last_shas list """
   newshas = set() # Empty Set
   for root, dirs, files in os.walk(startfrom):
       # You can skip some directories by removing them from dirs if present
       print(root, len(files), "Files", end="\r") # So we can see some progress
       for fname in files:
           sha = hashfile(os.path.join(root, fname))
           if sha not in last_shas:
               print("New/Changed file:", os.path.join(root, fname) # or some other action
           newshas.add(sha)
    return newshas

def main():
    """ Main Processing """
    started = datetime.datetime.now()
    sha_list = set() # Start with none
    if os.path.exists(SHA_FILE):
        sha_list = pickle.load(open(SHA_FILE, 'rb'))
    new_shas = check_tree(TREE_ROOT, sha_list)
    pickle.dump(new_shas, open(SHA_FILE, 'rb'), 4)
    print(f"\n\nCalculated {len(new_shas)} in {datetime.datetime.now() - started}")
    print(f"{len(new_shas.difference(sha_list)} New/Changed files")

if __name__ == "__main__":
    main()

• Free, Open Source & Gratis
• Just about any platform
• You may not have room to store all of the SHA values in RAM so may need to be more sophisticated
• Rather than printing the changed filenames you could perform whatever other action you need to do from within the script
• Suitable for a chron task on platforms that support it