1

So I am looking for a tool that can compare files in folders based on checksums (this is common, not hard to find); however, my use-case is that the files can exist in pretty deep folder paths that can change, I am expected to compare them every few months and ONLY create a package of the different files. I don't care what folders the files are in, the same file can move between folders regularly and files wouldn't change names much, only content (so checksums are a must).

My issue is that almost all of the tools I can find do care about the folder paths when they compare folders, I don't and I actually want it to ignore the folder paths. I rather not develop anything or at least only have to develop a small part of the process to save time.

To be clear the order I am looking for things to happen are:

Program scans directory from 1/1/2020 (A).
Program scans directory from 4/1/2020 (B)
Finds all checksums in B that don't exist in A and make a new folder with the files that are different (C).

Any ideas? Also, this needs to only happen every 4 months and would cover only about 47gb (32,000 files). If it runs for 18 hours thats totally fine. I just need it to work.

  • Welcome aboard :-) Which operating system should this run on? And are you willing to pay (if so, how much)? – Mawg says reinstate Monica Mar 18 at 8:01
  • 1
    I have it as a tag, but this is all windows server. I am willing to pay. Probably around $100 – DataforDays Mar 18 at 16:06
0

I would suggest using a more sophisticated hashing than simple check sums, to avoid the possibility of hash collisions. Possible hashing values would SHA-1, SHA-256, etc.

You can do this with a few lines of python on almost any platform by using the built-in libraries, specifically os.walk to traverse your directory structure & hashlib to compute the hash values. You could even create a zip of the new/changed files with zipfile. Personally I would do something along the lines of:

#!python # the following code assumes python 3.8 or higher
import os
import datetime
import pickle # used to store the dictionary between runs
import hashlib

CHUNK_SIZE = 1024*1024  # A megabyte at a time adjust if necessary
TREE_ROOT = "/top/of/tree" # Where ever that is
SHA_FILE = "/some/other/path/tree_shas.pickle" # Adjust as needed

def hashfile(filepath):
    """ Calculate the hash of a single file """
    with open(filepath, 'rb') as infile:
        sha = hashlib.sha256()
        while chunk := infile.read(CHUNK_SIZE): # This will only work for python >3.8
            sha.update(chunk)
    return sha.digest()

# The above tested with a 12 MB file and took 39 msecs on my laptop

def check_tree(startfrom, last_shas):
   """ Check the contents of a tree against the sha values in last_shas list """
   newshas = set() # Empty Set
   for root, dirs, files in os.walk(startfrom):
       # You can skip some directories by removing them from dirs if present
       print(root, len(files), "Files", end="\r") # So we can see some progress
       for fname in files:
           sha = hashfile(os.path.join(root, fname))
           if sha not in last_shas:
               print("New/Changed file:", os.path.join(root, fname) # or some other action
           newshas.add(sha)
    return newshas

def main():
    """ Main Processing """
    started = datetime.datetime.now()
    sha_list = set() # Start with none
    if os.path.exists(SHA_FILE):
        sha_list = pickle.load(open(SHA_FILE, 'rb'))
    new_shas = check_tree(TREE_ROOT, sha_list)
    pickle.dump(new_shas, open(SHA_FILE, 'rb'), 4)
    print(f"\n\nCalculated {len(new_shas)} in {datetime.datetime.now() - started}")
    print(f"{len(new_shas.difference(sha_list)} New/Changed files")

if __name__ == "__main__":
    main()
  • Free, Open Source & Gratis
  • Just about any platform
  • You may not have room to store all of the SHA values in RAM so may need to be more sophisticated
  • Rather than printing the changed filenames you could perform whatever other action you need to do from within the script
  • Suitable for a chron task on platforms that support it
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.