5

Can anyone recommend Windows software that will traverse a directory tree and compare every file byte-by-byte with another similarly structured tree.

To be considered identical, the files need to have the same name, content, and position in the two trees.

All I need to know is which files are different, and which are missing from the second tree. (If it reveals which files are in tree #2, but not in tree #1, that's a nice bonus.)

I need to compare tens of thousands of files, spanning hundreds of gigabytes, so speed and simplicity are important. Reliability and accuracy are essential.

This question is similar to this other question, but I don't need line-by-line reports of what is different within each file. I also need to go beyond 6 levels in a tree; the software must traverse the entire tree.

It is also similar to this question, but I need the actual bytes of the files compared. The software cannot assume two files with the same name (even with the same date and time) are identical.

Recommendations for gratis software are preferred, as are open-source, but all recommendations are welcome.

Improve this question
8
  • If I understand it right, you're looking for a duplicate finder.
    – Thomas Weller
    Commented Jul 19, 2016 at 10:53
  • Saying byte-by-byte: wouldn't a checksum+size compare do as well? Pretty much unlikely two files having the exact same size and checksum but "differences in bytes".
    – Izzy
    Commented Jul 19, 2016 at 11:22
  • @Thomas asked "If I understand it right, you're looking for a duplicate finder." Well, not really, but I see how you read that. I expect 99% of the files to be duplicates. Also, unlike a duplicate file finder, all files must be in the same position in the corresponding tree to be considered identical.
    – Amazon Dies In Darkness ♦
    Commented Jul 19, 2016 at 21:02
  • 1
    @Izzy asked "Saying byte-by-byte: wouldn't a checksum+size compare do as well?" Good question. I'm hesitant to say yes, because checksums are close to perfect, but not perfect. I have recently done some work in which I discovered how often different data sets can generate identical checksums (of course, the longer the checksum, the more unique). Also, generating checksums would still require each file to be read. All that said, I'm open to answers that use checksums, especially if there are real benefits.
    – Amazon Dies In Darkness ♦
    Commented Jul 19, 2016 at 21:08
  • 1
    @defaultlocale Yes. Actually, I prefer WinMerge2011 ( bitbucket.org/jtuc/winmerge2011/downloads ) which is an improved version. But, unfortunately, regardless of which WinMerge version is used, it seems to take a very long time to compare large files. IIRC, it takes about an hour to compare two 1GB files. So just 50 of those can take 2 days. Do you know of any settings for it that makes it faster for large files?
    – Amazon Dies In Darkness ♦
    Commented Jul 20, 2016 at 3:11

2 Answers 2

Reset to default
3

If the similarly structured tree is identical you can use Beyond Compare.

This software can easily compare folders and its content

Screenshot of application

Improve this answer
1
1

One of the simplest ways to do what you are looking for, not byte by byte but with a very low probability of missing anything, is to generate a complete tree listing for both trees with a fingerprint such as MD5 or SHA1 for every file.

You can then compare the two listings line by line. This can be done quickly with python.

You could even extend such a script to read in the "master" tree finger print file and to display, when processing the comparison tree, any files & directories that are:

  1. Missing from the comparison tree
  2. In the comparison tree but not in the master
  3. Files that are in both but with a different fingerprint (MD5/SHA1)

A little more work would create a tool that could update the entries from the master tree only if the time/date/size changed. And, of course, once you have the "fingerprint" it is quick and simple to compare your master tree with itself later to spot what has changed.

While Windows does not ship with python, it is readily available, free, from www.python.org.

Improve this answer
1
  • 3
    For me, "simplicity are important. Reliability and accuracy are essential" are not met with a solution I have to develop myself
    – Thomas Weller
    Commented Jul 19, 2016 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.