5

Can anyone recommend Windows software that will traverse a directory tree and compare every file byte-by-byte with another similarly structured tree.

To be considered identical, the files need to have the same name, content, and position in the two trees.

All I need to know is which files are different, and which are missing from the second tree. (If it reveals which files are in tree #2, but not in tree #1, that's a nice bonus.)

I need to compare tens of thousands of files, spanning hundreds of gigabytes, so speed and simplicity are important. Reliability and accuracy are essential.

This question is similar to this other question, but I don't need line-by-line reports of what is different within each file. I also need to go beyond 6 levels in a tree; the software must traverse the entire tree.

It is also similar to this question, but I need the actual bytes of the files compared. The software cannot assume two files with the same name (even with the same date and time) are identical.

Recommendations for gratis software are preferred, as are open-source, but all recommendations are welcome.

  • If I understand it right, you're looking for a duplicate finder. – Thomas Weller Jul 19 '16 at 10:53
  • Saying byte-by-byte: wouldn't a checksum+size compare do as well? Pretty much unlikely two files having the exact same size and checksum but "differences in bytes". – Izzy Jul 19 '16 at 11:22
  • @Thomas asked "If I understand it right, you're looking for a duplicate finder." Well, not really, but I see how you read that. I expect 99% of the files to be duplicates. Also, unlike a duplicate file finder, all files must be in the same position in the corresponding tree to be considered identical. – RockPaperLizard Jul 19 '16 at 21:02
  • 1
    @Izzy asked "Saying byte-by-byte: wouldn't a checksum+size compare do as well?" Good question. I'm hesitant to say yes, because checksums are close to perfect, but not perfect. I have recently done some work in which I discovered how often different data sets can generate identical checksums (of course, the longer the checksum, the more unique). Also, generating checksums would still require each file to be read. All that said, I'm open to answers that use checksums, especially if there are real benefits. – RockPaperLizard Jul 19 '16 at 21:08
  • 1
    @defaultlocale Yes. Actually, I prefer WinMerge2011 ( bitbucket.org/jtuc/winmerge2011/downloads ) which is an improved version. But, unfortunately, regardless of which WinMerge version is used, it seems to take a very long time to compare large files. IIRC, it takes about an hour to compare two 1GB files. So just 50 of those can take 2 days. Do you know of any settings for it that makes it faster for large files? – RockPaperLizard Jul 20 '16 at 3:11
3

If the similarly structured tree is identical you can use Beyond Compare.

This software can easily compare folders and its content

Screenshot of application

  • Thanks. Looks interesting, although expensive. – RockPaperLizard Jul 30 '16 at 18:00
1

One of the simplest ways to do what you are looking for, not byte by byte but with a very low probability of missing anything, is to generate a complete tree listing for both trees with a fingerprint such as MD5 or SHA1 for every file.

You can then compare the two listings line by line. This can be done quickly with python.

You could even extend such a script to read in the "master" tree finger print file and to display, when processing the comparison tree, any files & directories that are:

  1. Missing from the comparison tree
  2. In the comparison tree but not in the master
  3. Files that are in both but with a different fingerprint (MD5/SHA1)

A little more work would create a tool that could update the entries from the master tree only if the time/date/size changed. And, of course, once you have the "fingerprint" it is quick and simple to compare your master tree with itself later to spot what has changed.

While Windows does not ship with python, it is readily available, free, from www.python.org.

  • 2
    For me, "simplicity are important. Reliability and accuracy are essential" are not met with a solution I have to develop myself – Thomas Weller Jul 19 '16 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.