0

I need to check if the files contained in A folder (and its sub-folders) are present (as duplicates) in B folder or in one of its sub-folders BUT I don't want to see duplicate files present in B folder/sub-folders itself!

To make it more clear, take the following example scenario:

Folder A tree is:

A
|_ DIR A1
|  |_ file_a1_1
|
|_ DIR A2
|  |_ file_a2_1
|
|_ file_a_1
|_ file_a_2
|_ a_file_not_duplicated

Folder B tree is:

B
|_ DIR B1
|  |_ file_a1_1
|  |_ file_a_1
|
|_ DIR B2
|  |_ file_a2_1
|  |_ file_b_1   (<= only duplicated in B)
|  |_ file_b_2   (<= only duplicated in B)
|
|_ file_a_1
|_ file_a_2
|_ file_a1_1
|_ file_a2_1
|_ file_b_1
|_ file_b_2

So each file in A folder/sub-folders, with the exception of the "a_file_not_duplicated" one, has a duplicate in B folder or in one of its sub-folders. Now, a "traditional" duplicate finder software would list me, besides of those, even the files that are duplicates on B folder/sub-folders (file_b_1 and file_b_2 in the example) but that are not present in A folder/sub-folders and that are the ones I don't want to be shown and that haven't to be included among the results of the duplicates list I expect.

A free software for this would be great, but I'm prepared to consider/evaluate paid one too.

Any suggestion? Thank you!

P.S. In the example I used the same file names to point out which ones had duplicates in B folder/sub-folders but, of course, the duplicates identification method has to be based primarily on file contents/checksum (as usually occurs with duplicate finders).

EDIT: I need something that can run on Windows or Linux (both if possible).

SECOND EDIT:

To better clarify what I'm asking here, I'm adding two graphic representation with some additional explanations:

enter image description here

Legend:

  • A|B = the set of all files contained in A|B (root) folder plus all its sub-folders;

  • a/b n. d. = A/B set files that don’t have duplicates (unique ones);

  • type 1 dupes = A|B files that have (at least) one duplicate in B|A set and no duplicates in A|B set itself;

  • type 2 dupes = A|B files that have (at least) one duplicate in B|A set and (at least) one duplicate in A|B set also;

  • type 3a/b dupes = A/B files that have (at least) one duplicate in A/B set only and no duplicates in B/A set;

(Summing up type 1 & type 2 dupes = files that have (at least) one duplicate in the other “folder&sub-folders” files set and that might (or not) also have duplicates in their own “folder&sub-folders” files set.)

Another way to represent the situation is:

enter image description here

The darkened area represents, besides non duplicated files, the duplicate ones I don't want to be shown among results: I don't want "type 3b" duplicates, that is duplicate files contained in B plus its sub-folders and that have no duplicate in A plus its sub-folders.

2
  • If your question was answered to your satisfaction, you can accept an answer by clicking on the check mark under the voting arrows. Sep 1, 2017 at 17:07
  • @Kodiologist : please read back my question, I edited it to make it more clear. And, please, read my comments to your answer too.
    – danicotra
    Sep 4, 2017 at 20:02

3 Answers 3

1

Here's another possible solution to this in Python that can be run in both Linux and Windows:

#!/usr/bin/env python

#This script originates from the need to give an answer to this question on SoftwareRecs:
#https://softwarerecs.stackexchange.com/questions/45293/check-for-folder- subfolders-duplicates-in-another-folder-subfolder-but-not-check
#Credit goes to Kodiologist for having deeply inspired and motivated me with his answer: https://softwarerecs.stackexchange.com/a/45316
#(this script is based on his code)

#This uses scandir and requires Python >= 3.5
#
#USEFUL REFERENCES:
#http://benhoyt.com/writings/scandir/
#https://github.com/benhoyt/scandir

from os import scandir, path
import hashlib

class element_hashed(object):
    __slots__ = ('path', 'name')
    def __repr__(self):
        return "<element_hashed path:%s name:%s>" % (self.path, self.name)
    def __str__(self):
        return "element_hashed: path is %s, name is %s" % (self.path, self.name)

#Source dir data structure:
#dict[HASH] -> dict[id_1...n] -> element_hashed('path', 'name')

#Repository dir data structure:
#dict[HASH] -> dict[id_1...n] -> "a found item" # just a simple filler, this information isn't really important for the purpose

srcdir = 'A'
repdir = 'B'
dcount = 0

def hashes(topdir, repository=False):
    for entry in scandir(topdir):

        #I'm not going to follow symlinks, I want only "effective" files/directory
        #https://www.python.org/dev/peps/pep-0471/
        #there was basic consensus among the most involved participants, and this PEP's author [...] to warrant following symlinks by default
        #it's straightforward to call the relevant methods with follow_symlinks=False if the other behaviour is desired.

        if entry.is_dir(follow_symlinks=False):
            yield from hashes(entry.path)

        elif entry.is_file(follow_symlinks=False):
            with open(entry.path, "rb") as o:
                if repository:
                    eh = "a found item"
                else:
                    eh = element_hashed()
                    eh.path = topdir
                    eh.name = entry.name
                yield eh, hashlib.sha256(o.read()).hexdigest()


if __name__ == '__main__':
    import sys
    if len(sys.argv) != 3:
        sys.exit("usage: " + sys.argv[0] + " source_dir repository_dir \n(relative paths may be used)")
    else:
        if not (path.isdir(sys.argv[1]) and path.isdir(sys.argv[2])):
            sys.exit("ERROR: one or both non-existent directory!")
        srcdir = sys.argv[1]
        repdir = sys.argv[2]

source_hashes = dict()

for eh, key in hashes(path.abspath(srcdir)): #use source absolute path always
    if not key in source_hashes:
        source_hashes[key] = dict()
    source_hashes[key][path.join(eh.path, eh.name)] = eh

for _, h in hashes(path.abspath(repdir), True): #use repository absolute path always
    if h in source_hashes: # found _ 1b|2b type _ dup. item
        if len(source_hashes[h]) > 1:
            for eh in source_hashes[h]:
                print("Type 2a dup. :=> ", eh)
                dcount += 1
        else:
            print("Type 1a dup. :=> ", next(iter(source_hashes[h])))
            dcount += 1
        source_hashes.pop(h, None)

#now the remainders are all the "srcdir" files that don't have any duplicate in "repdir" = unique ones + type 3a dupes. I'm only interested in the latter
for fn in source_hashes.values():
    if len(fn) > 1:
        for eh in fn:
            print("Type 3a dup. :=> ", eh)
            dcount += 1

print("\nFound ", dcount, " dupes")
0

If I understand your question correctly, you want to verify that every file in A has at least one copy in B. Here's a Python program that computes a hash of every file in both directories and prints out the names of files that appear in A but not B. This program is free software and runs on both Windows and Linux.

import os, hashlib

source = 'A'
destination = 'B'

def hashes(topdir):
    for dirpath, _, filenames in os.walk(topdir):
       for fn in filenames:
           fn = os.path.join(dirpath, fn)
           with open(fn, "rb") as o:
               yield fn, hashlib.sha256(o.read()).hexdigest()

source_hashes = dict()
for fn, h in hashes(source):
    source_hashes[h] = fn
for _, h in hashes(destination):
    source_hashes.pop(h, None)

for fn in source_hashes.values():
   print("Didn't find a copy of", fn)
5
  • Hi there :) Thank you for your reply, your code doesn't achieve what I requested (please, see how I updated my question to clarify what I need). Also only last occurrence of duplicate files matching same hash are listed...yet you made my light bulb turn on! You gave me inspiration I didn't think of making something by my own at first but, after seeing your answer, I do now. It shouldn't be complicated to modify your code to achieve the desired result and I began working at it so to a have a python solution for this to provide a Linux solution + a second one for Windows (beside the one I found)
    – danicotra
    Sep 4, 2017 at 19:57
  • P.S. I can't upvote this by now, but I just upvoted a couple of your answers here on softwarerecs concerning python ;-D
    – danicotra
    Sep 4, 2017 at 20:00
  • @danicotra I'm afraid I don't understand the material added in the second edit of the question. I'm sorry, since you clearly put a lot of effort into it. You might try asking somebody who speaks your native language. But it sounds like you figured out how to use my code to achieve what you want, so I guess your problem is solved after all. I'm glad I was able to help. Sep 4, 2017 at 22:29
  • Ok, thank you and I'm sorry I couldn't be clear as I hoped... I should provide a new answer using python script inspired by your answer in few days (I do this in my spare time); most likely that'll be more understandable to you than any of my attempts to explain it with words, I guess :-D
    – danicotra
    Sep 4, 2017 at 22:40
  • I made it! :) It took me much less time than I expected, it's very impressive how Python lets you obtain such things so quickly and with so few coding... amazing! THANK YOU SO MUCH again for giving me the starting idea with the code in your answer! :)
    – danicotra
    Sep 5, 2017 at 20:02
0

I came to a solution to this for Windows O.S. I think I've tried almost all available free duplicates finder software out there but no one provide such a "folder self-scan exclusion" feature, so I began trying paid software and then came across the DoubleKiller Pro demo.

DK Pro lets you compare contents choosing which folders&sub-folders have to be considered as "library". This means, the elements in that folder won't be compared by themselves (i.e. directory self-scan exclusion), here's an usage example, first what happens if you use "classic" folder comparison:

B folder among the 'fresh' directory list failed to keep unwanted duplicate files off results

As you see, some duplicate files in B folder/sub-folders that I didn't want to be listed because they don't really has duplicates in A folder/sub-folders are there among results. Now, here's what happens putting B as "library" directory list instead:

B folder among the 'library' directory list now unwanted duplicate files don't appear anymore

You'll notice unwanted duplicate files now don't appear anymore and stay off the results. If I want to better highlight which A folder/sub-folders duplicate files are present in the result list I can now check 'by criteria...' (using an appropriate path matching mask) and then sort results by 'path' and I'll get:

enter image description here enter image description here

So, in the end, this did the trick. If I'd want to delete the selected files or export entries I'd have to buy a license, although it wouldn't be such a prohibitive expense (a private use license costs 14.95 € / 19.95 US-$ after all, quite affordable and reasonable price when worth it) but in the case you only need to see a list (when you're not dealing with hundreds of files or if you don't want to delete anything and don't even need to export entries listing) you can even save your money.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.