Check for folder/subfolders duplicates in another folder/subfolder BUT not checking for the duplicates in the second folder itself

Question

I need to check if the files contained in A folder (and its sub-folders) are present (as duplicates) in B folder or in one of its sub-folders BUT I don't want to see duplicate files present in B folder/sub-folders itself!

To make it more clear, take the following example scenario:

Folder A tree is:

A
|_ DIR A1
|  |_ file_a1_1
|
|_ DIR A2
|  |_ file_a2_1
|
|_ file_a_1
|_ file_a_2
|_ a_file_not_duplicated

Folder B tree is:

B
|_ DIR B1
|  |_ file_a1_1
|  |_ file_a_1
|
|_ DIR B2
|  |_ file_a2_1
|  |_ file_b_1   (<= only duplicated in B)
|  |_ file_b_2   (<= only duplicated in B)
|
|_ file_a_1
|_ file_a_2
|_ file_a1_1
|_ file_a2_1
|_ file_b_1
|_ file_b_2

So each file in A folder/sub-folders, with the exception of the "a_file_not_duplicated" one, has a duplicate in B folder or in one of its sub-folders. Now, a "traditional" duplicate finder software would list me, besides of those, even the files that are duplicates on B folder/sub-folders (file_b_1 and file_b_2 in the example) but that are not present in A folder/sub-folders and that are the ones I don't want to be shown and that haven't to be included among the results of the duplicates list I expect.

A free software for this would be great, but I'm prepared to consider/evaluate paid one too.

Any suggestion? Thank you!

P.S. In the example I used the same file names to point out which ones had duplicates in B folder/sub-folders but, of course, the duplicates identification method has to be based primarily on file contents/checksum (as usually occurs with duplicate finders).

EDIT: I need something that can run on Windows or Linux (both if possible).

SECOND EDIT:

To better clarify what I'm asking here, I'm adding two graphic representation with some additional explanations:

Legend:

• A|B = the set of all files contained in A|B (root) folder plus all its sub-folders;

• a/b n. d. = A/B set files that don’t have duplicates (unique ones);

• type 1 dupes = A|B files that have (at least) one duplicate in B|A set and no duplicates in A|B set itself;

• type 2 dupes = A|B files that have (at least) one duplicate in B|A set and (at least) one duplicate in A|B set also;

• type 3a/b dupes = A/B files that have (at least) one duplicate in A/B set only and no duplicates in B/A set;

(Summing up type 1 & type 2 dupes = files that have (at least) one duplicate in the other “folder&sub-folders” files set and that might (or not) also have duplicates in their own “folder&sub-folders” files set.)

Another way to represent the situation is:

The darkened area represents, besides non duplicated files, the duplicate ones I don't want to be shown among results: I don't want "type 3b" duplicates, that is duplicate files contained in B plus its sub-folders and that have no duplicate in A plus its sub-folders.

Answer 1

Here's another possible solution to this in Python that can be run in both Linux and Windows:

#!/usr/bin/env python

#This script originates from the need to give an answer to this question on SoftwareRecs:
#https://softwarerecs.stackexchange.com/questions/45293/check-for-folder- subfolders-duplicates-in-another-folder-subfolder-but-not-check
#Credit goes to Kodiologist for having deeply inspired and motivated me with his answer: https://softwarerecs.stackexchange.com/a/45316
#(this script is based on his code)

#This uses scandir and requires Python >= 3.5
#
#USEFUL REFERENCES:
#http://benhoyt.com/writings/scandir/
#https://github.com/benhoyt/scandir

from os import scandir, path
import hashlib

class element_hashed(object):
    __slots__ = ('path', 'name')
    def __repr__(self):
        return "<element_hashed path:%s name:%s>" % (self.path, self.name)
    def __str__(self):
        return "element_hashed: path is %s, name is %s" % (self.path, self.name)

#Source dir data structure:
#dict[HASH] -> dict[id_1...n] -> element_hashed('path', 'name')

#Repository dir data structure:
#dict[HASH] -> dict[id_1...n] -> "a found item" # just a simple filler, this information isn't really important for the purpose

srcdir = 'A'
repdir = 'B'
dcount = 0

def hashes(topdir, repository=False):
    for entry in scandir(topdir):

        #I'm not going to follow symlinks, I want only "effective" files/directory
        #https://www.python.org/dev/peps/pep-0471/
        #there was basic consensus among the most involved participants, and this PEP's author [...] to warrant following symlinks by default
        #it's straightforward to call the relevant methods with follow_symlinks=False if the other behaviour is desired.

        if entry.is_dir(follow_symlinks=False):
            yield from hashes(entry.path)

        elif entry.is_file(follow_symlinks=False):
            with open(entry.path, "rb") as o:
                if repository:
                    eh = "a found item"
                else:
                    eh = element_hashed()
                    eh.path = topdir
                    eh.name = entry.name
                yield eh, hashlib.sha256(o.read()).hexdigest()


if __name__ == '__main__':
    import sys
    if len(sys.argv) != 3:
        sys.exit("usage: " + sys.argv[0] + " source_dir repository_dir \n(relative paths may be used)")
    else:
        if not (path.isdir(sys.argv[1]) and path.isdir(sys.argv[2])):
            sys.exit("ERROR: one or both non-existent directory!")
        srcdir = sys.argv[1]
        repdir = sys.argv[2]

source_hashes = dict()

for eh, key in hashes(path.abspath(srcdir)): #use source absolute path always
    if not key in source_hashes:
        source_hashes[key] = dict()
    source_hashes[key][path.join(eh.path, eh.name)] = eh

for _, h in hashes(path.abspath(repdir), True): #use repository absolute path always
    if h in source_hashes: # found _ 1b|2b type _ dup. item
        if len(source_hashes[h]) > 1:
            for eh in source_hashes[h]:
                print("Type 2a dup. :=> ", eh)
                dcount += 1
        else:
            print("Type 1a dup. :=> ", next(iter(source_hashes[h])))
            dcount += 1
        source_hashes.pop(h, None)

#now the remainders are all the "srcdir" files that don't have any duplicate in "repdir" = unique ones + type 3a dupes. I'm only interested in the latter
for fn in source_hashes.values():
    if len(fn) > 1:
        for eh in fn:
            print("Type 3a dup. :=> ", eh)
            dcount += 1

print("\nFound ", dcount, " dupes")

Answer 2

If I understand your question correctly, you want to verify that every file in A has at least one copy in B. Here's a Python program that computes a hash of every file in both directories and prints out the names of files that appear in A but not B. This program is free software and runs on both Windows and Linux.

import os, hashlib

source = 'A'
destination = 'B'

def hashes(topdir):
    for dirpath, _, filenames in os.walk(topdir):
       for fn in filenames:
           fn = os.path.join(dirpath, fn)
           with open(fn, "rb") as o:
               yield fn, hashlib.sha256(o.read()).hexdigest()

source_hashes = dict()
for fn, h in hashes(source):
    source_hashes[h] = fn
for _, h in hashes(destination):
    source_hashes.pop(h, None)

for fn in source_hashes.values():
   print("Didn't find a copy of", fn)

Answer 3

I came to a solution to this for Windows O.S. I think I've tried almost all available free duplicates finder software out there but no one provide such a "folder self-scan exclusion" feature, so I began trying paid software and then came across the DoubleKiller Pro demo.

DK Pro lets you compare contents choosing which folders&sub-folders have to be considered as "library". This means, the elements in that folder won't be compared by themselves (i.e. directory self-scan exclusion), here's an usage example, first what happens if you use "classic" folder comparison:

As you see, some duplicate files in B folder/sub-folders that I didn't want to be listed because they don't really has duplicates in A folder/sub-folders are there among results. Now, here's what happens putting B as "library" directory list instead:

You'll notice unwanted duplicate files now don't appear anymore and stay off the results. If I want to better highlight which A folder/sub-folders duplicate files are present in the result list I can now check 'by criteria...' (using an appropriate path matching mask) and then sort results by 'path' and I'll get:

So, in the end, this did the trick. If I'd want to delete the selected files or export entries I'd have to buy a license, although it wouldn't be such a prohibitive expense (a private use license costs 14.95 € / 19.95 US-$ after all, quite affordable and reasonable price when worth it) but in the case you only need to see a list (when you're not dealing with hundreds of files or if you don't want to delete anything and don't even need to export entries listing) you can even save your money.