1

Sometimes I have the problem that I want to check for new files in one directory compared to another one. The situation in more detail:

I have a directory SRC with lots of files and I have a directory REF with lots if files. Now I want to know which files in SRC are new, i.e. are not in REF.

Note the file comparison should be regardless of the a file's location in the directory structure: e.g. if SRC/a/file1.txt and DEST/zq/file1.txt are identical, they should not be listed.

This is similar to duplicate detection, but kind of the opposite, too. I am looking for a command line tool which produce a list of the new, non-duplicate files in SRC. Under no circumstances it should do anything with the files in REF.

I had a look at a few duplicate finders: fdupes, rdfind, fastdupes, DedupeCopy and dupeGuru. From my understanding none of them offeres an direct way to list the non duplicates in SRC. The tool should run at least under OSX, but a platform-independent solution (maybe in Python) would be best.

Before I re-invent the wheel and write a script myself, did I overlook a command line option in the listed tools or is there another command line program achieving want I want?

  • Too bad you're asking for a command line tool only. Total Commander can do very flexible comparisons, including not, or reverting the selection after a compare. – Jan Doggen Jul 31 '14 at 8:00
2

It might not be exactly what you're looking for, but I'm curious if a variant on

rsync -vnr --size-only SRCDIR DESTDIR

Will give you what you want.

  • --size-only means that it will also list any files that are different sizes (in this case, the important part is that it shuts off timestamp comparisons)
  • -n suppresses the actual 'sync' part of rsync, making it not actually modify the DESTDIR
  • -v verbosifies it, so we get a printout of what would have happened
  • -r ecursive
  • You might hit a corner case where --ignore-existing would be handy

You may find other useful options


An alternative grep/find option that will ignore directories:

fgrep -vf <(find REF -type f -printf "%f\n") <(find SRC -type f -printf "%f\n")

  • Like your way of thinking, but unfortunately this does not work in all cases: If the folder structure in SRC and REF is different (as it is for me indeed), rsync will report wrong files as non-duplicates, just because there are under another path the other directory. – halloleo Jul 30 '14 at 12:35
  • @halloleo: Hmm... so, for example SRC/a/file1.txt and DEST/zq/file1.txt both exist and that counts as "duplicate"? If so I suspect you'll want to pipe find into diff or perhaps awk... – jkerian Jul 30 '14 at 12:48
  • Yep, same file in different directories is stil the same file. I edit the question accordingly. - That's why I ask for a tool which finds non-duplicates fast (i.e. with a smart (hash?) algorithm) - similar to the mentioned duplicate finders... – halloleo Jul 30 '14 at 21:51
  • Sorry to spoil your many ideas, but the find/grep approach does not check the content of the files at all (neither a real content comparison nor at least a file size check). – halloleo Aug 6 '14 at 2:21
1

Writing the full script for you and making it robust will take some more time, but here are the elements you need.

find $SRC -type f -exec basename \{\} \; | sort > SRCfiles.txt
find $REF -type f -exec basename \{\} \; | sort > REFfiles.txt
diff SRCfiles.txt REFfiles.txt | grep "^<"

Gives you a list of file names that appear only in SRC.

Edit: If you want a list of files that are new to SRC or the files are different, then you need to perform a checksum, using the md5 command. Then, reformat the line and use join to find files that differ by name or content.

Perl file basehash.plx:

use File::Basename;

while (<>) {
  @F = split;
  print basename($F[1]),'/',$F[0],' ',$F[1];
}

Run this script

find $SRC -type f -print | xargs -n 1 md5 -r | perl basehash.plx | sort >SRCfiles.txt
find $REF -type f -print | xargs -n 1 md5 -r | perl basehash.plx | sort >REFfiles.txt
join -a1 -j1 SRCfiles.txt REFfiles.txt

gives you

forcing.txt/1706a9b5433058d1d5992b99fe3f7198 src/infl/sample/forcing.txt

if either forcing.txt occurs only in SRC or has different contents than that of REF.

  • This answer has the same issues as the fgrep / rsync answer: The commands do no check the content of the files at all (neither a real content comparison nor at least a file size check). – halloleo Aug 6 '14 at 2:23
  • @halloleo the md5 calculates the file hash. Same hash = same content. – jm666 Aug 7 '14 at 21:35
  • Ups, yes, with Perl script you are correct. – halloleo Aug 8 '14 at 1:21
0

I recently used a python script to find all the duplicate files in several directories by using os.walk and hashlib.md5 to generate a default dictionary of all the files with matching MD5 values, (MD5 as key & list of paths as the value), then output all the entries with more than one path, you could do the same but simply output all the paths where the list has a length of 1.

  • Free
  • Cross Platform
  • Surprisingly fast
  • Very few lines of code.

Like this:

import sys
import os
import hashlib


def file_md5(file_path):
    """ Get the MD5 of a file."""
    block_size=2**20
    md5 = hashlib.md5()
    with open(file_path, "b") as infile:  # Must read all files as binart
        for data = infile.read(block_size):
            md5.update(data)
    return md5.digest()

def get_tree_hash_dict(startdir):
    """ Return a dictionary of MD5 values vs paths for a directory tree."""
    md5dict = {}
    for root, dirs, files in os.walk(startdir):
        for filename in files:
            thepath = os.path.join(root, filename)
            md5 = file_md5(thepath)
            md5dict[md5] = thepath

    return md5dict

def print_delta_paths(refdict, otherdict):
    """ Find the unmatching items."""
    refset = set(refdict.keys())
    otherset = set(otherdict.keys())
    print '-' * 80
    print "Unique to Ref:"
    for md5 in (refset - otherset):  # Set arithmatic
        print refdict[md5]
    print '-' * 80
    print "Unique to Other:"
    for md5 in (otherset - refset):  # Set arithmatic
        print refdict[md5]
    print '-' * 80

if __name__ == "__main__":
    if len(sys.argv) < 3:
        print "Call with a referece directory and one or more to check"
    else:
        print "Getting Ref Keys"
        refdict = get_tree_hash_dict(sys.argv[1])
        for other in sys.argv[2:]:
            print "Getting keys for", other
            otherdict = get_tree_hash_dict(other)
            print_delta_paths(refdict, otherdict)
        print "DONE!"

I am sure that the above could be improved upon, e.g. if ref never changes you could pickle the dictionary for it, etc.

  • Sounds great. Bring it on! :-) – halloleo Aug 6 '14 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.