3

I am looking for a program that I can run on windows that can check for duplicate (or larger/smaller resolution of same image) images (possibly different extensions), and If there are exact duplicates replace all with a hard-link. Otherwise if they are same image, but different resolution, I need to keep the largest one, and replace the rest with hard-links.

I know that there are lots of programs to determine duplicate files and create hard-links. However I am unaware of any that can find duplicate images with different resolution and create hard-links based on a given parameter.

Just for an idea of how many images are being worked with, its around 50,000. I would estimate that there is at least 20% duplicates (10,000 Hard-Links).

If there isn't any software already made that can do this, Could someone point me to a good C# Lib that can handle image comparisons.

Improve this question
1
  • that complex software could be imagemagick I think. – onurcano Apr 7 '17 at 12:05
1

I ended up googling a bit more (after failing with python for a while), and found a program called AllDup (freeware). It has successfully satisfied all of my needs, and more. There only seems to be two downsides with it. The first being a moderately complicated gui (Since its not just for images, there is a large number of unrelated options). The second is, AllDup is not fully automated. However its as close as I need.

Improve this answer
1

I've written a Python script to find visually similar images and delete all but the one with the largest size.

You can maybe look to modify it to "replace with hard-link" instead of deleting:

https://github.com/AnirudhKishan/DeleteVisuallyRedundant

Improve this answer
0

Not C# but python + Pillow can quickly walk the directory tree, extracting the image information and some sort of finger print such as the MD5 of a fixed, reduced, resolution of each image to locate exact duplicates that differ only in scale. It can handle most image file types, other than raw, and you could also specify that it compare files that are in different formats - possibly with a preference of which to keep in the event of a duplicate of similar dimensions.

It can also handle file deletion and hard link creation for you but before deletion I would suggest a compare of the pairs of images, scaled to the lower of the two resolutions, to make sure that they are duplicates.

I do strongly recommend making a backup of all of your images first.

Improve this answer
3
  • I will have to look into python and pillow a bit (I've have yet to work with python despite how many times I've heard of it's usefulness). Also is pillow capable of Finding images of similiarity (eg 99% similiar after being downscaled?) Since a good number of the pictures are almost identical, but have very minor differences (differences of only a few pixels) or would they still be considered identical in this case? – Jason Brown Apr 7 '17 at 7:55
  • You can measure the difference between 2 images, (once they have been scaled the same), but given the number of files I would suggest reducing in size quite drastically and generating an MD5 to locate candidates, (otherwise you are looking at many millions of comparisons of whole images. If the reduction was to say 128x128 then images that only differ by a few pixels are likely to be given as canidate matches - you could then do a pairwise compare for how different are they (once scaled to the same). – Steve Barnes Apr 7 '17 at 9:10
  • 1
    Ok, I will give it a try, and post back with results/source if it works well enough. – Jason Brown Apr 7 '17 at 10:08
0

The tool for you is Duplicate Photo Finder by MindGems, but it can not create symlinks. You can do that easily though via a bat file or something or external application as it can export the results to CSV or XML. Should be quite easy

This is the only tool that is really maintained and working properly - I work with photos and scan 100 000+ images. This is the only tool that will not crash on such volumes and will produce proper results. Over the years I tried many tools, but none of them does what is advertised so give it a try.

enter image description here

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.