3

I am looking for a program that I can run on windows that can check for duplicate (or larger/smaller resolution of same image) images (possibly different extensions), and If there are exact duplicates replace all with a hard-link. Otherwise if they are same image, but different resolution, I need to keep the largest one, and replace the rest with hard-links.

I know that there are lots of programs to determine duplicate files and create hard-links. However I am unaware of any that can find duplicate images with different resolution and create hard-links based on a given parameter.

Just for an idea of how many images are being worked with, its around 50,000. I would estimate that there is at least 20% duplicates (10,000 Hard-Links).

If there isn't any software already made that can do this, Could someone point me to a good C# Lib that can handle image comparisons.

Improve this question
1
  • that complex software could be imagemagick I think.
    – onurcano
    Apr 7 '17 at 12:05
1

I ended up googling a bit more (after failing with python for a while), and found a program called AllDup (freeware). It has successfully satisfied all of my needs, and more. There only seems to be two downsides with it. The first being a moderately complicated gui (Since its not just for images, there is a large number of unrelated options). The second is, AllDup is not fully automated. However its as close as I need.

Improve this answer
1

I've written a Python script to find visually similar images and delete all but the one with the largest size.

You can maybe look to modify it to "replace with hard-link" instead of deleting:

https://github.com/AnirudhKishan/DeleteVisuallyRedundant

Improve this answer
0

Not C# but python + Pillow can quickly walk the directory tree, extracting the image information and some sort of finger print such as the MD5 of a fixed, reduced, resolution of each image to locate exact duplicates that differ only in scale. It can handle most image file types, other than raw, and you could also specify that it compare files that are in different formats - possibly with a preference of which to keep in the event of a duplicate of similar dimensions.

It can also handle file deletion and hard link creation for you but before deletion I would suggest a compare of the pairs of images, scaled to the lower of the two resolutions, to make sure that they are duplicates.

I do strongly recommend making a backup of all of your images first.

Improve this answer
3
  • I will have to look into python and pillow a bit (I've have yet to work with python despite how many times I've heard of it's usefulness). Also is pillow capable of Finding images of similiarity (eg 99% similiar after being downscaled?) Since a good number of the pictures are almost identical, but have very minor differences (differences of only a few pixels) or would they still be considered identical in this case?
    – Jason Brown
    Apr 7 '17 at 7:55
  • You can measure the difference between 2 images, (once they have been scaled the same), but given the number of files I would suggest reducing in size quite drastically and generating an MD5 to locate candidates, (otherwise you are looking at many millions of comparisons of whole images. If the reduction was to say 128x128 then images that only differ by a few pixels are likely to be given as canidate matches - you could then do a pairwise compare for how different are they (once scaled to the same).
    – Steve Barnes
    Apr 7 '17 at 9:10
  • 1
    Ok, I will give it a try, and post back with results/source if it works well enough.
    – Jason Brown
    Apr 7 '17 at 10:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.