Compare two image files for identical data - excluding metadata?

Question

I have lots of photos from the web I’d like to add GPS exif data to, but I rely on removing duplicates by using comparison tools like cmp, md5sum andduff.

If I edit the exif data, then no tools I know of will recognize duplicate images. I’m looking for a tool (or reliable 1-line pipeline command) that can either:

• (preferred) generate a digest that is only based on the body and not any metadata
• report 2 images as identical pixel-for-pixel

Does such a tool exist? (Or is there any article that elaborates on why it is a problem not worth solving?)

Further info

The reason I prefer a digest over a black-box duplicate checker is that I like to create flat .txt files as a database of other interesting information about pictures (e.g. ranking images).

Answer 1

You can use exiftool to removed all of the metadata and to output what is left, i.e. the image to standard out in binary format with the command:

exiftool filename.jpg  -all= -o - -b

And of course md5sum will except standard in as an input so you should be able to construct a pipe such as:

exiftool filename.jpg  -all= -o - -b | md5sum -

Answer 2

For some reason exiftool did not work for me as it seemed to leave some metadata on, but replacing it with imagemagick convert worked great:

convert -strip filename.jpg -| md5sum

Answer 3

FWIW here is a quick one-liner to remove files where the image matches but the metadata does not:

find *_201* *_202[0-3] -type f -print | egrep -i "(jpg|arw|dng|tif|crw|cr3|nef|hif|heif|cs2|jpeg|png)" | while read file; do ofile="../$file"; if [ -f "$ofile" ]; then sha=`magick "$file" - 2>/dev/null | shasum`; osha=`magick "$ofile" - 2>/dev/null | shasum`; if [ "$sum" -eq "$osum" ]; then /bin/echo -n "-"; rm "$file"; else echo "$ofile differs - $osum vs $sum" >&2; fi; else echo "$ofile does not exist" >&2; fi; done 2> 20240527_exceptions.txt

I filtered out all the non-image c**p. Also I should not that the files I am removing are in a sub-folder. Each folder has the year appended and I don't want to delete anything from 2024. I write exceptions to a file called exceptions.txt and a string of "-" to stdout so I can see that it's doing stuff. There will be a short pause ...

I'm running this on a Mac under Sonoma so you may need to change a few details if you are a Linux person or other platform.

Answer 4

Here's how to generate a digest based only on pixel values using ImageMagick.

First, create some images, in various formats, i.e. BMP, GIF, PNG

magick -size 64x48 xc:red +write a.png +write a.gif a.bmp

Now get hex digests over just the pixels:

magick identify -format "%#:%f\n" a.*  
                      
dd2631e2587efb4fad5891366d906fa2cf3f554cc5c2f4da8ea0495c97b92909:a.bmp 
dd2631e2587efb4fad5891366d906fa2cf3f554cc5c2f4da8ea0495c97b92909:a.gif
dd2631e2587efb4fad5891366d906fa2cf3f554cc5c2f4da8ea0495c97b92909:a.png

You can check it ignores metadata such as the timestamp by creating two PNGs with different timestamps:

magick -size 64x48 xc:red a.png
sleep 2                             # to ensure a second has elapsed
magick -size 64x48 xc:red b.png

Now check the hashes still match despite different time:

identify -format "%#:%f\n" [ab].png

dd2631e2587efb4fad5891366d906fa2cf3f554cc5c2f4da8ea0495c97b92909:a.png
dd2631e2587efb4fad5891366d906fa2cf3f554cc5c2f4da8ea0495c97b92909:b.png

See what md5sum makes of them:

md5sum [ab].png
5d3c269e627b518c33148479018a1530  a.png
47717fe7262c465fca2a99b411f37c9f  b.png

Answer 5

There is also a Python project called jpegdupes (previously called imgdupes) that is specifically made to check .jpg files with identical image content, ignoring any changes in metadata (e.g. EXIF and other metadata).

It will check for image files and create a .signatures directory that will record the hashes for the image content. After it's done, it will display a list of the found duplicates as well as some options to delete them.

It is only specific to jpeg rather than all image types, though.