I have lots of photos from the web I’d like to add GPS exif data to, but I rely on removing duplicates by using comparison tools like cmp, md5sum andduff.

If I edit the exif data, then no tools I know of will recognize duplicate images. I’m looking for a tool (or reliable 1-line pipeline command) that can either:

  • (preferred) generate a digest that is only based on the body and not any metadata
  • report 2 images as identical pixel-for-pixel

Does such a tool exist? (Or is there any article that elaborates on why it is a problem not worth solving?)

Further info

The reason I prefer a digest over a black-box duplicate checker is that I like to create flat .txt files as a database of other interesting information about pictures (e.g. ranking images).

  • 1
    Strictly EXIF data is Metadata not a header as it can potentially occur at locations other than the start of the file. – Steve Barnes Jul 9 '18 at 0:38
  • Thanks I will remove the word header from the post – Sridhar Sarnobat Jul 9 '18 at 0:59

You can use exiftool to removed all of the metadata and to output what is left, i.e. the image to standard out in binary format with the command:

exiftool filename.jpg  -all= -o - -b

And of course md5sum will except standard in as an input so you should be able to construct a pipe such as:

exiftool filename.jpg  -all= -o - -b | md5sum -
  • 1
    Note that the above does not change the file the file is read, metadata stripped and then the image data output into md5sum! You use sys::stdout rather than /temp/ – Steve Barnes Jul 9 '18 at 1:50
  • Oh wait, I didn’t read properly. If it uses sdtin then there will be no permanent damage – Sridhar Sarnobat Jul 9 '18 at 1:50
  • Come to think of it, when removing duplicate files I may end up removing the more desirable variant of the file. Maybe what really want is instead of deleting one of them, merging the exif data such that the md5sum output becomes identical – Sridhar Sarnobat Jul 9 '18 at 1:53
  • 1
    You only need to merge the EXIF data if the files are otherwise identical so your first step is to locate all of the duplicates (ignoring the EXIF data) then you can potentially use exiftool to extract all of the metadata from both files merge it and set one or both files to contain it - this will not be a one liner. – Steve Barnes Jul 9 '18 at 2:11
  • Yeah, a one liner is unrealistic after thinking this through more. It would be more of a shell script or groovy script if I want to do it reliably – Sridhar Sarnobat Jul 9 '18 at 2:38

For some reason exiftool did not work for me as it seemed to leave some metadata on, but replacing it with imagemagick convert worked great:

convert -strip filename.jpg -| md5sum

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.