11

What I am searching for is a software to find duplicate files in Linux. I am expecting it meets these requirements.

Must haves

  • Runs natively on Linux (not through Wine or PlayOnLinux)
  • Is able to find duplicates based on content (size, name or date don't matter)
  • Manual selection and batch deletion of duplicates is possible
  • Able to scan both ext4 and NTFS drives

Optionals

  • GUI
  • Sorting on the basis of size, time, no. of duplicates etc.
  • Is open source

Things that don't matter

  • Time it takes to scan drives
  • Resources it eats while its scanning (CPU, GPU or RAM)

So any recommendations?

Improve this question

5 Answers 5

Reset to default
8

rmlint (http://rmlint.readthedocs.org/en/latest/)

  • Runs natively on Linux
  • Is able to find duplicates based on content
  • Manual selection and batch deletion of duplicates is possible (creates a batch script for you to review / edit / run)
  • Able to scan both ext4 and NTFS drives
  • GUI (in develop branch)
  • Sorting on the basis of size, time, no. of duplicates etc.
  • Is open source

Also, even though they don't matter:

  • Is very fast
  • Is light on RAM even with huge datasets

Extra points of note:

  • Includes option for byte-by-byte file matches instead of SHA1 checksum
  • Can output to csv or json so you can do further processing and analysis
Improve this answer
2
  • Thanks for suggestion, rmlint definitely is faster than fdupes and loved how it made a shellscript for review. Its gonna be my primary dupe finder now ;).
    – H. Freeze
    Oct 15, 2015 at 12:14
  • // , I confirm the light CPU and RAM requirements. I recommend using the command line version, if you know how to do the whole ctrl+alt+t thing. youtube.com/watch?v=xLg4HWzlsE8 <<< This guy has a good video on it, although he doesn't discuss installation.
    – Nathan Basanese
    Sep 12, 2016 at 4:08
4

fdupes should do what you want.

From Wikipedia:

fdupes is a program written by Adrian Lopez to scan directories for duplicate files, with options to list, delete or replace the files with hardlinks pointing to the duplicate. It first compares file sizes, partial MD5 signatures, full MD5 signatures, and then performs a byte-by-byte comparison for verification.

fdupes is written in C and is released under the MIT License.

Improve this answer
3
  • +1 for a great suggestion, Its seems to be exactly the software i needed, a gui would have been good but it works great without it nonetheless.
    – H. Freeze
    Sep 29, 2015 at 2:07
  • 1
    If you like fdupes, you might also like the samesame suite
    – baitisj
    Oct 12, 2015 at 1:23
  • 1
    @baitisj Tried using samesame, its a good alternative for fdupes but it has its learning curve.
    – H. Freeze
    Oct 15, 2015 at 12:19
1

If you want a GUI, KDE has komparator:

http://komparator.sourceforge.net/

It uses the MD5 checksum and hasn't been updated since 2013, but it still works okay.

Improve this answer
1
  • +1,Thanks for suggestion, it does work great.Though was using gnome which made installing it dependency hell for me ;)
    – H. Freeze
    Oct 9, 2015 at 12:08
1

Please check findimagedupes (http://tuxdiary.com/2012/05/20/find-visually-similar-images-in-ubuntu/), a visually similar image finder on Linux. In addition to duplicates, it finds the same images with different resolutions, formats etc.

Improve this answer
1
  • Welcome to Software Recommendations – and thanks for sharing! While it certainly contains a solution, this post does not contain enough information to be considered a quality answer. Please read our discussion on what makes an answer high quality to see if you can incorporate some of these improvements into your answer. Especially, your answer should show how the product meets the requirements and what makes it recommendable. Also it would be preferable to include the core behind that link (apt-get)
    – Izzy ♦
    Oct 22, 2015 at 20:44
1

fslint: http://www.pixelbeat.org/fslint/

  • runs natively and is open source: is there in Debian

  • GUI and selection of dups to delete:

GUI and selection of dups to delete

  • intelligent detection of duplicates, as said in the FAQ:

Q. What algorithm is used to check for duplicate files?

A. On a standard install, you can see the script used in /usr/share/fslint/fslint/findup. In summary the algorithm is:

  1. exclude files with unique lengths

  2. handle files that are hardlinked to each other

  3. exclude files with unique md5(first_4k(file))

  4. exclude files with unique md5(whole file)

  5. exclude files with unique sha1(whole file) (in case of md5 collisions).

  • tools other than file duplicates (bad symlinks, empty directories etc)
Improve this answer
1
  • fslint is still switching over from Python 2 dependencies, so isn;t a great option at the moment. Hopefully it'll be upgrade soon.
    – JSCard
    Aug 17, 2020 at 21:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.