8

What I am searching for is a software to find duplicate files in Linux. I am expecting it meets these requirements.

Must haves

  • Runs natively on Linux (not through Wine or PlayOnLinux)
  • Is able to find duplicates based on content (size, name or date don't matter)
  • Manual selection and batch deletion of duplicates is possible
  • Able to scan both ext4 and NTFS drives

Optionals

  • GUI
  • Sorting on the basis of size, time, no. of duplicates etc.
  • Is open source

Things that don't matter

  • Time it takes to scan drives
  • Resources it eats while its scanning (CPU, GPU or RAM)

So any recommendations?

6

rmlint (http://rmlint.readthedocs.org/en/latest/)

  • Runs natively on Linux
  • Is able to find duplicates based on content
  • Manual selection and batch deletion of duplicates is possible (creates a batch script for you to review / edit / run)
  • Able to scan both ext4 and NTFS drives
  • GUI (in develop branch)
  • Sorting on the basis of size, time, no. of duplicates etc.
  • Is open source

Also, even though they don't matter:

  • Is very fast
  • Is light on RAM even with huge datasets

Extra points of note:

  • Includes option for byte-by-byte file matches instead of SHA1 checksum
  • Can output to csv or json so you can do further processing and analysis
  • Thanks for suggestion, rmlint definitely is faster than fdupes and loved how it made a shellscript for review. Its gonna be my primary dupe finder now ;). – H. Freeze Oct 15 '15 at 12:14
  • // , I confirm the light CPU and RAM requirements. I recommend using the command line version, if you know how to do the whole ctrl+alt+t thing. youtube.com/watch?v=xLg4HWzlsE8 <<< This guy has a good video on it, although he doesn't discuss installation. – Nathan Basanese Sep 12 '16 at 4:08
3

fdupes should do what you want.

From Wikipedia:

fdupes is a program written by Adrian Lopez to scan directories for duplicate files, with options to list, delete or replace the files with hardlinks pointing to the duplicate. It first compares file sizes, partial MD5 signatures, full MD5 signatures, and then performs a byte-by-byte comparison for verification.

fdupes is written in C and is released under the MIT License.

  • +1 for a great suggestion, Its seems to be exactly the software i needed, a gui would have been good but it works great without it nonetheless. – H. Freeze Sep 29 '15 at 2:07
  • 1
    If you like fdupes, you might also like the samesame suite – baitisj Oct 12 '15 at 1:23
  • 1
    @baitisj Tried using samesame, its a good alternative for fdupes but it has its learning curve. – H. Freeze Oct 15 '15 at 12:19
1

If you want a GUI, KDE has komparator:

http://komparator.sourceforge.net/

It uses the MD5 checksum and hasn't been updated since 2013, but it still works okay.

  • +1,Thanks for suggestion, it does work great.Though was using gnome which made installing it dependency hell for me ;) – H. Freeze Oct 9 '15 at 12:08
0

Please check findimagedupes (http://tuxdiary.com/2012/05/20/find-visually-similar-images-in-ubuntu/), a visually similar image finder on Linux. In addition to duplicates, it finds the same images with different resolutions, formats etc.

  • Welcome to Software Recommendations – and thanks for sharing! While it certainly contains a solution, this post does not contain enough information to be considered a quality answer. Please read our discussion on what makes an answer high quality to see if you can incorporate some of these improvements into your answer. Especially, your answer should show how the product meets the requirements and what makes it recommendable. Also it would be preferable to include the core behind that link (apt-get) – Izzy Oct 22 '15 at 20:44
0

fslint: http://www.pixelbeat.org/fslint/

  • runs natively and is open source: is there in Debian

  • GUI and selection of dups to delete:

GUI and selection of dups to delete

  • intelligent detection of duplicates, as said in the FAQ:

Q. What algorithm is used to check for duplicate files?

A. On a standard install, you can see the script used in /usr/share/fslint/fslint/findup. In summary the algorithm is:

  1. exclude files with unique lengths

  2. handle files that are hardlinked to each other

  3. exclude files with unique md5(first_4k(file))

  4. exclude files with unique md5(whole file)

  5. exclude files with unique sha1(whole file) (in case of md5 collisions).

  • tools other than file duplicates (bad symlinks, empty directories etc)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.