Software to find duplicate files

Question

I need recommendations for a software for de-duplication.

Requirements:

• Runs on Windows

• Ability to find duplicate archives: files that have exactly the same content (and, so, the same size). Names, paths or dates are irrelevant.

• Options to perform automatic actions (like to keep the newest, the oldest, or files in some path) and also the manual action "do nothing, prompt for action". This last case (manual action) could be performed on more than one file

• Option to create symlinks: instead of simply removing all-but-one ocurrence, create symlink to one version.

• Graphical interface (and not only command prompt)

Answer 1

Duplicate Files Finder : http://doubles.sourceforge.net/

For Windows and Linux. Lets user find dup files (by their content, not only their names). The search can include and exclude files (ex.: exclude all *.tmp) and restrict which file sizes are searched.

Can remove dup files by deleting them or by creating links.

Instead of using hashes, it uses the following algorithm:

The algorithm is as follows: First, all files are sorted by their size, because files can be only equal, if they have the same size (logically). Then the files are compared with each other, and thus the equal files are determined. If two files are not equal from a given point on, reading is interrupted; no more has to be read for determining that these files are not equal. Because of this the results are determined much faster than in programs which use hashing algorithms, for which all files have to be read completely. Additional caching of the contents of the files additionally improves performance.

It's easy to use and fast. Can store the results (list of file names) in a .txt file, so that you can process it further with some other technique.

License: GPL

Answer 2

Well, for years now, I have used Duplicate Cleaner Free. It has an intuitive, but powerful, GUI, and lots of features, including "Turn duplicate file into hard links".

 -    Find duplicate files, fast!  
 -    Cleaner hard drives means faster performance.  
 -    Fast Scanning and accurate duplicate comparison.  
 -    Locate duplicate music (includes MP3, iTunes (M4A) and more.)  
 -    Find Similar images (resized, rotated, edited and more.)  
 -    Find duplicate folders.  
 -    Flexible search settings - search the way you want to.  
 -    Intelligent Selection Assistant.  
 -    Search home or networked drives.  
 -    Search multiple locations.  
 -    Export / Import your results.  
 -    View all file information in a detailed list.  
 -    Large, resizable windows for easy browsing.  
 -    Handy image preview window with side-by-side comparison.  
 -    Delete duplicate files to the Recycle bin.  
 -    Move dupe files to a new location.  
 -    ***Turn duplicate file into hard links***.  
 -    Protection for Windows and system files.  
 -    Supports multiple languages.  
 -    Comprehensive user manual.  
 -  For Microsoft Windows XP, Windows Vista, Windows 7 and 8.  

Answer 3

I have been using Manyprog Find Duplicate Files to remove duplicate files successfully for many years.

Answer 4

Total Commander: Alt+F7, advanced, search for duplicates: [x] size, [x] content. It uses similar algorithm like Duplicate Files Finder.

Licence: shareware, but you can use it forever, it only asks you to press a button on start. Full text:

Total Commander for Windows is a Shareware (demo version) program. This means that you can test it for a period of 30 days. After testing the program, you must either order the full version, or delete the program from your harddisk.

Far manager can probably make it too.

Answer 5

MindGems Fast Duplicate File Finder is worth a look: https://www.mindgems.com/products/Fast-Duplicate-File-Finder/Fast-Duplicate-File-Finder-About.htm

No affiliation.

Not sure if it does do these that you are asking for:

Options to perform automatic actions (like to keep the newest, the oldest, or files in some path) and also the manual action "do nothing, prompt for action". This last case (manual action) could be performed on more than one file

Option to create symlinks: instead of simply removing all-but-one ocurrence, create symlink to one version.

However what makes the program distinct, I feel, is that it allows you to keep one of the folders in the comparison intact. In that, with 2 given folders, if mutual duplicate files are found in both folders, the program can be instructed to delete only the files in one of the folders.

This feature is great if you have a master copy of a set of files, that you want to keep intact, and are determining what other folders contain that have the same files.