3

While we have a tag for , that's not exactly what I am looking for.

I have a 12tB drive which is almost full (don't ask), which I fear contains many duplicated, or almost duplicated directories (including sub-directories, so .directory trees).

It may be that a file or two has been added, a READ_ME.txt has been deleted, or an .INI file has been updated in a directory which I had previously accidentally copied.

While substantially identical, those directories would not be identified as duplicates by tools which I know.

I am seeking a tool which will help me identify directories which I might want to examine and consider merging them or deleting one.

So, basically, find directories which are X pre cent or more identical. May allow me to define X, maybe allow me to define file extinctions to compare or exclude (e.g exclude *.ini, but include *.xls).

I don't care too much about the features. I just want something that will flag to me that I might want to take a look.

While I would strongly prefer to find something off the shelf, I fear that this is yet another entry on my personal "code this" list . If you would like such a tool, please add feature requests in comments, in case I do actually ever get round to coding it.

Improve this question
7
  • 1
    It's eerie how much of a kindred spirit we are on this issue. I too asked a similar question a while ago (don't even know which site), and while I use TreeSize, I am still seeking a solution. Anywhere we can communicate directly and maybe build one - I need someone to guide my thought on how to approach this.
    – Lockszmith
    Dec 25, 2021 at 15:53
  • 1
    You could start a chat. I am not sure how much input I would have though. Just some measure of number of identical files, probably a s a percentage, and overall size of identical files, again, probably as a percentage, with a user-defined threshold to define similarity. Possibly, a green/yellow/read colo(u)r code, or similar. The tricky parts is text files. Do you just want to consider if they are identical? Or if a line/lines was added/removed/changed? It's easier with binary files, I think; more clear cut. If you don't get much help here defining the algorithm, try Reddit, for discussion.
    – Mawg says reinstate Monica
    Dec 26, 2021 at 8:55
  • Mawg, none of these do exactly what you want (hence not adding an answer), but may be of help to you and Lockszmith: FreeFileSync, WinMerge, SyncBackFree. If you have trouble finding any of them, just let me know, and I'll get you the official links.
    – End Anti-Semitic Hate ♦
    Jan 17, 2022 at 3:25
  • @Lockszmith Pinging you if the comment I just posted above helps you as well.
    – End Anti-Semitic Hate ♦
    Jan 17, 2022 at 3:25
  • 2
    While looking into implementation aspects, I stumbled upon a hashing algorithm called xxHash, searching for projects using the algorithm I found these 2 projects that in tandem might be able to address my use case: github.com/razum2um/xxhashdir_comm which relies on github.com/lunatic-cat/xxhashdir . I'll try this out and update here
    – Lockszmith
    Jan 26, 2022 at 14:17

2 Answers 2

Reset to default
1

If you can code something, a possible solution (inspired by Benjamin Pierce's unison tool) could be to compute some hash (e.g. MD5) for every file, keep it in some database (perhaps using sqlite or redis or gdbm), sort that database by MD5 hashes, and compare the size and contents of every file with the same MD5 hash.

You could also get some inspiration (or borrow some code) from omake.

You should expect the software to run for several hours, during which you should not run any other program on your computer.

Improve this answer
2
  • This is along the lines of what I've been thinking. With matching folders being rated by how many similar hashes they contain, and how many of those similar hashes have the same names.
    – Lockszmith
    Jan 20, 2022 at 20:54
  • In case you're interested in following up. See my comment to the question
    – Lockszmith
    Jan 26, 2022 at 14:18
0

I can't think of an ideal solution, but:

  • Treesize can show you directories sorted by size - it would give you an idea on which folders are similar. It also shows number of files/folders within each folder

enter image description here

Improve this answer
1
  • Thanks KERR, yes, I actually use WinDirStat for stuff like this, and indeed it is somewhat an indicator - looks like developing a tool is the only recourse at this point. If my mind will allow me some focus I might try and tackle a command line tool for this at first
    – Lockszmith
    Jan 22, 2022 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.