8

Note: I've read through the other questions and do not believe this is a duplicate (pun intended). Most "file de-duplicators" seem to want to help you reduce filespace. Instead, I want to consolidate and reorganize the directory structure; which will also result in removal of file duplicates.

Over the years I've collect a large number of files (now residing on various Windows 8 machines, USB drives, and a Stora NAS). In the past, I have had various backup strategies. This has resulted in many duplicate files and directories. Further, I've modified some directories, but of course haven't modified each and every backup of that directory.

I don't expect an automated tool to clean things up for me. De-duplication tools, for instance are of little value … I think partially because my files and directories are in such a mess.

I want a tool that will catalog all my files and help me to organize them. The final result would be a pristine new directory tree.

I've tried this before manually. The problem is that there are so many files, I can't do everything in one sitting. I also forget which directories I copied over and which I didn't. Do you know of a tool that could help?

Here are some of the features that I would like:

  • Build a tree structure of the directories on multiple drives.
  • Traverse tree and find duplicate sub-trees. Ideally, "auto merge" them if they really are duplicates. But also allow me to examine each one and make fine-grained decisions.
  • Don't actually do any moves. The first phase is just to setup a mapping of source files and destination files. Then after many days of work, I can say "go" for the entire process to work.
  • I don't want anything being deleted. at some point, I'll trust the pristine area, but not for a while.
  • For each source directory, I can either decide to copy it to the pristine area, or I can mark it as "unnecessary to move"
  • After the creation of the pristine area, I could then have the tool go back through the old file systems and report what was not copied, so that I could audit the list and adjust as needed.

Since I didn't easily find an answer to this while searching, does it mean I'm asking for something unreasonable? It seems like it would be a common problem.

Improve this question
3
  • I think the title should be reworded to focus more on directory trees. It will make the question stand out among others about duplicate files
    – Tymric
    Aug 18, 2014 at 9:10
  • 1
    You are making it hard on yourself with your requirement "Don't actually do any moves". If you do actual piecemeal moves it's a non-issue to track how far you are. All the material will go trough your hands, and all though you may not get it 100% correct in one go, it is fast, and has hardly any overhead. But I'm also sure your I can say "go" for the entire process to work will not give you a 100% perfect result either. Aren't you over-engineering this? How much material are we talking about anyway? - numbers help.
    – user416
    Jan 26, 2015 at 12:17
  • @JanDoggen, you are most likely correct. The total size is around 200G, which I guess isn't that much anymore. However, there are lots of old directories with .5M pictures in them. Ultimately, I did essentially as you describe...I copied all that I knew about and then have iterated a couple of times to try to catch everything. I also limited my consolidation, at the moment, to only image and movie files. That simplified things greatly. Thanks.
    – Robert Lugg
    Jan 27, 2015 at 17:43

1 Answer 1

Reset to default
1

You can use a general purpose automation tool like Linx to build a process that suits you. A process that might work for you can look like:

  1. Import file info to Excel or database.
  2. Use Excel or sql to manipulate the data. Add additional fields to store what you want done with specific files or folders.
  3. When you're ready, let Linx or the tool of your choice do the moving around.

Start small. Once you've got the process right move on to the bigger problems.

Here is a screenshot of the Linx Designer:

LinxDesigner

If that doesn't look too scary then

  1. Get Linx from linx.twenty57.com
  2. Download a working sample using Excel as the file info database: FileSystemCleanup.zip

Disclaimer: I'm on the team that develops Linx.

Improve this answer
3
  • Could you please explain in the answer how does it solve the OP's problem? Otherwise, your answer wouldn't be relevant to this specific question and does not belong here
    – Tymric
    Aug 29, 2014 at 13:20
  • Timmy, I appreciate your review. Any other ideas or options would be great too. I will look into this tool to see what I can do. Its a bit lower level than I was interested in, but may be easier that a scripted solution that I was considering. Besides, it does look like fun.
    – Robert Lugg
    Aug 31, 2014 at 12:59
  • After trying it out, Linx doesn't meet the requirements of my question. It could potentially simplify a scripting solution, but doesn't seem to have the interactive capabilities that I need.
    – Robert Lugg
    Sep 2, 2014 at 6:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.