5

I am looking for a file explorer that works on Windows 7 and can open a folder with millions of files quickly (e.g. 50 million). It should be GUI and list all the files. (The computer has SSD).

Windows Explorer takes around one minute for each million of files (has a progression bar), and Explorer++ takes a bit longer (no progression bar).

Improve this question
4
  • What is your file system, NTFS? How long does a dir /b > nul 2>&1 take?
    – Thomas Weller
    Commented Dec 6, 2015 at 20:52
  • And just out of curiosity: why would one place so many files in a single directory? What is the use case?
    – Thomas Weller
    Commented Dec 6, 2015 at 20:53
  • 1
    @ThomasWeller NTFS, around 10 seconds on a folder with 1.2 million files. Just a data set split into many JSON files, each file being between 5 and 15 KB. (they could be merged into larger JSON files, but it's more convenient to have them separated)
    – Franck Dernoncourt
    Commented Dec 7, 2015 at 6:08
  • Ok, those 10 seconds are probably the minimum you can get, dominated by the file system driver. Compared to 1 minute/million I guess that's acceptable.
    – Thomas Weller
    Commented Dec 7, 2015 at 17:08

1 Answer 1

Reset to default
2

It looks you are attempting to solve an X-Y Problem. I don't know of any operating system which actively (i.e. by special means) supports having more than few hundred files together in one directory/folder. Technically, you CAN do it everywhere, but the result is like you described in the question.

Solution 1: if you want to keep the information in the files:

Find a key to distribute files into directories and write a script to do it. It can be based on filename, file content, file date or just on a newly created unique value.

Example with key based on original filenames:

from folder containing files 10000000.dat thru 99999999.dat, move file
12345678.dat
into
12\34\56\78.dat (or 12\34\56\12345678.dat) (having 100 files/dirs in directory)
or into
12\345\678.dat (or 12\345\12345678.dat) (having 1000 files/dirs in directory).

Simple, fast, elegant, file access times return close to zero. (I recommend not exceeding 255 items in directory – first of above cases.)

Solution 2: if you can transfer your content to more suitable processing engine:

Create a suitable relational database and import the content of the files inside. Add indexes on fields you use for searching and you got efficient storage and retrieval mechanism for your data. Microsoft SQL Server Express is free and known for superior performance. Or go with MariaDB for a free solution from MySQL world.

Again, searching for "file explorer that can open a folder with millions of files quickly" is the same as searching for "2-D game library for Microsoft Excel which can produce more than 30 frames per second. I have SSD drives and fast CPU." The programmer who would write such a thing would waste his effort on a tool solving nonsense problem which could be easily avoided by proper thinking and organizing the data.

Improve this answer
2
  • 2
    There are probably people who are stuck with folders containing millions of files and can't modify the folder structure due to legacy software, for example an old enterprise application that puts its data in a folder and is being used way more than was ever imagined. A valid solution could be implemented rather easily, it would load files in a different thread, display the first results almost immediately and use caching+paging.
    – Nicolas Raoul ♦
    Commented Nov 21, 2018 at 7:56
  • @NicolasRaoul – Yes, that's a good idea. If anyone knows such a software, please share it here. Concerning this Q/A, the OP did not post a follow-up nor accepted the answer, so we don't know if he managed this somehow.
    – miroxlav
    Commented Nov 23, 2018 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.