Distributed file system over network

Question

I'm looking for a system that will be cross platform allowing me to utilize excess disk space on commodity hardware to create a single large disk.

For example, if I have 400 GB available across 10 machines, each machine should be able to see a single 4 TB drive (or less with replication).

I'm not certain that Software RAID is what this is and I've examined the Comparison_of_distributed_file_systems Wikipedia article.

• Network latency is not an issue as it will be primarily for archiving
• Preferably this would run in software and run parallel to the existing
• FOSS preferred, but proprietary can be considered

Any recommendations or experience?

Answer 1

I could recommend the Lustre Filesystem. It's GPL licensed and used by Cray for their XC line of supercomputers. I have written parallel applications that heavily leveraged this particular FS across about 100 nodes; though I don't know how to set it up or configure it. Using it is just as you would expect with everything appearing just as files and directories though; the files were moved across the nodes seamlessly after they were written.

The pdf of the operator's manual can be viewed here.

Answer 2

Off the top of my head, any one of these sounds like it might meet your requirements:

• GlusterFS: GNU GPL3
• MongoDB: Server Side Public License (SSPL)
• InterPlanetary File System: MIT license, Apache license 2.0

The GlusterFS I usually use is set up so that each file in the special "DATA" folder is stored twice somewhere on a 3-server cluster, each one running Red Hat, so if any one entire server is disconnected or unavailable, the data can still be retrieved from the other servers. So 10 machines, each with 400 GB reserved for Gluster, could act more or less like a single 10*400MB/2 = 2 TB drive. As a normal user on any one machine, my "DATA" folder appears to act the same as the other normal local folders in Red Hat (permissions, sub-folders, etc.), except that any changes appear to instantly show up in the corresponding "DATA" folder on any other machine in my cluster.

The MongoDB I usually use is a key:document datastore without any permissions, subfolders, etc.; but I hear that other people use the MongoDB GridFS to add those things.

Have you had a chance to look at the Wikipedia article on distributed file systems and the Wikipedia list of distributed file systems? I'm glad you've already seen the Wikipedia article Comparison of distributed file systems; that sounds like exactly what you want.

I'm not familiar enough with the other file systems in those articles to tell whether they are better or worse for your application.