Is there a two-way streaming file system or cloud provider that would allow offloading files from my local hard drive to a NAS or the cloud?

Question

Background

Is there some kind of a file system or cloud service that would allow me to offload things from my local hard drive (to a NAS or cloud), and still makes it seem that all my files are still available locally? Then when I try to access one of these files, it would stream them to me. If I save, then it would stream those changes back, keeping them sync. But: it would manage space efficiently, meaning files I access often would stay cached locally and things that I haven't touched for many months will be cleaned from the local cache. And maybe I could also specify how much I allot for local cache, e.g. 10..100 GB.

Requirements

I'm mainly looking for (free and) open source software, but I'd like to know if such commercial and/or closed source software exists as well -- especially if no (F)OSS versions are available.

I am mainly looking for web-based or linux-compatible apps, but I'd like to know if such software exists for other platforms as well -- especially if no web/linux versions are available.

So far

I know about Dropbox and Google Drive and alike, but AFAIK they don't have any kind of smart caching or file streaming mechanism. The best I can do with these is manually select some folders to be synced or not to be synced. And after that I have to wait to open files after it starts downloading.

I've heard about this tool: torrent-mount, which mounts a virtual file system of a torrent, immediately letting you look around it. And when you open up any file it starts streaming that file to you. What's really cool about this is that this process is all transparent to local applications: they just think that your hard drive is seeking and slow at reading, they have no idea the data comes from the web. All the network communication is abstracted away.

Downsides: AFAIK this is a one-way streaming solution, and the torrent protocol is read-only and can't really support streaming modifications upstream. Also, AFAIK, it doesn't keep track of how how many times you open a file, and you can't set a cache limit either. It's more of a cool tech demo showing: "Hey, look, it's possible to mount torrent files as virtual file systems." But the reading and streaming user experience is quite similar to what I am looking for.

Questions

Is there any software that you've used, tried or just know about that might fit for what I am looking for?

Or if you haven't heard about a tool like that and you also would like something similar, I would be glad if you told me -- maybe as a comment -- so at least I know I am not alone in wishing for something like this.

Answer 1

You prefered free and/or open source solutions, but not many answers have been posted. So I'll provide a non-free suggestion.

Google's updated Disk offering for Google Suite customers, called Drive File Stream, seems to fit most of your requirements, except the price tag.

It is a client that is installed on your computer, and makes files available in the operating system's file finder. When you access a file, it will be streamed from your Google Drive, and when you save a file, the updates will be streamed back up again.

You do not have to store the file locally on your computer. But if you know that you are going to be off line, and need to work on some documents, you can select them to be available off line,

The downside: It requires you to have a Google Suite subscription. The prices are currently €4/user/month for 30 GB storage, or €8/user/month for unlimited storage.

Answer 2

I have two options for you...

Resilio Sync (Windows, OS X, Linux, FreeBSD)

This is a utility for file sharing and synchronization based on torrent technology. It differs from mounting a torrent folder in that the communication is bi-directional, so you can modify.

The Pro version will give you control over what to synchronize, allowing you to download only the files you want. In addition, the pro version allow you to set permission per file.

Resilio has different levels of integration with each OS. If I recall correctly, on Linux it does not have a device to mount on the target folder, nor have placeholder files.

WebDrive (Windows, OS X, iOS, Android)

This is a utility to mount remote storage. Thus, it does not have to synchronize a folder to access its contents. Instead, it will offer the files using a custom device driver that hides the network operations.

You do not have to synchronize, but you can. It allows you to create sync tasks and schedule them to synchronize folders with the remote location.

More importantly, it can work offline if the connection is broken or you manually turn it to offline mode (you can even configure it to start a connection in offline mode, which will mount whatever is in cache※). Furthermore, you can configure how much cache storage you want for each remote location.

※: When doing this, it will mount without going to the network, so you will not see not cached files.

That is not all! The special thing is that WebDrive will let you use a variety of protocols and services, including Amazon Cloud, Dropbox, Google Drive, and of course any FTP server.

The downsides are that it is paid (20 day free trial) and there is no Linux version.