I need to copy a few hundred GB of data between two machines. Due to limited diskspace and time I would far prefer known-good transfers on the first go, so I want something that verifies the data as it's transferring. (I will most definitely still be checksumming the files when they're done to make sure they really did transfer okay, but in-flight verification significantly minimizes the likelihood anything will have gone wrong, and I highly value this.)

The machine with the data on it is running Linux; the destination is a laptop running Windows 7 (at least until I get Linux installed on it as well). So, the tool needs to at least support Win7 and Linux.

My Wi-Fi is largely fairly stable but has the occasional hitch/jitter; it also does abruptly go out to lunch every so often, and of course power cycling it kills all active connections. So, I'm looking for something that gracefully handles "completely disconnected" and continually retries to reconnect until successful.

I've weighed up encryption vs no encryption and I've decided that while I don't really need it, I also don't see the point in making do with (or suggesting) programs that don't implement encryption, given the ubiquity of HTTPS.

Transparent compression is okay but not critically needed; I'm unsure if the data I'm transferring will compress well.

I'm currently banging together a barebones file-transfer script that fulfills my requirements, but if there's something already out there that does this I'd love to know because the script is taking a bit more time than I expected it would.

Here's what I've already looked into:

  • BitTorrent: perfect (client designed around network transience and poor quality), except for the lack of inherent file encryption and the fact that P2P is widely frowned upon, so I generally avoid using it as a rule

  • FTP/FTPS, HTTP/HTTPS: don't support verification :(

  • Metalink: the protocol is perfect (and it runs over HTTPS!), but I'm unaware of any clients that actually support piece hashes. I don't really think the whole idea of "...and...done! wait, your 100GB file's hash is wrong, we need to start over" makes any sense whatsoever.

  • Resilio Sync: closed-source, probably implements tracking; very reluctant to use :/

  • 1
    "at least until I get Linux installed on it as well" - is this relevant to the question? If it is going to happen soon, could you hold off on the data transfer and accept a Linux only solution? – Mawg says reinstate Monica Sep 24 '18 at 11:54

Syncthing seems to almost exactly fit what you're asking for here.

It provides:

  • Active verification of blocks of data as they are transferred, with automatic refetching of bad blocks.
  • In-transit encryption of the data.
  • High resilience to networking issues (it will reconnect automatically and pick up copying data where it left off).
  • Transparent compression of the data on-the-wire, though use of this si generally discouraged as it often does not save much time.
  • Runs on Linux and Windows (though it's a lot easier to work with on Linux).

Internally, it works very similarly to BitTorrent (it was designed originally as a competitor to BTSync).

The one possible downside is that it is designed for continuous synchronization of data more than one-shot transfers like you appear to want. This means that initial setup is going to be a bit more involved than some other options, and that you will have to be careful to not modify any of the files on the destination system while things are syncing (otherwise the modifications may get copied back to the source system), but otherwise it should work reasonably well for what you want.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.