3

What are some software solutions for transferring data between servers that are not located on the same LAN network which can open multiple download connections and has error correction to make sure files are not corrupted during transfer (or when pieced together).

Right now I am simply using HTTP (Apache) to download the data via Axel, but always needing to verify hashes on these large files is time consuming. And if a large 8GB+ file is corrupted I am either re-downloading, or converting it to torrent so that the torrent client can patch the file.

I tried the torrent protocol approach but also having to deal with creating the torrent files and then adding them to a client on both ends is time consuming.

I have tried rsync, although since it does not utilize multiple connections it doesn't seem to saturate my entire downlink at home. (Don't ask, the server is on gigabit fiber and both server ends are fine when testing to other networks and other users don't report the problem. My ISP won't admit there is a problem with their network route going to my server)

So is there any solutions that will allow me to easily transfer files ranging from 600mb to 8gb+ in size that allows multiple connections to be made that also includes error correction? I am just looking for a way to simply this process to not be as involved or time consuming than it already is.

Also as a bonus encryption support for the traffic would be a nice feature.

Thanks!

  • and when we have gigabit fibers, a small number of 8GiB retransfer is not a big problem. – 把友情留在无盐 May 11 '15 at 16:11
  • Well, typically ISP limitations are on the upload side, so I'd check the sending/server side first, not the receiving side. Also - what's in between can matter, but it may be hard/impossible to check. – Dan Cornilescu May 16 '15 at 4:47
  • 1
    What is "Axel"? – guettli Mar 16 '16 at 7:10
  • AFAIK you are not faster if you do parallel download. I would falsify your statement that rsync does not saturate the downlink. Please run five rsyncs in parallel. Do you get a better throughput? – guettli Mar 16 '16 at 7:13
2

A possibility could be plain scp (secure copy) which is part of the openssh package on most Linux distributions. It gives you integrity check, compression, encryption and you can launch as many scp commands in parallel as you like. Not as advanced as rsync but I found it more convenient in some cases.

  • Does it fix failures without re-downloading everything? – Nicolas Raoul Mar 16 '16 at 5:50
  • Transfer failures yes. But failures related to inability to write the files at destination no, those files are identified in error messages and have to be re-tried (i.e. re-downloaded) manually. Re-running the same scp cmd will re-download everything. – Dan Cornilescu Mar 16 '16 at 14:06
2

Is the data proprietary or sensitive? Do you need encryption?

If you need encryption, then you'll need a protocol that runs over ssh, like rsync.

If not, you should just script the creation of torrents, right?

Actually, I'd wager ssh prevented rsync from saturating your download link, so maybe specify an alternative transport with the --rsh options.

An interesting option might be using rsync locally but with a network block device as the back end, either running over ssh or not. I donno if hook a block device into an unencrypted connection is wise, but hey.

A wild option might be configuring a local dm-cache or dm-snapshot with a network block device back end, so your files stay remote but get cached locally.

Just fyi, CernVM-FS looks amazing (docs,github), but focusses on large numbers of small file.

  • Does it fix failures without re-downloading? – Nicolas Raoul Mar 16 '16 at 5:49
0

You can try with aria2. Copied from the website:

aria2 is a lightweight multi-protocol & multi-source command-line download utility. It supports HTTP/HTTPS, FTP, BitTorrent and Metalink. aria2 can be manipulated via built-in JSON-RPC and XML-RPC interfaces. Download

Features

  • Multi-Connection Download. aria2 can download a file from multiple sources/protocols and tries to utilize your maximum download bandwidth. Really speeds up your download experience.

  • Lightweight. aria2 doesn’t require much memory and CPU time. When disk cache is off, the physical memory usage is typically 4MiB (normal HTTP/FTP downloads) to 9MiB (BitTorrent downloads). CPU usage in BitTorrent with download speed of 2.8MiB/sec is around 6%.

  • Fully Featured BitTorrent Client. All features you want in BitTorrent client are available: DHT, PEX, Encryption, Magnet URI, Web-Seeding, Selective Downloads, Local Peer Discovery and UDP tracker.

  • Metalink Enabled. aria2 supports The Metalink Download Description Format (aka Metalink v4), Metalink version 3 and Metalink/HTTP. Metalink offers the file verification, HTTP/FTP/BitTorrent integration and the various configurations for language, location, OS, etc.

  • Remote Control. aria2 supports RPC interface to control the aria2 process. The supported interfaces are JSON-RPC (over HTTP and WebSocket) and XML-RPC.

Intended to be used from command line, but the RPC allows the usage of several GUI tools to handle it.

  • Does it fix failures without re-downloading? – Nicolas Raoul Mar 16 '16 at 5:48
0

Although you ask for a software solution, there might be solutions using standard packages which you can script much easier.

What I'm wondering is whether you can chop the files into chunks of up to size 100MB. (That's an arbitrary figure, choose a different size if you prefer). Then it would be easy to queue them into multiple instances of ncat or netcat, and send them that way.

You'll always have to ccheck the hashes at the far end (there's no certainty the data is correct unless you check the data). But a job that hashes a large number of 100MB files has a big advantage - like bittorrent, if one of them doesn't hash as expected, you have to ask the sender to put one chunk back on the queue, not the whole file.

It would take a bit of scripting, but not much, because it's all pretty standard *nix tools.

-2

To verify the checksum, along with the HTTP transfer:

curl http://1.1.1.1/1 | tee >(crc32 - >1.crc32) >1

HTTP over TCP is reliable in most cases.

  • 1
    This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. – Ken Herbert May 12 '15 at 4:54
  • 1
    @winterblood OP ask for means of file transfer, and this post do provide a solution of file transfer. – 把友情留在无盐 May 12 '15 at 5:24
  • 1
    @winterblood curl is a piece of software, so soubunmei might just have missed to include the source (apt-get install curl, Linux users know that). So this is a "software solution" as requested. // soubunmei: you might wish to extend a bit, though, how it meets the requirements (encryption→https? multiple connections? error correction?) and include the install part I've mentioned? – Izzy Mar 15 '16 at 21:50
  • 1
    This does not fix failures without re-downloading, right? – Nicolas Raoul Mar 16 '16 at 5:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.