I occasionally need to copy a single 180GB file over a shared WAN, and robocopy just isn't cutting it. Is there any software (paid is fine, for a reasonable fee) that will automatically multithread the copy of a single huge file (preferably using UDP, and managing it's own validation)? A Google search turns up a lot of old posts, dead links, programs aimed at copying many smaller files, or sketchy Russian web sites.

The file is already compressed.

(Using 7zip to split the file into multiple pieces, having robocopy /mt:16 copy the "slices" and then reassembling them using 7zip on the other side, would speed things up, but that adds a lot of overhead and still goes over TCP 445.)

If relevant, the two servers are running Windows Server 2016.

  • BitTorrent seems like it would check some of those "boxes". But could you explain the insistence on UDP? Is that a perceived or measured bottleneck? Is it a bottleneck at all? With TCP rsync over SSH would be yet another option. Due to those rolling hashes it should be capable of doing this ... – 0xC0000022L Jun 4 '20 at 7:55
  • @0xC0000022L while I agree that bt is a great way to get parallelism, there's no way the RISK team would approve opening the bt firewall ports. I mentioned UDP because of it's lower overhead. – RonJohn Jun 4 '20 at 13:45
  • okay, but my main point is that the lower overhead of UDP will not blankly translate into a performance boost. If these servers are in the same network (the mentioning of firewall makes me doubt that), BT is no issue. If they aren't any UDP-based protocol causes risks when it comes to telling flooding apart from legit high-bandwidth traffic. If you are using a VPN and do that over TCP ... well, there goes your performance boost. So that's why I was asking pointed questions in regard to whether you'd been actually measuring the performance. – 0xC0000022L Jun 4 '20 at 14:58
  • @0xC0000022L is a DC-DC copy. I haven't been measuring the performance of bittorrent or any other UDP-based tool because it's not installed on those servers. – RonJohn Jun 4 '20 at 17:56
  • But you have measured the performance of raw TCP throughput, gauged that against physical bandwidth, and you also measured the TCP throughput when copying via robocopy (presumably CIFS?) and gauged that against what you'd expect as raw throughput? – 0xC0000022L Jun 5 '20 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.