It will be a long story :)
For the last decade the engineering team (us) have maintained homebrew tool for deployment engineering team written in C# to transfer files across datacenters and recently, cloud. It is quite flexible and capable of transferring large amounts of data. however, this is obsolete code with low maintainability, in addition writing such a tool is not SE team expertise and main business. I've spent couple of days lookin for such a tool, to no avail. None doing all the stuff we need and since it is very obvious tasks I think I just overlooked it or looked in the wrong place.
So, what I was looking for?
The tool should be capable of copying files from multiple sources to one destination and vice versa, copy (or move) files from one source to multiple destinations. Sometimes (usually when sending over network outside the datacenter) it should be compressed prior to be sent, the tool on the receiving end should be capable of uncompressing incoming files and dispatching it further to single/multiple destinations.
In addition it should be capable of uploading/downloading files to/from FTP (compressing before upload, uncompressing after download). It should be configurable to support back-up of files to be sent, for example, compress files, upload to FTP, upload uncompressed files to cloud (S3 for example) for backup, delete local source files.
As for the robustness, it should be capable to detect partially uploaded/copied files (test compressed file integrity for example). In addition, when running CPU intense operations (like compressing) it should be capable to employ multiple readers/writers otherwise the bandwidth is not fully utilized. And the last but not least, multiple instances of such a tool running on several machines should be able to pull files from common shared folder (yep, rocket science)

Typical use cases:
Thingy instance01 - Copy files from multiple folders on server to multiple folders on NAS, replicating the folder structure.
Thingy instance02 – compress files copied previously to NAS, upload to FTP, upload to cloud, make backup copies to another NAS storage, delete source.
Thingy instance03 – Download from FTP, uncompress, copy files to shared folder.
Thingy instance04..10 – pull files from shared folder to another machine where file parser waits for incoming stuff.
Thingy instance11 – gather files processed in previous step, compress and copy to DWH machine, upload to Hadoop, copy to backup location

Now the question, since all this “byte transfer” back and forth is not something unusual to happen in a big internet company, is there COTS tool that can perform all the above?

  • Do you need compression with any FTP server, even those that don't implement Mode Z? – Nicolas Raoul Feb 25 '16 at 6:19
  • What do you mean by "common shared folder"? CIFS? – Nicolas Raoul Feb 25 '16 at 6:20
  • Yes, since we pack multiple files in one compressed archive to achieve better compression ratios – kreuzerkrieg Feb 25 '16 at 6:26
  • @NicolasRaoul, common shared folder is a folder on a physical medium, accessible using UNC path for many instances of the "file transferer", so it should take care of the case when two or more instances are trying to pull the same file. – kreuzerkrieg Feb 25 '16 at 6:30
  • If the non-Mode-Z FTP server is only accessible via FTP, you won't be able to uncompress after uploading. Is this server also available via SSH for instance, or can you install a sort of agent on it? – Nicolas Raoul Feb 25 '16 at 8:00

Hey I think I have a solution for you - Cloud FastPath, it connects to all of those systems and has proprietary compression and WAN optimization. It has really great reporting for partially transferred and failed files, as well as a ton of automation.

*edit - I work for Cloud FastPath

| improve this answer | |
  • Wow sounds like THE thing we need. Will check it later. Allow couple of days before I mark your reply as answer – kreuzerkrieg Feb 25 '16 at 17:30
  • it is not cheap, I have to admit :) hundreds of TB per month will cost us more than recruiting dedicated developer for our homemade solution – kreuzerkrieg Feb 28 '16 at 10:08
  • Accepting as an answer despite SaaS is not what we are looking for. – kreuzerkrieg Mar 14 '16 at 5:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.