It will be a long story :)
For the last decade the engineering team (us) have maintained homebrew tool for deployment engineering team written in C# to transfer files across datacenters and recently, cloud. It is quite flexible and capable of transferring large amounts of data. however, this is obsolete code with low maintainability, in addition writing such a tool is not SE team expertise and main business. I've spent couple of days lookin for such a tool, to no avail. None doing all the stuff we need and since it is very obvious tasks I think I just overlooked it or looked in the wrong place.
So, what I was looking for?
The tool should be capable of copying files from multiple sources to one destination and vice versa, copy (or move) files from one source to multiple destinations. Sometimes (usually when sending over network outside the datacenter) it should be compressed prior to be sent, the tool on the receiving end should be capable of uncompressing incoming files and dispatching it further to single/multiple destinations.
In addition it should be capable of uploading/downloading files to/from FTP (compressing before upload, uncompressing after download). It should be configurable to support back-up of files to be sent, for example, compress files, upload to FTP, upload uncompressed files to cloud (S3 for example) for backup, delete local source files.
As for the robustness, it should be capable to detect partially uploaded/copied files (test compressed file integrity for example). In addition, when running CPU intense operations (like compressing) it should be capable to employ multiple readers/writers otherwise the bandwidth is not fully utilized. And the last but not least, multiple instances of such a tool running on several machines should be able to pull files from common shared folder (yep, rocket science)
Typical use cases:
Thingy instance01 - Copy files from multiple folders on server to multiple folders on NAS, replicating the folder structure.
Thingy instance02 – compress files copied previously to NAS, upload to FTP, upload to cloud, make backup copies to another NAS storage, delete source.
Thingy instance03 – Download from FTP, uncompress, copy files to shared folder.
Thingy instance04..10 – pull files from shared folder to another machine where file parser waits for incoming stuff.
Thingy instance11 – gather files processed in previous step, compress and copy to DWH machine, upload to Hadoop, copy to backup location
Now the question, since all this “byte transfer” back and forth is not something unusual to happen in a big internet company, is there COTS tool that can perform all the above?
It will be a long story :)
Hey I think I have a solution for you - Cloud FastPath, it connects to all of those systems and has proprietary compression and WAN optimization. It has really great reporting for partially transferred and failed files, as well as a ton of automation.
*edit - I work for Cloud FastPath