In short I wanted to have a local folder be in sync with a folder in a remote server.
The long story:
I have a folder called
master where I store a bunch of data. This data is organized in folders that contain the actual text/json files. Assume
master contains folders with names
folder2, etc. What happens is that
master gets updated with new data contained in
folderi for the
ith folder. These new folders can be added at any time. I could manually
scp (as I've been) and send all the data to my computer. However, its extremely tedious to do so. Ideally I would want to be able to use something like
git where the program by itself figures out what actually needs to be updated. The catch with git is that it stores things in the cloud and version controls. I am not (totally) against storing things in the cloud but having version control is not required, however, ideally I only send data to myself (unfortunately, github puts quotas on storing data so I can't use that).
I've been using
scp -r user@loc:/loc/master . to send all my data but it seems that it doesn't actually check what data I already has and just goes ahead and resends everything. This is really time consuming because the algorithm should run in order the additions rather than in the history of the contents of
master. Does anyone have any software/tool/library etc that can work easily on the terminal to update my
mater directory? The idea requirements are:
- Updates the main (or
master) folder per update/change in the folder rather than in the whole history or all contents.
- I don't have to manually specify each folder, just the
masterfolder and it only send whatever needs to be updated (essentially like git but without the version controlling)
Ideally it would do the syncing seamlessly with the remote file.
I have thought of using my institutions dropbox which essentially means that I some storage to space, however, I am not 100% how top set up such an automatic sycing with dropbox only using the terminal.
Not sure if this will be an issue but some of these files/data is generated by scripts. Not sure if dropbox will but if it freaks out in the middle of my files executing because its trying to sync that would be super annoying (or if it decides to declare merge conflicts during execution that would also be really annoying). For these reasons I wasn't sure if it would be a bad idea to use dropbox but it might actually be just fine if it doesn't behave to weirdly. If it seamlessly syncs thats cool but if I have to specify only occasionally (only every time there is a change in the contents of
master) that it has to update due to a new folder, thats also cool. I'm just worried about weird syncing issue that could arise when the file is actually being written...maybe dropbox can do this I don't know how though.
The system should work for Unix file systems. Essentially I am using locally OS X and remotely some sort of Linux.