1

In short I wanted to have a local folder be in sync with a folder in a remote server.

The long story:

I have a folder called master where I store a bunch of data. This data is organized in folders that contain the actual text/json files. Assume master contains folders with names folder1, folder2, etc. What happens is that master gets updated with new data contained in folderi for the ith folder. These new folders can be added at any time. I could manually scp (as I've been) and send all the data to my computer. However, its extremely tedious to do so. Ideally I would want to be able to use something like git where the program by itself figures out what actually needs to be updated. The catch with git is that it stores things in the cloud and version controls. I am not (totally) against storing things in the cloud but having version control is not required, however, ideally I only send data to myself (unfortunately, github puts quotas on storing data so I can't use that).

I've been using scp -r user@loc:/loc/master . to send all my data but it seems that it doesn't actually check what data I already has and just goes ahead and resends everything. This is really time consuming because the algorithm should run in order the additions rather than in the history of the contents of master. Does anyone have any software/tool/library etc that can work easily on the terminal to update my mater directory? The idea requirements are:

  1. Updates the main (or master) folder per update/change in the folder rather than in the whole history or all contents.
  2. I don't have to manually specify each folder, just the master folder and it only send whatever needs to be updated (essentially like git but without the version controlling)

Ideally it would do the syncing seamlessly with the remote file.


I have thought of using my institutions dropbox which essentially means that I some storage to space, however, I am not 100% how top set up such an automatic sycing with dropbox only using the terminal.

Not sure if this will be an issue but some of these files/data is generated by scripts. Not sure if dropbox will but if it freaks out in the middle of my files executing because its trying to sync that would be super annoying (or if it decides to declare merge conflicts during execution that would also be really annoying). For these reasons I wasn't sure if it would be a bad idea to use dropbox but it might actually be just fine if it doesn't behave to weirdly. If it seamlessly syncs thats cool but if I have to specify only occasionally (only every time there is a change in the contents of master) that it has to update due to a new folder, thats also cool. I'm just worried about weird syncing issue that could arise when the file is actually being written...maybe dropbox can do this I don't know how though.


The system should work for Unix file systems. Essentially I am using locally OS X and remotely some sort of Linux.

2

I recommend lsyncd for this purpose. It monitors your local directory for any new files and then uses rsync to transfer data to a destination. rsync performs an incremental backup, what means that it only transfer files that have changed between source and destination. It can work in daemon mode in the background. I use it on a daily basis to synchronize a local directory with with remote server like this and it works great:

$ lsyncd -rsync /media/data/music /mnt/r/music

I only use it on Linux but it also supports MacOS.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.