4

I've got quite a few machines with quite a few directories that I want to keep in sync. For example, I have /var/backups on some servers that I want to take a copy of on a nightly basis. rsync is a tool I've been used so far, and initially sounds like a natural choice, because I need the following;

  • Synchronization of directories over WAN connections (eg, home server to sync with public web server)
  • Easily available in repositories for most Linux distributions (big bonus, makes updating easy)
  • Incremental transfers
  • Works without any additional ports (just SSH/22) open

I'm running rsync as a standard cronjob at the moment. However, these are the limitations I'm trying to address with a recommendation;

  • If /var/backups (or wherever I'm copying from) is empty, or if a file cannot be copied due to permissions, I get no warning or notification.
  • For large directory sync's (eg, 3-4 Tb), it's difficult to see the status of that sync at a glance.
4
  • 1
    Have you tried the --progress option to show progress, maybe also couple it with --verbose and --stats to print out more details?
    – Tymric
    Oct 30 '14 at 21:01
  • I've just added a suitable tag to your question. While waiting for your dedicated answers, you might wish to browse the 33 other questions tagged file-synchronization, there might be already a solution for you.
    – Izzy
    Oct 30 '14 at 22:46
  • you may consider using a distributed file system with a high number of replicas. It is not exactly what you asked for, but may serve the purpose rather well.
    – Martin
    Oct 30 '14 at 23:34
  • Sorry, I'm in the office & can't get past the firewall to post details. But you can now use BitTorrent to efficiently synch almost any kid of device. See getsync.com Jan 30 '15 at 14:49
3

You may find ownCloud useful

Synchronization of directories over WAN connections (eg, home server to sync with public web server)

Check (Main purpose)

Easily available in repositories for most Linux distributions (big bonus, makes updating easy)

Check

Incremental transfers

In Progress?

Works without any additional ports (just SSH/22) open

You can work around this with SSH if you need

If /var/backups (or wherever I'm copying from) is empty, or if a file cannot be copied due to permissions, I get no warning or notification.

Check (Activity Feed)

For large directory sync's (eg, 3-4 Tb), it's difficult to see the status of that sync at a glance.

Check?

2

Have you considered Gluster (www.gluster.org) ?

Now, wait a sec. When you look at this thing, if you're like me, it's going to strike you as a really huge thing -- like you're driving to the corner store in your aircraft carrier or something. But in usage it strikes me as /pretty/ light, despite a few processes running. I found it really alien to set up the first time (took hours-not-minutes) but subsequent configs from scratch were a matter of minutes after the responses and outputs were familiar.

In its /geo/replicated mode, it seems to watch a tree for changes and then push changed files into a queue for ONE-WAY replication (so far; bidirectional georeplication is vapoured periodically and then everyone sobers up) to the remote host; I think it's vanilla rsync, but it all kinda handles that in the background.

It will need a few ports open; I'm sorry. I think you can mitigate that with some port forwarding in a running ssl socket.

Gluster can also be used for live bidirectional synch with X hosts over a LAN, with locks and all that; the geo (rsync) replication doesn't do locking/etc, and is really best used for a remote-backup or 'lukewarm' (mostly-cold) DR solution.

I've advised over the application of gluster to replicate terabytes to a location thousands of kilometers from the source. The customer replicates back by (manually) disabling the A->B replication and re-enabling the B->A georepl profile already configured.

So. Grab a paint-by-numbers, allocate some time, take a shot, take a breath, and give it a go?

0

Unison File Synchronizer

Unison is a cross-platform file synchronization tool that allows you to keep two or more replicas of a directory on two or more host machines or disk drives. Some of it's features are:

  • Synchronizes files between any pair of machines communicating over either a direct socket link or tunneling over an ssh connection
  • Available in Linux repositories (Debian, Ubuntu, Fedora, Arch Linux, etc...)
  • Incremental transfers similar to rsync:

    Transfers of small updates to large files are optimized using a compression protocol similar to rsync

  • Works without additional ports1
For the additional requirements in the question:

  • Displays detailed status through both texual and graphical interfaces
  • Displays warnings in console window or as an advisory pop-up window

1On Linux. Might require additional configuration on Windows

1
  • I considered Unison for my own stuff, I had to say. 90% of the time it runs really well, too. For the 10% where one has to dive in for something, though, we decided to skip it. If the user can handle the care and feeding that's very-occasionally (read: monitor this!) required, then it stands as a great user-space option over vanilla rsync. May 23 '17 at 1:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.