2

I need to do a daily backup of a set of large files (>3GB each) that have a, maybe, 20 ΜΒ difference each day, mostly overwritten parts.

The problem is both the time needed to complete the daily backup but also bandwidth usage:

Our landline Internet is bad and goes off at times, an LTE backup kicks in but there's a monthly quota that gets easily eaten up with one or two backups leaving us severely throttled during work hours, if the landline goes off again.

I need a solution that can be run in the CLI/batch files and makes a differential backup, on Windows, without requiring cygwin. If I didn't have the cygwin limitation, rsync would be a prime candidate.

  • You should look into Robocopy as well – Dave Aug 1 '17 at 9:46
  • 1
    Hi Dave, Robocopy is our current solution but doesn't do differential copies. Sure it's MIR option will copy anything changed etc but if a 3GB file has a 1 byte changed it will still copy all 3 GBs. – kntouskos Aug 2 '17 at 11:56
2

I use xcopy (built in Windows command) with the following options to achieve a similar thing:

xcopy /H /D /E /Y <source> <destination>

/H - Copy system and hidden files.
/D - Only copy files that are newer on the source than the destination.
/E - Copy all directories and subdirectories even if empty.
/Y - Suppress overwrite confirmations.

It has loads of other options too that are worth looking at.

This will still only copy entire files rather than partial ones, but only the ones that have been modified. There are Windows versions of rysnc out there, (some even with GUIs) that don't require Cygwin, but I don't have experience with them.

  • 1
    Thanks for the input. The problem is all files, 9 of them, change each day with small bits added or edited. So that's about 30GB of data each day that I want to cut down on. I'd be really interested in a CLI version of rsync for windows that doesn't require cygwin. Do you happen to remember a name? – kntouskos Aug 2 '17 at 11:50
  • @kntouskos DeltaCopy is one, but I’m not sure if it’s got a CLI. Found this just by searching for “windows rsync CLI” - acrosync.com – Darren Aug 2 '17 at 12:03
0

I had similar problem, but I needed to incrementally backup 50 GB files, so I guess you could use this.

1. Taking snapshot to preserve consistency

This is completely optional and depend on your situation. If the files you want to backup are in use, you want to make point-in-time snapshot of them and run backup on the snapshot. Otherwise the files may change during backup.

I already have some easy to follow instructions here.

2. Backing up

I've got three options here for you:

  • Unison: mainly a synchronization tool, but also rock solid alternative for rsync on Windows (and other main platforms). It implements rsync protocol internally and has no dependencies.
  • zpaq: offline incremental archiver, deduplicator. It creates small diffs, which you can/have to transfer manually. Effective and finishes in constant time.
  • Restic: modern alternative to zpaq. Adds encryption. Not as space efficient as zpaq but twice as fast.
  • xdelta3: binary diff tool. Very effective, creates diffs as large as the actually changed data. Usually it takes about the same time as zpaq, but occasionally its runtime explodes to 3-5x of that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.