Differential copy solution for Windows

Question

I need to do a daily backup of a set of large files (>3GB each) that have a, maybe, 20 ΜΒ difference each day, mostly overwritten parts.

The problem is both the time needed to complete the daily backup but also bandwidth usage:

Our landline Internet is bad and goes off at times, an LTE backup kicks in but there's a monthly quota that gets easily eaten up with one or two backups leaving us severely throttled during work hours, if the landline goes off again.

I need a solution that can be run in the CLI/batch files and makes a differential backup, on Windows, without requiring cygwin. If I didn't have the cygwin limitation, rsync would be a prime candidate.

Answer 1

I use xcopy (built in Windows command) with the following options to achieve a similar thing:

xcopy /H /D /E /Y <source> <destination>

/H - Copy system and hidden files.
/D - Only copy files that are newer on the source than the destination.
/E - Copy all directories and subdirectories even if empty.
/Y - Suppress overwrite confirmations.

It has loads of other options too that are worth looking at.

This will still only copy entire files rather than partial ones, but only the ones that have been modified. There are Windows versions of rysnc out there, (some even with GUIs) that don't require Cygwin, but I don't have experience with them.

Answer 2

I had similar problem, but I needed to incrementally backup 50 GB files, so I guess you could use this.

1. Taking snapshot to preserve consistency

This is completely optional and depend on your situation. If the files you want to backup are in use, you want to make point-in-time snapshot of them and run backup on the snapshot. Otherwise the files may change during backup.

I already have some easy to follow instructions here.

2. Backing up

I've got three options here for you:

• Unison: mainly a synchronization tool, but also rock solid alternative for rsync on Windows (and other main platforms). It implements rsync protocol internally and has no dependencies.
• zpaq: offline incremental archiver, deduplicator. It creates small diffs, which you can/have to transfer manually. Effective and finishes in constant time.
• Restic: modern alternative to zpaq. Adds encryption. Not as space efficient as zpaq but twice as fast.
• xdelta3: binary diff tool. Very effective, creates diffs as large as the actually changed data. Usually it takes about the same time as zpaq, but occasionally its runtime explodes to 3-5x of that.