7

I'm looking for software to implement backups from a NAS (which runs a vanilla Linux distribution) to a cloud storage/backup provider. The NAS stores 1-3TB of document & media files. In particular, I need something that will:

  • Run on Linux (more specifically, either directly on Debian Linux or via a Docker image).
  • Store versioned backups (i.e. allows me to go back in time).
  • Store backups encrypted (and only I should be able to access the files; the data storage provider should not be able to decrypt the contents).
  • Efficiently (incrementally) syncs files to an offsite location on at least a daily basis. Should only require a full initial backup, and no periodic full backups after that.
  • Compressed file transfer.
  • I'm flexible on which cloud provider to backup to (e.g. S3, Google Drive, Crashplan, etc.), as long as the recurring cost is fairly low (under $200/yr for 1+ TB of storage).
  • Open source is preferred, but commercial software is OK, too.
  • Must provide a CLI interface (if there's a GUI, it should be optional)

Duplicity comes pretty close to meeting these requirements, except that it requires a full backup periodically to clean out old files and shrink backup sizes and restore times.

Borg Backup looks like it has a lot of potential. Unfortunately, cloud storage is currently not supported out of the box (https://github.com/borgbackup/borg/issues/1070). One could spin up a VM somewhere in the cloud and backup to it via e.g. NFS, but that gets very pricey.

Improve this question
8
  • 1
    OK, then you might wish to take a look at Borg Backup. Didn't yet try it myself, just stumbled upon it last week. Supports encryption, deduplication, compression – and all that e.g. via ssh to a NAS, so it might work with "external cloud providers".
    – Izzy ♦
    Dec 12, 2016 at 13:05
  • Thanks! Borg Backup looks really interesting. Unfortunately, cloud storage is currently not supported out of the box. I could spin up a VM somewhere and backup via e.g. NFS, but that gets very pricey.
    – Alex
    Dec 13, 2016 at 2:58
  • If speed is not an issue, many providers allow you to use WebDAV to access your "shares" – so you could use e.g. davfs2 to mount them. There's also a way to mount Amazon cloud. Cannot be more specific as you didn't mention the provider :)
    – Izzy ♦
    Dec 13, 2016 at 7:26
  • Great point. I'm a bit wary of doing this though, as integrating cloud support into backup tools often requires architectural changes to the software / backup format, which are sidestepped when using this approach. Also, I've tested S3 FUSE filesystems in the past and every one that I tested suffered from either performance problems or reliability problems. That said, I haven't done any testing with other cloud storage providers such as Google/Amazon Drive, that might be worth looking into.
    – Alex
    Dec 14, 2016 at 13:38
  • @Izzy, the word many here is quite overstatement:) Call me at least two. Among big cloud vendors only OneDrive provide WebDAV access, and their access is a total shit! Try to load 150Mb file via their WebDAV and you'll be very frustrated.
    – Suncatcher
    Mar 13, 2017 at 9:49

4 Answers 4

Reset to default
3

Use Borg (or another incremental backup program) alongside a cloud access tool

Consider using a combination of utilities that each do a part of what you want, instead of a single giant program that would do all of this. You could essentially get away with 2 programs: the "cloud access layer" and Borg (if this is what you prefer). I think the following setup would (mostly) do what you need:

Cloud storage service recommendation

Try Amazon Cloud Drive. It's quite cheap (I think about $55/year) and it works all right for me. They have a 3-month trial I believe, and you pay on a yearly basis with unlimited storage (to the extent that people store 10s of terabytes without issues). There are also several other unlimited storage services that are affordable, so please check out /r/DataHoarder on Reddit for further recommendations.

Connecting to the cloud storage from your computer

rclone allows you to use a variety of cloud storage services (including Amazon Cloud Drive) from a single interface. There is another utility with similar features, acd_cli, which only supports Amazon Cloud Drive. Both support managing your storage from the command line (upload, download, list files), and to various degrees as a mounted filesystem.

Accessing your cloud storage as a mounted filesystem

This can be quite neat, since you can browse the contents of your cloud drive as it were a locally mounted filesystem. Both rclone and acd_cli support FUSE-based mounting of the cloud drive in a directory on your computer.

  • Read-only access works quite well, although acd_cli is a bit faster since it minimises round-tripping to the cloud drive service, something rclone can't do at the moment.
  • Read-write support exists, but is somewhat unreliable due to the nature of cloud services and their internal implementation.

If in your tests Borg (or whichever incremental backup solution you choose) works well with FUSE mounts set to read-write, that's great. If you have any issues with that, maybe your backup tool can export its incremental backup as an archive file, which you can then upload via the rclone or acd_cli command line.

Encryption

You can pick any number of the below, but one layer of strong encryption is sufficient. I recommend not using more than one layer, as it adds additional complexity in case something breaks and/or if you need to recover data (I'm speaking from experience, even one layer is painful enough to handle there).

  • rclone supports adding an additional file-level encryption layer on top of any cloud storage, with or without filename encryption. This may be the simplest option, and it's what I have picked personally for now.

  • acd_cli may have (or had) something like this, but I couldn't find any conclusive information online. Some acd_cli tutorials online refer to using it together with EncFS, which a recent security audit found to be insecure.

  • BorgBackup seems to have some sort of encryption built-in, but I cannot comment on whether it's strong or not as I haven't used it.

  • File-based encryption layer is another option. If you choose to set up your cloud drive as a FUSE mount, using eCryptFS on top of that should be OK. There is a different piece of software called CryFS, but I haven't looked into it in detail. Do NOT use encfs, as I mentioned before, it was found to be weak.

Compression

  • BorgBackup appears to have an option to use compression, so that would appear to be the easiest option if you decide to go with that.
  • If you go with a different application which supports compression, try to make sure it has 'xz' or LZMA2 compression, as these have a really good compression ratio (although they may be too much for really weak machines).
  • If you go with a different application that does not support compression, I recommend compressing its backup snapshot files into a .tar.xz or .7z archives before uploading them to the cloud.

I apologise if this doesn't meet your needs, but please let me know via comments if you have any questions (also thanks for pointing me at Borg, I may try it out).

Improve this answer
2

Have a look at duplicati.

It does fit all of your conditions if I'm not mistaken :

  • It works with standard protocols like FTP, SSH, WebDAV as well as popular services like Microsoft OneDrive, Amazon Cloud Drive & S3, Google Drive, box.com, Mega, hubiC and many others.
  • AES encryption is done on your device, then encrypted files are stored
  • plus it's open source (LGPL)
  • (I won't repeat all the features here since they're nicely presented just one click away)

Combine it with a few mega.nz accounts (okay, for terabytes you might need a lot of those) and you have backups for completely free too.

Improve this answer
1
  • Duplicati checks all the right boxes from a features perspective but unfortunately, v2.0 has been in beta for multiple years now and the previous stable version is unsupported so it doesn't appear to be a very safe choice.
    – Yeroc
    Jul 2, 2018 at 16:11
1

I just went through the process of evaluating a number of backup products with the exact same list of requirements. In the end I settled on Duplicacy (source code). It covers all the points you mention and offers the ability to prune older backups to minimize storage costs. If you have multiple machines it can apparently do de-duplication across those machines if they're backing up to the same storage repository. Be aware that although it's open source it's not a standard open source license and requires a license for commercial use.

Improve this answer
0

I ended up settling on restic, backing up to Backblaze B2 at a cost of $5/TB/mo. I've been using it for a few months and it's been robust. It meets the requirements above, except that it doesn't support compression (though this is in discussion and a part of their v2 roadmap). It supports deduplication across hosts, and overall I like their design, thoughtfulness, and pace of development.

Improve this answer
1
  • I considered restic as well but because I'm backing up mostly text files (mail server) the lack of compression was a show-stopper for me.
    – Yeroc
    Jul 5, 2018 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.