40

I have a large amount of files that should never change, including RAW photographs and video files. I'm worried about silent bit rot.

I do have backups to restore lost/corrupted files, but comparing current files against backups is not practical (for example, video files are on digital tapes). Also, my backup software does not provide functionality for this.

Is there a software that scans list of folders, stores reliable checksums and can validate that selection for added/removed/modified (corrupted) files?

There's about 3TB and 21 million files (a large portion of that is really small files, obviously), so memory consumption is important. It should run on Linux, and preferably on OS X too.

Note: on Linux, I'm already running ZFS, which has robust checksums, and scrub for detecting bitflips. However, it's not possible or practical to use either that or btrfs on OS X / optical disks / USB disks that should be portable (i.e FAT). I highly prefer filesystem agnostic solution.

  • 2
    If there isn't any software that does this automatically, since you're on OSX and Linux, you could pretty easily set up a bash script and a cron job to do this with md5sum to check all your files, build a report, etc. – dotVezz Feb 5 '14 at 1:28
  • I suggest that you keep an eye on the next-generation file system btrfs: plus.google.com/+AaronSeigo/posts/ZAmXwESunL2 – Gallaecio Feb 5 '14 at 22:48
19

I started using AIDE:

AIDE (Advanced Intrusion Detection Environment) is a file and directory integrity checker.

It tracks changed, added and modified files, and file attributes. Supports variety of checksum algorithms, including sha256/sha512.

On Ubuntu, aide package is available from base repository (apt-get install aide). On OS X, compiling failed with mysterious errors, but installation with macports succeeded:

sudo port install aide

Example configuration file is available at /opt/local/etc/aide.conf. Running is simple:

aide --init   # Initializes the database - calculates checksums
aide --check  # Checks files against the database
aide --update # Checks files against the database, and updates the database

All data is stored in plaintext file (which is obviously vulnerable to corruption, but keeping a copy is easy), so switching the tool to something else should be straightforward.


Positive things:

  • Fast
  • Supports multiple strong checksum algorithms. Use of md5 is highly discouraged, as it's basically broken.
  • Easy to run on cron
  • Based on short testing, no issues so far. Detected all changes (on content and on configured file attributes) properly, as well as added and removed files.
  • Supports complicated file excludes: for example, there's no point on checksumming temporary files, or any file that should change.
  • Calculates multiple checksums (configurable). This provides relatively good guarantees for future - even if one hashing algorithm is compromised, integrity database is still useful, even against intentional modifications (vs. bit rot).
  • Checksums are stored in plaintext, and headers include field definitions. This is useful if configuration file is lost, or if it is parsed with another program.
  • Easy to store configuration file and checksum database on each disk/CD/folder (structure). With that, all configuration options are automatically stored, and it is easy to run integrity check again.

Negative points:

  • Configuring requires editing the config file on text editor, versus having a nice UI. Similarly, checking output is straight to the terminal.
  • Latest release is from 2010, but on the other hand it is feature complete, so there is no need for constant updates.
  • Checksum database integrity is not automatically validated. Fortunately, doing that separately is easy (sha1sum checksums.db > checksums.db.sha1sum)
  • Works well on Windows? – Pacerier Mar 31 '15 at 23:14
  • aide --init (or anything else I tried to do with aide) doesn't work on Ubuntu - the package maintainers broke it. You have to use aideinit instead. – reinierpost May 3 '16 at 14:43
  • 2
    MD5 has security issues, but why does that matter for the OP's purpose? He's not looking to securely hash sensitive data. In fact MD5 is very commonly used as a file checksum - example. That aside, +1 for a good answer. – JBentley Jun 5 '16 at 15:32
11

I've been using cfv for this for years.

  • It supports recursive operations (you can create one checksum file per directory, or one checksum file for a complete subtree).
  • It could ignore case and fix path separator options for cross platform use which is quite useful if you want to check/create checksums on different file systems and/or operating systems.
  • It's a console application but it has a nice progress bar (unlike plain md5sum).
  • It can detect added files (-u option).

I don't know if it runs or Mac OS X or not, but it's in MacPorts.

  • Hmm. sha1sum * > files.sha1sum; cfv works fine. According to man, -r -m -u is proper set of options to check for files with no hashes. That seems to work fine, BUT if file with checksum is modified, it still outputs OK for that. If file with checksum is removed, it properly prints "1 not found, 1 unverified". I was about to open bug ticket, but didn't bother registering to sourceforge. – Olli Feb 7 '14 at 9:07
  • @Olli: I don't think you need the -m flag. According to the manual, -m don't compare checksums. Try cfv -T -uu -f test1.sha1 (if you have one sha1 file for a complete directory tree). – palacsint Feb 7 '14 at 11:13
  • for some reason, that checks everything twice (and yes, I know comments are not a support forum). – Olli Feb 7 '14 at 12:47
7

No school like oldschool. Admittedly, I have a bit of a bias here from my forensics background but you could run md5 sums and compare them.

While this SO thread has some interesting methods, I'm rather fond of hashdeep. Memory usage is low, though its chewing through a bunch of processor cycles, but through the magic of audit mode, it does everything you ask of it through the command line automatically - by taking in a text file, and comparing it to what it lists.

While the site has windows binaries, quite a few distributions and mac package managers have ports - you can find a full list here

  • hashdeep - including time estimates - sounded nice, but I don't really like the interface: a) on mac terminal, progress bar/estimates is seriously broken, b) for folders, you have to give --recursive. Otherwise in --audit it just traverses recursively and prints Known file not used for each file, c) if file is copied, according to hashdeep it's moved (and old one is reported as "No match"), instead of "added", d) audit and update requires two passes. – Olli Feb 7 '14 at 8:59
  • "While this SO thread has some interesting methods […]": Was there supposed to be a link? – unor Apr 9 '14 at 23:30
  • I think he meant this SR thread. – Andrew Cheong May 8 '14 at 8:04
  • I just tried hashdeep and I expected it to report the files that I've added or removed since creating the hash table. It doesn't appear to have options to report them at all. Strange. – reinierpost Apr 28 '16 at 8:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.