I have a large amount of files that should never change, including RAW photographs and video files. I'm worried about silent bit rot.

I do have backups to restore lost/corrupted files, but comparing current files against backups is not practical (for example, video files are on digital tapes). Also, my backup software does not provide functionality for this.

Is there a software that scans list of folders, stores reliable checksums and can validate that selection for added/removed/modified (corrupted) files?

There's about 3TB and 21 million files (a large portion of that is really small files, obviously), so memory consumption is important. It should run on Linux, and preferably on OS X too.

Note: on Linux, I'm already running ZFS, which has robust checksums, and scrub for detecting bitflips. However, it's not possible or practical to use either that or btrfs on OS X / optical disks / USB disks that should be portable (i.e FAT). I highly prefer filesystem agnostic solution.

  • 2
    If there isn't any software that does this automatically, since you're on OSX and Linux, you could pretty easily set up a bash script and a cron job to do this with md5sum to check all your files, build a report, etc.
    – dotVezz
    Commented Feb 5, 2014 at 1:28
  • I suggest that you keep an eye on the next-generation file system btrfs: plus.google.com/+AaronSeigo/posts/ZAmXwESunL2
    – Gallaecio
    Commented Feb 5, 2014 at 22:48
  • On a different alignment from the question as stated: you might be interested in testing git-annex. Its "archivist" use case covers both detection of corruptions and repair from other copies. Yes, you can ask it to semi-automatically copy data between storage areas to ensure enough copies are available. Default setup is a little "invasive" as all files are replaced with symlinks to dedicated read-only directories to prevent unintended writes (can be disabled). Primarily a Linux tool, it looks like it is available as app bundle and homebrew. Commented Sep 23, 2019 at 16:48

4 Answers 4


I started using AIDE:

AIDE (Advanced Intrusion Detection Environment) is a file and directory integrity checker.

It tracks changed, added and modified files, and file attributes. Supports variety of checksum algorithms, including sha256/sha512.

On Ubuntu, aide package is available from base repository (apt-get install aide). On OS X, compiling failed with mysterious errors, but installation with macports succeeded:

sudo port install aide

Example configuration file is available at /opt/local/etc/aide.conf. Running is simple:

aide --init   # Initializes the database - calculates checksums
aide --check  # Checks files against the database
aide --update # Checks files against the database, and updates the database

All data is stored in plaintext file (which is obviously vulnerable to corruption, but keeping a copy is easy), so switching the tool to something else should be straightforward.

Positive things:

  • Fast
  • Supports multiple strong checksum algorithms. Use of md5 is highly discouraged, as it's basically broken.
  • Easy to run on cron
  • Based on short testing, no issues so far. Detected all changes (on content and on configured file attributes) properly, as well as added and removed files.
  • Supports complicated file excludes: for example, there's no point on checksumming temporary files, or any file that should change.
  • Calculates multiple checksums (configurable). This provides relatively good guarantees for future - even if one hashing algorithm is compromised, integrity database is still useful, even against intentional modifications (vs. bit rot).
  • Checksums are stored in plaintext, and headers include field definitions. This is useful if configuration file is lost, or if it is parsed with another program.
  • Easy to store configuration file and checksum database on each disk/CD/folder (structure). With that, all configuration options are automatically stored, and it is easy to run integrity check again.

Negative points:

  • Configuring requires editing the config file on text editor, versus having a nice UI. Similarly, checking output is straight to the terminal.
  • Latest release is from 2010, but on the other hand it is feature complete, so there is no need for constant updates.
  • Checksum database integrity is not automatically validated. Fortunately, doing that separately is easy (sha1sum checksums.db > checksums.db.sha1sum)
  • Works well on Windows?
    – Pacerier
    Commented Mar 31, 2015 at 23:14
  • aide --init (or anything else I tried to do with aide) doesn't work on Ubuntu - the package maintainers broke it. You have to use aideinit instead. Commented May 3, 2016 at 14:43
  • 2
    MD5 has security issues, but why does that matter for the OP's purpose? He's not looking to securely hash sensitive data. In fact MD5 is very commonly used as a file checksum - example. That aside, +1 for a good answer.
    – JBentley
    Commented Jun 5, 2016 at 15:32
  • Another point - text file configuration and CLI output would be a positive for many people rather than a negative.
    – JBentley
    Commented Jan 8, 2020 at 13:13

I've been using cfv for this for years.

  • It supports recursive operations (you can create one checksum file per directory, or one checksum file for a complete subtree).
  • It could ignore case and fix path separator options for cross platform use which is quite useful if you want to check/create checksums on different file systems and/or operating systems.
  • It's a console application but it has a nice progress bar (unlike plain md5sum).
  • It can detect added files (-u option).

I don't know if it runs or Mac OS X or not, but it's in MacPorts.

  • Hmm. sha1sum * > files.sha1sum; cfv works fine. According to man, -r -m -u is proper set of options to check for files with no hashes. That seems to work fine, BUT if file with checksum is modified, it still outputs OK for that. If file with checksum is removed, it properly prints "1 not found, 1 unverified". I was about to open bug ticket, but didn't bother registering to sourceforge.
    – Olli
    Commented Feb 7, 2014 at 9:07
  • @Olli: I don't think you need the -m flag. According to the manual, -m don't compare checksums. Try cfv -T -uu -f test1.sha1 (if you have one sha1 file for a complete directory tree).
    – palacsint
    Commented Feb 7, 2014 at 11:13
  • for some reason, that checks everything twice (and yes, I know comments are not a support forum).
    – Olli
    Commented Feb 7, 2014 at 12:47

No school like oldschool. Admittedly, I have a bit of a bias here from my forensics background but you could run md5 sums and compare them.

While this SO thread has some interesting methods, I'm rather fond of hashdeep. Memory usage is low, though its chewing through a bunch of processor cycles, but through the magic of audit mode, it does everything you ask of it through the command line automatically - by taking in a text file, and comparing it to what it lists.

While the site has windows binaries, quite a few distributions and mac package managers have ports - you can find a full list here

  • hashdeep - including time estimates - sounded nice, but I don't really like the interface: a) on mac terminal, progress bar/estimates is seriously broken, b) for folders, you have to give --recursive. Otherwise in --audit it just traverses recursively and prints Known file not used for each file, c) if file is copied, according to hashdeep it's moved (and old one is reported as "No match"), instead of "added", d) audit and update requires two passes.
    – Olli
    Commented Feb 7, 2014 at 8:59
  • "While this SO thread has some interesting methods […]": Was there supposed to be a link?
    – unor
    Commented Apr 9, 2014 at 23:30
  • I think he meant this SR thread. Commented May 8, 2014 at 8:04
  • I just tried hashdeep and I expected it to report the files that I've added or removed since creating the hash table. It doesn't appear to have options to report them at all. Strange. Commented Apr 28, 2016 at 8:00

Verifying backups and checking for bit rot are the driving motivations which led to my creating Datimprint, which I just released. I had used hashdeep (mentioned in another answer) before, and it worked OK. But I wanted something written in a modern, cross-platform language with the ability to create recursive hashes (i.e. one fingerprint to ensure the entire tree was valid). I also wanted to distinguish between content changes and filename case/timestamp changes.

Datimprint is written in Java so it runs on Linux, Mac, and Windows. It uses the latest file APIs and encoding standards, and is multithreaded for quick file system traversal and checksum processing. For example the Unicode difficulties experienced by hashdeep simply isn't an issue with Datimprint, which uses Unicode with correct encoding from top to bottom.

Datimprint creates a "data imprint" of a directory tree and stores it in datim file.

datimprint generate /var/data --output /var/imprints/data-2022-11-12.datim

You can later check the data tree against the imprint file, to verify a backup or to check for data degradation, for example.

datimprint check /var/backup/data --imprint /var/imprints/data-2022-11-12.datim

The source code is publicly available via its GitHub project.

If Datimprint is helpful to you, please let me know. You can request new features or report bugs by filing issues. You can provide other feedback in discussions about the program.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.