File integrity checker

Question

I have a large amount of files that should never change, including RAW photographs and video files. I'm worried about silent bit rot.

I do have backups to restore lost/corrupted files, but comparing current files against backups is not practical (for example, video files are on digital tapes). Also, my backup software does not provide functionality for this.

Is there a software that scans list of folders, stores reliable checksums and can validate that selection for added/removed/modified (corrupted) files?

There's about 3TB and 21 million files (a large portion of that is really small files, obviously), so memory consumption is important. It should run on Linux, and preferably on OS X too.

Note: on Linux, I'm already running ZFS, which has robust checksums, and scrub for detecting bitflips. However, it's not possible or practical to use either that or btrfs on OS X / optical disks / USB disks that should be portable (i.e FAT). I highly prefer filesystem agnostic solution.

Answer 1

I started using AIDE:

AIDE (Advanced Intrusion Detection Environment) is a file and directory integrity checker.

It tracks changed, added and modified files, and file attributes. Supports variety of checksum algorithms, including sha256/sha512.

On Ubuntu, aide package is available from base repository (apt-get install aide). On OS X, compiling failed with mysterious errors, but installation with macports succeeded:

sudo port install aide

Example configuration file is available at /opt/local/etc/aide.conf. Running is simple:

aide --init   # Initializes the database - calculates checksums
aide --check  # Checks files against the database
aide --update # Checks files against the database, and updates the database

All data is stored in plaintext file (which is obviously vulnerable to corruption, but keeping a copy is easy), so switching the tool to something else should be straightforward.

---

Positive things:

• Fast
• Supports multiple strong checksum algorithms. Use of md5 is highly discouraged, as it's basically broken.
• Easy to run on cron
• Based on short testing, no issues so far. Detected all changes (on content and on configured file attributes) properly, as well as added and removed files.
• Supports complicated file excludes: for example, there's no point on checksumming temporary files, or any file that should change.
• Calculates multiple checksums (configurable). This provides relatively good guarantees for future - even if one hashing algorithm is compromised, integrity database is still useful, even against intentional modifications (vs. bit rot).
• Checksums are stored in plaintext, and headers include field definitions. This is useful if configuration file is lost, or if it is parsed with another program.
• Easy to store configuration file and checksum database on each disk/CD/folder (structure). With that, all configuration options are automatically stored, and it is easy to run integrity check again.

Negative points:

• Configuring requires editing the config file on text editor, versus having a nice UI. Similarly, checking output is straight to the terminal.
• Latest release is from 2010, but on the other hand it is feature complete, so there is no need for constant updates.
• Checksum database integrity is not automatically validated. Fortunately, doing that separately is easy (sha1sum checksums.db > checksums.db.sha1sum)

Answer 2

I've been using cfv for this for years.

• It supports recursive operations (you can create one checksum file per directory, or one checksum file for a complete subtree).
• It could ignore case and fix path separator options for cross platform use which is quite useful if you want to check/create checksums on different file systems and/or operating systems.
• It's a console application but it has a nice progress bar (unlike plain md5sum).
• It can detect added files (-u option).

I don't know if it runs or Mac OS X or not, but it's in MacPorts.

Answer 3

No school like oldschool. Admittedly, I have a bit of a bias here from my forensics background but you could run md5 sums and compare them.

While this SO thread has some interesting methods, I'm rather fond of hashdeep. Memory usage is low, though its chewing through a bunch of processor cycles, but through the magic of audit mode, it does everything you ask of it through the command line automatically - by taking in a text file, and comparing it to what it lists.

While the site has windows binaries, quite a few distributions and mac package managers have ports - you can find a full list here

Answer 4

Verifying backups and checking for bit rot are the driving motivations which led to my creating Datimprint, which I just released. I had used hashdeep (mentioned in another answer) before, and it worked OK. But I wanted something written in a modern, cross-platform language with the ability to create recursive hashes (i.e. one fingerprint to ensure the entire tree was valid). I also wanted to distinguish between content changes and filename case/timestamp changes.

Datimprint is written in Java so it runs on Linux, Mac, and Windows. It uses the latest file APIs and encoding standards, and is multithreaded for quick file system traversal and checksum processing. For example the Unicode difficulties experienced by hashdeep simply isn't an issue with Datimprint, which uses Unicode with correct encoding from top to bottom.

Datimprint creates a "data imprint" of a directory tree and stores it in datim file.

datimprint generate /var/data --output /var/imprints/data-2022-11-12.datim

You can later check the data tree against the imprint file, to verify a backup or to check for data degradation, for example.

datimprint check /var/backup/data --imprint /var/imprints/data-2022-11-12.datim

The source code is publicly available via its GitHub project.

If Datimprint is helpful to you, please let me know. You can request new features or report bugs by filing issues. You can provide other feedback in discussions about the program.