1

I'm looking for a utility that ensures that a file's data integrity is maintained, ideally in a similar way that the ZFS filesystem handles data integrity:

Keep two or more copies of the file, checksum every 4K block, and then fix bit corruption over time when it's detected (when the block-level checksum of the file being read differs from the archived checksum for the block, go to the other files to see if their blocks match and use the first one that has the correct checksum, fix the wrong one while we're at it).

Traditionally, you can perform a checksum of the entire file and just detect if the new checksum doesn't match. But block-level checksums would allow for both copies of the data to be corrupt, and so long as the corruption isn't to the same block in each file, the original data can be recovered without error. That is a big advantage.

The OS can be either Windows 10 or Linux.

  • 1
    This is typically not done on file level on one disk, because if the disk breaks, you can't recover anything, no matter how good your integrity data is. Google for RAID (redundant array of inexpensive/idenpendent disks). – Thomas Weller Sep 15 '16 at 6:23
  • No, RAID doesn't do what I suggested. It is good for when an entire disk fails, but not for data integrity. If a bit flips from 0 to 1 on one of your RAID disks, to the RAID controller, everything is fine. – Steve Weigand Sep 15 '16 at 14:38
2

Have you tried parchive (short for parity archive) checksum files (https://en.wikipedia.org/wiki/Parchive). There are several implementations of generators around for different OSs.

In this model the files under consideration are cut into blocks and parity information is stored in separate files with choosable redundancy.

Let's check your points:

  1. Keep two or more copies of the file: Partly: not exact copy, but parity files
  2. checksum every 4K block: Check! Block size freely selectable
  3. and then fix bit corruption over time when it's detected: Check! Some implementations integrate a file watcher, some have to be started manually and the repair damaged files.
  4. allow for both copies of the data to be corrupt, and so long as the corruption isn't to the same block in each file: Check! Original as well as parity files can be damaged as long as the curruptions are unrelated blockwise
  5. the original data can be recovered without error: Check! Depending on selected redundancy even some completely missing files can be recovered in the extreme case

BTW: Some file archivers as RAR also include error correcting redundancy codes in their archive format

  • Thanks so much for that. I had completely forgotten about par/par2 archives. I remember it from back in the good old usenet days, where it performed admirably. It is exactly what I was looking for. I do wonder if there are other utilities that provide similar levels of data integrity protection. If anyone out there is reading this comment and knows of something else, please don't hesitate to post it even though I've selected this as the "best" answer (it's presently the only answer). Thanks! – Steve Weigand Sep 16 '16 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.