12

I am a little concerned about one of my disk drives and afraid that I might be overwriting good backups of it with corrupt files.

I would like a tool which recursively scans all directories and tests the integrity of each individual file.

Unfortunately, that means a different check for each file type, so such a tool may either not exist or may be limited, but I would like to know what is out there.

In rough order of what I want to verify:

  • photos
  • videos
  • eBooks
  • archives
  • MS office documents
  • mp3 / flac / ogg
  • anything else is a bonus

A huge bonus would be if the tool were to attempt to repair corrupted files, but that is probably too much to ask.


[Update] many posters don't seem to read or understand the question. I am not concerned with MD5 or checksum; I am talking about files which have a known internal structure, where it can be detected if that internal structure is not consistent/corrupt.

And, as a bonus, which can be repaired if that internal structure is (slightly) corrupted - for instance, rescue the photo from a JPEG if EXIF info or thumbnail is corrupt, rescue music from MP if the IDtags are corrupt; there are programs available which will repair corrupt ZIP files, MS Word & Excel etc

3
  • Make sure you have a working backup before running such a tool. Anything that goes around and touches every file will strain your hard drive, and if it's already in a bad shape, then the software could do more harm than good
    – Tymric
    Jul 26, 2015 at 22:17
  • This program also assumes that the current state of the data is valid, but it offers a repair feature. quickpar.org.uk If you PAR2 your data files the program will create data recovery files. The size/percentage of the PAR2 files determines the amount of damage can be recovered. So if you get bad sectors you can fix those and PAR2 will rebuild the missing data. Unfortunately, they have to be updated every time you legitimately update the files.
    – cybernard
    Aug 7, 2015 at 4:17
  • 1

5 Answers 5

2

The File Information Tool Set (FITS) bundles an array of tools for identifying and validating files, as well as extracting technical metadata. Basically, it generates an XML file, with the outputs from the different Tools.

FITS works in different stages as shown in the image below.

enter image description here

To use it recursively, you'd have to write the command the following way:

./fits.sh -i /input-files-directory -o /output-directory -r -n
  • -r causes FITS to recursively process all files when the input is a directory.
  • -n outputs the corresponding XML file report in the same nested way as the files are nested in the input directories.

There are more options available, which are documented here.

1
  • FITS requires Java JRE.
    – AcK
    Feb 4, 2023 at 12:29
1

There is a ready-made solution. Use Veretino. A cross-platform open source app for calculating and verifying a large number of checksums, and generally verifying the integrity of a large amount of data. It can also find/test damaged files (including certain types) in a folder.

0

Exact file looks like a program that will do what you need to do. You have to set it up to scan a folder but you could just set it up to scan the folder c or whatever drive you are scanning.

Exact File
Features:

  • A file integrity verification tool:
    • Use it to make sure files copied to CD-ROM are bit-perfect copies,
    • Use it to make sure backups copied from one drive to another are just right,
    • Use it to make sure files haven’t been changed or damaged over time.
  • Multi-threaded, so your extra CPU cores get used when scanning multiple files and work gets done faster.
  • Happy with Unicode file names, so it doesn’t fail when you’re using it on files named in Japanese, Hebrew, Chinese, or any other language.
  • Supports multiple checksum routines (hashes), like MD5, SHA1, CRC32, RIPEMD and others.
  • Supports recursive directory scanning.
  • Supports Very Big Files — If it’s on your hard drive, ExactFile can handle it.
  • Does everything popular file summer utilities do, like fsum, md5sum, sha1sum, sfv, etc, but better!
  • Compatible with popular file checksum digest formats.
  • For Windows 2000, XP, Vista, and 7.
  • GUI. Easy to use to get checksums for individual files, create checksum digests, and test checksum digests. Does not require the console version or any external DLLs.
  • FREE.

Exact File User Interface

4
  • And I am supposed to google for it? With a unique name like that?
    – Mawg
    Jul 26, 2015 at 15:36
  • 1
    "looks like" a good program to do this - have you even tried it? Please read How to Answer
    – Mawg
    Jul 26, 2015 at 15:37
  • 4
    I googled "Exact file" and found it, but it looks like a checksum,sha, etc based program. It hash each file once and then it compares the hashes later. However, if the files are damaged before you run the program the first time it will not detected that.
    – cybernard
    Jul 26, 2015 at 17:04
  • Aye, there’s the rub. As such, I am afraid that it is of no use to me :-( I suppose that I could code my own ...
    – Mawg
    Jul 27, 2015 at 7:14
0

You can use Total Commander to create a checksum file.
This works very well for entire directories or single files.

Download the software from www.ghisler.com (for Windows) or https://www.codeweavers.com/via/totalcommander8 (for Mac)

1) To scan and take a snapshot, go to Menu => Files => Create Checksum Files.
2) To verify, go to Menu => Files => Verify Checksums (from Checksum Files)

3
  • This does not meet the requirement I would like a tool which recursively scans all directories...
    – user416
    Aug 26, 2015 at 6:44
  • That's not the point though ()not my downvote, btw). So many posters don't seem to read or understand the question. I am not concerned with MD5 or checksum; i am talking about files which have a known internal structure and which can be repaired if that internal structure is (Slightly) corrupted - for instance, rescue the photo from a JPEG if EXIF info or thumbnail is corrupt, rescue music from MP if the IDtags are corrupt; there are programs available which will repair corrupt ZIP files, MS Word & Excel etc
    – Mawg
    Aug 26, 2015 at 8:34
  • 1
    Apart from that, a checksum doesn't even match the OP's first criterium: "I am a little concerned about one of my disk drives and afraid that I might be overwriting good backups of it with corrupt files." A checksum here would indicate a changed file (which might need to be backed-up) – but not tell whether that means the file is corrupt. It might even be an indicator that the file needs to be backed-up (as it has changed). So this approach doesn't fit at all, sorry.
    – Izzy
    Aug 26, 2015 at 9:25
0

I use Bvckup2 to mirror drives. The author is also concerned about bit-rot and currently has a small mailing list to discuss prevention, with the view to producing a tool.

If anyone interested contacts him, he will (I am sure) be happy to add you to the discussion.

This is the closest to an answer that I have been able to find

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.