1

I have millions of files on a filesystem. Most of them are of mainstream file formats (JPEG, DOCX, ZIP, MP3 etc.). Some of them are corrupted, i.e. some blocks of the file have been overwritten by random data. I need a tool which lists, reads and analyzes all files recursively, and gives me a list of the corrupted files. For each file it should: autodetect the file format; if the file contains compressed parts, decompress those parts in memory, and report failures; if the file contains checksums, verify the checksums, and report failures; scan the entire file for structural errors (e.g. the byte following a JPEG segment is not 0xff), and report them. It doesn't have to understand each file format to the tiniest detail, but it should understand a few dozen mainstream file formats, and be able to detect the most blatant corruptions (e.g. the file is truncated; or a 4 KiB block was overwritten with random data, causing a checksum mismatch).

The operating system can be Windows, macOS or Linux, whichever the tool works on.

What I don't need:

  • A tool which can only autodetect the file format by looking at the first few bytes of the file, and comparing it against known file format signatures. That's because this tool can detect corruption only near the begininning of the file. I've already run such tools, and I already have a list a files with undetected formats.

  • A tool which has gigabytes of dependencies, or it requires installing specific software for opening files (e.g. installing Microsoft Word just to determine whether a DOCX file is corrupt).

  • A tool which does a physical I/O scan of the HDD/SSD/DVD. My storage medium works fine, there are no I/O errors.

  • A tool which does a filesystem check and/or repair. The filesystem is fine, there are no errors.

  • A tool which rereads files on a checksummed filesystem (e.g. ZFS or Btrfs) or checksummed RAID volume. I don't have such a filesystem.

  • A tool which finds deleted files and undeletes them. None of the files I'm interested in are deleted.

  • A tool which computes whole-file checksums (e.g. CRC-32, MD5, SHA-256) and compares files against known checksums. I don't have known-good checksums for these files.

  • A tool which finds files with the wrong extension (i.e. extension not corresponding to the file format). I've already run such a tool, and all the files have the right extension.

  • A tool which repairs corrupted files without asking first. I don't want to make changes to any of the files (yet). Maybe later I use some file repair software.

  • A tool which isn't able to give me a list of corrupted files, which I can copy-paste to somewhere else. I need a full list which I can study, and make decisions about repairs later.

The tutorials I was able to find with a Google search are full of advice for something related, but which ultimately I don't need (see the list above).

Preferably, I'd like to get a tool recommended by a user who have used the tool successfully at least once for this purpose.

I was able to find the commercial software Repairit. It supports dozens of file formats, but it's not obvious from the web page whether it can do what I need.

13
  • 1
    Have you consider to change the filesystem with some which support error correction (so you will have no corrupted files)? Jun 25, 2023 at 6:53
  • 1
    Write to support at RepairIt Jun 25, 2023 at 13:09
  • 1
    If writing a program is an option, most of the work can be done using (LEADTOOLS SDKs)[leadtools.com/sdk/formats/product-comparison-chart]. (Disclosure: I work for the vendor). The RasterCodecs.Load() method supports hundreds of image, document and vector formats. Set the ThrowExceptionsOnInvalidImages Property to true, then loop through the millions of files you have. The method loads all pages from a file: it doesn't just examine the header, but converts the whole file to images and if it finds problems, raises an exception, which the program can catch and record in a log file.
    – Amin Dodin
    Jul 11, 2023 at 15:02
  • 1
    I didn't post as answer because the SDKs are not a ready solution. If you decide to build a solution, they can help you with much of the work but not all of it. For example, they support a wide range of image formats, document formats like PDF, DOCX and XLS, and vector drawing files like DWG and DXF, but they don't support things like ZIP or RAR archives. Also, there's no low-level access to the internals of every format. If a DOCX file is valid, it can be converted to other document formats or rendered as images, but you can't access individual contents of the encapsulating ZIP.
    – Amin Dodin
    Jul 12, 2023 at 15:41
  • 1
    That's why there's a fully-functional free evaluation. Before you commit to investing time and money into building a solution using LEADTOOLS, you get a chance to try any functions or classes you might need, and you get to discuss with our support and sales teams issues like license cost, code performance and optimal ways to use the SDKs (all of this with no strings attached).
    – Amin Dodin
    Jul 12, 2023 at 15:41

1 Answer 1

2

Many zip programs like 7zip have a built-in tester, but only for zip files as zip files contain CRC or similar for each file. This is where you would use a dos or bash for loop to automate the process.

7z t <archive-name>

Linux

for i in `find / -iname "*.zip"`;do 7z t $i>>log.txt;done

More good news docx files are compressed .doc files so 7z t can check them also.

1
  • This is great! 7-Zip can extract dozens of archive file formats, thus with this command I can check many file formats.
    – pts
    Jul 14, 2023 at 8:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.