Determine file type from its content (not trusting the .extension)

Question

I have millions of files like report.bin which are maybe a Word file or maybe a PDF file.
I can not trust the filename extension.
Also, I must distinguish between PDF/A and PDF/X, and other distinct formats that are often saved with overlapping filename extensions.

Requirements:

• Free always, for any use. Ideally open source
• Identify as many file formats as possible, using only the file's data/metadata
• Works on Linux, Mac, and Windows
• Works offline
• Must be usable from a script or program, any library interface or command-line interface is fine
• Fast

TrID is not free for commercial use.

Answer 1

file is available on Linux and Mac, plus Windows plus Cygwin or similar.

It is free, open source and identifies files via magic number mostly.

Its database of formats is not very extensive.

It performs fast but does not try very hard.

Command-line interface.

File tests each argument in an attempt to classify it. There are three sets of tests, performed in this order: filesystem tests, magic number tests, and language tests. The first test that succeeds causes the file type to be printed.

The type printed will usually contain one of the words text (the file contains only printing characters and a few common control characters and is probably safe to read on an ASCII terminal), executable (the file contains the result of compiling a program in a form understandable to some UNIX kernel or another), or data meaning anything else (data is usually “binary” or non-printable). Exceptions are well-known file formats (core files, tar archives) that are known to contain binary data.

Answer 2

I used to work for the French National Library, to build an digital archive system that contains not only digitized books but also millions of digital artefacts with all kinds of strange file types. We used JHOVE to recognize file formats.

JHOVE is free and open source, it is maintained by JSTOR and the Harvard University Library.

JHOVE can be used from the command line, via a Java library, or using a GUI.

It is especially strong with TIFF, JPG and other digital preservation formats.

Answer 3

There are several tools out there that could be useful. Here is a page from forensicsWiki that provides a consolidated list.

Apache Tika seems to be the most promising since it is open source, cross platform (Java) and it will be well supported in the future as it is a apache top level project.