Document Triage platform

Question

I'm looking for some kind of software platform (or possibly open source API would do most of this) to perform document triage.

Imagine you get 3,000 documents (word, pdf, powerpoint, mixed formats) on a cluster of 5 different topics. We'd want a partially automated solution that would help:

• Determine the rough topic coverage in the body of the documents
• Help a person prioritize which to read first, and what order to read them in, to minimize the amount of time sifting through junk
• Search all of the documents across formats for certain keywords or phrases, optimally with some ability to define some synonyms (for example, "acquisition" and "purchasing")
• Determine which documents are copies/versions of one another so that only one copy would be examined

Answer 1

First, install Alfresco and the Calais integration (it might take a day depending on your experience).
Then upload all of your documents to Alfresco.

Calais is a library/API developed by Reuters to extract semantic information from human text.

You will now be able to:

• Find all documents about purchasing, with a nice tag cloud.
• Quickly search for all documents that contain a particular keyword. You can also combine this search with conditions on tags, filename, author, date, etc.
• You can prioritize by "starring" the documents you need to read first.

Alfresco used to have a module for finding duplicates but I can't find anymore.

Answer 2

I would suggest that you would first need to get a copy of all of the documents in a plain text format, possibly markdown.

Assuming that you have tools to open most of them that can output into plain text, possibly automated via python with win32com, N.B. for the pdf documents a lot depends on the type - if they are scanned documents that just have the images of the pages you are out of luck - if they have been generated by software such as print to pdf then you could use pdfminer. You should also capture, along side, meta information such as last update date, etc.

Once you have all of the files in a plain text format you can then use tools such as NLTK to fingerprint each of your text files by parsing them to extract significant items such as nouns & verbs then count each of those items. Looking for your keywords in these lists should give an indication of which of the original files is most, and least, worth looking at. Files with very similar lists of the significant items and similar counts are probably near copies of each other.