In Short

I would like a software recommendation that summarizes a long text document and produces keywords for that text document. Ideally, it will exclude words like "with, and, the, etc." and include more important words. Ideally so i can script using Python3

I am unfamiliar with search engines but I know they have indexing techniques to rank keywords that summarize documents/webpages. My particular scenario does not require such a sophisticated suite (SQL databases and indexing entire filesystems, etc.). All my documents that I am interest to find are in one directory, so linux terminal utilities like grep are sufficient to parse. Maybe a full suite will perform better?

I have looked at PyLucene but it seems to have a lot more than I need. Can someone please give some recommendations for software or what kind of field of study this is, so I can do some of my own research, is it called indexing?

Explanation of issue

I have hundreds of text documents (with a unique made-up file extension, .crp). Each of these .crp files describe, in plain text, a scenario for an experiment (basically a README) for the logs that my program generated.

For example I have directory called scanario_1 which contains a description.crp and .log files for the experiment. The description.crp describes the conditions under which the .log files were generated.

I write a each detailed description.crp file manually and also manually generate a keywords.crp file which summarizes the description.crp file. So I can perform queries on the keywords.crp files and not have to search the lengthy description.

Problem is, as I analyze the logs for performance and make notes on it (update the description.crt) I have to then go an update each keyword.crp file. And I would like to automate the creation of the keyword.crp files by having something read the description.crt file and produce the keywords

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.