I would like a software recommendation that summarizes a long text document and produces keywords for that text document. Ideally, it will exclude words like "with, and, the, etc." and include more important words. Ideally so i can script using Python3
I am unfamiliar with search engines but I know they have indexing techniques to rank keywords that summarize documents/webpages. My particular scenario does not require such a sophisticated suite (SQL databases and indexing entire filesystems, etc.). All my documents that I am interest to find are in one directory, so linux terminal utilities like
grep are sufficient to parse. Maybe a full suite will perform better?
I have looked at PyLucene but it seems to have a lot more than I need. Can someone please give some recommendations for software or what kind of field of study this is, so I can do some of my own research, is it called indexing?
Explanation of issue
I have hundreds of text documents (with a unique made-up file extension,
.crp). Each of these
.crpfiles describe, in plain text, a scenario for an experiment (basically a
README) for the logs that my program generated.
For example I have directory called
scanario_1which contains a
.logfiles for the experiment. The
description.crpdescribes the conditions under which the
.logfiles were generated.
I write a each detailed
description.crpfile manually and also manually generate a
keywords.crpfile which summarizes the
description.crpfile. So I can perform queries on the
keywords.crpfiles and not have to search the lengthy description.
Problem is, as I analyze the logs for performance and make notes on it (update the
description.crt) I have to then go an update each
keyword.crpfile. And I would like to automate the creation of the
keyword.crpfiles by having something read the
description.crtfile and produce the keywords