In Short
I would like a software recommendation that summarizes a long text document and produces keywords for that text document. Ideally, it will exclude words like "with, and, the, etc." and include more important words. Ideally so i can script using Python3
I am unfamiliar with search engines but I know they have indexing techniques to rank keywords that summarize documents/webpages. My particular scenario does not require such a sophisticated suite (SQL databases and indexing entire filesystems, etc.). All my documents that I am interest to find are in one directory, so linux terminal utilities like grep
are sufficient to parse. Maybe a full suite will perform better?
I have looked at PyLucene but it seems to have a lot more than I need. Can someone please give some recommendations for software or what kind of field of study this is, so I can do some of my own research, is it called indexing?
Explanation of issue
I have hundreds of text documents (with a unique made-up file extension,
.crp
). Each of these.crp
files describe, in plain text, a scenario for an experiment (basically aREADME
) for the logs that my program generated.For example I have directory called
scanario_1
which contains adescription.crp
and.log
files for the experiment. Thedescription.crp
describes the conditions under which the.log
files were generated.I write a each detailed
description.crp
file manually and also manually generate akeywords.crp
file which summarizes thedescription.crp
file. So I can perform queries on thekeywords.crp
files and not have to search the lengthy description.Problem is, as I analyze the logs for performance and make notes on it (update the
description.crt
) I have to then go an update eachkeyword.crp
file. And I would like to automate the creation of thekeyword.crp
files by having something read thedescription.crt
file and produce the keywords