I have a book written in Markdown/Markua, separated into chapters. Currently, it consists of about 26 individual *.md files. I'm planning to write an index for the book, but I'm unable to find a command-line utility or Ruby gem that will do the following:
- Provide a word-frequency list from the .md input files that I can sort alphabetically or by count. (NB: This is the relatively easy part, as Ruby can use
str.split.tallyto generate a hash. I can roll my own for this part, if necessary.)
- Provide a reliable list of editable stopwords to remove from the frequency list. I've found several on the web, including this one from MySQL, but they all vary enough to seem non-canonical.
- Allow treating certain phrases ("connected words") that are relevant to the domain as individual items. For example, "Scrum Master" ought to be treated as a single word for indexing, rather than two.
- Support for inflections and contractions, e.g.
octopus => octopiand
Scrum's, Scrums => Scrum.
- (Optional) Bonus points if the indexing can point to source files or line numbers as well. This is less critical because I can use ack, grep, ag, or other similar tools to locate the indexed items, but it's a "nice to have."
A GUI like IndexMakerS doesn't seem helpful, as it only works on PDF files, and doesn't seem to allow command-line editing of the stopwords and phrases. As a programmer (but not an NLP expert) I've also tried to integrate stopwords, connected words, and inflections into a coherent hash when rolling my own code, but it's a little more complex than I'd anticipated.
Most importantly, the end goal isn't to just generate the frequency list, but to use the top X% of the filtered list to programmatically insert index and cross-link markup into my Markdown/Markua documents. While it would be nice if there was a tool that did that cross-linking for me, that's probably a step too far as Markua is a fairly specialized use case. I'll cross that bridge myself if I can find something to do the heavy lifting of properly indexing the words and phrases I need.
As long as I have the editable lists and can extract the words/phrases I'm looking for, I can programmatically insert the markup I need. I've tried searching for phrases like
command line index generator on the web, but haven't turned up any meaningful results.
Is there a Linux, Mac, or BSD tool that will do what I'm trying to accomplish?