I have a book written in Markdown/Markua, separated into chapters. Currently, it consists of about 26 individual *.md files. I'm planning to write an index for the book, but I'm unable to find a command-line utility or Ruby gem that will do the following:

  1. Provide a word-frequency list from the .md input files that I can sort alphabetically or by count. (NB: This is the relatively easy part, as Ruby can use str.split.tally to generate a hash. I can roll my own for this part, if necessary.)
  2. Provide a reliable list of editable stopwords to remove from the frequency list. I've found several on the web, including this one from MySQL, but they all vary enough to seem non-canonical.
  3. Allow treating certain phrases ("connected words") that are relevant to the domain as individual items. For example, "Scrum Master" ought to be treated as a single word for indexing, rather than two.
  4. Support for inflections and contractions, e.g. octopus => octopi and Scrum's, Scrums => Scrum.
  5. (Optional) Bonus points if the indexing can point to source files or line numbers as well. This is less critical because I can use ack, grep, ag, or other similar tools to locate the indexed items, but it's a "nice to have."

A GUI like IndexMakerS doesn't seem helpful, as it only works on PDF files, and doesn't seem to allow command-line editing of the stopwords and phrases. As a programmer (but not an NLP expert) I've also tried to integrate stopwords, connected words, and inflections into a coherent hash when rolling my own code, but it's a little more complex than I'd anticipated.

Most importantly, the end goal isn't to just generate the frequency list, but to use the top X% of the filtered list to programmatically insert index and cross-link markup into my Markdown/Markua documents. While it would be nice if there was a tool that did that cross-linking for me, that's probably a step too far as Markua is a fairly specialized use case. I'll cross that bridge myself if I can find something to do the heavy lifting of properly indexing the words and phrases I need.

As long as I have the editable lists and can extract the words/phrases I'm looking for, I can programmatically insert the markup I need. I've tried searching for phrases like command line index generator on the web, but haven't turned up any meaningful results.

Is there a Linux, Mac, or BSD tool that will do what I'm trying to accomplish?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.