A Linux tool / utiltiy for identifying non-stopwords in English text

Question

Is there a tool I can sent a text file or other file into that will identify unusual words (not in a stopword list) in Linux?

For example, words that are generally in the English language will be excluded, but technical jargon words will be displayed as non-stop (unusual) words with a count. It should also be case sensitive.

Here is an example of what I mean...

If I have the sentence:

"Tommy programmed in Powershell, PowerShell, C++, and SQL, while using VirtualBox, on his VirtualBox."

Then it would would return a result like this:

1, Powershell
1, PowerShell
1, C++
1, SQL
2, VirtualBox

I guess what I'm trying to do is standardize the case of the unusual words, and the tool will take the text and try to find it, so when I correct it, I know I've gotten it all.

I'm aware you can search for things with vim, but it's a lot easier to not miss anything if you already have a list of words to ignore (stopwords) that make it easy to separate the stopwords from the unique ones.

(And of course you can always remove any words like "Ruby" from the stopword list so they show up too when you need them to)

I mean really all you need is a list of words and a script to run through them, but I figured I'd ask if there wasn't already a pre-built tool for this.

Answer 1

The python Natural Language Tool Kit nltk is probably your best bet here. There are examples of the sort of thing that you are trying to do in the online book such as the unusual words example.

• There is plenty of support for counting occurences
• I would suggest that you use case insensitive checks for removing the "common" words before counting case sensitively.
• It is python so available for Linux and just about everything else.