10

I have this list with 1600 English words where I don't know the grammatical type. Now I would like to know for each word in this list whether it is a noun, verb, adverb or other grammatical type. If a word can be used with several types, I'd prefer to have them all (but the most common one is enough).

Is there a library with a recognition algorithm for this case? I'd prefer something where I can get started easily — read one tutorial and write 20 lines of code would be ideal.

Requirements:

  • Guess the grammatical type of an English word
  • Any programming language
  • Any license
  • Free
  • Isn't this question better suited for Programmers? – Izzy Jun 18 '14 at 16:15
  • @Izzy No, programmers is more theoretical. I don't know what site this question would fit on however. – Seth Jun 18 '14 at 17:37
  • @Izzy Software Engineering would shriek at a question asking to recommend a programming language. I think the question is fine here, actually — it's more of a library recommendation at the core, and the programming language will follow. – Gilles 'SO- stop being evil' Jun 18 '14 at 18:02
  • 1
    @Gilles Library recommendations should already have a language.. "Suggest me a library in a random language that can do x" is probably too broad, as Tim points out in his answer. – Seth Jun 18 '14 at 18:56
  • @Seth Tim's answer doesn't say anything about library recommendations having to specify a language. Many languages have easy forms of cross-language bindings, so it's fairly common to be able use a library written in language A in a program written in language B. Tim does say that “Recommend what language I should use to build this project is just way too broad”, but the “project” here is basically calling one library function in a loop. – Gilles 'SO- stop being evil' Jun 18 '14 at 19:03
7

You are looking for a POS-tagger(= part-of-speech tagger).

One of the most accurate and widely used is the Stanford Part-Of-Speech Tagger:

  • free (except for commercial non-open source software)
  • open source, written in Java
  • well documented
  • trained models for English, Arabic, Chinese, French, and German are available.
  • Binding available in many other languages: Ruby, Python (NLTK (2.0+) contains an interface to the Stanford POS tagger), PHP, F#/C#/.NET, etc.

Example:

import java.io.BufferedReader;
import java.io.FileReader;
import java.util.List;

import edu.stanford.nlp.ling.Sentence;
import edu.stanford.nlp.ling.TaggedWord;
import edu.stanford.nlp.ling.HasWord;
import edu.stanford.nlp.tagger.maxent.MaxentTagger;

class TaggerDemo {

  private TaggerDemo() {}

  public static void main(String[] args) throws Exception {
    if (args.length != 2) {
      System.err.println("usage: java TaggerDemo modelFile fileToTag");
      return;
    }
    MaxentTagger tagger = new MaxentTagger(args[0]);
    List<List<HasWord>> sentences = MaxentTagger.tokenizeText(new BufferedReader(new FileReader(args[1])));
    for (List<HasWord> sentence : sentences) {
      List<TaggedWord> tSentence = tagger.tagSentence(sentence);
      System.out.println(Sentence.listToString(tSentence, false));
    }
  }

}

Other tools: http://en.wikipedia.org/wiki/Part-of-speech_tagging

5

It's not a built-in function, but you can do so with python and nltk.

A simple code would be like this :

import nltk

with open(file) as f:
    for line in f:
        tmp = nltk.word_tokenize(line)
        print nltk.pos_tag(tmp)

You can find explanation for each tag here (figure 5.1).

Trouble is, it will return the most probable tag and not everyone of them.

  • If I start this it sais: Resource 'taggers/maxent_treebank_pos_tagger/english.pickle' not found. and if I start nltk.download() in the python console, what do I have to download? – rubo77 Jun 18 '14 at 16:18
  • I'm affraid I can't answer this, since I just downloaded everything when I started using it. – fxm Jun 18 '14 at 16:25
  • How do you download everything? I used apt-get install python-nltk – rubo77 Jun 18 '14 at 16:25
  • Once you entered nltk.download(), select all and click download :) However, please note this is highly overkill for your task. – fxm Jun 18 '14 at 16:28
  • I selected book there now, but that takes ages but it worked! the english.pickle in there. this is stored in my user folder /home/rubo77/nltk_data can I delete the other folders there? – rubo77 Jun 18 '14 at 16:30
4

You can use the Apache OpenNLP library:

  • free and open source, written in Java
  • supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech tagging, named entity extraction, chunking, parsing, and coreference resolution.
  • includes maximum entropy and perceptron based machine learning.

Part-of-Speech Tagger Documentation:

Loading a model:

InputStream modelIn = null;

try {
  modelIn = new FileInputStream("en-pos-maxent.bin");
  POSModel model = new POSModel(modelIn);
}
catch (IOException e) {
  // Model loading failed, handle the error
  e.printStackTrace();
}
finally {
  if (modelIn != null) {
    try {
      modelIn.close();
    }
    catch (IOException e) {
    }
  }
}      

Tagging:

POSTaggerME tagger = new POSTaggerME(model);
String sent[] = new String[]{"Most", "large", "cities", "in", "the", "US", "had",
                             "morning", "and", "afternoon", "newspapers", "."};         
String tags[] = tagger.tag(sent);
double probs[] = tagger.probs(); // confidence scores for each tag
Sequence topSequences[] = tagger.topKSequences(sent); // Some applications need to retrieve the n-best pos tag sequences and not only the best sequence
2

You can use IBM LanguageWare (Wikipedia):

  • free (you need to register to be able to download it)
  • Java
  • not sure how active the project is, the latest version is 2011-10-21.
  • Note that LanguageWare does not currently provide part of speech (POS) disambiguation and therefore all ambiguities are passed back up to the calling application.
  • UIMA-based (UIMA = Unstructured Information Management Architecture)
  • AFAIK often not the first choice in academia circles but IBM has made some strong contributions to NLP.

Also available, LanguageWare Resource Workbench is an Eclipse application for building custom language analysis into IBM LanguageWare resources and their associated UIMA annotators. UIMA (see the Apache UIMA project too) is the only industry standard for content analytic and was used by IBM Watson to win the Jeopardy Challenge. UIMA was first developed by IBM, and is now in open source.

A nice presentation of IBM Languageware: Natural language processing and early-modern dirty data: applying IBM Languageware to the 1641 depositions

1

You can use TextBlob (open source, MIT License):

TextBlob is a Python (2 and 3) library for processing textual data. It provides a simple API for diving into common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, classification, translation, and more.

Example:

from textblob import TextBlob

text = '''
The titular threat of The Blob has always struck me as the ultimate movie
monster: an insatiably hungry, amoeba-like mass able to penetrate
virtually any safeguard, capable of--as a doomed doctor chillingly
describes it--"assimilating flesh on contact.
Snide comparisons to gelatin be damned, it's a concept with the most
devastating of potential consequences, not unlike the grey goo scenario
proposed by technological theorists fearful of
artificial intelligence run rampant.
'''

blob = TextBlob(text)
blob.tags           # [('The', 'DT'), ('titular', 'JJ'),
                    #  ('threat', 'NN'), ('of', 'IN'), ...]

blob.noun_phrases   # WordList(['titular threat', 'blob',
                    #            'ultimate movie monster',
                    #            'amoeba-like mass', ...])

for sentence in blob.sentences:
    print(sentence.sentiment.polarity)
# 0.060
# -0.341

blob.translate(to="es")  # 'La amenaza titular de The Blob...'

Features:

  • Noun phrase extraction
  • Part-of-speech tagging
  • Sentiment analysis
  • Classification (Naive Bayes, Decision Tree)
  • Language translation and detection powered by Google Translate
  • Tokenization (splitting text into words and sentences)
  • Word and phrase frequencies
  • Parsing
  • n-grams
  • Word inflection (pluralization and singularization) and lemmatization
  • Spelling correction
  • Add new models or languages through extensions
  • WordNet integration

Installation:

pip install -U textblob
python -m textblob.download_corpora
0

You can use spaCy:

  • Python
  • Open source
  • free for research (GNU Affero General Public License v3), 5kUSD/year for production
  • Linux/ Mac OS X. Windows is not supported.
  • first release in January 2015

Install:

pip install spacy
python -m spacy.en.download

Or:

conda install spacy
python -m spacy.en.download

Demo:

from spacy.parts_of_speech import ADV

def is_adverb(token):
    return token.pos == spacy.parts_of_speech.ADV

# These are data-specific, so no constants are provided. You have to look
# up the IDs from the StringStore.
NNS = nlp.vocab.strings['NNS']
NNPS = nlp.vocab.strings['NNPS']
def is_plural_noun(token):
    return token.tag == NNS or token.tag == NNPS

def print_coarse_pos(token):
    print(token.pos_)

def print_fine_pos(token):
    print(token.tag_)
0

You can use the Python package polyglot, which is a natural language pipeline that supports massive multilingual applications:

  • gratis (GPLv3 license)
  • open source

It does part-of-speech tagging:

import polyglot
from polyglot.text import Text, Word

text = Text(u"O primeiro uso de desobediência civil em massa ocorreu em setembro de 1906.")

print("{:<16}{}".format("Word", "POS Tag")+"\n"+"-"*30)
for word, tag in text.pos_tags:
    print(u"{:<16}{:>2}".format(word, tag))
Word            POS Tag
------------------------------
O               DET
primeiro        ADJ
uso             NOUN
de              ADP
desobediência   NOUN
civil           ADJ
em              ADP
massa           NOUN
ocorreu         ADJ
em              ADP
setembro        NOUN
de              ADP
1906            NUM
.               PUNCT

The POS tagger model is explained in Al-Rfou, Rami, Bryan Perozzi, and Steven Skiena. "Polyglot: Distributed word representations for multilingual nlp." arXiv preprint arXiv:1307.1662 (2013).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.