2

Say I have a string representing text in a natural language, for example:

She is an effective teacher. Most students have found her reasonably helpful.

Is there a programming library or command line program that will take this as input and return a tree structure, the kind that is similar to syntax trees produced by parsers for formal/programming languages?

I know of POS taggers, but these taggers only seem to analyse the text at the word level. I need the constituents of all other levels.

Hence, the tree structure should reflect the structure of the text at all levels ie discourse, sentence, clause, phrase, group, word, and even morpheme (both inflectional and derivational) and I should be able to traverse this tree from the bottom-most node to the "root" node... So traversing the whole tree of the above-mentioned string sample should give something like:

  • Constituent/Element----Constituent/Element Type/Part Of Speech----Level
  • She----Pronoun----Word
  • Is----Verb----Word
  • An----Determiner----Word
  • Effective----Adjective----Word
  • teacher----Noun----Word
  • ...
  • She----Subject/Noun Phrase----Phrase
  • Is----Predicate/Verb Phrase----Phrase
  • An effective teacher----Object/Noun Phrase----Phrase
  • ...
  • She is an effective teacher.----Simple sentence----Sentence
  • Most students have found her reasonably helpful.----Simple sentence----Sentence

This sort of library and/or program should support the following languages at the very least:

  • English
  • Chinese Mandarin
  • Korean

It would be good if it also supports:

  • Cantonese
  • Malay / Indonesian
  • Russian
  • Arabic
  • ...and more!...

For languages that have inflectional morphology and particles, the parser should also recognise the elements below the level of the word, and the particles. Eg. In Korean, a word like 하모일어서 should be parsed as 하모일/NNG+이/VCP+어서/EC

Actually, I have also thought of achieving the abovementioned by creating my own parser using a parser generator like ANTLR, which takes in a BNF grammar as input, and produce a parser program/executable as output.. But I haven't been able to find BNFs for natural languages so far.. If one can provide BNF resources for natural languages that can be used with ANTLR, that can also be accepted..

System requirements:

PHP,Javascript,Java,Python (or any other language) (listed in order of preference from most preferred to least) AND/OR Command line program

For use on MacOS 10.13.6

Open source where possible

1 Answer 1

1

What about Apache OpenNLP and its SentenceDetector??

However OpenNLP expects from you that you train and finetune your own models, because it cannot know which types of sentences you intend to test it with. For example, you cannot expect a model trained on English sentences to accurately classify test-sentences given in the German language.

I think there are European-Language corpora available, and also tutorials. I think more models (for Asian languages) can be found by searching the internet and the academic literature.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.