I want to import a huge database of short English sentences into an educational software.
I need to categorize sentences into a few levels of "difficulty" (difficulty for a non-native to understand the sentence).
- "My name is Joe and I like ponies, dogs, cats, rabbits and mushrooms." → 0.1
- "I want to evaluate the complexity of this sentence." → 0.5
- "Beware those whose voice supersedes this very edict." → 0.9
... or any similar system of scoring.
A good algorithm would take into account:
- the difficulty of each term (Coelacanth is more difficult than goldfish because it is used less often in books, notably children/teen books. Will is more difficult than willingness because it has synonyms. etc)
These would have to be retrieved from a pre-established database (this might help).
- the difficulty of the grammar and conjunctions (that is more difficult than and).
- ideally the complexity of in-sentence references (sentence where several expressions refer to each other, forming a complex graph with the former at first unclear but the latter later understood... or not)
- Sentence length. This is for memorization, so a sentence twice as long might as well be considered twice as difficult.
- other parameters might be useful too.
- Able to process around 1,000,000 entries within a few weeks
- Web service acceptable, but bonus if runnable on-premise.
- Bonus for Open Source.
- Bonus if supports other languages too.
OGTE asks me for a sentence and a school level, then tells me whether it fits or not. I would like a tool that asks me for a sentence, and tells me what school level it fits.