i know this is a question that has already been asked before ( Old question on stack overflow ) but I've done my homeworks and share the results, and ask if there can be any other method appliable to detect a language from a string block.

First of all, I'm running a global pseudo-crawler that downloads web home pages, and I need to detect the language based on the text-only version of that homepage. So, text blocks are usually long ( more than 256 characters ).

Currently I'm running this pseudo-crawler on a LAMP stack.

I specify that i know there is Google and other good APIs that can do this work, but they're not free, and if you have to run this detection through - say - 1 billion string blocks, it quickly becomes expensive in a way that can not be considered a good approach.

Here's what I've tried to detect the language:

1) Text_LanguageDetect pear package.

Upsides: very quick detection algorithm. Plug'n'play on a LAMP stack. Downsides: it behaves strangely on different versions of PHP. I was able to run it flawlessly on an environment with PHP 5.3, but on PHP 5.5 and PHP 7.0 it returns "english" for the 99% of the strings, even if the encoding is correct ( for arabic, chinese, and so on )

2) Vocabulary-driven detection ( home-made )

Upsides: very accurate detection. Downsides: extremely slow algorithm, and you have to create a vocabulary for every language you want to detect. In my case, for every language in the world.

3) NTextCat (c#)

Upsides: it's the piece of code that's the youngest I've come to find on the web, it's been on active developing on 2013. Downsides: you have to run it behind mono on debian, because it's a .net application, and this adds a negative overhead ( it's way slower on mono that running it under windows ). Plus, it detects only 14 languages instead of the sponsorized 280+, experimented on a dry run of this application through 300000 text blocks of the "real life".

Every other implementation of language detection I've come to find on the web ( java-based primarily ), uses the "N-Gram-Based Text Categorization", along with the 2 ones I've mentioned above.

So, question is: do you know any other open source project, written in any language, that can be integrated as an external module on linux that can accurately detect the language from a long string? Remember that, I can't use any external API ( if it's paid ), because it's slow, and the detection calls would be literally millions. I know I could write an hybrid system that caches the results of the calls for very short strings and check whether I have that detection in cache or not, but in this case "the cost" would be in terms of detection speed.

  • Why do you consider (2) slow? Looking up words in a table is pretty fast. Creating a vocabulary for many languages isn't a problem: word lists are easier to find than more advanced data. Is the problem agglutinative or otherwise not-exactly-word-based languages? (That's a problem I've never had to deal with.) – Gilles 'SO- stop being evil' Feb 15 '16 at 21:56
  • I didn't verified this information but time ago I were looking for some things related about language syntax lexers for an specific project with Scintilla, then I heard that notepad++ comes with keyword lists for each language they support, so maybe that could be is a starting point for (2). – ElektroStudios Feb 15 '16 at 22:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy