i know this is a question that has already been asked before ( Old question on stack overflow ) but I've done my homeworks and share the results, and ask if there can be any other method appliable to detect a language from a string block.
First of all, I'm running a global pseudo-crawler that downloads web home pages, and I need to detect the language based on the text-only version of that homepage. So, text blocks are usually long ( more than 256 characters ).
Currently I'm running this pseudo-crawler on a LAMP stack.
I specify that i know there is Google and other good APIs that can do this work, but they're not free, and if you have to run this detection through - say - 1 billion string blocks, it quickly becomes expensive in a way that can not be considered a good approach.
Here's what I've tried to detect the language:
Upsides: very quick detection algorithm. Plug'n'play on a LAMP stack. Downsides: it behaves strangely on different versions of PHP. I was able to run it flawlessly on an environment with PHP 5.3, but on PHP 5.5 and PHP 7.0 it returns "english" for the 99% of the strings, even if the encoding is correct ( for arabic, chinese, and so on )
2) Vocabulary-driven detection ( home-made )
Upsides: very accurate detection. Downsides: extremely slow algorithm, and you have to create a vocabulary for every language you want to detect. In my case, for every language in the world.
Upsides: it's the piece of code that's the youngest I've come to find on the web, it's been on active developing on 2013. Downsides: you have to run it behind mono on debian, because it's a .net application, and this adds a negative overhead ( it's way slower on mono that running it under windows ). Plus, it detects only 14 languages instead of the sponsorized 280+, experimented on a dry run of this application through 300000 text blocks of the "real life".
Every other implementation of language detection I've come to find on the web ( java-based primarily ), uses the "N-Gram-Based Text Categorization", along with the 2 ones I've mentioned above.
So, question is: do you know any other open source project, written in any language, that can be integrated as an external module on linux that can accurately detect the language from a long string? Remember that, I can't use any external API ( if it's paid ), because it's slow, and the detection calls would be literally millions. I know I could write an hybrid system that caches the results of the calls for very short strings and check whether I have that detection in cache or not, but in this case "the cost" would be in terms of detection speed.