Offline language detection library or script

Question

One of my Python services was using textblob for language detection. Last month Google made a breaking change to the API service which textblob was calling. Steps have been provided on SO to update textblob to make it work again. But I have decided that I want my service to not be vulnerable to third-party breaking changes. I am therefore looking for an offline solution.

This is the function in question as it is currently. Not much to it:

def is_english(text):
    blob = TextBlob(text)
    language = blob.detect_language()
    return language == "en"

The solution should:

1. Be capable of determining whether a page of text (~~300 words) is written in English or not (just True/False. If it isn't English I don't particularly care what language it is, I'm just dropping it).
2. Be capable of working without an internet connection.
3. Be either a Python library or a script/app that can run on a Linux OS and be called from the CLI.
4. May use ML models but could also just be a library that checks the text against an English word corpus and returns False if not enough of the words are English.

If this doesn't exist I could probably write the corpus checker in (4) above myself but I'd rather use something if it exists.

Answer 1

The polyglot library has a Language Detection package polyglot.detect that seems to fit your requirements perfectly:

1. Be capable of determining whether a page of text (~~300 words) is written in English or not (just True/False. If it isn't English I don't p3articularly care what language it is, I'm just dropping it).

Looking at the introductory example polyglot returns the best match using only 2 lines so you can immediately test for English.

2. Be capable of working without an internet connection.

After installing polyglot locally you can use it offline.

3. Be either a Python library or a script/app that can run on a Linux OS and be called from the CLI.

polyglot is a Python library and can be installed on Linux. It also has a convenient CLI.

4. May use ML models but could also just be a library that checks the text against an English word corpus and returns False if not enough of the words are English.

polyglot depends on "CLD2 is a Naïve Bayesian classifier" that comes with "table-driven scoring is highly optimized for both space and speed, running about 10x faster than other detectors and covering over 70 languages in 1.8MB of x86 code and tables". So there's no need for an additional corpus because the necessary tables com built-in.