One of my Python services was using textblob for language detection. Last month Google made a breaking change to the API service which textblob was calling. Steps have been provided on SO to update textblob to make it work again. But I have decided that I want my service to not be vulnerable to third-party breaking changes. I am therefore looking for an offline solution.
This is the function in question as it is currently. Not much to it:
def is_english(text):
blob = TextBlob(text)
language = blob.detect_language()
return language == "en"
The solution should:
- Be capable of determining whether a page of text (~~300 words) is written in English or not (just True/False. If it isn't English I don't particularly care what language it is, I'm just dropping it).
- Be capable of working without an internet connection.
- Be either a Python library or a script/app that can run on a Linux OS and be called from the CLI.
- May use ML models but could also just be a library that checks the text against an English word corpus and returns False if not enough of the words are English.
If this doesn't exist I could probably write the corpus checker in (4) above myself but I'd rather use something if it exists.