Python Speech Recognition Library

Question

I'm looking for a library in python that has fairly accurate speech recognition. It would preferably return a string indicating what was said, so that I can work with the string to do other things. Thanks!!!

I looked at this related question, but I don't think we're asking the same thing.

Answer 1

You can use CMU Sphinx:

• free and open source
• recognizer library written in C but provides Python bindings
• often mentioned as one of the best open source speech recognition engines

When looking for a speech recognition software for Linux a while ago, I was told that CMU Sphinx' accuracy is significantly lower than Dragon (I'd be curious if anyone here has a benchmark Sphinx vs Dragon). However, if your voice recordings are in a well restricted domain, you might be able to train CMU Sphinx well enough.

Answer 2

OpenAI's Whisper (off-line, MIT license, Python 3.9, CLI) yields some highly accurate transcription. To use (tested on Ubuntu 20.04 x64 LTS):

conda create -y --name whisperpy39 python==3.9
conda activate whisperpy39
pip install git+https://github.com/openai/whisper.git 
sudo apt update && sudo apt install ffmpeg
whisper recording.wav
whisper recording.wav --model large

If using an Nvidia 3090 GPU, add the following after conda activate whisperpy39

pip install -f https://download.pytorch.org/whl/torch_stable.html
conda install pytorch==1.10.1 torchvision torchaudio cudatoolkit=11.0 -c pytorch

it can be used as a Python lib, e.g.:

import whisper

model = whisper.load_model("base")

# load audio and pad/trim it to fit 30 seconds
audio = whisper.load_audio("audio.mp3")
audio = whisper.pad_or_trim(audio)

# make log-Mel spectrogram and move to the same device as the model
mel = whisper.log_mel_spectrogram(audio).to(model.device)

# detect the spoken language
_, probs = model.detect_language(mel)
print(f"Detected language: {max(probs, key=probs.get)}")

# decode the audio
options = whisper.DecodingOptions()
result = whisper.decode(model, mel, options)

# print the recognized text
print(result.text)

---

Performance info below.

Model inference time:

Size	Parameters	English-only model	Multilingual model	Required VRAM	Relative speed
tiny	39 M	tiny.en	tiny	~1 GB	~32x
base	74 M	base.en	base	~1 GB	~16x
small	244 M	small.en	small	~2 GB	~6x
medium	769 M	medium.en	medium	~5 GB	~2x
large	1550 M	N/A	large	~10 GB	1x

WER on several corpus from https://cdn.openai.com/papers/whisper.pdf:

WER on several languages from https://github.com/openai/whisper/blob/main/language-breakdown.svg: