I have around 2 hours of educational videos for 5-10 year children. I want to dub the videos into different languages and accents with computer voices. I have audio tracks and the script of the audio.

Maestra and Glitch.digital seem very easy to use and produce some initial results, and could serve for documentaries, but do not good enough to my ears to hold a young audience's attention. Google WaveNet and Amazon Polly have very realistic voices and allow further fine-tuning, such as rate, pitch, and emphasis.

I am looking for a solution to translate videos into multiple languages with these characteristics, by order of importance:

  • automatic timestamping of the original script with the audio track, and the final audio has translated sentences "lip-synced" to those timestamps
  • target accent within a language (e.g. UK English or US English)
  • different voices for different speakers (e.g. male and female)
  • fine-tuning of the target language input, e.g. emphasis of a sentence, adjusting pauses, pitch and rate of certain words
  • add vocal fillers like "mmm", "uuh"
  • translation of the original script to the target language

If possible, I would also like to have the voice sing the songs in the videos, but I can also remove the songs.

I could program all of these (except singing) with Python, Speech-to-text, and text-to-speech on Google Cloud. That would take me weeks, so I'm curious about easier options.

Do existing solutions allow this fine-tuning for polishing videos in multiple languages?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.