I'm an animator trying to get a program that will use machine learning to automate the lip syncing process. The program I need made will have two files given to it: a seed, and an audio file. The seed will be a video made in another program, it will be anywhere between a few minutes to a few hours of hand synced audio, done in a different program. The seed will be made uniquely for any new speaker, so as to train the program for every person. A transcript of the video can be added if it would help, but I'm okay removing this if you believe it won't. Then the audio will be added, this is the one needing to be synced. It will then need to go through, listen frame by frame and compare the sounds to the seed video. When it finds a match, it will save whatever image is in the video at that frame into a sequence. The final output will be a png image sequence that can be played at 30 frames per second in a video editor along with the audio. For the sake of quality, I would prefer that the seed also be an image sequence I upload into it with the seed's audio, this way the exported image sequence will have transparency, if that's possible. Also when the audio is being recorded, there is a video being recorded as well of the speaker, if adding some sort of facial tracking could also help, I'd like that as well. Lastly, I'd prefer this to run as quickly as possible. So not running real-time, seeing as the audio being synced can sometimes be up to several hours long. I will down the line like an ability to do lip syncing live (with perhaps a 5 second delay) and it just displays in a window and syncs to a chosen microphone or audio device.
eg: So for example, when the program is listening to the audio that needs to be synced, let's say it hears the "Ah" sound. It will then search the seed audio for the best match to this sound, let's say it's the first time the word "and" is said. It hears a matching Ah" sound, and then looks to the video to see what image is currently being displayed. It copies that image, then moves on to the next frame/sound. In the end it should produce an image sequence that could play back at 30 frames per second in an editor alongside the audio.
So which program or DSP techniques will help in this?
