Free software for background/foreground image extraction

Question

I have many videos that I would like to replace the background in with my own custom background. I have a picture of the background in the video without any of the foreground objects that I want to remove.

I would like to be able to either remove the existing background from either the entire video or I could do it frame by frame using PNGs.

Microsoft's Background Cut algorithm seems the perfect fit, but I can't find any implementation of it.

I've also seen various computer vision software that can identify foreground regions that have changed compared to the blank background, but nothing that I have seen does a good job of cleanly extracting the foreground objects.

Is there any free software, preferably for Linux or OS X, that can do this? It would be nice for it to be open-source, but anything that is free as in $0.00 would be acceptable.

Answer 1

To my knowledge, there is no public software that does this well yet. I come from a background in video production where this process is called "keying." To perform the current operation you've outlined would require "rotoscoping," i.e., going through frame by frame and manually tracing the outline of your subject. Rotoscoping is typically a slow, tedious process.

You're probably aware that color is typically used to pull keys automatically, e.g., with blue or green screens. This is called chroma keying and even it can suffer from difficulty with edge detection due to color bleeding.

I've heard the process you want called "depth keying." Here's another paper by Microsoft that explores the topic: research.microsoft.com/en-us/um/people/eyalofek/depth key/depthkey.pdf I believe that Adobe is currently working on similar technology, though I am not sure when such tools will be publicly available (let alone free or open-source).