I'd like to write a script performing the following task: read a number of videos, determine a common chunk of frames that appear in all of them, then delete that common chunk from each video.
What are some libraries that provide this kind of functionality for manipulating videos and images (either closely matched functionalities, or low-level blocks that I can make use of)? I'm ok with the library being in any language.
The most used tool/library for video manipulation is FFMPEG - It can read, slice, dice & write just about every video format that is out there and it is free, gratis & open source and cross platform.
All of that power comes at a price in usability so there are lots of projects that wrap things up to make them a little more presentable - one of my favourites is MoviePy - also free & cross platform and well worth a look which can greatly simplify cutting and splicing your files.
The big problem is recognising the "identical" blocks within each file. This is a problem because, due to encoding and compression, even though they look identical to you they will not be to the computer. Which leads us on to Computer Vision as you are going to have to search for visually similar blocks within the files.
The go-to tool for Image Recognition is OpenCV, again free, gratis, open source & cross platform and there are 1000s of books, articles & papers on how to use it. It is primarily C++ but there are python and other bindings.
It would be far too much to describe how you would need to search for visually similar blocks in multiple files but the process will consist of extracting a low definition, monocrome, fingure print at a given interval through each film and then comparing those & refining the results with multiple passes.
You can expect your computer to be spending a lot of time processing looking for the "Identical" blocks unless you can give it a hint or two - if, as I suspect, you are looking to remove the adverts from a set of videos recorded off air then you can usually, on any given station, set your watch by when the adverts are going to be - also many stations use a cue block in the top right corner of the screen when the ads are about to start. Both of these can be useful to narrow the search.
If you are really lucky, or have not recorded your footage yet, you will have the subtitles track on each - this can really speed things up as searching text for identical patters is a lot faster than frames for similarities - VideoGrep is a python script that uses the subtitle track to extract movie clips that match requirements and is probably a very good place to start.
For thinking outside of the box on selecting video segments you might also get some ideas from this MoviePy script which selects and joins together the interesting bits of a soccer game based on the noise the crowd is making.
You could check out the LEADTOOLS Multimedia SDK to implement this type of task. It supports C, C++, .NET and will work with other languages that support COM objects. This SDK supports decompressing video files and provides a video callback filter to get the individual frames to do custom image processing on. This would allow you to use the same algorithm for processing without needing special code to handle different file formats. If the files you're processing use MPEG-2 or H.264 compression, you would also be able to utilize (natively) hardware decompression to offload some of the work to the GPU.
Here is a forum post on the LEADTOOLS Support Forums that has a sample project illustrating how to use the Video Callback: HOW TO: OCR Video using the callback filter
A key part of this forum post is in adding the video callback. Here is the relevant code on how to do that:
Processor videoCallback = playctrl.VideoProcessors.Callback;
playctrl.SelectedVideoProcessors.Add(videoCallback);
lmvCallback = (LMVCallback)playctrl.GetSubObject(PlayObject.SelVideoProcessor);
lmvMyUserCallback = new LMVMyUserCallback();
lmvCallback.ReceiveProcObj = lmvMyUserCallback;
The other key part of this demo is the implementation of the ReceivePro() method, as this is the invoked callback function. The code to pay attention to is this:
public void ReceiveProc(int pData, int lWidth, int lHeight, int lBitCount, int lSize, int bTopDown)
{
try
{
if (m_bSnapshot)
{
// Get the background image
RasterViewPerspective viewPerspective;
if (bTopDown == 1)
viewPerspective = RasterViewPerspective.TopLeft;
else
viewPerspective = RasterViewPerspective.BottomLeft;
RasterImage img = new RasterImage(
RasterMemoryFlags.User, //A combination of the RasterMemoryFlags enumeration members indicating the type of memory to allocate for the image data.
lWidth, // Width of the image in pixels.
lHeight, //Height of the image in pixels.
lBitCount, //The number of bits per pixel.
RasterByteOrder.Bgr, //Color order for 16-, 24-, 32-, 48- and 64-bit images.
viewPerspective, //Specifies where the beginning of the image is stored.
null, //The palette that the image will use. You can specify your own palette, or use null (Nothing in Visual Basic) for LEAD's fixed palette. The palette member is used only when bitsPerPixel is less than or equal to 8.
IntPtr.Zero, //Unmanaged data pointer that will contain the image data when flags is RasterMemoryFlags.User.
0); //Length in bytes of the data passed to userData. Only when used when userData is not IntPtr.Zero and flags is RasterMemoryFlags.User.
img.SetUserData(new IntPtr(pData), lSize);
As this project also illustrates, you can use features from the LEADTOOLS Recognition Imaging SDK if the common frames you are looking for contain text. This SDK also contains the image processing class CorrelationCommand, which is designed to find specific image data within another image. Here is some sample code illustrating how you would set up and run this method:
RasterCodecs codecs = new RasterCodecs();
// Prepare the command
RasterImage DstImage = codecs.Load("source.png");
CorrelationCommand command = new CorrelationCommand();
command.CorrelationImage = DstImage;
command.Threshold = 70;
command.XStep = 1;
command.YStep = 1;
command.Points = new LeadPoint[90];
//Apply the correlation filter.
command.Run(img);
With access to each frame in the video stream, you would be able to do any type of image processing you wanted to do on the frame.
Disclaimer: I am an employee of the company that wrote this library.