I would suggest taking a look at MoviePy which is:
- Free, Gratis & Open Source
- Cross Platform including Linux, Mac & Windows (there is even a docker image available).
It allows a wide range of video compositing (a.k.a. non-linear editing), video processing, or creation of advanced effects. It can read and write the most common video formats, including GIF.
Basically anything that you can draw with python + the various drawing, graphing & text libraries, (which is a lot), can be overlaid onto a video clip. Ideally you have a function that returns the image of the overlay at a given time. For a nice example take a look at the Star Worms demo.
Note: low res & frame rate for upload here!
You may also wish to take a look at some of the matplotlib examples and specifically at the XKCD Font which you will probably like for this sort of work.
I would also strongly suggest taking a look at the cross platform mathematical video creation python library Manim which can produce the overlay videos for a wide range of mathematical usage.