Does anyone know of a free typesetting program that can use I can to do layouts of old Jewish books?
I.e. a page of the Talmud like this (tractate shabbat 3a):
Latex is a free and high quality typesetting system with highly customizable layouts. The one you linked could be replicated by use of the custom environments. It might take time to learn especially if you have no previous experience with markup languages, but you can find excellent community support on the StackExchange TeX and LaTeX website.
Here is a very similar question on TeX.SE where this issue was successfully addressed
LaTeX is cross-platform, and you could write it using any text editor. I recommend TeXStudio since it is easy to setup and has a beginner-friendly interface with auto-complete, syntax highlighting, and a built-in viewer. TeXStudio is also free and cross-platform