I have a single 1.3gb html file that contains the entirety of the Encyclopedia Britannica. I want to convert it to markdown in a way that preserves subheadings, tables, and UTF-8 special characters.

Unfortunately, this file is so incredibly massive that all the usual conversion tools fail. Pandoc runs for a couple of hours before getting SIGKiLLED by my OS for using more than 60 GB of RAM. Html2text is somewhat less RAM hungry, but after running for more than 48 hours it still hadn't completed, and given the lack of a progress display I had to assume that it was hanging.

I was able to successfully use links to convert the file to plain text, but I would greatly prefer markdown. Subheads and especially tables are formatted terribly this way.

My machine has a Skylake processor clocked at 4.4 Ghz as well as 20 GB of RAM, so I have a decent amount of resources to work with. macOS-compatible software is preferred, but at this point, I wouldn't complain if I had to install and boot into Haiku. Anything to get this file converted.

  • I suspect that making the software leaner isn't the solution. I would try making the data smaller, by breaking up the one big file into several (maybe many!) smaller ones. Granted, that task itself may require its own tooling (ready-made software or write your own), but to me it sounds more achievable than a one-shot conversion of the entire HTML to Markdown.
    – John Y
    Feb 19, 2018 at 16:18
  • @JohnY If you can suggest software to auto split, convert, then recombine, please do—I would consider that a solution! I tried splitting into ten files manually, and all the software I tried still chocked, so as you surmise I would need to automate it somehow. Feb 20, 2018 at 16:56
  • I don't know of any. Some Googling turns up a blog describing HTMLDOC, though its links to HTMLDOC are obsolete (it's here now). I was actually thinking mainly along the lines of writing your own, such as this person did.
    – John Y
    Feb 20, 2018 at 22:40
  • Before you do embark on automated splitting, you may want to do more experimentation with manual splitting, to get a better idea of how big is too big and how small is small enough. For example, just try one encyclopedia topic to see if it's small enough. And check that the automated conversion to Markdown is working the way you want it to even on small-enough pieces. I can imagine tables in particular being a potential trouble spot, since the original Markdown didn't really support them. Some newer, more feature-laden Markdown dialects do though.
    – John Y
    Feb 20, 2018 at 22:48


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.