Convert PDF files to wiki?

Question

I'm looking to convert a very messy* set of reference PDFs to wikitext. Ideally, I'd like to make this process a simple one-and-done application, but understand that such a tool may not be available. Using a series of intermediary tools is possible.

It's preferred that intermediary tools meet the following specifications (The first two are merely preferences, and can be disregarded in favor of functionality):

• Convert to HTML rather than Word
• Have a GUI rather than a command line.
• Can handle a couple hundred pages of PDF.
• Are open source (though "free trial" is acceptable.)
• MediaWiki is the preferred output format, but other formats would be acceptable given that:
  • it can be hosted on LAMP stack or is a free SaaS-wiki
  • multiple user access
  • the ability to make the wiki private

*The PDFs in question are both difficult to use, as they frequently refer to pages within themselves and within each other, without internal linking. They also happen to use a two column formatting, which most of the convert-to-HTML/text tools I've tried so far (Calibre/FoxItPDF) can't seem to handle - They either export the text on the same line, or they create paragraph elements that alternate between the two columns - it might be possible to transform these output files using some jQuery, but that's a lot of work for little gain.

Windows 10 or Linux (Ubuntu) would be considered available platforms. Web apps or google drive are also possibilities.

Answer 1

I suggest you perform the following steps:

PDF → Word → HTML → MediaWiki

PDF to Word

There are a few free online tools which perform the conversion with a very good quality. The one I like most is SmallPDF. I think this approach is better, compared to pdftotext, because it has good heuristics to detect headings and preserve essential formatting (bold, italic).

Word to HTML

You can use Pandoc to perform the conversion from DOCX to HTML:

pandoc -i file.docx -s -o file.html

The -s (standalone) switch is important, because it will save encoding information in the HTML file. If you open the file with a web browser, you want to see it correctly.

A caveat I've noticed while testing is that slightly indented paragraphs might be considered <blockquote>s by Pandoc. Keep that in mind while performing your conversions. You might also want to check the HTML file before performing the final step. Formulas will probably be broken.

HTML to MediaWiki

You can do this step using Pandoc, too. Theoretically, the HTML step may not be necessary, however IMHO it is useful to check that everything is OK using a browser for a quick preview.

pandoc -i file.html -t mediawiki -o file.wiki.txt

Example

I tested this method with the paper Imperfect Forward Secrecy: How Diffie-Hellman Fails in Practice. The original PDF looks like this:

The HTML file (after the first two steps) is quite good:

The indentation comes from paragraphs treated as quotes. I believe it can be fixed by tweaking the DOCX file, if needed.

Finally, here's an excerpt from the MediaWiki code:

<blockquote>'''Imperfect Forward Secrecy: How Diffie-Hellman Fails in Practice'''

David Adrian''¶'' Karthikeyan Bhargavan''∗'' Zakir Durumeric''¶'' Pierrick Gaudry''†'' Matthew Green''§''

J. Alex Halderman''¶'' Nadia Heninger''‡'' Drew Springall''¶'' Emmanuel Thomé''†'' Luke Valenta''‡'' Benjamin VanderSloot''¶'' Eric Wustrow''¶'' Santiago Zanella-Béguelin''&quot;'' Paul Zimmermann''†''
</blockquote>
== ''∗'' INRIA Paris-Rocquencourt ''†'' INRIA Nancy-Grand Est, CNRS, and Université de Lorraine ==

<blockquote>''&quot;'' Microsoft Research ''‡'' University of Pennsylvania ''§'' Johns Hopkins ''¶'' University of Michigan

For additional materials and contact information, visit [https://weakdh.org/ WeakDH.org.]

'''ABSTRACT'''

We investigate the security of Diffie-Hellman key exchange as used in popular Internet protocols and find it to be less secure than widely believed. First, we present Logjam, a novel flaw in TLS that lets a man-in-the-middle downgrade connections to “export-grade” Diffie-Hellman. To carry out this attack, we implement the number field sieve discrete log algorithm. After a week-long precomputation for a specified 512-bit group, we can compute arbitrary discrete logs in that group in about a minute. We find that 82% of vulnerable servers use a single 512-bit group, allowing us to compromise connections to 7% of Alexa Top Million HTTPS sites. In response, major browsers are being changed to reject short groups.

We go on to consider Diffie-Hellman with 768- and 1024-bit groups. We estimate that even in the 1024-bit case, the com- putations are plausible given nation-state resources. A small number of fixed or standardized groups are used by millions of servers; performing precomputation for a single 1024-bit group would allow passive eavesdropping on 18% of popular HTTPS sites, and a second group would allow decryption of traffic to 66% of IPsec VPNs and 26% of SSH servers. A close reading of published NSA leaks shows that the agency’s attacks on VPNs are consistent with having achieved such a break. We conclude that moving to stronger key exchange methods should be a priority for the Internet community.
</blockquote>
= INTRODUCTION =

<blockquote>Diffie-Hellman key exchange is widely used to establish session keys in Internet protocols. It is the main key exchange mechanism in SSH and IPsec and a popular option in TLS. We examine how Diffie-Hellman is commonly implemented and deployed with these protocols and find that, in practice, it frequently offers less security than widely believed.

Answer 2

After some experimenting, I've come up with a multi-software solution on Linux shell. It preserved formatting very well in my attempts, so I can say I'm pleased with the results of the HTML conversion. The mediawiki output may still need some cleanup on occasion, but overall we got a very good result.

Installing Tools

We'll be using the following command-line tools:

• Poppler
• Pandoc

These can be installed using the following commands (Ubuntu Linux 14.04 assumed. Adjust directions for your version of Linux. Some of these tools might also work on Windows, but I'm not providing install or usage instructions for them.)

For Poppler: sudo apt-get install poppler-utils

For Pandoc: Installation Guide

Pandoc specifically recommends downloading the .deb and installing from it - however, if you don't mind an older version and are willing to risk any bugs associated with it sudo apt-get install pandoc might work just fine.

Using Poppler To Extract PDF Text as HTML

Poppler includes a number of command line tools to extract things like images from PDF, and they are better detailed here.

Steps:

1. Navigate to the directory holding your PDF(s) for conversion.
2. Make a subdirectory for the output files: sudo mkdir dirname.

3. Run the following command:

   sudo pdftohtml -s -p -fmt png -nodrm "file.pdf" "file/file.html"

This command will create a lot of files, which is why we contain the results in their own directory. It will extract any images in the file, so all of those will be saved there. It also creates two HTML files, one of which will be an outline, and the other of which will contain all of the text in a formatting very close to the original.

You can type pdftohtml -h to gain a better understanding of available parameters.

I've explained the parameters used here for the sake of understanding the command:

• -s contains all of the output within one HTML document (excluding the outline.
• -p attempts to replaces pdf internal linking with html links.
• -fmt controls the output format of images, with png and jpg being valid options.
• -nodrm igores download rights management restrictions on the PDF.
• -i ignores images. I didn't use this, but it felt prudent to mention as in some cases it may massively speed your output format.

Alternative Method: Poppler pdftotext

Poppler also has a pdftotext command. This was the only tool I've found so far that handled PDF extraction well in an instance of having two columns of text. While other tools were printing straight across from left to right or alternating lines of text from the two columns, Poppler put the text together in the right order.

Run the following command:

    pdftotext -htmlmeta "file.pdf" "file.html"

 Replace "file" with the name of the file you want to parse and with the name of the HTML file you want to write your text output to. 
 The `-htmlmeta` option creates an HTML version of the text in your PDF. (This is much less fancy than the previous command and only puts the text in `pre` tags). You should see an HTML file in your directory which you can open to check the results of. Depending on the formatting of your source PDF file, you may find that Poppler is variable in it's effectiveness. You can try running `pdftotext -h` for information on other command options that may improve or worsen your results. 

Using Pandoc to Convert HTML to MediaWiki

(or just about any other format!) Pandoc is a very useful command-line program that converts an input file in just about any format to just about any different output format. Staying inside the same directory, simply run the following command:

 pandoc file.html -f html -t mediawiki -s -o file.txt

This command simply takes the HTML file and writes it in equivalent MediaWiki format to a txt file. I've provided some breakdown of the parameters for basic usage in case you need to convert to another format.

• -f The input format of the file.
• -t The format of the output file.
• -s Standalone adds a header and footer to the document, rather than producing a document fragment.
• -o The name of the output file.

For more info on Pandoc, read the user guide.

It is possible you may run into an error with Pandoc, presumably caused by your file being too large. I ran into this error and some fixes can be found here.

Optional: Clean Up Poor Encoding

Depending on your PDF encoding, you may find strange Unicode charecters in your HTML output. This step is intended to clean up this output to the best possible degree of accuracy. ftfy, stands for fixes text for you, and it's a Python library with a command-line interface. We'll be using the command line to clean our files. This step is preformed before using Pandoc.

To install ftfy:

git clone https://github.com/LuminosoInsight/python-ftfy.git
cd python-ftfy
sudo python setup.py install

Or, if you system has pip, pip install ftfy. Note that if you want to use a version of 5.0 (most recent available at time of writing) or later, you need Python 3. I used Python 2.x with ftfy 4.1.1 for this answer. Using the same directory, type the following command:

 ftfy -o file_clean.html --preserve-entities file.html

Optionally, you may include the --guess option to have ftfy guess your encoding, or --encoding if you know your encoding. This may produce better results.