rough comparison of files

Question

I want to compare two editions of one book. The book is very long and the editions are close but not identical.

I have extracted the texts with OCR. The obtained text-files have mistakes and many things left from text formating. Now I want to make a rough comparison of the text-files --- I want to ignore most of the things, but I need to capture something like a sentence or at least new paragraph.

The file comparison (like meld) does not seem to be useful for this (ignoring white space does not help at all).

Any suggestion?

Answer 1

I think a good buzzword here would be "fuzzy diff".

Possible answer: https://github.com/google/diff-match-patch

online tool to evaluate: https://neil.fraser.name/software/diff_match_patch/demos/diff.html (activate efficiency cleanup)

Technical explanations of the algorithm: https://neil.fraser.name/writing/diff/

Other possibilities and related questions:

https://stackoverflow.com/questions/3108274/generate-fuzzy-difference-of-two-files-in-python-with-approximate-comparison

https://pypi.python.org/pypi/fuzzywuzzy

Looking at the motivation for the question it might also be possible to take some software which can compare two pdfs visually (or based on available ocr layer) such as https://github.com/zeliboba/DiffPDF-app newer versions are commercial: http://www.qtrac.eu/diffpdf.html

alternatives (really different in spite of similar names): https://github.com/JoshData/pdf-diff http://vslavik.github.io/diff-pdf/

I have personally only tried the DiffPDF-app and at least in my case it did not seem work 100% perfectly in my case.

Answer 2

I would use KDiff3 for such purpose.

Some benefits for your need:

1. Thanks to the summary of differences in the vertical columns, you can easily see where the differences are.
2. You can see within the same line of text if there are differences.
3. You can easily reload one or the other file (by clicking in its path field and hitting "Enter") if it was edited with another editor and the comparison will refresh automatically.

Some screenshots here.

I like using Kdiff3 to compare, edit with PSPad, and refresh in Kdiff3.