Text formatter which understands grammer

Question

I have converted some PDF documents which contained JPEG into ASCII text.

Unfortunately, there are many
line
breaks in the middle of
sentences, leaving some lines with only a few words and the
text generally difficult to read.

Is there any
tool which will reformat the
text, taking into account sentences and
paragraphs
which might be broken up like
this. It should
recognize English sentence structure - that they generally
end with
period, question
mark full
stop, etc

Answer 1

You can do this in most editing software (word processors/text editors) that lets you search/replace paragraph marks or line breaks. Here's is an example strategy described in Word syntax where ^p is a paragraph break, assuming no distinction between new-line characters and page breaks (which is the case for text files), and in which I will use _ for space characters.

First insert extra paragraph marks after sentence punctuation:

• Replace . with .^p
• Replace ? with ?^p
• Replace ! with !^p

Remove leading and trailing spaces, keep repeating until no more found:

• Replace _^p with ^p
• Replace ^p_ with ^p

Reduce the number of consecutive paragraph breaks to two at the most. Keep repeating until no more found:

• Replace ^p^p^p with ^p^p

Additional steps if you want to stitch sentences together into one parapraph:

• Replace ^p^p with $$##$$ (or any string that is not in your document)
• Replace ^p with _
• Replace $$##$$ with ^p

Additional notes:

• Put all this in a keyboard macro if you need to do it in multiple documents.
• You may wish to reduce to one paragraph break only. Then you need a variation of this strategy.
• You can use regular expressions as well, but I'm not fluent with them.
• For that additional step your software must be able to handle very long lines, because you will have an intermediate result which is basically just one line.
• Do all this on copies of your files so that you have a backup.