I would like a tool that takes input text and returns which sentences are duplicates and very similar (e.g., by informational distance as in Levenshtein distance). For example, if two sentences contain > 80 % same word (fuzzy) then find them.
The best way I found was: Replace all dots (sentence terminators) with a newline. Then sort the file with sort and output it. It works but only for exact sentences.
In R there is a tokenizers package which you can use to grab sentences from a body of text. Documentation.
However I have tried it, and sometimes it gets things wrong. In particular there it assigns a sentence break after some Abbreviation ("J. Smith of Univ. Cambridge says..."), and converts the text to two (or even three) sentences when there should be only one.
Sometimes (depending on whitespace co-occurrence?) is mistreats decimal separators: it thinks there is a sentence break in the middle of Pi (3.141), or any floating point number.
I am sure there are similar libraries available for other languages but I assume these have similar problems as tokenizers.
I haven't studied the parsed output for similarity, except by sorting and visual inspection, as you did. You can however compare sentences by line length, wordcount, tokenize them into words, and compare the words on a per-sentence basis... This is a creative data-science / natual-language-processing task.
