15

I want to detect sentences, long phrases, and possibly paragraphs that have been repeated in a document. I've been working on a document, and want to make sure I haven't copied the same or similar text into more than one spot.

Ideally, the application should be available online, or easily installable on an OS X Mavericks computer with Pages but not Microsoft Word installed.

I came across Pro Writing Aid, but its "Repeat Words & Phrases" seems to me very noisy - some individual words are highlighted by it merely for occurring a fair bit.

I have also seen Online-Utility.org's Text Analyzer, which isn't bad. However, there is some redundancy in its information. If there's a seven word phrase that's repeated twice, it also mentions the two six word phrases consisting of words one to six and two to seven as occurring twice. Also, it's hard to visualise the results, and see if there are particular sections that have a high amount of duplicated text.

This question is different from Program to search for word repetitions in text documents and Program to search for word repetitions in Word documents in that they're asking about one word being repeated by the following word.

  • If your document is or can be simplified to a text file, you can probably use the regular expressions search in Sublime Text (or alternative text editors). Regex can easily detect duplicate words and lines; with a bit of work, you might be able to detect duplicate phrases and sentences. – Huey Apr 14 '15 at 13:19
  • This is a very good one and I am not a very good at grammar and this should help me to ehance my writing .... – PROBERT May 23 '16 at 14:43
  • There are many diff tools – paparazzo Dec 27 '16 at 2:37
  • You might want to try the Levenshstein distance – Niklas Rosencrantz Mar 12 '17 at 5:23
1

Maybe you can try this code:

https://github.com/raypereda/repeating-phrases (I'm not the developer of "repeating-phrases")

Other option is to try some shell script (macOS, linux).

cat ./THEFILE.txt | tr '\n' ' ' | tr -d '\b\r' | tr -s ' ' | gsed 's/\([.!?]\) \(.\)/\1\n\2/g' | grep -v -e '^$' | sort | uniq -c | sort -nr | head -5

Note: using "gsed" because of macOS. Explanation:

tr '\n' ' ' -> strip "\n" (merge lines)
tr -d '\b\r' -> remove control chars if exists
tr -s ' ' -> multiple spaces in sequence became just one space
gsed 's/\([.!?]\) \(.\)/\1\n\2/g' -> one phrase in each line (punctuation based)
grep -v -e '^$' -> strip blank lines
sort | uniq -c | sort -nr | head -20 -> statistics

Both solutions assume plain text files for input.

0

Textanz can do that (find and highlight repeated text blocks in a document).

http://www.textanz.com/index.php

https://i.imgur.com/1giwJZ1.jpg

protected by Community Oct 15 at 20:46

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.