I have several text files that have partially intersecting contents. I want to combine them in the manner to have all contents in one file, but have as little duplicates as possible. This tool not has to process all files at once, I'll be satisfied with processing file pairs one by one.

The main problem is when I'm using tools like Compare++ or SmartSynchronize and kdiff3 they are trying to overwrite some blocks in one file with blocks from other, while other times, just add blocks, as needed. Most of times they even correctly find place where text must be inserted.

Thought files are large, I'm even ready to merge them in manual mode, but I lack the function that allows me to add the block from, say, right panel to left panel instead of replacing it. Some blocks "recommended" for replace make some sense, others seem random.

I don't want to replace one string to another when they are completely different, I want to have both those strings in resulting file!

For sure, the tool must be as 'smart' as possible in detecting places for insertion. Because lines of text may be arranged arbitrary, in the next case




the tool must not treat A from right panel as diff and must clearly understand, that it is the same A as in left panel.

Platform is Windows.

Is there good tool to find (almost) duplicate blocks inside a single file? Like having some "fuzziness" preset, allowing blocks to differ by N symbols.

  • KDiff3 does have an option to insert text from a then from b at each diff but the problem will be when it finds two paragraphs only differ on one line it will suggest the difference/merge at line level rather than paragraph. You could possibly get around this by using the input filters to remove line feeds within paragraphs and then re-wrapping later. Jul 1 '15 at 5:15
  • @SteveBarnes yeah, it also "shuffles" strings in strange way (it looks like it for me). and it cant resolve a problem with blocks of text swapped in files (my example with AB BC CA). seems like i will have to take my 3-4 hours and do everything semi-manualy. just gotto develop methodology. also, pls see my edit Jul 1 '15 at 6:09
  • 1
    maybe WinMerge could be interesting for you... Jul 2 '15 at 6:21

It's a two step process. First combine all text files into one. Upload the file into a tool like http://text-analyser.com for Natural Language Processing (NLP). The tool will parse all the sentences into rows. From within the text-analyzer tool click the Export button. That generates an Excel file. Then remove the other columns in the Excel file so that you isolate the column of sentences and then save that as .TAB separated file. Upload that file into http://text-analyser.com and click the Remove Duplicates option. During the import the tool will remove all duplicate sentences. From the File tab click the File Name in the Html table...that downloaded file is a clean/duplicate free file of sentences! Hope this helps someone looking for a solution to this problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.