Software to analyze files and find similar strings

Question

I'm looking for a software that, given several text files (a few thousands actually), it will find all the occurrences of strings that are exactly the same without knowing the string in advance (it probably needs to set up a string length to analyze).

For example, let's say I have

file1.txt

1234567890 hello test

Lorem ipsum dolor sit amet

file2.txt

hello test

file3.txt

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut faucibus turpis urna. Donec vel tincidunt enim. Phasellus blandit massa at dolor faucibus ultricies.

And then I somehow "ask" this software to "find all the occurrencies of equal strings that are 10 characters long" it will tell me that file1 and file share the "hello test" string while file1 and file3 have the "Lorem ipsu" string in common.
I then ask for equal strings of length 20, it will show me that file1 and file3 share the string "Lorem ipsum dolor si" while file2 has no string of such length.
Mind that the files might be completely different except for one string, so using a diff tool on all the files to find those that are most similar will not work

Extra but not required:

1) Be able to specify the length in words rather than characters (so that i can specify "find the strings that have 6 words in common"

2) Be smart enough to even find similar strings (much like some diff tools), that is, strings that might differ for a word or two, or maybe they have some non-shared words in the middle of a string. (like, for example "Lorem ipsum test 123 test dolor sit amet" and "Lorem ipsum dolor sit amet" will be matched)

I'm looking for something that can be used on windows, but i have installed bash on this system so i can run linux command line tools if required.

I'm also open to paid solutions if they don't cost too much (no more than 40€) and do not require a recurring subscription

Thanks in advance

EDIT: I have found out that this kind of software is used to check plagiarism on papers, research documents and similar. I have found Plagiarism Checker x https://plagiarismcheckerx.com/ Which does what I need (using the cross comparison) and costs about 40 dollars, i bought it but when i tried to add all my documents (4k, and it lets you add about 150 at a time) it crashed when I was almost done adding the documents. Then i tried to analyze 150 documents and it took almost half a day. But it did what I required, except not as efficiently as I thought. I'm not posting this as answer because it didn't complete the task I need to do

Answer 1

I don't know of such a program, but it sounds like a relatively simple program to write, if you're looking for equal strings.

For each file:

1. Find all strings in that file matching your criteria.
2. Put them in a dictionary. The key is the string, the value is a set of filenames where that string was found. Use a set so that each filename is present at most once.

Loop over the keys of the cumulative dictionary, looking for those entries which have more than one filename associated with them.

To handle "similar" strings, convert each string to a "canonical" form before inserting it in the dictionary. For example, to ignore each string's case, convert each string to lowercase before inserting it in the dictionary.

It should take only a few seconds to process each group of files, about the same amount of time as to read the files, depending on how expensive it is to locate all the strings in your files and convert them to "canonical" form.

Answer 2

If you don't find an out-of-the-box solution (and I doubt you will), you might be able to build your own using Apache Lucene (https://lucene.apache.org/) with a custom tokenizer for wildcard searches.