I am seeking recommendations for a Windows tool to find and count duplicate lines in a text file containing 500,000-999,999 lines.
The tool can be a CLI or GUI.
If the duplicate lines are consecutive or non-consecutive, they need to be counted as duplicates.
For example, if we have the following text file:
I am a lizard. I am a dog. I am a cat. I am a bird. I am a lizard. I am a lizard. I am a dog. I am a cat. I am a dog. I am a lizard.
The output will look something like:
I am a lizard. (4) I am a dog. (3) I am a cat. (2)
The exact formatting of the output is not important, as long as I can determine the duplicates (and number of each) from it.
Bonus Features (none required):
- Cross-compatibility with Linux.
- Portable software (no installer) is a big plus.
- Ability to set the minimum number of occurrences lines to be classified as a duplicate.
- Ability to only find strings matching a regex (Regular Expression). This is not critical, as I can pre-process each file with a regex tool.
- The software only needs to handle a single file at a time. Although I have multiple files to process, I don't currently need to find duplicates across files. If the tool can handle multiple files as inputs, that's a bonus, but not required.