Software to find & count duplicate strings in a text file

Question

I am seeking recommendations for a Windows tool to find and count duplicate lines in a text file containing 500,000-999,999 lines.

The tool can be a CLI or GUI.

If the duplicate lines are consecutive or non-consecutive, they need to be counted as duplicates.

For example, if we have the following text file:

I am a lizard.
I am a dog.
I am a cat.
I am a bird.
I am a lizard.
I am a lizard.
I am a dog.
I am a cat.
I am a dog.
I am a lizard.

The output will look something like:

I am a lizard. (4)
I am a dog. (3)
I am a cat. (2)

The exact formatting of the output is not important, as long as I can determine the duplicates (and number of each) from it.

Requirements:

• Gratis.
• Windows-compatibility.

Bonus Features (none required):

• Cross-compatibility with Linux.
• Portable software (no installer) is a big plus.
• Ability to set the minimum number of occurrences lines to be classified as a duplicate.
• Ability to only find strings matching a regex (Regular Expression). This is not critical, as I can pre-process each file with a regex tool.
• The software only needs to handle a single file at a time. Although I have multiple files to process, I don't currently need to find duplicates across files. If the tool can handle multiple files as inputs, that's a bonus, but not required.

Answer 1

You can do this with any Linux terminal (or WSL in Windows):

Basic counts:

user@host:~$ cat tmp.txt | sort | uniq -c | sort -r

      4 I am a lizard.
      3 I am a dog.
      2 I am a cat.
      1 I am a bird.

Match lines with a regexp:

user@host:~$ cat tmp.txt | egrep '(lizard|dog)' | sort | uniq -c | sort -r

      4 I am a lizard.
      3 I am a dog.

Input multiple files:

user@host:~$ cp tmp.txt tmp2.txt
user@host:~$ cat tmp.txt tmp2.txt | egrep '(lizard|dog)' | sort | uniq -c | sort -r

      8 I am a lizard.
      6 I am a dog.

Answer 2

This free website looks like it does what you're looking for: https://www.somacon.com/p568.php

It's not technically downloadable software but it will of course work on Linux too.

Answer 3

I do believe the crisp text editor can do this. https://crisp.com Most comprehensive text editor I have come across.

Not free (commercial code) but has windows and linux, and I believe mac versions.

I think that if you found an editor that could read your file that had a comprehensive (multi-column) sort function that you could sort your big file and then manually count up the dups. That is ok if you have one big file but if you have to do alot of separate files that could get tedious. Might be ok for a one-time no cost deal.