0

I am seeking recommendations for a Windows tool to find and count duplicate lines in a text file containing 500,000-999,999 lines.

The tool can be a CLI or GUI.

If the duplicate lines are consecutive or non-consecutive, they need to be counted as duplicates.

For example, if we have the following text file:

I am a lizard.
I am a dog.
I am a cat.
I am a bird.
I am a lizard.
I am a lizard.
I am a dog.
I am a cat.
I am a dog.
I am a lizard.

The output will look something like:

I am a lizard. (4)
I am a dog. (3)
I am a cat. (2)

The exact formatting of the output is not important, as long as I can determine the duplicates (and number of each) from it.

Requirements:

  • Gratis.
  • Windows-compatibility.

Bonus Features (none required):

  • Cross-compatibility with Linux.
  • Portable software (no installer) is a big plus.
  • Ability to set the minimum number of occurrences lines to be classified as a duplicate.
  • Ability to only find strings matching a regex (Regular Expression). This is not critical, as I can pre-process each file with a regex tool.
  • The software only needs to handle a single file at a time. Although I have multiple files to process, I don't currently need to find duplicates across files. If the tool can handle multiple files as inputs, that's a bonus, but not required.
2

3 Answers 3

0

I do believe the crisp text editor can do this. https://crisp.com Most comprehensive text editor I have come across.

Not free (commercial code) but has windows and linux, and I believe mac versions.

I think that if you found an editor that could read your file that had a comprehensive (multi-column) sort function that you could sort your big file and then manually count up the dups. That is ok if you have one big file but if you have to do alot of separate files that could get tedious. Might be ok for a one-time no cost deal.

0

You can do this with any Linux terminal (or WSL in Windows):

Basic counts:

user@host:~$ cat tmp.txt | sort | uniq -c | sort -r

      4 I am a lizard.
      3 I am a dog.
      2 I am a cat.
      1 I am a bird.

Match lines with a regexp:

user@host:~$ cat tmp.txt | egrep '(lizard|dog)' | sort | uniq -c | sort -r

      4 I am a lizard.
      3 I am a dog.

Input multiple files:

user@host:~$ cp tmp.txt tmp2.txt
user@host:~$ cat tmp.txt tmp2.txt | egrep '(lizard|dog)' | sort | uniq -c | sort -r

      8 I am a lizard.
      6 I am a dog.
0

This free website looks like it does what you're looking for: https://www.somacon.com/p568.php

It's not technically downloadable software but it will of course work on Linux too.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.