Text processing app to delete duplicate lines in text file

Question

I have a large text file with over 20,000 lines. The order of the lines is important. I want to remove all duplicate non-blank lines in the file, without disturbing the order. What windows software can accomplish this task?

If possible, I would like to manually confirm/deny the deletion of each duplicate line.

For this task:

• I prefer gratis software. This is not a requirement, but the software must cost no more than a milkshake.
• I prefer portable software (no installer), but this is not a requirement.

Answer 1

You can try this: Remove Duplicate Lines. It's an online app that will delete duplicate text lines.

However, it will not allow you to manually confirm/deny the deletion of each duplicate line.

Answer 2

Honestly, my inclination for this kind of (relatively simple) text processing is to write a quick and dirty script. So here is a quick and dirty Powershell script you could modify to your needs. For 20 000 line files I think that the get-content command should be okay but if your files are too big then you may need other optimizations (hashing lines as dictionary keys, using the .NET System.IO.StreamReader class...). There are other ways to improve things as well.

# Create a file consisting of an original file with duplicate lines 
# removed, preserving line order.
# Run this at your own risk!

# Paul "Worthless" Nijjar, 2016-03-29

param(
   [parameter(Mandatory=$true)][string] $InFile,

   [parameter(Mandatory=$true)][string] $OutFile,

   [parameter()][switch] $ConfirmRemovals
)

if ($InFile -eq $OutFile) { 
    write-error("Input and Output files should be different: $OutFile")
}

# Store all unique lines in a dictionary!
$linemap = @{}
$lineNumber = 0

# I hope you didn't need OutFile!
echo $null > $OutFile

foreach ($currLine in (Get-Content $InFile)) {
    $lineNumber++
    $outline =  "{0}:`"{1}`"" -f $lineNumber, $currLine
    write-debug $outline
    if ($currLine.trim() -eq "" ) { 
        # Line is blank. Preserve it. 
        $currLine >> $OutFile
    } elseif (! $linemap.ContainsKey($currLine)) { 
        # Not a duplicate. Record in map and write to file.
        $linemap.Set_Item($currLine, $lineNumber)
        $currLine >> $OutFile
    } else { 
        $outline = "Line {0} `"{1}`" is a duplicate of line {2}" -f $lineNumber, $currLine, $linemap.Get_Item($currLine)
        write-host $outline

        if ($ConfirmRemovals) { 
            $omitIt = read-host -prompt "Omit it? (y/n)"
            if ($omitIt -ne "y") { 
                write-host "Not deleting."
                $currLine >> $OutFile
            } else { 
                write-host "Omitting by confirmation."

            } # end if omitIt check
        } else { 
            write-host "Omitting."
        } # End if confirm removals

    } # end if linemap contains key

} # end get-content 

I saved the file as Select-NonDuplicates.ps1 .

Here is a sample input file (in.txt):

cat
bat
Cat
cat
The above two WILL be duplicates!
Powershell is case insensitive for dictionary strings.
frogg
frog
Rabbit
 Rabbit
The above will not be a duplicate.
Be careful of leading spaces. trim() can help here.
 Rabbit

Blank lines will be preserved.


Monkey
Monkey  
The above will not be a duplicate.
Be careful of trailing spaces, too.
frog
frog
Rabbit
Cat
Mountain goat

and here the corresponding output file (out.txt):

cat
bat
The above two WILL be duplicates!
Powershell is case insensitive for dictionary strings.
frogg
frog
Rabbit
 Rabbit
The above will not be a duplicate.
Be careful of leading spaces. trim() can help here.

Blank lines will be preserved.


Monkey
Monkey  
Be careful of trailing spaces, too.
Mountain goat

I invoke this by opening a command prompt and starting powershell as:

powershell -executionpolicy remotesigned

and then at the prompt typing

& Select-NonDuplicates.ps1 -InFile in.txt -Outfile out.txt 

To confirm removals, type:

& Select-NonDuplicates.ps1 -InFile in.txt -Outfile out.txt -ConfirmRemovals

(The usual disclaimers apply: I am a mediocre programmer and there are probably bugs, the code is not well tested and might ruin everything, I apologise if this level of detail comes across as patronizing, Windows costs more than a milkshake.)

Answer 3

AkelPad and Geany can do that. Both handle 20,000 lines easily.

AkelPad: Options → Plug-ins... → Format::LineRemoveDuplicates → Call

It won't be interactive, but you can see the lines to be removed by using Format::LineGetDuplicates

^{The installer comes with plug-ins, but if you are going to use the portable version, make sure to install Format plug-in.}

Geany:
Tools → Plugin Manager → enable Line Operations → Close
Tools → Line Operations → Remove Duplicate Lines, Ordered