3

I have a large text file with over 20,000 lines. The order of the lines is important. I want to remove all duplicate non-blank lines in the file, without disturbing the order. What software can accomplish this task?

If possible, I would like to manually confirm/deny the deletion of each duplicate line.

For this task:

  • I prefer software. This is not a requirement, but the software must cost no more than a milkshake.
  • I prefer software (no installer), but this is not a requirement.
2

3 Answers 3

1

You can try this: Remove Duplicate Lines. It's an online app that will delete duplicate text lines.

However, it will not allow you to manually confirm/deny the deletion of each duplicate line.

1

Honestly, my inclination for this kind of (relatively simple) text processing is to write a quick and dirty script. So here is a quick and dirty Powershell script you could modify to your needs. For 20 000 line files I think that the get-content command should be okay but if your files are too big then you may need other optimizations (hashing lines as dictionary keys, using the .NET System.IO.StreamReader class...). There are other ways to improve things as well.

# Create a file consisting of an original file with duplicate lines 
# removed, preserving line order.
# Run this at your own risk!

# Paul "Worthless" Nijjar, 2016-03-29

param(
   [parameter(Mandatory=$true)][string] $InFile,

   [parameter(Mandatory=$true)][string] $OutFile,

   [parameter()][switch] $ConfirmRemovals
)

if ($InFile -eq $OutFile) { 
    write-error("Input and Output files should be different: $OutFile")
}

# Store all unique lines in a dictionary!
$linemap = @{}
$lineNumber = 0

# I hope you didn't need OutFile!
echo $null > $OutFile

foreach ($currLine in (Get-Content $InFile)) {
    $lineNumber++
    $outline =  "{0}:`"{1}`"" -f $lineNumber, $currLine
    write-debug $outline
    if ($currLine.trim() -eq "" ) { 
        # Line is blank. Preserve it. 
        $currLine >> $OutFile
    } elseif (! $linemap.ContainsKey($currLine)) { 
        # Not a duplicate. Record in map and write to file.
        $linemap.Set_Item($currLine, $lineNumber)
        $currLine >> $OutFile
    } else { 
        $outline = "Line {0} `"{1}`" is a duplicate of line {2}" -f $lineNumber, $currLine, $linemap.Get_Item($currLine)
        write-host $outline

        if ($ConfirmRemovals) { 
            $omitIt = read-host -prompt "Omit it? (y/n)"
            if ($omitIt -ne "y") { 
                write-host "Not deleting."
                $currLine >> $OutFile
            } else { 
                write-host "Omitting by confirmation."

            } # end if omitIt check
        } else { 
            write-host "Omitting."
        } # End if confirm removals

    } # end if linemap contains key

} # end get-content 

I saved the file as Select-NonDuplicates.ps1 .

Here is a sample input file (in.txt):

cat
bat
Cat
cat
The above two WILL be duplicates!
Powershell is case insensitive for dictionary strings.
frogg
frog
Rabbit
 Rabbit
The above will not be a duplicate.
Be careful of leading spaces. trim() can help here.
 Rabbit

Blank lines will be preserved.


Monkey
Monkey  
The above will not be a duplicate.
Be careful of trailing spaces, too.
frog
frog
Rabbit
Cat
Mountain goat

and here the corresponding output file (out.txt):

cat
bat
The above two WILL be duplicates!
Powershell is case insensitive for dictionary strings.
frogg
frog
Rabbit
 Rabbit
The above will not be a duplicate.
Be careful of leading spaces. trim() can help here.

Blank lines will be preserved.


Monkey
Monkey  
Be careful of trailing spaces, too.
Mountain goat

I invoke this by opening a command prompt and starting powershell as:

powershell -executionpolicy remotesigned

and then at the prompt typing

& Select-NonDuplicates.ps1 -InFile in.txt -Outfile out.txt 

To confirm removals, type:

& Select-NonDuplicates.ps1 -InFile in.txt -Outfile out.txt -ConfirmRemovals

(The usual disclaimers apply: I am a mediocre programmer and there are probably bugs, the code is not well tested and might ruin everything, I apologise if this level of detail comes across as patronizing, Windows costs more than a milkshake.)

1
  • Thanks Paul! Nothing patronizing at all! Since powershell is free with Windows, and I've already paid for it, the overall financial cost is zero. I'll take a look at what you've created as soon as I can. It might be perfect! :-) Mar 29, 2016 at 4:52
1

AkelPad and Geany can do that. Both handle 20,000 lines easily.

AkelPad: Options → Plug-ins... → Format::LineRemoveDuplicates → Call

It won't be interactive, but you can see the lines to be removed by using Format::LineGetDuplicates

The installer comes with plug-ins, but if you are going to use the portable version, make sure to install Format plug-in.

Geany:
Tools → Plugin Manager → enable Line Operations → Close
Tools → Line Operations → Remove Duplicate Lines, Ordered

3
  • Thank you very much! I gave AkelPad a try, and it was very fast. Unfortunately, it also wants to remove about 500 blank lines interspersed throughout the file. Is it possible to ignore the blank lines in AkelPad? Feb 3, 2017 at 7:58
  • @RockPaperLizard, not that I know of. I totally missed that requirement, sorry. It should be doable via Scripts plug-in, you may want to ask on the forum.
    – user
    Feb 3, 2017 at 15:26
  • No problem at all. Thank you for the recommendations! Feb 4, 2017 at 7:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.