Honestly, my inclination for this kind of (relatively simple) text processing is to write a quick and dirty script. So here is a quick and dirty Powershell script you could modify to your needs. For 20 000 line files I think that the get-content
command should be okay but if your files are too big then you may need other optimizations (hashing lines as dictionary keys, using the .NET System.IO.StreamReader
class...). There are other ways to improve things as well.
# Create a file consisting of an original file with duplicate lines
# removed, preserving line order.
# Run this at your own risk!
# Paul "Worthless" Nijjar, 2016-03-29
param(
[parameter(Mandatory=$true)][string] $InFile,
[parameter(Mandatory=$true)][string] $OutFile,
[parameter()][switch] $ConfirmRemovals
)
if ($InFile -eq $OutFile) {
write-error("Input and Output files should be different: $OutFile")
}
# Store all unique lines in a dictionary!
$linemap = @{}
$lineNumber = 0
# I hope you didn't need OutFile!
echo $null > $OutFile
foreach ($currLine in (Get-Content $InFile)) {
$lineNumber++
$outline = "{0}:`"{1}`"" -f $lineNumber, $currLine
write-debug $outline
if ($currLine.trim() -eq "" ) {
# Line is blank. Preserve it.
$currLine >> $OutFile
} elseif (! $linemap.ContainsKey($currLine)) {
# Not a duplicate. Record in map and write to file.
$linemap.Set_Item($currLine, $lineNumber)
$currLine >> $OutFile
} else {
$outline = "Line {0} `"{1}`" is a duplicate of line {2}" -f $lineNumber, $currLine, $linemap.Get_Item($currLine)
write-host $outline
if ($ConfirmRemovals) {
$omitIt = read-host -prompt "Omit it? (y/n)"
if ($omitIt -ne "y") {
write-host "Not deleting."
$currLine >> $OutFile
} else {
write-host "Omitting by confirmation."
} # end if omitIt check
} else {
write-host "Omitting."
} # End if confirm removals
} # end if linemap contains key
} # end get-content
I saved the file as Select-NonDuplicates.ps1
.
Here is a sample input file (in.txt
):
cat
bat
Cat
cat
The above two WILL be duplicates!
Powershell is case insensitive for dictionary strings.
frogg
frog
Rabbit
Rabbit
The above will not be a duplicate.
Be careful of leading spaces. trim() can help here.
Rabbit
Blank lines will be preserved.
Monkey
Monkey
The above will not be a duplicate.
Be careful of trailing spaces, too.
frog
frog
Rabbit
Cat
Mountain goat
and here the corresponding output file (out.txt
):
cat
bat
The above two WILL be duplicates!
Powershell is case insensitive for dictionary strings.
frogg
frog
Rabbit
Rabbit
The above will not be a duplicate.
Be careful of leading spaces. trim() can help here.
Blank lines will be preserved.
Monkey
Monkey
Be careful of trailing spaces, too.
Mountain goat
I invoke this by opening a command prompt and starting powershell as:
powershell -executionpolicy remotesigned
and then at the prompt typing
& Select-NonDuplicates.ps1 -InFile in.txt -Outfile out.txt
To confirm removals, type:
& Select-NonDuplicates.ps1 -InFile in.txt -Outfile out.txt -ConfirmRemovals
(The usual disclaimers apply: I am a mediocre programmer and there are probably bugs, the code is not well tested and might ruin everything, I apologise if this level of detail comes across as patronizing, Windows costs more than a milkshake.)