Software to search for duplicate lines within a text file

Question

I have a text file containing 20K URL's. Within the file, 1-2 blank lines are present between many of the URL's. I need to find all duplicate URL's within the file and remove them.

The order of each URL within the file is important, and cannot be changed. Also, single blank lines must be maintained. (Multiple consecutive blank lines can be maintained or converted into a single blank line.)

It doesn't matter which occurrence of the duplicate URL is kept, but I would prefer to keep the first occurrence.

What software can help me accomplish this task?

The software must be gratis and work in Windows. Open-source is preferred, but definitely not required.

Answer 1

CudaText editor is almost what you need. It has plugin Sort which is called by menu item "Plugins / Sort", there you see several menu items, including "Remove duplicate lines".

"Almost what you need" because "Remove duplicate lines" removes also duplicate blank lines. Try to ask at support forum, maybe this can be changed, ie blanks can be left.

Plugin also gives dialog (Plugins / Sort / Sort dialog) to do the same:

Answer 2

On macOS, the free version of BBEdit (also available as TextWrangler) has a built-in function to replace duplicate lines, which I use quite often. Unfortunately, the product is not available for Windows.

You might already have Microsoft Excel, which has a similar three-click function, but it's not free software.

I found this article explaining how to write a small Google Apps Script to add the same functionality in Google Sheets. Maybe that will work for you?

/**
 * Removes duplicate rows from the current sheet.
 */
function removeDuplicates() {
  var sheet = SpreadsheetApp.getActiveSheet();
  var data = sheet.getDataRange().getValues();
  var newData = [];
  for (var i in data) {
    var row = data[i];
    var duplicate = false;
    for (var j in newData) {
      if (row.join() == newData[j].join()) {
        duplicate = true;
      }
    }
    if (!duplicate) {
      newData.push(row);
    }
  }
  sheet.clearContents();
  sheet.getRange(1, 1, newData.length, newData[0].length).setValues(newData);
}

_{(source: Romain Vialard on Google Apps)}