1

I have a text file containing 20K URL's. Within the file, 1-2 blank lines are present between many of the URL's. I need to find all duplicate URL's within the file and remove them.

The order of each URL within the file is important, and cannot be changed. Also, single blank lines must be maintained. (Multiple consecutive blank lines can be maintained or converted into a single blank line.)

It doesn't matter which occurrence of the duplicate URL is kept, but I would prefer to keep the first occurrence.

What software can help me accomplish this task?

The software must be gratis and work in Windows. Open-source is preferred, but definitely not required.

0

On macOS, the free version of BBEdit (also available as TextWrangler) has a built-in function to replace duplicate lines, which I use quite often. Unfortunately, the product is not available for Windows.

You might already have Microsoft Excel, which has a similar three-click function, but it's not free software.

I found this article explaining how to write a small Google Apps Script to add the same functionality in Google Sheets. Maybe that will work for you?

/**
 * Removes duplicate rows from the current sheet.
 */
function removeDuplicates() {
  var sheet = SpreadsheetApp.getActiveSheet();
  var data = sheet.getDataRange().getValues();
  var newData = [];
  for (var i in data) {
    var row = data[i];
    var duplicate = false;
    for (var j in newData) {
      if (row.join() == newData[j].join()) {
        duplicate = true;
      }
    }
    if (!duplicate) {
      newData.push(row);
    }
  }
  sheet.clearContents();
  sheet.getRange(1, 1, newData.length, newData[0].length).setValues(newData);
}

(source: Romain Vialard on Google Apps)

  • Thanks! Your post eventually led me to sortmylist.com, which then reminded me of textmechanic.com, which I believe has the functionality I need. Unfortunately, it's online (as opposed to an offline application), but I'm looking at the code, and I think it may work correctly offline. – RockPaperLizard Apr 5 at 8:33
  • Well, TextMechanic has the exact functionality I need, but so far, I haven't been able to get it to work offline. I also discovered it has a limit to the number of times you can use the service per hour. I'll take a look at some of the solutions you posted. BTW, I'm about to update my question to include a couple additional relevant details. – RockPaperLizard Apr 5 at 8:48
  • Well, none of the recommendations you provided will work for me right now, but a big thumbs up for the effort... sincere thanks for trying to help me. Hopefully a recommendation meeting all the criteria will be posted. – RockPaperLizard Apr 6 at 7:12
  • You're welcome. It's a relatively straightforward task; in your case I'd probably given up after 30 minutes of searching and spend the next 15 minutes to write a console program in C# or Java doing the task for me. (I realize this might not be an option for other users having the same problem - also my naive solution might fail for very large files.) – Glorfindel Apr 6 at 8:23
0

CudaText editor is almost what you need. It has plugin Sort which is called by menu item "Plugins / Sort", there you see several menu items, including "Remove duplicate lines".

"Almost what you need" because "Remove duplicate lines" removes also duplicate blank lines. Try to ask at support forum, maybe this can be changed, ie blanks can be left.

Plugin also gives dialog (Plugins / Sort / Sort dialog) to do the same:

enter image description here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.