Do we have or is there a program/software that has regex and deletes duplicate lines of text file that has 1 or more GB?

I have Emeditor but when I have a file that has 80 million lines and I want it to delete duplicate lines in that file, I think it goes more than 5 movements or 2 or more days before it deletes all duplicate lines.

I want to know if there's a software or program that takes probably less than 2 or 3 days before it deletes all duplicate lines.

  • 1
    What part does the regex play? Are you talking about something like "duplicate except for timestamps"? – CPerkins Mar 6 '17 at 15:35
  • 1
    if I understood, I wouldn't have asked. Do you expect to give the regex to the program? What do you want the program to do with the regex? Are you really looking for exact duplicate lines? If so, what part does the regex play in what the program does? – CPerkins Mar 7 '17 at 13:42
  • 1
    The reason the role of the regex matters is that without it, the problem is trivially easy: I had a few spare minutes while I was waiting for something, so I wrote a quick utility to make files unique by whole lines, preserving original order. Took me around 20 minutes to write (because I'm fussy and like flexibility). It takes right around 2 minutes to process a file I created with 80,636,920 lines (of which 116058 lines are unique - files with more unique lines will take a bit longer, limited by file io). But you mentioned regexes, so I'm trying to find out what it is you need. – CPerkins Mar 7 '17 at 15:00
  • 1
    If I now understand you, the regex has nothing with the program you seek. You just want a tool to make a file of 80 million lines unique that's reasonably fast (minutes rather than days), preserving order. Is that correct? – CPerkins Mar 7 '17 at 21:46
  • 1
    Indeed, I wrote a simple one yesterday. I'll publish it to my github+bitbucket and update you. On my machine, test data like yours (80 million lines of which 70 million are unique, average of around 70 chars per line) take around half an hour to run, even without investigating higher-performant collections. That's better than your couple of days. – CPerkins Mar 8 '17 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.