I have a text file containing 20K URL's. Within the file, 1-2 blank lines are present between many of the URL's. I need to find all duplicate URL's within the file and remove them.
The order of each URL within the file is important, and cannot be changed. Also, single blank lines must be maintained. (Multiple consecutive blank lines can be maintained or converted into a single blank line.)
It doesn't matter which occurrence of the duplicate URL is kept, but I would prefer to keep the first occurrence.
What software can help me accomplish this task?
The software must be gratis and work in Windows. Open-source is preferred, but definitely not required.