So is there a software to
union (not simple merge, that is duplicate lines will be removed) multiple text file one after another to a single big file?
I need it for a research purpose. No GUI required.
If you are not allergic to ports of unix tools, use http://unxutils.sourceforge.net/ and then do
sort.exe -u file1.txt file2.txt file3.txt > output
This will destroy the ordering, but with omitting duplicated lines is going to be slightly messy anyway.