0

So is there a software to union (not simple merge, that is duplicate lines will be removed) multiple text file one after another to a single big file?

I need it for a research purpose. No GUI required.

Thanks.

Improve this question
  • If I recall my database lectures correctly, that's more a join instead of a union – Thomas Weller Sep 2 '16 at 21:46
  • How does the accepted solution remove duplicate lines? – Thomas Weller Sep 4 '16 at 18:48
  • Following up on earlier remarks: your question is ambiguous, please edit it. Does that is refer to the 'union' or to the 'simple merge', i.e. do you want duplicate lines removed or not? It is not even clear if you want the result file sorted (and also, are your input files sorted? - that may make a difference for the possible results). It helps if you give an example of two input files + one output file. – user416 Sep 5 '16 at 11:21
1

You can use http://www.filesplitter.org as long as your files have the same name and extension .chunk001 etc

And then use Notepad++ to remove the duplicate lines, as in this answer https://stackoverflow.com/a/16293580/8157

Improve this answer
0

If you are not allergic to ports of unix tools, use http://unxutils.sourceforge.net/ and then do

sort.exe -u file1.txt file2.txt file3.txt > output

This will destroy the ordering, but with omitting duplicated lines is going to be slightly messy anyway.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.