I have a bunch of tables with some common columns but completely independent data. I want to create a single large table by appending the data from all of the tables. However, I only want the columns that are common between all of these tables.
One issue is that there are hundreds of columns and about a million rows each. So each table is around 1gb. I was running into memory issues when trying to manipulate the data in some software like Rapidminer.
Any suggestions about a software that could do this? Oh yes, and I run linux x64, so hopefully the software would run on that platform as well.