1

I have a bunch of tables with some common columns but completely independent data. I want to create a single large table by appending the data from all of the tables. However, I only want the columns that are common between all of these tables.

One issue is that there are hundreds of columns and about a million rows each. So each table is around 1gb. I was running into memory issues when trying to manipulate the data in some software like Rapidminer.

Any suggestions about a software that could do this? Oh yes, and I run linux x64, so hopefully the software would run on that platform as well.

  • Please give an exact example – Jan Doggen Nov 1 '16 at 8:12
1

Take a look at Pandas:

  • Can handle large data sets well
  • Cross Platform - including Linux
  • Free, Gratis & Open Source

I would suggest coming up with a list of the "Common" columns first and then for each dataset loading it, deleting the unneeded columns and then merging.

  • 2
    Thanks steve. I ended up using a combo of just sql with some pandas and sqlalchemy. I was able to run an intersection and reduce on the column names between tables. So that worked out well. Thanks for the tip. – krishnab Nov 1 '16 at 16:23
  • 1
    In order to help others in future, pelase either accept Steve's answer, possibly editing it if necessary, or post and accept your own answer. – Mawg Dec 1 '16 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.