Software to search for correlation in data

Question

I'm searching for a software where I can import a large amount of data (5000 columns, unknown number of rows, about 20gb) and the software gives me a estimate correlation (does not have to be very accurate). Google translate says this is has to be the "correlation coefficient" (german "korrelationskoeffizient").

Use case: I work in a (rough translation) "waste heat power plant" as a student employee and want to optimize some processes. The 5000 columns are measurements taken from sensors. I want to know what value(s) will rise if I change another. Of course, in the end, this will be checked by people who understand this topic better than me.

The operating system does not matter. Preferred is windows or mac (workplace) but linux is ok, too (private).

Answer 1

If your computer has enough RAM, you could completely read and process the file with R, optionally using the data.table package.

From your description, you have a long time series of sensor readings and would like to model the system to be able to predict behaviour. This is a very complex topic that I'm not familiar with, but R packages, like "forecast" exist for building such models.

For a start, it would be probably a good idea to cut a small part of the big 20Gb file and to analyse just that manageable portion. Doing such a cut is trivial with Unix/Linux tools in a terminal:

head -n 5000 bigfile.txt > first5000lines.txt

It is always a good idea to actually look at a fraction of the data in a spreadsheet software and you can reduce the number of columns by using:

cut -f 1:50 first5000lines.txt > 50colsfirst5000lines.txt

A lot of research and effort to understand data analysis will be required for the task you describe and I don't think that there is any kind of software able to automagically do the work.

R is cross-platform and versions for Windows, Mac and Linux exist.

Answer 2

Eureqa is designed for finding correlations (including non-linear multi-variable complicated ones). It does not work for big dataset hovewer, so you can use it on partial data.

You can try inserting limited set of data into the online version to see "low hanging fruit" correlations in your data.

Unfortunately, it is not FOSS.

Answer 3

I also suggest R as a starting point based on what I understand it to be designed to do. The downside may be that it requires you to work at a command-line interface.

A different approach is found with tools that are more UI based and more spread-sheet oriented. Origin is an example of perhaps one of the better tools to use when these two criteria are important for you. I cannot speak to the data size limit of Origin.

Finally, perhaps you want to design your own UI widgets around a self-styled data analysis package to put forward for future users to just "point, click, and get good-looking results". Igor Pro would be my recommendation in this case.