How to analyse and visualize complex data?

Question

I have a Python dataframe with roughly 50.000 rows and 30 columns.

This dataframe is the result of a computer simulation.

25 of the columns are input parameters of this computer simulation, the rest are output variables. I would like to identify the "best" parameters, but as the output variables are dependent from each other and also variable in their acceptable limits, I cannot define a compound cost function and have to do this task manually.

I would like to analyse this dataframe by visualizing the dependency of various input to output variables and started doing so by setting up different small matplotlib based code snippets that show the dependency between two variables. I could extend this to three variables to get a three dimensional plot, but still I would not get the big picture. I also sorted my dataframe and extracted individual variable sets, but still this does not help much for the big picture.

I am looking for a Python library or a set of (matplotlib) commands that help me to find dependencies between variables more easily.

Alternatively, I could image to load my dataframe in a sql database to use any packages or visualisation tools there.

Answer 1

if your columns are strictly numeric, and ideally non-null, you can try Principal Components Analysis, PCA. That's one of the oldest techniques of multiimensional analysis.

Performing the PCA can be done in a single line of code. After all, it's a variant of a matrix factorization. If I remember correctly.

However, the interpretation of the components is kind of an art and requires some experience.

What such an interpretation looks like can be seen here in one of David Robinson's screencasts of an analysis of Bob Ross paintings. Bob Ross was a TV celebrity in the 1980s. He has drawn many, many landscapes paintings in his TV show that ran for decades.

Here is a video of a PCA analysis of what motifs go togehter in these paintings. FOr instance, the first Principal Component represents "River" vs "SNow", they tend to NOT go together but each of them is related to other features, e.g. "Mountain" with "Conifer", and "Ocean" with "waves".

https://youtu.be/sD993H5FBIY?t=2669

I'm a big fan of D. Robinson who is a well known package author in the R community.

Answer 2

First, how do you plot single 50000-long vectors ? What are they -- time series, geographic maps ... ? datashader may help.

Once you can plot single 50k vectors, regress b = Ax + res (A 50k x 25, x 25 x 5, b Ax and res 50k x 5) and plot 3 x 5 panels with rows b Ax res, columns b0 .. b4. Sort x on row and columns, plot it as a heatmap.

Looking at pairs or triples of 5 vecs: there are only 10 pairs AB AC .. DE of 5 objects, so 10 subplots are easy, might show up strongly or weakly correlated pairs. Their complements are the 10 triples CDE BDE .. ABC: if AB is weak, CDE might be strong, or the other way around.

Could you point to data similar to yours on the web ?

(Not your question, but it's important to clean and scale the raw data early. Scaling: there are lots of ways, log, signed power, histogram equalize ... Tiny SVD singular values: rescale those columns, or drop them. Outliers: ... )

Links: seaborn regression, seaborn plotting-pairwise-data-relationships

stats stack might be a better place for this question.

Answer 3

One way would be to plot the real part vs the imaginary part of the data. There is probably no dependence but you may find the real part is a function of the imaginary part (maybe linearly). And also the PCA as sussested by @knb.