2

I have a Python dataframe with roughly 50.000 rows and 30 columns.

This dataframe is the result of a computer simulation.

25 of the columns are input parameters of this computer simulation, the rest are output variables. I would like to identify the "best" parameters, but as the output variables are dependent from each other and also variable in their acceptable limits, I cannot define a compound cost function and have to do this task manually.

I would like to analyse this dataframe by visualizing the dependency of various input to output variables and started doing so by setting up different small matplotlib based code snippets that show the dependency between two variables. I could extend this to three variables to get a three dimensional plot, but still I would not get the big picture. I also sorted my dataframe and extracted individual variable sets, but still this does not help much for the big picture.

I am looking for a Python library or a set of (matplotlib) commands that help me to find dependencies between variables more easily.

Alternatively, I could image to load my dataframe in a sql database to use any packages or visualisation tools there.

0
1

if your columns are strictly numeric, and ideally non-null, you can try Principal Components Analysis, PCA. That's one of the oldest techniques of multiimensional analysis.

Performing the PCA can be done in a single line of code. After all, it's a variant of a matrix factorization. If I remember correctly.

However, the interpretation of the components is kind of an art and requires some experience.

What such an interpretation looks like can be seen here in one of David Robinson's screencasts of an analysis of Bob Ross paintings. Bob Ross was a TV celebrity in the 1980s. He has drawn many, many landscapes paintings in his TV show that ran for decades.

Here is a video of a PCA analysis of what motifs go togehter in these paintings. FOr instance, the first Principal Component represents "River" vs "SNow", they tend to NOT go together but each of them is related to other features, e.g. "Mountain" with "Conifer", and "Ocean" with "waves".

https://youtu.be/sD993H5FBIY?t=2669

I'm a big fan of D. Robinson who is a well known package author in the R community.

0

One way would be to plot the real part vs the imaginary part of the data. There is probably no dependence but you may find the real part is a function of the imaginary part (maybe linearly). And also the PCA as sussested by @knb.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.