I have recently encountered the following interesting task and I am wondering someone could help me find the best solution, or the best software with which I can find solution.
Say I have an M x N matrix, M number of rows/observations and N number of columns/fields. I need to choose K columns from the N number of columns, and use the remaining M x K dataset to do a certain thing (could be any statistical computation, like regression, correlation matrix, summation of all the number, etc, etc).
The issue is, I need to do the above thing for EVERY "N choose K" combination !
Let N = 30
K = 8 There will be 5.8 million combinations (30 choose 8)
K = 9 There will be 14.3 million combinations (30 choose 9)
And I will need to loop through all the combinations for k = 1, 2, 3, ..., 30, respectively.
Software wise, I have been using MatLab for a while but it slows down significantly when N gets larger than 25. As a result, I did some research and it seems SAS is known for dealing with large data sets, and its industry-grade statistics packages. So then I put together a bunch of resources and references and started picking up SAS. But my concern is that I spend so much time only to hit a hard rock and get stuck, or only to find out that SAS is no better solution to my issue above.
How To Help Me
It would be much appreciated if someone could ...
- Advise on what software suite solves my problem in the more efficient way
- General tricks and tips you have, while using SAS, MatLab, R, etc. I really need to know their real differences, strengths, and weaknesses !! Any references, links, or discussion regarding the pros and cons of those software animals would be much appreciated.
- Advise on how mathematically or algorithm-wise I can reduce the steps of completing the above task
- Any other reference, link that you think may relate to my issue. Anything you want me to study, look into, and research, that may point me to the right directions.
Thank you for your patience : )