1

Question

I have recently encountered the following interesting task and I am wondering someone could help me find the best solution, or the best software with which I can find solution.

Say I have an M x N matrix, M number of rows/observations and N number of columns/fields. I need to choose K columns from the N number of columns, and use the remaining M x K dataset to do a certain thing (could be any statistical computation, like regression, correlation matrix, summation of all the number, etc, etc).

The issue is, I need to do the above thing for EVERY "N choose K" combination !

Example

Let N = 30

K = 8 There will be 5.8 million combinations (30 choose 8)
K = 9 There will be 14.3 million combinations (30 choose 9)

And I will need to loop through all the combinations for k = 1, 2, 3, ..., 30, respectively.

Struggle

Software wise, I have been using MatLab for a while but it slows down significantly when N gets larger than 25. As a result, I did some research and it seems SAS is known for dealing with large data sets, and its industry-grade statistics packages. So then I put together a bunch of resources and references and started picking up SAS. But my concern is that I spend so much time only to hit a hard rock and get stuck, or only to find out that SAS is no better solution to my issue above.

How To Help Me

It would be much appreciated if someone could ...

  1. Advise on what software suite solves my problem in the more efficient way
  2. General tricks and tips you have, while using SAS, MatLab, R, etc. I really need to know their real differences, strengths, and weaknesses !! Any references, links, or discussion regarding the pros and cons of those software animals would be much appreciated.
  3. Advise on how mathematically or algorithm-wise I can reduce the steps of completing the above task
  4. Any other reference, link that you think may relate to my issue. Anything you want me to study, look into, and research, that may point me to the right directions.

Thank you for your patience : )

2
  • Whatever you choose, consider using parallel computations. The only option I know, matlab, has parfor which lets you run independent loop iterations in parallel. Do you have access to a computer cluster, or at least to a computer with multiple physical cores? (Disclaimer: I don't have any experience with processing data of this multitude, Big Data guys are probably called for) – Andras Deak Sep 28 '15 at 19:58
  • I do have a multi-core PC at my consumption but it's the software that has been the bottleneck for my working progress. I don't have access to a customize-able cloud yet, but I am very much interested in getting one. What options do I have? – user4564894 Oct 13 '15 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.