3

I have a .csv file with 3.5 GB of data (around 8 million rows and 86 columns), and I need to build a regression model on this dataset.

The thing is, just trying to read the CSV using pandas, and then subsequently doing any kind of operation on it (even just taking the number of rows), takes a ridiculous amount of time. So, considering I need to visualize the data to know how it looks, then preprocess it, and etc etc, I don't really think it's feasible to do it the same way I've always done for files with 50-100 MBs-ish.

I tried looking into python's multiprocessing module, but while it did help me to calculate the number of rows, I don't see it helping me with most of the other things I need to do (like building the model for example).

So, does anyone know how I should tackle this? I am doing it on a python notebook.

Improve this question
  • In R there are some packages build to deal with huge single-file data-sets, see this post – Llopis Nov 23 '16 at 11:42
  • The obvious suggestion would be to do it on another PC. Can you do that? Also, is this just once, or a recurring thing? – Mawg says reinstate Monica Oct 5 '18 at 8:34
4

Take a look at the Dask library. (http://dask.pydata.org/en/latest/). It extends pandas' DataFrames and Numpy arrays for larger-than-memory computation.

Here's a blog post about using Dask with scikit-learn: https://www.continuum.io/blog/developer-blog/dask-and-scikit-learn-3-part-tutorial

Improve this answer
  • Looks quite nice, I'll certainly take a look at it! – Berne Nov 23 '16 at 13:36
1

Several suggestions:

  • Delete rows you don't need. Perhaps some are not relevant to the model
  • Delete columns you don't need e.g ones that are constant or low variance or just not important
  • Change all the int64s and floats to int8 (IF they fit)
  • Take a sample of rows
  • Use feature reduction e.g. PCA if you can; or a univariate method like chi2
Improve this answer
  • 1
    While I do acknowledge these are nice things to do to make my data smaller, it's still something that will take quite some time to run. I was hoping to be able to do something before that stage, exactly so I can make the preprocessing faster. Which is why I was thinking about using the cloud, but was unsure about what to use exactly. – Berne Nov 23 '16 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy