I have a .csv file with 3.5 GB of data (around 8 million rows and 86 columns), and I need to build a regression model on this dataset.
The thing is, just trying to read the CSV using pandas, and then subsequently doing any kind of operation on it (even just taking the number of rows), takes a ridiculous amount of time. So, considering I need to visualize the data to know how it looks, then preprocess it, and etc etc, I don't really think it's feasible to do it the same way I've always done for files with 50-100 MBs-ish.
I tried looking into python's multiprocessing module, but while it did help me to calculate the number of rows, I don't see it helping me with most of the other things I need to do (like building the model for example).
So, does anyone know how I should tackle this? I am doing it on a python notebook.