1

I deal a lot with time-series-related tasks (primarily forecasting, but also clustering, EDA etc.), and, because of sheer amounts of data available, I've noticed that relying on Pandas and Numpy as primary tools of data preprocessing has become somewhat slowish. I'm not a CS engineer and my programming skills and knowledge are limited but if I understand it correctly, once you start working with large volumes of data, it makes sense to find some other useful tools that can help you overcome Big Data challenges.

Restrictions I find important:

  1. I would prefer a python-based tool since I am already familiar with the language;
  2. It should be suitable for handling large volumes of data;
  3. Ideally, it should have some useful ways of working with time series because unlike with tabular data partitioning and feature engineering can be a bit trickier.

What I have considered:

  1. Dask and Dask-ML;
  2. Pyspark.

Both options seem quite reasonable. However, before diving into something completely new to me, I would love to hear your thoughts on it. Any help is greatly appreciated!

Improve this question
2
  • Could you please quantify how big your datasets are and what kind of resources your PC has available? The cause of the slowdown could be your dataset not fitting in the available RAM, causing your system to swap... or it could be purely CPU-limited, in which case Spark and other Hadoop-like systems would only slow it down further.
    – aitap
    Jan 23 at 13:10
  • 1
    Sure. Datasets normally have more than 50 million rows. In some cases it's simply impossible to fit them in the RAM, so I have to process them in chunks. My PC has 32 gb of RAM and Core i7-13700K. But I can use multiple local machines.
    – User
    Jan 23 at 14:03

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.