0

Problem

I have a Parquet file that is much larger than my memory and would like to sort it according to two columns.

Specifically let's say I have 64GB of memory, a 1TB Parquet file with the columns user_id (string) and timestamp (integer) which I'd like sorted, in that order.

Requirements

The solution must be able to

  • work on a locally stored file
  • handle the data with the limited available memory
  • be robust in that I can be sure it won't run out of memory or stall

The solution should be FOSS, though non-FOSS recommendations would also be appreciated.

Preferred, but not required

  • Solution that finishes in a reasonable time (let's say a couple of days at most)
  • Output the sorted data to Parquet

The ideal solution would use Python, but at this point, I'm very happy to use absolutely any technology.

What I've tried

Dask

  • Does not natively support sorting by two columns
  • Memory usage often just blows up with bigger data (potentially solvable)

Vaex

  • Very low CPU utilization, seemed to be very slow
  • I had problems with Parquet IO, Vaex seemed to simply stall.
  • Admittedly, the reasons above are a bit vague as I haven't tried Vaex enough. If you think Vaex might be up for the task, then that could still be the answer.
2
  • Have you think about using hive tables (if you have hadoop of course)? Aug 4 at 16:46
  • 1
    @RomeoNinov I do not have hadoop, but I might consider looking into that option, if nothing else crops up. Feel free to answer with more details.
    – Dahn
    Aug 5 at 14:41
1

One possible solution about your case is to use external software. My idea is to use Hive with Parquet tables and create the sort there.

Here you can find sample instruction how to create table for Parquet. You can also check this Q/A in Stack Overflow about load the data directly from Parquet file.

Here you will find how to build statement with sort by column. Here you will see some detailed documentation.

But generally the command will be something like:

select * from table order by col1,col2;

This solution can work with huge tables because it split the work in jobs which deal with much smaller data than the source file/table.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.