I have a Parquet file that is much larger than my memory and would like to sort it according to two columns.
Specifically let's say I have 64GB of memory, a 1TB Parquet file with the columns
user_id (string) and
timestamp (integer) which I'd like sorted, in that order.
The solution must be able to
- work on a locally stored file
- handle the data with the limited available memory
- be robust in that I can be sure it won't run out of memory or stall
The solution should be FOSS, though non-FOSS recommendations would also be appreciated.
Preferred, but not required
- Solution that finishes in a reasonable time (let's say a couple of days at most)
- Output the sorted data to Parquet
The ideal solution would use Python, but at this point, I'm very happy to use absolutely any technology.
What I've tried
- Does not natively support sorting by two columns
- Memory usage often just blows up with bigger data (potentially solvable)
- Very low CPU utilization, seemed to be very slow
- I had problems with Parquet IO, Vaex seemed to simply stall.
- Admittedly, the reasons above are a bit vague as I haven't tried Vaex enough. If you think Vaex might be up for the task, then that could still be the answer.