I am looking for a minimal interface to distributed ndarrays. I'd like to chunk my data in arbitrary blocks, scatter them across a cluster, run computations on the local numpy array, rechunk my data in different blocks, know which process owns which block(s) etc.

I have in mind, for example, the block-cyclic distribution of matrices over grids of MPI processes that you use in ScaLAPACK

The array size can be O(10) TB, scattered across O(100) nodes. Every MPI rank owns one or multiple blocks of the array and operates over it with many threads.

(With my great surprise) it seems to me that there is no standard library for this purpose.

Considered libraries:

  • dask.array does all I need with an amazingly simple interface, but
    • I (actually other libraries I call) use MPI. dask "wastes" two ranks for the scheduler and running the script. This overhead is unacceptable when I have few ranks per node (and plenty of cores on each node).
    • I don't need most of the (great!) functionalities of dask (delayed computation, dynamic scheduling, etc.)
    • It is unclear who owns which block.
  • distarray
    • It is not maintained anymore

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.