I am looking for a minimal interface to distributed ndarrays. I'd like to chunk my data in arbitrary blocks, scatter them across a cluster, run computations on the local numpy array, rechunk my data in different blocks, know which process owns which block(s) etc.
I have in mind, for example, the block-cyclic distribution of matrices over grids of MPI processes that you use in ScaLAPACK
The array size can be O(10) TB, scattered across O(100) nodes. Every MPI rank owns one or multiple blocks of the array and operates over it with many threads.
(With my great surprise) it seems to me that there is no standard library for this purpose.
dask.arraydoes all I need with an amazingly simple interface, but
- I (actually other libraries I call) use MPI. dask "wastes" two ranks for the scheduler and running the script. This overhead is unacceptable when I have few ranks per node (and plenty of cores on each node).
- I don't need most of the (great!) functionalities of dask (delayed computation, dynamic scheduling, etc.)
- It is unclear who owns which block.
- It is not maintained anymore