Python has a very handy multiprocessing built-in module. Particularly its Pool class allows simple parallelization. However, it only supports a shared-memory multiprocessing model, not a distributed memory model like is used in cluster computing (like MPI). There are tools like mpi4py that allow distributed memory processing in Python, but the ones I have been able to find use substantially different semantics.

Is anyone aware of a distributed memory package for Python that offers a roughly drop-in replacement for the built-in multiprocessing.Pool class?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.