I search a parallel execution library for Python.

There are two in the standard library:

I discovered, that simple tasks like handling timeouts are not easy with the standard library tools. Handle multiprocessing.TimeoutError in multiprocessing

Some years ago I used celery, but this is too big for my current use case.

Wanted features

  • Serverless on local machine
  • Should run on linux. Windows/Mac support not needed.
  • Sane handling of ctrl-c
  • Later, extending to compute on other machines should require only few source code changes.
  • Optional: GUI for browsing job results.
  • Situation is not clear, nor updated. Too many libraries, not all propriate for clusters. – jaromrax Jan 16 '17 at 13:50
  • @jaromrax do you have a hint? – guettli Jan 16 '17 at 15:32
  • The question here is not clear. Specifically, why is the solution using multiprocessing's apply_async as suggested in the comment on the linked SO question is not an option ? Also, what does Sane handling of ctrl-c mean ? Would it be sane to kill the current process but wait for all other forked processes to complete or would it be sane to also send ctrl-c to the processes ? The decision should be left to the library user rather than the module implementation, imo. – lonetwin Apr 26 '19 at 12:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.