I'm currently looking for a job queuing/job scheduling system.

What I plan to do:

  • On a single, powerful multi-core machine (16 Cores, 128RAM),
  • queue and run tasks (~ run time from hours to a day),
  • to which the number of cores are explicitly assigned.

Software requirements

  • Should be able to execute the queued tasks,
  • distribute the workload to the available cores, or queue if no ressources are free.
  • Freeware or open source
  • Written in Python or having at least Python bindings

In a few words, I want to use something like PBS, where I can simply submit via qsub. But as I need such a system only for a single machine, I think that PBS is way to bloated.

I had a brief look into celery, and RQ, but I'm not sure if they fully meeet the requirements (i.e. I'm not sure if the ressource managemenets works as I need).

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.