I'm currently looking for a job queuing/job scheduling system.

What I plan to do:

  • On a single, powerful multi-core machine (16 Cores, 128RAM),
  • queue and run tasks (~ run time from hours to a day),
  • to which the number of cores are explicitly assigned.

Software requirements

  • Should be able to execute the queued tasks,
  • distribute the workload to the available cores, or queue if no ressources are free.
  • Freeware or open source
  • Written in Python or having at least Python bindings

In a few words, I want to use something like PBS, where I can simply submit via qsub. But as I need such a system only for a single machine, I think that PBS is way to bloated.

I had a brief look into celery, and RQ, but I'm not sure if they fully meeet the requirements (i.e. I'm not sure if the ressource managemenets works as I need).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.