I'm currently looking for a job queuing/job scheduling system.
What I plan to do:
- On a single, powerful multi-core machine (16 Cores, 128RAM),
- queue and run tasks (~ run time from hours to a day),
- to which the number of cores are explicitly assigned.
- Should be able to execute the queued tasks,
- distribute the workload to the available cores, or queue if no ressources are free.
- Freeware or open source
- Written in Python or having at least Python bindings
In a few words, I want to use something like PBS, where I can simply submit via qsub. But as I need such a system only for a single machine, I think that PBS is way to bloated.
I had a brief look into celery, and RQ, but I'm not sure if they fully meeet the requirements (i.e. I'm not sure if the ressource managemenets works as I need).