Looking for a small-scale (<1000 concurrent jobs) scheduling server to satisfy these requirements:

  • Server maintains a minimum number of worker nodes (e.g., always keep 10 workers alive; implies a keepalive check for workers)
  • Workers run in user-space (no sudo/root access available)
  • Worker nodes are launched via a specified shell command (behaves like SSH)
  • Gratis

Other things to note:

  • Communication between server and workers can be over sockets or over file-system (all systems are over NFS)
  • Can even be a module for Python/Perl/Go (since these are available on worker systems)

It takes multiple minutes for a worker node to spin up, so I'd like to always keep a certain number of worker nodes online to handle incoming tasks.

I am able to compile server and worker binaries (or modules) on an architecturally similar system and copy over to target systems if needed.

A GUI is not required; in fact, I'd prefer a purely commandline tool.

So far, I've looked at Zookeeper, Jenkins, Kala, Chronos, but they either require root on worker nodes, or do not have the concept of keeping a minimum number of workers alive.

Given that there are dozens (hundreds?) of scheduling systems, I'm reasonably sure such a system exists, but I'm lacking the right terminology to find it.

Thanks for your input.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.