Looking for a small-scale (<1000 concurrent jobs) scheduling server to satisfy these requirements:
- Server maintains a minimum number of worker nodes (e.g., always keep 10 workers alive; implies a keepalive check for workers)
- Workers run in user-space (no sudo/root access available)
- Worker nodes are launched via a specified shell command (behaves like SSH)
- Gratis
Other things to note:
- Communication between server and workers can be over sockets or over file-system (all systems are over NFS)
- Can even be a module for Python/Perl/Go (since these are available on worker systems)
It takes multiple minutes for a worker node to spin up, so I'd like to always keep a certain number of worker nodes online to handle incoming tasks.
I am able to compile server and worker binaries (or modules) on an architecturally similar system and copy over to target systems if needed.
A GUI is not required; in fact, I'd prefer a purely commandline tool.
So far, I've looked at Zookeeper, Jenkins, Kala, Chronos, but they either require root on worker nodes, or do not have the concept of keeping a minimum number of workers alive.
Given that there are dozens (hundreds?) of scheduling systems, I'm reasonably sure such a system exists, but I'm lacking the right terminology to find it.
Thanks for your input.
