I'm looking for a simple, lightweight parallel job scheduling engine, for the following workflow (in this example, from a shell):
$ createpool -n 2 # create a worker pool with 2 workers $ submitjob -options # submit a job, starts immediately $ submitjob -options # submit a job, starts immediately $ submitjob -options # submit a job, waits for worker to become free $ queryjob -options # query status of a job
Ideally, this would enabled by a simple script or Python module that we could ship with our driver code - no separate installation needed. We only need execution on a local worker pool, no cluster applications.
The scheduler needs to allow me to create a custom initialization script, say to copy some files in place. Then it should shut the pool down once idle.
OS: Mac, Windows
Price: free, open source