3

I have a couple of machine learning experiments which I start running when my machine is free long enough. Each job takes several hours.

I would like to have a simple "scheduler" which executes commands one after another. When one job is finished, the execution time should get recorded and the next job is started. If I kill the current job / scheduler (which should be possible without problems), then the next time it is started the last (not finished) job should be started again.

It would be nice if the commands could be stored / entered in a file which is easy to edit and could be put under version control (e.g. JSON).

The software has to be gratis and run under Ubuntu 16.04

Improve this question

1 Answer 1

Reset to default
2

gnu parallel should be fully sufficient here. E.g. this command:

parallel -j1 --joblog logfile time :::: joblist

reads your jobs from the textfile "joblist", where each line is regarded as one command, runs exactly one job at a time and measures the time used for each of the jobs. It also saves run jobs in the text file "logfile" for later use

Should you later decide to kill it you can resume with

parallel resume --joblog logfile -j1 time :::: joblist
Improve this answer
2
  • Re: "one command, runs exactly one job": please clarify if "job" and "command" mean the same thing here.
    – agc
    Apr 4, 2017 at 14:50
  • The meaning of "job" is more comprehensive here: It can either be a single command or a small shell script that is to be run for each line in the input file
    – jf1
    Apr 4, 2017 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.