4

I am looking for software to improve my workflow, when running computational (numerical) exeriements.

After seeing how things are done in the super-computer world, with PBS, I want to get some of the benifits for myself.

I want to make job scripts, and then leave them. (Rather than my current, run a job and watch it work, like a mother hen. which is a waste of time.). I want to schedule multiple jobs, and have them run when computer capacity is free.

Basically I have say 8 computation experiments, I can't run them all at once as 2 of them take up 24Gb of RAM, and the rest use say 10Gb each, (and my system only has 32Gb of RAM). My current work is memory limited, but I would expect any job schedualling software to also support CPU (/thread) limited tasks.

I only have one server, but it is very grunty. I am not the only user, however all other users are only doing simple things like running command line chat clients and text editors, most of the time.

  • OS run on (Wheezy) Debian Linux
  • Price: Free as in beer. (Gratis). I'm good if it is free for Not for profit or for Research Use.
  • Must Not: Require it(/its jobs) to be the only thing going on on the server
  • Should: support python jobscrips
  • Should: Email me the results. (Including standard out, etc)
  • Should: Be installable via a debain package manager. Wether that is apt-get, or pip or rubygems. (This isn't a deal breaker, but it would be extremely nice to have)
  • Does Not: require any kind of support for clustering
  • Does Not: require any kind of GUI

Only thing I've heard of is TORQUE, and I don't know anyone who has used it, or know much about it.

Improve this question
4
  • Anything wrong with Cron and at, which ship with all Linux distros? If it's the "lack of a GUI", then it's a lack in your list of requirements :)
    – Izzy
    Commented Mar 19, 2014 at 9:30
  • @Izzy: to my knowledge Cron and at are time based tool. That is they let you schedual a job to run, at particular times. I don't know how long my jobs will run for, or which can run at the same time as each other (because they don't complete for resources.) I could math hammer out an estimate (maybe), but I know tools exits. I've editted the question to hopefully be more clear.
    – Frames Catherine White
    Commented Mar 19, 2014 at 17:36
  • If the only thing standing against that is you've to ensure they don't overlap, you might wish to look into semaphores (in easy terms: put a "flag-file" on script start, delete it at script end, and check for it before the job gets processed; optionally throw a warning when "old semaphores" are found indicating a job hangs/crashed). Apart from that, while waiting for answers: aunt Google tells linux job-scheduler concurrency, maybe :)
    – Izzy
    Commented Mar 20, 2014 at 10:41
  • Izzy: if I go that far, I might as well go a little further and just implement a job scheduling as a python program. Honestly it would probably be 1 hr work. But TORQUE exists and it is the open source version of the tool I have had a few hours training in. (I suspect it is over powered as it does think in clusters.) There is a thread about torque on a single computer here [ubuntuforums.org/showthread.php?t=1512061]. If i end up using it I'll answer my own question.
    – Frames Catherine White
    Commented Mar 20, 2014 at 15:20

2 Answers 2

Reset to default
1

Take a look at SLURM.

  • OS has a Debian package (source)
  • Price: again, it has a Debian package
  • other things might run on the server
  • you can submit everything (with as many parameters as you like) as long as it is executable
  • Email: I don't know if there is direct support for that. But I think that might rather be part of the script you submit. Eventually you can write a wrapper script, that gets submitted to SLURM and does nothing than calling the actual script, receiving the results when this is ready and sending an email.
  • GUI: I think there are GUIs, but I never have used one as I use SLURM at university over SSH.
  • Clustering is supported (I'm running speech recognition over a HPC cluster)

Important commands are:

Improve this answer
0

Have a look at Hudson or its (maybe more active) Jenkins fork. They are "Continuous Integration" servers which are intended for controlling automatic software builds, but can also be used to start automatic tests etc. I'm not intimately familiar with the software, but here's an overview:

  • OS: Java/Tomcat-based; runs on Linux; Debian package available
  • Price: free (as in freedom and in beer)
  • Email: supported
  • GUI: web interface for configuration etc.; configured jobs can then be started by accessing their URLs with wget/curl
  • runs any shell command
  • for every host, the number of allowed concurrent jobs can be specified; jobs will be queued until a slot is available
  • doesn't bother if other things are running on the hosts
    • AFAIK there is no "global" system load check, though
Improve this answer
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.