7

if there is a better stack to ask this on, please let me know.

My Problem

I have a simulation that takes 3-4 days to run. I want to run many of these, with different parameters defined in text files on remote linux machines via ssh (most probably). I have already been doing this via a combination of bash, python, ssh and screen. My scripts send (or broadcasts many sets of) the input files to a directory on a remote machines and starts up the simulation in a detached screen. I have a script that checks to see whether the simulation has finished, and when it has, I have another script that post processes the results remotely and then gathers them.

Currently, I have a few problems. Some of the remote machines are lab computers, which means even though they are not supposed to be switched off, they inevitably are switched off from time to time. The simulation writes the outputs every 10 timesteps, so there's no great loss of work, but the simulation needs to be restarted once the computer is switched back on. Sometimes, the simulation crashes, the last recorded time step needs to be deleted and and the simulation manually restarted. On a related note, I can query the remote machines with a script to see if the simulation has finished successfully, but I don't know how to check if the simulation is running or waiting to be restarted without using ssh and top.

What I would like

I would like to make this process more sophisticated and streamlined. The model itself was developed by someone else (it's an OpenFoam CFD model if that makes a difference) and I have inherited it in order to use it for meta-modelling and optimisation work and it's not in my field of expertise to really go into the model much myself.

Ideally, I would like to build a dashboard or CLI that I can use to robustly, remotely, easily manage these simulations. I'd like to use Python if possible, because I'd be most comfortable with that, and I'm quite happy with bash too, though my work in bash is a lot more hacky.

I'm looking for design patterns or tools that would allow me to query the pool of machines I have in a programmatic way, to check the state of my simulations in a programmatic way and then program behaviour based upon those statuses.

Example

For instance: when a computer is turned off with one of my simulations running, normally I don't realise until I look for results, otherwise the process is quite time consuming, so there's a lot of lost computing time. Is there a bash tool or do fabric or paramiko allow me to be notified when a machine is switched off? Is it silly to set up a task on a machine I know will not be turned off that will continuously query each of my machines in order to check if they're still on?

Alternatives

If this does not seem feasible, is it worth moving to the cloud? My worry is first that this will cost me (I don't think I have a budget for this) and that it will be a lot of hassle to get set up with the right environment, but am I wrong? It's not something I'm familiar with.

TL:DR I want to be able to programmatically query the status of and restart simulations on remote machines over ssh.

Improve this question
8
  • 3
    I guess you want a batch system like it is usually used on clusters
    – MagunRa
    Dec 7, 2015 at 14:03
  • 1
    Thanks, I'll give that a go. Looks like ps might be a useful command line utility. This link looks quite useful to build upon: anyexample.com/linux_bsd/bash/…
    – Ogaday
    Dec 7, 2015 at 17:55
  • To be clear - you want a tool/library that allows you to query the status of running jobs on Linux machines over SSH through Python?
    – Undo
    Dec 7, 2015 at 18:06
  • Not necessarily through python. Bash is fine. I would like to build an application out of it, so that it's less manual checking, more automated checking and automated notifications a computer is shut down or if a job finishes, the computer can be allocated a new one without me having to manually do it.
    – Ogaday
    Dec 7, 2015 at 18:07
  • 1
    I would suggest taking a look at the render farm job management software, (much of it in python), that is already out there possibly with a view to adapting it to your use case. The problems are very similar.
    – Steve Barnes
    Dec 7, 2015 at 20:58

2 Answers 2

Reset to default
1

I suggest to look at this. I have not used it myself, but I will the next time I need to do things like this:

http://python-rq.org/

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It is backed by Redis and it is designed to have a low barrier to entry. It can be integrated in your web stack easily.

One hint: You say simulations take 3-4 days to run. I would try to cut this into small pieces. Is there a way to do parallel processing? This has many benefits:

  • If you compute small chunks, you don't loose much if one node fails.
  • It is faster.
Improve this answer
4
  • I think that this is probably what I wanted. I'm no longer doing this sort of thing, but I think I could have had a python script that uses a standard parallel library to manage child processes and subprocess to wrap the simulation itself. It's a pretty simple pattern, and I assume RQ does exactly that and more, with a queue? What are the pitfalls to look out when using such a pattern? How resilient is it: ie. if a computer running a child process gets turned off a connection lost, is it robust enough to keep everything else going?
    – Ogaday
    Mar 9, 2016 at 16:41
  • @Ogaday I have not used python-rq.org myself. I used celery once, and I think it has too many features and some drawbacks. The drawbacks of celery are according to my point of view: One man show (just look at the commit history at github) and restarting is not easy (I asked about this year and about three years ago, but nothing has changed).
    – guettli
    Mar 9, 2016 at 20:51
  • Fabric also seem like a good option, but python 2 only? Seeing as I was doing stuff over ssh, it might make a lot of sense.
    – Ogaday
    Mar 9, 2016 at 21:26
  • @Ogaday Fabric is not for parallel execution. It is like remote-shell-scripting (something I learned, and I will avoid it in the future). We are currently in the process of switching from fabric to salt (for configuration management).
    – guettli
    Mar 9, 2016 at 22:01
0

Grid Computing

I'm no expert on this, but I suspect Grid computing is the technical term for your needs.

One of the first such tools was Xgrid from Apple, based on Zilla by Next, Inc. Now discontinued.

HTCondor seems to be a solution that might work for you. It is an open-source long-running project at a university. It runs on Linux and other OSes.

Oracle Grid Engine is another such project. Forked into various related projects.

Wikipedia has this list of Grid computing projects.

Improve this answer
1
  • One constraint I forgot to mention was that I'm not a super user on the network. Grid computing does look worth investigating, however. I've heard of HTC Condor before, and I think there might be something set up like that on the network, so I'll do some digging, thanks.
    – Ogaday
    Dec 8, 2015 at 12:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.