if there is a better stack to ask this on, please let me know.
I have a simulation that takes 3-4 days to run. I want to run many of these, with different parameters defined in text files on remote linux machines via ssh (most probably). I have already been doing this via a combination of bash, python, ssh and screen. My scripts send (or broadcasts many sets of) the input files to a directory on a remote machines and starts up the simulation in a detached screen. I have a script that checks to see whether the simulation has finished, and when it has, I have another script that post processes the results remotely and then gathers them.
Currently, I have a few problems. Some of the remote machines are lab computers, which means even though they are not supposed to be switched off, they inevitably are switched off from time to time. The simulation writes the outputs every 10 timesteps, so there's no great loss of work, but the simulation needs to be restarted once the computer is switched back on. Sometimes, the simulation crashes, the last recorded time step needs to be deleted and and the simulation manually restarted. On a related note, I can query the remote machines with a script to see if the simulation has finished successfully, but I don't know how to check if the simulation is running or waiting to be restarted without using ssh and top.
What I would like
I would like to make this process more sophisticated and streamlined. The model itself was developed by someone else (it's an OpenFoam CFD model if that makes a difference) and I have inherited it in order to use it for meta-modelling and optimisation work and it's not in my field of expertise to really go into the model much myself.
Ideally, I would like to build a dashboard or CLI that I can use to robustly, remotely, easily manage these simulations. I'd like to use Python if possible, because I'd be most comfortable with that, and I'm quite happy with bash too, though my work in bash is a lot more hacky.
I'm looking for design patterns or tools that would allow me to query the pool of machines I have in a programmatic way, to check the state of my simulations in a programmatic way and then program behaviour based upon those statuses.
For instance: when a computer is turned off with one of my simulations running, normally I don't realise until I look for results, otherwise the process is quite time consuming, so there's a lot of lost computing time. Is there a bash tool or do fabric or paramiko allow me to be notified when a machine is switched off? Is it silly to set up a task on a machine I know will not be turned off that will continuously query each of my machines in order to check if they're still on?
If this does not seem feasible, is it worth moving to the cloud? My worry is first that this will cost me (I don't think I have a budget for this) and that it will be a lot of hassle to get set up with the right environment, but am I wrong? It's not something I'm familiar with.
TL:DR I want to be able to programmatically query the status of and restart simulations on remote machines over ssh.