I have several large, intensive programs that I need to run one after the other, with the input of one flowing to the output of the next. I want to automate this process as it would be occurring pretty often and its a pain to run one then the next.
I've been told to look into workflow engines for this, so I've had a look at Luigi, Airflow and Amazon SWF, but none of them really seem to be aimed at the right thing. The programs I need to run are all self contained applications, rather than a few lines of code that need to be run thousands of times a second. These tools seem to all be geared towards doing Hadoop jobs, which I dont think is really what I need to do.
Does anyone have anything they could suggest?