Been told to look into Workflow Engines to automate the running of some processes, I'm not sure its correct for my use case

Question

I have several large, intensive programs that I need to run one after the other, with the input of one flowing to the output of the next. I want to automate this process as it would be occurring pretty often and its a pain to run one then the next.

I've been told to look into workflow engines for this, so I've had a look at Luigi, Airflow and Amazon SWF, but none of them really seem to be aimed at the right thing. The programs I need to run are all self contained applications, rather than a few lines of code that need to be run thousands of times a second. These tools seem to all be geared towards doing Hadoop jobs, which I dont think is really what I need to do.

Does anyone have anything they could suggest?

Thanks

Answer 1

Well you are right, I think a workflow engine / job scheduler would meet your need. I am not familiar with those mentioned but ProActive from ActiveEon would work.

To give more details, you'll be able to visually create dependencies between your different programs and allocate resources according to their individual needs.

Do you only have an objective of automation or are you also trying to optimize the resource utilization?