I have several large, intensive programs that I need to run one after the other, with the input of one flowing to the output of the next. I want to automate this process as it would be occurring pretty often and its a pain to run one then the next.

I've been told to look into workflow engines for this, so I've had a look at Luigi, Airflow and Amazon SWF, but none of them really seem to be aimed at the right thing. The programs I need to run are all self contained applications, rather than a few lines of code that need to be run thousands of times a second. These tools seem to all be geared towards doing Hadoop jobs, which I dont think is really what I need to do.

Does anyone have anything they could suggest?


1 Answer 1


Well you are right, I think a workflow engine / job scheduler would meet your need. I am not familiar with those mentioned but ProActive from ActiveEon would work.

To give more details, you'll be able to visually create dependencies between your different programs and allocate resources according to their individual needs.

Do you only have an objective of automation or are you also trying to optimize the resource utilization?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.