I have a set of image processing jobs that is somewhat similar to a build process with multiple dependencies of software. I keep updating source code that makes up multiple (mostly sequential) steps of analyses, as well as updating input data and parameters.

I thought it might be useful if I use some kind of dependency analysis (using a directed acyclic graph) just like build systems such as Buck and Bazel do. It will make it easy to do repeated, exploratory data analysis tasks.

Is there any package for doing that easily in Python or any other languages? Even more, is there any information for "building" data analysis jobs in general?


There are several pipeline modelling packages in Python, which can help you define dependencies and thus design data flow(and processing) pipelines. Some are:

Please look at a similar answer, for such libraries in more languages.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.