Is there any framework or library for managing “dependencies” of data processing?

Question

I have a set of image processing jobs that is somewhat similar to a build process with multiple dependencies of software. I keep updating source code that makes up multiple (mostly sequential) steps of analyses, as well as updating input data and parameters.

I thought it might be useful if I use some kind of dependency analysis (using a directed acyclic graph) just like build systems such as Buck and Bazel do. It will make it easy to do repeated, exploratory data analysis tasks.

Is there any package for doing that easily in Python or any other languages? Even more, is there any information for "building" data analysis jobs in general?

Answer 1

There are several pipeline modelling packages in Python, which can help you define dependencies and thus design data flow(and processing) pipelines. Some are:

• Spotify's Luigi
• Airbnb's Airflow

Please look at a similar answer, for such libraries in more languages.