I'm working with a start-up as a Data Engineer. We have python automatized scripts for loading data to db's through different VM's per project, so far we've been using Slack to get notifications when the process fails for any script in order to fix and re-run it. When this happens, we proceed manually.

I want to find a tool/framework/something to help me in the daily verification of all the tables (hosted in BigQuery) succesfully loaded; in my head I visualize it as a check-list. The simplest idea I got is to use a table that refreshes everytime a target table is loaded; this also looks as the longer and messy solution. Another idea is to create a bot, but I know almost anything about the topic.

Any help will be very appreciated. Thanks in advance.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.