0

Are there any end to end open source industry standard ETL projects written in Python along with other technologies such as Airflow, cron etc. Writing an efficient and scalable ETL program is not an easy task. I am looking for an open source project that I can learn from and can be used as a framework to work on ETL projects. My main aim is to learn from an industry standard example. Something that covers the below.

1) Metadata driven control 2) Configuration capability 3) Activity Logging 4) Automating & Scheduling 5) ETL Monitoring 6) Error Alerting 7) Deployment Etc

Any GitHub links or resources would be appreciated.

Improve this question
2
  • It would help a great deal if you explain what ETL means in your context and ideally which industry you are looking for it to be a standard in. – Steve Barnes Jan 6 '20 at 5:07
  • A general ETL job e.g. extracting data from a source such as file, API or DB. Transforming it and loading it to a db. I am keen to understand the code layout and best practices of an existing solution. In terms of industry wise it doesn’t really matter. – Sharingan Jan 6 '20 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.