1

I am planning to work on an project that does analysis on the Stock prices. The daily data gets fetched from an API end point. Before running the analytics and predictions, this data needs to be loaded into a database so that the process of running the analytics is easier.

The question here is that: I am planning to run this on a local machine which will be a laptop. The data source here is an API end point. At any given point in time the database contains the all the stocks OHLC daily data for the last 10 years. The analytics will be run on the database itself using Stored Procedures or Python Scripts. Looking forward to automate this process end-to-end.

This will be a daily batch run i.e. it will be run only once per day.

Is Airflow and Postgresql a good choice?

What should be the preferred ETL tool and database for this? (I am looking only for open-source tools because this a personal project).

The final that I am looking for is an light weight ETL tool (or similar software/setup) and a database to handle 10Million records efficiently with minimal hardware requirements.

1 Answer 1

1

I have been banging my head to find a solution for this one. Randomly I ended up reading https://makebook.io/ execution part. This has made things clear. I doesn't matter much with which tool I get started now because most of the modern tools are very powerful and we rarely end up reaching their limits. So Airflow and Postgresql is the choice I have chosen for now and will start on the development using the same.

1
  • 1
    But I am still open to other tools that are being suggested by the community members so that I get to know about them and probably can make use of them in future projects. Mar 21, 2021 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.