I am planning to work on an project that does analysis on the Stock prices. The daily data gets fetched from an API end point. Before running the analytics and predictions, this data needs to be loaded into a database so that the process of running the analytics is easier.
The question here is that: I am planning to run this on a local machine which will be a laptop. The data source here is an API end point. At any given point in time the database contains the all the stocks OHLC daily data for the last 10 years. The analytics will be run on the database itself using Stored Procedures or Python Scripts. Looking forward to automate this process end-to-end.
This will be a daily batch run i.e. it will be run only once per day.
Is Airflow and Postgresql a good choice?
What should be the preferred ETL tool and database for this? (I am looking only for open-source tools because this a personal project).
The final that I am looking for is an light weight ETL tool (or similar software/setup) and a database to handle 10Million records efficiently with minimal hardware requirements.
