I'm looking for recommendation for the following:
I currently have approximately 10 Million csv file each with the following identical data structure as file attached:
I want to transfer all this data from the csv files into a database.
From the said database I want to be able to retrieve with Python in the fastest way possible all the ID for the files matching the search criteria which I am going to insert.
The criteria will be based on either the average or median from the "Product Quantity" or "Product Price".
An example of typical queries would be:
- Retrieve from the entire database all ID which have AVG Product Quantity minor or equal to 50.
- Retrieve from the entire database all ID which have AVG Product Price for Product No. from 1 to 10, below or equal to 30 USD?
What is the best database to do this? SQL, Mysql, SqlLite, MongoDB, Orcale or any other?