1

I'm looking for recommendation for the following:

I currently have approximately 10 Million csv file each with the following identical data structure as file attached:

Typical data structure

I want to transfer all this data from the csv files into a database.

From the said database I want to be able to retrieve with Python in the fastest way possible all the ID for the files matching the search criteria which I am going to insert.

The criteria will be based on either the average or median from the "Product Quantity" or "Product Price".

An example of typical queries would be:

  • Retrieve from the entire database all ID which have AVG Product Quantity minor or equal to 50.

or

  • Retrieve from the entire database all ID which have AVG Product Price for Product No. from 1 to 10, below or equal to 30 USD?

What is the best database to do this? SQL, Mysql, SqlLite, MongoDB, Orcale or any other?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.