I am trying to perform deep learning on very large datasets which have to be stored in a NewSQL database because it has to be distributed. However, for the training process, I currently import the data in pandas using pd.read_sql_query. However, this runs the query and stores the result set in memory. However, if I did so with the large dataset, I would run out of memory very quickly and the time taken for the data to transfer would be very large.

Is there a NewSQL database system which allows me to run a query and access only the parts I need when I need it? So I imagine, I would run a query, and the database stores the result set in memory while I need it. Meanwhile, if I read the first row, it will only send the first row of data to the client. So this way, I could train the model using the data stored on a database without running out of memory.

I would appreciate if anyone knows how to do this. The NewSQL functionality is optional but would be very good if it was there. The priority is this "remote reading" functionality.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.