I am trying to perform deep learning on very large datasets which have to be stored in a NewSQL database because it has to be distributed. However, for the training process, I currently import the data in pandas using pd.read_sql_query. However, this runs the query and stores the result set in memory. However, if I did so with the large dataset, I would run out of memory very quickly and the time taken for the data to transfer would be very large.
Is there a NewSQL database system which allows me to run a query and access only the parts I need when I need it? So I imagine, I would run a query, and the database stores the result set in memory while I need it. Meanwhile, if I read the first row, it will only send the first row of data to the client. So this way, I could train the model using the data stored on a database without running out of memory.
I would appreciate if anyone knows how to do this. The NewSQL functionality is optional but would be very good if it was there. The priority is this "remote reading" functionality.