I am currently working on a machine learning application that requires users to upload datasets, so naturally I want a database that has very fast batch-reads. Writing is only a one time process, but the learning model can be iteratively changed, and most optimization algorithms like mini-batch gradient descent use batch data for input.

The data will be in the form of tables. The dataset can involve any numerical set (floating and integer), like image sets. That would mean 10,000 columns for a dataset of 100*100 images like the CIFAR-100 dataset. So upto a few millions of columns is an upper bound.

MySQL is my preferred option here. But I am ready to work with any NoSQL database if it outperforms SQL database for this use case. Memory should not be a problem, as I am only expecting to do batch reads of upto 1000 rows at a time (and 100 on average).

  • You might try asking in DBA. You should include some sample data with a few rows (if that concept applies). Can you specify how many columns are involved and how much data each column will hold? How big is the initial data set and how much growth of data do you anticipate over time? SQL Server 2016 offers in-memory processing for some types of data. For reads, nothing beats in-memory processing. Also here on when it is best to use traditional database vs NoSQL: stackoverflow.com/questions/10553860/… – Sun Mar 16 '16 at 14:58
  • @sunk818 I added column details. So if I store data in mysql and then replicate it in an in-memory store like Redis, I should get the best performance? I am not expecting more than 1GB of data, and the server is AWS. So I am good to go? – goelakash Mar 16 '16 at 15:15
  • I'm not familiar with Redis. Try it with and without and see if you see a difference. With so few rows, I'm not sure if it'll make much of a difference. MySQL has an upper hard limit of 4096 columns: dba.stackexchange.com/questions/3972/too-many-columns-in-mysql – Sun Mar 16 '16 at 15:23
  • It seems that I could either normalise the rows (create table for each column) or create extra rows per sample(create ID, FeatureID and FeatureValue columns). – goelakash Mar 16 '16 at 16:30
  • Like I said, I think you'll get better help from DBA.stackexchange.com – Sun Mar 17 '16 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.