I work usually on processing data from databases such as Postgres, mysql, and mongoldb where I connect to the database through python script process the data then re-upload it.

However, my issue is currently I'm working with a project where I have 10+ million records and growing everyday. Pulling such data for processing then uploading it back is very expensive.

I'm novice in the database world and my question might not make sense, but I know most of the solutions for manipulating such big datasets follow a sql(Hadoop/hive) solution. However, is there a solution where I can run python scripts on a database directly such as Mongodb/presto without having to go through fetching and re-uploading on my local machine?


Yes; you simply need to define your workflow process in terms of operations to be carried out by a succession of SQL statements, and use the Python connection to send those SQL statements to the database. For example, rather than fetching the data into a Python data structure, one of those SQL statements might put the data into a temporary table, and subsequent SQL statements will then process it as necessary. This approach only requires enough SQL expertise to express your workflow as a sequence of SQL statements.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.