I work usually on processing data from databases such as Postgres, mysql, and mongoldb where I connect to the database through python script process the data then re-upload it.
However, my issue is currently I'm working with a project where I have 10+ million records and growing everyday. Pulling such data for processing then uploading it back is very expensive.
I'm novice in the database world and my question might not make sense, but I know most of the solutions for manipulating such big datasets follow a sql(Hadoop/hive) solution. However, is there a solution where I can run python scripts on a database directly such as Mongodb/presto without having to go through fetching and re-uploading on my local machine?