0

I am working on a Python application where I need to update multiple databases with historical stock data. Here’s a breakdown of the requirements:

I have 5 databases, each storing candlestick data for different time intervals:

  • 1-minute candles

  • 5-minute candles

  • 15-minute candles

  • 30-minute candles

  • Day candles

Each database contains 50 stocks/collections, and each collection has approximately 2000 documents (representing historical data for each stock).

I need to update all the databases in parallel to ensure the historical data is processed efficiently.

My goal is to leverage GPU (Nvidia) to speed up the process.

  • How can I efficiently update all 5 databases in parallel?

  • How can I utilize the GPU (Nvidia) in Python for this task, especially to offload the computations (data processing) to the GPU?

  • What libraries or techniques should I use to manage parallelism and GPU utilization for this type of workload?

I am currently using MongoDB for the databases. I am familiar with Python libraries like multiprocessing and threading for parallelism but not sure how to integrate GPU processing with these. I have an Nvidia GPU and am aware of libraries like PyCUDA and CuPy, but I am unsure how to use them for database updates.

Improve this question
2
  • It sounds like your problem is IO bound, rather than processing bound, in that the limiting factors are the speed of reading through files and accessing databases. This generally means that trying to use the GPU will probably slow things down.
    – Steve Barnes
    Commented Oct 6 at 9:09
  • I would make sure I had a nvme storage before I tried to leverage a GPU for database.
    – cybernard
    Commented Oct 11 at 16:33

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.