1

We have built an MVP in Python Data ecosystem, but the performance is much less than desired.

Our approach:

  • multiprocessing module for task parallelisation (which is awkwardly done in Python - pickle/unpickle each task)
  • scikit-learn library
  • pandas for data transport
  • it's not about dealing with big data or training models on large data, after profiling we figured out most of the time is taken by a regular computing tasks, but we are doing a lot.

Hence:

  1. We wonder what else can we use to make computing faster? After some research number of candidates popped up: Julia, Lua, Rust etc.
    • Requirements:
      • FASTER
      • Parallelisation support by design (not some workaround like Python does)
      • Existence of scientific libraries, not all of Python but major selection
      • Compiled (I guess)
  2. But before we move to another platform - we wonder if there's a way of speeding up existing Python code?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.