We have built an MVP in Python Data ecosystem, but the performance is much less than desired.
Our approach:
- multiprocessing module for task parallelisation (which is awkwardly done in Python - pickle/unpickle each task)
- scikit-learn library
- pandas for data transport
- it's not about dealing with big data or training models on large data, after profiling we figured out most of the time is taken by a regular computing tasks, but we are doing a lot.
Hence:
- We wonder what else can we use to make computing faster? After some research number of candidates popped up: Julia, Lua, Rust etc.
- Requirements:
- FASTER
- Parallelisation support by design (not some workaround like Python does)
- Existence of scientific libraries, not all of Python but major selection
- Compiled (I guess)
- Requirements:
- But before we move to another platform - we wonder if there's a way of speeding up existing Python code?
