I have a pipeline of independent Python programs. They load their models on start (expensive operation, and not threadsafe) then wait for input. The input can be splitted into chunks which the program must process in as a whole. Each chunk can be processed independently by the same program, but some programs are depend on others output (as in a pipeline). The problem is, that some program is slower than the other which creates a bottleneck in the pipe causing the waste of the resources. The programs are can be imagined as microservices each running on a isolated single core process.
The single core version would be something like this:
cat input.txt | ./program_A | ./program_B | ./program_C | ... > output.txt
I want to upscale the pipeline above to multiple core and machine and multiple user at the same time. My ideal workflow should be something like this:
cat input.txt | ./split_to_chunks | ./program_A | MULTIPLE INSTANCE OF ./SLOW_program_B AS NEEDED TO REDUCE THE BOTTLENECK | ./program_C | ./put_chunks_together > output.txt
| (pipe) means possibly multiple machines,
> means some REST API.
./put_chunks_together are logical steps.
The constraints are the following:
- Only have a small number of machines (e.g. 3-5 machines each have 4 cores.)
- The programs have high memory usage, so I would avoid any virtualization/containerisation to squeeze all juice from the machines
- Keep the order of the cunks at the end.
- Big amount of data, multiple users, but small chunks
- The programs should stay in the memory and not allowed to shut down, after processing each chunk. They are stateless at the interchunk level.
Possible solutions (I'm not an expert on the topic):
- Batch processing/job spooling
- Mostly Hadoop, which I prefer not to use
- Mr Job and others: What about automatic scaling?
- Message Queuing
- Sounds complex, I can not find an example similar to my problem
- I don't see how it scales automatically
Tutorials are welcome!