I'm currently developing a, rather simple, event-driven service architecture using Kafka and Python. One of the service performs very heavy machine learning computation and I would like to distribute the load over multiple machines. I'm quite confident I would be able to implement this in Kafka maintaining the stateless trait of the current architecture (not a requirement but you know, state sucks) but it feels a lot like an anti-pattern.
Therefore I'm considering some more specialized software that would be used exclusively by this service. Do you believe it makes sense? And if so, which software should I use?
As I said I'm using Python and I have Zookeeper available because it comes with Kafka. Apparently though there are no mature options that support Zookeeper as a broker. We are a small company and we work mostly on premise so keeping the number of softwares to learn and maintain is a priority.