I'm wondering which simple tools could help me implementing a standard Master <-> N Workers stateful job dispatch in Java. Master would be polling jobs from a broker and distributing it to worker nodes of a private cluster. The Worker instance would be executing jobs. Each worker would have a stateful and (sadly) non-serializable inner state related to the jobs entities it is processing.

So jobs related to a certain entity would need to be all sequentially dispatched to the same Worker instance - and be all replayed on another living node in case it crashes before the "end signal" job.

As a first intention, I wanted to use the Akka toolbox (actors / clusters) - but since the toolbox is not well known / liked by the rest of the team, I was wondering if there was not a more simple / standard way of achieving this with the current team stack (Amazon Cloud, Spring Cloud) or with traditional distributed job processing framework (Spark / Flink) ?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.