2

I'm searching for a message queue solution containing the most achievable reliability of whole message life-cycle.

My system consists of a set of applications emitting events - messages. I need to have all these events propagated to a message system. A set of consumers then read those messages. The throughput is about 100 events/sec. Each event is about 2k-1MB.

My thought is to have a producer integrated to all my event emitting applications...

  • When any of these apps dies it simply stops producing events. That's OK.
  • When broker(s) dies the messages should be persisted on client. Retries should deliver the messages (ideally with no duplicates).

I've tried Apache Kafka, although it seems they don't really support such small installations where all brokers could possibly die together (no geo separation, etc.). Thus resulting in producers simply not retrying when broker(s) not being reachable.

Any suggestions? Thanks.

1

For an on-premises solution, I have seen enterprises use RabbitMQ with good results. I think it fulfills your whole spec, and you can build a network of machines running Rabbit for resilience.

For a hosted solution, I would look at Azure Service Bus but note that 1MB is the absolute limit on message size, even on the Premium tier.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.