I'm searching for a message queue solution containing the most achievable reliability of whole message life-cycle.
My system consists of a set of applications emitting events - messages. I need to have all these events propagated to a message system. A set of consumers then read those messages. The throughput is about 100 events/sec. Each event is about 2k-1MB.
My thought is to have a producer integrated to all my event emitting applications...
- When any of these apps dies it simply stops producing events. That's OK.
- When broker(s) dies the messages should be persisted on client. Retries should deliver the messages (ideally with no duplicates).
I've tried Apache Kafka, although it seems they don't really support such small installations where all brokers could possibly die together (no geo separation, etc.). Thus resulting in producers simply not retrying when broker(s) not being reachable.
Any suggestions? Thanks.