Kafka as a Message Queue - can you do it, and should you do it?
This talk has been (or will be) presented at:
Using Kafka's offset commit mechanism we can implement a message processing system with at-least-once delivery guarantees. However, we can only acknowledge processing of all messages up to a given point (offset). That’s very often enough, but not always.
Sometimes we need selective, out-of-order message acknowledgments, like in a “traditional” message queue. If a message is not acknowledged for a given period of time, it should be re-delivered. Can this be implemented on top of Kafka? Sure! (By the way: this is similar to how Amazon’s SQS works.)
In the talk I’ll describe the architecture and implementation of a message queue built on Kafka: kmq. We’ll go through two crucial components: the queue client and the message redelivery tracker. There will be some live coding, some slides, and a couple of demos.
We’ll also look at the performance (which is surprisingly good) & latency, as well as possible problems that using such an approach can cause, such as “error-flooding”.
Kmq is open-source and available at github.com/softwaremill/kmq.
Slides
Video
If you want to dig deeper and compare characteristics of Kafka and other message queues read this article: Evaluating persistent, replicated message queues.