RabbitMQ vs Kafka: a working web dev's honest take
Every few months someone drops "we should put Kafka on this" into a planning meeting, and every few months I have to remind myself what Kafka actually is versus what RabbitMQ is versus what my trusty Drupal queue does at 2am while nobody's watching. So I read the docs, the AWS and CloudAMQP comparisons, and a Confluent benchmark, and wrote down the version I wish someone had handed me.
Short answer: they look similar (both move messages between things) but they are built on genuinely different philosophies, and picking the wrong one is how you end up operating a distributed log to send welcome emails.
Two tools, two philosophies
RabbitMQ is a traditional message broker. It speaks AMQP, and its whole personality is a smart broker, dumb consumer. The broker does the thinking: it routes messages through exchanges, tracks acknowledgements, handles dead-letter logic, and pushes messages out to consumers over long-lived connections. A message gets consumed, the consumer ACKs, the broker deletes it. Job done, queue drains.
Kafka flips that. It's a distributed, append-only commit log with a dumb broker, smart consumer model. The broker mostly appends bytes to disk files (partitions) and does not track who read what. The intelligence lives in the consumer, which pulls batches and tracks its own position (an offset). Messages hang around for a retention window whether or not anyone read them.
Smart broker vs dumb broker, in practice
The difference shows up the moment you need routing. RabbitMQ gives you four exchange types out of the box (direct, fanout, topic, header), so you can fan a message to many queues or route by pattern without writing code. Kafka has no built-in routing; producers write to a topic, messages land in partitions by order received, and any "routing" is logic you build with consumer groups or Kafka Streams.
Push vs pull matters too. RabbitMQ pushes to consumers (you set a prefetch limit so you don't drown a slow worker). Kafka consumers pull batches from an offset, which changes how you think about backpressure.
The core split is persistence: Kafka is a log you can re-read; RabbitMQ is a queue that empties as you consume it.
The bit that actually matters: retention and replay
This is the deciding question for most projects. Do you need to replay messages after they've been processed? Event sourcing, CDC, audit trails, feeding three different analytics consumers from one stream? That's Kafka's native superpower, and it's awkward-to-impossible with classic RabbitMQ queues.
Worth knowing: RabbitMQ closed part of this gap. Since v3.9 it ships Streams (the rabbitmq_stream plugin), an append-only log with offset-based consumption that looks a lot like Kafka's model, built into the same broker. So "RabbitMQ can't replay" is now a misconception; its default queues can't, but Streams can.
Throughput, latency, ordering, delivery
Ballpark figures, and please treat these as orders of magnitude, not gospel:
- Throughput: Kafka appears to reach ~millions of messages/sec per broker via sequential disk I/O; RabbitMQ typically does thousands to tens of thousands and scales out with more brokers.
- Latency: below roughly 10 MB/s RabbitMQ often gives lower p50/p99 latency; a Confluent benchmark reports RabbitMQ latency degrading past ~30 MB/s while Kafka holds low latency at high throughput.
- Ordering: Kafka guarantees order within a partition, not across partitions. RabbitMQ guarantees order within a single queue with a single consumer, but competing consumers break it.
- Delivery: RabbitMQ offers at-most-once, at-least-once, and transactional. Kafka producers tune durability with
acks=0|1|alland support exactly-once semantics in-stream. - Ecosystem: Kafka has Kafka Connect and Kafka Streams; RabbitMQ has a plugin culture (management, shovel, federation, streams). Managed options: CloudAMQP and Amazon MQ for RabbitMQ; Confluent Cloud and Amazon MSK for Kafka.
A tiny producer, because talk is cheap
Here's a RabbitMQ producer in plain PHP via php-amqplib. It's basically what a Drupal queue worker does under the hood, minus the framework:
<?php
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;
$conn = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');
$channel = $conn->channel();
// Durable queue so jobs survive a broker restart.
$channel->queue_declare('emails', false, true, false, false);
$msg = new AMQPMessage(
json_encode(['to' => 'user@example.com', 'template' => 'welcome']),
['delivery_mode' => AMQPMessage::DELIVERY_MODE_PERSISTENT]
);
$channel->basic_publish($msg, '', 'emails');
echo "queued one email job\n";
$channel->close();
$conn->close();
The Kafka equivalent is conceptually different: you'd write to a topic like user-events split into, say, 6 partitions keyed by user ID, and any number of consumer groups read it independently at their own offsets. Same key always lands in the same partition, which is how you get per-user ordering.
So which one? (Honest answer: probably neither yet)
The most common misconception I ran into is that Kafka "just scales better" and is therefore the safe default. It scales horizontally more naturally, sure, but most of us never touch the volume where a well-fed RabbitMQ struggles. Success here is use-case fit, not architectural bragging rights.
My rough decision tree as a web dev:
- Background jobs, task queues, per-message workflow, complex routing? RabbitMQ.
- Need replay, long retention, many independent consumers, or genuine firehose throughput? Kafka (or RabbitMQ Streams if you only need a bit of it).
- Just deferring some work in a Drupal or Laravel app? Your framework's queue API on a database or Redis is fine. Don't page yourself for a message broker you didn't need.
Kafka is a fantastic tool. It's also a distributed system you now operate. Reach for it when the problem is actually shaped like a log, and let RabbitMQ (or your humble queue) handle the rest.