Kafka vs RabbitMQ vs Pulsar: A Comparison

Kafka, RabbitMQ and Pulsar all move messages from producers to consumers, and all three can be made to do each other's jobs. But each was designed around a different idea, and choosing well means knowing which idea fits your workload. The central distinction is queue versus log.

Queue and log, on live brokers

A queue hands each message to one consumer and removes it once acknowledged. A log keeps every event in order and lets each reader keep its own position. The script sends the same 1,000 events, the first 1,000 trips of a public taxi dataset, to a RabbitMQ queue and a Kafka topic, then reads each with two independent readers, a billing service and an audit service:

import json
import urllib.request
import uuid
from pathlib import Path

import pika
import pyarrow.parquet as pq
from confluent_kafka import Consumer, Producer
from confluent_kafka.admin import AdminClient, NewTopic

name = "yellow_tripdata_2024-01.parquet"
if not Path(name).exists():
    urllib.request.urlretrieve(f"https://d37ci6vzurychx.cloudfront.net/trip-data/{name}", name)
rows = pq.read_table(name, columns=["PULocationID", "fare_amount"]).slice(0, 1000).to_pylist()
events = [json.dumps({"zone": r["PULocationID"], "fare": r["fare_amount"]}) for r in rows]
run = uuid.uuid4().hex[:8]                                       # fresh queue and topic names for each run

# RabbitMQ: a queue. A message is delivered to one consumer and removed once acknowledged.
channel = pika.BlockingConnection(pika.ConnectionParameters("localhost")).channel()
queue = f"trips-{run}"
channel.queue_declare(queue=queue, durable=True)
for event in events:
    channel.basic_publish(exchange="", routing_key=queue, body=event)

def drain(label):
    count = 0
    while True:
        method, _, body = channel.basic_get(queue=queue)
        if method is None:
            break
        channel.basic_ack(method.delivery_tag)                   # acknowledged: the broker deletes it
        count += 1
    print(f"RabbitMQ {label}: {count} messages")

drain("billing service reads")
drain("audit service reads")

# Kafka: a log. Events stay on the topic; each consumer group has its own position.
BROKER, topic = "localhost:9092", f"trips-{run}"
admin = AdminClient({"bootstrap.servers": BROKER})
admin.create_topics([NewTopic(topic, num_partitions=1, replication_factor=1)])[topic].result()
producer = Producer({"bootstrap.servers": BROKER})
for event in events:
    producer.produce(topic, value=event)
producer.flush()

def read(group):
    consumer = Consumer({"bootstrap.servers": BROKER, "group.id": f"{group}-{run}", "auto.offset.reset": "earliest"})
    consumer.subscribe([topic])
    count, idle = 0, 0
    while idle < 5:
        msg = consumer.poll(1.0)
        if msg is None:
            idle += 1
        elif not msg.error():
            count, idle = count + 1, 0
    consumer.close()
    print(f"Kafka {group} group reads: {count} events")

read("billing")
read("audit")

RabbitMQ billing service reads: 1000 messages
RabbitMQ audit service reads: 0 messages
Kafka billing group reads: 1000 events
Kafka audit group reads: 1000 events

The RabbitMQ queue gave all 1,000 messages to the first reader. Once acknowledged they were gone, and the second reader found the queue empty. The Kafka topic gave all 1,000 events to both groups, because events are retained by policy and not deleted when consumed (Kafka docs: introduction).

Neither behaviour is a defect. A queue is right when a message is a task that must be done once: send this email, resize that image. A log is right when events are facts that several systems need: an order was placed, and billing, analytics and fraud detection each want to know.

RabbitMQ: flexible routing for tasks

RabbitMQ implements a routing model. Publishers send messages to exchanges, and exchanges route copies to queues according to bindings, with direct, fanout, topic and headers exchange types (RabbitMQ docs: AMQP concepts). A queue is a first-in-first-out collection; a message is removed once a consumer acknowledges it, and several consumers can share a queue to spread work (RabbitMQ docs: queues).

So the way to give both billing and audit a copy in RabbitMQ is not to share a queue but to bind two queues to a fanout exchange, one per service. That works well, but each consumer must exist before the message is published; a service added next month cannot read last month's messages. RabbitMQ's strengths are routing flexibility, per-message acknowledgement and redelivery, and simple operation for moderate volumes. Replicated quorum queues provide durability. It also offers streams, an append-only log queue type, for cases that need replay.

Kafka: a durable, replayable log

Kafka's unit is the partitioned topic. Events with the same key go to the same partition and are read in the order written, and consumer groups track their own offsets; Apache Kafka explained shows this in detail. Its strengths follow from the log: very high throughput, long retention, replay for new consumers and reprocessing, and a large ecosystem of connectors and stream processors. Its costs: no per-message routing logic inside the broker, ordering only within a partition, partition counts that must be planned, and more operational weight than a simple queue.

Pulsar: log and queue with separated storage

Pulsar aims to cover both models. It is publish-subscribe, with four subscription types: exclusive and failover for ordered, log-style reading, shared for queue-style round-robin delivery across consumers, and key_shared, which delivers messages with the same key to one consumer (Pulsar docs: messaging). By default acknowledged messages are deleted, as in a queue, and a retention policy keeps them, as in a log.

Its architecture also differs. Brokers are stateless and serve messages, while Apache BookKeeper stores them, with a separate metadata store; multi-tenancy and geo-replication are built in (Pulsar docs: architecture). Separating serving from storage lets each scale independently. The price is more components to run and a smaller community and ecosystem than Kafka's.

Side by side

How to choose

• Background jobs and service-to-service tasks, with routing rules and per-message retries: RabbitMQ.
• Event streams feeding analytics, several consumers per event, replay and high volume: Kafka. It is also the default for data engineering, so skills and tooling are plentiful; building a streaming pipeline with Kafka and Spark shows a typical use.
• One platform for both queueing and streaming, many tenants, or geo-replication as a core need: evaluate Pulsar.
• Already running one of them well? Staying put is usually right. Each can cover part of the others' ground, and operating two messaging systems costs more than most feature gaps.

Whichever you choose, delivery is normally at-least-once, so consumers must tolerate duplicates. How the choice fits a wider low-latency design is covered in real-time analytics: architecture and tools.

Apache Kafka basics, stream processing concepts and event-driven architecture are part of the streaming module of the Data Engineering course.

Sources

• Introduction (Apache Kafka documentation) — Apache Kafka
• Queues (RabbitMQ documentation) — RabbitMQ
• AMQP 0-9-1 model explained (RabbitMQ documentation) — RabbitMQ
• Architecture overview (Apache Pulsar documentation) — Apache Pulsar
• Messaging concepts (Apache Pulsar documentation) — Apache Pulsar