Kafka Interview Questions Explained

Kafka interview questions are rarely about installing Kafka. They test whether you can predict what happens to a message: which partition it lands in, in what order consumers see it, whether a crash can make it be processed twice, and how far behind a consumer is. The answers below come from a real Kafka 4 broker, running in KRaft mode without ZooKeeper, fed with a real hour of New York taxi pickups, 08:00 to 09:00 on 16 January 2025 (NYC TLC: trip record data), each message keyed by its pickup zone.

import time
from collections import defaultdict

import pandas as pd
from confluent_kafka import Consumer, Producer, TopicPartition
from confluent_kafka.admin import AdminClient, ConfigResource, NewTopic

BROKER = {"bootstrap.servers": "127.0.0.1:9092"}               # a single Kafka 4 broker in KRaft mode, in Docker
admin = AdminClient(BROKER)
admin.delete_topics(["trips"])
for future in admin.delete_consumer_groups(["audit", "billing"]).values():   # start from a clean slate on every run
    try:
        future.result()
    except Exception:
        pass                                                     # the groups do not exist yet on the first run
time.sleep(2)
admin.create_topics([NewTopic("trips", num_partitions=3, replication_factor=1)])["trips"].result()

trips = pd.read_parquet("yellow_tripdata_2025-01.parquet", columns=["tpep_pickup_datetime", "PULocationID"])
hour = trips[(trips.tpep_pickup_datetime >= "2025-01-16 08:00") & (trips.tpep_pickup_datetime < "2025-01-16 09:00")]
hour = hour.sort_values("tpep_pickup_datetime")                  # NYC TLC: a real busy hour, in time order

producer = Producer({**BROKER, "enable.idempotence": True})     # idempotent producer: no duplicates from retries
for i, (ts, zone) in enumerate(zip(hour.tpep_pickup_datetime, hour.PULocationID)):
    producer.produce("trips", key=str(zone), value=f"{i}|{ts:%H:%M:%S}")
producer.flush()


def consume(group, limit, commit_after=None, reset="earliest"):
    c = Consumer({**BROKER, "group.id": group, "auto.offset.reset": reset, "enable.auto.commit": False})
    c.subscribe(["trips"])
    got, waited = [], 0
    while len(got) < limit:
        m = c.poll(1)
        if m is None:                                            # joining the group takes a few seconds at first
            waited += 1
            if got or waited > 30:
                break
            continue
        got.append(m)
        if commit_after and len(got) == commit_after:
            c.commit(asynchronous=False)                         # offsets saved up to here
    return c, got


print(f"K1 {len(hour):,} real pickups produced, keyed by pickup zone, to a topic with 3 partitions")
reader, everything = consume("audit", len(hour) + 10)
partitions_per_key, per_partition = defaultdict(set), defaultdict(int)
for m in everything:
    partitions_per_key[m.key()].add(m.partition())
    per_partition[m.partition()] += 1
print(f"   keys found in more than one partition: {sum(len(p) > 1 for p in partitions_per_key.values())} of {len(partitions_per_key)}")
print(f"   messages per partition: {dict(sorted(per_partition.items()))}")

seq = defaultdict(list)
for m in sorted(everything, key=lambda m: (m.partition(), m.offset())):
    seq[m.key()].append(int(m.value().decode().split("|")[0]))
print(f"\nK2 per-key order preserved within its partition: {all(s == sorted(s) for s in seq.values())}")
reader.close()

print("\nK3 at-least-once: a consumer commits after 50 of 100 messages, then crashes before committing again")
crashing, first = consume("billing", 100, commit_after=50)
crashing.close()                                                 # no final commit: the work after 50 is not recorded
restarted, second = consume("billing", 100)
seen_twice = {(m.partition(), m.offset()) for m in first} & {(m.partition(), m.offset()) for m in second}
print(f"   processed before the crash: {len(first)} | messages processed again after restart: {len(seen_twice)}")
restarted.close()

print("\nK4 consumer lag for the 'billing' group (end offset minus committed offset)")
probe = Consumer({**BROKER, "group.id": "billing"})
committed = probe.committed([TopicPartition("trips", p) for p in range(3)])
lag = sum(probe.get_watermark_offsets(tp)[1] - max(tp.offset, 0) for tp in committed)
print(f"   total lag: {lag:,} messages")
probe.close()

config = admin.describe_configs([ConfigResource("topic", "trips")])
retention = list(config.values())[0].result()["retention.ms"].value
print(f"\nK5 topic retention.ms: {int(retention):,} ({int(retention) / 86_400_000:.0f} days); replay = a new group from 'earliest'")

K1 6,677 real pickups produced, keyed by pickup zone, to a topic with 3 partitions
   keys found in more than one partition: 0 of 192
   messages per partition: {0: 2348, 1: 2693, 2: 1636}

K2 per-key order preserved within its partition: True

K3 at-least-once: a consumer commits after 50 of 100 messages, then crashes before committing again
   processed before the crash: 100 | messages processed again after restart: 50

K4 consumer lag for the 'billing' group (end offset minus committed offset)
   total lag: 6,627 messages

K5 topic retention.ms: 604,800,000 (7 days); replay = a new group from 'earliest'

K1. How does Kafka decide which partition a message goes to?

By its key. Messages with the same key always go to the same partition (Kafka: introduction). Across 6,677 pickups from 192 zones, no zone appeared in more than one partition. The cost is uneven load: the three partitions received 2,348, 2,693 and 1,636 messages, because zones differ hugely in traffic and their hashes do not balance perfectly. Messages without a key are spread across partitions instead, which balances load but gives up per-key ordering. A hot key, such as one very busy zone, can overload one partition however many partitions the topic has.

K2. Does Kafka guarantee ordering?

Within a partition, yes; across partitions, no. Every zone's messages were read back in the order they were produced, because each zone lives in a single partition. A consumer reading three partitions sees them interleaved, though, so there is no global order across zones. If order matters per customer or per device, use that ID as the key; if it matters globally, you need a single partition and accept its throughput limit.

K3. Can a message be processed twice?

Yes, with the usual at-least-once setup. A consumer that processes messages and then commits its position can crash in between; its replacement starts from the last committed offset and repeats the work (Kafka: design). Here a consumer processed 100 messages but committed after 50, then crashed. The restarted consumer in the same group received the last 50 again. Committing before processing would avoid duplicates but could lose messages, which is at-most-once.

What interviewers want to hear: make processing idempotent, for example with upserts by message key or a store of processed IDs, so redelivery is harmless. On the producing side, the idempotent producer used here stops retries from writing duplicates into the log.

K4. What is consumer lag?

The number of messages between the end of a partition and the group's committed offset, summed across partitions: how far behind the group is. After the crash, the billing group had committed only 50 offsets, leaving a lag of 6,627 messages. Rising lag is the main health signal for a streaming consumer: it means processing is slower than production, or that a consumer has stopped.

K5. How long does Kafka keep messages, and how do you replay them?

Until the retention limit, by time or size, whether or not they have been consumed. The topic's retention.ms was 604,800,000, seven days, the default. Consuming does not delete anything, which is what makes replay possible: a new consumer group with auto.offset.reset=earliest reads the topic from the start, as the audit consumer in K1 did. That is how teams rebuild a downstream table after a bug, as long as the retention period is longer than the time it takes to notice the bug.

Other questions to expect

• How many consumers can share the work? Up to one per partition in a group; extra consumers sit idle. Partition count is therefore a ceiling on parallelism.
• What does replication add? Each partition is copied to several brokers; with acks=all and enough in-sync replicas, a write survives a broker failure. A single-broker setup, like this one, cannot show that.
• What happens during a rebalance? When a consumer joins, leaves or stops responding, the group's partitions are reassigned. The consumers that lose partitions should commit what they have processed first, or their replacements will repeat it, the same effect as the crash in K3.
• How does exactly-once processing work? The producer writes its output and the consumer's offsets in one transaction, and downstream consumers read only committed transactions. It covers Kafka-to-Kafka processing; writes to an external database still need idempotent upserts.
• What is log compaction? Retention that keeps only the latest message per key, so a topic can represent the current state of each key.

How to answer Kafka questions

• Choose keys for the ordering you need, and check the skew they create.
• Assume at-least-once and make consumers idempotent.
• Watch lag as the first sign of trouble.

Pipeline design with Kafka is covered in building a streaming pipeline with Kafka and Spark and real-time analytics architecture. Capacity and partition skew are worked in system design interview questions for data engineers, and Google Cloud's messaging service in GCP interview questions for data engineers.

Streaming with Kafka is part of the Data Engineering course.

Sources

• Introduction (Apache Kafka documentation) — The Apache Software Foundation
• Design (Apache Kafka documentation) — The Apache Software Foundation
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission