Apache Kafka Explained: Streaming Data Basics

Apache Kafka is the backbone of most streaming data platforms. Its own introduction describes event streaming as capturing data in real time from sources such as databases, sensors and applications as streams of events, and Kafka as a platform to publish and subscribe to those streams, store them durably, and process them (Kafka docs: introduction). The storage part is what sets it apart from a traditional message queue, and it is easiest to understand by watching a broker work.

The vocabulary

• Event: a record of something that happened, with a key, a value and a timestamp.
• Topic: a named stream of events, such as trips.
• Partition: a topic is split into partitions, each an append-only log. Partitions are how Kafka scales.
• Offset: an event's position within its partition.
• Producer and consumer: applications that write and read events, fully decoupled from each other.
• Consumer group: a set of consumers sharing the work of reading a topic, with their progress tracked per group.
• Broker: a Kafka server. A cluster has several, and partitions are replicated across them.

Kafka in action

The script creates a topic with three partitions on a local broker, produces the first 1,000 trips of January 2024 from New York's taxi data (NYC TLC: trip record data) as events keyed by pickup zone, then reads them with consumer groups. A single broker is enough to try it: docker run -d -p 9092:9092 apache/kafka:latest starts one.

import json
import urllib.request
import uuid
from collections import Counter
from pathlib import Path
import pyarrow.parquet as pq
from confluent_kafka import Consumer, Producer
from confluent_kafka.admin import AdminClient, NewTopic

name = "yellow_tripdata_2024-01.parquet"
if not Path(name).exists():
    urllib.request.urlretrieve(f"https://d37ci6vzurychx.cloudfront.net/trip-data/{name}", name)
trips = pq.read_table(name, columns=["tpep_pickup_datetime", "PULocationID", "fare_amount"]).slice(0, 1000).to_pylist()

BROKER = "localhost:9092"
topic = f"trips-{uuid.uuid4().hex[:8]}"                      # a fresh topic for each run
admin = AdminClient({"bootstrap.servers": BROKER})
admin.create_topics([NewTopic(topic, num_partitions=3, replication_factor=1)])[topic].result()

# produce: the key decides the partition, so all events for one zone stay in order
producer = Producer({"bootstrap.servers": BROKER})
for trip in trips:
    event = {"pickup": trip["tpep_pickup_datetime"].isoformat(), "zone": trip["PULocationID"], "fare": trip["fare_amount"]}
    producer.produce(topic, key=str(trip["PULocationID"]), value=json.dumps(event))
producer.flush()

def read_all(group):
    consumer = Consumer({"bootstrap.servers": BROKER, "group.id": group, "auto.offset.reset": "earliest",
                         "enable.auto.commit": False})
    consumer.subscribe([topic])
    messages, idle = [], 0
    while idle < 5:
        msg = consumer.poll(1.0)
        if msg is None:
            idle += 1
        elif not msg.error():
            messages.append(msg)
            idle = 0
    if messages:
        consumer.commit(asynchronous=False)                  # record how far this group has read
    consumer.close()
    return messages

group = f"dashboard-{uuid.uuid4().hex[:8]}"
first = read_all(group)
per_partition = Counter(m.partition() for m in first)
print("events read:", len(first), "| per partition:", dict(sorted(per_partition.items())))
zones = {}
for m in first:
    zones.setdefault(m.key().decode(), set()).add(m.partition())
print("zones:", len(zones), "| zones spread over more than one partition:", sum(len(p) > 1 for p in zones.values()))
jfk = [m.offset() for m in first if m.key() == b"132"]
print("JFK events:", len(jfk), "| offsets strictly increasing:", jfk == sorted(set(jfk)))
print("same group reads again:", len(read_all(group)), "events")
print("a new group reads:", len(read_all(f"audit-{uuid.uuid4().hex[:8]}")), "events")

events read: 1000 | per partition: {0: 256, 1: 478, 2: 266}
zones: 66 | zones spread over more than one partition: 0
JFK events: 21 | offsets strictly increasing: True
same group reads again: 0 events
a new group reads: 1000 events

What the output shows

Keys decide partitions, and partitions decide order. The 1,000 events spread across the three partitions, 256, 478 and 266, but not at random: each of the 66 pickup zones landed entirely in one partition, and none was split. Kafka's guarantee is that events with the same key are written to the same partition and read in exactly the order they were written. The 21 JFK Airport events have strictly increasing offsets, so a consumer sees them in order. Across different partitions there is no ordering guarantee, which is why the choice of key matters: key by the entity whose events must stay in sequence, a customer, an order, a device.

The split is uneven, 478 against 256. Partitions balance keys, not volume. Partition 1 happens to hold the busier zones. Hot keys cause the same skew in Kafka as in any partitioned system.

Reading does not remove anything. The first group read all 1,000 events and committed its offsets. Reading again with the same group returned nothing, because the group resumes after its committed position. A brand-new group read all 1,000 from the beginning. Kafka's documentation is explicit that events are not deleted after consumption; they are kept for a configured retention period, so any number of applications can read the same stream independently, and a new one can replay history. That one property is what makes Kafka a log and not a queue.

Consumer groups and scaling

Within a group, each partition is read by one consumer at a time. Run three consumers in the group above and each takes one partition; run one and it takes all three. That is how consumption scales, and it sets a ceiling: more consumers than partitions leaves the extras idle, so partition count is chosen with future parallelism in mind.

Durability and delivery guarantees

In production, each partition is replicated across brokers, commonly three copies, so a broker can fail without losing data. Producers choose how many replicas must acknowledge a write before it counts as sent.

Delivery is normally at-least-once. The example commits offsets after reading; if a consumer crashes after processing events but before committing, the group reads those events again on restart. Microsoft's guidance on the idempotent consumer pattern lists Kafka among the brokers where this applies, and recommends making consumers safe to repeat (Microsoft Learn: idempotent consumer). Committing before processing gives at-most-once instead: no duplicates, but events can be lost.

The wider platform

Kafka has five core APIs: Admin, Producer and Consumer, used above, plus Kafka Connect, which moves data between Kafka and databases or storage with ready-made connectors, and Kafka Streams, a library for transforming streams. Many teams process Kafka topics with Spark or Flink instead; building a streaming pipeline with Kafka and Spark shows that.

When Kafka is the wrong tool

• A few jobs exchanging tasks: a simple queue is easier; Kafka vs RabbitMQ vs Pulsar compares them.
• Data needed hourly or daily: batch is simpler and cheaper; see batch vs streaming.
• No team to operate it: a cluster needs monitoring, capacity planning and upgrades; managed services reduce but do not remove that work.

Where Kafka sits in a full low-latency design is covered in real-time analytics: architecture and tools, and the ideas underneath it, partitioning, replication and retries, in distributed systems basics.

Apache Kafka basics, stream processing concepts, real-time data pipelines and event-driven architecture are part of the streaming module of the Data Engineering course.

Sources

• Introduction (Apache Kafka documentation) — Apache Kafka
• Idempotent Consumer pattern (Azure Architecture Center, Microsoft Learn) — Microsoft Learn
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission