Lambda vs Kappa Architecture Explained

Many data platforms need two things that pull in opposite directions: complete, accurate results over all history, and fresh results within seconds. The Lambda and Kappa architectures are the two classic designs for getting both. They differ in one decision: whether batch and streaming are separate systems or the same one.

Lambda: two paths, merged

Lambda sends every incoming event down two paths (Microsoft Learn: big data architectures):

• The batch layer, or cold path, stores all raw data, immutable and append-only, and periodically recomputes accurate views over the whole dataset.
• The speed layer, or hot path, processes recent events in real time, favouring low latency over accuracy.
• The serving layer combines the two: batch views for history, speed-layer results for the period the batch has not yet covered.

The strength of the design is robustness. Raw data is never changed, so any view can be recomputed from scratch, and a bug in the streaming code is corrected by the next batch run.

The cost of two code paths

The weakness is that the same business logic lives in two places, usually in two frameworks. Jay Kreps, one of Kafka's creators, put it bluntly: maintaining code that needs to produce the same result in two complex distributed systems is exactly as painful as it seems (Kreps: Questioning the Lambda Architecture). The simulation below shows what that looks like in data, using a week of New York taxi trips (NYC TLC: trip record data) with the seventh day as "today":

import urllib.request
from pathlib import Path
import pandas as pd

name = "yellow_tripdata_2024-01.parquet"
if not Path(name).exists():
    urllib.request.urlretrieve(f"https://d37ci6vzurychx.cloudfront.net/trip-data/{name}", name)
events = pd.read_parquet(name, columns=["tpep_pickup_datetime", "fare_amount"])
events.columns = ["at", "fare"]
events = events[events["at"].between("2024-01-01", "2024-01-08", inclusive="left")].sort_values("at")
today = events["at"] >= "2024-01-07"                 # the day still in progress; earlier days are history

# Lambda: the same metric written twice, once per layer, by different code
def batch_layer(history):                            # nightly job: accurate, applies the fare rule
    return history[history["fare"] >= 0].groupby(history["at"].dt.date).size()

def speed_layer(recent):                             # streaming job: fast, written separately; the rule was missed
    return recent.groupby(recent["at"].dt.date).size()

serving = pd.concat([batch_layer(events[~today]), speed_layer(events[today])])
print("Lambda, today from the speed layer: ", serving.iloc[-1])
print("Lambda, after tonight's batch run:  ", batch_layer(events).iloc[-1])

# Kappa: one piece of logic over one ordered log; history is recomputed by replaying it
def stream_job(log, rule):
    counts = {}
    for at, fare in zip(log["at"].dt.date, log["fare"]):
        if rule(fare):
            counts[at] = counts.get(at, 0) + 1
    return pd.Series(counts)

v1 = stream_job(events, lambda fare: fare >= 0)
print("Kappa, today:                       ", v1.iloc[-1])
v2 = stream_job(events, lambda fare: fare > 0)       # the rule changes: replay the log with the new code
print("Kappa, replayed with a new rule:    ", v2.iloc[-1], "| days recomputed:", len(v2))
print("week total: batch layer", batch_layer(events).sum(), "| Kappa v1", v1.sum())

Lambda, today from the speed layer:  67543
Lambda, after tonight's batch run:   66561
Kappa, today:                        66561
Kappa, replayed with a new rule:     66551 | days recomputed: 7
week total: batch layer 601270 | Kappa v1 601270

In the Lambda design, the batch job applies a rule, exclude negative fares, that the separately written streaming job lacks. During the day the dashboard shows 67,543 trips; after the nightly batch replaces the speed layer's figure, the same day shows 66,561. Nothing crashed, yet the number moved by 982 overnight. Teams running Lambda systems commonly meet versions of this: rounding done differently, a filter added to one path only, time zones handled in two ways.

Kappa: one path, replayed

Kappa keeps the idea of an immutable log and drops the second path. All data flows through a single stream processing system; when history must be recomputed, the stream is replayed. In the simulation, one function holds the logic, so today's count is 66,561 from the start, with no overnight correction. When the rule changes, to exclude zero fares as well, the log is replayed through the new code and all seven days are recomputed consistently.

Kreps's recipe for doing this in production: retain the full log, start a second instance of the stream job from the beginning with the new code, write its output to a new table, switch readers to that table once it has caught up, then stop the old job. The requirement is a log that keeps enough history, such as Kafka with long retention or events mirrored to cheap storage, and a stream processor that can replay it quickly in parallel.

Side by side

What changed since these were named

The debate was sharpest when batch and streaming needed different engines. Two developments have softened it:

• Unified engines. Frameworks such as Spark and Flink run the same transformation code over bounded and unbounded data, so a team can keep batch and streaming jobs without writing the logic twice.
• Lakehouse table formats. Tables that accept streaming writes and batch reads, with transactions, let one stored copy serve both a low-latency job and a nightly recomputation; data lake vs data warehouse vs lakehouse explains the foundation.

Many current platforms are therefore neither pure design: a single set of transformation logic, a streaming path for freshness, and scheduled jobs that restate recent periods to absorb corrections and late events, as handling late-arriving data describes.

How to choose

• Do you need results in seconds at all? If hourly is fine, a well-run batch pipeline is simpler than either architecture; batch vs streaming covers that decision.
• Is your data naturally a stream of events, with a durable log? Kappa keeps one code path; Apache Kafka explained covers the log.
• Do you depend on large historical recomputations or logic that is hard to express in streaming? Lambda, with shared libraries and tests that run the same cases through both paths.
• Whichever you choose, keep raw events immutable and replayable. That property, common to both designs, is what makes mistakes recoverable.

For the wider picture of how analytical platforms are organised, see data warehouses explained.

Batch versus real-time processing, Apache Kafka, stream processing concepts and a real-time streaming pipeline project are part of the Data Engineering course.

Sources

• Big data architectures (Azure Architecture Center, Microsoft Learn) — Microsoft Learn
• Questioning the Lambda Architecture (Jay Kreps, O'Reilly Radar, 2014) — O'Reilly
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission