Batch vs Streaming Data Processing: A Clear Comparison

Batch and streaming are two answers to the question "when do we process data?". Batch waits, collects a bounded set of data, and processes it in one go. Streaming processes each event, or each small group of events, as it arrives. The data and often the logic are the same; what changes is how fresh the result is, and what that freshness costs.

How each works

Batch processing runs jobs over data at rest: read files or tables for a period, transform them, write the output. Microsoft's architecture guide describes it as long-running jobs that filter, aggregate and prepare data for analysis, and notes the drawback plainly: if processing takes hours, results are hours old (Microsoft Learn: big data architectures).

Stream processing handles unbounded data in motion. Events are captured by a message ingestion layer, such as Kafka, and a stream processor filters, aggregates and enriches them continuously, writing to a sink. Where ETL and ELT operate on datasets in scheduled batches, streaming processes data as it arrives (Microsoft Learn: ETL).

Micro-batching sits between them: a stream is cut into very small batches, every few seconds or minutes, and each is processed with batch-style logic. Spark Structured Streaming works this way by default.

Freshness, measured

Abstract talk about latency hides how different the options feel. Take one real day, 15 January 2024, of New York taxi trips (NYC TLC: trip record data), and ask what each kind of pipeline would know when someone opens a dashboard at 12:30:

import urllib.request
from pathlib import Path
import pandas as pd

name = "yellow_tripdata_2024-01.parquet"
if not Path(name).exists():
    urllib.request.urlretrieve(f"https://d37ci6vzurychx.cloudfront.net/trip-data/{name}", name)
trips = pd.read_parquet(name, columns=["tpep_pickup_datetime", "tpep_dropoff_datetime", "fare_amount"])
day = trips[trips["tpep_pickup_datetime"].between("2024-01-15", "2024-01-16", inclusive="left")]
done = day["tpep_dropoff_datetime"]                       # a trip is known to the system once it ends

asked_at = pd.Timestamp("2024-01-15 12:30")               # someone opens the dashboard at 12:30
truth = int((done <= asked_at).sum())
print(f"trips completed today by 12:30: {truth}")
print(f"{'pipeline':<28}{'last run':>9}{'trips it knows':>16}{'missing':>9}{'runs a day':>12}{'trips per run':>15}")
pipelines = [  # (name, when it last ran, trips of today it has loaded, runs in a day)
    ("nightly batch at 01:00", pd.Timestamp("2024-01-15 01:00"), 0, 1),            # that run loaded yesterday's trips
    ("hourly batch", asked_at.floor("1h"), None, 24),
    ("micro-batch every 5 minutes", asked_at.floor("5min"), None, 288),
    ("streaming, per event", asked_at, None, len(day)),
]
for label, last, loaded, runs in pipelines:
    known = loaded if loaded is not None else int((done <= last).sum())
    print(f"{label:<28}{last.strftime('%H:%M'):>9}{known:>16}{truth - known:>9}{runs:>12}{len(day) / runs:>15.1f}")

trips completed today by 12:30: 23090
pipeline                     last run  trips it knows  missing  runs a day  trips per run
nightly batch at 01:00          01:00               0    23090           1        77033.0
hourly batch                    12:00           20504     2586          24         3209.7
micro-batch every 5 minutes     12:30           23090        0         288          267.5
streaming, per event            12:30           23090        0       77033            1.0

By 12:30, 23,090 trips had finished. The nightly batch knows none of them; today is invisible until tomorrow. The hourly batch last ran at noon and is missing 2,586 trips, about eleven percent. The five-minute micro-batch and the per-event stream are both complete at this moment.

The right-hand columns show the other side of the trade. The nightly job runs once over 77,033 trips. The hourly job runs 24 times. The five-minute pipeline runs 288 times a day, each over a few hundred trips, and the per-event stream handles each trip individually. More runs mean more scheduling, more small writes, and compute that must be available all day, not for an hour at night.

Notice also where the gains flatten. Going from nightly to hourly recovers most of the day, 89 percent of the trips. Going from hourly to five minutes recovers the last eleven percent. Going from five minutes to per-event recovers, at 12:30, nothing a dashboard reader would notice. Each step costs more than the one before and delivers less.

What streaming really costs

Freshness is not the hard part of streaming; correctness is. A batch job sees a complete, closed period. A streaming job never does, so it must handle problems batch can ignore:

• Late and out-of-order events. A window cannot simply close at the top of the hour; a watermark decides how long to wait, and results may be delayed or revised (Spark docs: streaming APIs). Handling late-arriving data measures how much arrives late.
• Duplicates on restart. Recovery replays events since the last checkpoint, so outputs must be idempotent.
• State. Aggregations keep running state that must be stored, checkpointed and bounded.
• Operations. A stream is always on. Failures at 3 am need alerts and someone to respond; a failed nightly job can usually wait until morning.
• Reprocessing. Fixing a bug means replaying history through the stream, or running a parallel batch job; Lambda vs Kappa covers both approaches.

Microsoft's guidance for streaming reliability lists exactly these: checkpointing, idempotent transformations, watermarks for late events and dead-letter queues.

Side by side

How to choose

Start from the decision the data supports, not from the technology:

• Who acts on this, and how quickly? A weekly planning report needs daily data. A fraud check needs an answer before the payment completes.
• What does an hour of delay cost? If nothing changes, batch is correct, not merely cheaper.
• Can the source deliver events? Streaming needs changes as they happen, from event producers or change data capture.
• Can the team run it? An always-on pipeline needs monitoring and on-call cover.

A practical default: build batch first, shorten the schedule to hourly or to micro-batches when freshness matters, and reserve true event-at-a-time streaming for the few uses that need a reaction in seconds. Many platforms end up with all three, each where it pays.

To see a stream built end to end, read building a streaming pipeline with Kafka and Spark; for the platform underneath, Apache Kafka explained.

Batch versus real-time processing, Spark Streaming, Apache Kafka basics and real-time data pipelines are part of the Data Engineering course.

Sources

• Big data architectures (Azure Architecture Center, Microsoft Learn) — Microsoft Learn
• Extract, transform, load (ETL) (Azure Architecture Center, Microsoft Learn) — Microsoft Learn
• Structured Streaming APIs on DataFrames and Datasets (Apache Spark documentation) — Apache Spark
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission