Batch vs Real-Time Model Inference: Tradeoffs

Every model that goes into production answers one design question early: are predictions made ahead of time, in batches, or at the moment someone asks for them? A churn score can be recalculated nightly. A fraud check must happen while the payment waits. Most cases are less obvious, and the choice decides cost, complexity and how stale a prediction can be.

This guide measures the trade-off on real data and turns it into a decision guide.

Two patterns

Cloud platforms offer both, and options between them. Amazon's SageMaker, for example, offers real-time endpoints for interactive low-latency use, serverless and asynchronous options, the latter for large payloads and long processing (AWS: deploy models), and batch transform for large datasets when no persistent endpoint is needed (AWS: batch transform).

Measuring what freshness is worth

The test uses hourly bike rentals from the UCI Bike Sharing data, where the weather is recorded every hour, released under CC BY 4.0 (UCI: Bike Sharing). One model scores July to December 2012 two ways. The nightly batch predicts every hour of a day at midnight, so it can only use midnight's weather. The real-time version predicts each hour as it comes, with that hour's weather.

import time

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error

# UCI Bike Sharing, hourly counts (CC BY 4.0)
hour = pd.read_csv("data/hour.csv")
FEATURES = ["season", "yr", "mnth", "hr", "holiday", "weekday", "workingday", "weathersit", "temp", "atemp", "hum", "windspeed"]
WEATHER = ["weathersit", "temp", "atemp", "hum", "windspeed"]
train, test = hour[hour.dteday < "2012-07-01"], hour[hour.dteday >= "2012-07-01"].copy()
model = HistGradientBoostingRegressor(random_state=42).fit(train[FEATURES], train.cnt)

# Real-time: each prediction is made when it is needed, with the weather of that hour
realtime = model.predict(test[FEATURES])

# Nightly batch: every hour of the day is scored at midnight, with the weather known at midnight
midnight = test.groupby("dteday")[WEATHER].transform("first")
batch_input = test[FEATURES].copy()
batch_input[WEATHER] = midnight
nightly = model.predict(batch_input)

changed = (test.weathersit != midnight.weathersit).mean()
print(f"{len(test):,} hours scored, July to December 2012")
print(f"hours whose weather category differed from midnight's: {changed:.0%}")
print(f"MAE with nightly batch predictions: {mean_absolute_error(test.cnt, nightly):6.1f} rentals an hour")
print(f"MAE with real-time predictions:     {mean_absolute_error(test.cnt, realtime):6.1f} rentals an hour")

worst = test.assign(gap=np.abs(nightly - realtime)).sort_values("gap", ascending=False).iloc[0]
print(f"largest single difference: {worst.dteday} {int(worst.hr)}:00, "
      f"weather category {int(midnight.loc[worst.name, 'weathersit'])} at midnight, {int(worst.weathersit)} by then")

# The cost side: scoring everything at once versus one request at a time
rows = test[FEATURES]
start = time.perf_counter()
model.predict(rows)
bulk = time.perf_counter() - start
start = time.perf_counter()
for i in range(500):
    model.predict(rows.iloc[[i]])
single = (time.perf_counter() - start) / 500
print(f"speed on this laptop: all {len(rows):,} rows in {bulk * 1000:.0f} ms, one row at a time {single * 1000:.2f} ms each")

4,376 hours scored, July to December 2012
hours whose weather category differed from midnight's: 35%
MAE with nightly batch predictions:   63.2 rentals an hour
MAE with real-time predictions:       44.5 rentals an hour
largest single difference: 2012-10-04 17:00, weather category 3 at midnight, 1 by then
speed on this laptop: all 4,376 rows in 14 ms, one row at a time 7.79 ms each

What the output shows

• The weather often changed after midnight. In 35% of hours the weather category was different from midnight's.
• Fresh inputs cut the error substantially. Real-time predictions were off by 44.5 rentals an hour on average; nightly ones by 63.2, about 42% more.
• The worst miss was a classic. On 4 October 2012 it was raining at midnight and clear by 5 pm, the evening rush. The nightly prediction planned for rain.
• Batch scoring is extremely efficient. All 4,376 rows took 14 ms in one call, about 3 microseconds a row, while single-row calls took about 8 ms each on this laptop, thousands of times more per prediction. The exact timings vary between runs, but the gap does not.

So the question is not which pattern is better, but whether the value of fresh inputs is worth the cost of computing on demand.

Measure freshness on your own data

The comparison above can be repeated for almost any model before anything is built. Replay a period of history twice. In the first pass, give the model only what it would have known at batch time, such as yesterday's or midnight's values. In the second, give it what it would know at the moment of the request. The difference in error is the value of freshness for your problem.

Then translate it into money or outcomes. Fewer stock-outs, fewer missed fraud cases or happier riders may or may not pay for an always-on service. Both passes must use only data that existed at the time of each prediction, or the comparison will flatter the fresh version.

Choosing

Hybrids

Many systems mix the two:

• Precompute, then adjust: a nightly score corrected at request time with a few fresh features.
• Precompute, then refresh on change: rescore an item only when its key inputs change.
• Cache real-time results for repeated requests within a short window.

Each hybrid needs care to keep features consistent between the batch and real-time paths. A feature computed one way at night and another way on request quietly breaks the model. See real-time feature engineering.

Operating each pattern

Batch jobs need scheduling, retries and a check that outputs are complete and fresh. Real-time services need autoscaling, timeouts, fallbacks and latency monitoring. For building the service, see serving ML models with FastAPI and Docker. For watching either kind in production, see monitoring ML models in production.

Common mistakes

• Defaulting to real time for predictions nobody needs instantly.
• Defaulting to batch when key inputs change during the day.
• Never measuring the value of freshness, as the comparison above does.
• Different feature code for batch and real-time paths.
• No fallback when the real-time service is slow or down.

Batch versus real-time inference and deployment strategies are covered in the Model Deployment module of the MLOps course.

Sources

• Deploy models for inference (Amazon SageMaker AI documentation) — Amazon Web Services
• Batch transform for inference (Amazon SageMaker AI documentation) — Amazon Web Services
• Bike Sharing dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository