Model Drift and Data Drift: Detection and Response

Models fail in production for two different reasons. Sometimes the inputs change: different customers, different seasons, a new market. Sometimes the inputs stay the same but the right answer changes. Most drift tooling watches only the first.

This guide shows both on real data. Three Januaries of New York taxi trips contain a change that left every input statistic flat while the model's error grew more than fivefold, and a second change that hid inside an average.

The kinds of drift

Lu and colleagues' review defines concept drift as a change in the joint distribution of inputs and target, which can come from three sources. The inputs can change while the relationship to the target stays put, which they call virtual drift because the decision boundary does not move. The target can change given the same inputs, actual drift, which lowers accuracy. Or both can happen at once (Lu et al.: Learning under Concept Drift).

Model drift is the informal name for the result: a model whose performance has decayed, for any of these reasons.

The experiment

The data are NYC Taxi and Limousine Commission trip records for yellow taxis (NYC TLC: trip record data). Using January of 2022, 2024 and 2025 compares the same season each year. The model predicts what a rider is charged before any tip, from trip distance, pickup hour, weekday, pickup and drop-off zones and rate code, much like an upfront price estimate. The target is total amount minus tip, since the data dictionary notes that total amount excludes cash tips (NYC TLC: data dictionary).

import numpy as np
import pandas as pd
from scipy.stats import ks_2samp
from sklearn.ensemble import HistGradientBoostingRegressor

FEATURES = ["trip_distance", "hour", "weekday", "pulocationid", "dolocationid", "ratecodeid"]
CATEGORICAL = {"hour", "weekday", "pulocationid", "dolocationid", "ratecodeid"}


def january(year):
    trips = pd.read_parquet(f"yellow_tripdata_{year}-01.parquet")
    trips.columns = [c.lower() for c in trips.columns]
    minutes = (trips.tpep_dropoff_datetime - trips.tpep_pickup_datetime).dt.total_seconds() / 60
    trips = trips[trips.tpep_pickup_datetime.between(f"{year}-01-01", f"{year}-01-31 23:59:59")
                  & trips.trip_distance.between(0.01, 50) & trips.fare_amount.between(0.01, 300)
                  & minutes.between(1, 180)].copy()
    trips["charged"] = trips.total_amount - trips.tip_amount   # what the rider paid before any tip
    trips["hour"] = trips.tpep_pickup_datetime.dt.hour
    trips["weekday"] = trips.tpep_pickup_datetime.dt.weekday
    return trips


def psi(reference, current, categorical):
    if categorical:
        ref, cur = reference.value_counts(normalize=True), current.value_counts(normalize=True)
        keys = ref.index.union(cur.index)
        ref, cur = ref.reindex(keys, fill_value=0).to_numpy(), cur.reindex(keys, fill_value=0).to_numpy()
    else:
        edges = np.unique(np.quantile(reference, np.linspace(0, 1, 11)))
        edges[0], edges[-1] = -np.inf, np.inf
        ref = np.histogram(reference, edges)[0] / len(reference)
        cur = np.histogram(current, edges)[0] / len(current)
    ref, cur = np.clip(ref, 1e-4, None), np.clip(cur, 1e-4, None)
    return float(np.sum((cur - ref) * np.log(cur / ref)))


def report(name, model, trips):
    error = model.predict(trips[FEATURES]) - trips.charged
    print(f"  {name:<42} MAE {error.abs().mean():5.2f}  bias {error.mean():+5.2f}")


y2022, y2024, y2025 = january(2022), january(2024), january(2025)
train = y2022.sample(200_000, random_state=0)
holdout = y2022.drop(train.index).sample(100_000, random_state=1)
later = y2024.sample(100_000, random_state=2)
model = HistGradientBoostingRegressor(random_state=0).fit(train[FEATURES], train.charged)

print("1. Input drift, January 2022 -> January 2024 (PSI)")
for column in FEATURES:
    print(f"  {column:<14} {psi(train[column], later[column], column in CATEGORICAL):.3f}")
print(f"  {'predictions':<14} {psi(pd.Series(model.predict(train[FEATURES])), pd.Series(model.predict(later[FEATURES])), False):.3f}")
ks = ks_2samp(train.trip_distance, later.trip_distance)
print(f"  KS test on trip_distance: statistic {ks.statistic:.3f}, p-value {ks.pvalue:.1e}")

print("\n2. The 2022 model, judged against what riders paid")
report("January 2022 (held-out trips)", model, holdout)
report("January 2024", model, later)
print(f"  median fare per mile: 2022 ${(y2022.fare_amount / y2022.trip_distance).median():.2f}, "
      f"2024 ${(y2024.fare_amount / y2024.trip_distance).median():.2f}")

print("\n3. Response: retrain on 1-14 January 2024, test on 15-31 January")
first = y2024[y2024.tpep_pickup_datetime < "2024-01-15"].sample(200_000, random_state=3)
second = y2024[y2024.tpep_pickup_datetime >= "2024-01-15"].sample(100_000, random_state=4)
report("old model (trained on 2022)", model, second)
report("retrained model", HistGradientBoostingRegressor(random_state=0).fit(first[FEATURES], first.charged), second)

print("\n4. January 2025, model retrained on all of January 2024")
recent = y2024.sample(200_000, random_state=5)
current = HistGradientBoostingRegressor(random_state=0).fit(recent[FEATURES], recent.charged)
y2025 = y2025.sample(100_000, random_state=6)
tolled = y2025.cbd_congestion_fee > 0
print(f"  trips charged the new congestion fee: {tolled.mean():.0%}")
report("all trips", current, y2025)
report("trips charged the fee", current, y2025[tolled])
report("trips not charged the fee", current, y2025[~tolled])
adjusted = current.predict(y2025[FEATURES]) + 0.75 * tolled
print(f"  {'charged the fee, after adding $0.75':<42} bias {(adjusted - y2025.charged)[tolled].mean():+5.2f}")

1. Input drift, January 2022 -> January 2024 (PSI)
  trip_distance  0.008
  hour           0.006
  weekday        0.018
  pulocationid   0.036
  dolocationid   0.023
  ratecodeid     0.012
  predictions    0.008
  KS test on trip_distance: statistic 0.021, p-value 7.2e-27

2. The 2022 model, judged against what riders paid
  January 2022 (held-out trips)              MAE  1.27  bias +0.00
  January 2024                               MAE  6.85  bias -6.72
  median fare per mile: 2022 $5.00, 2024 $7.18

3. Response: retrain on 1-14 January 2024, test on 15-31 January
  old model (trained on 2022)                MAE  6.90  bias -6.78
  retrained model                            MAE  2.00  bias -0.48

4. January 2025, model retrained on all of January 2024
  trips charged the new congestion fee: 66%
  all trips                                  MAE  2.11  bias -0.26
  trips charged the fee                      MAE  1.96  bias -0.66
  trips not charged the fee                  MAE  2.40  bias +0.53
  charged the fee, after adding $0.75        bias +0.09

Inputs flat, error up fivefold

Every input's PSI between January 2022 and January 2024 was 0.036 or less, and the predictions moved by 0.008. Judged by input monitoring, nothing happened.

Yet the 2022 model's average error rose from $1.27 to $6.85, and it under-predicted by $6.72 a trip. The cause was a rule change, not a data change. The TLC's first fare increase in ten years took effect on 19 December 2022, with a $3.00 initial charge, 70 cents for each additional unit and a $1.00 improvement surcharge (NYC TLC: Industry Notice 22-02). The median fare per mile went from $5.00 to $7.18. Same trips, different answers: actual drift.

Prediction drift could not catch it either. The model's outputs depend only on its inputs, so when the inputs hold still, so do the predictions.

Significant is not the same as important

SciPy's two-sample Kolmogorov-Smirnov test compares two samples' distributions and returns a statistic and a p-value (SciPy: ks_2samp). For trip distance the p-value was 7.2e-27, overwhelming evidence that the distributions differ. The statistic, the largest gap between the two cumulative distributions, was 0.021: about two percentage points.

With hundreds of thousands of rows, almost any difference is significant. Alert on effect sizes such as the KS statistic or PSI, and decide in advance how big a change matters.

Responding by retraining

Retraining on the first two weeks of January 2024 cut the error on the last two weeks from $6.90 to $2.00. When the relationship itself has changed, recent labelled data is the fix.

That works here because a fare is known the moment a trip ends. For loan defaults or customer churn, labels take months, and the model is wrong for all that time. In those settings, watch for external changes and sample early labels.

Responding to a change you know about

The second change was announced in advance. From 5 January 2025 the data carry a per-trip charge for the MTA's Congestion Relief Zone, and the TLC's fare page lists it as 75 cents for yellow and green taxis in Manhattan south of and including 60th Street (NYC TLC: taxi fare).

The model retrained on January 2024 looked healthy on January 2025 overall: a bias of -$0.26, which most checks would pass. Split by the change, the picture differed. Trips charged the fee, 66% of the sample, were under-predicted by $0.66, while the rest were over-predicted by $0.53. The average hid two opposite errors. Adding the published 75 cents to trips charged the fee brought that group's bias to +$0.09.

Two lessons follow. Slice by whatever changed. And a known change does not need detecting: put it into the model, the features or the pricing rules on the day it starts. Differences of around half a dollar either way, such as the -$0.48 left after retraining in 2024, are the ordinary variation your thresholds have to tolerate.

A response playbook

Choosing detection methods

• PSI compares binned distributions and works for numbers and categories.
• The KS statistic suits numeric features; ignore its p-value on large samples.
• Prediction drift is cheap and immediate, but blind to actual drift, as shown above.
• Error and bias by slice is the only direct evidence of actual drift, and it needs labels.
• The reference window matters: compare like seasons, or use rolling windows.

For the wider monitoring setup, see monitoring ML models in production. For data checks that catch broken feeds, see testing machine learning code and data. For retraining on a trigger, see continuous training, and to roll back to a known-good version, see the model registry.

Common mistakes

• Treating stable inputs as proof of a healthy model.
• Using p-values on very large samples.
• Watching only averages, which can hide opposite errors.
• Retraining on every drift alarm, including on broken data.
• Ignoring announced changes in prices, rules or products.

Model drift detection, performance monitoring and retraining pipelines are taught in the Model Monitoring and Maintenance module of the MLOps course.

Sources

• Learning under Concept Drift: A Review (IEEE TKDE, 2018) — Lu et al.
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC TLC
• Data Dictionary: Yellow Taxi Trip Records (NYC TLC) — NYC TLC
• Industry Notice 22-02: New Taxi and SHL Meter Fare Structure Effective 12/19/22 (NYC TLC) — NYC TLC
• Taxi Fare (NYC TLC) — NYC TLC
• scipy.stats.ks_2samp (SciPy documentation) — SciPy