Handling Concept Drift in Live ML Systems

Concept drift is what happens when the relationship your model learned stops being true. The inputs may look the same, but the outcome they lead to has changed. Evidently describes it as a change in the input-output relationships a model has learned, detectable through model quality metrics when labels arrive, and usually handled by retraining (Evidently: concept drift). "Retrain the model" is easy to say. Which retraining actually works is the harder question, and it is what this article tests.

A real case of concept drift

The UCI Bike Sharing dataset has hourly rentals from Washington D.C.'s Capital Bikeshare system for 2011 and 2012, with weather and calendar information (UCI: Bike Sharing). The system grew quickly between the two years. At 5 pm on a clear working day, the average number of rentals rose from 431 in 2011 to 709 in 2012: the same hour, the same weather, two-thirds more demand. A model that learned 2011's relationship between conditions and rentals will under-predict 2012.

The experiment trains a gradient boosting model on January to June 2011, then walks forward month by month to December 2012. Each month is predicted, scored, and then added to the history before the strategy decides whether to retrain.

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error

# UCI Bike Sharing, hourly rentals in Washington D.C., 2011-2012
bikes = pd.read_csv("hour.csv", parse_dates=["dteday"])
bikes["month"] = bikes.dteday.dt.to_period("M")
features = ["season", "mnth", "hr", "holiday", "weekday", "workingday", "weathersit", "temp", "atemp", "hum", "windspeed"]

peak = bikes[(bikes.hr == 17) & (bikes.workingday == 1) & (bikes.weathersit == 1)]
print("D1 same conditions, different demand (5 pm, working day, clear weather): mean rentals")
print("   " + " | ".join(f"{y}: {v:.0f}" for y, v in peak.groupby(bikes.dteday.dt.year).cnt.mean().items()))


def fit(rows, cols=features):
    return HistGradientBoostingRegressor(max_iter=200, random_state=0).fit(rows[cols], rows.cnt)


months = sorted(bikes.month.unique())
first = [m for m in months if m <= pd.Period("2011-06", "M")]
later = [m for m in months if m > pd.Period("2011-06", "M")]
history = bikes[bikes.month.isin(first)]
baseline = mean_absolute_error(history[history.month == first[-1]].cnt,
                               fit(history[history.month < first[-1]]).predict(history[history.month == first[-1]][features]))


def run(strategy):
    cols = features + ["yr"] if "year feature" in strategy else features
    model, retrains, errors = fit(history, cols), 0, []
    seen = history.copy()
    for m in later:
        month = bikes[bikes.month == m]
        err = mean_absolute_error(month.cnt, model.predict(month[cols]))
        errors.append((m, err))
        seen = pd.concat([seen, month])
        triggered = err > 1.5 * baseline
        if strategy in ("monthly, all history", "monthly, all + year feature") or (strategy == "if error > 1.5x, all history" and triggered):
            model, retrains = fit(seen, cols), retrains + 1
        elif strategy == "monthly, last 6 months" or (strategy == "if error > 1.5x, last 6 months" and triggered):
            model, retrains = fit(seen[seen.month > m - 6], cols), retrains + 1
    return errors, retrains


print(f"\nD2 mean absolute error (rentals per hour) after training on Jan-Jun 2011; baseline error {baseline:.0f}")
print(f"   {'strategy':<32}{'2011 H2':>9}{'2012 H1':>9}{'2012 H2':>9}{'overall':>9}{'retrains':>10}")
for strategy in ["never retrain", "monthly, all history", "if error > 1.5x, all history", "monthly, last 6 months",
                 "if error > 1.5x, last 6 months", "monthly, all + year feature"]:
    errors, retrains = run(strategy)
    e = pd.Series(dict(errors))
    half = lambda a, b: e[(e.index >= pd.Period(a, "M")) & (e.index <= pd.Period(b, "M"))].mean()
    print(f"   {strategy:<32}{half('2011-07', '2011-12'):>9.0f}{half('2012-01', '2012-06'):>9.0f}"
          f"{half('2012-07', '2012-12'):>9.0f}{e.mean():>9.0f}{retrains:>10}")

D1 same conditions, different demand (5 pm, working day, clear weather): mean rentals
   2011: 431 | 2012: 709

D2 mean absolute error (rentals per hour) after training on Jan-Jun 2011; baseline error 29
   strategy                          2011 H2  2012 H1  2012 H2  overall  retrains
   never retrain                          32       87       92       70         0
   monthly, all history                   30       85       91       69        18
   if error > 1.5x, all history           32       85       91       69        12
   monthly, last 6 months                 31       42       46       39        18
   if error > 1.5x, last 6 months         32       49       49       43         7
   monthly, all + year feature            30       45       39       38        18

What the results show

Doing nothing fails badly. The original model's error rose from 32 rentals per hour in late 2011 to around 90 in 2012, three times its baseline.

Retraining more often on all history barely helped. Retraining every month on everything seen so far brought the overall error from 70 only to 69. The model now saw 2012 data, but it was outnumbered by 2011 data that described a different relationship, and none of the features told the model which year a row came from. Triggering the same retraining only when error passed 1.5 times the baseline gave the same result with fewer retrains.

The training window mattered more than the frequency. Retraining monthly on only the last six months cut the overall error to 39, because the model learned the current relationship instead of averaging old and new.

Triggers can save effort. Retraining on the last six months only when error exceeded 1.5 times the baseline reached an overall error of 43 with 7 retrains instead of 18.

Fixing the features can beat any schedule. Adding the year indicator, a feature known in advance, and retraining monthly on all history gave the lowest overall error, 38, with the best late-2012 performance. When drift has an identifiable cause, a feature that represents it lets the model use old data without being misled by it.

Why all-history retraining failed here

A tree-based model can only separate rows that its features can tell apart. Without a feature for the year, a clear Tuesday at 5 pm in 2011 and the same hour in 2012 look identical, so the model learns their average. As 2012 months were added, that average moved only slowly towards the new level, because six to eighteen months of 2011 data still dominated. A sliding window removes the old rows; the year feature lets the model keep them but treat them separately. Both work by stopping the old relationship from diluting the new one, which is the core idea behind any good response to concept drift.

Choosing a retraining response

1. Confirm it is concept drift. Check model error on labelled recent data, not only input distributions. Inputs here looked normal; the relationship changed.
2. Ask whether the old data still describes today. If not, a sliding window or down-weighting old rows usually beats more frequent retraining on everything.
3. Look for a cause you can represent. Growth, price changes, a new product line or a policy change can often become a feature. That is more robust than discarding history.
4. Decide the trigger. Fixed schedules are simple and predictable; error-based triggers save compute and review effort. Either way, set the threshold from a measured baseline.
5. Validate before you replace. Compare each retrained model with the current one on the latest labelled period, and keep the previous version ready to roll back.

Practical constraints in production

Labels often arrive late. Bike rentals are counted the same hour, but loan defaults or churn take months. When labels lag, use proxy signals, such as prediction distributions and input drift, as the Evidently guide suggests, to decide when to investigate, and retrain once enough labelled data exists. Record every retrain with its data window, features and evaluation, so you can explain later why the model changed. Experiment tracking with MLflow makes this straightforward; see MLflow for data scientists.

For detecting drift in the first place, see model drift and data drift: detection and response. For automating the retraining pipeline itself, see continuous training, and for the wider monitoring picture, monitoring ML models in production.

Model monitoring and retraining pipelines are taught in the MLOps course.

Sources

• Bike Sharing Dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository
• What is concept drift in ML, and how to detect and address it (Evidently AI) — Evidently AI