Monitoring ML Models in Production: A Complete Guide

A model that passed validation can still fail in production without a single error message. The code keeps running, the predictions keep arriving, and they slowly stop being right. Monitoring is how you find out, ideally before your users do.

This guide covers what to monitor, which signals deserve an alert, and how to work when the true answers arrive late. It replays a year of real monitoring, in which the loudest alert turned out to be the least useful.

What goes wrong after launch

• The pipeline breaks quietly. Google's Rules of Machine Learning call these silent failures: a joined table stops updating, or a feature that was filled in for 90% of examples drops to 60%, and the model keeps answering. Google Play once ran with a table six months stale, and refreshing it alone raised install rate by 2% (Google: Rules of ML).
• The world changes. Inputs move away from the training data, or the link between inputs and the answer shifts.
• Training and serving disagree. The same feature computed by two code paths produces different values.
• The service itself fails. Latency climbs, requests error, capacity runs out.

Four layers to monitor

Google's ML Test Score turns this into seven monitoring tests: dependency changes cause notification, data invariants hold, training and serving are not skewed, models are not too stale, models are numerically stable, computing performance has not regressed, and prediction quality has not regressed (Breck et al.: The ML Test Score).

A year of monitoring, replayed

The UCI Air Quality dataset holds 9,358 hourly readings from five low-cost metal oxide sensors at road level in an Italian city, with true carbon monoxide concentrations from a certified reference analyser alongside. Missing values are marked -200, and the dataset's description, citing De Vito and colleagues (2008), notes both concept drift and sensor drift (UCI: Air Quality).

The model is a "virtual sensor": it estimates CO from the cheap sensors and the weather. It is trained on five weeks of spring 2004 and validated on the next two. Then each month from May 2004 to March 2005 gets a monitoring report, as if run at month end, with five checks: complete inputs for at least 95% of hours, true values for at least 70% of scored hours, no input with a population stability index (PSI) above 0.25, mean absolute error (MAE) within 1.2 times validation, and average bias below 0.3 mg/m3.

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor

air = pd.read_csv("AirQualityUCI.csv", sep=";", decimal=",").dropna(axis=1, how="all").dropna(how="all")
air["ts"] = pd.to_datetime(air.Date + " " + air.Time, format="%d/%m/%Y %H.%M.%S")
air = air.replace(-200, np.nan)                                   # the dataset marks missing values as -200
INPUTS = ["PT08.S1(CO)", "PT08.S2(NMHC)", "PT08.S3(NOx)", "PT08.S4(NO2)", "PT08.S5(O3)", "T", "RH", "AH"]
TARGET = "CO(GT)"                                                 # reference analyser, mg/m3

fit = air[air.ts < "2004-04-17"].dropna(subset=INPUTS + [TARGET])
val = air[(air.ts >= "2004-04-17") & (air.ts < "2004-05-01")].dropna(subset=INPUTS + [TARGET])
model = RandomForestRegressor(n_estimators=200, min_samples_leaf=3, random_state=0, n_jobs=-1).fit(fit[INPUTS], fit[TARGET])
baseline = (val[TARGET] - model.predict(val[INPUTS])).abs().mean()
print(f"trained on {len(fit)} hours, validation MAE {baseline:.2f} mg/m3\n")


def psi(reference, current, bins=10):
    edges = np.unique(np.quantile(reference, np.linspace(0, 1, bins + 1)))
    edges[0], edges[-1] = -np.inf, np.inf
    ref = np.clip(np.histogram(reference, edges)[0] / len(reference), 1e-4, None)
    cur = np.clip(np.histogram(current, edges)[0] / len(current), 1e-4, None)
    return float(np.sum((cur - ref) * np.log(cur / ref)))


LIMITS = {"inputs": 0.95, "labels": 0.70, "psi": 0.25, "mae": 1.2 * baseline, "bias": 0.30}
air["month"] = air.ts.dt.to_period("M")
live = air[(air.ts >= "2004-05-01") & (air.ts < "2005-04-01")]
counts = {}
print(f"{'month':<8} {'inputs':>7} {'labels':>7} {'pred':>5} {'actual':>6} {'bias':>5} {'MAE':>5}  {'top drift (PSI)':<18} alerts")
for month, rows in live.groupby("month"):
    complete = rows.dropna(subset=INPUTS)                         # rows the model can score
    labelled = complete.dropna(subset=[TARGET])                   # rows whose true value has arrived
    error = labelled[TARGET] - model.predict(labelled[INPUTS])
    drift = {c: psi(fit[c], complete[c]) for c in INPUTS}
    top = max(drift, key=drift.get)
    checks = {
        "inputs": len(complete) / (month.days_in_month * 24) < LIMITS["inputs"],
        "labels": len(labelled) / len(complete) < LIMITS["labels"],
        "drift": drift[top] > LIMITS["psi"],
        "MAE": error.abs().mean() > LIMITS["mae"],
        "bias": abs(error.mean()) > LIMITS["bias"],
    }
    alerts = [name for name, fired in checks.items() if fired]
    for name in alerts:
        counts[name] = counts.get(name, 0) + 1
    print(f"{str(month):<8} {len(complete) / (month.days_in_month * 24):>7.1%} {len(labelled) / len(complete):>7.1%} "
          f"{model.predict(labelled[INPUTS]).mean():>5.2f} {labelled[TARGET].mean():>6.2f} {-error.mean():>+5.2f} "
          f"{error.abs().mean():>5.2f}  {top + f' {drift[top]:.1f}':<18} {', '.join(alerts)}")

print("\nmonths with each alert:", ", ".join(f"{k} {v}" for k, v in counts.items()))

trained on 777 hours, validation MAE 0.34 mg/m3

month     inputs  labels  pred actual  bias   MAE  top drift (PSI)    alerts
2004-05    98.1%   81.1%  2.27   1.94 +0.33  0.40  T 1.6              drift, bias
2004-06    94.7%   89.0%  2.33   1.93 +0.40  0.42  T 6.5              inputs, drift, MAE, bias
2004-07    99.9%   75.1%  2.25   1.78 +0.46  0.49  T 6.8              drift, MAE, bias
2004-08    93.4%   61.6%  1.58   1.25 +0.33  0.33  T 7.6              inputs, labels, drift, bias
2004-09    97.5%   79.1%  2.40   2.16 +0.24  0.32  T 3.9              drift
2004-10    99.9%   52.8%  3.10   2.72 +0.37  0.46  AH 6.9             labels, drift, MAE, bias
2004-11   100.0%   94.3%  2.62   2.69 -0.07  0.44  AH 1.1             drift, MAE
2004-12    89.2%   84.5%  2.27   2.58 -0.31  0.43  PT08.S4(NO2) 1.6   inputs, drift, MAE, bias
2005-01    91.8%   90.2%  2.20   2.11 +0.09  0.43  T 3.0              inputs, drift, MAE
2005-02    88.7%   98.7%  1.84   1.99 -0.16  0.30  AH 3.5             inputs, drift
2005-03    99.9%   97.3%  2.04   2.04 -0.00  0.34  PT08.S4(NO2) 1.5   drift

months with each alert: drift 11, bias 6, inputs 5, MAE 6, labels 2

The pred, actual, bias and MAE columns use the hours whose true values had arrived. Bias is the average prediction minus the average true value, so positive means the model over-estimated.

What the output shows

• The drift alert fired every month, 11 of 11. Temperature and humidity followed the seasons, with PSI reaching 7.6 against an alarm level of 0.25. An alert that always fires is soon muted. The ML Test Score warns that drift thresholds need careful tuning to balance false positives against false negatives.
• The real problem was bias. From May to August the model over-estimated CO by 0.33 to 0.46 mg/m3, between 17% and 26% of the true level, and MAE reached 0.49 against 0.34 at validation. Trained on spring, it had never seen summer.
• Winter brought the opposite error. In December it under-estimated by 0.31.
• Not all drift was weather. In December and March the largest shift was in one sensor's own readings, PT08.S4. Telling sensor ageing apart from a real change in pollution needs the reference values.
• By February and March 2005, MAE was back to 0.30 and 0.34, in the same season as the training data. Seasonality, not decay, drove most of the error.
• The pipeline had problems of its own. In five months fewer than 95% of hours had complete inputs, and in October only 53% of scored hours had true values, so that month's error rests on half the data.

Designing alerts people act on

• Page on outcomes and pipeline health. Performance and bias, once labels exist, and missing or stale data deserve a person's attention. Send drift statistics to a dashboard for diagnosis.
• Compare like with like. A drift reference from the same season, or a rolling window, separates expected change from surprise.
• Watch bias in aggregate and by slice. The ML Test Score recommends measuring the average prediction against reality as a canary for problems.
• Show label coverage next to every quality metric. An error figure from half the data is weaker evidence.
• Track model age. The same paper suggests alerting when a model or one of its feature tables grows too old. Google's Rule 8 asks how much quality you lose with a day-old, week-old or quarter-old model.
• Check before release. Rule 9 is to catch problems before a model is exported to serving, where an error becomes user-facing.

When labels arrive late

True answers often come days or months later: a loan defaults, a customer churns, a reference analyser is read. Until then, watch inputs, prediction distributions and bias against any known totals. Log every prediction with an ID, its inputs and the model version, so the true value can be joined when it arrives. For tasks where labels never arrive, budget for people to label a sample.

Tools

Most teams combine pieces. Service metrics usually go to Prometheus and Grafana; see Prometheus and Grafana for ML monitoring. Prediction logs go to a log store; see logging and observability for ML systems. Drift reports often come from libraries such as Evidently.

Cloud products change. AWS documents that SageMaker Model Monitor is no longer open to new customers and will get no new features, and points to open-source monitoring solutions built on SageMaker AI MLflow Apps and Evidently AI, with QuickSight dashboards and CloudWatch, as its replacement (AWS: Model Monitor availability change).

A short runbook for alerts

1. Check the pipeline first. Missing inputs, late labels and stale tables cause more alerts than models do.
2. Check what changed. A deployment, a dependency or an upstream schema.
3. Find the slice. A time of day, a region or a customer group often carries the whole problem.
4. Decide the response: fix the data, roll back the model, retrain, or accept and document. For drift statistics in depth, see model drift and data drift. For automated retraining, see continuous training.

Common mistakes

• Alerting on every drift statistic, until the team ignores the pager.
• Monitoring servers but not predictions.
• Reporting accuracy without label coverage.
• No baseline saved at launch, so nothing to compare against.
• No owner for each alert.

Drift detection, performance monitoring, logging and alerts, and retraining pipelines make up the Model Monitoring and Maintenance module of the MLOps course.

Sources

• Rules of Machine Learning (Google for Developers) — Google
• The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction (IEEE Big Data 2017) — Breck et al., Google
• Air Quality dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository
• Amazon SageMaker Model Monitor availability change (AWS documentation) — Amazon Web Services