Continuous Training: Automating Model Retraining
Continuous training retrains models automatically, but retraining is only as good as the checks around it. On three years of real household energy data, this guide compares four policies: never retraining, monthly retraining, adding data checks, and adding a champion gate that was right only 21 times in 34.
GloryTecks Team
Training & Placement Faculty
Last updated: 8 October 2026
Models age. Continuous training answers by retraining them automatically: new data comes in, a pipeline trains a candidate, and the candidate replaces the current model. The idea is simple. The hard part is the set of checks around it, because an automated pipeline retrains just as happily on broken data, and promotes a worse model just as quickly as a better one.
This guide compares four retraining policies over three years of real data, and measures how good each check actually was.
What continuous training means
Google's MLOps guide describes continuous training as automating the training pipeline so it can run without a person. It lists the usual triggers: on demand, on a schedule, when new training data arrives, when model performance degrades, and when data distributions change significantly. It also puts two checks around training: data validation first, to decide whether to retrain or stop, and model validation after, to confirm the new model is good enough to deploy (Google Cloud: MLOps).
| Trigger | Suits | Watch out for |
|---|---|---|
| Schedule | Data that arrives regularly | Retraining when nothing changed |
| New data | Irregular data deliveries | Partial or broken deliveries |
| Performance drop | Labels that arrive quickly | Needs reliable monitoring |
| Distribution change | Labels that arrive late | Drift that does not hurt the model |
The experiment
The UCI household power data hold 2,075,259 minute readings from one house in Sceaux, near Paris, from December 2006 to November 2010, with about 1.25% of rows missing (UCI: household power consumption). The task is a daily forecast of tomorrow's electricity use from recent days and the calendar.
Four policies run from January 2008 to November 2010 and are scored on the same 1,040 complete days:
- Never retrain: a model trained on 2007 serves for three years.
- Retrain monthly, no checks: each month, train on everything so far, with the data as logged, so missing minutes count as zero.
- Retrain monthly, with data checks: days missing more than 10% of readings become unknown, and when a forecast's inputs are incomplete, a simple fallback answers: the average of the last 28 complete days.
- As 3, plus a champion gate: the challenger trains without the last 28 days and must beat the current model on them to replace it.
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
power = pd.read_csv("household_power_consumption.txt", sep=";", na_values="?", # UCI, one household, 2006-2010
usecols=["Date", "Time", "Global_active_power"], dtype={"Global_active_power": float})
power.index = pd.to_datetime(power.Date + " " + power.Time, format="%d/%m/%Y %H:%M:%S")
daily = power.Global_active_power.resample("D").agg(["sum", "count", "size"]).loc["2006-12-17":"2010-11-25"]
daily["missing"] = 1 - daily["count"] / daily["size"]
daily["as_logged"] = daily["sum"] / 60 # kWh, with missing minutes silently counted as zero
daily["checked"] = daily.as_logged.where(daily.missing <= 0.10) # data check: a day missing over 10% is unknown
FEATURES = ["yesterday", "week_ago", "mean_7d", "mean_28d", "weekday", "day_of_year"]
def table(usage): # features at day d, target = usage on day d + 1
tomorrow = usage.index + pd.Timedelta(days=1)
return pd.DataFrame({"yesterday": usage, "week_ago": usage.shift(6),
"mean_7d": usage.rolling(7, min_periods=5).mean(), "mean_28d": usage.rolling(28, min_periods=20).mean(),
"weekday": tomorrow.dayofweek, "day_of_year": tomorrow.dayofyear, "target": usage.shift(-1)})
def train(rows, before):
known = rows.loc[:before - pd.Timedelta(days=1)].dropna()
return HistGradientBoostingRegressor(random_state=0, max_iter=200).fit(known[FEATURES], known.target)
def mae(model, rows):
rows = rows.dropna()
return np.abs(model.predict(rows[FEATURES]) - rows.target).mean()
tables = {"as_logged": table(daily.as_logged), "checked": table(daily.checked)}
fallback = daily.checked.rolling(28, min_periods=1).mean() # when inputs are incomplete: recent average
truth = daily.checked.shift(-1)
months = pd.date_range("2008-01-01", "2010-11-01", freq="MS")
days = daily.loc["2008-01-01":"2010-11-24"].index
after_gap = days.isin([g + pd.Timedelta(days=k) for g in daily.index[daily.missing > 0.10] for k in range(8)])
print(f"{(daily.missing > 0.10).sum()} days with over 10% of readings missing; scoring {truth.reindex(days).notna().sum():,} "
f"complete days, January 2008 to November 2010\n")
print(f"{'policy':<45} {'MAE kWh':>7} {'week after a gap':>17} notes")
for name, source, retrain, gate in (("never retrain (trained on 2007)", "as_logged", False, False),
("retrain monthly, no checks", "as_logged", True, False),
("retrain monthly, data checks", "checked", True, False),
("retrain monthly, data checks + champion gate", "checked", True, True)):
rows, forecast = tables[source], pd.Series(np.nan, index=days)
model, promoted, rejected, sound, fallbacks = train(rows, months[0]), 0, 0, 0, 0
for i, month in enumerate(months):
last = months[i + 1] - pd.Timedelta(days=1) if i + 1 < len(months) else days[-1]
if i and retrain and not gate:
model = train(rows, month)
elif i and gate: # challenger trained without the last 28 days, then compared
holdout = rows.loc[month - pd.Timedelta(days=28):month - pd.Timedelta(days=1)]
challenger = train(rows, month - pd.Timedelta(days=28))
keep_old = mae(challenger, holdout) > mae(model, holdout)
sound += (mae(model, rows.loc[month:last]) <= mae(challenger, rows.loc[month:last])) == keep_old
promoted, rejected = promoted + (not keep_old), rejected + keep_old
model = model if keep_old else challenger
block = rows.loc[month:last]
ready = block.dropna(subset=FEATURES)
forecast.loc[ready.index] = model.predict(ready[FEATURES])
gaps = block.index.difference(ready.index)
forecast.loc[gaps], fallbacks = fallback.reindex(gaps).to_numpy(), fallbacks + len(gaps)
error = (forecast - truth.reindex(days)).abs()
notes = f"{fallbacks} fallback days" if fallbacks else ""
if gate:
notes += f"; {promoted} promoted, {rejected} rejected; right call for the next month {sound} of {promoted + rejected}"
print(f"{name:<45} {error.mean():>7.2f} {error[after_gap].mean():>17.2f} {notes}")22 days with over 10% of readings missing; scoring 1,040 complete days, January 2008 to November 2010
policy MAE kWh week after a gap notes
never retrain (trained on 2007) 6.31 5.65
retrain monthly, no checks 5.53 5.20
retrain monthly, data checks 5.44 4.42 46 fallback days
retrain monthly, data checks + champion gate 5.48 4.35 46 fallback days; 16 promoted, 18 rejected; right call for the next month 21 of 34What the output shows
- Retraining paid. Monthly retraining cut the mean absolute error from 6.31 kWh to 5.53, 12% lower than never retraining.
- Data checks paid most where data broke. In the week after a gap, error fell from 5.20 to 4.42 kWh, and overall to 5.44. The checked pipeline answered 46 days with the fallback rather than feeding the model inputs built from missing readings. Counting missing minutes as zero is a silent failure: the totals look like real days, just low ones.
- The champion gate did not earn its keep. It rejected 18 of 34 challengers, picked the model that did better over the following month only 21 times in 34, and ended slightly worse overall, 5.48 against 5.44, though slightly better right after gaps. Twenty-eight days of daily errors is a noisy judge.
A gate is still worth having, but for a different job. Google's Rules of ML recommend sanity checks right before a model is exported, making sure its performance is reasonable on held-out data (Google: Rules of ML). That catches broken candidates: a failed training run, a corrupted dataset, a bug. Choosing between two decent models needs far more evaluation data, a tolerance for noise, and a record of how often the gate was right.
Designing a retraining pipeline
- Pick the trigger from how labels arrive. Schedules are simple; performance triggers need fast labels; drift triggers fill the gap when labels are slow. See model drift and data drift.
- Validate data before training, and stop the pipeline on schema breaks, missing data or impossible values.
- Record what each model was trained on: data version, code commit and settings. See reproducibility in ML.
- Validate the candidate in two layers: absolute sanity thresholds always, and comparison with the current model only on enough data, with a tolerance.
- Promote through a registry that keeps the previous version one step away; see the model registry.
- Keep watching after promotion; see monitoring ML models in production.
The pipeline itself belongs in the same delivery process as the rest of the code; see CI/CD for machine learning.
Common mistakes
- Retraining on whatever arrived, without data checks.
- Treating missing values as zero.
- Judging challengers on too little data.
- Promoting without a rollback path.
- Scheduling retraining before the labels exist.
What is continuous training in MLOps?
How often should a model be retrained?
What checks should run before a retrained model is deployed?
Is continuous training the same as online learning?
Continuous training and retraining pipelines are covered in the CI/CD for Machine Learning and Model Monitoring and Maintenance modules of the MLOps course.
Sources
- MLOps: Continuous delivery and automation pipelines in machine learning (Google Cloud) — Google Cloud
- Rules of Machine Learning (Google for Developers) — Google
- Individual Household Electric Power Consumption dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master MLOps?
Join GloryTecks’ expert-led MLOps training in Hyderabad with real-time projects and 100% placement support.
Related articles
Model Versioning with MLflow and DVC
A model version is only useful if you can say what produced it. This guide combines DVC for data with MLflow for models, so each registered version records its data hash and Git commit, then rebuilds an old version from that record alone and checks that the predictions come out identical.
Model Drift and Data Drift: Detection and Response
Most drift tooling watches a model's inputs. This guide shows, on three Januaries of real New York taxi trips, a change that left every input statistic flat while the model's error grew fivefold, why significance tests mislead at scale, and how to respond to drift you detect and to change you know is coming.
MLOps Roadmap 2026: Skills, Tools and Career Path
MLOps sits between data science, software engineering and operations, which makes it hard to know where to start. This roadmap orders the skills from foundations to production operations, gives each stage a project that proves the skill, and explains what the MLOps engineer role actually involves.