CI/CD for Machine Learning: Pipelines That Work
CI/CD for machine learning has to test three things that change: code, data and models. This guide runs five commits through a staged pipeline on real wine data. It stops a broken delivery, a feature bug and a weaker model at different stages, and shows duplicates had inflated the AUC from 0.822 to 0.896.
GloryTecks Team
Training & Placement Faculty
Last updated: 8 October 2026
Continuous integration and delivery made software releases routine: every change is built, tested and deployed by a pipeline instead of by hand. Machine learning needs the same discipline, but more is changing at once. A new model can break because of the code, because of the data it learned from, or simply because it is worse.
CI/CD for machine learning means testing all three, in an order that stops bad changes as cheaply as possible. This guide builds that pipeline and runs five commits through it.
What changes in ML
The Thoughtworks article that named Continuous Delivery for Machine Learning puts it simply: ML applications change along three axes, the code, the model and the data (martinfowler.com: CD4ML). Google's MLOps guide spells out what that does to the pipeline. Integration must also test and validate data, schemas and models. Delivery ships a training pipeline that in turn deploys a prediction service. And continuous training is a new property of its own (Google Cloud: MLOps).
| Stage | Catches | Typical cost |
|---|---|---|
| Unit tests | Bugs in feature and training code | Seconds |
| Data validation | Schema breaks, impossible values, missing data | Seconds |
| Training | Nothing by itself; must be reproducible | Minutes to hours |
| Quality gate | Models below a threshold or worse than the live one | Minutes |
| Smoke test | Broken or unloadable artefacts | Seconds |
| Deployment | Handled by a release strategy and monitoring | Varies |
The order matters: run the cheap stages first so the expensive one runs only for changes worth training.
The pipeline
The data are the UCI Wine Quality sets: red and white vinho verde samples from northern Portugal with physicochemical measurements and a sensory quality score, under CC BY 4.0 (UCI: Wine Quality). The model predicts whether a wine is rated 7 or higher. A new model deploys only if its AUC is at least 0.80 and no more than 0.005 below the model already deployed.
Five commits follow, each simulated as a change to the data, the feature code or the model settings, and each running the same stages.
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
red = pd.read_csv("winequality-red.csv", sep=";").assign(red=1)
white = pd.read_csv("winequality-white.csv", sep=";").assign(red=0)
delivery = pd.concat([red, white], ignore_index=True) # UCI Wine Quality (CC BY 4.0)
SCHEMA = {c: "float64" for c in delivery.columns} | {"quality": "int64", "red": "int64"}
RANGES = {"pH": (2.5, 4.5), "alcohol": (7.0, 16.0), "density": (0.98, 1.05)}
state = {"champion": None, "version": 0, "trained": 0}
def add_features(wine): # feature code, versioned with everything else
return wine.assign(free_share=wine["free sulfur dioxide"] / wine["total sulfur dioxide"])
def add_features_refactored(wine): # a refactor that swapped the ratio by mistake
return wine.assign(free_share=wine["total sulfur dioxide"] / wine["free sulfur dioxide"])
def unit_tests(make_features): # stage 1: fast checks on the code
sample = delivery.head(50)
out = make_features(sample)
checks = {"keeps every row": len(out) == len(sample), "free_share is a fraction": out.free_share.between(0, 1).all(),
"no missing values": not out.isna().any().any()}
return [name for name, passed in checks.items() if not passed]
def validate(data): # stage 2: checks on the data
problems = [f"{c} is {data[c].dtype}, expected {t}" for c, t in SCHEMA.items() if str(data[c].dtype) != t]
if not problems:
problems += [f"{c} outside {low}-{high}" for c, (low, high) in RANGES.items() if not data[c].between(low, high).all()]
problems += ["more than 1% missing"] if data.isna().mean().max() > 0.01 else []
return problems
def fit_and_score(wine, params=None): # train, then score on a fixed held-out split
wine = wine.assign(good=wine.quality >= 7).drop(columns="quality")
train, test = train_test_split(wine, test_size=0.25, random_state=0, stratify=wine.good)
features = [c for c in wine.columns if c != "good"]
model = HistGradientBoostingClassifier(random_state=0, **(params or {})).fit(train[features], train.good)
return model, roc_auc_score(test.good, model.predict_proba(test[features])[:, 1]), test[features]
def pipeline(commit, data=delivery, make_features=add_features, params=None):
if failed := unit_tests(make_features):
return print(f"{commit:<40} stopped at unit tests: {'; '.join(failed)}")
if problems := validate(data):
return print(f"{commit:<40} stopped at data validation: {'; '.join(problems)}")
model, auc, test_x = fit_and_score(make_features(data.drop_duplicates()), params) # duplicates would leak
state["trained"] += 1
floor = max(0.80, state["champion"] - 0.005) if state["champion"] else 0.80 # stage 3: quality gate
if auc < floor:
return print(f"{commit:<40} stopped at quality gate: AUC {auc:.3f}, needed {floor:.3f}")
smoke = model.predict_proba(test_x.head(3))[:, 1] # stage 4: smoke test of the packaged model
assert np.all((smoke >= 0) & (smoke <= 1))
state["champion"], state["version"] = auc, state["version"] + 1
print(f"{commit:<40} deployed as v{state['version']} (AUC {auc:.3f})")
print(f"{len(delivery):,} wines, {delivery.duplicated().sum():,} exact duplicate rows; target: rated 7 or higher")
print(f"baseline AUC if the duplicates are left in: {fit_and_score(add_features(delivery))[1]:.3f}\n")
text_alcohol = delivery.assign(alcohol=delivery.alcohol.map(lambda value: f"{value}%"))
pipeline("1. baseline model")
pipeline("2. new delivery, alcohol as text", data=text_alcohol)
pipeline("3. refactor of the feature code", make_features=add_features_refactored)
pipeline("4. smaller, faster model", params={"max_depth": 2, "max_iter": 20})
pipeline("5. tuned model", params={"learning_rate": 0.05, "max_iter": 400, "max_leaf_nodes": 63})
print(f"\nmodels trained: {state['trained']} of 5 commits; versions deployed: {state['version']}")6,497 wines, 1,177 exact duplicate rows; target: rated 7 or higher
baseline AUC if the duplicates are left in: 0.896
1. baseline model deployed as v1 (AUC 0.822)
2. new delivery, alcohol as text stopped at data validation: alcohol is str, expected float64
3. refactor of the feature code stopped at unit tests: free_share is a fraction
4. smaller, faster model stopped at quality gate: AUC 0.797, needed 0.817
5. tuned model deployed as v2 (AUC 0.826)
models trained: 3 of 5 commits; versions deployed: 2What the output shows
- The data had a trap before any commit. 1,177 rows were exact duplicates. Left in, they pushed the baseline AUC to 0.896, against 0.822 once removed, because copies of the same wine landed on both sides of the train-test split. A check that would never fail a build still changed every number downstream.
- Commit 2 never reached training. Data validation stopped a delivery in which alcohol arrived as text, such as "9.4%".
- Commit 3 never reached the data. A unit test caught a refactor that inverted the free-to-total sulfur dioxide ratio; a share above 1 is impossible. Without the test, the model would have trained happily on a meaningless feature.
- Commit 4 failed on quality. The smaller, faster model passed every other check, but its AUC fell to 0.797 against the 0.817 required. Speeding a model up is fine; losing quality without anyone deciding to is not, and the gate turns that judgement into a rule.
- Commit 5 shipped. The tuned model reached 0.826 and deployed as version 2.
- Only 3 of 5 commits trained a model. The cheap stages stopped the rest.
Setting the gate
A quality gate needs three decisions written down before it ever fails. The first is the metric and the data it is measured on: a fixed, versioned test set, so that a change in the score means a change in the model rather than in the sample. The second is an absolute floor below which nothing ships, whatever the current model scores. The third is a tolerance against the deployed model, sized to the normal noise in the metric; here it is 0.005 of AUC. When the gate blocks a change the team still wants, change the rule in a reviewed commit instead of skipping the stage.
From script to CI system
In a real setup each stage becomes a job in a CI service such as GitHub Actions, GitLab CI or Jenkins; see GitHub Actions for ML. Around the jobs:
- Version the data and register models, so every deployment traces back to its inputs; see data versioning with DVC and the model registry.
- Write tests for data and models, not just code; see testing machine learning code and data.
- Release with a strategy, such as shadow mode or a canary; see shadow deployment and canary releases.
- Keep credentials out of the pipeline's code and logs; see secrets management for ML pipelines.
- Retrain automatically when needed; see continuous training.
Common mistakes
- Testing only the code.
- Training before the cheap checks have run.
- Gates with no tolerance, or a test set that moves between runs.
- Ignoring duplicates and leakage in evaluation.
- A manual step between the gate and production.
What is CI/CD for machine learning?
What tests should an ML pipeline run?
How is CI/CD for ML different from regular CI/CD?
Which tools are used for CI/CD in machine learning?
The CI/CD for Machine Learning module of the MLOps course covers CI/CD fundamentals, automated model deployment, testing ML pipelines and continuous training.
Sources
- Continuous Delivery for Machine Learning (martinfowler.com) — Danilo Sato, Arif Wider and Christoph Windheuser
- MLOps: Continuous delivery and automation pipelines in machine learning (Google Cloud) — Google Cloud
- Wine Quality dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master MLOps?
Join GloryTecks’ expert-led MLOps training in Hyderabad with real-time projects and 100% placement support.
Monitoring ML Models in Production: A Complete Guide
NextKubernetes for ML: Deploying Models at Scale
Related articles
Docker for Machine Learning: A Practical Guide
How to containerise a machine-learning project so it runs the same everywhere: pinning the exact library versions a saved model depends on, a Dockerfile ordered for fast rebuilds, keeping data out of images, multi-stage builds, non-root users and GPU access.
What Is MLOps? DevOps for Machine Learning Explained
MLOps is the practice of building machine learning systems that keep working after launch. This guide defines it, explains why ML needs more than DevOps, and shows the problem it solves on real data: a model built in 2011 that was 38% wrong through 2012, and how much the way you retrain it matters.
Kubeflow Pipelines: An Introduction for ML Engineers
Kubeflow Pipelines turns Python functions into containerised steps that pass datasets and models to each other. This introduction builds a four-step training pipeline on real Beijing air-quality data, compiles it, runs it locally in Docker, and shows a data check stopping a run before training.