What Is MLOps? DevOps for Machine Learning Explained

A data scientist trains a model, it scores well on test data, and it goes live. Six months later it is quietly wrong, and nobody knows. This is the most common way machine learning disappoints in practice, and the reason MLOps exists.

Google's architecture guide defines MLOps as an engineering culture and practice that unites the development and the operation of ML systems, with automation and monitoring at every step. It adds the point that matters most: the real challenge is not building a model but building an integrated system and operating it continuously in production (Google Cloud: MLOps).

Why machine learning needs its own practice

Ordinary software does what its code says until someone changes the code. A model does what its training data taught it, and the world keeps moving after training ends. That one difference creates most of the work:

• Data changes, so a model can get worse without any change to its code.
• Results depend on data, code and settings together, so reproducing a model needs all three.
• Quality is statistical, so passing tests is not the same as being right.
• Only a small part of the system is model code. Google's guide and the paper behind it note that the surrounding configuration, data pipelines, serving and monitoring are vast by comparison.

That paper, presented at NeurIPS in 2015, argued that quick wins from machine learning are not free and that real ML systems commonly incur massive ongoing maintenance costs (Sculley et al.: Hidden Technical Debt). MLOps is largely the discipline of paying that debt down.

The problem, measured

The script shows model decay on real data. It predicts daily bike rentals from the UCI Bike Sharing dataset, released under CC BY 4.0 (UCI: Bike Sharing). One model is trained on 2011 and never touched again. Two others are retrained at the start of each month of 2012, one on the latest 12 months and one on the latest 3.

import pandas as pd
from sklearn.ensemble import RandomForestRegressor

day = pd.read_csv("data/day.csv")                               # UCI Bike Sharing (CC BY 4.0)
day["month"] = day.dteday.str[:7]
FEATURES = ["season", "mnth", "holiday", "weekday", "workingday", "weathersit", "temp", "atemp", "hum", "windspeed"]


def wape(actual, predicted):
    # total error as a share of total rentals: robust to a single near-zero day
    return (actual - predicted).abs().sum() / actual.sum()


def train(data):
    return RandomForestRegressor(n_estimators=200, random_state=42).fit(data[FEATURES], data.cnt)


frozen = train(day[day.dteday < "2012-01-01"])                  # built once on 2011, never updated
months = sorted(day.month.unique())
results = []
print(f"{'month':<8} {'frozen':>7} {'retrained, last 12 months':>26} {'retrained, last 3 months':>25}")
for i in range(12, len(months)):
    now = day[day.month == months[i]]
    row = [wape(now.cnt, frozen.predict(now[FEATURES]))]
    for window in (12, 3):                                      # retrain at the start of each month
        recent = day[day.month.isin(months[i - window:i])]
        row.append(wape(now.cnt, train(recent).predict(now[FEATURES])))
    results.append(row)
    print(f"{months[i]:<8} {row[0]:>7.0%} {row[1]:>26.0%} {row[2]:>25.0%}")

mean = pd.DataFrame(results).mean()
print(f"{'2012':<8} {mean[0]:>7.0%} {mean[1]:>26.0%} {mean[2]:>25.0%}")
print(f"\naverage rentals a day: 2011 {day[day.dteday < '2012-01-01'].cnt.mean():,.0f}, "
      f"2012 {day[day.dteday >= '2012-01-01'].cnt.mean():,.0f}")

month     frozen  retrained, last 12 months  retrained, last 3 months
2012-01      54%                        54%                       38%
2012-02      47%                        45%                       14%
2012-03      46%                        39%                       30%
2012-04      42%                        39%                       13%
2012-05      29%                        28%                       12%
2012-06      30%                        29%                       11%
2012-07      33%                        33%                       10%
2012-08      34%                        35%                        7%
2012-09      36%                        36%                        8%
2012-10      41%                        38%                       16%
2012-11      34%                        32%                       24%
2012-12      35%                        28%                       30%
2012         38%                        36%                       18%

average rentals a day: 2011 3,406, 2012 5,600

What the output shows

• The world moved. Average rentals rose from 3,406 a day in 2011 to 5,600 in 2012. The frozen model had learned the old level.
• The frozen model was wrong by 38% of all rentals across 2012, worst in January at 54%. Nothing in its code changed; the data did.
• Retraining is not automatically the fix. Retrained every month on the last 12 months, the error barely moved, to 36%, because a year of history still described the old level.
• Retraining well halved the error. On the latest 3 months, error fell to 18% across the year, and to 7% in August.
• There was a trade-off. In November and December the short window got worse, 24% and 30%, because it had learned from summer months. How to retrain is a modelling decision.

None of this is visible without monitoring. The frozen model would have kept answering confidently all year.

The MLOps lifecycle

The loop closes: monitoring feeds retraining, which feeds validation and deployment again. Microsoft's maturity model describes the most advanced teams as those where production metrics trigger retraining automatically (Microsoft: MLOps maturity model). See MLOps maturity levels.

What MLOps is not

• Not a product. Tools help, but the practice is how a team builds, releases and watches models.
• Not only for large companies. One data scientist with one model benefits from tracking, versioning and a way to notice decay.
• Not full automation by default. Many good setups keep a person approving each release.
• Not a one-off project. Like the models it supports, it needs regular attention.

Where to start

Track every experiment, version the data, and put a model behind a tested release process with a way back. Then add monitoring, and only then automate retraining. For the tools behind each step, see the MLOps tools landscape. For how MLOps relates to DevOps and DataOps, see MLOps vs DevOps vs DataOps. For the learning path, see the MLOps roadmap.

Common mistakes

• Treating launch as the finish line.
• Retraining on a schedule without checking whether the new model is better.
• Monitoring servers but not predictions.
• Automating before the basics, such as versioning and tests, are in place.
• Assuming more data is always better, when recent data may matter more.

The ML lifecycle, MLOps versus DevOps and the challenges of ML in production open the MLOps course.

Sources

• MLOps: Continuous delivery and automation pipelines in machine learning (Google Cloud) — Google Cloud
• Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015) — Sculley et al.
• MLOps maturity model (Microsoft Learn) — Microsoft
• Bike Sharing dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository