Shadow Deployment and Canary Releases for ML
Shadow mode runs a new model on real traffic without letting it answer; a canary lets it answer for a few users first. On real e-commerce sessions, this guide shows a unit bug that 89% agreement hid, the flag-rate guardrail that caught it, and why per-stage business metrics were too noisy to gate on.
GloryTecks Team
Training & Placement Faculty
Last updated: 8 October 2026
Releasing a new model is a bet that what worked offline will work on live traffic. Two techniques make that bet smaller. A shadow deployment runs the new model on real requests without letting it answer anyone. A canary release lets it answer for a small share of users first, and widens the share only while guardrails hold.
This guide does both on real e-commerce sessions, with a challenger carrying a realistic bug. It shows what each step caught, and two signals that would have misled.
Two ways to reduce launch risk
In shadow mode, the new model receives a copy of each live request and its answers are logged but never returned. AWS's SageMaker shadow tests work this way: a copy of each request goes to the shadow variant in real time, only the production variant's response reaches the caller, and the shadow's responses can be logged for comparison (AWS: shadow tests).
Google's SRE workbook defines canarying as a partial, time-limited deployment of a change together with its evaluation. The part that gets the change is the canary and the rest is the control (Google SRE: canarying releases). Martin Fowler's site describes it as rolling a change out to a small subset of users before everyone (martinfowler.com: canary release).
| Shadow deployment | Canary release | |
|---|---|---|
| Users affected | None | A small, growing share |
| What you learn | Correctness, agreement, output distributions, latency | Safety with real users, early outcome signals |
| Main cost | Running two models on all traffic | Some users meet the new model first |
| Cannot show | How users react to the new answers | Small effects, on small samples |
The setup
The UCI Online Shoppers data hold 12,330 sessions on an online shop over a year, each labelled by whether it ended in a purchase, under CC BY 4.0 (UCI: Online Shoppers). The champion is a logistic regression and the challenger a gradient-boosted model, both trained on February to September. The service flags sessions likely to buy, for example to offer live chat.
The challenger reads its inputs from a new feature service, and that service carries a bug injected on purpose: it sends bounce and exit rates as percentages, while the model was trained on fractions. October's sessions run in shadow mode. November and December run as a four-stage canary, with one guardrail: the canary's flag rate must stay within 5 points of the control's.
import hashlib
import numpy as np
import pandas as pd
from sklearn.compose import make_column_transformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
shop = pd.read_csv("online_shoppers_intention.csv") # UCI Online Shoppers Purchasing Intention (CC BY 4.0)
shop["Weekend"] = shop.Weekend.astype(int)
shop["session"] = [f"s{i:05d}" for i in range(len(shop))]
CATEGORIES = ["VisitorType"]
NUMBERS = [c for c in shop.columns if c not in CATEGORIES + ["Month", "Revenue", "session"]]
FEATURES = NUMBERS + CATEGORIES
train = shop[shop.Month.isin(["Feb", "Mar", "May", "June", "Jul", "Aug", "Sep"])]
october, holidays = shop[shop.Month == "Oct"], shop[shop.Month.isin(["Nov", "Dec"])]
champion = make_pipeline(make_column_transformer((OneHotEncoder(handle_unknown="ignore"), CATEGORIES),
(StandardScaler(), NUMBERS)),
LogisticRegression(max_iter=2000)).fit(train[FEATURES], train.Revenue)
challenger = make_pipeline(make_column_transformer((OneHotEncoder(handle_unknown="ignore"), CATEGORIES),
remainder="passthrough"),
HistGradientBoostingClassifier(random_state=0)).fit(train[FEATURES], train.Revenue)
def new_feature_service(sessions): # the challenger's new input path, with a deliberately injected unit bug
sessions = sessions.copy()
sessions[["BounceRates", "ExitRates"]] *= 100 # sends percentages; training used fractions
return sessions
def flags(model, sessions): # flag likely buyers, for example to offer live chat
return model.predict_proba(sessions[FEATURES])[:, 1] >= 0.5
print("1. Shadow mode in October: the champion answers, the challenger only logs")
served = flags(champion, october)
for name, inputs in (("challenger, new feature service", new_feature_service(october)),
("challenger, unit bug fixed", october)):
shadow = flags(challenger, inputs)
print(f" {name:<32} agrees on {np.mean(shadow == served):.0%} of sessions; "
f"flags {shadow.mean():.1%} against the champion's {served.mean():.1%}")
print("\n2. October outcomes, once they arrive")
buyers, fixed = october.Revenue.to_numpy(), flags(challenger, october)
for name, flagged in (("champion", served), ("challenger", fixed)):
print(f" {name:<10} {buyers[flagged].mean():.0%} of flagged sessions bought; {flagged[buyers].mean():.0%} of buyers flagged")
print("\n3. Canary in November and December; guardrail: flag rate within 5 points of the control")
bucket = holidays.session.map(lambda s: int(hashlib.sha256(f"canary:{s}".encode()).hexdigest(), 16) % 100).to_numpy()
STAGES = ((5, 1000), (25, 1000), (50, 1000), (100, 1725)) # percent of traffic, sessions in the stage
def canary(prepare):
start = 0
for percent, size in STAGES:
window, to_canary = holidays.iloc[start:start + size], bucket[start:start + size] < percent
start += size
new = flags(challenger, prepare(window[to_canary]))
bought = window[to_canary].Revenue.to_numpy()[new]
detail = f"{bought.mean():.0%} of {new.sum()} flags bought" if new.any() else "no flags"
if percent == 100:
print(f" 100%: {len(new):>4} sessions, flag rate {new.mean():5.1%}; {detail} -> fully released")
return
old = flags(champion, window[~to_canary])
gap = new.mean() - old.mean()
verdict = "continue" if abs(gap) <= 0.05 else "ROLL BACK"
print(f" {percent:>3}%: {len(new):>4} sessions, flag rate {new.mean():5.1%} vs {old.mean():5.1%} "
f"({gap * 100:+.1f} points); {detail} -> {verdict}")
if verdict == "ROLL BACK":
print(f" sessions served by the faulty model: {len(new)} of the {size:,} in this stage")
return
print(" the fixed challenger:")
canary(lambda sessions: sessions)
print(" the buggy challenger, as if shadow mode had been skipped:")
canary(new_feature_service)1. Shadow mode in October: the champion answers, the challenger only logs
challenger, new feature service agrees on 89% of sessions; flags 1.3% against the champion's 12.4%
challenger, unit bug fixed agrees on 91% of sessions; flags 16.2% against the champion's 12.4%
2. October outcomes, once they arrive
champion 76% of flagged sessions bought; 45% of buyers flagged
challenger 74% of flagged sessions bought; 57% of buyers flagged
3. Canary in November and December; guardrail: flag rate within 5 points of the control
the fixed challenger:
5%: 54 sessions, flag rate 11.1% vs 10.9% (+0.2 points); 83% of 6 flags bought -> continue
25%: 255 sessions, flag rate 9.8% vs 8.6% (+1.2 points); 68% of 25 flags bought -> continue
50%: 498 sessions, flag rate 9.8% vs 9.0% (+0.9 points); 80% of 49 flags bought -> continue
100%: 1725 sessions, flag rate 9.9%; 75% of 170 flags bought -> fully released
the buggy challenger, as if shadow mode had been skipped:
5%: 54 sessions, flag rate 3.7% vs 10.9% (-7.2 points); 100% of 2 flags bought -> ROLL BACK
sessions served by the faulty model: 54 of the 1,000 in this stageWhat shadow mode caught
- Agreement hid the bug. The faulty challenger agreed with the champion on 89% of sessions, which sounds healthy. But it flagged 1.3% of sessions against 12.4%. Most sessions are easy non-buyers that any model gets right, so overall agreement says little. Compare decision rates and score distributions.
- The bug was training/serving skew, the same feature arriving differently in training and in serving. The ML Test Score calls the skew test perhaps the most important and least implemented, and blames it for production issues across many teams (Breck et al.: The ML Test Score).
- Fixed, the challenger looked promising. It flagged 16.2% and agreed on 91%. Once October's outcomes arrived, 74% of its flagged sessions had bought, against the champion's 76%, and it caught 57% of buyers against 45%.
- Nobody was exposed. Every answer customers received came from the champion.
What the canary showed
- The fixed challenger passed every stage. Its flag rate stayed within 1.2 points of the control's at 5%, 25% and 50% of traffic, and it was fully released.
- Business metrics were too noisy to gate early stages. The share of flagged sessions that bought swung between 68% and 83% on 6 to 49 flags per stage. Gate early stages on fast, high-volume signals, and judge business effect at larger stages or in a proper A/B test.
- Without shadow mode, the canary still limited the damage. The faulty challenger fell 7.2 points below the control at the first stage and was rolled back after serving 54 of the stage's 1,000 sessions.
- One guardrail would have failed silently. Both of the faulty model's two flags were buyers, a perfect 100%. A precision-only guardrail would have waved it through.
The SRE workbook adds a caution: a canary is an A/B comparison, and both parts can change together, so absolute checks against service-level objectives are needed too.
Designing your own
- Mirror requests asynchronously, so the shadow model can never slow or break a response.
- Strip side effects from the shadow. It must not send messages, charge cards or write to production tables.
- Log both answers with the request ID, then compare decision rates, score distributions, disagreement by segment, latency and errors.
- Route canary traffic by a stable hash of the user or session, so each person sees one model throughout.
- Write stages, guardrails and rollback rules before starting, and automate the rollback.
- Hand effect sizes to an A/B test; see A/B testing ML models in production.
For switching whole environments instead, see blue-green deployments. For the serving layer itself, see serving ML models with FastAPI and Docker, and for keeping the previous version ready to restore, see the model registry.
Common mistakes
- Judging shadow results by overall agreement.
- Shadow models with side effects.
- Gating early canary stages on sparse business metrics.
- Routing canary traffic per request instead of per user.
- Rollbacks that need a meeting instead of an automated rule.
What is shadow deployment in machine learning?
What is the difference between a canary release and an A/B test?
How long should each canary stage last?
What should I compare in shadow mode?
Deployment strategies such as these are part of the Model Deployment module of the MLOps course.
Sources
- Shadow tests (Amazon SageMaker AI documentation) — Amazon Web Services
- Canarying Releases (Google SRE workbook) — Google
- Canary Release (martinfowler.com) — Danilo Sato, martinfowler.com
- The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction (IEEE Big Data 2017) — Breck et al., Google
- Online Shoppers Purchasing Intention dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master MLOps?
Join GloryTecks’ expert-led MLOps training in Hyderabad with real-time projects and 100% placement support.
Related articles
MLOps Roadmap 2026: Skills, Tools and Career Path
MLOps sits between data science, software engineering and operations, which makes it hard to know where to start. This roadmap orders the skills from foundations to production operations, gives each stage a project that proves the skill, and explains what the MLOps engineer role actually involves.
MLflow Tutorial: Tracking Experiments and Models
MLflow records what produced each model so you never have to remember. This tutorial trains three models on real bike-sharing data, logs every run, compares them by querying the tracking store, registers the best one, points a champion alias at it and loads it back by that alias, all on a laptop.
Docker for Machine Learning: A Practical Guide
How to containerise a machine-learning project so it runs the same everywhere: pinning the exact library versions a saved model depends on, a Dockerfile ordered for fast rebuilds, keeping data out of images, multi-stage builds, non-root users and GPU access.