A/B Testing ML Models in Production
A model that wins offline still has to prove itself with real customers. On real bank marketing outcomes, this guide shows why a one-point offline gain needed about 36,000 calls per arm to confirm, how to split traffic and check the split, and how checking results early turned 4% false alarms into 18%.
GloryTecks Team
Training & Placement Faculty
Last updated: 8 October 2026
Offline evaluation tells you which model scores better on historical data. An A/B test asks a different question: does the new model make things better for real customers, measured on the outcome the business cares about? Only a live comparison can answer that, because models change what customers see and do.
A/B testing ML models goes wrong in a few predictable ways: samples too small to detect the gain, splits that silently break, and results read too early. This guide shows each one on real outcomes from a bank's phone campaigns.
When an A/B test is the right tool
| Method | Question it answers | Who sees the new model |
|---|---|---|
| Shadow deployment | Does it run correctly on real traffic? | Nobody |
| Canary release | Is it safe at small scale? | A few percent |
| A/B test | Is it better, on the outcome that matters? | A randomised share, often half |
| Blue-green deployment | Can we switch and switch back instantly? | Everyone, at the switch |
Shadow and canary releases check safety; an A/B test measures effect. See shadow deployment and canary releases and blue-green deployments.
Designing the test
- Randomise people, not requests. Assign each customer to one arm by hashing a stable ID with a salt for this experiment, so a person always sees the same model and each experiment gets an independent split.
- Pick one primary metric that reflects the goal, here subscriptions per call, plus guardrails such as complaints, latency and call volume.
- Fix the sample size in advance from the smallest lift worth detecting.
- Run whole business cycles, at least full weeks, so weekday effects cancel out.
The data and the models
The UCI Bank Marketing data hold 41,188 phone contacts by a Portuguese bank from May 2008 to November 2010, with whether each client subscribed a term deposit, under CC BY 4.0. The dataset notes warn that call duration is known only after a call and should be dropped for a realistic model, so it is dropped here (UCI: Bank Marketing).
Half the rows train a champion, logistic regression, and a challenger, gradient boosting. The other half is the experiment's population: in each arm, that arm's model calls the 20% of customers it rates most likely to subscribe. Every customer in the data was actually called, so the outcome of calling each one is known. This is a replay of historical outcomes, not a live test, which is what makes it reproducible.
import hashlib
import numpy as np
import pandas as pd
from scipy.stats import chisquare, norm
from sklearn.compose import make_column_transformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
bank = pd.read_csv("bank-additional-full.csv", sep=";").drop(columns="duration") # call length is known only afterwards
bank["customer"] = [f"c{i:05d}" for i in range(len(bank))]
bank["subscribed"] = (bank.pop("y") == "yes").astype(int)
NUMERIC = [c for c in bank.columns if pd.api.types.is_numeric_dtype(bank[c]) and c != "subscribed"]
CATEGORIES = [c for c in bank.columns if c not in NUMERIC + ["customer", "subscribed"]]
FEATURES = CATEGORIES + NUMERIC
in_history = np.random.default_rng(0).random(len(bank)) < 0.5
train, population = bank[in_history], bank[~in_history]
champion = make_pipeline(make_column_transformer((OneHotEncoder(handle_unknown="ignore"), CATEGORIES),
(StandardScaler(), NUMERIC)),
LogisticRegression(max_iter=2000)).fit(train[FEATURES], train.subscribed)
challenger = make_pipeline(make_column_transformer((OneHotEncoder(handle_unknown="ignore"), CATEGORIES),
remainder="passthrough"),
HistGradientBoostingClassifier(random_state=0)).fit(train[FEATURES], train.subscribed)
def call_list(model, customers, share=0.2): # each model calls the 20% it rates most likely to subscribe
scores = model.predict_proba(customers[FEATURES])[:, 1]
return customers.iloc[np.argsort(-scores)[: int(len(customers) * share)]]
def calls_needed(p1, p2, alpha=0.05, power=0.8): # per arm, for a two-sided test of two proportions
z = norm.ppf(1 - alpha / 2) + norm.ppf(power)
return int(np.ceil(z ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2))
print("1. Offline, on the same customers")
for name, model in (("champion, logistic regression", champion), ("challenger, gradient boosting", challenger)):
auc = roc_auc_score(population.subscribed, model.predict_proba(population[FEATURES])[:, 1])
print(f" {name:<30} AUC {auc:.3f}, top-20% conversion {call_list(model, population).subscribed.mean():.1%}")
base = call_list(champion, population).subscribed.mean()
print("\n2. Calls needed per arm (5% significance, 80% power)")
for lift in (0.01, 0.03, 0.05):
print(f" to detect {base:.1%} -> {base + lift:.1%}: {calls_needed(base, base + lift):,}")
print("\n3. The experiment, customers split by a salted hash of their ID")
EXPERIMENT = "call-model-2026-10" # salt: each experiment gets its own split
arm = population.customer.map(lambda c: int(hashlib.sha256(f"{EXPERIMENT}:{c}".encode()).hexdigest(), 16) % 2)
sizes = arm.value_counts().sort_index()
print(f" customers per arm {sizes[0]:,} and {sizes[1]:,}; sample ratio check p = {chisquare(sizes).pvalue:.3f}")
results = []
for label, model, group in (("A, champion", champion, 0), ("B, challenger", challenger, 1)):
called = call_list(model, population[arm == group])
results.append((called.subscribed.sum(), len(called)))
print(f" {label:<14} {len(called):,} calls, {called.subscribed.sum():,} subscriptions, {called.subscribed.mean():.1%}")
(x_a, n_a), (x_b, n_b) = results
rate_a, rate_b = x_a / n_a, x_b / n_b
pooled = (x_a + x_b) / (n_a + n_b)
z = (rate_b - rate_a) / np.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
margin = norm.ppf(0.975) * np.sqrt(rate_a * (1 - rate_a) / n_a + rate_b * (1 - rate_b) / n_b)
diff = (rate_b - rate_a) * 100
print(f" B minus A {diff:+.1f} points, 95% interval {diff - margin * 100:+.1f} to {diff + margin * 100:+.1f} points, "
f"p = {2 * norm.sf(abs(z)):.2f}")
print("\n4. Peeking: 2,000 A/A tests with the same model in both arms")
outcomes = call_list(champion, population).subscribed.to_numpy()
rng = np.random.default_rng(1)
at_end = at_any_look = 0
for _ in range(2000):
y, group = outcomes[rng.permutation(len(outcomes))], rng.integers(0, 2, len(outcomes))
flags = []
for share in np.linspace(0.1, 1.0, 10): # look after every tenth of the calls
n = int(len(y) * share)
a, b = y[:n][group[:n] == 0], y[:n][group[:n] == 1]
p = y[:n].mean()
flags.append(abs(b.mean() - a.mean()) / np.sqrt(p * (1 - p) * (1 / len(a) + 1 / len(b))) > norm.ppf(0.975))
at_end += flags[-1]
at_any_look += any(flags)
print(f" 'significant' when checked once, at the planned end: {at_end / 2000:.1%}")
print(f" 'significant' at any of 10 looks: {at_any_look / 2000:.1%}")1. Offline, on the same customers
champion, logistic regression AUC 0.786, top-20% conversion 34.8%
challenger, gradient boosting AUC 0.798, top-20% conversion 35.8%
2. Calls needed per arm (5% significance, 80% power)
to detect 34.8% -> 35.8%: 35,838
to detect 34.8% -> 37.8%: 4,029
to detect 34.8% -> 39.8%: 1,465
3. The experiment, customers split by a salted hash of their ID
customers per arm 10,481 and 10,197; sample ratio check p = 0.048
A, champion 2,096 calls, 740 subscriptions, 35.3%
B, challenger 2,039 calls, 729 subscriptions, 35.8%
B minus A +0.4 points, 95% interval -2.5 to +3.4 points, p = 0.76
4. Peeking: 2,000 A/A tests with the same model in both arms
'significant' when checked once, at the planned end: 4.2%
'significant' at any of 10 looks: 18.4%Reading the results
Offline, the challenger looked better. Its AUC was 0.798 against 0.786, and its top 20% converted at 35.8% against 34.8%: one percentage point.
That point is expensive to confirm. Detecting a rise from 34.8% to 35.8%, at 5% significance with 80% power, needs 35,838 calls per arm. A three-point lift needs 4,029, and five points needs 1,465. Decide before the test which lift is worth finding.
Check the split before the results. The arms held 10,481 and 10,197 customers, and the sample ratio check gave p = 0.048. Microsoft runs this check on every A/B test before showing any results, using a threshold it describes as conservative to avoid false alarms, and reports that about 6% of the tests it analysed had a sample ratio mismatch (Microsoft Research: sample ratio mismatch). A check run on every test needs a strict threshold, or one healthy test in 20 gets flagged. Here nobody could drop out after assignment, so the imbalance is chance. In a live system, a borderline value is a reason to look at segments and assignment logs before trusting anything else.
The test itself was inconclusive. Each arm made about 2,000 calls. B converted 35.8% against A's 35.3%; the gap, 0.4 points before the rates are rounded, came with a 95% interval from -2.5 to +3.4 points and p = 0.76. The right summary is not "no difference" but "this test could not tell".
Peeking manufactures winners. In 2,000 A/A tests with the same model in both arms, 4.2% looked significant when checked once at the planned end, close to the 5% expected. Checking after every tenth of the data and stopping at the first significant result raised that to 18.4%. Evan Miller explains that significance levels assume a sample size fixed in advance, and suggests sequential designs, with checkpoints planned beforehand, for anyone who must look early (Evan Miller: how not to run an A/B test).
When the test is too small
- Ask whether the lift is worth detecting. If not, ship on offline evidence with a guardrail-only canary.
- Expose more traffic or run longer until the planned sample size is reached.
- Choose a more sensitive metric that still reflects the goal.
- Use a sequential design when early stopping is a real need.
Once a challenger wins, promote it through the registry so rollback stays one step away; see the model registry. Keep watching it afterwards; see monitoring ML models in production.
Common mistakes
- Randomising requests instead of people.
- No sample-size plan, so the test is too small to decide anything.
- Stopping at the first significant result.
- Reading results before the sample ratio check.
- Calling an underpowered test proof of no difference.
How do you A/B test a machine learning model?
How long should an A/B test of a model run?
What is a sample ratio mismatch?
Why not just compare models offline?
Deployment strategies, including how new models are released alongside old ones, are part of the Model Deployment module of the MLOps course.
Sources
- Bank Marketing dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository
- How Not To Run an A/B Test (Evan Miller) — Evan Miller
- Diagnosing Sample Ratio Mismatch in A/B Testing (Microsoft Research) — Microsoft Research
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master MLOps?
Join GloryTecks’ expert-led MLOps training in Hyderabad with real-time projects and 100% placement support.
Continuous Training: Automating Model Retraining
NextModel Registry: Managing the ML Model Lifecycle
Related articles
Monitoring ML Models in Production: A Complete Guide
A model that passed validation can fail quietly in production. This guide sets out what to monitor at each layer, how to cope with labels that arrive late and how to design alerts people act on, using a year of real sensor data in which the loudest alert was the least useful one.
Kubernetes for ML: Deploying Models at Scale
Kubernetes runs copies of your model service, replaces the ones that die and swaps versions without downtime, if the manifest is right. This guide deploys a concrete-strength model to a local cluster, kills a pod, rolls out a new version under continuous traffic with no failed requests, and rolls back.
Production ML Systems: Architecture and Best Practices
In a production machine learning system the model is the smallest part. This guide maps the components around it, compares the three serving patterns, explains the ML-specific technical debt that architecture has to contain, and sets out the practices that keep a system maintainable as it grows.