A/B Testing for Data Scientists: A Practical Guide

An A/B test answers one question: does a change cause a better outcome than what you have now? Randomly assign users to the current version (A) or the new one (B), measure the same outcome in both, and compare. The mechanics take an afternoon to learn. The discipline is in the design and in resisting the urge to read the results early, and that is where most tests that "won" but changed nothing went wrong.

Design before you launch

Write four things down before any traffic is split:

1. The hypothesis: "Showing delivery dates on the product page increases the share of visitors who start checkout."
2. One primary metric that decides the test, here checkout-start rate. Other metrics are guardrails (page load time, refunds) or background reading, not additional chances to win.
3. The unit of randomisation: usually the user, not the page view, so one person always sees the same version.
4. The minimum effect worth detecting and the test's duration. If a lift smaller than one percentage point would not change any decision, there is no point designing a test to detect it.

This is standard hypothesis testing (NIST e-Handbook of Statistical Methods): a null hypothesis of no difference, a significance level (usually 5%), and power, the probability of detecting the effect if it is real (usually 80%).

Sample size: calculate it, then commit to it

Small effects need large samples. For comparing two conversion rates, the required users per group follow from the baseline rate, the effect size, the significance level and the power:

import math
import numpy as np
from scipy.stats import chisquare, norm

def users_per_group(p_control, p_variant, alpha=0.05, power=0.80):
    z_alpha = norm.ppf(1 - alpha / 2)
    z_power = norm.ppf(power)
    p_bar = (p_control + p_variant) / 2
    top = (z_alpha * math.sqrt(2 * p_bar * (1 - p_bar))
           + z_power * math.sqrt(p_control * (1 - p_control) + p_variant * (1 - p_variant))) ** 2
    return math.ceil(top / (p_variant - p_control) ** 2)

print(users_per_group(0.10, 0.11), users_per_group(0.10, 0.12))

14751 3841

Detecting a lift from 10% to 11% needs about 14,750 users in each group; detecting 10% to 12% needs about 3,840. Halving the effect roughly quadruples the sample. norm.ppf gives the critical values of the normal distribution (SciPy: norm). Divide the total by your daily traffic to get the duration, round up to whole weeks so every weekday is represented, and do not stop before then.

Analysing a conversion test

The simulation below plays the part of a finished test: 15,000 users per group, with a true rate of 10.0% for A and 11.2% for B.

rng = np.random.default_rng(7)
n = 15_000
a = rng.binomial(1, 0.100, n)
b = rng.binomial(1, 0.112, n)

rate_a, rate_b = a.mean(), b.mean()
pooled = (a.sum() + b.sum()) / (2 * n)
z = (rate_b - rate_a) / math.sqrt(pooled * (1 - pooled) * 2 / n)
p_value = 2 * (1 - norm.cdf(abs(z)))
se_diff = math.sqrt(rate_a * (1 - rate_a) / n + rate_b * (1 - rate_b) / n)
low, high = (rate_b - rate_a) - 1.96 * se_diff, (rate_b - rate_a) + 1.96 * se_diff
print(f"A {rate_a:.4f}  B {rate_b:.4f}  lift {rate_b - rate_a:+.4f}  z={z:.2f}  p={p_value:.4f}  95% CI [{low:+.4f}, {high:+.4f}]")

A 0.1035  B 0.1145  lift +0.0110  z=3.06  p=0.0022  95% CI [+0.0039, +0.0181]

The two-proportion z-test gives p = 0.0022, well under 0.05, so the difference is unlikely to be chance. The confidence interval is the more useful output: the lift is probably between 0.4 and 1.8 percentage points. If the smallest lift that justifies shipping were 1.5 points, this result would be significant but not conclusive for the business. Always report the interval, not just the p-value.

When the metric is not a conversion

For averages, such as revenue per user or session length, compare the two groups with Welch's t-test, which does not assume equal variances; SciPy's ttest_ind with equal_var=False runs it, as the Python for data science guide shows. Revenue is usually skewed, with a few very large orders, which widens the interval and inflates the sample you need. Common remedies are capping extreme values at a high percentile, agreed before the test, or analysing a transformed metric such as whether the user bought at all. Whatever you choose, decide it in advance so the analysis cannot drift towards the answer you hoped for.

Check the split before trusting anything

If you planned a 50/50 split, the group sizes should be close to equal. A large imbalance, called a sample ratio mismatch, means something in the assignment or logging is broken, for example users of one variant crashing before they are counted, and the results cannot be trusted however significant they look. A chi-square goodness-of-fit test flags it (SciPy: chisquare):

print(chisquare([15_000, 14_620]))

Power_divergenceResult(statistic=4.87508440243079, pvalue=0.027247108371904246)

With p = 0.027, a split of 15,000 against 14,620 is unlikely under a true 50/50 assignment. Investigate before reading the outcome metric.

Traps that produce confident wrong answers

• Peeking. Checking the p-value every day and stopping as soon as it dips below 0.05 inflates the false-positive rate far above 5%. Fix the sample size in advance, or use a sequential testing method designed for continuous monitoring.
• Many metrics. Test twenty metrics at 5% and, on average, one will be "significant" by chance. Decide on the primary metric in advance and correct for multiple comparisons when several really matter.
• Novelty and learning effects. Users react to change itself. Run long enough to see behaviour settle, and look at the trend over the test.
• Wrong unit. Randomising by session while measuring per user lets one person see both versions and violates the independence the test assumes.
• Segment fishing. Slicing results by city, device and age after the fact until something wins is the multiple-metrics problem in disguise. Segment analyses are for generating the next hypothesis, not for declaring this one.

When an A/B test is the wrong tool

Some questions cannot be randomised: a change that affects everyone at once (a price change announced publicly), network effects where treated users influence the control group, or decisions with too little traffic to ever reach the required sample. In those cases, quasi-experimental methods, such as comparing before and after against a control region, or qualitative research, give better evidence than an underpowered test.

The statistics underneath, including distributions, p-values and confidence intervals, are covered in statistics for data science and probability for data science. Hypothesis testing and inferential statistics are part of the mathematics and statistics module of the Data Science course.

Sources

• NIST/SEMATECH e-Handbook of Statistical Methods — NIST
• scipy.stats.norm — SciPy documentation
• scipy.stats.chisquare — SciPy documentation