Bias-Variance Tradeoff Explained with Real Examples

Every model's error on new data comes from three sources. Bias is error from wrong assumptions: a straight line fitted to a curve is wrong no matter how much data you give it. Variance is error from sensitivity to the particular training sample: a model that bends to fit every point changes shape every time the sample changes. Noise is randomness in the data itself, which no model can remove. Making a model more flexible lowers bias and raises variance; the tradeoff is finding the flexibility where their sum is smallest.

That description is familiar. It is more convincing measured, so here is an experiment that measures it.

The experiment

The true relationship is a sine curve, and each training set is 40 noisy observations of it. For each model complexity, polynomial degree 1, 3, 5 and 9, the code draws 200 independent training sets, fits a model to each, and records its predictions at 50 fixed test points (scikit-learn: PolynomialFeatures). The average prediction across the 200 models shows systematic error, bias; the spread of predictions shows variance.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

rng = np.random.default_rng(0)

def true_f(x):
    return np.sin(2 * np.pi * x)

x_test = np.linspace(0.1, 0.9, 50)
noise = 0.3

for degree in (1, 3, 5, 9):
    preds = []
    for _ in range(200):                                    # 200 independent training sets
        x = rng.uniform(0, 1, 40)
        y = true_f(x) + rng.normal(0, noise, 40)
        model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(x[:, None], y)
        preds.append(model.predict(x_test[:, None]))
    preds = np.array(preds)
    bias2 = np.mean((preds.mean(axis=0) - true_f(x_test)) ** 2)
    variance = np.mean(preds.var(axis=0))
    print(f"degree {degree}: bias^2 {bias2:.3f}  variance {variance:.3f}  total error {bias2 + variance + noise**2:.3f}")

degree 1: bias^2 0.151  variance 0.014  total error 0.255
degree 3: bias^2 0.003  variance 0.007  total error 0.101
degree 5: bias^2 0.000  variance 0.012  total error 0.103
degree 9: bias^2 0.001  variance 0.142  total error 0.233

The total is the expected squared error on new data: bias squared plus variance plus the noise variance of 0.09, which no model can go below.

Reading the results

• Degree 1 (a straight line) has almost all its error in bias. Its 200 fitted lines agree closely with each other (low variance) and are all wrong in the same way, because a line cannot follow a sine wave. More data would not help.
• Degree 3 removes nearly all the bias while keeping variance low. Its total error, 0.101, is barely above the noise floor of 0.09: this is the sweet spot for this problem.
• Degree 5 is about as good: its bias is essentially zero, and its slight extra variance is the price of flexibility it did not need.
• Degree 9 has almost no bias but ten times the variance of degree 3. Each model bends to fit its own 40 noisy points, so the 200 models disagree wildly between observations, and the total error more than doubles.

That is the tradeoff in numbers: as complexity rises, bias falls and variance rises, and total error traces a U. scikit-learn's own underfitting and overfitting example shows the same effect visually (scikit-learn: underfitting vs overfitting).

Recognising each in a real project

You never have 200 training sets, and you never know the true function. You do have training and validation scores, and their pattern tells you which problem dominates (scikit-learn: validation and learning curves):

A learning curve, error plotted against training-set size, adds one more clue. With high bias, both curves flatten out early at a high error, and more data does not help. With high variance, the gap between them narrows as data grows, so more data is a real fix.

A diagnosis, step by step

Suppose a churn model built with logistic regression on five customer attributes scores about the same on training and validation data, and both scores are disappointing. The two numbers being close rules out overfitting, so variance is not the problem, and collecting more of the same data will not help. The model is too simple for the pattern, so the right moves are richer features, such as usage trends over time instead of a single snapshot, or a more flexible model. Now suppose a gradient-boosted model on 200 features scores almost perfectly on training data and much worse on validation. That gap is variance: reduce the number of features, constrain the trees, add regularisation or add data, and watch whether the gap closes.

Fixing high bias

• Use a more flexible model: add features or interactions, use a non-linear model such as a tree ensemble, or reduce regularisation.
• Engineer better features. A model cannot use information it is never given; see feature engineering.
• Train longer, for iterative models that stopped too early.

Fixing high variance

• Collect more training data, if the learning curve says it will help.
• Increase regularisation: a larger penalty in ridge or lasso regression, fewer leaves or a shallower depth in trees, dropout in neural networks.
• Reduce the number of features, removing noisy or redundant ones.
• Average many models. Bagging and random forests reduce variance by averaging trees trained on different samples; ensemble learning explains why that works.

Choose the fix after diagnosing, not before. Adding data to a high-bias model and making a high-variance model more complex are the two most common ways to spend a week without improving anything.

A note on modern models

Very large models, such as deep networks with more parameters than training examples, can generalise well despite enormous capacity, which the simple U-curve does not predict. Regularisation, training procedures and data scale all play a part. For the tabular problems most data scientists work on, the classical picture above remains a reliable guide to diagnosis.

The practical fixes for each side are collected in overfitting and underfitting: causes and fixes. Model evaluation, including overfitting and underfitting, is part of the model evaluation and optimisation module of the Data Science course.

Sources

• Underfitting vs. Overfitting (scikit-learn example) — scikit-learn documentation
• Validation curves and learning curves (scikit-learn) — scikit-learn documentation
• PolynomialFeatures — scikit-learn documentation