Overfitting and Underfitting: Causes and Fixes

A model can fail in two opposite ways. It can be too simple to capture the pattern in the data, which is underfitting, or so flexible that it memorises noise in the training set, which is overfitting. Both give disappointing results on new data, but they need opposite fixes, so the first job is telling them apart. This guide diagnoses both on a real dataset and measures what each fix does.

The data is 19,020 events from the MAGIC gamma telescope dataset, generated by simulation: ten numeric features describing the image each particle shower leaves in the telescope, labelled as a gamma signal or hadron background (UCI: MAGIC gamma telescope). All scores are accuracy under 5-fold cross-validation.

The diagnosis: compare training and validation scores

The key measurement is simple: score the model on the data it trained on and on data it did not see.

from sklearn.datasets import fetch_openml
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate, learning_curve
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.tree import DecisionTreeClassifier

data = fetch_openml(data_id=1120, as_frame=True)
X, y = data.data, (data.target == "h").astype(int)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

def diagnose(name, model):
    scores = cross_validate(model, X, y, cv=cv, return_train_score=True)
    train, valid = scores["train_score"].mean(), scores["test_score"].mean()
    print(f"{name:<28} train {train:.3f} | validation {valid:.3f} | gap {train - valid:.3f}")

diagnose("logistic regression", make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)))
diagnose("decision tree, no limits", DecisionTreeClassifier(random_state=0))

logistic regression          train 0.791 | validation 0.790 | gap 0.001
decision tree, no limits     train 1.000 | validation 0.816 | gap 0.184

The two models show the two failure patterns:

• Underfitting: the logistic regression scores about 79% on both. No gap, but both scores are low. It cannot represent the pattern, so it does equally poorly everywhere.
• Overfitting: the unrestricted tree is perfect on its training data and 18 points worse on new data. It has memorised the training rows, noise included.

To judge "low", compare with a stronger model on the same data. As shown further down, other models reach about 88% validation accuracy here, so 79% leaves a lot on the table.

Find the balance with a validation curve

Model complexity controls the trade-off. For a decision tree, maximum depth is the simplest control:

for depth in (2, 4, 6, 8, 10, 15):
    diagnose(f"tree, max depth {depth}", DecisionTreeClassifier(max_depth=depth, random_state=0))

tree, max depth 2            train 0.794 | validation 0.792 | gap 0.003
tree, max depth 4            train 0.824 | validation 0.815 | gap 0.008
tree, max depth 6            train 0.854 | validation 0.836 | gap 0.018
tree, max depth 8            train 0.882 | validation 0.845 | gap 0.036
tree, max depth 10           train 0.909 | validation 0.845 | gap 0.064
tree, max depth 15           train 0.961 | validation 0.837 | gap 0.124

Read it from top to bottom. Shallow trees underfit: train and validation are close and low. As depth grows, both rise until depth 8 to 10, where validation peaks at 0.845. Beyond that, training accuracy keeps climbing while validation falls: the extra depth is fitting noise. This scan of a complexity setting is a validation curve, the standard tool for finding the balance point (scikit-learn: validation and learning curves). The bias-variance trade-off explains the theory behind the shape.

Fixing overfitting: what each remedy does

diagnose("tree, min 50 per leaf", DecisionTreeClassifier(min_samples_leaf=50, random_state=0))
diagnose("tree, cost-complexity pruned", DecisionTreeClassifier(ccp_alpha=0.0005, random_state=0))
diagnose("random forest", RandomForestClassifier(n_estimators=200, random_state=0, n_jobs=-1))
diagnose("boosting, 1000 rounds", HistGradientBoostingClassifier(max_iter=1000, early_stopping=False, random_state=0))
diagnose("boosting, early stopping", HistGradientBoostingClassifier(max_iter=1000, early_stopping=True, random_state=0))

tree, min 50 per leaf        train 0.862 | validation 0.841 | gap 0.021
tree, cost-complexity pruned train 0.871 | validation 0.847 | gap 0.025
random forest                train 1.000 | validation 0.880 | gap 0.120
boosting, 1000 rounds        train 1.000 | validation 0.886 | gap 0.114
boosting, early stopping     train 0.929 | validation 0.882 | gap 0.047

• Constraining the model, by requiring at least 50 samples per leaf or by cost-complexity pruning, which removes branches that add little (scikit-learn: decision trees), turns the memorising tree into one that generalises: validation 0.841 and 0.847, gaps of about 2 points.
• Averaging many trees in a random forest lifts validation to 0.880 without constraining the trees at all.
• Boosting reaches 0.886.

Notice something important in the last three lines. The random forest and 1,000 rounds of boosting both score 100% on their training data, the textbook sign of overfitting, yet they are the best models on validation data. A large gap is a warning to investigate, not a verdict. What matters is the validation score, and whether a change raises or lowers it. Early stopping, which halts boosting when an internal validation score stops improving (scikit-learn: HistGradientBoostingClassifier), halves the gap and uses far fewer rounds, for almost the same validation accuracy. That makes it cheaper to train and serve, not more accurate.

Would more data help? Read the learning curve

A learning curve plots scores against training set size:

sizes, train_scores, valid_scores = learning_curve(
    DecisionTreeClassifier(max_depth=10, random_state=0), X, y, cv=cv,
    train_sizes=[0.05, 0.2, 0.5, 1.0], shuffle=True, random_state=0)
for n, t, v in zip(sizes, train_scores.mean(axis=1), valid_scores.mean(axis=1)):
    print(f"{n:>6} training rows: train {t:.3f} | validation {v:.3f}")

   760 training rows: train 0.983 | validation 0.788
  3043 training rows: train 0.945 | validation 0.820
  7608 training rows: train 0.924 | validation 0.840
 15216 training rows: train 0.909 | validation 0.845

With little data, the tree memorises its 760 rows and generalises poorly. As data grows, training accuracy falls and validation rises: the gap closes. But the validation gain from doubling the data, from 7,608 to 15,216 rows, is only half a point, so more of the same data would help this model only slightly. A better model, as the forest and boosting showed, helps far more.

One trap to know: this dataset is stored sorted by class. Without shuffle=True, the learning curve's smaller training sets contain only one class and every score is meaningless. Always check how your data is ordered before taking subsets.

Fixing underfitting: give the model more capacity

The logistic regression underfit because it can only draw a straight boundary. Adding squared and interaction terms of the features lets it draw curved ones:

for degree in (2, 3):
    diagnose(f"logistic, degree-{degree} features", make_pipeline(
        StandardScaler(), PolynomialFeatures(degree), StandardScaler(), LogisticRegression(max_iter=5000)))

logistic, degree-2 features  train 0.862 | validation 0.861 | gap 0.001
logistic, degree-3 features  train 0.877 | validation 0.870 | gap 0.007

Validation rises from 0.790 to 0.870 while the gap stays tiny: the classic cure for underfitting. Other routes to more capacity are better features, a more flexible model such as trees, or less regularisation. Feature engineering shows how domain features do the same job.

A diagnosis table

Mistakes that cause hidden overfitting

• Tuning on the test set. Every decision made by looking at test scores leaks into the model. Keep one test set untouched until the end; cross-validation is for the choices before it.
• Preprocessing before splitting. Fitting a scaler or encoder on all the data lets test information into training. Put preprocessing inside a pipeline.
• Leaked features. A column that encodes the answer produces great validation scores and a useless model.
• Trying hundreds of models and reporting the best validation score without a final check on fresh data.

Overfitting and underfitting, train and test splits and cross-validation are part of the model evaluation and optimisation module of the Data Science course.

Sources

• MAGIC Gamma Telescope (UCI Machine Learning Repository) — UCI Machine Learning Repository
• Validation curves and learning curves (scikit-learn) — scikit-learn documentation
• Decision trees: tips on practical use and minimal cost-complexity pruning (scikit-learn) — scikit-learn documentation
• HistGradientBoostingClassifier (scikit-learn) — scikit-learn documentation