Handling Imbalanced Datasets: Techniques That Work

When one class is rare, a classifier can score well by mostly ignoring it. Fraud, equipment failures, rare diseases and customer churn all look like this. The usual advice is to rebalance the data with oversampling, undersampling, SMOTE or class weights. Those techniques do something real, but it is often not what people expect, and there are simpler options that can work better.

This guide measures the techniques on a real problem: 11,183 mammography cases described by six numeric features, of which only 260, or 2.3%, are positive (OpenML: mammography). Every result uses 5-fold cross-validated predictions, so each case is scored by a model that never saw it.

The data and the evaluation

import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, precision_recall_curve, precision_score, recall_score
from sklearn.model_selection import StratifiedKFold, cross_val_predict
from sklearn.preprocessing import StandardScaler
from imblearn.over_sampling import SMOTE, RandomOverSampler
from imblearn.pipeline import make_pipeline
from imblearn.under_sampling import RandomUnderSampler

data = fetch_openml(data_id=310, as_frame=True)
X, y = data.data.to_numpy(float), (data.target == "1").astype(int).to_numpy()
print(X.shape, "| positives:", y.sum(), f"({y.mean():.1%})")

(11183, 6) | positives: 260 (2.3%)

Two measurements are reported for each model. Average precision (AP) summarises the whole precision-recall curve: how well the model ranks positive cases above negative ones, independent of any threshold. Then, at the default threshold of 0.5, precision (the share of flagged cases that are real), recall (the share of real cases flagged) and the number flagged. Accuracy is useless here: predicting "negative" for everyone scores 97.7%. Evaluation metrics explained covers why.

What class weights and resampling actually change

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

def evaluate(model):
    proba = cross_val_predict(model, X, y, cv=cv, method="predict_proba")[:, 1]
    flagged = proba >= 0.5
    print(f"AP {average_precision_score(y, proba):.3f} | at 0.5: precision {precision_score(y, flagged, zero_division=0):.3f}, "
          f"recall {recall_score(y, flagged):.3f}, flagged {flagged.sum()}")
    return proba

def logistic(**options):
    return LogisticRegression(max_iter=1000, **options)

techniques = {
    "no treatment": make_pipeline(StandardScaler(), logistic()),
    "class weights": make_pipeline(StandardScaler(), logistic(class_weight="balanced")),
    "oversampling": make_pipeline(StandardScaler(), RandomOverSampler(random_state=0), logistic()),
    "SMOTE": make_pipeline(StandardScaler(), SMOTE(random_state=0), logistic()),
    "undersampling": make_pipeline(StandardScaler(), RandomUnderSampler(random_state=0), logistic()),
}
probas = {}
for name, model in techniques.items():
    print(f"{name:>13}:", end=" ")
    probas[name] = evaluate(model)

 no treatment: AP 0.615 | at 0.5: precision 0.784, recall 0.404, flagged 134
class weights: AP 0.534 | at 0.5: precision 0.166, recall 0.850, flagged 1330
 oversampling: AP 0.535 | at 0.5: precision 0.166, recall 0.850, flagged 1334
        SMOTE: AP 0.512 | at 0.5: precision 0.168, recall 0.846, flagged 1309
undersampling: AP 0.526 | at 0.5: precision 0.170, recall 0.838, flagged 1285

The untreated model is cautious: it flags 134 cases, 78% of them correctly, but finds only 40% of the positives. Each technique works on the training data differently:

• Class weights make each positive case count more in the loss.
• Random oversampling repeats positive cases.
• SMOTE creates synthetic positive cases along lines between neighbouring real ones (Chawla et al.: SMOTE).
• Random undersampling discards negative cases.

All four do nearly the same thing. They push predicted probabilities for positives upward, so at the 0.5 threshold recall jumps to about 85% while precision collapses to about 17%, with roughly ten times as many cases flagged. And their average precision falls, from 0.615 to between 0.51 and 0.54: on this data, none of them made the model better at ranking cases. Mostly they moved the decision boundary.

The simpler alternative: move the threshold

If the goal is higher recall, you can keep the untreated model and lower the threshold at which a case is flagged (scikit-learn: tuning the decision threshold). Comparing every model at the same recall of 85%:

def at_recall(proba, target=0.85):
    precision, recall, thresholds = precision_recall_curve(y, proba)
    i = np.where(recall[:-1] >= target)[0][-1]
    return f"threshold {thresholds[i]:.3f}, precision {precision[i]:.3f}, flagged {(proba >= thresholds[i]).sum()}"

for name in ("no treatment", "class weights", "SMOTE"):
    print(f"{name:>13} at recall 0.85:", at_recall(probas[name]))

 no treatment at recall 0.85: threshold 0.039, precision 0.219, flagged 1007
class weights at recall 0.85: threshold 0.500, precision 0.166, flagged 1328
        SMOTE at recall 0.85: threshold 0.489, precision 0.162, flagged 1368

To catch 85% of positives, the untreated model with a threshold of 0.039 flags 1,007 cases. The class-weighted and SMOTE models flag over 1,300 for the same catch rate: about 320 to 360 more false alarms for reviewers to clear. Threshold moving also leaves the model's probabilities meaningful, whereas rebalanced training inflates them, so a "50% risk" from a rebalanced model does not mean a one-in-two chance.

One caution: here the thresholds are read off the same out-of-fold predictions used for scoring, which is slightly optimistic for every model alike. In a real project, choose the threshold on validation data and confirm it on a held-out test set; scikit-learn's TunedThresholdClassifierCV automates that with cross-validation.

A better model beats a rebalanced one

print("gradient boosting:", end=" ")
boosted = evaluate(HistGradientBoostingClassifier(random_state=0))
print("gradient boosting at recall 0.85:", at_recall(boosted))

gradient boosting: AP 0.728 | at 0.5: precision 0.807, recall 0.581, flagged 187
gradient boosting at recall 0.85: threshold 0.004, precision 0.301, flagged 735

Gradient-boosted trees rank cases better, lifting average precision to 0.728, and at 85% recall they flag 735 cases instead of 1,007. No resampling technique came close to that improvement. When a rare class is hard to find, a better model and better features usually do more than rebalancing the data.

The mistake that fakes a great result

Resampling must happen inside cross-validation, on training folds only. The imblearn pipeline used above does exactly that. Here is what happens if SMOTE is applied to the whole dataset first:

X_wrong, y_wrong = SMOTE(random_state=0).fit_resample(X, y)
proba = cross_val_predict(make_pipeline(StandardScaler(), logistic()), X_wrong, y_wrong, cv=cv, method="predict_proba")[:, 1]
print("SMOTE before splitting, AP:", round(average_precision_score(y_wrong, proba), 3), "| rows:", len(y_wrong))

SMOTE before splitting, AP: 0.944 | rows: 21846

An average precision of 0.944, far above anything real. Two things went wrong. The test folds now contain synthetic points built from training points, so the model is tested on near-copies of what it learned. And the test data is balanced, half positive, so it no longer resembles reality, where 2.3% of cases are positive. The imbalanced-learn documentation warns against exactly this (imbalanced-learn: common pitfalls). Always evaluate on data with the real class balance.

When rebalancing is worth trying

The results above are for one dataset and are not a universal rule. Resampling and class weights can help when:

• The model or library has no usable probability output, so you cannot move a threshold.
• The training algorithm struggles to learn anything about the rare class at all, for example with very few positives per batch in neural network training.
• A fixed decision rule is imposed downstream and cannot be changed.

Treat them as options to test with the same cross-validation, not as a required step.

A practical order of work

1. Measure with the right metrics: average precision and precision or recall at a chosen operating point, never accuracy alone.
2. Use stratified splits so every fold contains positives.
3. Try a stronger model and better features before anything else.
4. Choose the threshold from the costs of false alarms and misses, on validation data.
5. Test class weights or resampling inside a pipeline, and keep them only if they improve the metric that matters.
6. Get more positive examples if you can; nothing helps as reliably as more real data about the rare class.

For extremely rare events with few labels, framing the problem as anomaly detection can be the better route.

The confusion matrix explained shows how moving the threshold reshapes the four kinds of outcome. Evaluation metrics and techniques for model evaluation are part of the model evaluation and optimisation module of the Data Science course.

Sources

• mammography (OpenML dataset 310) — www.openml.org
• SMOTE: Synthetic Minority Over-sampling Technique — Journal of Artificial Intelligence Research, 2002
• Common pitfalls and recommended practices (imbalanced-learn) — imbalanced-learn.org
• Tuning the decision threshold for class prediction (scikit-learn) — scikit-learn documentation