Regression vs Classification: When to Use Each

Every supervised machine learning problem starts with a framing decision. Predicting a number, such as a price, a demand or a delivery time, is regression. Predicting a category, such as spam or not, churn or stay, or which product line, is classification. Most algorithms come in both versions; scikit-learn, for example, provides a classifier and a regressor for most of its model families (scikit-learn: supervised learning). The definitions are simple. The judgement lies in the grey areas, where a target could be treated either way, and in matching the framing to the decision the model supports.

The quick rule

Classification models usually output a probability for each class, and a threshold turns that into a label (Google: classification). Regression models output a number directly.

A target that can go either way

Wine quality is a good test case. The UCI repository's description of the wine quality data says it can be viewed as either a classification or a regression task, and notes that the classes are ordered and unbalanced (UCI: wine quality). Here are the 1,599 red wines, each scored by tasters, with eleven chemical measurements (OpenML: wine-quality-red):

import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.ensemble import HistGradientBoostingClassifier, HistGradientBoostingRegressor
from sklearn.metrics import accuracy_score, average_precision_score, mean_absolute_error, precision_score, recall_score
from sklearn.model_selection import StratifiedKFold, cross_val_predict

wine = fetch_openml(data_id=40691, as_frame=True)
X, quality = wine.data, wine.target.astype(int)
print(len(quality), "wines, quality counts:", quality.value_counts().sort_index().to_dict())

1599 wines, quality counts: {3: 10, 4: 53, 5: 681, 6: 638, 7: 199, 8: 18}

Most wines score 5 or 6; grades 3, 4 and 8 are rare. Now model it both ways with the same algorithm family, gradient-boosted trees, and score both on the same terms:

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
as_number = cross_val_predict(HistGradientBoostingRegressor(random_state=0), X, quality, cv=cv)
as_rounded = np.clip(np.round(as_number), 3, 8)
as_class = cross_val_predict(HistGradientBoostingClassifier(random_state=0), X, quality, cv=cv)

print(f"always predict 5:        MAE {mean_absolute_error(quality, np.full(len(quality), 5)):.3f}, accuracy {(quality == 5).mean():.3f}")
print(f"regression, rounded:     MAE {mean_absolute_error(quality, as_rounded):.3f}, accuracy {accuracy_score(quality, as_rounded):.3f}")
print(f"classification (6 grades): MAE {mean_absolute_error(quality, as_class):.3f}, accuracy {accuracy_score(quality, as_class):.3f}")
for grade in (3, 4, 8):
    mask = quality == grade
    print(f"grade {grade} ({mask.sum()} wines): regression right {(as_rounded[mask] == grade).mean():.0%}, "
          f"classification right {(as_class[mask] == grade).mean():.0%}")
print(f"off by two grades or more: regression {(np.abs(quality - as_rounded) >= 2).sum()}, "
      f"classification {(np.abs(quality - as_class) >= 2).sum()}")

always predict 5:        MAE 0.727, accuracy 0.426
regression, rounded:     MAE 0.347, accuracy 0.684
classification (6 grades): MAE 0.353, accuracy 0.679
grade 3 (10 wines): regression right 0%, classification right 0%
grade 4 (53 wines): regression right 6%, classification right 0%
grade 8 (18 wines): regression right 0%, classification right 11%
off by two grades or more: regression 48, classification 50

On overall accuracy and error, the two framings are effectively tied, and both beat the do-nothing baseline by a wide margin. The differences are in the details. In principle, regression uses the order of the grades, while classification treats 3 and 8 as unrelated labels; here, though, both framings make large mistakes on almost the same number of wines, 48 and 50 off by two grades or more. Both almost never predict the rare extreme grades, the ones a wine buyer might care about most. The honest conclusion for ordered targets: try both, and judge them on the errors that matter to the user of the model.

When the decision is yes or no, model the decision

Suppose the real question is simpler: which wines are good, scoring 7 or more? You could reuse the regression and cut its prediction at 6.5, or train a classifier directly on the yes-or-no target:

good = (quality >= 7).astype(int)
p_good = cross_val_predict(HistGradientBoostingClassifier(random_state=0), X, good, cv=cv, method="predict_proba")[:, 1]
print(f"'good' wines (7 or more): {good.mean():.1%}")
print(f"classifier trained on the decision: average precision {average_precision_score(good, p_good):.3f}, "
      f"precision {precision_score(good, p_good >= 0.5):.3f}, recall {recall_score(good, p_good >= 0.5):.3f}")
print(f"regression score, cut at 6.5:       average precision {average_precision_score(good, as_number):.3f}, "
      f"precision {precision_score(good, as_number >= 6.5):.3f}, recall {recall_score(good, as_number >= 6.5):.3f}")

'good' wines (7 or more): 13.6%
classifier trained on the decision: average precision 0.749, precision 0.719, recall 0.590
regression score, cut at 6.5:       average precision 0.644, precision 0.684, recall 0.558

The dedicated classifier ranks good wines better, with average precision 0.749 against 0.644, and wins on precision and recall at its default threshold too. The regression spent its effort getting the whole scale right, including the difference between a 4 and a 5, which does not matter for this decision. When the business decision is binary, training directly for it usually pays off.

The reverse also holds. Turning a meaningful number into categories, such as splitting delivery time into "late" and "on time", throws away information you may later need. If different users need different cut-offs, predict the number and let each apply its own threshold.

Questions that settle the framing

1. What action will the prediction drive? A yes-or-no action suggests classification; a quantity to plan for, such as stock to order, suggests regression.
2. Are the differences between values meaningful? If a 4 is closer to a 5 than to a 1, the target has order that regression can use.
3. Which errors are costly? Being off by a little on every case, or missing rare but important cases? The answer chooses the metric, and often the framing.
4. Is a probability needed? Classifiers give probabilities of events, useful for expected-value decisions and risk scores.
5. How much data is there per category? Classes with only a handful of examples, like grade 3 here, will rarely be predicted by either framing without more data.

Common mistakes

• Regressing on arbitrary codes. Categories stored as numbers, such as product type 1, 2 and 3, have no order. Treating them as a quantity invents one.
• Judging a classifier by accuracy alone when one class dominates. Evaluation metrics explained covers what to use instead.
• Rounding regression output without checking the ends. Predictions often cluster in the middle, as they did here, so extreme values are rarely predicted.
• Fixing the framing too early. For ordered targets, a quick test of both framings with cross-validation costs little and removes guesswork.

For the algorithms that implement both framings, see the top 10 machine learning algorithms. Supervised learning with regression and classification models is part of the machine learning module of the Data Science course.

Sources

• Wine Quality (UCI Machine Learning Repository) — UCI Machine Learning Repository
• wine-quality-red (OpenML dataset 40691) — www.openml.org
• Supervised learning (scikit-learn) — scikit-learn documentation
• Classification (Google Machine Learning Crash Course) — Google for Developers