Evaluation Metrics: Accuracy, Precision, Recall and F1 Demystified

A model's score is only as useful as the metric behind it. The same classifier can look excellent by one measure and useless by another, and choosing the wrong one leads teams to ship models that fail at their real job. This guide works through the main classification metrics on a real problem, predicting diabetes from routine measurements, and ends with a practical way to choose.

The four core metrics come from counting four kinds of outcome: true positives, false positives, false negatives and true negatives. If those are new, the confusion matrix explained shows how to read them; here the focus is on what each metric tells you and when to trust it.

The four metrics in one sentence each

• Accuracy: the share of all predictions that were right.
• Precision: of the cases the model flagged as positive, the share that really were.
• Recall: of the cases that really were positive, the share the model caught.
• F1: a single score that is high only when precision and recall are both high.

Precision is about trusting the alarms; recall is about not missing cases. Improving one usually costs the other, which is why they are reported together (scikit-learn: metrics and scoring).

Why F1 uses the harmonic mean

F1 is the harmonic mean of precision and recall, not the simple average. The difference matters when one of them is poor:

precision, recall = 0.9, 0.1
print("arithmetic mean:", (precision + recall) / 2)
print("F1 (harmonic mean):", round(2 * precision * recall / (precision + recall), 2))

arithmetic mean: 0.5
F1 (harmonic mean): 0.18

A model that catches only one positive in ten is not half good, and F1 says so. The harmonic mean is pulled towards the smaller value, so a model cannot score well by maximising one metric and neglecting the other.

A real example: screening for diabetes

The Pima Indians diabetes dataset records eight measurements, such as glucose, body mass index and age, for 768 patients, 35% of whom tested positive (OpenML: diabetes). Compare a do-nothing model that always predicts "negative" with a logistic regression:

from sklearn.datasets import fetch_openml
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (accuracy_score, average_precision_score, f1_score, fbeta_score,
                             precision_score, recall_score, roc_auc_score)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = fetch_openml("diabetes", version=1, as_frame=True)
X, y = data.data, (data.target == "tested_positive").astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)

models = {"always negative": DummyClassifier(strategy="most_frequent"),
          "logistic regression": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))}
for name, model in models.items():
    pred = model.fit(X_train, y_train).predict(X_test)
    print(f"{name}: accuracy {accuracy_score(y_test, pred):.3f}, "
          f"precision {precision_score(y_test, pred, zero_division=0):.3f}, "
          f"recall {recall_score(y_test, pred):.3f}, F1 {f1_score(y_test, pred):.3f}")

always negative: accuracy 0.649, precision 0.000, recall 0.000, F1 0.000
logistic regression: accuracy 0.775, precision 0.738, recall 0.556, F1 0.634

The do-nothing model reaches 65% accuracy while finding no one with diabetes. That is accuracy's blind spot: when one class is common, predicting it every time earns a high score. Always compare accuracy with this baseline, or better, use a metric that focuses on the class you care about.

The logistic regression's precision of 0.738 means about three in four patients it flags really have diabetes. Its recall of 0.556 means it misses nearly half of those who do. For a screening test, that miss rate is the number that matters.

The threshold decides the trade-off

A classifier outputs a probability, and a threshold, 0.5 by default, turns it into a yes or no. Lowering the threshold flags more patients, raising recall and lowering precision. F-beta generalises F1: with beta = 2, recall is treated as twice as important as precision, which suits screening, where a missed case is worse than a follow-up test:

model = models["logistic regression"]
proba = model.predict_proba(X_test)[:, 1]
for threshold in (0.5, 0.4, 0.3, 0.2):
    pred = (proba >= threshold).astype(int)
    print(f"threshold {threshold}: precision {precision_score(y_test, pred):.3f}, recall {recall_score(y_test, pred):.3f}, "
          f"F1 {f1_score(y_test, pred):.3f}, F2 {fbeta_score(y_test, pred, beta=2):.3f}")

threshold 0.5: precision 0.738, recall 0.556, F1 0.634, F2 0.584
threshold 0.4: precision 0.671, recall 0.679, F1 0.675, F2 0.677
threshold 0.3: precision 0.657, recall 0.827, F1 0.732, F2 0.786
threshold 0.2: precision 0.576, recall 0.938, F1 0.714, F2 0.833

At a threshold of 0.2, the model catches 94% of patients with diabetes, at the cost of more false alarms: precision falls to 0.576. F1 prefers 0.3; F2, which weights recall more, prefers 0.2. Neither is correct in general; the right choice depends on the cost of each mistake. Choose the threshold on validation data, not the test set, and report it with the metrics, because a precision or recall figure means nothing without the threshold that produced it.

Threshold-free metrics: ROC AUC and average precision

To compare models before choosing a threshold, use a metric computed from the probabilities across all thresholds:

print("ROC AUC:", round(roc_auc_score(y_test, proba), 3))
print("average precision (PR-AUC):", round(average_precision_score(y_test, proba), 3),
      "| positive rate:", round(y_test.mean(), 3))

ROC AUC: 0.871
average precision (PR-AUC): 0.767 | positive rate: 0.351

ROC AUC is the probability that a randomly chosen positive case gets a higher score than a randomly chosen negative one; a random model scores 0.5. Average precision summarises the precision-recall curve, and a random model scores the positive rate, here 0.351. When positives are rare, ROC AUC can look reassuringly high because the many easy negatives dominate it, while average precision stays focused on how well the rare class is found. Saito and Rehmsmeier showed this effect across simulated and real imbalanced datasets (PLOS ONE: precision-recall vs ROC). For fraud, defects or rare diseases, prefer average precision, and see handling imbalanced datasets for training techniques.

One caution about this dataset: some measurements, such as blood pressure and body mass index, are recorded as zero when missing. A real project would clean them before modelling; here they are left as they are so the focus stays on metrics.

Multiclass: macro, micro and weighted averages

With more than two classes, precision, recall and F1 are computed per class and then averaged, and the averaging method changes the story. Take 100 support tickets: 80 billing, 15 technical, 5 legal, and a model that handles billing well and legal badly:

y_true = ["billing"] * 80 + ["technical"] * 15 + ["legal"] * 5
y_pred = (["billing"] * 76 + ["technical"] * 4      # billing tickets
          + ["technical"] * 10 + ["billing"] * 5    # technical tickets
          + ["legal"] * 1 + ["billing"] * 4)        # legal tickets
print("per class:", dict(zip(["billing", "legal", "technical"], f1_score(y_true, y_pred, average=None).round(3))))
for average in ("micro", "macro", "weighted"):
    print(f"{average} F1: {f1_score(y_true, y_pred, average=average):.3f}")

per class: {'billing': 0.921, 'legal': 0.333, 'technical': 0.69}
micro F1: 0.870
macro F1: 0.648
weighted F1: 0.857

• Micro pools every prediction together; with one label per item, it equals accuracy, and the big billing class dominates it.
• Macro averages the per-class scores equally, so the weak legal class pulls it down to 0.648. Use it when every class matters, however small.
• Weighted averages per-class scores by class size, so it sits close to micro.

Reporting only micro or weighted F1 here would hide that the model gets four of five legal tickets wrong. Report the per-class scores alongside any average (scikit-learn: metrics and scoring).

A note on regression metrics

For predicting numbers, the main choices are the mean absolute error (MAE) and the root mean squared error (RMSE). RMSE squares errors before averaging, so one large miss moves it far more:

from sklearn.metrics import mean_absolute_error, root_mean_squared_error

actual = [100, 100, 100, 100, 100]
predicted = [98, 102, 97, 103, 150]
print("MAE:", mean_absolute_error(actual, predicted), "| RMSE:", round(root_mean_squared_error(actual, predicted), 2))

MAE: 12.0 | RMSE: 22.48

Use MAE when every unit of error costs the same and you want a figure people can read directly. Use RMSE when large errors are disproportionately harmful. An end-to-end machine learning project shows MAE used to frame and evaluate a whole regression project.

How to choose

Whatever you choose, compute it with cross-validation rather than on one split, since a single test set can flatter or punish a model by chance; cross-validation in machine learning shows how, and scikit-learn accepts names such as "f1", "average_precision" and "recall" as its scoring argument.

Evaluation metrics, cross-validation and hyperparameter tuning together make up the model evaluation and optimisation module of the Data Science course.

Sources

• Metrics and scoring: quantifying the quality of predictions (scikit-learn) — scikit-learn documentation
• diabetes (OpenML dataset 37) — www.openml.org
• The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets — PLOS ONE, 2015