Cross-Validation in Machine Learning: A Practical Guide

A single train/test split gives you one number, and that number depends on which rows happened to land in the test set. Cross-validation replaces it with several scores from different splits. That tells you both how well the model does and how much the estimate moves. It only works if every split is honest: no information from a test fold may reach the model trained for it. This guide covers what scikit-learn does by default, how to choose a splitter for your data, and the leakage mistake that makes cross-validation scores meaningless.

What cross_val_score does by default

Call cross_val_score(model, X, y) without a cv argument and you get 5-fold cross-validation. For a classifier with a binary or multiclass target, the folds are stratified: each fold keeps roughly the class proportions of the whole dataset. For everything else, including regressors, the folds are plain contiguous K-fold splits (scikit-learn: cross_val_score).

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)          # 569 samples, 30 features

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

scores = cross_val_score(model, X, y, cv=cv, scoring="roc_auc")
print(scores.round(3), scores.mean().round(3), scores.std().round(3))

Report the mean and the spread. Two models whose mean scores differ by less than the fold-to-fold standard deviation are not reliably different on this data.

Choosing a splitter for your data

K-fold and shuffled splits assume every row is independent of every other and drawn from the same distribution (scikit-learn: cross-validation). Real datasets often break that in one of two ways, and each has its own splitter.

Grouped data

Suppose each customer contributes several transactions. With plain K-fold, one customer’s transactions end up in both training and test folds. The model can recognise the customer instead of learning the pattern, and the score overstates how it will do on customers it has never seen. GroupKFold keeps each group in exactly one test fold:

import numpy as np
from sklearn.model_selection import GroupKFold

groups = np.array(["c1", "c1", "c2", "c2", "c3", "c3", "c4", "c4"])   # customer id per row
X = np.arange(8).reshape(-1, 1)
y = np.array([0, 1, 0, 1, 0, 1, 0, 1])

for train_idx, test_idx in GroupKFold(n_splits=4).split(X, y, groups):
    print("test customers:", sorted(set(groups[test_idx])))
# each fold tests one whole customer: c4, c3, c2, c1

Time-ordered data

If you forecast next month from past months, a random split trains on the future and tests on the past, and the score will not hold up once the model is deployed. TimeSeriesSplit always tests on rows that come after the training rows, and each training set contains the ones before it (scikit-learn: cross-validation):

from sklearn.model_selection import TimeSeriesSplit

for train_idx, test_idx in TimeSeriesSplit(n_splits=3).split(np.arange(12)):
    print("train", train_idx, "test", test_idx)
# train [0 1 2]                 test [3 4 5]
# train [0 1 2 3 4 5]           test [6 7 8]
# train [0 1 2 3 4 5 6 7 8]     test [ 9 10 11]

The rows must already be sorted by time. For forecasting-specific modelling, see time series forecasting: a beginner-friendly introduction.

The leakage mistake: preprocessing before splitting

scikit-learn defines data leakage as using information at training time that would not be available at prediction time (scikit-learn: common pitfalls). The most common form in cross-validation is fitting a preprocessing step (scaling, imputation, feature selection, target encoding) on the full dataset and then cross-validating only the model. Each training fold has then already "seen" its test fold through the preprocessing.

The effect is easy to demonstrate. The data below is pure noise: 5,000 random features and labels with no relationship to them. An honest evaluation should score about 50%, a coin flip.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(42)
X = rng.normal(size=(200, 5000))       # 5,000 features of noise
y = rng.integers(0, 2, size=200)        # labels unrelated to X

# Leaky: pick the 20 "best" features using ALL rows, then cross-validate
X_selected = SelectKBest(f_classif, k=20).fit_transform(X, y)
leaky = cross_val_score(LogisticRegression(max_iter=1000), X_selected, y, cv=5)

# Honest: selection is part of the model, refit inside every training fold
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=1000))
honest = cross_val_score(pipe, X, y, cv=5)

print(leaky.mean().round(3), honest.mean().round(3))   # 0.845 0.545

With scikit-learn 1.8 and this seed, the leaky version reports 84.5% accuracy on data that contains no signal at all; the pipeline reports 54.5%, which is chance. Exact numbers vary with the seed and library version, but the gap does not. The fix is always the same: put every step that learns from data into a Pipeline and cross-validate the pipeline, so each step is refit on the training fold only.

• StandardScaler, SimpleImputer, encoders, SelectKBest, PCA: inside the pipeline.
• Deduplication, fixing data types, dropping columns you decided on before looking at the target: fine to do up front.
• Anything computed from the target (y), such as target encoding or oversampling: always inside the fold, never before.

Cross-validation and hyperparameter tuning

When you use cross-validation scores to choose hyperparameters, those scores become optimistic: you picked the settings that happened to do best on those folds. The scikit-learn guide warns that tuning can overfit the evaluation data in this way, and recommends keeping a test set for the final evaluation (scikit-learn: cross-validation). In practice: split off a test set first, run GridSearchCV or RandomizedSearchCV on the rest, and score the chosen model on the test set once. Grid search vs random search vs Bayesian optimisation compares the search strategies.

A checklist before you trust a CV score

1. Is a test set held out and untouched until the end?
2. Does the splitter match the data: stratified for imbalanced classes, grouped for repeated entities, time-ordered for forecasting?
3. Is shuffling set deliberately, with a random_state so results are reproducible?
4. Is every learned preprocessing step inside the pipeline?
5. Is the metric the one that matters for the problem? For rare positives, accuracy is rarely it; see accuracy, precision, recall and F1.
6. Are you reporting the spread across folds, not just the mean?

Where to go next

Cross-validation measures overfitting; overfitting and underfitting: causes and fixes covers what to do about it. Model evaluation, cross-validation and tuning are part of the model evaluation module of our Data Science course.

Sources

• Cross-validation: evaluating estimator performance — scikit-learn user guide
• sklearn.model_selection.cross_val_score — scikit-learn API reference
• Common pitfalls and recommended practices — scikit-learn user guide